Neural Networks with Approximated Activation Functions

Abstract

The present disclosure relates to a neural network including at least one neural network layer and a specific type of activation function connected to the output of the at least one neural network layer, where the activation function is implemented as an approximation function of a mathematically defined real-valued nonlinear activation function, the approximation function allowing processing only integers of a fixed-point representation of the input values ​​of the approximation function.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of data-based encoding and decoding in neural network architectures. In particular, some embodiments relate to methods and apparatus for such encoding and decoding of images and / or videos from bitstreams using multiple processing layers. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded block-by-block by predicting the block and further coding only the difference between the original block and its prediction. In particular, such coding may include transforming, quantizing, and generating a bitstream, and typically includes some entropy coding. Typically, the three components of a hybrid coding method, namely transforming, quantizing, and entropy coding, are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use a transformed representation to code the residual signal after prediction.

[0003] In recent years, neural network architectures have been applied to image and / or video coding. In general, these neural network (NN)-based approaches can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize some parts of the end-to-end coding framework, such as prediction parameter selection or compression. Moreover, several neural network-based approaches have also been discussed for use in hybrid image and video coding frameworks, for example, for implementation as trained deep learning models for intra- or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications discussed above have in common that they generate some feature map data to be communicated between an encoder and a decoder.

[0005] A neural network is a machine learning model that employs one or more layers of nonlinear units that can predict an output for a received input based on it. Some neural networks include one or more hidden layers in addition to an output layer. As an output of each hidden layer, a corresponding feature map can be provided. Such corresponding feature map of each hidden layer can be used as an input to a subsequent layer in the network, i.e., a subsequent hidden layer or an output layer. Each layer of the network generates an output from a received input according to the current value of a respective parameter set. In a neural network that is split between different devices, for example, between an encoder and a decoder, or between a device and a cloud, the feature map at the output of the location of the split (e.g., a first device) is compressed and transmitted to the remaining layers of the neural network (e.g., to a second device).

[0006] Further improvements in encoding and decoding using trained network architectures may be desirable. Summary of the Invention [Means for solving the problem]

[0007] The present disclosure provides a method and apparatus for improving device interoperability of devices / platforms of different architectures, including neural networks. Device interoperability means that the execution of the same procedure with the same input data on two devices / platforms gives the same result for both devices / platforms. In particular, in the context of entropy model-based coding and / or data, such as image data compression and decompression, it is an important issue to provide substantially bit-exact processing results on the encoding side and the decoding side, respectively, to provide the same or complementary technical effect.

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0009] Particular implementations are outlined in the accompanying independent claims, while other implementations are defined in the dependent claims.

[0010] To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear and relatively simple nonlinear activation functions, such as the ReLU function that essentially defines the clipping procedure, this requirement can be met relatively easily. For more complex nonlinearities, especially

number

[0011] Thus, according to a first aspect, there is provided a neural network comprising at least one neural network layer and an activation function connected to an output of the at least one neural network layer, the activation function being implemented as an approximation function of a mathematically defined real-valued non-linear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing processing only integers of a fixed-point representation of input values ​​of the approximation function.

[0012] In fixed-point representation / arithmetic, real numbers are mantissa * base 指数 where both the base and exponent are fixed, and thus a fixed-point representation of a fraction is essentially an integer (see also the detailed explanation below).

[0013] The approximation function approximates a mathematically defined real-valued nonlinear activation function used in the art (which often includes one or more exponential functions (with base e) and which processes floating-point representations of the input values) in a way that allows processing only integers in a fixed-point representation of the input values ​​(e.g., like a ReLU function).

[0014] Integer-only processing allows fixed-point neural networks in that bit-exact behavior on different platforms can be obtained. This allows interoperability between different platforms / devices to be significantly improved compared to that of the state of the art. The provision of such approximated (approximate) activation functions may allow bit-exact reproduction of important numerical operations at the encoder and decoder sides, respectively, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, if bit-exact behavior can be obtained at both sides, a region of an image (a still image or a frame of a video sequence) can be (entropy) encoded at the encoder side and reconstructed without corruption at the decoder side. Furthermore, the internal operation (state) of the involved devices is defined in a substantially same way by providing neural networks including the approximated activation functions according to the first aspect for different devices that communicate with each other. The approximated activation functions implemented in the neural network of the first aspect can be selected in different ways as described below, although the present disclosure is not limited to one of these specific ways.

[0015] According to one implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation of the neural network of the first aspect, the mathematically defined nonlinear activation function (approximated by the approximated activation function) is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function. For all of these mathematically defined nonlinear activation functions that may be suitable for practical applications, an approximation function can be found that allows processing only integers of a fixed-point representation of input values.

[0016] According to one implementation, the approximation function (approximated activation function) includes a finite number of Taylor series summands determined based on at least one of: a) expected input values ​​of data input to at least one neural network layer or approximation function, b) accumulator register sizes of accumulator registers included in the neural network for buffering the summation result, and c) the approximation function.

[0017] According to another implementation, the approximation function is a polynomial function, the maximum degree of which is determined based on at least one of: a) expected input values ​​of data input to at least one neural network layer or approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering the summation result, and c) the approximation function.

[0018] According to another implementation, the approximation function is a rational function, and the maximum degree of the polynomials in the numerator and denominator of the rational function is determined based on at least one of: a) expected input values ​​of data input to at least one neural network layer or approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering the summation result, and c) the approximation function.

[0019] Taylor series of a mathematically defined nonlinear activation function f(x)

number

number

[0020] 2 n-1 -1 and -2 n-1 Note that is an example of a restriction on accumulator register size only, and may be replaced in the condition by other restrictions that appropriately define the accumulator register size.

[0021] The activation functions that are particularly important in many applications are

number

[0022] According to an implementation of the neural network of the first aspect, the mathematically defined nonlinear activation function is a Softmax function, and the approximation function (the approximated activation function) is

number

[0023] According to another implementation, the approximated activation function is

number

[0024] The positive constant ε can be easily selected depending on, for example, the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11 may be in the range.

[0025] Numerical experiments prove that these kinds of approximation functions based on the ReLU function for the Softmax function can ensure the bit-exact behavior of the application of activation functions on different platforms.

[0026] The implementation of the approximated activation function can advantageously be realized in any neural network, in particular the neural network may be a convolutional or fully connected neural network or a convolutional or fully connected neural network layer.

[0027] According to a second aspect, there is provided (with the same advantages as provided by the neural network of the first aspect) a method of operating a neural network (e.g. a convolutional or fully connected neural network) comprising at least one neural network layer, comprising the step of implementing, as an activation function of the at least one neural network layer, an approximation function of a mathematically defined real-valued non-linear activation function, wherein the approximation function allows processing only integers of a fixed-point representation of input values ​​of the approximation function.

[0028] According to one implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation, the mathematically defined nonlinear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function.

[0029] The approximation function may include a finite number of Taylor series summands determined based on expected input values ​​of the data input to at least one neural network layer or approximation function.

[0030] Taylor series of a mathematically defined nonlinear activation function f(x)

number

number

[0031] The mathematically defined non-linear activation function may be a Softmax function, in which case the approximation function implemented according to the method of the second aspect or an implementation thereof may be

number

[0032] Alternatively, the approximation function is

number

[0033] The positive constant ε can be easily selected depending on, for example, the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11 A second alternative, an approximated activation function that represents the Softmax function, is

number

[0034] The approximated activation functions implemented in the neural network of the method of the first and second aspects may be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as above. Thus, according to the third aspect, a method of encoding data is provided, comprising steps of the method of operating a neural network according to the first aspect or any implementation thereof (with the same advantages as above). In one implementation of the method of the third aspect, encoding the data comprises providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, the providing of the entropy model comprising performing steps of the method of operating a neural network according to the second aspect or any implementation thereof. The entropy encoding may comprise entropy encoding by an arithmetic encoder.

[0035] A substantially bit-exact reproduction of the entropy model used on both the encoder and decoder side is essential for reliable reconstruction of the (compressed) data in the context of entropy coding interoperability between different platforms.

[0036] The entropy model provides statistical (probability) properties of the symbols to be encoded or decoded, such as mean, variance, (cross-)correlation, etc., and it may be provided by one of: a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of hyper-prior distribution and autoregressive prior distribution of the variational autoencoder. Thus, the method of the third aspect can be advantageously implemented in a (variational) autoencoding system (see also the detailed description below).

[0037] According to a fourth aspect (and complementary to the third aspect), a method for decoding encoded data is provided, comprising steps of a method for operating a neural network according to the second aspect or any implementation thereof (with the same advantages as above). According to one implementation of the fourth aspect, the decoding of the data comprises providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, the entropy decoding of the data comprising steps of a method for operating a neural network according to the second aspect or any implementation thereof. The entropy decoding may comprise entropy decoding by an arithmetic decoder. Also on the decoder side, according to one implementation of the fourth aspect, the entropy model is provided by one of a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of the variational autoencoder.

[0038] The data processed by the method of the third or fourth aspect may be, for example, image data representing a still image or a frame of a video sequence.

[0039] According to a fifth aspect, a method for encoding at least a part of an image is provided, comprising the steps of converting tensors representing components of the image into latent tensors, providing an entropy model, and processing the latent tensors by a neural network to generate a bitstream based on the provided entropy model, where the step of providing the entropy model comprises performing the steps of the method according to the second aspect or any of its implementations (with the same advantages as above). The latent tensors may be processed by an arithmetic encoder. The processing of the latent tensors to obtain tensors representing components of the image may comprise performing the steps of the method according to the first aspect or any of its implementations.

[0040] The entropy model may be provided by one of: a) a hyper-prior distribution for the variational autoencoder, b) an autoregressive prior distribution for the variational autoencoder, and c) a combination of the hyper-prior distribution and the autoregressive prior distribution for the variational autoencoder.

[0041] According to a sixth aspect, a method for reconstructing at least a part of an image is provided, comprising the steps of providing an entropy model, processing a bit stream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image, and processing the latent tensors to obtain tensors representing components of the image, where the step of providing the entropy model comprises performing steps of a method for operating a neural network according to the second aspect or any implementation thereof (with the same advantages as above). The processing of the bit stream may be performed by an arithmetic decoder. The entropy model may be provided by one of: a) a hyper-prior distribution of a variational autoencoder, b) an autoregressive prior distribution of a variational autoencoder, and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder. Furthermore, the processing of the latent tensors to obtain tensors representing components of the image may comprise performing steps of a method according to the twenty-first aspect or an implementation thereof.

[0042] The components of the image represented by the tensors according to the methods of the fifth and sixth aspects may be Y, U or V components, or alternatively R, G, B components.

[0043] According to a seventh aspect, an approximated activation function (approximating a Softmax function)

number

[0044] According to an eighth aspect, an approximated activation function (approximating a Softmax function)

number

[0045] The positive constant ε can be easily selected depending on, for example, the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11 may be in the range.

[0046] These new activation functions can replace the traditional Softmax function and may prove advantageous in terms of device interoperability (as encoders and decoders).

[0047] According to a ninth aspect, there is provided an apparatus for encoding data, comprising a neural network according to the first aspect or any implementation thereof.

[0048] According to a tenth aspect, there is provided an apparatus for decoding data, comprising a neural network according to the first aspect or any implementation thereof.

[0049] According to an eleventh aspect, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network according to the first aspect or any implementation thereof.

[0050] According to a twelfth aspect, there is provided an apparatus for decoding at least a part of an image, comprising a neural network according to the first aspect or any implementation thereof.

[0051] According to a thirteenth aspect, there is provided an apparatus according to one of the twenty-second to twelfth aspects, comprising a hyperprior for a variational autoencoder, the hyperprior comprising a neural network according to the first aspect or any implementation thereof.

[0052] According to a fourteenth aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including a processing circuit configured to convert tensors representing components of the image into latent tensors, provide an entropy model by a neural network according to the first aspect or any implementation thereof, and process the latent tensors based on the provided entropy model to generate a bitstream.

[0053] According to a fifteenth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus including a processing circuit configured to provide an entropy model by a neural network according to the first aspect or any implementation thereof, process a bitstream based on the provided entropy model to obtain latent tensors representing components of the image, and process the latent tensors to obtain tensors representing components of the image.

[0054] The components of an image may be Y, U or V components or R, G, B components.

[0055] According to a sixteenth aspect, the present disclosure relates to a method for operating a neural network including at least one neural network layer including or connected to an accumulator register for buffering a summation result, the method comprising the steps of: defining lower and upper integer thresholds for values ​​of integers included in a data entity of input data of the at least one neural network layer, the method further comprising: clipping the values ​​of integers included in the data entity of the input data to the defined lower integer threshold if the values ​​of integers included in the data entity of the input data are smaller than the defined lower integer threshold; and clipping the values ​​of integers included in the data entity of the input data to the defined upper integer threshold if the values ​​of integers included in the data entity of the input data are larger than the defined upper integer threshold, such that an integer overflow of the accumulator register is avoided.

[0056] According to the method of the sixteenth aspect, due to clipping of integer-valued data (including integers) input to the neural network layer, interoperability between different platforms / devices can be significantly improved compared to the state of the art. Clipping may enable bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, an area of ​​an image (a still image or a frame of a video sequence) can be (entropy) encoded at the encoder side and reconstructed without corruption at the decoder side, if integer overflows, which may result in non-deterministic behavior on different platforms, can be reliably avoided. In general, there is no standardized handling of overflow situations, and it is hardly possible to imagine a procedure that is commonly followed in this regard. Integer overflow occurs when an arithmetic operation attempts to create a number outside the range that can be represented with a given number of digits, i.e., a number that is either higher than the maximum value or lower than the minimum representable value. This situation can be handled differently for different compilers, devices (CPUs, GPUs), etc. In order to achieve bit-exact results of integer arithmetic on different platforms, integer overflow should be avoided. According to the method of the 16th aspect, integer overflow of accumulator registers can be avoided on both the encoder side and the decoder side. Furthermore, the internal operation (state) of the involved devices is defined in substantially the same way by providing clipping of input data values ​​according to the method of the 16th aspect.

[0057] According to one implementation, the method of the sixteenth aspect further comprises a step of scaling the data entities of the input data (having real numbers or integers) by a first scaling factor (i.e. the data entities of the input data are multiplied by the first scaling factor) to obtain scaled values ​​of the data entities of the input data. The scaling may be performed to improve the handling of the input data values ​​on different devices. In particular, the scaling can be complemented by rounding the scaled values ​​of the data entities of the input data to the respective nearest integer values ​​to obtain integer values ​​of the data entities of the input data. The rounding can be performed by a floor function or a ceil function. Thereby, for example, real input data that may be provided initially can be processed by the disclosed method of operating a neural network.

[0058] The output data can be obtained by processing the input data by at least one neural network layer, and according to one implementation, once the first scaling factor has been applied, the output data entities included in the output data are divided by a third scaling factor to obtain an unscaled result. The unscaling may be performed directly on the output of the at least one neural network layer or after processing by the activation function. Thus, according to one implementation, the input data is processed by the at least one neural network layer to obtain output data including the output data entities, the output data entities are processed by the activation function to obtain an output of the activation function, and the output of the activation function is divided by the third scaling factor.

[0059] According to another implementation form, the method of the sixteenth aspect further comprises the steps of processing the input data by at least one neural network layer to obtain output data including an output data entity, factoring the third scaling factor into a first part and a second part, dividing the output data entity by the first part of the factored third scaling factor to obtain a partially unscaled output data entity, processing the partially unscaled output data entity by an activation function to obtain an output of the activation function, and dividing the output of the activation function by the second part of the factored third scaling factor. Since all options of unscaling may be implemented, the unscaling process can be performed with high flexibility.

[0060] The above-mentioned integer lower threshold used in the method of the sixteenth aspect may be less than or equal to 0, and the integer upper threshold may be greater than or equal to 0. Suitable possible choices for the thresholds employed are -2 for the integer lower threshold; k-1 , 2 for upper integer threshold k-1 -1, and k represents the predefined bit depth (bit size) of the input data.

[0061] The method of the sixteenth aspect can be advantageously applied to any kind of neural network and neural network layer. According to one implementation, at least one neural network layer is or includes one of a fully connected neural network layer and a convolutional neural network layer. Furthermore, at least one neural network layer may include an attention mechanism (see detailed description below).

[0062] According to one implementation, at least one neural network layer includes integer-valued weights, which can further improve platform / device interoperability and further reduce the risk of integer overflow of accumulator register sizes (see also below).

[0063] According to one implementation, the weights, real-valued weights (e.g., weights including only real numbers) or integer-valued weights, are originally provided and scaled by a second scaling factor to obtain scaled weights, and the scaled weights are rounded to the respective nearest integer value to obtain integer-valued weights of at least one neural network. This allows the method of the sixteenth aspect to be used for originally provided (e.g., user-dependent) real-valued weights.

[0064] The second scaling factor is

number

number

[0065] According to another implementation, the second scaling coefficient s for the j-th output channel of at least one neural network layer is jteeth,

number

[0066] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0067] According to another implementation, the condition is:

number

number

[0068] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0069] As already mentioned, the method of the 16th aspect can be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as above. In particular, data coding / decoding based on entropy coding represents an elaborate process where at least some interoperability of procedures performed on different devices gives the same results in numerical precision. Thus, according to the 17th aspect, a method of encoding data is provided, comprising steps of the method of operating a neural network according to the 16th aspect or any implementation thereof (with the same advantages as above). In one implementation of the method of the 17th aspect, encoding the data comprises providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, where providing the entropy model comprises performing steps of the method of operating a neural network according to the 16th aspect or any implementation thereof. The entropy encoding may comprise entropy encoding by an arithmetic encoder.

[0070] A substantially bit-exact reproduction of the entropy model used on both the encoder and decoder side is essential for reliable reconstruction of the (compressed) data in the context of entropy coding interoperability between different platforms.

[0071] The entropy model provides statistical (probability) properties of the symbols to be encoded or decoded, such as mean, variance, (cross-)correlation, etc., and it may be provided by one of: a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of hyper-prior distribution and autoregressive prior distribution of the variational autoencoder. Thus, the method of the seventeenth aspect can be advantageously implemented in a (variational) autoencoding system (see also the detailed description below).

[0072] The method of encoding data as described above may further include a step of signaling the defined lower threshold and the defined upper threshold to the decoder side. Thus, the decoder side can be easily provided with the necessary information for the same processing regarding the clipping procedure of the input data performed by the encoder side. Similarly, at least one of the first scaling factor, the second scaling factor and the third scaling factor may be signaled to the decoder side in the bitstream.

[0073] Further, the method may include signaling the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold to the decoder side. Further, the method may include signaling at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0074] According to an eighteenth aspect (and complementary to the seventeenth aspect), a method for decoding encoded data is provided, comprising steps of a method for operating a neural network according to the sixteenth aspect or any implementation thereof (with the same advantages as above). According to one implementation of the eighteenth aspect, the decoding of the data comprises providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, the entropy decoding of the data comprising steps of a method for operating a neural network according to the sixteenth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding by an arithmetic decoder. Also on the decoder side, according to one implementation of the eighteenth aspect, the entropy model is provided by one of a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of the variational autoencoder.

[0075] Information regarding the defined lower threshold and the defined upper threshold may be received at the decoder side in the bitstream from the encoder side. Similarly, information regarding at least one of the first scaling factor, the second scaling factor and the third scaling factor may be received at the decoder side in the bitstream from the encoder side.

[0076] Furthermore, the method may include receiving, at the decoder side, information regarding the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold.Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to a predefined first scaling factor and the difference relative to a predefined second scaling factor in the bitstream.

[0077] The data processed by the method of the seventeenth or eighteenth aspects may be, for example, image data representing a still image or a frame of a video sequence.

[0078] According to a nineteenth aspect, a method for encoding at least a part of an image is provided, comprising: converting a tensor representing a component of the image into a latent tensor; providing an entropy model; and processing the latent tensor by a neural network based on the provided entropy model to generate a bitstream, wherein the step of providing the entropy model comprises performing the steps of the method according to the sixteenth aspect or any of its implementations (with the same advantages as above). The latent tensor may be processed by an arithmetic encoder. The processing of the latent tensor to obtain a tensor representing a component of the image may comprise performing the steps of the method according to the sixteenth aspect or any of its implementations.

[0079] The entropy model may be provided by one of: a) a hyper-prior distribution for the variational autoencoder; b) an autoregressive prior distribution for the variational autoencoder; and c) a combination of the hyper-prior distribution and the autoregressive prior distribution for the variational autoencoder.

[0080] According to one implementation, the method of the nineteenth aspect further comprises a step of signaling, within the bitstream, the defined lower threshold and the defined upper threshold to the decoder side.

[0081] Furthermore, at least one of the first scaling factor, the second scaling factor and the third scaling factor may be signaled to the decoder side in the bitstream.

[0082] Furthermore, the method may include signaling the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold to the decoder side. Furthermore, the method may include signaling at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0083] According to a twentieth aspect, there is provided a method for reconstructing at least a part of an image, comprising the steps of providing an entropy model, processing a bitstream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image, and processing the latent tensors to obtain tensors representing components of the image, wherein the step of providing the entropy model comprises performing steps of a method for operating a neural network according to the sixteenth aspect or any implementation thereof (with the same advantages as above). The processing of the bitstream may be performed by an arithmetic decoder. The processing of the latent tensors to obtain tensors representing components of the image may comprise performing steps of a method according to the sixteenth aspect or any implementation thereof.

[0084] The entropy model may be provided by one of: a) a hyper-prior distribution for the variational autoencoder; b) an autoregressive prior distribution for the variational autoencoder; and c) a combination of the hyper-prior distribution and the autoregressive prior distribution for the variational autoencoder.

[0085] According to an implementation, the method of the twentieth aspect includes reading information about the defined lower threshold and the defined upper threshold from the bitstream. Further, information about at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be read from the bitstream.

[0086] Furthermore, the method may include receiving, at the decoder side, information regarding the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold.Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to a predefined first scaling factor and the difference relative to a predefined second scaling factor in the bitstream.

[0087] The components of the image represented by the tensors according to the methods of the third and nineteenth aspects may be Y, U or V components, or they are R, G, B components.

[0088] According to a twenty-first aspect, the present disclosure relates to a method for operating a neural network including a neural network layer including or connected to an accumulator register having a predefined accumulator register size for buffering a summation result. The method includes a step of defining an integer lower threshold A and an integer upper threshold B for integer values ​​included in a data entity (e.g., a number, a vector, or a tensor) of input data of the neural network layer. The method further includes a step of clipping the integer value included in the data entity of the input data to the defined integer lower threshold if the integer value included in the data entity of the input data is smaller than the defined integer lower threshold, and a step of clipping the integer value included in the data entity of the input data to the defined integer upper threshold if the integer value included in the data entity of the input data is larger than the defined integer upper threshold. The method further includes a step of determining integer value weights (i.e., weights including integers, e.g., weights only including integers) of the neural network layer based on the defined integer lower threshold, the defined integer upper threshold, and the predefined accumulator register size, so that integer overflow of the accumulator register can be avoided.

[0089] According to the method of the twenty-first aspect, the integer weights of the neural network layers of the neural network are adjusted such that integer overflow of the accumulator registers can be avoided when the input data is clipped to a defined threshold. This allows the interoperability between different platforms / devices to be significantly improved compared to that of the prior art. The adjustment of the weights may enable bit-exact reproduction of important numerical operations at the encoder and decoder sides, respectively, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, if integer overflow of the accumulator registers can be avoided at both sides, a region of an image (a still image or a frame of a video sequence) can be (entropy) encoded at the encoder side and reconstructed without corruption at the decoder side. Furthermore, the internal operation (state) of the involved devices is defined in substantially the same way by providing the adjusted weights determined according to the method of the twenty-first aspect. As described below, the integer weights can be determined such that integer overflow of the accumulator registers can be avoided in various ways, although the present disclosure is not limited to one of these specific ways.

[0090] The method of the 21st aspect works for any kind of neural network layer, including fully connected neural network layers and convolutional neural network layers. The method of the 21st aspect may also be advantageously implemented in a transformer architecture, where the neural network layer includes some attention mechanism (see also the detailed description below). The accumulator register size may be n bits, where n is a positive integer value, for example, n=32 bits or n=16 bits. The stored value may be -2 n-1 ~2 n-1 Range: -1 or 0 to 2 n-1 The accumulator register size can be fixed size or dynamically allocated.

[0091] According to one implementation, the integer lower threshold is less than or equal to 0, and the integer upper threshold is greater than or equal to 0. Thus, the restriction on non-negative integer values ​​of the input data is deemed appropriate for practical applications, so that possible or negative integer input values ​​can be allowed.

[0092] According to one implementation, the integer lower threshold A is −2 k-1 and the upper integer threshold B is 2 k-1 −1, where k represents a predefined bit depth of the layer input data, which is typically known for the particular application and configuration used, and can therefore be easily used to define the clipping thresholds to be applied to the input data.

[0093] The adjusted weights are typically used in the summation process performed by the neural network layer. According to one implementation, the neural network layer performs summation (accumulated in the so-called accumulator registers mentioned above).

number

[0094] A specific concrete formulation for determining the integer weights such that integer overflow of the accumulator registers can be avoided is contemplated herein. According to one implementation, the integer weights of the neural network layers {w i} is the condition

number

[0095] Considering that the input data value is limited to be between the defined lower threshold A and upper threshold B, if these conditions are met, integer overflow of the accumulator register can be reliably avoided. n-1 -1 and -2 n-1 Note that is given as an example of a constraint on accumulator register size only, and may be replaced by other constraints that appropriately define the accumulator register size.

[0096] According to another implementation, the integer weights {w i} is used to ensure that the condition is true so that integer overflow of the accumulator register is avoided.

number

[0097] According to another implementation, the integer weights {w i} is used to ensure that the condition is true so that integer overflow of the accumulator register is avoided.

number

[0098] The sum of these implementations

number

number

[0099] For a two-dimensional layer, the sum is

number

number

[0100] For all of these sums, the above conditions must be satisfied for each of the output channels of the neural network layer.

[0101] In particular, the user-dependent application weights of the neural network layer of the neural network may be provided as real-valued weights (i.e. the weights include real numbers, e.g. only real numbers). In this case too, the method of the 21st aspect and its implementations may be advantageously used. In this case, according to one implementation, the provided real-valued weights may be scaled by a first scaling factor to obtain scaled weights, and the scaled weights may be rounded to the respective nearest integer value to obtain integer-valued weights. The scaling factor may be selected with high flexibility. 2 sjIt may be considered appropriate to have a first scaling factor given by: j represents the number of bits to represent the fractional part of the real-valued weight.

[0102] In particular, according to one implementation of the method of the twenty-first aspect, a first scaling coefficient s for a j-th output channel of at least one neural network layer is j teeth,

number

[0103] According to another implementation, the second scaling coefficient s for the j-th output channel of at least one neural network layer is j teeth,

number

[0104] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0105] According to another implementation, the condition is:

number

number

[0106] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0107] Similarly, according to one implementation, the method includes scaling the data entities of the input data by the second scaling factor to obtain scaled values ​​of the data entities, the scaled values ​​of the data entities may be rounded to a respective nearest integer value to obtain integer values ​​of the data entities.

[0108] As already mentioned, the method of the 21st aspect can be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as above. Thus, according to the 22nd aspect, a method of encoding data is provided, comprising the steps of the method of operating a neural network according to the 21st aspect or any implementation thereof. In one implementation of the method of the 22nd aspect, the encoding of data comprises providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, the providing of the entropy model comprises performing the steps of the method of operating a neural network according to the 21st aspect or any implementation thereof. With regard to the substantially bit-exact reproduction of the entropy model used both on the encoder side and on the decoder side, in the context of entropy coding interoperability between different platforms, it is essential for reliable reconstruction of (compressed) data.

[0109] The entropy model provides statistical (probability) properties of the symbols to be encoded or decoded, such as mean, variance, (cross-)correlation, etc., and it may be provided by one of: a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of hyper-prior distribution and autoregressive prior distribution of the variational autoencoder. Thus, the method of the 22nd aspect can be advantageously implemented in a (variational) autoencoding system (see also the detailed description below).

[0110] According to a 23rd aspect (and complementary to the 22nd aspect), a method for decoding encoded data is provided, comprising the steps of the method for operating a neural network according to the 21st aspect or any implementation thereof. According to one implementation of the 23rd aspect, the decoding of the data comprises providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, the entropy decoding of the data comprising the steps of the method for operating a neural network according to the 21st aspect or any implementation thereof. Also on the decoder side, according to one implementation of the 23rd aspect, the entropy model is provided by one of: a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of the hyper-prior distribution and the autoregressive prior distribution of the variational autoencoder.

[0111] The data processed by the method of the twenty-second or twenty-third aspect may be, for example, image data representing a still image or a frame of a video sequence.

[0112] According to a 24th aspect, there is provided a method for encoding at least a portion of an image, comprising: converting tensors representing components of the image into latent tensors; providing an entropy model; and processing the latent tensors by a neural network based on the provided entropy model to generate a bitstream, wherein the step of providing the entropy model comprises performing steps of a method according to the 21st aspect or an implementation thereof (having the same or similar advantages as described above). According to an implementation of the 24th aspect, the entropy model is provided by one of: a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of the hyper-prior distribution and the autoregressive prior distribution of the variational autoencoder.

[0113] According to a twenty-fifth aspect, there is provided a method for reconstructing at least a portion of an image, comprising the steps of providing an entropy model, processing a bitstream by a neural network based on the provided entropy model to obtain latent tensors representative of components of the image, and processing the latent tensors to obtain tensors representative of components of the image, wherein providing the entropy model comprises performing the steps of a method according to the twenty-first aspect or an implementation thereof (having the same or similar advantages as described above). Again, the entropy model may be provided by one of a) a hyper-prior distribution of the variational autoencoder, b) an autoregressive prior distribution of the variational autoencoder, and c) a combination of the hyper-prior distribution and the autoregressive prior distribution of the variational autoencoder.

[0114] The image components represented in the above tensors may be, for example, Y, U, or V components, or R, G, B.

[0115] According to a twenty-sixth aspect, there is provided a neural network including a neural network layer configured to process input data to obtain output data, an activation function configured to process the output data to obtain activation function output data, a first unit for scaling, rounding and clipping the input data input to the neural network layer, and a second unit for unscaling at least one of the output data and the activation function output data. The first unit may be divided into sub-units each configured for different operations such as scaling, rounding and clipping the input data, and the on and off operations of the first unit and each sub-unit may be switchable, respectively.

[0116] The first unit may be configured to perform steps of the methods according to the sixteenth to twenty-fifth aspects and implementations thereof.

[0117] According to one implementation, the neural network of the 26th aspect is configured to perform the steps of the methods according to the 16th to 26th aspects and their implementations.

[0118] According to a 27th aspect, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network according to the 26th aspect and implementations thereof. The apparatus of the 27th aspect may comprise one of: a) a hyper-prior distribution of a variational autoencoder, b) an autoregressive prior distribution of a variational autoencoder, and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder, each of the priors comprising the neural network of the 26th aspect or implementations thereof.

[0119] According to a 28th aspect, there is provided an apparatus for decoding at least a portion of an encoded image, comprising a neural network according to the 26th aspect and implementations thereof. The apparatus may be an apparatus for decoding at least a portion of an image, and may comprise one of: a) a hyper-prior distribution of a variational autoencoder, b) an autoregressive prior distribution of a variational autoencoder, and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder, each of the priors comprising the neural network of the 26th aspect or implementations thereof.

[0120] According to a twenty-ninth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0121] According to a 30th aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0122] According to a thirty-first aspect, there is provided a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to any of the above-mentioned aspects associated with the method and any implementation thereof.

[0123] The method according to any of the above aspects relating to the method and any implementation thereof can be implemented in an apparatus configured to perform the steps of such method (with the same advantages as described above).

[0124] According to a thirty-second aspect, there is provided an apparatus for encoding data, the apparatus including a processing circuit configured to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof not limited to decoding.

[0125] According to a thirty-third aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including: a processing circuit configured to convert tensors representing components of the image into latent tensors; provide an entropy model including performing any step of the method according to any of the above aspects relating to the method and any implementation thereof not limited to decoding; and process the latent tensors by a neural network based on the provided entropy model to generate a bitstream.

[0126] According to a thirty-fourth aspect, there is provided an apparatus for decoding data, the apparatus including a processing circuit configured to perform any step of the method according to any of the above-mentioned aspects relating to the method not limited to encoding and any implementation form thereof.

[0127] According to a thirty-fifth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus including a processing circuit configured to provide an entropy model including performing any step of the method according to any of the above aspects relating to the method not limited to encoding and any implementation form thereof, process the bitstream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image, and process the latent tensors to obtain tensors representing components of the image.

[0128] The functions of the above-mentioned devices may be implemented by hardware, or by hardware executing corresponding software.

[0129] According to a further aspect, the present disclosure relates to a video stream encoding device, including a processor and a memory for storing instructions for causing the processor to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof, not limited to decoding.

[0130] According to a further aspect, the present disclosure relates to a video stream decoding device, including a processor and a memory for storing instructions for causing the processor to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof, not limited to encoding.

[0131] The methods and apparatus of the above-described aspects and implementations can be readily combined with one another where considered appropriate.

[0132] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0133] According to a twenty-ninth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0134] According to a 30th aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0135] According to a thirty-first aspect, there is provided a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to any of the above-mentioned aspects associated with the method and any implementation thereof.

[0136] The method according to any of the above aspects relating to the method and any implementation thereof can be implemented in an apparatus configured to perform the steps of such method (with the same advantages as described above).

[0137] According to a thirty-second aspect, there is provided an apparatus for encoding data, the apparatus including a processing circuit configured to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof not limited to decoding.

[0138] According to a thirty-third aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including: a processing circuit configured to convert tensors representing components of the image into latent tensors; provide an entropy model including performing any step of the method according to any of the above aspects relating to the method and any implementation thereof not limited to decoding; and process the latent tensors by a neural network based on the provided entropy model to generate a bitstream.

[0139] According to a thirty-fourth aspect, there is provided an apparatus for decoding data, the apparatus including a processing circuit configured to perform any step of the method according to any of the above-mentioned aspects relating to the method not limited to encoding and any implementation form thereof.

[0140] According to a thirty-fifth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus including a processing circuit configured to provide an entropy model including performing any step of the method according to any of the above aspects relating to the method not limited to encoding and any implementation form thereof, process the bitstream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image, and process the latent tensors to obtain tensors representing components of the image.

[0141] The functions of the above-mentioned devices may be implemented by hardware, or by hardware executing corresponding software.

[0142] According to a further aspect, the present disclosure relates to a video stream encoding device, including a processor and a memory for storing instructions for causing the processor to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof, not limited to decoding.

[0143] According to a further aspect, the present disclosure relates to a video stream decoding device, including a processor and a memory for storing instructions for causing the processor to perform any step of the method according to any of the above-mentioned aspects relating to the method and any implementation thereof, not limited to encoding.

[0144] The methods and apparatus of the above-described aspects and implementations can be readily combined with one another where considered appropriate.

[0145] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0146] The following embodiments of the present disclosure will be described in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0147] [Figure 1] FIG. 2 is a schematic diagram illustrating channels processed by layers of a neural network. [Diagram 2] FIG. 1 is a schematic diagram illustrating an autoencoder type neural network. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example network architecture in which the encoder and decoder sides include hyper-prior models. [Figure 3B] FIG. 1 is a schematic diagram illustrating a general network architecture in which the encoder side includes a hyper-prior model. [Figure 3C] FIG. 1 is a schematic diagram illustrating a general network architecture in which the decoder side includes a hyper-prior model. [Figure 4] FIG. 1 is a schematic diagram illustrating an example network architecture in which the encoder and decoder sides include hyper-prior models. [Diagram 5] FIG. 1 is a block diagram illustrating the structure of a cloud-based solution for machine-based tasks, such as machine vision tasks. [Figure 6] FIG. 1 is a block diagram illustrating an end-to-end video compression framework based on neural networks. [Figure 7] Illustrate a transformer. [Figure 8] 1 illustrates a neural network layer. [Figure 9] 1 illustrates an exemplary neural network according to one embodiment. [Figure 10] 1 is a flow chart illustrating a method of operating a neural network including clipping of input data. [Figure 11] 1 illustrates a neural network that includes units for clipping, scaling, and rounding, and a unit for unscaling, according to one embodiment. [Figure 12] 1 is a flow chart illustrating a method of operating a neural network including adjusting neural network layer weights. [Figure 13] 1 is a flow chart illustrating a method of operating a neural network that includes an implementation of an approximate activation function. [Figure 14] 1 illustrates an exemplary neural network including an approximated activation function, according to one embodiment. [Figure 15] 1 shows an apparatus configured to perform the method steps illustrated in FIGS. [Figure 16] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 17] 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 18] 1 is a block diagram showing an example of an encoding device or a decoding device. [Figure 19] 1 is a block diagram illustrating another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0148] Like reference numbers and designations in different drawings may indicate like elements.

[0149] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and show, by way of example, specific aspects of the disclosed embodiments or specific aspects in which the disclosed embodiments may be used. It is understood that the disclosed embodiments may be used in other ways and may include structural or logical changes not shown in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0150] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, where one or more particular method steps are described, the corresponding device may include one or more units, e.g., functional units, for performing the described one or more method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the multiple steps, respectively), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, where a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps performing the function of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it is understood that the features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise noted.

[0151] The following provides an overview of some of the terminology used and the framework within which embodiments of the present disclosure may be employed.

[0152] Artificial Neural Networks An artificial neural network (ANN), or connectionist system, is a computing system loosely inspired by the biological neural circuits that make up animal brains. Such systems "learn" to do a task by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, an ANN may learn to identify images that contain cats by analyzing example images that have been manually labeled as "cat" or "not cat" and using the results to identify cats in other images. The ANN does this without any prior knowledge of cats, e.g., that cats have fur, tails, whiskers, and cat-like faces. Instead, the ANN automatically generates discriminative characteristics from the examples that the ANN processes.

[0153] ANNs are based on a collection of connected units or nodes called artificial neurons that loosely model neurons in the biological brain. Each connection can send signals to other neurons, similar to synapses in the biological brain. An artificial neuron that receives a signal can then process the signal and signal to neurons connected to it.

[0154] In an ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically each have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons may have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layer multiple times.

[0155] The original goal of the ANN approach was to solve problems in the same way that the human brain solves them. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a variety of tasks including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even in activities traditionally considered to be exclusively human, such as painting.

[0156] The name "convolutional neural network" (CNN) indicates that the network employs a mathematical operation called convolution, which is a special kind of linear operation. Convolutional networks are neural networks that use convolution instead of the common matrix multiplication in at least one of their layers.

[0157] FIG. 1 illustrates a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer and an output layer, as well as multiple hidden layers. The input layer is the layer to which the input (such as a portion of an input image 11 as shown in FIG. 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The output of the layer is one or more feature maps (illustrated by empty solid rectangles), sometimes called channels. There may be resampling (such as subsampling) involved in the operation of some or all of the layers. As a result, the feature maps may be smaller, as illustrated in FIG. 1. Note that convolutions with strides may also reduce the size (resampling) of the input feature maps. The activation function in a CNN is typically a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called a convolutional layer, but this is only by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index in the matrix, in that it affects how the weights are determined at that particular index point.

[0158] When programming a CNN to process images, the input is a tensor with dimensions (number of images) × (image width) × (image height) × (image depth), as shown in Figure 1. It should be understood that the image depth can be composed by the channels of the image. After passing through the convolutional layer, the image is abstracted into a feature map with dimensions (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network should have the following attributes: a convolution kernel defined by width and height (hyperparameters); the number of input and output channels (hyperparameters); the depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0159] In the past, traditional multi-layer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, MLP models suffer from high dimensionality and do not scale well with high-resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently and conveniently processed with full connectivity. Also, such network architectures do not take into account the spatial structure of the data, treating input pixels that are far away from each other the same as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Thus, for purposes such as image recognition, where spatially local input patterns dominate, full connectivity of neurons is wasteful.

[0160] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of CNNs. The parameters of the layer consist of a set of learnable filters (kernels, as mentioned above) that have small receptive fields but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns filters that activate when it detects some particular type of feature at some spatial location in the input.

[0161] Stacking the activation maps of all filters along the depth dimension forms the full output volume of the convolutional layer. Thus, every entry in the output volume can also be interpreted as the output of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation have the same meaning. It is called an activation map in some papers because it is a mapping that corresponds to the activation of different parts of the image, and also a mapping of when a certain type of feature can be found in the image, so it is also called a feature map. High activation means that a certain feature has been found.

[0162] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0163] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0164] A pooling layer operates independently on each depth slice of the input and resizes it spatially. The most common form is a pooling layer with a filter of size 2 × 2, applied with a stride of 2 for each depth slice of the input by 2 along both the width and height, discarding 75% of the activations. In this case, all max operations are over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can also use other functions such as average pooling and L2 norm pooling. Average pooling was often used in the past, but has recently fallen out of favor in comparison to max pooling, which often works better in practice. For aggressive reduction of the size of the representation, the recent trend is to use smaller filters or to discard the pooling layer altogether. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling, where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0165] The ReLU above is short for rectified linear unit and applies a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting them to zero. ReLU increases the non-linearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions, such as saturated hyperbolic tangent and sigmoid functions, are also used to increase the non-linearity. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly compromising generalization accuracy.

[0166] After several convolutional and max-pooling layers, higher-level inference in neural networks is performed via fully connected layers, whose neurons are connected to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as affine transformations, matrix multiplications followed by bias offsets (vector addition of a learned or fixed bias term).

[0167] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions suitable for different tasks may be used: Softmax loss is used to predict a single class out of K mutually exclusive classes; Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0168] In summary, FIG. 1 shows the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map containing multiple channels corresponding to several filters in the set of learnable filters of this layer. The feature map is then subsampled, for example using a pooling layer, which reduces the dimensionality of each channel in the feature map. The data then reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish the connectivity of the network, these parameters need to be synchronized between two connected layers, such that the number of input channels of the current layer is equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is usually equal to the number of channels of the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer(s)) may be passed to an output layer. Such an output layer may be convolutional or resampled in some implementations. In an exemplary and non-limiting implementation, the output layer is a fully connected layer.

[0169] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. Its schematic diagram is shown in Figure 2. An autoencoder includes an encoder side 210 where an input x is input to the input layer of an encoder sub-network 220, and a decoder side 250 where an output x' is output from a decoder sub-network 260. The goal of an autoencoder is to learn a representation (encoding) 230 of a dataset x, usually for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Together with the reduction (encoder) side sub-network 220, the reconstruction (decoder) side sub-network 260 is trained, and the autoencoder tries to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, hence its name. In the simplest case, given one hidden layer, the encoder stage of the autoencoder takes the input x and maps it to h, h = σ(Wx+b) It is.

[0170] This image h is typically called the code 230, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are typically initialized randomly and then updated iteratively during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x: x'=σ'(W'h'+b') where σ', W', and b' of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0171] Variational autoencoder models make strong assumptions about the distribution of the latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is then fed into a directed graphical model p θ(x|h), and the encoder estimates the posterior distribution p θ Approximation q for (h|x) φ Suppose we are learning (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than standard autoencoders. The objective of a VAE has the form

number

[0172] In the formula, D KL represents the Kullback-Leibler divergence. The prior distribution for the latent variables is a centrally isotropic multivariate Gaussian distribution p θ It is usually set that (h) = N(0,Ι). In general, the shapes of the variational and likelihood distributions are chosen such that they are factored Gaussians: q φ (h|x)=N(ρ(x),ω 2 (x)I) p φ (x|h)=N(μ(h),σ 2 (h)I) where ρ(x) and ω 2 (x) is the encoder output, μ(h) and σ 2 (h) is the decoder output.

[0173] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have enabled researchers to apply neural network-based techniques to the task of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0174] Accordingly, data compression is considered a fundamental and well-studied problem in engineering, and is generally formulated with the goal of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0175] In this context, known as the lossy compression problem, we must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0176] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space is intractable. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a reversible entropy code. This scheme is called transform coding because of the central role of the transform.

[0177] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method, transform, quantization, and entropy code, are optimized separately (often by manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. Several transforms are used for that purpose, such as the discrete cosine transform and discrete sine transform (DCT, DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0178] Variational Image Compression The variable autoencoder (VAE) framework can be regarded as a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in Figure 3A, which shows the VAE framework.

[0179] The transformation process can be divided into four main parts, and FIG. 3A illustrates the VAE framework. In FIG. 3A, the Encoder 101 maps the input image x to a latent representation (represented by y) via a function y=f(x). This latent representation may be referred to below as a part of the "latent space" or a point in the "latent space". The function f() is a transformation function that converts the input signal x to a more compressible representation y. The Quantizer 102 maps the latent representation y using Q, which represents a quantization function.

number

number

number

[0180] A latent space can be understood as a compressed representation of data where similar data points are close to each other in the latent space. Latent spaces are useful for learning data features and finding simpler representations of data for analysis. The quantized latent representation T,

number

number

number

number

number

number

[0181] In FIG. 3A, component AE105 is an arithmetic encoding module that encodes the quantized latent representation

number

number

number

number

[0182] Arithmetic decoding (AD) 106 is the process of undoing the binarization process, where binary numbers are converted back to sample values. Arithmetic decoding is provided by arithmetic decoding module 106.

[0183] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0184] In FIG. 3A there are two sub-networks connected to each other. A sub-network in this context is a logical division between parts of the total network. For example, in FIG. 3A modules 101, 102, 104, 105 and 106 are called the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generation) and decoding (analysis) of a first bitstream "Bitstream 1". The second network in FIG. 3A includes modules 103, 108, 109, 110 and 107 and is called the "hyperencoder / decoder" sub-network. The second sub-network is responsible for generating a second bitstream "Bitstream 2". The purpose of the two sub-networks is different.

[0185] The first sub-network is Transformation 101 of an input image x into its latent representation y (this makes it easier to compress x), · The latent representation y is converted to a quantized latent representation

number

number

number

[0186] The purpose of the second sub-network is to obtain statistical properties of the samples of "Bitstream 1" (e.g., mean, variance and correlation between samples of Bitstream 1) so that the compression of Bitstream 1 by the first sub-network is more efficient. The second sub-network generates a second bitstream "Bitstream 2" that includes said information (e.g., mean, variance and correlation between samples of Bitstream 1).

[0187] The second network is the quantized latent representation

number

number

number

number

number

number

number

number

number

number

number

number

number

[0188] 3A describes an example of a VAE (Variational Autoencoder), the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler that targets extracting cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0189] 3A shows the encoder and the decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and the decoder may, and very often are, integrated into different devices.

[0190] FIG. 3B shows an encoder, and FIG. 3C shows the decoder component of the VAE framework in isolation. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels, motion information channels, etc. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of a first sub-network of the encoder, and Bitstream 2 is the output of a second sub-network of the encoder.

[0191] Similarly, in FIG. 3C, two bitstreams, Bitstream 1 and Bitstream 2, are received as input, and the reconstructed (decoded) image is

number

[0192] Specifically, as seen in Figure 3B, the encoder includes an encoder 121 that converts an input x into a signal y that is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic encoding module 125 and a hyper-encoder 123. The hyper-encoder 123 provides bitstream 2, already described above, to a hyper-decoder 147, which in turn provides this information to the arithmetic encoding module 105 (125).

[0193] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the output of the encoding of the signal, which are then provided (sent) to the decoding process. Unit 101 (121) is called "encoder", but the complete sub-network described in FIG. 3B can also be called "encoder". An encoder generally means a unit (module) that converts an input into an encoded (e.g., compressed) output. As can be seen from FIG. 3B, unit 121 can in fact be considered as the core of the entire sub-network, since it performs the conversion of the input x into y, which is a compressed version of x. The compression in encoder 121 can be achieved, for example, by applying a neural network, or in general any processing network with one or more layers. In such a network, the compression can be performed by cascading processes, including downsampling, which reduces the size and / or number of channels of the input. Thus, the encoder can be called, for example, a neural network (NN)-based encoder, etc.

[0194] The remaining parts in the figure (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE 125, in combination with the hyperencoder 123 and hyperdecoder 127 used to configure the AE 125, may perform binarization that may further compress the quantized signal by lossless compression. Therefore, the entire sub-network of FIG. 3B may be called an "encoder".

[0195] Most deep learning (DL) based image / video compression systems reduce the dimensionality of a signal before converting it to binary numbers (bits). For example, in the VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height and thus a smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress the signal y. Note that in general, an encoder does not necessarily need to reduce the size of both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size of only one dimension (or generally a subset of dimensions).

[0196] In J. Balle, L. Valero Laparra, and EP Simoncelli (2015) "Density Modeling of Images Using a Generalized Normalization Transformation", In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016 (hereafter referred to as "Balle"), the authors propose a framework for end-to-end optimization of image compression models based on nonlinear transformations. They optimize for the mean squared error (MSE) but use a more flexible transformation built from a cascade of linear convolutions and nonlinearities. In particular, they use the Generalized Divided Normalization (GDN) combined nonlinearity, inspired by models of neurons in the biological visual system and proven effective for Gaussianized image densities. This cascade transformation is followed by a uniform scalar quantization (i.e., each element is rounded to the nearest integer), effectively implementing a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0197] Such an example of a VAE framework is shown in Figure 4, which utilizes six downsampling layers marked 401-406. The network architecture includes a hyper-prior model. On the left (g a ,g s ) shows the image autoencoder architecture, and on the right (h a ,h s ) corresponds to an autoencoder that implements a hyper-prior. The factorized prior model is then transformed into an analytic and synthetic transformation g a and g s We use the same architecture for both. Q represents quantization, and AE and AD represent arithmetic encoder and decoder, respectively. The encoder applies g a, resulting in responses y (latent representations) with spatially varying standard deviations. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function.

[0198] The response is h a to summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. The encoder then

number

number

number

number

number

number

number

[0199] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer, has N channels, and the size of the convolution kernel is k1×k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As above, 2↓ means that downsampling by a factor of two is performed in this layer. Downsampling by a factor of two results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of two. Since there are six downsampling layers, if the width and height of the input image 414 (also represented by x) are given by w and h, the output signal z^ 413 has a width and height equal to w / 64 and h / 64, respectively. The modules represented by AE and AD are arithmetic encoders and arithmetic decoders, which are described with reference to FIG. 3A to FIG. 3C. The arithmetic encoder and decoder are specific implementations of entropy coding. The AE and AD can be replaced by other means of entropy coding. In information theory, entropy encoding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, and is a reversible process. Also, the "Q" in the figure corresponds to the quantization operation referenced in connection with FIG. 4 above, and further described in the "Quantization" section above. Also, the quantization operation and corresponding quantization unit as part of the components 413 or 415 are not necessarily present and / or can be replaced with another unit.

[0200] Also shown in Figure 4 is a decoder including upsampling layers 407-412. A further layer 420, implemented as a convolutional layer but which does not provide upsampling to the received input, is provided between upsampling layers 411 and 410 in the processing order of the input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer does not change the size of the input, but may be provided in a NN to perform operations on the input that change certain characteristics. However, it is not necessary that such a layer be provided.

[0201] When viewed in processing order of the bitstream 2 through the decoder, the upsampling layers are in reverse order, i.e. from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here as providing upsampling with an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, other upsampling ratios such as 3, 4, 8, etc. may be used. Layers 407-412 are implemented as convolutional layers (conv). In particular, they may be intended to provide, on the input, an inverse operation of the encoder's operation, and thus the upsampling layers may apply a deconvolution operation to the received input such that its size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two neighboring samples, or by nearest neighbor sample copying, etc.

[0202] In the first sub-network, some convolutional layers (401-403) are followed by Generalized Divided Normalization (GDN) on the encoder side and Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLU. It should be noted that the present disclosure is not limited to such implementations and in general, other activation functions may be used instead of GDN or ReLU.

[0203] Cloud solutions for machine tasks Machine-oriented video coding (VCM) is another computer science direction that is popular today. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms like object segmentation, detection and recognition. In contrast to traditional image and video coding, which is intended for human perception, the quality characteristic is not the reconstruction quality but the performance of a computer vision task, e.g. object detection accuracy. This is illustrated in Figure 5.

[0204] Machine-directed video coding, also called collaborative intelligence, is a relatively new paradigm for efficient deployment of deep neural networks across a mobile cloud infrastructure. By partitioning the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computational nodes, e.g., between devices, but generally between any functionally defined nodes. Here, the term "node" does not refer to the neural network nodes described above. Rather, a (computational) node here refers (physically or at least logically) to a separate device / module that implements a part of a neural network. Such devices may be different servers, different end-user devices, a mix of servers and / or user devices, and / or a cloud and / or a processor, etc. In other words, the computational nodes may be considered as nodes that belong to the same neural network and communicate with each other to convey coding data within / for the neural network. For example, one or more layers may be executed on a first device (such as a device on the mobile side 510) and one or more layers may be executed on another device (such as a cloud server on the cloud side 590) to be able to perform complex calculations. However, the distribution may also be finer and a single layer may be executed on multiple devices. In this disclosure, the term "multiple" refers to two or more. In some existing solutions, part of the neural network function is executed on a device (such as a user device or an edge device) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device running part of the neural network. The concept of collaborative intelligence has also been extended to model training.In this case, data flows in both directions: from cloud to mobile during backpropagation in training, and during the forward pass in training, as well as from mobile to cloud (illustrated in Figure 5) in inference.

[0205] Some works have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization was presented, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590, rather than sending the compressed natural image data to the cloud and using the reconstructed image to perform object detection. Thus, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, e.g. arithmetic coding, are common approaches for compression of deep features (i.e., feature maps).

[0206] Today, video content contributes to over 80% of Internet traffic, and the percentage is expected to further increase. Therefore, it is important to build an efficient video compression system to generate higher quality frames for a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection and video object tracking, are sensitive to the quality of the compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, the techniques of video compression also help in action recognition and model compression. However, in the past decades, video compression algorithms, as mentioned above, rely on hand-crafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences. Although each module is well designed, the entire compression system is not optimized end-to-end. It is desirable to further improve the video compression performance by optimizing the entire compression system together.

[0207] End-to-end image or video compression DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations that are not used in traditional approaches. However, it is not trivial to directly apply these techniques to build end-to-end learning systems for video compression. First, learning how to generate and compress tuned motion information for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0208] A straightforward solution is to represent the motion information using learning-based optical flow. However, current learning-based optical flow approaches aim to generate a flow field that is as accurate as possible. Precise optical flow is often not optimal for a particular video task. Moreover, the amount of data for optical flow increases significantly when compared with the motion information in traditional compression systems, and directly applying existing compression approaches to compress the optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective for both the residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of the reconstructed frames given the number of bits (or bit rate) for compression. RDO is critical for video compression performance. To take advantage of the power of end-to-end training for learning-based compression systems, an RDO strategy is needed to optimize the entire system.

[0209] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao; "DVC: An End-to-end Deep Video Compression Framework", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0210] Such an encoder is illustrated in Fig. 6. In particular, Fig. 6 shows the overall structure of an end-to-end trainable video compression framework. To compress the motion information, we use the optical flow vt The corresponding representation m that is suitable for better compression. t The CNN was specified to convert

[0211] Transformers In recent years, Transformers have attracted increasing attention in the fields of both language processing (e.g., text translation) and image processing. Video coding can be facilitated by the adoption of Transformer-based neural networks. Transformers can be used for image enhancement and classification purposes. Transformers do not include recurrent or convolutional neural networks, but rely on self-attention. Transformers may also be implemented, for example, in the configuration shown in FIG. 6. In particular, Transformers may be combined with recurrent or convolutional neural networks.

[0212] FIG. 7 illustrates an example of a Transformer 700. The Transformer 700 includes a neural layer 810 (Transformer layer). The Transformer 700 may include an encoder-decoder architecture including an encoder neural layer and a decoder neural layer. Alternatively, the Transformer 700 may include an encoder stack of neural layers only. Input data is input to the Transformer and output data is output. For example, the Transformer 700 may be configured for image processing and output an enhanced image. The input data may include, for example, patches of images or words of a sentence. For example, the tokenizer generates tokens in the form of patches from an image or words from a sentence to be processed. These tokens can be converted to embeddings (of continuous values) by some embedding algorithm. According to the example shown in FIG. 7, the linear projection layer 720 converts the input patches to tensor representations (embeddings) of the parts of the object to be processed (in the latent space). These tensor representations of the signal input are processed by the Transformer 700. The provision of the positional encoding layer 730 provides information about the position of parts of the processed object (e.g. an image or a sentence) relative to one another, for example the position of patches of an image or words of a sentence relative to one another. A sine function for the positional encoding may be used.

[0213] The final layer of the neural layer 710 outputs data tensors in latent space that are converted back to object space (e.g., image or sentence space) by the linear backprojection layer 740.

[0214] Processing by the neural layers (transformer layers) 710 is based on the concept of self-attention. Each of the neural layers 710 of the transformer 700 includes a multi-head self-attention layer and a (fully connected) feed-forward neural network. The self-attention layer helps the encoder stack to look at other parts of the object (e.g., image or word patches) when encoding a particular part of the object (e.g., a patch of an image or a word of a sentence). The output of the self-attention layer is fed into the feed-forward neural network. The decoder stack also has both of its components, and an additional "encoder-decoder") attention layer between them, that helps the decoder stack to focus on the relevant part of the input data. Each part of the object (e.g., a patch of an image or a word of a sentence) being processed at each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. However, the feed-forward layer does not have these dependencies, so various paths can be executed in parallel while flowing through the feed-forward layer.

[0215] In the multi-head self-attention layer of the encoder stack, we take the query Q, the key K, and the value V tensors, as well as the self-attention

number

[0216] Each sub-layer (self-attention layer and feedforward neural network) within each encoder and decoder layer has residual connections around it, followed by a normalization layer.

[0217] The output of the top encoder layer is transformed into a set of attention vectors, K and V, which are used by each decoder layer in that encoder-decoder attention layer. The encoder-decoder attention layer operates similarly to the multi-head self-attention layer in the encoder stack, except that it creates a query matrix from each of the layers below and obtains the key and value matrices from the output of the encoder stack.

[0218] The decoder stack outputs a vector of floating point numbers, which are converted to object parts (e.g. patches of an image or words of a sentence) by a final linear layer (a fully connected neural network) that outputs logits, followed by a Softmax layer that produces the highest probability output.

[0219] As mentioned above, the adoption of neural networks (e.g., CNNs) is becoming increasingly important in the field of image coding systems, and indeed in data compression systems in general, e.g. in the context of neural network-based compression of images, video, audio, 3D-PCC, etc.

[0220] In practice, neural networks in practical applications are implemented on various platforms / devices that differ from each other in terms of numerical architecture. For example, one device may be a CPU or contain a CPU, while another device in data communication with the device that is a CPU or contains a CPU may be a GPU or contain a GPU. Different platforms (e.g., one that contains a CPU on one device and a GPU on another device) usually deal with sophisticated integer and real situations in different ways, especially floating point arithmetic but also fixed point arithmetic. One class of sophisticated real situations is overflow of the registers / memories used. There is no standardized handling of overflow situations and it is hardly possible to imagine a procedure that is commonly followed in this regard.

[0221] Integer overflow occurs when an arithmetic operation attempts to create a number outside the range that can be represented with a given number of digits, i.e., either higher than the maximum value or lower than the minimum representable value. This situation can be handled differently for different compilers, devices (CPU, GPU), etc. In order to achieve bit-exact results of integer operations on different platforms, integer overflow should be avoided. In this context, it is noted that obtaining bit-exact results on different platforms is not important for all processing applications. However, in some applications it is. For example, in the context of autoencoding, e.g., the variational autoencoder as described above with reference to Figures 2-4, an entropy model for lossless data compression is used at the encoder side and the decoder / receiver side to model the symbol probabilities of the arithmetic encoder and the decoder, respectively. The encoder side and the decoder / receiver side may employ different kinds of platforms. It is important for reliable reconstruction of the encoded image that the entropy model is applied in an identical manner on both sides. In fact, even a small deviation of the entropy model used on the decoder / receiver side from the entropy model used on the encoder side can lead to a complete breakdown of the image decoding / reconstruction process due to incorrect symbol interpretation. For example, in the configuration illustrated in FIG. 4, the entropy model

number

[0222] With regard to real arithmetic, a distinction must be made between fixed-point and floating-point arithmetic. Fixed-point arithmetic is arithmetic in which fractional (non-integers) numbers are represented by storing a fixed number of fractional digits. In general, this is integer arithmetic, and to convert such fixed-point values ​​to real numbers, division by a scaling factor corresponding to the number of bits used to store the fractional part is used. Floating-point arithmetic is a representation of mantissa*base. 指数 is an arithmetic that uses a representation of real numbers of the form mantissa * base, where the mantissa is an integer, the base is an integer greater than or equal to 2, and the exponent is also an integer. The base is fixed, and the pair of mantissa and exponent represents a number. The adoption of a non-fixed exponent makes it possible to achieve a trade-off between dynamic range and precision, compared to fixed-point arithmetic. Essentially, in both fixed-point and floating-point arithmetic, real numbers are expressed as mantissa * base. 指数The only difference is that in fixed-point arithmetic, both the base and the exponent are fixed, while in floating-point arithmetic, only the base is fixed and the exponent is part of the number representation. Thus, in fixed-point arithmetic, the precision of the operation result (the exponent) does not depend on the precision of the arguments, which means, for example, that the result of a sum does not depend on the order of the summation (if no overflow occurs, the operation is associative, like a normal integer summation). On the other hand, in floating-point arithmetic, the exponent is part of the number representation, which is calculated during the arithmetic operation based on the exponents of the arguments. This causes the summation to be non-associative. For example, if a large number of very small numbers are added one by one to a larger number, the result will be equal to the larger number because the effect of the relatively small numbers is lost after rounding with the exponent of the larger number. Conversely, if the smaller numbers are added to each other first, the effect of adding each number is not lost by rounding and the resulting sum may not be completely lost after rounding with the exponent of the relatively large, larger number. Thus, unless there is an overflow, the summation in fixed-point arithmetic is associative, while in principle floating-point arithmetic is not associative, i.e., in general, (A+B)+C≠A+(B+C). Since the order of summation may be different or even not completely predetermined on the encoder and decoder sides, this creates a problem as to whether identical results are achieved on both sides. Therefore, fixed-point arithmetic may be preferred over floating-point arithmetic. In the context of entropy coding-based data compression to ensure proper decoding on various platforms, especially on systems with large parallelism, it may be preferable to avoid the use of floating-point arithmetic at least in the entropy part of the image coding network and use fixed-point arithmetic instead. In this case, the potential problem of different implementations of floating-point arithmetic on different devices is solved, since fixed-point (integer) arithmetic is much more portable. However, overflow cannot be avoided by simply restricting to fixed-point arithmetic.To guarantee bit-exact behavior on different platforms, it is not sufficient to use fixed-point arithmetic, but it is important to guarantee that there is no integer overflow.

[0223] More specifically, consider a neural network layer as illustrated in FIG. 8. The neural network layer may be included in, for example, a convolutional neural network or a fully connected neural network. i are input to different input channels, and the weights w ij where index i is C in represents the input channel of the input channel, and the index j of C out The jth channel's output is

number

[0224] The accumulator register may be used for summation buffering.

[0225] The (usually trainable) bias D is ignored in the following for simplicity. For example, for a convolutional neural network layer, the output of the jth channel is

number

[0226] The accumulator registers of a neural network used for summation buffering have a predefined size, i.e., some number of bits (size) n of the accumulator register (e.g., n=16 bits or 32 bits). To avoid integer overflow, for example, the following condition

number

[0227] A) Clipping and scaling According to one embodiment, for the fixed-point arithmetic used, the input data x processed by a neural network layer of a neural network (e.g., a convolutional neural network) is i , and the real-valued weights w ij is an integer value

number

number

number

number

number

number

[0228] To avoid the performance loss caused by this conversion to fixed-point arithmetic,

number

number

number

[0229] Input value x i The range of is not generally known. To satisfy the condition of Eq. 5, the following restrictions are

number

[0230] Thus, according to this embodiment, the following units are added to the neural network (inference pipeline): units configured for scaling, rounding and clipping input data values ​​before being input to the neural network layers, and a unit for dividing the output of the neural network layers by a scaling factor. If it is guaranteed that only integer-valued data (data containing only integers) is processed, the scaling and / or rounding units may be omitted (or not operated). The units may be added to neural networks used for data compression / coding, in particular to (variational) autoencoder or transformer architectures as described above.

[0231] An exemplary embodiment is illustrated in FIG. 9. FIG. 9 shows the neural network layers of a convolutional neural network for exemplary purposes. The neural network includes a stack of convolutional layers followed by an activation function, in this case a LeackyReLU function. The black arrows in FIG. 9 indicate the inputs of new units added to a conventional neural network. Before the convolutional layers, the input values ​​of the real-valued input data are multiplied by a scaling factor, and the result of the multiplication process is rounded to the nearest integer. If the input data is already integer-valued, the scaling and / or rounding process can be omitted. If the resulting integer value is smaller than a predefined lower integer threshold −A, the integer value is clipped to a predefined lower integer threshold −A, and if the integer value is larger than a predefined upper integer threshold B, the integer value is clipped to an upper integer threshold B. Thus, the resulting input values ​​are processed by the convolutional neural network layers, and the outputs of these layers are input. According to one embodiment, the output of the activation function is divided by a scaling factor applied to the input data. According to another embodiment, the output of the convolutional neural network layer is divided by a scaling factor and then fed to the activation function. According to a further embodiment, a factorization of each of the scaling factors into two parts is performed, the division by the first part can be performed before the activation function and the division by the second part can be performed after the activation function.

[0232] A neural network such as that shown in Figure 9, including units for scaling, rounding, and clipping the outputs of the neural network layers, and dividing them by a scaling factor (before and / or after the respective activation functions), may be provided on both the encoder and decoder sides of a coding system, e.g., to an encoder for encoding an image and a decoder for decoding the encoded image. For example, this can be employed by the encoder / decoder included in the configurations illustrated in Figures 2 to 4, 6 and 7.

[0233] B) Weight The condition in Equation 5 can be transformed into another condition for the weights of the neural network layers. In the following, as described above, the input data x i and weight w ij It is assumed that A, B, C, and D have already been converted to integers (if necessary) by scaling and rounding. It is further assumed that clipping to −A and B, respectively, has been performed if necessary according to Equation 6. The condition in Equation 5 (for each output channel if different output channels are provided, the index j indicating the output channel is omitted below) is

number

[0234] If the condition in Equation 7 is satisfied, the inference, including clipping of the input data to the lower and upper thresholds, respectively, will not cause the overflow of the accumulator registers of the neural network. If the model weights in the data compression system can be user-defined, the condition in Equation 7 must be satisfied if bit-exact behavior of the encoder-side and decoder-side neural networks needs to be guaranteed.

[0235] According to one embodiment, the integer lower threshold -A is -2 k-1 and the upper integer threshold B is 2 k-1 −1, where k represents the predefined bit depth of the layer input data.

[0236] In this case, the weighting conditions that are easy to check are:

number

number

[0237] This form of weight condition can be particularly easily computed for user-defined weights to check whether overflow is guaranteed to never occur during inference. The conditions according to Equation 8 and Equation 9 must be satisfied for each output j of a neural network layer.

[0238] The bias D can in principle be set to 0, in which case Equations 8 and 9 are even easier to check.

[0239] In particular, for one-dimensional convolutional neural network layers, the sum

number

number

[0240] For a two-dimensional convolutional neural network layer, the sum

number

number

number

[0241] C) Weight scaling factor Weight scaling factor s j (More precisely,

number

number

number

[0242] According to another implementation, the second scaling coefficient s for the j-th output channel of at least one neural network layer is j teeth,

number

[0243] In these conditions, n represents the bit size of the accumulator register, and wij represents the real-valued weights, k represents the predefined bit depth of the input data, and b j represents the bias value (which can be 0).

[0244] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0245] According to another implementation, the condition is:

number

number

number

[0246] To convert the entire neural network pipeline to fixed-point arithmetic, the parameters p and k must be known. These parameters can be, for example, in some calibration data set, for a chosen k and a predefined n, (p,k), 0≦p≦k≦n, and the corresponding s obtained using Eq. (10). j The loss function can be selected by checking all possible combinations of p, k. The minimum of the predefined loss function corresponds to the best pair (p, k). The loss function can, for example, express the number of bits required to encode some image or its likelihood estimate. If the analysis and synthesis parts (see, for example, the left side of the configuration shown in Figure 4) are converted to fixed-point arithmetic, not only the entropy part but also the distortion can be used as part of the loss function used.

[0247] D) Activation function To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear and relatively simple nonlinear activation functions, such as the ReLU function that essentially defines the clipping procedure, this requirement can be met relatively easily. For more complex nonlinearities, especially

number

[0248] According to an embodiment, the mathematically defined nonlinear activation function is replaced with an approximation function selected from the group consisting of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function and a parametric ReLU function. The mathematically defined nonlinear activation function to be replaced may be a Softmax function, a Sigmoid function, a Hyperbolic Tangent function, a Swish function, a Gaussian Error Linear Unit Function, or a Scaled Exponential Linear Unit Function.

[0249] The approximation function may be employed, for example, by an encoder / decoder included in the configurations illustrated in Figures 2 to 4, 6 and 7, and may be used in the context of data compression / coding in general.

[0250] In general, for any nonlinear activation function, the sum of a few first elements of its Taylor series can be used. The Taylor series of a mathematically defined function f(x) centered on a predefined value a is given by

number

number

number

[0251] It should be noted that, depending on the practical application, approximations of mathematically defined non-linear activation functions by rational or polynomial functions may be considered particularly appropriate.

[0252] In particular, according to one embodiment, a vector x operates on i components

number

number

number

number

[0253] The Softmax function defined by Equation 15 and Equation 16 may be advantageously used as the activation function of the last one of a stack of (e.g., convolutional) neural network layers to provide, for example, the classification results obtained by the neural network.

[0254] In particular, an embodiment is provided relating to a method 1000 for operating a neural network in which integer overflow can be avoided. The neural network includes a neural network layer including or connected to an accumulator register with a predefined accumulator register size for buffering summation results. The method 1000 includes a step S1010 of defining lower and upper integer thresholds for values ​​of integers included in a data entity (e.g., a number, a vector or a tensor) of input data of the neural network layer. Furthermore, the method includes a step S1020 of clipping the value of the integer included in the data entity of the input data to the defined lower integer threshold if the value of the integer included in the data entity of the input data is smaller than the defined lower integer threshold, and a step S1020 of clipping the value of the integer included in the data entity of the input data to the defined upper integer threshold if the value of the integer included in the data entity of the input data is larger than the defined upper integer threshold, in order to avoid integer overflow of the accumulator register. By the method 1000 as illustrated in FIG. 10, when performing the same procedure with the same input data on two devices / platforms, the same results can be obtained for both devices / platforms since integer overflow can be avoided by employing a clipping procedure.

[0255] The method illustrated in FIG. 10 can be implemented in any kind of neural network, for example, a neural network including one or more fully connected neural network layers or one or more convolutional neural network layers. In particular, a neural network 1100 illustrated in FIG. 11 is provided. The neural network 1100 can utilize the method 1000. The neural network 1100 includes a unit 1110 for scaling, rounding, and clipping. The unit 1110 may be divided into subunits configured for different operations, such as scaling, rounding, and clipping of input data, respectively, and the on and off operation of the first unit and each subunit may be switchable, respectively. The unit 1110 may be configured to receive real-valued input data (data including real numbers) and convert it to integer-valued input data by multiplying the input data value by a scaling factor and rounding the multiplication result to the next integer as described above. The resulting integer value is clipped to integer lower and upper thresholds as necessary.

[0256] The merged and clipped data is input to the neural network layer 1120. Downstream of the neural network layer 1120, a unit for unscaling 1130 is provided. The unit for unscaling 1130 can divide the output of the neural network layer 1120 by a scaling factor. The output of the unit for unscaling 1130 may be input to the activation function 1140. Alternatively, the output of the neural network layer 1120 may be directly input to the activation function 1140, and the unit for unscaling 1130 is provided downstream of the activation function 1140 and divides its output. Alternatively, the unscaling by the unit for unscaling 1130 is performed partially on the output of the neural network layer 1120 (by a first factorization part of the scaling factor) and then partially on the output of the activation function 1140 (by a second factorization part of the scaling factor).

[0257] In particular, an embodiment is provided for a method 1200 for operating a neural network based on adjusted weights as illustrated in Fig. 12. The neural network includes a neural network layer including or connected to an accumulator register having a predefined accumulator register size for buffering summation results. The method 1200 includes a step S1210 of defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in a data entity (e.g., a number, a vector or a tensor) of input data of the neural network layer. Furthermore, the method includes a step S1220 of clipping the integer value included in the data entity of the input data to the defined lower integer threshold if the integer value included in the data entity of the input data is smaller than the defined lower integer threshold, and a step S1220 of clipping the integer value included in the data entity of the input data to the defined upper integer threshold if the integer value included in the data entity of the input data is larger than the defined upper integer threshold. Further, the method includes a step S1230 of determining integer-valued weights (i.e., including integers, e.g., integer-only weights) of the neural network layers based on a defined integer lower threshold, a defined integer upper threshold, and a predefined accumulator register size, such that integer overflow of the accumulator register can be avoided.

[0258] By the method 1200 as illustrated in Fig. 12, when performing the same procedure with the same input data on two devices / platforms, the same result can be obtained for both devices / platforms. In particular, in the context of entropy model-based coding and / or data, e.g., image data compression and decompression, it is an important issue to provide substantially bit-exact processing results on each of the encoding side and the decoding side to provide the same or complementary technical effect that can be obtained by the method 1200 illustrated in Fig. 12.

[0259] In particular, according to one embodiment, a method 1300 of operating a neural network (see FIG. 13) is provided. The method 1300 comprises a step 1310 of implementing, as an activation function of at least one neural network layer, an approximation function of a mathematically defined real-valued nonlinear activation function, the approximation function allowing processing of only integers of a fixed-point representation of the input values ​​of the approximation function. The approximation function may comprise at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined nonlinear activation function may be selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function. In particular, the approximated activation function may be

number

number

[0260] The provision of the above-mentioned approximated (approximate) activation functions can enable bit-exact reproduction of important numerical operations on the encoder side and on the decoder side, respectively, since in particular the calculation of exponential functions can be avoided so that the technical effects obtained by these numerical operations are the same or complementary to each other.

[0261] According to one embodiment, a neural network 1400 is provided as illustrated in Fig. 14. The neural network 1400 includes at least one neural network layer 1410 and an activation function 1420 connected to the output of the at least one neural network layer 1410, the activation function 1420 being implemented as an approximation function of a mathematically defined real-valued nonlinear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing processing only integers of a fixed-point representation of the input values ​​of the approximation function.

[0262] At least some of the steps of the methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13 may be included in a method for encoding or decoding at least a part of data, e.g. an image. According to an embodiment, the encoding or decoding is based on an entropy model providing statistical (probability) properties of the symbols to be encoded or decoded, e.g. mean, variance, (cross)correlation, etc. The entropy model may be provided by a hyper-prior distribution of the variational autoencoder, an autoregressive prior distribution of the variational autoencoder, or a combination of a hyper-prior distribution and an autoregressive prior distribution of the variational autoencoder. The employment of the methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13 for data coding, e.g. image coding, may prove advantageous with regard to high-quality reconstruction of the encoded / compressed data without suffering serious corruption.

[0263] The methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13 may be implemented in an apparatus 1500 as shown in Figure 15, which may be configured to perform steps of these methods. According to an embodiment, the apparatus 1500 includes a processing circuit configured to perform steps of the methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13. Furthermore, the apparatus 1500 may include at least one of the neural networks illustrated in Figures 9, 11 and 14. The apparatus 1500 may be included in an encoder (e.g., the encoder 20 shown in Figures 16 and 17) or a decoder (e.g., the decoder 20 shown in Figures 16 and 17), or may be included by the video coding device 8000 shown in Figure 18 or the apparatus 9000 shown in Figure 19.

[0264] The apparatus 1500 may be an apparatus for encoding or decoding data, for example at least a portion of an image. Furthermore, the apparatus 1500 may include or be included in a (variational) autoencoder or a transformer as described above.

[0265] Although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking or parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0266] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the appended figures do not necessarily require the particular order or sequence depicted to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0267] Some exemplary implementations in hardware and software A corresponding system in which the above-mentioned encoder-decoder processing chain can be deployed is illustrated in FIG. 16. FIG. 16 is a schematic block diagram illustrating an example coding system, e.g., video, image, audio, and / or other coding system (or short coding system), that can utilize the techniques of the present application. The video encoder 20 (or short encoder 20) and the video decoder 30 (or short decoder 30) of the video coding system 10 represent examples of devices that can be configured to perform the techniques according to various examples described in the present application. For example, the video coding and decoding may employ neural networks that can be distributed and that can apply the above-mentioned bitstream analysis and / or bitstream generation to communicate feature maps between distributed computational nodes (two or more).

[0268] As shown in FIG. 16, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding, for example, encoded picture data 13.

[0269] The source device 12 includes an encoder 20, and may additionally, i.e. optionally, include a picture source 16, a pre-processor (or pre-processing unit) 18, for example a picture pre-processor 18, and a communications interface or unit 22.

[0270] Picture source 16 may include or be any kind of picture capture device, such as a camera for capturing real-world pictures, and / or any kind of picture generation device, such as a computer graphics processor for generating computer-animated pictures, or any kind of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any kind of memory or storage device that stores any of the aforementioned pictures.

[0271] In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17 .

[0272] The pre-processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It will be understood that the pre-processing unit 18 may be an optional component. It is noted that the pre-processing may also employ a neural network (as shown in any of Figures 1 to 7) using presence indicator signaling.

[0273] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0274] Communications interface 22 of source device 12 may be configured to receive encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via communications channel 13 to another device, such as destination device 14 or any other device, for storage or direct reconstruction.

[0275] The destination device 14 includes a decoder 30 (e.g., a video decoder 30), and may additionally, i.e. optionally, include a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0276] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. an encoded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0277] The communications interface 22 and the communications interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communications link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network or any combination thereof, any type of private and public network, or any type of combination thereof.

[0278] The communications interface 22 may be configured, for example, to package the encoded picture data 21 in a suitable format, e.g., packets, and / or process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0279] Communications interface 28, forming a counterpart of communications interface 22, may for example be configured to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or unpackaging to obtain encoded picture data 21.

[0280] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrows for communication channel 13 in FIG. 16 pointing from source device 12 to destination device 14, or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, e.g., to set up connections, and to ascertain and exchange any other information related to the communication link and / or data transmission, e.g., encoded picture data transmission. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded pictures 31.

[0281] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display, e.g., by a display device 34.

[0282] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, for example for displaying the picture to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed picture, for example an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0283] 16 illustrates source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0284] As will be apparent to a person skilled in the art based on the description, the presence and (exact) division of functions or features of different units within source device 12 and / or destination device 14 as shown in FIG. 16 may vary depending on the actual device and application.

[0285] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both the encoder 20 and the decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including a neural network or portions thereof. The decoder 30 may be implemented via processing circuitry 46 to embody any coding system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and may execute instructions in hardware to perform the technology of the present disclosure using one or more processors. Either the video encoder 20 and the video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG. 17.

[0286] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiving device, a broadcast transmitting device, and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0287] In some cases, the video coding system 10 illustrated in FIG. 16 is only an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between encoding and decoding devices. In other examples, data may be retrieved from local memory or streamed over a network. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data in memory and / or retrieve data from memory and decode it.

[0288] 18 is a schematic diagram of a video coding device 8000 according to one embodiment of the present disclosure. The video coding device 8000 is suitable for implementing embodiments of the disclosure described herein. In one embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 16, or an encoder, such as the video encoder 20 of FIG. 16.

[0289] The video coding device 8000 includes a receive port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing the data, a transmitter unit (Tx) 8040 and a transmit port 8050 (or output port 8050) for transmitting data, and a memory 8060 for storing data. The video coding device 8000 may also include optical-electrical (OE) and electrical-optical (EO) components coupled to the receive port 8010, the receiver unit 8020, the transmitter unit 8040, and the transmit port 8050 for transmitting or receiving optical or electrical signals.

[0290] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGA, ASIC, and DSP. The processor 8030 communicates with the receiving port 8010, the receiver unit 8020, the transmitter unit 8040, the transmitting port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the above-described embodiments of the disclosure. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the neural network-based codec 8070 provides a substantial improvement to the functionality of the video coding device 8000 and provides the transformation of the video coding device 8000 into different states. Alternatively, the neural network based codec 8070 is implemented as instructions stored in memory 8060 and executed by the processor 8030.

[0291] The memory 8060 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0292] FIG. 19 is a simplified block diagram of an apparatus that may be used as either or both of source device 12 and destination device 14 from FIG. 16, according to an example embodiment.

[0293] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device, or multiple devices, now existing or later developed, capable of manipulating or processing information. Although implementations of the disclosure may be implemented using a single processor as shown, e.g., processor 9002, advantages in speed and efficiency may be achieved using multiple processors.

[0294] The memory 9004 in the apparatus 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 that is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010, which include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1 through N, which further include a video coding application that performs the methods described herein.

[0295] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with touch-sensing elements operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0296] Although shown here as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, the secondary storage may be directly coupled to other components of the device 9000 or may be accessed over a network, and may include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, the device 9000 may be implemented in a wide variety of configurations. [Explanation of symbols]

[0297] 1 bit stream, 2 input bit stream, 10 video coding system, 11 part of input image, 12 source device, 13 encoded picture data, communication channel, 14 destination device, 16 picture source, 17 raw picture data, 18 picture preprocessor, preprocessing unit, 19 picture data, 20 video encoder, short encoder, 21 encoded picture data, 22 communication interface, communication unit, 28 communication interface, communication unit, 30 video decoder, short decoder, 31 decoded picture data, 32 postprocessor, postprocessing unit, 33 picture data, 34 display device, 46 processing circuit, 101 encoder, transformation unit, module, 102 quantizer, module, 103 hyperencoder / decoder, hyper prior distribution, module, 104 decoder, module, 105 arithmetic encoder, arithmetic encoding module, 106 arithmetic decoder, arithmetic decoding module, 108 module, 109 module, 110 module, 121 NN encoder, unit, 122 quantizer, 123 hyperencoder, 125 arithmetic encoding module, 127 hyperdecoder, 147 hyperdecoder, 210 encoder side, 220 encoder sub-network, reduction side sub-network, 230 encoding, code, 250 decoder side, 260 decoder sub-network, reconstruction side sub-network, 322 quantizer, 401 downsampling layer, 402 downsampling layer, 403 downsampling layer, 404 downsampling layer, 405 downsampling layer, 406 downsampling layer, 407 upsampling layer, 408 upsampling layer, 409 upsampling layer, 410 upsampling layer, 411 upsampling layer, 412 upsampling layer, 413 building blocks, 414 input image, 415 building blocks, 420 further layers, 430 convolutional layer, 510 Mobile side, mobile part, 520 quantization layer, 550 hidden layer, 560 inverse quantization layer, 590 cloud side, 700 transformer, 710 neural layer, 720Linear projection layer, 730 positional encoding layer, 740 linear backprojection layer, 810 neural layer, 1000 method, 1100 neural network, 1110 unit for scaling, rounding and clipping, 1120 neural network layer, 1130 unit for unscaling, 1140 activation function, 1200 method, 1300 method, 1400 neural network, 1410 neural network layer, 1420 activation function, 1500 apparatus, 1510 processing circuit, 8000 video coding device, 8010 input port, 8010 receiving port, 8020 receiver unit, 8030 processor, 8040 transmitter unit, 8050 transmitting port, output port, 8060 memory, 8070 codec, 9000 apparatus, 9002 processor, 9004 memory, 9006 data, 9008 Operating system, 9010 Application program, 9012 Bus, 9018 Display

Claims

1. A neural network (1400), At least one neural network layer (1410); an activation function (1420) connected to the output of the at least one neural network layer (1410); Including, A neural network (1400) in which the activation function (1420) is implemented as an approximation function of a mathematically defined real-valued nonlinear activation function, the approximation function allowing processing of only integers in a fixed-point representation of input values ​​of the approximation function.

2. The neural network (1400) of claim 1 , wherein the approximation function comprises at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function.

3. 3. The neural network (1400) of claim 1 or 2, wherein the mathematically defined nonlinear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function.

4. 3. The neural network of claim 2, wherein the approximation function includes a finite number of Taylor series summands determined based on at least one of: a) an expected input value of a data input to the at least one neural network layer or the approximation function; b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result; and c) the approximation function.

5. 3. The neural network (1400) of claim 2, wherein the approximation function is a polynomial function, and a maximum degree of the polynomial function is determined based on at least one of: a) an expected input value of a data input to the at least one neural network layer (1410) or the approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result, and c) the approximation function.

6. 3. The neural network (1400) of claim 2, wherein the approximation function is a rational function, and a maximum degree of a polynomial in a numerator and a denominator of the rational function is determined based on at least one of: a) an expected input value of a data input to the at least one neural network layer (1410) or the approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result, and c) the approximation function.

7. Taylor series of the mathematically defined nonlinear activation function f(x) [0010] Regarding [0025] 7. The neural network (1400) of claim 2 or any one of claims 4 to 6, wherein k represents the number of summands of a Taylor series used to approximate the mathematically defined nonlinear activation function f(x) and n represents the bit depth of an accumulator register included in the at least one neural network layer (1410).

8. The mathematically defined nonlinear activation function is the Softmax function, and the approximation function is [0030] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

9. The mathematically defined nonlinear activation function is the Softmax function, and the approximation function is [0045] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

10. The neural network (1400) of any one of claims 1 to 9, wherein the neural network (1400) is a convolutional or fully connected neural network.

11. The neural network (1400) of any one of claims 1 to 10, wherein the neural network layer (1410) is a convolutional or fully connected neural network layer.

12. An apparatus (1500) for encoding data, comprising a neural network (1400) according to any one of claims 1 to 11.

13. An apparatus (1500) for decoding data, comprising a neural network (1400) according to any one of claims 1 to 11.

14. An apparatus (1500) for encoding at least a part of an image, comprising a neural network (1400) according to any one of claims 1 to 11.

15. An apparatus (1500) for decoding at least a part of an image, comprising a neural network (1400) according to any one of claims 1 to 11.

16. The apparatus (1500) of claim 14 or 15, comprising one of: a) a hyper-prior for a variational autoencoder comprising the neural network (1400) of any one of claims 1 to 11; b) an autoregressive prior for a variational autoencoder comprising the neural network (1400) of any one of claims 1 to 11; and c) a combination of a hyper-prior and an autoregressive prior for a variational autoencoder comprising the neural network (1400) of any one of claims 1 to 11, wherein at least one of the hyper-prior and the autoregressive prior for the variational autoencoder.

17. 12. An apparatus (1500) for encoding at least a portion of an image, comprising: a processing circuit (1510) configured to convert tensors representing components of the image into latent tensors, provide an entropy model by a neural network (1400) according to any one of claims 1 to 11, and process the latent tensors based on the provided entropy model to generate a bitstream.

18. An apparatus (1500) for decoding at least a portion of an encoded image, the apparatus (1500) comprising: a processing circuit (1510) configured to provide an entropy model by a neural network (1400) according to any one of claims 1 to 11; to process a bitstream based on the provided entropy model to obtain latent tensors representative of components of the image; and to process the latent tensors to obtain tensors representative of the components of the image.

19. 19. Apparatus according to claim 17 or 18, wherein the components of the image are Y, U or V components or are R, G, B components.

20. A method (1300) of operating a neural network (1400) including at least one neural network layer (1410), comprising: Implementing (S1310) an approximation of a mathematically defined real-valued non-linear activation function as an activation function (1420) of said at least one neural network layer (1410), said approximation function allowing processing only integers of a fixed-point representation of input values ​​of said approximation function; A method (1300).

21. 21. The method (1300) of claim 20, wherein the approximation function comprises at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function.

22. 22. The method (1300) of claim 20 or 21, wherein the mathematically defined non-linear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function.

23. 22. The method (1300) of claim 21, wherein the approximation function includes a finite number of Taylor series summands determined based on at least one of: a) expected input values ​​of data input to the at least one neural network layer (1410) or the approximation function; b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result; and c) the approximation function.

24. 22. The method (1300) of claim 21, wherein the approximation function is a polynomial function, and a maximum degree of the polynomial function is determined based on at least one of: a) an expected input value of a data input to the at least one neural network layer (1410) or the approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result, and c) the approximation function.

25. 22. The method (1300) of claim 21, wherein the approximation function is a rational function, and a maximum degree of polynomials in a numerator and a denominator of the rational function is determined based on at least one of: a) expected input values ​​of data input to the at least one neural network layer (1410) or the approximation function, b) an accumulator register size of an accumulator register included in the neural network for buffering a summation result, and c) the approximation function.

26. Taylor series of the mathematically defined nonlinear activation function f(x) [0050] Regarding [006] 24. The method (1300) of claim 21 or 23, wherein k represents the number of summands of a Taylor series used to approximate the mathematically defined nonlinear activation function f(x) and n represents a bit depth of an accumulator included in the at least one neural network layer (1410).

27. The mathematically defined nonlinear activation function is the Softmax function, and the approximation function is [0070] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

28. The mathematically defined nonlinear activation function is the Softmax function, and the approximation function is [0080] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

29. 24. The method (1300) of any one of claims 17 to 23, wherein the neural network (1400) is a convolutional or fully connected neural network.

30. 30. A method of encoding data comprising the steps of a method (1300) for operating a neural network (1400) according to any one of claims 20 to 29.

31. 31. The method of claim 30, wherein the encoding of the data comprises providing an entropy model by a neural network (1400) and entropy encoding the data based on the provided entropy model, and wherein providing the entropy model comprises performing steps of a method (1300) for operating a neural network (1400) according to any one of claims 20 to 29.

32. 32. The method of claim 31 , wherein the entropy model is provided by one of: a) a hyper-prior distribution of a variational autoencoder; b) an autoregressive prior distribution of a variational autoencoder; and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder.

33. 30. A method for decoding encoded data comprising the steps of a method (1300) for operating a neural network (1400) according to any one of claims 20 to 29.

34. 34. The method of claim 33, wherein the decoding of the data comprises providing an entropy model by a neural network (1400) and entropy decoding the data based on the provided entropy model, and wherein entropy decoding the data comprises the steps of a method (1300) for operating a neural network (1400) according to any one of claims 20 to 29.

35. 35. The method of claim 34, wherein the entropy model is provided by one of: a) a hyper-prior distribution of a variational autoencoder; b) an autoregressive prior distribution of a variational autoencoder; and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder.

36. 36. The method of any one of claims 30 to 35, wherein the data is image data.

37. 1. A method of encoding at least a portion of an image, comprising the steps of: converting tensors representing components of the image into latent tensors; Providing an entropy model; processing the latent tensors by a neural network (1400) based on the provided entropy model to generate a bitstream; Including, 30. A method, wherein the step of providing an entropy model comprises performing the steps of the method of any one of claims 20 to 29.

38. 38. The method of claim 37, wherein the entropy model is provided by one of: a) a hyper-prior distribution of a variational autoencoder; b) an autoregressive prior distribution of a variational autoencoder; and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder.

39. 1. A method for reconstructing at least a portion of an image, comprising the steps of: Providing an entropy model; processing the bitstream by a neural network (1400) based on the provided entropy model to obtain latent tensors representing components of the image; processing the latent tensors to obtain tensors representing the components of the image; Including, 30. A method, wherein the step of providing an entropy model comprises performing the steps of the method of any one of claims 20 to 29.

40. 1. A method for reconstructing at least a portion of an image, comprising the steps of: Providing an entropy model; processing the bitstream by a neural network (1400) based on the provided entropy model to obtain latent tensors representing components of the image; processing the latent tensors to obtain tensors representing the components of the image; Including, 30. A method, wherein processing the latent tensor to obtain tensors representing the components of the image comprises performing the steps of the method of any one of claims 20 to 29.

41. 41. The method of claim 40, wherein the entropy model is provided by one of: a) a hyper-prior distribution of a variational autoencoder; b) an autoregressive prior distribution of a variational autoencoder; and c) a combination of a hyper-prior distribution and an autoregressive prior distribution of a variational autoencoder.

42. 42. A method according to any one of claims 37 to 41, wherein the component is a Y, U or V component or an R, G, B component.

43. 43. A computer program product comprising program code stored on a non-transitory medium, the program being configured to perform the method of any one of claims 20 to 42 when executed on one or more processors.

44. An activation function (1420) for a neural network layer (1410) of a neural network (1400), comprising: [0090] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

45. An activation function (1420) for a neural network layer (1410) of a neural network (1400), comprising: [0010] where i represents the i-th component of an input vector x, the sum is over all K components of the input vector, and ε represents a positive constant that avoids division by zero.

Images