Behavior of neural networks with clipped input data

Abstract

The present disclosure relates to a method for operating a neural network with clipped input data, the method including: defining lower and upper integer thresholds for integer values ​​contained in data entities of input data for at least one neural network layer; clipping the integer values ​​contained in the data entities of input data to the defined lower integer threshold if the integer values ​​contained in the data entities of input data are less than the defined lower integer threshold; and clipping the integer values ​​contained in the data entities of input data to the defined upper integer threshold if the integer values ​​contained in the data entities of input data are greater than the defined upper integer threshold, such that integer overflow of an accumulator register is avoided.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for so encoding and decoding images and / or video from a bitstream using multiple processing layers. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, the signal is typically encoded block-by-block by predicting the block and further coding only the difference between the original block and its prediction. In particular, such coding may involve transforming, quantizing, and generating a bitstream that usually includes some entropy coding. Typically, the three components of a hybrid coding method - transforming, quantizing, and entropy coding - are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use the transformed representation to code the residual signal after prediction.

[0003] Recently, neural network architectures have been applied to image and / or video coding. In general, these neural network (NN)-based techniques can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize several parts of the end-to-end coding frameworks, such as prediction parameter selection or compression. Furthermore, several neural network-based techniques have also been discussed for use in hybrid image and video coding frameworks, for example, for implementation as trained deep learning models for intra- or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications mentioned above have in common that they generate some feature map data to be communicated between the encoder and the decoder.

[0005] A neural network is a machine learning model that uses one or more layers of underlying nonlinear units that can predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map may be provided as an output of each hidden layer. Such corresponding feature map of each hidden layer may be used as an input to a subsequent layer in the network, i.e., a subsequent hidden layer or an output layer. Each layer of the network generates an output from a received input according to the current value of a respective set of parameters. In a neural network that is split between different devices, for example, between an encoder and a decoder, or between a device and a cloud, the feature map at the output of the site of the split (e.g., a first device) is compressed and transmitted to the remaining layers of the neural network (e.g., to a second device).

[0006] Further improvements in encoding and decoding using the trained network architecture may be desirable. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure provides a method and apparatus for improving device interoperability of devices / platforms of different architectures with neural networks. Device interoperability means that executing the same procedure with the same input data on two devices / platforms gives the same result for both devices / platforms. In particular, in the context of coding and / or compression and decompression of data based on an entropy model, e.g., image data, it is a crucial issue to provide substantially bit-accurate processing results on the encoding side and the decoding side, respectively, to provide the same or complementary technical effect. [Means for solving the problem]

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0009] Particular implementations are outlined in the accompanying independent claims, while other implementations are defined in the dependent claims.

[0010] According to a first aspect, the present disclosure relates to a method for operating a neural network comprising at least one neural network layer comprising or connected to an accumulator register for buffering a summation result, the method comprising defining lower and upper integer thresholds for values ​​of integers contained in a data entity of input data for said at least one neural network layer, the method further comprising clipping the values ​​of integers contained in the data entity of input data to the defined lower integer threshold if the values ​​of integers contained in the data entity of input data are smaller than the defined lower integer threshold, and clipping the values ​​of integers contained in the data entity of input data to the defined upper integer threshold if the values ​​of integers contained in the data entity of input data are larger than the defined upper integer threshold, so as to avoid integer overflow of the accumulator register.

[0011] According to the method of the first aspect, due to clipping of integer-valued data (including integers, e.g., only integers) input to the neural network layers, interoperability between different platforms / devices can be significantly improved compared to the prior art. Clipping allows bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, such that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, a region of an image (a still image or a frame of a video sequence) can be (entropy) coded at the encoder side and reconstructed without corruption at the decoder side, while integer overflows, which can lead to non-deterministic behavior on different platforms, can be reliably avoided. In general, there is no standardized way of handling overflow situations, and one can hardly imagine a commonly followed procedure in this regard. Integer overflow occurs when an arithmetic operation attempts to create a number that is outside the range that can be represented with a given number of digits (greater than the maximum representable value or smaller than the minimum representable value). This situation can be handled differently for different compilers, devices (CPUs, GPUs), etc. In order to achieve bit-accurate results of integer arithmetic on different platforms, integer overflow should be avoided. According to the method of the first aspect, integer overflow of accumulator registers can be avoided on both the encoder side and the decoder side. Furthermore, the internal operation (state) of the involved devices is defined in substantially the same way by providing clipping of input data values ​​according to the method of the first aspect.

[0012] According to an implementation, the method of the first aspect further comprises scaling the data entities of the input data (having real or integer numbers) by a first scaling factor (i.e. the data entities of the input data are multiplied by the first scaling factor) to obtain scaled values ​​of the data entities of the input data. The scaling may be performed to improve the handling of the input data values ​​on different devices. In particular, the scaling may be supplemented by rounding the scaled values ​​of the data entities of the input data to the respective nearest integer value to obtain the integer values ​​contained in the data entities of the input data. The rounding may be performed by a floor or ceiling function, so that for example real numbers of the input data that may originally be provided can be processed by the disclosed method of operation of a neural network.

[0013] The output data may be obtained by processing the input data by the at least one neural network layer, and when a first scaling factor is applied, according to an implementation, the output data entities contained by the output data are divided by a third scaling factor to obtain an unscaled result. Unscaling may be performed directly on the output of the at least one neural network layer or after processing by an activation function. Thus, according to an implementation, the input data is processed by the at least one neural network layer to obtain output data containing output data entities, the output data entities are processed by an activation function to obtain an output of the activation function, and the output of the activation function is divided by the third scaling factor.

[0014] According to another implementation, the method of the first aspect further comprises the steps of processing the input data by the at least one neural network layer to obtain output data including an output data entity, factoring a third scaling factor into a first part and a second part, dividing the output data entity by the first part of the factored third scaling factor to obtain a partially scaled out output data entity, processing the partially scaled out output data entity through an activation function to obtain an output of the activation function, and dividing the output of the activation function by the second part of the factored third scaling factor. Since all options of descaling can be implemented, the descaling process can be performed with high flexibility.

[0015] The above-mentioned integer lower threshold used in the method of the first aspect may be less than or equal to 0, and the integer upper threshold may be greater than or equal to 0. Suitable possible choices for the thresholds employed are -2 for the integer lower threshold and k-1 , 2 for the upper integer threshold k-1 -1, where k denotes the predefined bit depth (bit size) of the input data.

[0016] The method of the first aspect may be advantageously applied to any kind of neural network and neural network layer. According to one implementation, the at least one neural network layer is or includes one of a fully connected neural network layer and a convolutional neural network layer. Furthermore, the at least one neural network layer may include an attention mechanism (see detailed description below).

[0017] According to one implementation, the at least one neural network layer includes integer-valued weights (integers, e.g., weights that include only integers), which can further improve platform / device interoperability and further reduce the risk of integer overflow of accumulator register sizes (see also below).

[0018] According to one implementation, weights, real-valued weights (e.g., weights including only real numbers) or integer-valued weights are originally provided and scaled by a second scaling factor to obtain scaled weights, and the scaled weights are rounded to their nearest integer value to obtain integer-valued weights of the at least one neural network, such that the method of the first aspect may be used, for example, with originally provided (e.g., user-dependent) real-valued weights.

[0019] The second scaling factor is

number

number

[0020] According to another implementation, the second scaling factor s for the j-th output channel of the at least one neural network layer is j teeth,

number

[0021] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0022] According to another implementation, the condition is

number

[0023] As already mentioned, the method of the first aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as mentioned above. In particular data coding and decoding based on entropy coding represents a delicate process where interoperability of at least some of the procedures executed on different devices should give the same results with respect to numerical precision. Thus, according to the second aspect, a method of encoding data is provided (with the same advantages as mentioned above) comprising the steps of the method of operating a neural network according to the first aspect or any implementation thereof. In an implementation of the method of the second aspect, encoding the data comprises the steps of providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, the step of providing the entropy model comprising performing the steps of the method of operating a neural network according to the first aspect or any implementation thereof. The entropy encoding may comprise entropy encoding by an arithmetic encoder.

[0024] In the context of entropy coding, interoperability between different platforms is essential for reliable reconstruction of the (compressed) data in terms of virtually bit-accurate reproduction of the entropy model used both on the encoder and decoder side.

[0025] The entropy model provides statistical properties of the symbols to be encoded or decoded, such as mean, variance, (cross)correlation, etc., and may be provided by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder. Thus, the method of the second aspect may advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0026] The method of encoding data may further include signaling the defined lower threshold and the defined upper threshold to a decoder side, so that the decoder side can be easily provided with the information required for the same processing performed by the encoder side regarding the clipping procedure of the input data. Similarly, at least one of the first scaling factor, the second scaling factor and the third scaling factor may be signaled to the decoder side in the bitstream.

[0027] Further, the method may include signaling the difference from the predefined lower threshold and the difference from the predefined upper threshold to the decoder side. Further, the method may include signaling at least one of the difference from the predefined first scaling factor and the difference from the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0028] According to a third aspect (as a complement to the second aspect), there is provided (with the same advantages as above) a method of decoding encoded data, comprising the steps of the method of operating a neural network according to the first aspect or any implementation thereof. According to one implementation of the third aspect, the decoding of the data comprises providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, where entropy decoding the data comprises the steps of the method of operating a neural network according to the first aspect or any implementation thereof. The entropy decoding may comprise entropy decoding by an arithmetic decoder. Also, on the decoder side, according to one implementation of the third aspect, the entropy model is provided by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of the variational autoencoder.

[0029] Information about the defined lower threshold and the defined upper threshold may be received in the bitstream at the decoder side from the encoder side. Similarly, information about at least one of the first scaling factor, the second scaling factor and the third scaling factor may be received in the bitstream at the decoder side from the encoder side.

[0030] Further, the method may include receiving at the decoder side information regarding a difference from a predefined lower threshold and a difference from a predefined upper threshold.Further, the method may include receiving at the decoder side information regarding at least one of a difference from a predefined first scaling factor and a difference from a predefined second scaling factor in the bitstream.

[0031] The data processed by the methods of the second or third aspect may be, for example, image data representing a still image or a frame of a video sequence.

[0032] According to a fourth aspect, there is provided a method of encoding at least a part of an image, comprising: converting tensors representing components of the image into latent tensors; providing an entropy model; and processing the latent tensors by a neural network based on the provided entropy model to generate a bitstream, wherein providing the entropy model comprises performing the steps of the first aspect or a method of implementation thereof (with the same advantages as described above). The latent tensors may be processed by an arithmetic encoder. Processing the latent tensors to obtain tensors representing components of the image may comprise performing the steps of the method according to the first aspect or any of its implementations.

[0033] The entropy model can be provided by one of a) the hyperprior of a variational autoencoder, b) the autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of a variational autoencoder.

[0034] According to one implementation, the method of the fourth aspect further includes signaling the defined lower threshold and the defined upper threshold in the bitstream to the decoder side. sj where s j represents the real fractional part of a real-valued weight (note that bases other than 2 are possible).

[0035] Furthermore, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be signaled to the decoder side in the bitstream.

[0036] Further, the method may include signaling the difference from the predefined lower threshold and the difference from the predefined upper threshold to the decoder side. Further, the method may include signaling at least one of the difference from the predefined first scaling factor and the difference from the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0037] According to a fifth aspect, there is provided a method for reconstructing at least a part of an image, the method comprising the steps of providing an entropy model, processing the bitstream by a neural network based on the provided entropy model to obtain latent tensors representative of components of the image, and processing the latent tensors to obtain tensors representative of components of the image, the step of providing the entropy model comprising performing (with the same advantages as above) the steps of the method for operating a neural network according to the first aspect or any of its implementations. The processing of the latent tensors to obtain tensors representative of components of the image may comprise performing the steps of the method according to the first aspect or any of its implementations. The processing of the bitstream may be performed by an arithmetic decoder. The entropy model may be provided by one of a) a hyper-prior of a variational autoencoder, b) an auto-regressive prior of a variational autoencoder, and c) a combination of the hyper-prior and the auto-regressive prior of a variational autoencoder. Furthermore, the processing of the latent tensors to obtain tensors representative of components of the image may comprise performing the steps of the method according to the twenty-first aspect or its implementations.

[0038] According to an implementation, the method of the fifth aspect includes reading information about the defined lower threshold and the defined upper threshold from the bitstream. Further, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be read from the bitstream.

[0039] The components of an image represented by a tensor according to the methods of the third and fourth aspects may be Y, U, or V components, or alternatively R, G, B components.

[0040] According to a sixth aspect, the present disclosure relates to a method for operating a neural network comprising a neural network layer comprising or connected to an accumulator register for buffering a summation result, the accumulator register having a predefined accumulator register size. The method includes defining a lower integer threshold A and an upper integer threshold B for integer values ​​contained in data entities (e.g., numbers, vectors, or tensors) of input data for the neural network layer. The method further includes clipping the integer values ​​contained in the data entities of the input data to the defined lower integer threshold if the integer values ​​contained in the data entities of the input data are smaller than the defined lower integer threshold, and clipping the integer values ​​contained in the data entities of the input data to the defined upper integer threshold if the integer values ​​contained in the data entities of the input data are larger than the defined upper integer threshold. The method further includes determining integer weights (i.e., weights containing integers, e.g., only integers) of the neural network layer based on the defined lower integer threshold, the defined upper integer threshold, and the predefined accumulator register size, such that integer overflow of the accumulator register can be avoided.

[0041] According to the method of the sixth aspect, the integer weights of the neural network layers of the neural network are trimmed such that integer overflow of the accumulator registers can be avoided if the input data is clipped to a defined threshold. This allows for a significant improvement in interoperability between different platforms / devices compared to the prior art. The trimming of the weights allows for bit-exact reproduction of key numerical operations at the encoder and decoder side, respectively, such that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, a region of an image (a still image or a frame of a video sequence) can be (entropy) encoded at the encoder side and reconstructed without corruption at the decoder side if integer overflow of the accumulator registers can be avoided on both sides. Furthermore, the internal operation (state) of the involved devices is defined in a substantially similar manner by providing trimmed weights determined according to the method of the sixth aspect. The integer weights can be determined in different ways such that integer overflow of the accumulator registers can be avoided, as described below, and the present disclosure is not limited to one of these specific ways.

[0042] The method of the sixth aspect works for any kind of neural network layer, including fully connected neural network layers and convolutional neural network layers. The method of the sixth aspect is also advantageously implemented in a Transformer architecture, where the neural network layer includes some attention mechanism (see also the detailed description below). The accumulator register size may be n bits, where n is a positive integer value, e.g., n=32 bits or n=16 bits. The stored values ​​may be -2 n-1 ~2 n-1 In the range of -1 or 0 to 2 n-1 The accumulator register size may be a fixed size or may be dynamically allocated.

[0043] According to one implementation, the lower integer threshold is less than or equal to 0, and the upper integer threshold is greater than or equal to 0. Thus, input data can be constrained to non-negative integer values, or negative integer input values ​​may be allowed as deemed appropriate for practical applications.

[0044] In one implementation, the integer lower threshold A is -2 k-1 and the upper integer threshold B is 2 k-1 −1, where k denotes a predefined bit depth of the layer input data, which is usually known for the particular application and configuration used and can therefore be easily used to define the clipping thresholds applied to the input data.

[0045] The trimmed weights are typically used in the summation process performed by the neural network layer. In one implementation, the neural network layer uses the summation (accumulated in the so-called accumulator registers mentioned above)

number

[0046] A specific concrete formulation for determining the integer weights such that integer overflow of the accumulator registers is avoided is envisioned herein. According to one implementation, the integer weights {w i} is the condition

number

[0047] If the input data value is constrained to be between a defined lower threshold A and an upper threshold B, then integer overflow of the accumulator register can be reliably avoided if these conditions are met. n-1 -1 and -2 n-1 are given only as examples of accumulator register size limits and may be replaced by other limits that properly define the accumulator register size.

[0048] Another implementation uses integer weights {w i} is the condition

number

[0049] Another implementation uses integer weights {w i} is the condition

number

[0050] Sum

number

number

[0051] For a 2-dimensional layer, the sum is

number

number

[0052] For all of these sums, the above conditions must be satisfied for each of the output channels of the neural network layer.

[0053] In particular, the user-dependent application weights of the neural network layers of the neural network may be provided as real-valued weights (i.e. the weights include real numbers, e.g. only real numbers). The sixth aspect and its implementation method can also be used advantageously in this case. In this case, according to an implementation, the provided real-valued weights may be scaled by a first scaling factor to obtain scaled weights, and the scaled weights may be rounded to the respective nearest integer value to obtain integer-valued weights. The scaling factor can be selected with high flexibility. The first scaling factor can be multiplied by 2 sj It seems appropriate to consider the sigma-based approach to be given by j denotes the number of bits that represent the fractional part of the real-valued weight.

[0054] In particular, according to one implementation of the method of the sixth aspect, the first scaling factor s for the j-th output channel of the at least one neural network layer is j is the condition

number

[0055] According to another implementation, The second scaling factor s for the jth output channel of at least one neural network layer j teeth,

number

[0056] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0057] According to another implementation, the condition is

number

[0058] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0059] Similarly, according to one implementation, the method includes scaling data entities of the input data by a second scaling factor to obtain scaled values ​​of the data entities. The scaled values ​​of the data entities may be rounded to their nearest integer values ​​to obtain integer values ​​of the data entities. The rounding may be performed by a floor function or a ceiling function.

[0060] As already mentioned, the method of the sixth aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as mentioned above. Thus, according to the seventh aspect, there is provided a method of encoding data comprising the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. In an implementation of the method of the seventh aspect, encoding the data comprises the steps of providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, the providing of the entropy model comprising performing the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. In the context of entropy coding, interoperability between different platforms in terms of substantially bit-accurate reproduction of the entropy model used on both the encoder side and the decoder side is essential for a reliable reconstruction of the (compressed) data.

[0061] The entropy model provides statistical (probability) properties of the symbols to be encoded or decoded, such as mean, variance, (cross)correlation, etc., and may be provided by one of: a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder. Thus, the method of the seventh aspect may advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0062] According to an eighth aspect (complementary to the seventh aspect), there is provided a method of decoding encoded data comprising the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. According to one implementation of the eighth aspect, decoding the data comprises providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, the entropy decoding of the data comprising the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. Also, on the decoder side, according to one implementation of the eighth aspect, the entropy model is provided by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of the variational autoencoder.

[0063] The data processed by the methods of the seventh or eighth aspect may be, for example, image data representing a still image or a frame of a video sequence.

[0064] According to a ninth aspect, there is provided a method of encoding at least a portion of an image, comprising the steps of converting tensors representing components of the image into latent tensors, providing an entropy model, and processing the latent tensors by a neural network based on the provided entropy model to generate a bitstream, wherein providing the entropy model comprises performing (with the same or similar advantages as described above) the steps of a method according to the sixth aspect or an implementation thereof. According to an implementation of the ninth aspect, the entropy model is provided by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of the variational autoencoder.

[0065] According to a tenth aspect, there is provided a method for reconstructing at least a part of an image, comprising the steps of providing an entropy model, processing a bitstream by a neural network based on the provided entropy model to obtain latent tensors representative of components of the image, and processing the latent tensors to obtain tensors representative of components of the image, wherein providing the entropy model comprises performing (with the same or similar advantages as described above) the steps of a method according to the sixth aspect or an implementation thereof. Again, the entropy model may be provided by one of a) a hyper-prior of a variational autoencoder, b) an auto-regressive prior of a variational autoencoder, and c) a combination of the hyper-prior and the auto-regressive prior of the variational autoencoder.

[0066] The above-mentioned image components represented by tensors may be, for example, Y, U or V components, or alternatively R, G, B components.

[0067] According to an eleventh aspect, there is provided a neural network comprising: a neural network layer configured to process input data to obtain output data; an activation function configured to process the output data to obtain activation function output data; a first unit for scaling, rounding and clipping input data input to the neural network layer; and a second unit for unscaling at least one of the output data and the activation function output data. The first unit may be divided into sub-units configured for different operations, respectively, such as scaling, rounding and clipping of the input data, and the on and off operations of the first unit and each sub-unit may be switchable, respectively.

[0068] The first unit may be configured to perform the steps of the methods according to the first to sixth aspects and implementations thereof.

[0069] According to an implementation, the neural network of the eleventh aspect is configured to perform the steps of the methods according to the first to sixth aspects and implementations thereof.

[0070] According to a twelfth aspect, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network according to the eleventh aspect and an implementation thereof. The apparatus of the twelfth aspect may include one of: a) a hyper prior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of a hyper prior and an autoregressive prior of a variational autoencoder, at least one of which comprises the neural network of the eleventh aspect or an implementation thereof.

[0071] According to a thirteenth aspect, there is provided an apparatus for decoding at least a portion of an encoded image comprising a neural network according to the eleventh aspect and implementations thereof, which may be an apparatus for decoding at least a portion of an image and may include one of a) a hyper prior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of a hyper prior and an autoregressive prior of a variational autoencoder, at least one of which includes the neural network of the eleventh aspect or an implementation thereof.

[0072] To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear and relatively simple nonlinear activation functions, such as the ReLU function that essentially defines the clipping procedure, this requirement is relatively easy to meet. For more complex nonlinearities, in particular

number

[0073] Thus, according to a fourteenth aspect, there is provided a neural network comprising at least one neural network layer and an activation function connected to an output of the at least one neural network layer, the activation function being implemented as an approximation function of a mathematically defined real-valued non-linear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing for integer-only processing of a fixed-point representation of input values ​​of the approximation function.

[0074] In fixed-point representation / arithmetic, a real number is a multiplication factor of its mantissa and its radix. 指数 where both the base and the exponent are fixed, and thus a fixed-point representation of a fraction is essentially an integer (see also the detailed explanation below).

[0075] The approximation function approximates a mathematically defined real-valued non-linear activation function used in the art (which often involves one or more exponential (base e) functions and operates on floating-point representations of the input values), thereby allowing operation only on integers in a fixed-point representation of the input values ​​(as in, for example, the ReLU function).

[0076] Processing with integers only allows fixed-point neural networks in that bit-exact behavior on different platforms can be obtained. This allows for significantly improved interoperability between different platforms / devices compared to the prior art. Providing such approximated (approximate) activation functions allows for bit-exact reproduction of key numerical operations at the encoder and decoder side, respectively, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, a region of an image (a still image or a frame of a video sequence) can be (entropy) encoded at the encoder side and reconstructed without corruption at the decoder side, if bit-exact behavior on both sides is obtained. Also, by providing neural networks including approximated activation functions according to the fourteenth aspect for different devices communicating with each other, the internal operation (state) of the involved devices can be defined in substantially the same way. The approximated activation functions implemented in the neural network of the fourteenth aspect can be selected in different ways, as described below, and the present disclosure is not limited to one of these specific ways.

[0077] According to an implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation of the neural network of the fourteenth aspect, the mathematically defined nonlinear activation function (i.e., the one approximated by the approximated activation function) is selected from the group consisting of a Softmax function, a Sigmoid function, a Hyperbolic Tangent function, a Swish function, a Gaussian Error Linear Unit function, and a Scaled Exponential Linear Unit function. For all of these mathematically defined nonlinear activation functions that may be considered suitable for practical applications, an approximation function can be found that allows working only with integers of a fixed-point representation of the input values.

[0078] According to one implementation, the approximation function (approximated activation function) includes a finite number of Taylor series summation terms determined based on at least one of: a) expected input values ​​of data input to the at least one neural network layer or approximation function; b) an accumulator register size of an accumulator register for buffering summation results included in the neural network; and c) the approximation function.

[0079] According to another implementation, the approximation function is a polynomial function, the maximum degree of which is determined based on at least one of: a) expected input values ​​of data input to the at least one neural network layer or approximation function, b) an accumulator register size of an accumulator register for buffering summation results included in the neural network, and c) the approximation function.

[0080] According to another implementation, the approximation function is a rational function, and a maximum degree of the polynomial in the numerator and denominator of the rational function is determined based on at least one of a) expected input values ​​of data input to the at least one neural network layer or approximation function, b) an accumulator register size of an accumulator register for buffering summation results included in the neural network, and c) the approximation function.

[0081] Taylor series for a mathematically defined nonlinear activation function f(x)

number

number

[0082] Note 2 n-1 -1 and -2 n-1 are only examples for accumulator register size limits and may be replaced by other limits that appropriately define the accumulator register size in these conditions.

[0083] The activation functions that are particularly important in many applications are

number

[0084] According to the neural network implementation of the 14th aspect, the mathematically defined nonlinear activation function is a Softmax function, and the approximation function (the approximated activation function) is

number

[0085] According to another implementation, the approximated activation function is

number

[0086] The positive constant ε can be easily selected, for example, depending on the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11 may be in the range.

[0087] Numerical experiments prove that these kinds of approximation functions for the Softmax function based on the ReLU function can ensure bit-precise behavior of the application of activation functions on different platforms.

[0088] The implementation of the approximated activation function may advantageously be realized in any neural network, in particular the neural network may be a convolutional or fully connected neural network or a convolutional or fully connected neural network layer.

[0089] According to a fifteenth aspect, there is provided (with the same advantages as provided by the neural network of the fourteenth aspect) a method of operating a neural network (e.g. a convolutional or fully connected neural network) comprising at least one neural network layer, the method comprising implementing as an activation function of said at least one neural network layer an approximation function of a mathematically defined real-valued non-linear activation function, the approximation function allowing working only with integers in a fixed-point representation of input values ​​of the approximation function.

[0090] According to one implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation, the mathematically defined nonlinear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function.

[0091] The approximation function may include a finite number of Taylor series sum terms determined based on expected input values ​​of data input to the at least one neural network layer or approximation function.

[0092] Taylor series for a mathematically defined nonlinear activation function f(x)

number

number

[0093] The mathematically defined non-linear activation function may be a Softmax function, in which case the approximation function implemented according to the fifteenth aspect or a method of implementation thereof may be

number

[0094] Alternatively, the approximation function is

number

[0095] The positive constant ε can be easily selected depending on, for example, the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11A second alternative to the approximate activation function, which represents the Softmax function, is the approximation for small values ​​of x close to 0.

number

[0096] The approximate activation functions implemented in the neural network of the 14th aspect and in the method of the 15th aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as described above. Thus, according to the 16th aspect, there is provided a method of encoding data (with the same advantages as described above) comprising the steps of the method of operating a neural network according to the 14th aspect or any implementation thereof. In an implementation of the method of the 16th aspect, encoding the data comprises the steps of providing an entropy model by the neural network and entropy encoding the data based on the provided entropy model, wherein providing the entropy model comprises performing the steps of the method of operating a neural network according to the 15th aspect or any implementation thereof. The entropy encoding may comprise entropy encoding by an arithmetic encoder.

[0097] In the context of entropy coding, interoperability between different platforms in terms of virtually bit-accurate reproduction of the entropy model used on both the encoder and decoder side is essential for reliable reconstruction of the (compressed) data.

[0098] The entropy model provides statistical (probability) properties of the symbols to be encoded or decoded, such as mean, variance, (cross)correlation, etc., and may be provided by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder. Thus, the method of the sixteenth aspect may advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0099] According to a seventeenth aspect (complementary to the sixteenth aspect), there is provided (with the same advantages as above) a method of decoding encoded data comprising the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof. According to one implementation of the seventeenth aspect, the decoding of the data comprises the steps of providing an entropy model by the neural network and entropy decoding the data based on the provided entropy model, the entropy decoding of the data comprising the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding by an arithmetic decoder. Also, on the decoder side, according to one implementation of the seventeenth aspect, the entropy model is given by one of a) a hyperprior of a variational autoencoder, b) an autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of the variational autoencoder.

[0100] The data processed by the method of the sixteenth or seventeenth aspect may be, for example, image data representing a still image or a frame of a video sequence.

[0101] According to an eighteenth aspect, there is provided a method for encoding at least a part of an image, comprising the steps of converting tensors representing components of the image into latent tensors, providing an entropy model, and processing the latent tensors by a neural network based on the provided entropy model to generate a bitstream, wherein providing the entropy model comprises performing (with the same advantages as described above) the steps of the fifteenth aspect or a method of implementation thereof. The latent tensors may be processed by an arithmetic encoder.

[0102] The entropy model can be provided by one of a) the hyperprior of a variational autoencoder, b) the autoregressive prior of a variational autoencoder, and c) a combination of the hyperprior and the autoregressive prior of a variational autoencoder.

[0103] According to a nineteenth aspect, there is provided (with the same advantages as above) a method of reconstructing at least a part of an image, comprising the steps of providing an entropy model, processing the bit stream by a neural network based on the provided entropy model to obtain latent tensors representative of components of the image, and processing the latent tensors to obtain tensors representative of components of the image, wherein providing the entropy model comprises performing steps of a method of operating a neural network according to the fifteenth aspect or any implementation thereof. The processing of the bit stream may be performed by an arithmetic decoder. The entropy model may be provided by one of: a) a hyper-prior of a variational autoencoder, b) an auto-regressive prior of a variational autoencoder, and c) a combination of the hyper-prior and the auto-regressive prior of a variational autoencoder. Furthermore, the processing of the latent tensors to obtain tensors representative of components of the image may comprise performing steps of a method according to the twenty-first aspect or an implementation thereof.

[0104] The components of an image represented by a tensor according to the methods of the eighteenth and nineteenth aspects may be Y, U or V components, or alternatively R, G, B components.

[0105] According to a twentieth aspect, an approximate activation function (which approximates a Softmax function)

number

[0106] According to a twenty-first aspect, an approximate activation function (which approximates a Softmax function)

number

[0107] The positive constant ε can be easily selected depending on, for example, the expected values ​​of the components of the input vector x. The positive constant ε can be, for example, 10 -15 ~10 -11 may be in the range.

[0108] These new activation functions may replace the traditional Softmax function and may prove to be better in terms of device interoperability (such as encoders and decoders).

[0109] According to a twenty-second aspect, there is provided an apparatus for encoding data, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0110] According to a twenty-third aspect, there is provided an apparatus for decoding data comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0111] According to a twenty-fourth aspect, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0112] According to a twenty-fifth aspect, there is provided an apparatus for decoding at least a portion of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0113] According to a 26th aspect there is provided an apparatus according to one of the 22nd to 25th aspects comprising one of a) a hyperprior of a variational autoencoder, the hyperprior comprising a neural network according to the 14th aspect or any implementation thereof; b) a hyperprior of an autoregressive prior of a variational autoencoder, the hyperprior comprising a neural network according to the 14th aspect or any implementation thereof; and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder, at least one of the hyperprior and the autoregressive prior comprising a neural network according to the 14th aspect or any implementation thereof.

[0114] According to a twenty-seventh aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus comprising a processing circuit configured to convert tensors representing components of the image into latent tensors, provide an entropy model by a neural network according to the fourteenth aspect or any implementation thereof, and process the latent tensors based on the provided entropy model to generate a bitstream.

[0115] According to a twenty-eighth aspect, there is provided an apparatus for decoding at least a part of an encoded image, the apparatus comprising a processing circuit configured to provide an entropy model by a neural network according to the fourteenth aspect or any implementation thereof, process the bitstream based on the provided entropy model to obtain latent tensors representative of components of the image, and process the latent tensor to obtain tensors representative of the components of the image.

[0116] The components of an image may be Y, U, or V components, or alternatively R, G, B components.

[0117] According to a twenty-ninth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementations thereof.

[0118] According to a thirtieth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementations thereof.

[0119] According to a thirty-first aspect, there is provided a computer-readable storage medium storing instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to any of the above method-related aspects and any implementations thereof.

[0120] A method according to any of the above method-related aspects and any implementations thereof may be implemented in an apparatus, providing an apparatus configured to perform the steps of such a method (with the same advantages as described above).

[0121] According to a thirty-second aspect, there is provided an apparatus for encoding data, the apparatus comprising processing circuitry configured to perform steps of any method according to any of the above method-related aspects and any implementations thereof, including but not limited to decoding.

[0122] According to a thirty-third aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus comprising a processing circuit configured to convert tensors representing components of the image into latent tensors and provide an entropy model, including but not limited to performing the steps of any of the methods according to any of the above aspects relating to the method and any implementations thereof, and to process the latent tensors by a neural network based on the provided entropy model to generate a bitstream.

[0123] According to a thirty-fourth aspect, there is provided an apparatus for decoding data, the apparatus comprising processing circuitry configured to perform the steps of any of the methods according to any of the above aspects and any implementations thereof, including but not limited to methods for encoding.

[0124] According to a thirty-fifth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus comprising: a processing circuit configured to perform the steps of any of the methods according to any of the above aspects and any implementations thereof relating to a method for encoding, including providing an entropy model; processing the bitstream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image.

[0125] The functions of the above-mentioned devices may be implemented by hardware, or by hardware executing corresponding software.

[0126] According to a further aspect, the present disclosure relates to a video stream encoding device including a processor and a memory storing instructions for causing the processor to perform steps of any of the methods according to any of the above aspects and any implementations thereof, including but not limited to the method of decoding.

[0127] According to a further aspect, the present disclosure relates to a video stream decoding device including a processor and a memory storing instructions for causing the processor to perform steps of any of the methods according to any of the above aspects and any implementations thereof, including but not limited to the encoding method.

[0128] The above-described aspects and implementation methods and apparatus can be readily combined with one another where considered appropriate.

[0129] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, the drawings, and the claims. [Brief description of the drawings]

[0130] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying figures and drawings.

[0131] [Figure 1] FIG. 2 is a schematic diagram showing channels processed by layers of a neural network. [Diagram 2] A schematic diagram showing an autoencoder type of neural network. [Figure 3A] FIG. 1 is a schematic diagram illustrating an exemplary network architecture in which the encoder and decoder sides include a hyper-prior model. [Figure 3B] A schematic diagram showing a general network architecture where the encoder side includes a hyper-prior model. [Figure 3C]A schematic diagram showing a general network architecture where the decoder side includes a hyperprior model. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary network architecture in which the encoder and decoder sides include a hyper-prior model. [Diagram 5] FIG. 1 is a block diagram showing the structure of a cloud-based solution for machine-based tasks, such as machine vision tasks. [Figure 6] 1 is a block diagram illustrating an end-to-end video compression framework based on neural networks. [Figure 7] FIG. 1 is a diagram showing a transformer. [Figure 8] FIG. 2 illustrates neural network layers. [Figure 9] FIG. 2 illustrates a neural network according to an embodiment. [Figure 10] 1 is a flow chart illustrating a method of operating a neural network, including clipping of input data. [Figure 11] FIG. 2 illustrates a neural network with units for clipping, scaling and rounding, and unscaling, according to an embodiment. [Figure 12] 1 is a flow chart illustrating a method of operating a neural network including adjusting neural network layer weights. [Figure 13] 1 is a flowchart illustrating a method of operating a neural network, including implementing an approximate activation function. [Figure 14] FIG. 2 illustrates a neural network including an approximated activation function, according to one embodiment. [Figure 15] FIG. 14 illustrates an apparatus configured to perform the steps of the methods illustrated in FIGS. 10, 12, and 13. [Figure 16] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 17] FIG. 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 18] FIG. 2 is a block diagram showing an example of an encoding device or a decoding device. [Figure 19] FIG. 2 is a block diagram showing another example of an encoding device or a decoding device.

[0132] Like reference numbers and designations in different drawings may indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0133] In the following description, reference is made to the accompanying drawings which form a part of this disclosure and which show by way of example certain aspects of the embodiments of the present disclosure or in which the embodiments of the present disclosure may be used. It is understood that the embodiments of the present disclosure may be used in other aspects and may include structural or logical changes not shown in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0134] For example, it is understood that disclosure related to a described method also applies to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more particular method steps are described, a corresponding device may include one or more units for performing the described one or more method steps (e.g., one unit performing the one or more steps, or multiple units each performing one or more of the multiple steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, when a particular apparatus is described based on one or more units, e.g., a functional unit, a corresponding method may include one step performing the function of the one or more units (e.g., one step performing the function of the one or more units, or multiple steps each performing the function of one or more of the multiple units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that the features of the various exemplary embodiments and / or aspects described herein can be combined with each other, unless otherwise specified.

[0135] The following provides an overview of some of the terminology and frameworks used within which embodiments of the present disclosure may be employed.

[0136] Artificial Neural Networks An artificial neural network (ANN), or connectionist system, is a computing system loosely inspired by the biological neural networks that make up animal brains. Such systems "learn" to perform a task by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, a system can learn to identify images that contain cats by analyzing example images that have been manually labeled as "cat" or "no cat," and then use the results to identify cats in other images. It does this without any prior knowledge of cats, such as that they have fur, tails, whiskers, and cat-like faces. Instead, it automatically generates discriminative characteristics from the examples it processes.

[0137] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely model neurons in a biological brain. Each connection can transmit signals to other neurons, like synapses in a biological brain. An artificial neuron that receives a signal can then process the signal and transmit it to the neurons connected to it.

[0138] In an ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically each have a weight that is adjusted as learning progresses. The weight increases or decreases the strength of the signal at the connection. Neurons may have a threshold so that a signal is sent only if the aggregate signal exceeds that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals go from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0139] The original goal of the ANN approach was to solve problems in the same way that the human brain does. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a variety of tasks, including activities traditionally considered the human specialty, such as computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even painting.

[0140] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution, which is a specialized type of linear operation. Convolutional networks are neural networks that use convolution instead of the more common matrix multiplication in at least one of their layers.

[0141] FIG. 1 shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is the layer where the input (such as a portion of an input image 11 as shown in FIG. 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The output of a layer is one or more feature maps (indicated by empty solid rectangles), sometimes called channels. There may be resampling (such as subsampling) involved in the operation of some or all of the layers. As a result, the feature maps become smaller, as shown in FIG. 1. Note that convolution with strides may also reduce (resampling) the size of the input feature maps. The activation function in a CNN is typically a ReLU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling, fully connected, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called a convolutional layer, but this is only by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index in the matrix in that it affects how the weights are determined at a particular index point.

[0142] When programming a CNN to process images, the input is a tensor with dimensions (number of images) × (image width) × (image height) × (image depth), as shown in Figure 1. It should be known that the image depth can be composed by the channels of the image. After passing through the convolutional layers, the image is abstracted into a feature map with dimensions (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layers in a neural network should have the following attributes: A convolution kernel defined by width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) should be equal to the number of channels of the input feature map (depth).

[0143] In the past, traditional multi-layer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, MLP models suffer from high dimensionality and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed feasibly at scale with full connectivity. Also, such network architectures do not take into account the spatial structure of the data, treating input pixels that are far away from each other the same as pixels that are close to each other. This ignores the locality of reference in the image data, both computationally and semantically. Thus, the full connectivity of neurons is useless for purposes such as image recognition, which are dominated by spatially local input patterns.

[0144] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of the layer consist of a set of learnable filters (kernels, as mentioned above) that have small receptive fields but extend across the full depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns a filter that activates when it detects some particular type of feature at some spatial location in the input.

[0145] The complete output volume of a convolutional layer is formed by stacking the activation maps for all filters along the depth dimension. Thus, every entry in the output volume can also be interpreted as the output of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation for a given filter. Feature map and activation have the same meaning. In some papers it is called an activation map because it is a mapping that corresponds to the activation of different parts of the image, and also a mapping of where in the image certain features are found, so it is called a feature map. High activation means that a feature has been found.

[0146] Another important concept in CNNs is pooling. Pooling is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, among which max pooling is the most common. It divides the input image into a set of non-overlapping rectangles and for each such subregion, outputs the maximum value.

[0147] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus also controlling overfitting. It is common to periodically insert pooling layers between successive convolutional layers in CNN architectures. The pooling operation provides another form of translational invariance.

[0148] A pooling layer operates independently on every depth slice of the input and resizes it spatially. The most common form is a pooling layer with a filter of size 2 × 2 applied by 2 along both the width and height with a stride of 2 per depth slice in the input, which discards 75% of the activations. In this case, each max operation is on 4 numbers. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can use other functions such as average pooling or L2 norm pooling. Average pooling was often used historically but has recently fallen out of favor compared to max pooling, which often performs better in practice. With aggressive reduction in the size of the representation, there is a recent trend towards using smaller filters or discarding the pooling layer altogether. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0149] ReLU stands for rectified linear unit and applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the non-linearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions are also used to increase the non-linearity, for example the saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalization accuracy.

[0150] After several convolutional and max pooling layers, high-level inference in neural networks is performed via fully connected layers. Neurons in fully connected layers have connections to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Thus, their activations can be computed as affine transformations with matrix multiplications followed by bias offsets (vector addition of learned or fixed bias terms).

[0151] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between the predicted (output) labels and the true labels, and is usually the final layer of a neural network. Different loss functions may be used that are suited to different tasks: Softmax loss is used to predict a single class out of K mutually exclusive classes; Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0152] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map containing a number of channels corresponding to the number of filters in the set of learnable filters of this layer. The feature map is then subsampled, for example using a pooling layer, which reduces the dimensionality of each channel in the feature map. Next, the data comes to another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish the connectivity of the network, these parameters need to be synchronized between two connected layers, so that the number of input channels for the current layer is equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., images, the number of input channels is usually equal to the number of channels of the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer(s)) may be passed to an output layer. Such an output layer may be convolutional or resampling in some implementations. In an exemplary and non-limiting implementation, the output layer is a fully connected layer.

[0153] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. Its schematic diagram is shown in Figure 2. An autoencoder includes an encoder side 210 with input x fed into the input layer of an encoder sub-network 220, and a decoder side 250 with output x' fed out of a decoder sub-network 260. The goal of an autoencoder is to learn a representation (encoding) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Together with the reduction (encoder) side sub-network 220, a reconstruction (decoder) side sub-network 260 is trained, where the autoencoder tries to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x. This is where the name comes from. In the simplest case, given one hidden layer, the encoder stage of an autoencoder takes an input x and maps it to h: h = σ(Wx+b) This image h is typically called the code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as the sigmoid function or the rectified linear unit. W is a weight matrix. b is a bias vector. The weights and biases are typically initialized randomly and then updated iteratively during training through backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x. x'=σ'(W'h'+b') Here, σ', W', b' for the decoder may be unrelated to the corresponding σ, W, b for the encoder.

[0154] Variational autoencoder models make strong assumptions about the distribution of the latent variables. They use a variational approach for learning the latent representations, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is then fed into a directed graphical model p θ(x|h), and the encoder estimates the posterior distribution p θ Approximation q for (h|x) φ Let us assume that we are learning (h|x), where φ and θ denote the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than standard autoencoders. The objective function of a VAE has the following form:

number

number

[0155] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have enabled researchers to interest in applying neural network-based techniques to image and video compression tasks. For example, an end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0156] Thus, data compression is considered a fundamental and well-studied problem in engineering, and is usually formulated with the goal of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized to a finite set of discrete values, which introduces errors.

[0157] In this context, known as a lossy compression problem, one must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0158] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space cannot be addressed. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a lossless entropy code. This scheme is called transform coding because of the central role of the transform.

[0159] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method - the transform, the quantizer, and the entropy code - are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use the transformed representation to code the residual signal after prediction. Several transforms are used for that purpose, such as the discrete cosine and sine transform (DCT, DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0160] Variational Image Compression The variable autoencoder (VAE) framework can be considered as a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in Figure 3A, which shows the VAE framework.

[0161] The transformation process can be divided into four main parts. Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (denoted by y) via a function y = f(x). This latent representation may be referred to below as a part or point in the "latent space". The function f() is a transformation function that transforms the input signal x to a more compressible representation y. The quantizer 102 transforms the latent representation y to a (discrete) valued quantized latent representation ^y by ^y = Q(y), where Q represents the quantization function. The entropy model or hyperencoder / decoder (also known as hyperprior) 103 estimates the distribution of the quantized latent representation ^y to obtain the minimum rate achievable in lossless entropy source coding.

[0162] A latent space can be understood as a representation of compressed data, where similar data points are closer to each other in the latent space. Latent spaces are useful for learning data features and for finding simpler representations of data for analysis. The quantized latent representation T, ^y and the side information ^z of the hyperprior 3 are included (binarized) in a bitstream 2 using arithmetic coding (AE). Furthermore, a decoder 104 is provided that converts the quantized latent representation into a reconstructed image ^x, where ^x=g(^y). The signal ^x is an estimate of the input image x. It is desirable that x is as close as possible to ^x, in other words, that the reconstruction quality is as high as possible. However, the higher the similarity between ^x and ^y, the higher the amount of side information that needs to be transmitted. The side information includes bitstream 1 and bitstream 2 shown in FIG. 3A, which are generated by the encoder and transmitted to the decoder. Typically, the higher the amount of side information, the higher the reconstruction quality. However, a higher amount of side information means a lower compression ratio. Therefore, one objective of the system described in FIG. 3A is to balance the reconstruction quality with the amount of side information conveyed in the bitstream.

[0163] In Fig. 3A, component AE 105 is an arithmetic encoding module, which converts samples of the quantized latent representation ^y and the side information ^z into a binary representation bitstream 1. The samples of ^y and ^z may comprise, for example, integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert (through a process of binarization) the sample values ​​into a string of binary digits (which are then included in a bitstream, which may contain further portions corresponding to the encoded image or further side information).

[0164] Arithmetic Decoding (AD) 106 is the process of reversing the binarization process, where the binary numbers are converted back to their original sample values. Arithmetic decoding is provided by an arithmetic decoding module 106.

[0165] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0166] In FIG. 3A, there are two sub-networks connected to each other. A sub-network in this context is a logical division between parts of the overall network. For example, in FIG. 3A, modules 101, 102, 104, 105, 106 are called the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generating) and decoding (parsing) the first bitstream "Bitstream 1". The second network in FIG. 3A includes modules 103, 108, 109, 110, and 107 and is called the "hyperencoder / decoder" sub-network. The second sub-network is responsible for generating the second bitstream "Bitstream 2". The purpose of the two sub-networks is different.

[0167] The first subnetwork is responsible for: Transform an input image x into its latent representation y (which is easier to compress than x)101 ●Quantize the latent representation y into a quantized latent representation ^y. Compress the quantized latent representation ^y using AE via arithmetic encoding module 105 to obtain a bitstream “Bitstream 1” Parse bitstream 1 through AD using arithmetic decode module 106 ● Use the parsed data to reconstruct 104 the reconstructed image (^x).

[0168] The purpose of the second sub-network is to obtain statistical properties of the samples of "Bitstream 1" (e.g., mean, variance and correlation between samples of Bitstream 1) so that the first sub-network can compress Bitstream 1 more efficiently. The second sub-network generates a second bitstream "Bitstream 2" that includes said information (e.g., mean, variance and correlation between samples of Bitstream 1).

[0169] The second network includes an encoding portion that includes converting 103 the quantized latent representation ^y to side information z, quantizing the side information z to quantized side information ^z, and encoding (e.g., binarizing) 109 the quantized side information ^z into a bitstream 2. In this example, the binarization is performed by arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD) 110 that converts the input bitstream 2 into decoded quantized side information ^z'. Since arithmetic encoding and decoding are lossless compression methods, ^z' can be identical to ^z. The decoded quantized side information ^z' is then converted 107 into decoded side information ^y', where ^y' represents a statistical property of ^y (e.g., the mean value of ^y samples or the variance of the sample values). The decoded latent representation ^y' is then provided to the arithmetic encoder 105 and arithmetic decoder 106 described above to control the probability model of ^y.

[0170] FIG. 3A illustrates an example of a VAE (Variational Auto-Encoder), the details of which may vary in different implementations. For example, in one particular implementation, there may be additional components to more efficiently obtain statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler that targets extracting cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0171] Figure 3A depicts the encoder and decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and decoder may, and very often are, embedded in different devices.

[0172] FIG. 3B illustrates the encoder component of the VAE framework in isolation, and FIG. 3C illustrates the decoder component of the VAE framework in isolation. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of a first sub-network of the encoder, and Bitstream 2 is the output of a second sub-network of the encoder.

[0173] Similarly, in Fig. 3C, two bitstreams, Bitstream1 and Bitstream2, are received as input and a reconstructed (decoded) image, ̂z, is produced at the output. As mentioned above, the VAE can be divided into different logical units performing different actions. This is illustrated in Fig. 3B and Fig. 3C, where Fig. 3B shows the components participating in the encoding of a video-like signal and the encoded information provided. This encoded information is then received by the decoder component shown in Fig. 3C, for example for decoding. It is noted that the encoder and decoder components, indicated with references 12x and 14x, may correspond in their function to the components described above in Fig. 3A and indicated with reference 10x.

[0174] In particular, as can be seen in figure 3B, the encoder comprises an encoder 121 which converts an input x into a signal y, which is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and to a hyper-encoder 123. The hyper-encoder 123 provides the bitstream 2 already mentioned above to a hyper-decoder 147 which provides its information to the encoding module 105 (125).

[0175] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the output of the encoding of the signal, which are then provided (sent) to the decoding process. Unit 101 (121) is called "encoder", but it is also possible to call the complete sub-network described in FIG. 3B "encoder". An encoder generally means a unit (module) that converts an input into an encoded (e.g., compressed) output. From FIG. 3B it can be seen that unit 121 can in fact be considered as the core of the entire sub-network, since it converts an input x into y, which is a compressed version of x. The compression in encoder 121 may be achieved, for example, by applying a neural network, or in general any processing network having one or more layers. In such a network, the compression may be performed by cascaded processing, including downsampling, which reduces the size and / or number of channels of the input. Thus, the encoder may be called, for example, a neural network (NN)-based encoder, etc.

[0176] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE 125 in combination with the hyperencoder 123 and hyperdecoder 127 used to constitute the AE 125 may perform a binarization that may further compress the quantized signal by lossless compression. Therefore, the entire sub-network of FIG. 3B can also be called an "encoder".

[0177] Most deep learning (DL) based image / video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits). For example, in the VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height and thus a smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress signal y. Note that in general, an encoder does not necessarily need to reduce the size in both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally a subset of the dimensions).

[0178] In Non-Patent Document 1 (hereafter referred to as "Balle"), the authors proposed a framework for end-to-end optimization of image compression models based on non-linear transforms. They optimize for mean squared error (MSE) but use a more flexible transform built from a cascade of linear convolutions and non-linearities. Specifically, they use a generalized divisive normalization (GDN) congruence non-linearity inspired by a model of neurons in the biological visual system and proven effective in Gaussianizing image densities. This cascaded transform is followed by a uniform scalar quantization (i.e., each element is rounded to the nearest integer) that effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric non-linear inverse transform. [Non-Patent Document 1] J. Balle, L. Valero Laparra, and EP Simoncelli (2015). “Density Modeling of Images Using a Generalized Normalization Transformation”, In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016

[0179] Such an example of a VAE framework is shown in Figure 4, which utilizes six downsampling layers marked 401-406. The network architecture includes a hyperprior model. a ,g s ) shows the image autoencoder architecture, and the right side (h a ,h s ) corresponds to an autoencoder that implements a hyperprior. The factorized prior model is then transformed into an analytic transformation g a and the composite transformation g sWe use the same architecture for each image. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder converts the input image x into g a to generate a response y (latent representation) with a spatially varying standard deviation. a contains multiple convolutional layers with subsampling and generalized division normalization (GDN) as the activation function.

[0180] The response is h a to summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. The encoder then uses the quantized vector ^z to estimate ^σ (the spatial distribution of the standard deviations is used to obtain probability (or frequency) values ​​for arithmetic coding (AE)) and uses it to compress and transmit the quantized image representation ^y (or latent representation). The decoder first recovers ^z from the compressed signal. Then, h s We obtain ^y using g, which also provides a correct probability estimate for successfully recovering ^y. We then convert ^y to g s to obtain a reconstructed image.

[0181] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer with N channels and the convolution kernel is of size k1×k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned above, 2↓ means that a downsampling of a factor 2 is performed in this layer. As a result of downsampling by a factor 2, one of the dimensions of the input signal is reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and the height of the input image are reduced by a factor of 2. Since there are 6 downsampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, the output signal z^ 413 has a width and height equal to w / 64 and h / 64, respectively. The modules denoted by AE, AD are the arithmetic encoder, arithmetic decoder described with reference to FIG. 3A to FIG. 3C. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD may be replaced by other means of entropy coding. In information theory, entropy encoding is a lossless data compression method used to convert the values ​​of symbols into a binary representation, which is an invertible process. Also, the "Q" in the diagram corresponds to the quantization operation described above in connection with FIG. 4 and further described above in the "Quantization" section. Also, the quantization operation and the corresponding quantization unit as part of component 413 or 415 may not necessarily be present and / or may be replaced with another unit.

[0182] Also shown in Figure 4 is a decoder including upsampling layers 407-412. Between upsampling layers 411 and 410 in the processing order of the inputs, a further layer 420 is provided which is implemented as a convolutional layer but which does not provide upsampling to the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer may be provided in a NN to perform operations on the inputs which do not change the size of the inputs but which change certain characteristics. However, it is not necessary to provide such a layer.

[0183] Viewed in the processing order of the bitstream 2 through the decoder, the upsampling layers are performed in reverse order, i.e. from upsampling layer 412 to upsampling layer 407. Here, each upsampling layer is shown to provide upsampling with an upsampling ratio of 2, denoted by ↑. Of course, it is not necessarily the case that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8, etc. may also be used. Layers 407-412 are implemented as convolutional layers (conv). In particular, they may be intended to provide an operation on the input that is the inverse of that of the encoder, so that the upsampling layers may apply a deconvolution operation to the received input, such that its size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in any other manner, such as by bilinear interpolation between two neighboring samples, or by nearest neighbor sample copying, or the like.

[0184] In the first sub-network, several convolutional layers (401-403) are followed by a Generalized Divided Normalization (GDN) on the encoder side and an Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLU. It should be noted that the present disclosure is not limited to such implementation and in general other activation functions may be used instead of GDN or ReLU.

[0185] ■ Cloud solutions for machine tasks Video Coding for Machines (VCM) is another computer science direction that is popular today. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms like object segmentation, detection and recognition. In contrast to traditional image and video coding, which are intended for human perception, the quality characteristic is not the reconstructed quality but the performance of the computer vision task, for example object detection accuracy. This is illustrated in Figure 5.

[0186] Machine-oriented video coding, also called collaborative intelligence, is a relatively new paradigm for efficient deployment of deep neural networks across mobile-cloud infrastructures. By splitting the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload such that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computational nodes (e.g., devices, but generally, any functionally defined nodes). Here, the term "node" does not refer to the nodes of the neural network described above. Rather, a (computational) node here refers to a separate device / module (physically or at least logically) that implements parts of a neural network. Such devices may be different servers, different end-user devices, a mix of servers and / or user devices and / or clouds and / or processors, etc. In other words, the computational nodes may be considered as nodes that belong to the same neural network and communicate with each other to transfer data encoded within / for the neural network. For example, one or more layers may run on a first device (e.g., a device on the mobile side 510) and one or more layers may run in another device (e.g., a cloud server on the cloud side 590) to perform complex calculations. However, the distribution may be finer and a single layer may run on multiple devices. In this disclosure, the term "multiple" refers to more than one. In some existing solutions, part of the neural network function is run on a device (such as a user device or an edge device) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device running said part of the neural network. The concept of collaborative intelligence has also been extended to model training.In this case, data flows in both directions: from cloud to mobile during backpropagation in training, and from mobile to cloud during forward passes in training and inference.

[0187] Some works have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization was presented, followed by Context-Based Adaptive Arithmetic Coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590, rather than sending the compressed natural image data to the cloud and using the reconstructed image to perform object detection. Thus, it may be advantageous to compress the data (features) generated by the mobile side 510, which for this purpose may include a quantization layer 520. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps benefits image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, e.g. arithmetic coding, are popular approaches to compression of deep features (i.e., feature maps).

[0188] Today, video content contributes to over 80% of Internet traffic, and the proportion is expected to further increase. Therefore, it is important to build an efficient video compression system to generate higher quality frames for a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of the compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, techniques in video compression are also useful for action recognition and model compression. However, in the past decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. Although each module is well designed, the entire compression system has not been optimized end-to-end. It is desirable to further improve the video compression performance by jointly optimizing the entire compression system.

[0189] ■ End-to-end image or video compression DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations that are not used in traditional methods. However, it is not trivial to directly apply these techniques to build end-to-end learning systems for video compression. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0190] A straightforward solution is to represent the motion information using learning-based optical flow. However, current learning-based optical flow methods aim to generate a flow field that is as accurate as possible. A precise optical flow is often not optimal for a particular video task. Moreover, the amount of data for optical flow increases significantly compared to the motion information in traditional compression systems, and directly applying existing compression methods to compress the optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective function for both the residual and motion information. Rate-distortion optimization (RDO) aims to achieve a higher quality reconstructed frame (i.e., less distortion) given the number of bits (or bitrate) for compression. RDO is critical for video compression performance. To leverage the power of end-to-end training for learning-based compression systems, an RDO strategy that optimizes the entire system is needed.

[0191] In Non-Patent Document 3, the authors proposed an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding. [Non-Patent Document 3] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao; "DVC: An End-to-end Deep Video Compression Framework". Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015

[0192] Such an encoder is shown in Figure 6. In particular, Figure 6 shows the overall structure of an end-to-end trainable video compression framework. To compress the motion information, we use the optical flow v t The corresponding representation m that is suitable for better compression. t The CNN was specified to convert

[0193] ■ Transformers In recent years, Transformers have attracted increasing attention in the fields of both language processing (e.g., text translation) and image processing. Video coding can be facilitated by using Transformer-based neural networks. Transformers can be used for image enhancement and classification purposes. Transformers do not include recurrent or convolutional neural networks, but rely on self-attention. Transformers can also be implemented, for example, in the configuration shown in Figure 6. In particular, Transformers may be combined with recurrent or convolutional neural networks.

[0194] FIG. 7 shows an example of a Transformer 700. The Transformer 700 includes a neural layer 810 (Transformer layer). The Transformer 700 may include an encoder-decoder architecture including an encoder neural layer and a decoder neural layer. Alternatively, the Transformer 700 may include only an encoder stack of neural layers. Input data is input to the Transformer and output data is output. For example, the Transformer 700 may be configured for image processing and output an enhanced image. The input data may include, for example, patches of images or words of a sentence. For example, a tokenizer generates tokens in the form of patches from the image to be processed or words from the sentence to be processed. These tokens can be converted into embeddings (of continuous values) by some embedding algorithm. According to the example shown in FIG. 7, a linear projection layer 720 converts the input patches into tensor representations (embeddings) of the parts of the object to be processed (in the latent space). These tensor representations of the signal input are processed by the Transformer 700. Providing a position encoding layer 730 provides information about the position of parts of an object to be processed (e.g., an image or a sentence) relative to one another, for example the position of patches of an image or words of a sentence relative to one another. A sine function may be used for the position encoding.

[0195] The last one of the neural layers 710 outputs an output data tensor in latent space, which is transformed back to object space (e.g., image or sentence space) by a linear backprojection layer 740.

[0196] The processing by the neural layers (transformer layers) 710 is based on the concept of self-attention. Each of the neural layers 710 of the transformer 700 comprises a multi-head self-attention layer and a (fully connected) feed-forward neural network. The self-attention layer helps the encoder stack to look at other parts of the object (e.g., an image patch or a word) when encoding a particular part of the object (e.g., an image patch or a word of a sentence). The output of the self-attention layer is fed into the feed-forward neural network. The decoder stack also has both of those components and an additional "encoder-decoder" attention layer between them that helps the decoder stack to focus on the relevant parts of the input data. Each part of the object to be processed at each position (e.g., an image patch or a word of a sentence) flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. However, the feed-forward layer does not have those dependencies and therefore the various paths can be executed in parallel while flowing through the feed-forward layer.

[0197] In the multi-head self-attention layer of the encoder stack, we take the query Q, key K, and value V tensors, as well as the self-attention

number

[0198] Each sub-layer in each encoder and decoder layer (self-attention attention layer and feedforward neural network) has residual connections around it, followed by a normalization layer.

[0199] The output of the top encoder layer is converted into a set of attention vectors, K and V, which should be used by each decoder layer in its encoder-decoder attention layer. The encoder-decoder attention layer works similarly to the multi-head self-attention layer of the encoder stack, except that it creates a query matrix from each layer below and takes the key and value matrices from the output of the encoder stack.

[0200] The decoder stack outputs a vector of floating point numbers that are converted into parts of objects (e.g. patches of an image or words of a sentence) by a final linear layer (a fully connected neural network) that outputs logits, followed by a Softmax layer that produces the highest probability output.

[0201] As mentioned above, the use of neural networks (e.g., CNNs) is gaining importance in the field of image coding systems, and indeed in data compression systems in general, based on neural networks, e.g. in the context of image, video, audio, 3D-PCC, etc. compression.

[0202] In practice, neural networks in real applications are implemented on a variety of platforms / devices that differ from each other in terms of numerical architecture. For example, one device may be or include a CPU, while another device in data communication with the device that may be or include a CPU may be or include a GPU. Different platforms (e.g., with a CPU on one device and a GPU on another device) usually handle delicate integer and real situations, especially floating point arithmetic, but also fixed point arithmetic, in different ways. One class of delicate real situations is the overflow of the registers / memories used. There is no standardized way of handling overflow situations, and it is hardly possible to imagine a procedure that is commonly followed in this regard.

[0203] Integer overflow occurs when an arithmetic operation attempts to create a number that is outside the range that can be represented with a given number of digits (greater than the maximum or smaller than the minimum representable value). This situation may be handled differently for different compilers, devices (CPU, GPU), etc. In order to achieve bit-exact results of integer operations on different platforms, integer overflow should be avoided. In this context, it is noted that obtaining bit-exact results on different platforms is not important for all processing applications. However, for some applications it is. For example, in the context of autoencoding, e.g., the variational autoencoder as described above with reference to Figures 2 to 4, an entropy model for lossless data compression is used on the encoder side and on the decoder / receiver side to model the symbol probabilities for the arithmetic encoder and the decoder, respectively. The encoder side and the decoder / receiver side may use different kinds of platforms. For reliable reconstruction of the encoded image, it is crucial that the entropy model is applied in the same way on both sides. In fact, even a small deviation of the entropy model used at the decoder / receiver side from the entropy model used at the encoder side can lead to a complete breakdown of the image decoding / reconstruction process due to incorrect symbol interpretation. For example, in the configuration shown in Fig. 4, the entropy model ^σ must be provided by the hyperprior on the arithmetic encoder AE at the encoder side and on the arithmetic decoder AD at the decoder side. Bit-accurate application of the same entropy model is required for reliable image reconstruction. The problem of incorrect symbol interpretation due to deviation of the entropy model used at the decoder / receiver side from the entropy model used at the encoder side arises not only for image coding systems, but also for all compression systems that use arithmetic coding and neural networks for the entropy part.

[0204] With regard to real arithmetic, a distinction must be made between fixed-point and floating-point arithmetic. Fixed-point arithmetic is arithmetic that represents fractional (non-integers) numbers by storing a fixed number of digits in the fractional part. In general, it is integer arithmetic, and to convert such fixed-point values ​​to real values, division by a scaling factor corresponding to the number of bits used to store the fractional part is used. Floating-point arithmetic is the mantissa x base. 指数 where the mantissa is an integer, the base is an integer greater than or equal to 2, and the exponent is also an integer. The base is fixed and the mantissa-exponent pair represents the number. By using a non-fixed exponent compared to fixed-point arithmetic, a trade-off between dynamic range and precision can be achieved. Essentially, for both fixed-point and floating-point arithmetic, a real number is the mantissa × base. 指数The only difference is that in fixed-point arithmetic, both the base and the exponent are fixed, while in floating-point arithmetic, only the base is fixed and the exponent is part of the number representation. Thus, for fixed-point arithmetic, the precision of the result (the exponent) does not depend on the precision of the arguments, which means, for example, that the result of a sum does not depend on the order of the sum (if no overflow occurs, the arithmetic is associative, like normal integer addition). On the other hand, for floating-point arithmetic, the exponent is part of the number representation and is calculated during the arithmetic operation based on the exponents of the arguments. This causes additions that are not associative. For example, if many very small numbers are added one by one to some larger number, the result is equal to the larger number, since the effect of the relatively small numbers is lost after rounding with the exponent of the larger number. Conversely, if the small numbers are added together first, the effect of adding each number is not lost by rounding, and the resulting sum can be relatively large so that it is not completely lost after rounding with the exponent of the larger number. Thus, sums in fixed-point arithmetic are associative in the absence of overflow, while in principle floating-point arithmetic is not associative. That is, in general, (A+B)+C≠A+(B+C). Since the order of additions may be different or even not completely pre-determined on the encoder and decoder sides, this presents a problem with regard to identical results being achieved on both sides. Therefore, fixed-point arithmetic may be preferred over floating-point arithmetic. In the context of entropy coding based data compression, it may be preferable to avoid the use of floating-point arithmetic at least for the entropy part of the net image coding and use fixed-point arithmetic instead to ensure proper decoding within different platforms, especially in systems with massive parallelism. In this case, the potential problem of implementing different floating-point arithmetic on different devices is solved, since fixed-point (integer) arithmetic is much more portable. However, overflow cannot be avoided by simply restricting to fixed-point arithmetic.To guarantee bit-accurate behavior on different platforms, it is not enough to simply use fixed-point arithmetic; it is also important to guarantee that there is no integer overflow.

[0205] More specifically, consider a neural network layer as shown in FIG. 8. The neural network layer may be included in, for example, a convolutional neural network or a fully connected neural network. i are input to different input channels, and weights w ij where index i is the in , and the index j indicates an input channel among C out jth channel output is

number

[0206] An accumulator register may be used for buffering the sum.

[0207] The (usually trainable) bias D is ignored in the following for simplicity. For example, for a typical convolutional neural network layer, the output of the jth channel is decomposed by the convolution operator * as

number

[0208] The accumulator registers of a neural network used for buffering sums have a predefined size, i.e., some accumulator register bit depth (size) n (for example, n = 16 or 32 bits). To avoid integer overflow, for example, the following conditions must be met:

number

[0209] A) Clipping and scaling According to one embodiment, since fixed-point arithmetic is used, the input data x processed by a neural network layer of a neural network (e.g., a convolutional neural network) i and the real-valued weights w of the neural network layers ij but

number

number

[0210] To avoid the performance loss caused by such conversion to fixed-point arithmetic,

number

number

number

[0211] Input value x i The range of is not generally known. To satisfy the condition according to Equation 5, the following constraint is applied using a lower integer threshold A and an upper integer threshold B:

number

[0212] Thus, according to this embodiment, the following units are added to the neural network (inference pipeline): units configured for scaling, rounding and clipping the input data values ​​before input to the neural network layers, and a unit for dividing the output of the neural network layers by a scaling factor. The scaling and / or rounding units may be omitted (or may not be operated) if it is guaranteed that only integer-valued data (data containing only integers) are processed. These units may be added to neural network networks used for data compression / coding, in particular for (variational) autoencoder or transformer architectures as described above.

[0213] An exemplary embodiment is shown in FIG. 9. FIG. 9 shows, for exemplary purposes, a neural network layer of a convolutional neural network. The neural network comprises a stack of convolutional layers followed by an activation function, in this case the LeackyReLU function. The black arrows in FIG. 9 indicate the input of a new unit added to a conventional neural network. Before the convolutional layers, the input values ​​of the real-valued input data are multiplied by a scaling factor and the result of the multiplication process is rounded to the nearest integer. If the input data is already integer-valued, the scaling and / or rounding process can be omitted. If the resulting integer value is smaller than a predefined lower integer threshold -A, the integer value is clipped to a predefined lower integer threshold -A, and if the integer value is larger than a predefined upper integer threshold B, the integer value is clipped to an upper integer threshold B. The input values ​​thus obtained are processed by convolutional neural network layers and the output of these layers is input. According to one embodiment, the output of the activation function is divided by the scaling factor. According to another embodiment, the output of the convolutional neural network layer is the divided scaling factor and then fed to the activation function. According to further embodiments, a factorization of each of the scaling factors into two parts is performed, where the division by the first part can be performed before the activation function and the division by the second part can be performed after the activation function.

[0214] A neural network as shown in Fig. 9, including units for scaling, rounding, and clipping and units for dividing the output of the neural network layer by a scaling factor (before and / or after the respective activation function), may be provided at both the encoder and decoder sides of a coding system, for example in an encoder for encoding an image and a decoder for decoding the encoded image, for example, in the encoder / decoder included in the configurations shown in Figs.

[0215] B) Weight The condition in Equation 5 can be transformed into another condition for the weights of the neural network layers. i and weight w ij Assume that A, B, C, and D have already been converted to integers by scaling and rounding (if necessary) as described above, and further clipped, if necessary, to -A and B, respectively, according to Equation 6. The terms of Equation 5 (for each output channel, if different output channels are provided, the index j indicating the output channel is omitted below) are transformed to:

number

[0216] If the condition in Equation 7 is satisfied, then the inference involving clipping the input data to the lower and upper thresholds, respectively, will not cause any overflow of the accumulator registers of the neural network. If the model weights in the data compression system are allowed to be user-defined, then the condition in Equation 7 must be satisfied if bit-accurate behavior of the encoder-side and decoder-side neural networks needs to be guaranteed.

[0217] According to one embodiment, the integer lower threshold -A is -2 k-1 and the upper integer threshold B is 2 k-1 −1, where k denotes the predefined bit depth of the layer input data.

[0218] In this case, an easy-to-check condition on the weights is

number

[0219] This form of the condition on the weights is particularly easy to compute for user-defined weights, to check whether overflow is guaranteed to never occur during inference. The conditions according to Equation 8 and Equation 9 must be satisfied for each output j of a neural network layer.

[0220] The bias D may in principle be zero, in which case equations 8 and 9 are easier to check.

[0221] In particular, for one-dimensional convolutional neural network layers,

number

[0222] For a 2D convolutional neural network layer,

number

number

[0223] C) Scaling factor for weights A larger scaling factor for the weights, s j (To be more precise, 2 sj ; see above) incurs a lower performance loss due to conversion to fixed-point arithmetic than smaller scaling factors (see above). j takes the maximum possible value that still ensures that the condition in Equation 8 or 9 is satisfied.

number

number

[0224] According to another implementation, the second scaling factor s for the jth output channel of the at least one neural network layer is j teeth,

number

[0225] Both conditions can advantageously ensure that integer overflow does not occur for the accumulator register.

[0226] According to another implementation, the condition is

number

number

number

[0227] To convert the entire neural network pipeline to fixed-point arithmetic, the parameters p and k should be known. These parameters can be chosen, for example, by checking all possible combinations of (p,k), 0≦p≦k≦n, and the corresponding s jis obtained using equation (10) for a selected k and a predefined n on some calibration data set. The minimum of the predefined loss function corresponds to the best pair (p,k). The loss function can represent, for example, the number of bits required to encode some image, or its likelihood estimate. If not only the entropy part but also the decomposition and synthesis parts (see, for example, the left side of the configuration shown in FIG. 4) are converted to fixed-point arithmetic, the distortion can also be used as part of the loss function used.

[0228] D) Activation function To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear and relatively simple nonlinear activation functions, such as the ReLU function that essentially defines a clipping procedure, this requirement is relatively easy to meet. For more complex nonlinearities, in particular

number

[0229] According to an embodiment, the mathematically defined nonlinear activation function is replaced by an approximation function selected from the group consisting of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The replaced mathematically defined nonlinear activation function may be a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, or a scaled exponential linear unit function.

[0230] The approximation functions may be used, for example, by encoders / decoders included in the configurations shown in Figures 2-4, 6 and 7, and generally in the context of data compression / encoding.

[0231] In general, for any nonlinear activation function, we can use the sum of the first few elements of its Taylor series. For a mathematically defined function f(x), the Taylor series around a predefined value is

number

number

number

[0232] It should be noted that, depending on the actual application, approximations of mathematically defined non-linear activation functions by rational or polynomial functions may be considered particularly appropriate.

[0233] In particular, according to one embodiment,

number

number

number

[0234] A Softmax function as defined by Equation 15 and Equation 16 may be advantageously used as an activation function for the last layer of a stack of (e.g., convolutional) neural network layers, e.g., to provide a classification result to be obtained by the neural network.

[0235] In particular, an embodiment is provided relating to a method 1000 for operating a neural network in which integer overflow may be avoided. The neural network comprises a neural network layer comprising or connected to an accumulator register for buffering summation results, the accumulator register having a predefined accumulator register size. The method 1000 includes defining S1010 a lower integer threshold and an upper integer threshold for values ​​of integers contained in a data entity (e.g., a number, a vector, or a tensor) of input data for the neural network layer. The method further includes a step S1020 of clipping the values ​​of integers contained in the data entity of the input data to the defined lower integer threshold if the values ​​of integers contained in the data entity of the input data are smaller than the defined lower integer threshold, and a step S1020 of clipping the values ​​of integers contained in the data entity of the input data to the defined upper integer threshold if the values ​​of integers contained in the data entity of the input data are larger than the defined upper integer threshold, in order to avoid integer overflow of the accumulator register. The method 1000 shown in FIG. 10 allows for the use of a clipping procedure to avoid integer overflow when performing the same procedure with the same input data on two devices / platforms, resulting in the same results for both devices / platforms.

[0236] The method shown in FIG. 10 may be implemented in any kind of neural network, for example, a neural network including one or more fully connected neural network layers or one or more convolutional neural network layers. In particular, a neural network 1100 shown in FIG. 11 is provided. The neural network 1100 may utilize the method 1000. The neural network 1100 comprises a unit 1110 for scaling, rounding, and clipping. The unit 1110 may be divided into subunits configured for different operations, such as scaling, rounding, and clipping of input data, respectively, and the on and off operation of the first unit and each subunit may be switchable, respectively. The unit 1110 may be configured to receive real-valued input data (data including real numbers) and convert the real-valued input data to integer-valued input data by multiplying the input data value by a scaling factor and rounding the multiplication result to the next integer, as described above. The resulting integer value is clipped to integer lower and upper thresholds, if necessary.

[0237] The integerized and clipped data is input to the neural network layer 1120. Downstream of the neural network layer 1120, a unit for unscaling 1130 is provided. The unit for unscaling 1130 may divide the output of the neural network layer 1120 by a scaling factor. The output of the unit for unscaling 1130 may be input to the activation function 1140. Alternatively, the output of the neural network layer 1120 may be directly input to the activation function 1140, and the unit for unscaling 1130 is provided downstream of the activation function 1140 and divides its output by the scaling factor. Alternatively, the unscaling by the unit for unscaling 1130 is performed partially on the output of the neural network layer 1120 (by a first factorized part of the scaling factor) and then partially on the output of the activation function 1140 (by a second factorized part of the scaling factor).

[0238] In particular, as shown in FIG. 12, an embodiment is provided for a method 1200 of operating a neural network based on adjusted weights. The neural network comprises or is connected to an accumulator register for buffering summation results, with a predefined accumulator register size, and comprises a neural network layer. The method 1200 includes defining S1210 a lower integer threshold A and an upper integer threshold B for integer values ​​contained in data entities (e.g., numbers, vectors, or tensors) of input data of the neural network layer. Furthermore, the method includes a step S1220 of clipping the integer values ​​contained in the data entities of the input data to the defined lower integer threshold if the integer values ​​contained in the data entities of the input data are smaller than the defined lower integer threshold, and a step S1220 of clipping the integer values ​​contained in the data entities of the input data to the defined upper integer threshold if the integer values ​​contained in the data entities of the input data are larger than the defined upper integer threshold. Further, the method includes a step S1230 of determining integer-valued weights (i.e., weights that include integers, e.g., only integers) of the neural network layers based on a defined integer lower threshold, a defined integer upper threshold, and a predefined accumulator register size, such that integer overflow of the accumulator registers may be avoided.

[0239] The method 1200 shown in Fig. 12 provides the same results for both devices / platforms when performing the same procedure with the same input data on the two devices / platforms. In particular, in the context of entropy model based coding and / or compression and decompression of data, e.g., image data, it is a key issue to provide substantially bit-accurate processing results at the encoder side and the decoder side, respectively, to provide the same or complementary technical effects obtainable by the method 1200 shown in Fig. 12.

[0240] In particular, according to an embodiment, a method 1300 of operating a neural network is provided (see FIG. 13). The method 1300 comprises implementing 1310 an approximation function of a mathematically defined real-valued nonlinear activation function as an activation function of said at least one neural network layer, the approximation function allowing for operation only with integers of a fixed-point representation of the input values ​​of the approximation function. The approximation function may comprise at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined nonlinear activation function may be selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaled exponential linear unit function. In particular, the approximated activation function is

number

[0241] Providing the above-mentioned approximated (approximate) activation functions may allow bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, in particular since the calculation of exponential functions may be avoided, such that the technical effects obtained by these numerical operations are the same or complementary to each other.

[0242] According to an embodiment, there is provided a neural network 1400 as shown in Fig. 14. The neural network 1400 comprises at least one neural network layer 1410 and an activation function 1420 connected to the output of said at least one neural network layer 1410, the activation function 1420 being implemented as an approximation function of a mathematically defined real-valued non-linear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing working only with integers in a fixed-point representation of the input values ​​of the approximation function.

[0243] At least some of the steps of the methods 1000, 1200 and 1300 described with reference to Fig. 10, Fig. 12 and Fig. 13 may be included in a method for encoding or decoding at least a part of data, e.g. an image. According to an embodiment, the encoding or decoding is based on an entropy model providing statistical (probability) properties of the symbols to be encoded or decoded, e.g. mean, variance, (cross)correlation, etc. The entropy model may be provided by a hyper-prior of a variational autoencoder, an auto-regressive prior of a variational autoencoder or a combination of a hyper-prior and an auto-regressive prior of a variational autoencoder. The use of the methods 1000, 1200 and 1300 described with reference to Fig. 10, Fig. 12 and Fig. 13 for data encoding, e.g. image encoding, may prove to be advantageous with regard to a high-quality reconstruction of the encoded / compressed data without suffering serious corruption.

[0244] The methods 1000, 1200, and 1300 described with reference to Figures 10, 12, and 13 may be implemented in an apparatus 1500 as shown in Figure 15, which may be configured to perform steps of these methods. According to an embodiment, the apparatus 1500 comprises a processing circuit configured to perform steps of the methods 1000, 1200, and 1300 described with reference to Figures 10, 12, and 13. Furthermore, the apparatus 1500 may comprise at least one of the neural networks shown in Figures 9, 11, and 14. The apparatus 1500 may be constituted by an encoder (e.g., the encoder 20 shown in Figures 16 and 17) or a decoder (e.g., the decoder 20 shown in Figures 16 and 17), or may be constituted by the video coding device 8000 shown in Figure 18 or the apparatus 9000 shown in Figure 19.

[0245] The apparatus 1500 may be an apparatus for encoding or decoding data, for example at least a portion of an image. Furthermore, the apparatus 1500 may comprise or be included in a (variational) autoencoder or a transformer as described above.

[0246] Although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the shown operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0247] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In certain embodiments, multitasking and parallel processing are advantageous.

[0248] Some Example Implementations in Hardware and Software A corresponding system in which the above-mentioned encoder-decoder processing chain may be deployed is shown in FIG. 16. FIG. 16 is a schematic block diagram illustrating an example coding system, e.g., a video, image, audio, and / or other coding system (or coding system for short), that may utilize the techniques of the present application. The video encoder 20 (or encoder for short) and the video decoder 30 (or decoder for short) of the video coding system 10 represent examples of devices that may be configured to perform the techniques according to various examples described herein. For example, the video coding and decoding may employ neural networks, which may be distributed, and may apply the above-mentioned bitstream parsing and / or bitstream generation to communicate feature maps between the distributed computational nodes (two or more).

[0249] As shown in FIG. 16, the coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14, for example for decoding the encoded picture data 13.

[0250] The source device 12 comprises an encoder 20 and may further, i.e. optionally, comprise a picture source 16 , a pre-processor (or pre-processing unit) 18 , for example a picture pre-processor 18 , and a communication interface or unit 22 .

[0251] The picture source 16 may comprise or be any kind of picture capture device, such as, for example, a camera capturing real-world pictures, and / or any kind of picture generation device, such as, for example, a computer graphics processor for generating computer animation pictures, or any kind of other device that obtains and / or provides real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures) and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also be any kind of memory or storage device that stores any of the aforementioned pictures.

[0252] To distinguish it from the preprocessor 18 and the processing performed by the preprocessing unit 18 , the picture or picture data 17 may also be referred to as raw picture or raw picture data 17 .

[0253] The pre-processor 18 is configured to receive the (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component. It is noted that the pre-processing may also employ a neural network (as in any of Figures 1-7) using presence indicator signaling.

[0254] A video encoder 20 is configured to receive the pre-processed picture data 19 and to provide encoded picture data 21 .

[0255] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0256] The destination device 14 comprises a decoder 30 (e.g., a video decoder 30), and may additionally, i.e. optionally, comprise a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0257] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. a storage device for the encoded picture data, and to provide the encoded picture data 21 to a decoder 30.

[0258] The communications interface 22 and the communications interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communications link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0259] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, e.g., packets, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0260] The communications interface 28, which forms a counterpart to the communications interface 22, may for example be configured to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging to obtain the encoded picture data 21.

[0261] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in Fig. 16, or as bidirectional communication interfaces, e.g., configured to send and receive messages, e.g., set up connections, receive, acknowledge and exchange any other information related to the communication link and / or data transmission, e.g., transmission of encoded picture data. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31.

[0262] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by, for example, a display device 34.

[0263] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, for example to display the picture to a user or viewer. Display device 34 may be or have any type of display for displaying the reconstructed picture, for example, an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0264] 16 illustrates source device 12 and destination device 14 as separate devices, an embodiment of the devices may include both or both functionality, i.e., source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0265] As will be apparent to those skilled in the art based on this document, the presence and (exact) division of functions or features of various units within the source device 12 and / or destination device 14 as shown in FIG. 16 may vary depending on the actual device and application.

[0266] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both the encoder 20 and the decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including neural networks or portions thereof. The decoder 30 may be implemented via processing circuitry 46 to embody any coding system or subsystem described herein. The processing circuitry may be configured to perform various operations, such as those described below. If the techniques are implemented in part in software, the device may store instructions for the software on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG.

[0267] Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0268] In some cases, the video coding system 10 shown in FIG. 16 is only an example, and the techniques herein may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between an encoding device and a decoding device. In other examples, data may be retrieved from a local memory, streamed over a network, or the like. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that simply encode data to memory and / or retrieve data from memory and decode it, without communicating with each other.

[0269] 18 is a schematic diagram of a video coding device 8000 according to an embodiment of the present invention. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 16, or an encoder, such as the video encoder 20 of FIG. 16.

[0270] The video coding device 8000 comprises an ingress port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data; a processor, logic unit, or central processing unit (CPU) 8030 for processing the data; a transmitter unit (Tx) 8040 and an egress port 8050 (or output port 8050) for transmitting the data; and a memory 8060 for storing the data. The video coding device 8000 may also comprise optical-electrical (OE) and electrical-optical (EO) components coupled to the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, and the egress port 8050 for the egress or ingress of optical or electrical signals.

[0271] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, the egress port 8050, and the memory 8060. The processor 8030 comprises a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 performs, processes, prepares, or provides various coding operations. Thus, the inclusion of the neural network-based codec 8070 provides a substantial improvement to the functionality of the video coding device 8000, resulting in the transformation of the video coding device 8000 into different states. Alternatively, the neural network based codec 8070 may be implemented as instructions stored in memory 8060 and executed by the processor 8030.

[0272] The memory 8060 may comprise one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device for storing programs when such programs are selected for execution, as well as for storing instructions and data read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0273] FIG. 19 is a simplified block diagram of an apparatus that may be used as either or both of the source device 12 and destination device 14 from FIG. 16, according to an exemplary embodiment.

[0274] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or devices, now existing or later developed, that can manipulate or process information. Although the disclosed embodiments may be implemented using a single processor as shown, such as processor 9002, advantages in speed and efficiency may be achieved using more than one processor.

[0275] The memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in some implementations. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 that is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010, which include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1-N, which further include a video coding application that performs the methods described herein.

[0276] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0277] Although shown here as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, secondary storage may be directly coupled to other components of the device 9000 or accessed over a network, and may comprise a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, the device 9000 may be implemented in a wide variety of configurations.

Claims

1. A method (1000) of operating a neural network comprising at least one neural network layer comprising or connected to an accumulator register for buffering a summation result, the method comprising: defining a lower integer threshold and an upper integer threshold for integer values ​​included in data entities of input data for the at least one neural network layer (S1010); and if the value of an integer contained in the data entity of the input data is smaller than a defined integer lower threshold, clipping the value of the integer contained in the data entity of the input data to the defined integer lower threshold (S1020), and if the value of an integer contained in the data entity of the input data is greater than a defined integer upper threshold, clipping the value of the integer contained in the data entity of the input data to the defined integer upper threshold (S1020), so as to avoid integer overflow of the accumulator register. Method (1000).

2. scaling data entities of the input data by a first scaling factor to obtain scaled values ​​of the data entities of the input data.

2. The method of claim 1 (1000).

3. 3. The method (1000) of claim 2, further comprising the step of rounding the scaled values ​​of the data entities of the input data to their nearest respective integer values ​​to obtain the integer values ​​contained in the data entities of the input data.

4. 4. The method (1000) of claim 2 or 3, further comprising: processing the input data by the at least one neural network layer to obtain output data including an output data entity; and dividing the output data entity by a third scaling factor.

5. 4. The method (1000) of claim 2 or 3, further comprising: processing the input data by the at least one neural network layer to obtain output data including output data entities; processing the output data entities through an activation function to obtain an output of the activation function; and dividing the output of the activation function by a third scaling factor.

6. 4. The method (1000) of claim 2 or 3, further comprising: processing the input data by the at least one neural network layer to obtain output data including an output data entity; factoring a third scaling factor into a first part and a second part; dividing the output data entity by the first part of the factored third scaling factor to obtain a partially de-scaled output data entity; processing the partially de-scaled output data entity through an activation function to obtain an output of the activation function; and dividing the output of the activation function by the second part of the factored third scaling factor.

7. The method (1000) of any one of claims 1 to 6, wherein the lower integer threshold is less than or equal to 0 and the upper integer threshold is greater than or equal to 0.

8. The lower threshold is −2 k-1 and for the upper threshold, 2 k-1 The method (1000) of any one of claims 1 to 7, wherein k is given by -1, where k denotes a predefined bit depth of the input data.

9. 9. The method (1000) of claim 1, wherein the at least one neural network layer is or includes one of a fully connected neural network layer and a convolutional neural network layer.

10. The method (1000) of any one of claims 1 to 9, wherein the at least one neural network layer includes an attention mechanism.

11. The method (1000) of any one of claims 1 to 10, wherein the at least one neural network layer includes integer-valued weights, including integers.

12. 12. The method (1000) of claim 11, further comprising providing weights, scaling the weights by a second scaling factor to obtain scaled weights, and rounding the scaled weights to the respective nearest integer values ​​to obtain the integer-valued weights of the at least one neural network.

13. The second scaling factor is [0010] where s j The method (1000) of claim 12 , wherein: represents a fractional portion of a real number contained in the real-valued weight.

14. s for the at least one output channel of the at least one neural network layer j is the condition [0025] Fulfilling Here, W j represents a subset of trainable weights of the at least one neural network layer, and |W j | is the subset W j n represents the bit size of the accumulator register; k represents the predefined bit depth of the input data; and b j The method (1000) of claim 13, wherein: represents the bias value.

15. s for the at least one output channel of the at least one neural network layer j teeth, [0030] The method (1000) of claim 14, wherein the method is given by:

16. The second scaling factor s for the j-th output channel of the at least one neural network layer j is the condition [0045] Fulfilling Here, C in is the number of input channels of the at least one neural network layer, n represents the bit size of the accumulator register, k represents a predefined bit depth of the input data, and b j The method (1000) of claim 13, wherein j denotes the bias value for the jth channel.

17. The second scaling factor s for the j-th output channel of the at least one neural network layer j teeth, [0050] 20. The method (1000) of claim 16, wherein the method is given by:

18. 18. A method of encoding data comprising steps of a method of operating a neural network according to any one of claims 1 to 17.

19. 20. A method according to claim 18, wherein encoding the data comprises providing an entropy model by a neural network and entropy encoding the data based on the provided entropy model, and wherein providing the entropy model comprises steps of a method of operating a neural network according to any one of claims 1 to 15.

20. 20. The method of claim 18, wherein the entropy model is provided by one of: a) a hyperprior of a variational autoencoder; b) an autoregressive prior of a variational autoencoder; and c) a combination of the hyperprior and the autoregressive prior of a variational autoencoder.

21. 21. The method of claim 19 or 20, wherein the entropy encoding comprises entropy encoding by an arithmetic encoder.

22. 22. The method according to any one of claims 19 to 21, further comprising signalling the defined lower threshold and the defined upper threshold to a decoder side.

23. 22. The method of claim 19, further comprising signaling at least one of the first scaling factor, the second scaling factor and the third scaling factor in a bitstream to a decoder side.

24. 22. The method according to any one of claims 19 to 21, further comprising signalling the difference from a predefined lower threshold and the difference from a predefined upper threshold to the decoder side.

25. 25. The method of claim 19, further comprising signaling at least one of the difference from the first predefined scaling factor and the difference from the second predefined scaling factor to a decoder side in the bitstream.

26. 26. The method of any one of claims 22 to 25, wherein Exponential-Golomb coding is used for the signaling.

27. 18. A method of decoding encoded data comprising the steps of a method of operating a neural network according to any one of claims 1 to 17.

28. 28. A method according to claim 27, wherein the decoding of the data comprises providing an entropy model by a neural network and entropy decoding the data based on the provided entropy model, and wherein entropy decoding the data comprises steps of a method of operating a neural network according to any one of claims 1 to 17.

29. 29. The method of claim 28, wherein the entropy model is provided by one of: a) a hyperprior of a variational autoencoder; b) an autoregressive prior of a variational autoencoder; and c) a combination of the hyperprior and autoregressive prior of a variational autoencoder.

30. 30. The method of claim 28 or 29, wherein the entropy decoding comprises entropy decoding by an arithmetic decoder.

31. The method according to any one of claims 27 to 30, further comprising receiving information about the defined lower threshold and the defined upper threshold in a bitstream from an encoder side.

32. 32. The method of claim 27, further comprising receiving information about at least one of the first and second scaling factors and a difference to a predefined value from an encoder side in a bitstream.

33. 31. The method of any one of claims 27 to 30, further comprising receiving information about the difference from a predefined lower threshold and the difference from a predefined upper threshold in the bitstream from an encoder side.

34. 34. The method of claim 27, further comprising receiving, at a decoder side, information about at least one of the difference from a first predefined scaling factor and the difference from a second predefined scaling factor in a bitstream.

35. 35. A method according to any one of claims 18 to 34, wherein the data is image data.

36. 1. A method for encoding at least a portion of an image, comprising the steps of: converting tensors representing components of the image into latent tensors; providing an entropy model; and processing the latent tensors through a neural network based on a provided entropy model to generate a bitstream; Providing the entropy model comprises performing the steps of a method according to any one of claims 1 to 17. method.

37. 37. The method of claim 36, wherein the entropy model is provided by one of: a) a hyperprior of a variational autoencoder; b) an autoregressive prior of a variational autoencoder; and c) a combination of the hyperprior and autoregressive prior of a variational autoencoder.

38. 38. The method of claim 36 or 37, wherein the processing of the latent tensors is performed by an arithmetic encoder.

39. 39. The method of any one of claims 36 to 38, further comprising signaling the defined lower threshold and the defined upper threshold in the bitstream to a decoder side.

40. 40. The method of any one of claims 36 to 39, further comprising signaling at least one of the first and second scaling factors to a decoder side in the bitstream.

41. 1. A method for reconstructing at least a portion of an image, comprising the steps of: providing an entropy model; processing the bitstream with a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing the components of the image; At least one of providing the entropy model and processing the latent tensors comprises performing steps of a method according to any one of claims 1 to 17. method.

42. 42. The method of claim 41, wherein processing the latent tensors to obtain tensors representing the components of the image comprises performing steps of the method of any one of claims 1 to 17.

43. 43. The method of claim 42, wherein the entropy model is provided by one of: a) a hyperprior of a variational autoencoder; b) an autoregressive prior of a variational autoencoder; and c) a combination of the hyperprior and autoregressive prior of a variational autoencoder.

44. 44. A method according to claim 42 or 43, wherein the processing of the bitstream is performed by an arithmetic decoder.

45. 45. The method of any one of claims 41 to 44, further comprising reading information regarding the defined lower threshold and the defined upper threshold from the bitstream.

46. 46. ​​The method of claim 41, further comprising reading at least one of the first scaling factor and the second scaling factor from the bitstream.

47. 47. A method according to any one of claims 36 to 46, wherein the components are Y, U or V components, or R, G, B components.

48. 48. A computer program product comprising program code stored on a non-transitory medium, said program code being adapted to perform a method according to any one of claims 1 to 47 when executed by one or more processors.

49. An apparatus (1500) for encoding data, comprising a processing circuit (1510) configured to perform steps of the method according to any one of claims 18 to 26 and 36 to 40.

50. 20. An apparatus (1500) for encoding at least a portion of an image, the apparatus having a processing circuit (1510) configured to perform the steps of converting tensors representing components of the image into latent tensors, providing an entropy model, the step comprising performing the steps of the method of any one of claims 1 to 17, and processing the latent tensors by a neural network based on the provided entropy model to generate a bitstream.

51. An apparatus (1500) for decoding data, comprising a processing circuit (1510) configured to perform steps of the method according to any one of claims 27 to 35 and 41 to 47.

52. 20. An apparatus (1500) for decoding at least a portion of an encoded image, the apparatus (1500) having a processing circuit (1510) configured to perform the steps of providing an entropy model, the step including performing the steps of the method of any one of claims 1 to 17, processing a bitstream by a neural network based on the provided entropy model to obtain latent tensors representing components of the image, and processing the latent tensors to obtain tensors representing the components of the image.

53. A neural network comprising: a neural network layer configured to process input data to obtain output data; and an activation function configured to process the output data to obtain activation function output data; a first unit for scaling, rounding and clipping the input data input to the neural network layer; a second unit for unscaling at least one of the output data and the activation function output data; A neural network comprising:

54. 54. The neural network of claim 53, wherein the first unit is configured to perform the steps of the method of claim 1 or 2.

55. A neural network according to claim 53 or 54, configured to carry out the steps according to any one of claims 1 to 17.

56. An apparatus (1500) for encoding at least a part of an image, comprising a neural network according to any one of claims 53 to 55.

57. 56. The apparatus (1500) of claim 52, comprising one of: a) a hyperprior of a variational autoencoder, the hyperprior comprising a neural network according to any one of claims 53 to 55; b) a hyperprior of an autoregressive prior of a variational autoencoder, the hyperprior comprising a neural network according to any one of claims 53 to 55; and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder, at least one of the hyperprior and the autoregressive prior comprising a neural network according to any one of claims 53 to 55.

58. An apparatus (1500) for decoding at least a part of an encoded image, comprising a neural network according to any one of claims 53 to 55.

59. 59. The apparatus (1500) of claim 58, comprising one of: a) a hyperprior of a variational autoencoder, the hyperprior comprising the neural network of any one of claims 53 to 55; b) a hyperprior of an autoregressive prior of a variational autoencoder, the hyperprior comprising the neural network of any one of claims 53 to 55; and c) a combination of a hyperprior and an autoregressive prior of a variational autoencoder, at least one of the hyperprior and the autoregressive prior comprising the neural network of any one of claims 53 to 55.

Images