Improved concept for representation of neural network parameter

Abstract

To support efficient encoding and / or decoding of neural network parameters.SOLUTION: An apparatus 100 generates an NN representation, configured to quantize an NN parameter onto a quantized value by determining a quantization parameter and a quantization value 152 for the NN parameter so that, from the quantization parameter 142, a multiplier 144 and a bit shift number 146 can be derived. The determining of the quantization parameter and the quantization value for the NN parameter is performed so that the quantized value of the NN parameter corresponds to a product between the quantization value and a factor 148, which depends on the multiplier, bit-shifted by the number of bits which depends on the bit shift number.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Embodiments according to the invention relate to an apparatus and method for encoding or decoding neural network parameters, using an improved concept of representation of the neural network parameters, whereby improvements in terms of inference and / or storage bitrate optimization can be achieved. [Background technology]

[0002] Neural networks, in their most basic form, consist of a series of affine transformations followed by element-wise nonlinear functions. They can be represented as directed acyclic graphs, as shown in Figure 1. Each node is accompanied by a specific value that is propagated forward to the next node by multiplication with the edge's respective weight value. Then, all input values ​​are simply aggregated.

[0003] Figure 1 shows an example of a graph representation of a feedforward neural network. Specifically, this two-layer neural network is a nonlinear function that maps a four-dimensional input vector to a solid line.

[0004] Mathematically, the neural network in Figure 1 calculates its output in the following way: output=L 2 (L 1 (input) Where: L i (X)=N i (B i (X) Wherein B i is the affine transformation of layer i, and N i is some nonlinear function of layer i.

[0005] Bias layer In the case of the so-called "bias layer", B i is the weight parameter (edge ​​weight) W associated with layer i i and the input X of layer i i Matrix multiplication with, followed by a bias bi It is the sum of. B i (X)=W i *X i +b i W i is the dimension n i ×k i is the weight matrix with i is the dimension k i ×m i The input matrix has the bias b i is a length n i The transpose vector of . The operator * denotes matrix multiplication. The bias b i The sum of W is an element-wise operation on the columns of a matrix. More precisely, i *X i +b i b i W i *X i This means that the value is added to each column of

[0006] So-called convolutional layers can also be used by casting them as a matrix-matrix product, as described in [1].

[0007] Hereafter, the procedure of computing outputs from given inputs is referred to as inference, and the intermediate results are referred to as hidden layers or hidden activations, which constitute linear transformations + element-wise nonlinearities, such as the first dot product + nonlinearity computation above.

[0008] Typically, neural networks contain millions of parameters and may require hundreds of megabytes for their representation. Their inference procedure therefore requires high computational resources to execute, as it involves the calculation of many dot product operations between large matrices. It is therefore very important to reduce the complexity of performing these dot products.

[0009] Batch Norm Layer A more sophisticated variant of the affine transformation of neural network layers is the so-called bias and batch norm operation: Formula 1:

number

[0010] If all vector elements of b are equal to 0, then Equation 1 refers to a batch norm layer. In contrast, ε and all vector elements of μ and β are set to 0, and γ and σ 2 If all elements of are set to 1, then the layer without batch norm (only bias) is processed.

[0011] Efficient representation of parameters Parameters W, b, μ, and σ 2 Let , γ, and β collectively denote the parameters of the layer. They typically need to be signaled in the bitstream. For example, they can be represented as 32-bit floating-point numbers or quantized to an integer representation. Note that ε is typically not signaled in the bitstream.

[0012] A particularly efficient approach for encoding such parameters uses a uniform reconstruction quantizer, where each value is represented as an integer multiple of a so-called quantization step size value. The corresponding floating-point number can be reconstructed by multiplying an integer by the quantization step size, which is usually a single floating-point number. However, efficient implementations for neural network inference (i.e., computing the output of a neural network given an input) use integer arithmetic wherever possible. Thus, it may be undesirable to have to reconstruct parameters into a floating-point representation. [Prior art documents] [Non-patent literature]

[0013] [Non-Patent Document 1] “cuDNN: Efficient Primitives for Deep Learning” (Sharan Chetlur, et al.; arXiv: 1410.0759, 2014) Summary of the Invention [Problem to be solved by the invention]

[0014] It is therefore desirable to improve the representation concepts of neural network parameters to support efficient encoding and / or decoding of such parameters. It may be desirable to reduce the bitstream in which neural network parameters are encoded to reduce signaling costs. Additionally or alternatively, it may be desirable to reduce the complexity of computational resources to improve neural network inference, e.g., to achieve an efficient implementation for neural network inference. [Means for solving the problem]

[0015] This is achieved by the subject matter of the independent claims of the present application.

[0016] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application. According to a first aspect of the present invention, the inventors of the present application have recognized that one problem faced by neural network (NN) representations arises from the fact that neural networks contain millions of parameters and therefore may require hundreds of megabytes for their representation. The inference procedure therefore requires high computational resources to execute, since it involves the calculation of many dot product operations between large matrices. According to a first aspect of the present application, this difficulty is overcome by using quantization of the NN parameters, which allows inference with little or no floating point operations. The inventors have found that it is advantageous to determine a quantization parameter from which the multiplier and bit shift numbers can be derived. This is based on the idea that it is efficient in terms of bit rate to signal only the quantization parameter and the quantization value instead of the 32-bit floating point value. Since the quantized values ​​of the NN parameters can be calculated using a multiplier, a bit shift number, and a quantization value, it is possible to perform calculations, such as sums of NN parameters and / or multiplications of NN parameters with vectors, in the integer domain instead of the floating-point domain. Thus, efficient calculations of inference can be achieved using the presented NN representation.

[0017] Thus, according to a first aspect of the present application, an apparatus for generating a NN representation, e.g., a data stream, is configured to quantize the NN parameters to quantized values ​​by determining quantization parameters and quantization values ​​for the NN parameters, such that the multipliers and bit shifts are derivable from the quantization parameters. The generated NN representation can be read / decoded by an apparatus for deriving NN parameters, e.g., quantized values ​​for the NN parameters, from the NN representation, e.g., a data stream. The apparatus for deriving NN parameters is configured to derive the quantization parameters and quantization values ​​from the NN representation, and to derive the multipliers and bit shifts from the quantization parameters. The multiplier can be derived from the quantization parameter based on the remainder of the division between the dividend derived by the quantization parameter and the divisor derived by the precision parameter, for example the precision parameter can be set to a default value, or multiple different integer values ​​of the precision parameter, such as natural numbers or powers of 2, can be tested by the device for the entire NN or for each part of the NN, such as each layer, and the best one is tested in terms of quantization error and bit rate with respect to its Lagrangian sum, obtaining the best value for the precision parameter and signaling this selection in the NN representation. The bit shift number can be derived from the quantization parameter based on the rounding of the quotient of the division. The NN parameter in the case of an apparatus for deriving NN parameters, or the quantized value of the NN parameter in the case of an apparatus for generating a NN representation, corresponds to the product between the quantized value and a factor depending on the multiplier (e.g., at least with respect to the absolute value of the quantized value with separate treatment of the sign in case of shifts, or additionally with respect to both the absolute value and the sign, such as when using two's complement representation and two's complement arithmetic for the product, its factors and shifts, respectively), bit-shifted by a number of bits depending on the bit shift number. Digital data can define a NN representation including the quantized parameter and the quantized value to represent the NN parameter, as described above.

[0018] It should be noted that the NN parameters derived by the device for deriving NN parameters correspond to the quantized values ​​of the NN parameters generated by the device for generating the NN representation, because the device for deriving NN parameters does not see the original NN parameters, and therefore sees the quantized values ​​of the NN parameters as the NN parameters from the perspective of the device for deriving NN parameters.

[0019] One embodiment relates to a device for performing inference using a NN, the device comprising a NN parameterizer configured to parameterize the NN. The NN parameterizer comprises an apparatus for deriving NN parameters from a NN representation as described above. In addition, the device comprises a computation unit configured to compute an inference output based on a NN input using the NN. As described above, the NN parameters can be derived based on a multiplier, a bit shift number and a quantization value, so that it is possible to perform computations, such as sums of NN parameters and / or multiplications of NN parameters with vectors, in the integer domain instead of the floating point domain. Thus, efficient computation of inference can be achieved by the device.

[0020] According to a second aspect of the present invention, the inventors of the present application have recognized that one problem encountered when performing inference using a neural network (NN) is due to the fact that the weight matrix used for inference may have quantization errors, and therefore only a low level of accuracy is achieved. According to a first aspect of the present application, this difficulty is overcome by using a transpose vector s, e.g. a scaling factor, which is multiplied element-wise with each column of the weight matrix W'. The inventors have found that arithmetic coding methods provide higher coding gains by using a scaling of the weight matrix and / or that the scaling of the weight matrix increases the neural network performance results, e.g. achieving higher accuracy. This is based on the idea that the transpose vector s can be efficiently adapted, e.g. depending on the weight matrix, e.g. a quantized weight matrix, to reduce the quantization error and thereby improve the prediction performance of the quantized neural network. Furthermore, the inventors have found that by factorizing the weighting parameters as a combination of the transpose vector s and the weighting matrix W', the two can be quantized independently, e.g., the transpose vector s and the weighting matrix W' can be quantized using different quantization parameters, thus increasing the efficiency of the representation, which is beneficial from a performance standpoint, but also from a hardware efficiency standpoint.

[0021] Thus, according to a second aspect of the present application, a device for performing inference using a NN is configured to use a NN to compute an inference output based on a NN input. The NN includes a pair of NN layers and inter-neuron activation feedforward from a first NN layer of the pair of NN layers to a second NN layer of the pair of NN layers. The device is configured to compute activations of neural network neurons of a second NN layer based on activations of neural network neurons of the first NN layer by forming a matrix X from activations of neural network neurons of the first NN layer and computing s·W'*X. The operator * denotes matrix multiplication, W' is a weight matrix of dimension n×m with n and m∈N, s is a transposed vector of length n, and the operator · denotes column-wise Hadamard multiplication between a matrix on one side of · and a transposed vector on the other side of ·.

[0022] According to a third aspect of the present invention, the inventors of the present application have recognized that one problem encountered when using batch norm layers arises from the fact that the batch norm parameters / elements of a batch norm operator are typically in floating point representation. However, efficient implementations for neural network inference (i.e., computing the output of a neural network given an input) use integer arithmetic whenever possible. This difficulty can be solved by assigning predetermined constant values ​​to the batch norm parameters / elements, e.g. b and μ and σ. 2 or σ. We have found that if the batch norm parameters / elements have a predefined constant value, they can be compressed very efficiently. This is based on the idea that it allows the use of a single flag to indicate whether all elements / parameters have a predefined constant value, and thus allows them to be set to the predefined constant value. In addition, we have found that the result of the batch norm operator is not changed by using a predefined constant value.

[0023] Therefore, according to a third aspect of the present application, the first embodiment relates to an apparatus for encoding NN parameters of a batch norm operator of a NN into a NN representation.

number

number

number

number

[0024] A parallel device that decodes the NN parameters of the batch norm operator of the NN from the NN representation derives γ and β from the NN representation and calculates σ' by a single signaling applied to all its components. 2 The device is configured to infer or derive σ′ :=θ and μ′ :=0 and b′ :=0, where θ is a predetermined parameter. The device may, for example, read a signaling, for example a flag, and derive σ′ 2 :=θ and μ′:=0 and b′:=0. The batch norm operator is defined as explained above with respect to the first embodiment of the third aspect.

[0025] Therefore, according to a third aspect of the present application, the second embodiment relates to an apparatus for encoding NN parameters of a batch norm operator of a NN into a NN representation.

number

number

number

number

[0026] A parallel device that decodes the NN parameters of the batch norm operator of the NN from the NN representation derives γ and β from the NN representation and, by a single signaling applied to all its components, σ 2 The device is configured to infer or derive σ :=1 and μ :=0 and b :=0. The device may, for example, read a signaling, for example a flag, and derive therefrom σ 2 The batch norm operator is configured to infer or derive μ :=1 and μ :=0 and b :=0. The batch norm operator is defined as explained above with respect to the second embodiment of the third aspect.

[0027] Therefore, according to a third aspect of the present application, a third embodiment relates to an apparatus for encoding NN parameters of a batch norm operator of a NN into a NN representation.

number

number

number

number

[0028] A parallel device that decodes the NN parameters of the batch norm operator of the NN from the NN representation derives γ and β from the NN representation and, by a single signaling applied to all its components, σ 2 The device is configured to infer or derive σ :=θ and μ :=0, where θ is a predetermined parameter. The device may, for example, read a signaling, for example a flag, and derive therefrom σ 2 :=θ and μ:=0. The batch norm operator is defined as explained above with respect to the third embodiment of the third aspect.

[0029] Therefore, according to a third aspect of the present application, a fourth embodiment relates to an apparatus for encoding NN parameters of a batch norm operator of a NN into a NN representation.

number

number

number

number

[0030] A parallel device that decodes the NN parameters of the batch norm operator of the NN from the NN representation derives γ and β from the NN representation and, by a single signaling applied to all its components, σ 2 The device is configured to infer or derive σ :=1 and μ :=0. The device may, for example, read a signaling, for example a flag, and derive therefrom σ 2 The batch norm operator is configured to infer or derive μ:=1 and μ:=0. The batch norm operator is defined as explained above with respect to the fourth embodiment of the third aspect.

[0031] The following method operates according to the principles described above.

[0032] One embodiment relates to a method for generating a NN representation, comprising quantizing an NN parameter to a quantized value by determining a quantization parameter and a quantization value for the NN parameter such that from the quantization parameter a multiplier is derivable based on a remainder of a division between a dividend derived by the quantization parameter and a divisor derived by a precision parameter, and a bit shift number is derivable from the quantization parameter based on a rounding of a quotient of the division. The quantization parameter is determined such that the quantized value of the NN parameter corresponds to a product of the quantization value and a factor depending on the multiplier, bit shifted by a number of bits depending on the bit shift number.

[0033] One embodiment relates to a method of deriving NN parameters from a NN representation, the method including deriving a quantization parameter and a quantization value from the NN representation. Additionally, the method includes deriving a multiplier from the quantization parameter based on a remainder of a division between a dividend derived by the quantization parameter and a divisor derived by a precision parameter, and deriving a bit shift number from the quantization parameter based on a rounding of a quotient of the division. The NN parameters correspond to a product of the quantization value and a factor depending on the multiplier, bit shifted by a number of bits depending on the bit shift number.

[0034] One embodiment relates to a method of performing inference using a NN, comprising parameterizing the NN using the above-described method for deriving NN parameters to derive NN parameters from a NN representation, in addition to the method of performing inference comprising using the NN to compute an inference output based on a NN input.

[0035] One embodiment relates to a method of performing inference using a NN, the method including using the NN to compute inference outputs based on NN inputs, the NN including a pair of NN layers and inter-neuron activation feedforward from a first NN layer of the pair of NN layers to a second NN layer of the pair of NN layers. The method includes computing activations of neural network neurons of a second NN layer based on activations of the neural network neurons of the first NN layer by forming a matrix X from activations of neural network neurons of the first NN layer and computing s·W'*X, where * represents matrix multiplication, W' is a weight matrix of dimension n×m with n and m∈N, s is a transpose vector of length n, and · denotes column-wise Hadamard multiplication between a matrix on one side of · and a transpose vector on the other side of ·.

[0036] One embodiment relates to a method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator being:

number

number

number

number

[0037] One embodiment relates to a method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator being:

number

number

number

number

[0038] One embodiment relates to a method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator being:

number

number

number

number

[0039] One embodiment relates to a method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator being:

number

number

number

number

[0040] One embodiment relates to a method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator being

number

[0041] One embodiment relates to a method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator being

number

[0042] One embodiment relates to a method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator being

number

[0043] One embodiment relates to a method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator being

number

[0044] As mentioned above, these methods are based on the same considerations as the apparatus or devices described above, and the methods may be completed with all the features and functions that are also described with respect to the apparatus or devices.

[0045] One embodiment relates to a digital storage medium containing digital data defining a NN representation generated by the method or apparatus for generating a NN representation as described above.

[0046] One embodiment relates to a computer program for carrying out one of the methods described above.

[0047] One embodiment relates to a data stream generated by a method or apparatus for generating a neural network representation, as described above.

[0048] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: [Brief description of the drawings]

[0049] [Figure 1] FIG. 1 illustrates a neural network. [Diagram 2] 1A-1D are schematic diagrams illustrating an apparatus for generating a NN representation, digital data defining the NN representation, and an apparatus for deriving NN parameters from the NN representation, according to an embodiment of the present invention; [Diagram 3]FIG. 1 is a schematic diagram of a feedforward neural network. [Figure 4] FIG. 1 illustrates a schematic diagram of a device for performing inference using a NN parameterizer, according to an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates a schematic diagram of a device for performing inference by factoring weight parameters as a composition of vectors and matrices, according to an embodiment of the present invention. [Figure 6] 1A and 1B illustrate schematic diagrams of an apparatus for encoding NN parameters into an NN representation and an apparatus for decoding NN parameters from the NN representation according to an embodiment of the present invention; [Figure 7] FIG. 2 shows a schematic diagram of a possible relationship between matrices X and W. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0050] Identical or equivalent elements, or elements having identical or equivalent functionality, are designated by the same or equivalent reference numbers in the following description, even if they appear in different figures.

[0051] In the following description, a number of details are set forth to provide a more complete description of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention. In addition, features of different embodiments described hereinafter may be combined with each other unless otherwise specified.

[0052] In the following, we present an efficient way to represent and signal the quantization step sizes of layer parameters that allows inference using only a small number of floating-point operations, or even no floating-point operations at all, i.e., a representation that is efficient in terms of bitrate, and can further be exploited for efficient computation of inference.

[0053] 2 illustrates an apparatus 100 for generating a NN representation 110. The apparatus 100 is configured to quantize the NN parameters 120 to a quantized value 130 by determining (140) a quantization parameter 142 and determining (150) a quantization value 152 for the NN parameters 120. The quantization value 152 may be determined (150) based on the quantization parameter 142. The determination (140) of the quantization parameter 142 may be performed by a quantization parameter determiner. The determination (150) of the quantization value 152 may be performed by a quantization value determiner.

[0054] A quantization parameter 142 is determined (140) such that a multiplier 144 and a number of bit shifts 146 can be derived from the quantization parameter 142. In determining (140) the quantization parameter 142, the apparatus 100 may, for example, already check whether the multiplier 144 and the number of bit shifts 146 can be derived from the determined quantization parameter 142.

[0055] Optionally, the apparatus 100 may be configured to derive the multiplier 144 from the quantization parameter 142 and to derive the bit shift number 146 from the quantization parameter 142, for example, to enable the apparatus 100 to determine the quantized value 130. However, this is not necessary, as the quantized value 130 may be represented by the quantization parameter 142 and the quantized value 152. The apparatus 100 does not need to explicitly determine the quantized value 130.

[0056] According to one embodiment, the generated NN representation 110 may include determined quantization parameters 142 and determined quantization values ​​152, such that the NN parameters 120, i.e., the quantized values ​​130 of the NN parameters 120, are derivable from the NN representation 110. For example, the apparatus 100 may be configured to encode the quantization parameters 142 and the quantization values ​​152 into the NN representation 110.

[0057] The multiplier 144 is derivable from the quantization parameter 142 based on the remainder of a division between a dividend derived by the quantization parameter 142 and a divisor derived by a precision parameter k 145 .

[0058] The bit shift number 146 can be derived from the quantization parameter 142 based on the rounding of the quotient of the division, i.e., based on the rounding of the quotient of the division between the dividend derived by the quantization parameter 142 and the divisor derived by the precision parameter k 145.

[0059] A determination (140) of the quantization parameters 142 is performed, whereby the quantized value 130 of the NN parameters 120 corresponds to the product between the quantization value 152 and a factor 148 depending on the multiplier 144, bit-shifted by a number of bits depending on the bit-shift number 146. The quantized value 130 of the NN parameters 120 corresponds to the product, for example at least with respect to the absolute value of the quantized value with separate treatment of the sign in case of the shift, or even with respect to both the absolute value and the sign, such as when using two's complement representation and two's complement arithmetic for the product, its factors and shifts, respectively. This is exemplarily and diagrammatically shown in unit 150.

[0060] According to one embodiment, the apparatus 100 is configured to provide quantized values ​​130 of NN parameters, e.g., NN parameters 120, by training the NN 20 using floating-point representations for the NN parameters and by determining quantization parameters 142 and quantization values ​​152 for the NN parameters by an iterative optimization scheme aimed at reducing quantization errors.

[0061] Apart from the device 100, Fig. 1 shows digital data 200 defining the NN representation 110 and a device 300 for deriving NN parameters, i.e. quantized values ​​130 of the NN parameters 120, from the NN representation 110. Due to the fact that the digital data 200 and the device 300 do not see the original values ​​of the NN parameters, the quantized values ​​130 are understood in this context as the values ​​of the NN parameters. For this reason, the NN parameters are shown as 130 for the following description of the digital data 200 and the device 300. It is clear that the NN parameters described herein can be represented by the original values ​​120 assigned to the NN parameters or by the quantized values ​​130 determined on the basis of the original values ​​120. Therefore, in the following, the NN parameters are shown as 120 / 130. This is to illustrate a feature that is generally applicable whether the NN parameters are represented by the original values ​​120 or by the quantized values ​​130.

[0062] The digital data 200 defines the NN representation 110, which includes a quantization parameter 142 and a quantization value 152 to represent the NN parameters 130, from which a multiplier 144 can be derived based on a remainder of a division between a dividend derived by the quantization parameter 142 and a divisor derived by a precision parameter k 145, and from which a bit shift number 146 can be derived based on a rounding of a quotient of the division. The NN representation 110 includes the quantization parameter 142 and the quantization value 152, so that the NN parameters 130 correspond to a product between the quantization value 152 and a factor 148 depending on the multiplier 144, bit shifted by a number of bits depending on the bit shift number 146.

[0063] The apparatus 300 for deriving the NN parameters 130 from the NN representation 110 is configured to derive the quantization parameter 142 from the NN representation 110, for example using a quantization parameter derivation unit 310, and to derive the quantization value 152 from the NN representation 110, for example using a quantization value derivation unit 320. In addition, the apparatus 300 is configured to derive the multiplier 144 and the bit shift number 146 from the quantization parameter 142. The apparatus 300 is configured to derive the multiplier 144 based on a remainder of a division between a dividend derived by the quantization parameter 142 and a divisor derived by the precision parameter 145, and to derive the bit shift number 146 based on a rounding of a quotient of the division. The derivation of the multiplier 144 may be performed using the multiplier derivation unit 330, and the derivation of the bit shift number 146 may be performed using a bit shift number derivation unit 340. The NN parameters 130 correspond to the product between the quantization value 152 and a factor 148 depending on the multiplier 144, bit-shifted by a number of bits depending on the bit-shift number 146 (see the corresponding description above for the apparatus 100 in FIG. 2 and the unit 150). The NN parameters 130 may for example be derived using an NN parameters derivation unit 350. The NN parameters derivation unit 350 may comprise the same features and / or functionality as the optional unit 150 of the apparatus 100.

[0064] In the following, embodiments and examples applicable to both the device 100 and the device 300 are presented.

[0065] According to one embodiment, the NN parameters 120 / 130 are one of a weight parameter, a batch norm parameter, and a bias. The weight parameter, e.g., a component w of W, may be usable to weight the interneuron activation feedforward between a pair of neurons, or alternatively, may represent a weight associated with an edge connecting a first neuron and a second neuron, weighting the forwarding of the activation of the first neuron in the sum of the inbound activation for the second neuron. The batch norm parameters, e.g., μ, σ2 , γ, β can be used to parameterize the affine transformation of the neural network layer, and the bias, e.g., b i may be usable to bias the sum of inbound inter-neuron activation feedforward for a given neural network neuron.

[0066] According to one embodiment, the NN parameters 120 / 130 are determined by the inter-neuron activation feedforward 12 of a plurality of inter-neuron activation feedforwards 122 of the NN, as shown, for example, in FIG. i , e.g., with respect to a component w of W. The apparatus 100 / 300 is configured to, for each of the plurality of inter-neuron activation feedforwards 122, encode / derive corresponding NN parameters 120 / 130 into / from the NN representation 110. The corresponding NN parameters 130 are included in the NN representation 110. In this case, the apparatus 100 encodes / derives corresponding inter-neuron activation feedforwards 122, e.g., with respect to a component w of W. i and the associated quantization parameters142 associated with each inter-neuron activation feedforward12 i The determination of the associated quantization parameters 142 may be configured to quantize the corresponding NN parameters 120 to the corresponding quantized values ​​130 by determining (140) an associated quantization value 152 associated with each inter-neuron activation feedforward 122 from the associated quantization parameters 142. i The feedforward activation between the respective neurons is calculated based on the remainder of the division between the divisor derived by the associated precision parameter 145 associated with i and the associated multiplier 144 associated with each neuron based on the rounding of the quotient of the division, and the feedforward activation 12 between the neurons. iThe corresponding apparatus 300 in this case is implemented to derive, for each of the plurality of inter-neuron activation feedforwards 122, the respective inter-neuron activation feedforwards 122 from the NN representation 110. i derive (310) the associated quantization parameters 142 associated with each inter-neuron activation feedforward 12 from the NN representation 110; i The derivation (310 and 320) may be performed, for example, by decoding from the NN representation 110, i.e., one for each edge. In addition, the apparatus 300 may derive from the associated quantization parameter 142, for each of the plurality of inter-neuron activation feedforwards 122, a dividend derived by the associated quantization parameter 142 and a quantization value 152 associated with the respective inter-neuron activation feedforward 122. i Based on the remainder of the division between the divisor derived by the associated precision parameter 145 associated with each neuron, the inter-neuron activation feedforward 12 i Based on the associated multiplier 144 associated with each of the divisions and the rounding of the quotient, the activation feedforward 12 between the neurons is calculated. i The method is then configured to derive (see 330 and 340) an associated bit shift number 146 associated with x, y, and x, y ...

[0067] According to another embodiment, the device 100 / device 300 is configured to subdivide the multiple inter-neuron activation feedforwards 122 of the NN 20 into subgroups 122a, 122b of inter-neuron activation feedforwards, whereby each subgroup is associated with an associated pair of NN layers of the NN, includes inter-neuron activation feedforwards between the associated pair of NN layers, and excludes inter-neuron activation feedforwards between a further pair of NN layers other than the associated pair of layers, and two or more subgroups are associated with a given NN layer (see, for example, FIG. 3). 1 and a pair of associated NN layers 114 and 116 1 The pair of layers 114 and 116 includes inter-neuron activation feedforward between the 1 Between the further pair of NN layers other than the further pair of NN layers 116, for example, 1 and 116 2 The subgroups 122a and 122b exclude the inter-neuron activation feedforward between the layers 116. 1 The subdivision of the inter-neuron activation feedforwards 122 of the NN 20 may be performed, for example, by indexing each edge / weight 12 in the NN 20 or by segmenting the edges 12 between each layer pair in other ways. The NN parameters 120 / 130 are associated with a single inter-neuron activation feedforward 12 of the inter-neuron activation feedforwards 122 of the NN 20. iFor each of the plurality of inter-neuron activation feedforwards 122, corresponding NN parameters 120 / 130 are included in the NN representation 110. The apparatus 300 is configured to derive, for each of the plurality of inter-neuron activation feedforwards 122, corresponding NN parameters 120 / 130 from the NN representation 110, for example by decoding from the NN representation, i.e. by decoding one subgroup per edge subgroup. The apparatus 100 / apparatus 300 is configured to determine (140) / derive (310) for each subgroup 122a, 122b of inter-neuron activation feedforwards an associated quantization parameter 142 associated with the respective subgroup 122a or 122b. The quantization parameter 142 is determined (140) by the apparatus 100 such that an associated multiplier 144 associated with each subgroup 122a or 122b is derivable from the quantization parameter 142 based on a remainder of a division between a dividend derived by the associated quantization parameter 142 and a divisor derived by an associated precision parameter 145 associated with each subgroup, and the quantization parameter 142 is determined (140) by the apparatus 100 such that an associated bit shift number 146 associated with each subgroup 122a or 122b is derivable from the quantization parameter 142 based on a rounding of a quotient of the division. The apparatus 300 is configured to derive the associated multiplier 144 and the associated bit shift number 146 from the NN representation 110. The apparatus 100 / apparatus 300 determines (140) a respective inter-neuron activation feedforward 122 from the NN representation 110 for each of the plurality of inter-neuron activation feedforwards 122. i NN representation 110, i.e., one for each edge is decoded. i The corresponding NN parameters 120 / 130 of each neuron are the associated quantization values ​​142 and the respective inter-neuron activation feedforward 12 i122a or 122b, which is a product of the factor 148 depending on the associated multiplier 144 associated with the subgroup, e.g., i corresponds to the product bit-shifted by a number of bits that depends on the associated bit-shift number 146 of the subgroup it is included in, e.g. 122a or 122b.

[0068] The associated accuracy parameters 145 may be, for example, across the NN 20 or for each NN layer 114, 116. 1 , and 116 2 Optionally, the apparatus 100 / 300 is configured to encode / derive an associated accuracy parameter 145 into / from the NN representation 110.

[0069] According to an embodiment, the apparatus 100 / 300 is configured to encode / derive the quantization parameters 142 in / from the NN representation 110 by using context-adaptive binary arithmetic encoding / decoding, or by directly writing / reading bits representing the quantization parameters 142 in / from the NN representation 110, or by encoding / deriving bits representing the quantization parameters 142 from the NN representation 110 via an equal probability bypass mode of the context-adaptive binary encoder / decoder of the apparatus 100 / 300. The apparatus 100 / 300 may be configured to derive the quantization parameters 142 from the NN representation 110 by binarizing / de-binarizing the bin string using a binarization scheme, e.g., exponential-Golomb coding.

[0070] According to one embodiment, the apparatus 100 is configured to determine (140) the quantization parameter 142 and encode it in the NN representation 110 in the form of a fixed-point representation, e.g., a two's complement representation. The apparatus 300 may be configured to derive (310) the quantization parameter 142 from the NN representation 110 in the form of a fixed-point representation, e.g., a two's complement representation. Optionally, the precision parameter 145 is 2 tand the bit length of the fixed-point representation, for example the two's complement representation, is set to be constant for the NN 20, or is set to be the sum of a base bit length that is constant for the NN 20 and t.

[0071] According to one embodiment, the apparatus 100 / 300 is configured to encode / derive the quantization parameter 142 into / from the NN representation 110 as an integer-valued syntax element.

[0072] According to one embodiment, the apparatus 100 is configured to determine the quantization value 152 and encode it in the NN representation 110 in the form of a fixed-point representation, e.g., a two's complement representation. The apparatus 300 may be configured to derive (320) the quantization value 152 from the NN representation 110 in the form of a fixed-point representation, e.g., a two's complement representation.

[0073] According to one embodiment, the apparatus 100 / 300 is configured to encode / derive the quantized values ​​152 to / from the NN representation 110 by binarizing / de-binarizing the quantized values ​​152 to / from a bin string according to a binarization scheme and encoding / decoding the bits of the bin string using context-adaptive arithmetic encoding / decoding.

[0074] According to one embodiment, the apparatus 100 / 300 is configured to encode / decode the quantized values ​​152 to / from the NN representation 110 by binarizing / de-binarizing the quantized values ​​152 to / from a bin string according to a binarization scheme, encoding / decoding a first bit of the bin string using context-adaptive arithmetic encoding / decoding, and encoding / decoding a second bit of the bin string using an equal probability bypass mode.

[0075] According to one embodiment, the quantization step size Δ149 may be derived by the device 100 and / or device 300 from the signed integer indicated by the quantization parameter QP 142 and the positive integer parameter k, i.e., the precision parameter 145, according to the following formula: mul=k+QP%k

number

[0076] The multiplier 144 is represented by mul, the bit shift number 146 is represented by shift, and the factor 148 is represented by mul / k.

[0077] The NN parameter 130 is (mul / k)·2 shift ·P, where P is the quantization value 152.

[0078] Floor Operator

number

number

number

[0079] Optionally, the device 100 and / or the device 300 may be configured to set the accuracy parameter k145 to a default value.

[0080] Alternatively, the apparatus 100 can optionally test multiple different integer values ​​of the precision parameter k145, such as natural numbers or powers of two. Different integer values ​​are tested, for example, for the entire NN or for each part of the NN, such as each layer, and the best precision parameter k145 is selected, for example, for its Lagrangian sum, with respect to quantization error and bit rate. The apparatus 100 may be configured, for example, to determine the precision parameter k145 and check, for example, in the determination (140), whether the multiplier 144 and the bit shift number 146 are derivable from the quantization parameter 142. Optionally, the precision parameter k145 selected by the apparatus 100 is signaled in the NN representation 110, for example, encoded in the NN representation 110. The apparatus 300 is configured, for example, to derive the precision parameter k145 from the NN representation 110.

[0081] According to one embodiment, the precision parameter 145 is a power of two.

[0082] According to one embodiment, the apparatus 100 / apparatus 300 is configured to encode / derive the precision parameter 145 into / from the NN representation 110 by writing / reading bits representing the precision parameter 145 directly into / from the NN representation 110 or by deriving bits representing the precision parameter 145 into / from the NN representation 110 via an equal probability bypass mode of the context-adaptive binary encoder / decoder of the apparatus 100 / apparatus 300.

[0083] Instead of signaling 32-bit floating point values ​​in the bitstream, e.g., digital data 200, only parameters QP 142 and k 145 need to be signaled. In some applications, it may be sufficient to signal QP 142 in the bitstream and set k 145 to some fixed value.

[0084] In a preferred embodiment, the parameter QP'=QP-QP 0is signaled in the bitstream instead of QP142, where the parameter QP 0 is a predetermined constant value. In other words, according to one embodiment, the device 100 / device 300 calculates the related quantization parameter QP 142 as the reference quantization parameter QP 0 The NN representation 110 is encoded / derived from the NN representation 110 in the form of a difference between the NN representation 110 and the NN representation 110.

[0085] In another preferred embodiment, k145 is 2 t In this way, the calculation of Δ149 can be performed without division as follows: Δ=mul 2 shift-t

[0086] This allows some calculations to be performed in the integer domain instead of the floating point domain, as exemplified below.

[0087] 4 illustrates a schematic diagram of a device 400 for performing inference using a NN 20. The device 400 comprises a NN parameterizer 410 configured to parameterize the NN 20. The NN parameterizer 410 comprises an apparatus 300 for deriving NN parameters 130 from a NN representation 110. The apparatus 300 for deriving NN parameters 130 may comprise the same or similar features as described with respect to the apparatus 300 of FIG. 2. The apparatus 300 may be understood as a NN parameter derivation unit. In addition, the device 400 comprises a calculation unit 420 configured to calculate an inference output 430 based on a NN input 440 using the NN 20, e.g., using a parameterization 450 of the NN 20 determined by the NN parameterizer 410.

[0088] Example 1: According to one embodiment, the NN parameterizer 410 is configured to derive, via the apparatus 300, at least one of a first NN parameter and a second NN parameter, whereby the first NN parameter corresponds to a product between a first quantized value and a first factor, bit-shifted by a first number of bits, and the second NN parameter corresponds to a product between a second quantized value and a second factor, bit-shifted by a second number of bits.

[0089] The first quantized value and the second quantized value both represent the quantized value shown at 152 in Figure 2. The first factor and the second factor both represent the factor shown at 148 in Figure 2.

[0090] For example, t=2 and k=2 t Let QP a A first QP, i.e., a first quantization parameter 142, denoted by a , i.e. the first bit shift number is 146, mul a , i.e., the first multiplier 144, and Δ a , i.e., the first quantization step size 149.

[0091] In addition, Q.P. b A second QP, i.e., a second quantization parameter 142, denoted by b , i.e. the second bit shift number 146, mul b , i.e., the second multiplier 144, and Δ b , i.e., the second quantization step size 149.

[0092] Although the "first" and "second" parameters are designated in this context by the same reference number, it is clear that they may have different values. They are only designated by the same reference number in order to clarify to which feature shown in FIG. 2 they belong.

[0093] C=Δ a C aThe first quantization matrix C for which a Think about it. D=Δ b D b The second quantization matrix D b Think about it. That is, C a QP a It is quantized using D b QP b It is quantized using Both matrices have the same dimensions. The quantization value 152 described in FIG. a or D b For example, C a may include a plurality of first quantization values ​​152, and D b may include a plurality of second quantized values ​​152.

[0094] Further assume that the sum C+D is calculated as follows:

number

[0095] The device 400 calculates a first quantized value C of a first NN parameter C. a The first multiplier is formed by a The first summand weighted by, say, mul a C a and the second quantized value D of the second NN parameter D b The second multiplier mul b and the difference between the first number of bits and the second number of bits, e.g.

number

number

number

[0096] Optionally, this determination / calculation may be performed by the calculation unit 420, where the calculation unit 420 is configured to add the first NN parameter C and the second NN parameter D to generate the final NN parameters of the NN 20, as described above.

[0097] As can be seen from the formula, there is no need to derive C and D, which may require floating-point arithmetic. Instead, C a The element of , i.e., the first quantized value 152, is simply mul a , i.e. multiplied by the first multiplier 144, D b The element of mul b , i.e., multiplied by the second multiplier 144,

number

[0098] According to one embodiment, the first NN parameter represents a base layer representation of the NN 20 and the second NN parameter represents an enhancement layer representation of the NN 20. Alternatively, the first NN parameter represents, for example, a current representation of the NN 20 and the second NN parameter represents an update of the current NN representation, i.e., an update of the current representation of the NN 20. Alternatively, for example, the first NN parameter represents a bias for biasing the sum of inbound inter-neuron activation feedforward of a given neural network neuron 10, i.e., b i The second NN parameters represent the components of the neural network layers 114, 116. 1 Or 116 2 The batch norm parameters μ, σ 2 , γ or β, for example b+μ.

[0099] Example 2: According to one embodiment, the NN parameterizer 410, via the apparatus 300, is configured to derive at least one of a third NN parameter and a fourth NN parameter, whereby the third NN parameter corresponds to a product between the third quantized value and the third factor, bit-shifted by a third number of bits, and the fourth NN parameter corresponds to a product between the fourth quantized value and the fourth factor, bit-shifted by a fourth number of bits.

[0100] The third quantized value and the fourth quantized value together represent the quantized value shown at 152 in FIG. 2. The third factor and the fourth factor together represent the factor shown at 148 in FIG.

[0101] For example, t=2 and k=2 t Let the first QP be, say, QP aThe third quantization parameter 142, denoted by a , i.e. the third bit shift number 146, mul a , i.e., the third multiplier 144, and Δ a , i.e., the third quantization step size 149.

[0102] Additionally, a second QP, e.g., QP b The fourth quantization parameter 142, denoted by b , i.e. the fourth bit shift number 146, mul b , i.e., the fourth multiplier 144, and Δ b , i.e., the fourth quantization step size 149.

[0103] Although the "third" and "fourth" parameters are denoted with the same reference number in this context, it is clear that they may have different values. They are only denoted with the same reference number in order to clarify to which feature shown in FIG. 2 they belong. The device 400 may be configured to derive only the third and / or fourth parameter or, as explained in Example 1 above, additionally the first and / or second parameter.

[0104] W=Δ a ·W a The quantization matrix W a Think about it. γ = Δ b ·γ b The quantized transposed vector γ b Think about it. That is, W a QP a quantized using γ b QP b It is quantized using The quantization value 152 described in FIG. a One component of or γ b For example, Wa may include multiple quantized values ​​152, and γ bmay include multiple quantized values ​​152.

[0105] Further assume that the element-wise product W γ is computed as follows:

number

[0106] This calculation / calculation may be performed, for example, by multiplying the third NN parameter W and the fourth NN parameter γ to obtain a third quantized value W a The first factor formed by and the third multiplier mul a and the fourth quantization value γ of the fourth NN parameter γ b The third factor formed by and the fourth multiplier mul b and a fourth factor formed by the third bit shift a The first addend and the fourth bit number shift b and a second addend formed by

number

[0107] As can be seen, there is no need to derive W and γ, which may require floating-point operations. Instead, the calculation mul a ·mul b ·W a ·γ b involves integer multiplication only,

number

[0108] According to one embodiment, the third NN parameter is the first neuron 10 of the first NN layer 114. 1 to the second NN layer 116 2 The second neuron 10 2 NN parameter 10 represents a weight parameter that weights the inter-neuron activation feedforward to the first neuron 10, e.g., a component w of W, or alternatively, 1 and the second neuron 10 2 Edge 12 connecting with i Related to the second neuron 10 2 The first neuron in the sum of inbound activations of 1 represents the weight that weights the transfer of activations of

[0109] The fourth NN parameter is, for example, the batch norm parameter, e.g., μ, σ 2 , γ or β. The batch norm parameter is, for example, 1 The first neuron for 1 The activation feed-forward amplification of the signal is adjusted, for example, by γ.

[0110] Quantization of input X According to one embodiment, the device 400 is configured to quantize the NN input X 440 by determining a fifth quantization parameter QP, i.e. the quantization parameter 142, and a fifth quantization value, e.g. X′, i.e. the quantization value 152, for the activations by quantizing the activations to a quantized value, e.g. X″, for example using the apparatus 300, whereby the derivation of the fifth multiplier mul, i.e. the multiplier 144 from the fifth quantization parameter QP results in the quantized value corresponding to the product between the fifth quantization value and a factor mul / k, i.e. the factor 148, which depends on the fifth multiplier, being bit-shifted by a fifth number of bits, which depends on the fifth bit-shift number, based on the remainder of the division between the dividend derived by the fifth quantization parameter and the divisor derived by the precision parameter k, i.e. the precision parameter 145, associated with the activation, and a fifth bit-shift number shift, i.e. the bit-shift number 146, based on the rounding of the quotient of the division.

[0111] In a preferred embodiment, the input X 440 of the bias layer or batch normalization layer is also quantized using the quantization method of the present invention, see the description of the device 100 in FIG. 2. That is, X″=Δ·X′=mul·2 shift-t ·X', the quantization parameter QP and related variables, associated shift, mul, and Δ(t=2 and k=2 t ) is selected and X is quantized to X'. Then, instead of using X to run a bias layer or a batch norm layer, X'' is used as the input. Note that X' can usually be represented with many fewer bits / elements than X, which is another advantage for efficient hardware or software implementation.

[0112] According to one embodiment, the NN parameterizer 410 is configured to derive the sixth NN parameter via the apparatus 300, whereby the sixth NN parameter is a product of the sixth quantization value and the sixth factor mul / k, bit-shifted by a sixth number of bits. The device 400 is configured to multiply the sixth NN parameter and the activations to generate a product by forming a product of a first factor formed by the sixth quantization value for the sixth NN parameter, a second factor formed by the sixth multiplier, a third factor formed by the fifth quantization value, and a fourth factor formed by the fifth multiplier, the product being bit-shifted by a number of bits corresponding to a sum including a first addend formed by the sixth number of bits and a second addend formed by the fourth number of bits.

[0113] According to one embodiment, the sixth NN parameter represents a weight parameter W that weights the input 440, so that the product W*X can be calculated / computed.

[0114] Referring back to FIG. 2, further optional features of device 100 and / or device 300 are described below.

[0115] Efficient encoding and decoding of the parameter QP In a preferred embodiment, the parameter QP, ie the quantization parameter 142, is encoded / decoded into / from the bitstream 200 by the device 100 / 300 using a signed exponential-Golomb code of order K according to the following definition:

[0116] Another preferred embodiment is the same as the previous preferred embodiment with the order K set to 0.

[0117] Exponential-Golomb coding of unsigned integers The unsigned exponential Golomb code of an unsigned integer complies with the decoding specifications of the syntax element ue(v) defined in the High Efficiency Video Coding (HEVC) standard.

[0118] The specification is briefly described below.

[0119] The decoding of the unsigned integer variable “decNum” from its binary representation encoded with an unsigned exponential-Golomb code of degree K is defined according to the following pseudocode: leadingZeroBits=-1 for(b=0;!b;leadingZeroBits++) b=read_bits(1)

[0120] Next, the variable codeNum is assigned as follows: decNum=(2 leadingZeroBits -1)*2 K +read_bits(leadingZeroBits+K)

[0121] The function read_bits(x) reads x bits from the bitstream and returns them as an unsigned integer, ordered from most significant bit (MSB) to least significant bit (LSB).

[0122] Exponential-Golomb coding of signed integers The unsigned exponential Golomb code of a signed integer shall follow the decoding specifications of the syntax element se(v) defined in the High Efficiency Video Coding (HEVC) standard.

[0123] The specification is briefly described below.

[0124] The decoding of the signed integer "signedDecNum" from the signed exponential-Golomb coded binary representation is as follows: First, the unsigned integer is decoded according to the HEVC ue(v) syntax element decoding process as described above. Then, the unsigned integer is converted to a signed integer according to the following formula:

number

number

[0125] Further preferred embodiments In a preferred embodiment, the parameter k, i.e., the precision parameter 145, is set to 2 t and the parameter t is encoded using an unsigned integer representation having bits_t bits (eg, having bits_t=3 or bits_t=4).

[0126] In another preferred embodiment, the parameter k, i.e., the precision parameter 145, is 2 t and the parameter t is encoded using Exponential-Golomb coding for unsigned integers.

[0127] In another preferred embodiment, the parameter QP, ie, the quantization parameter 142, is encoded using an exponential-Golomb code for signed integers.

[0128] In another preferred embodiment, the parameter k, i.e., the precision parameter 145, is 2 t and the parameter QP is encoded using a signed integer in two's complement representation using bits_qp bits. Bits_qp is set to a constant value, for example 12 or 13, or bits_qp is set to bits_qp0+t, where bits_qp0 is a non-zero constant integer value (for example, bits_qp0=6).

[0129] In the case of a CABAC encoded bitstream 200, the bits representing the parameters t and / or QP 142 may either be encoded as bypass bins (using the bypass mode of CABAC) or written directly into the bitstream 200.

[0130] In another preferred embodiment, the parameters W, b, μ, σ 2 , γ, and β are each quantized with a respective QP 142 value that is coded immediately prior to coding of the parameter.

[0131] In another preferred embodiment, a first QP 142 is coded into the bitstream 200 and associated with a subset of the parameters of the model. For each parameter x of this subset, one QP offset QP x is coded per parameter, and the effective QP 142 used to dequantize the parameters, i.e., the NN parameters 120, is QP + QP x It is given as QP x The binary representation of preferably uses fewer bits than the binary representation of QP. For example, x is encoded using a signed integer or an exponential-Golomb code for a fixed number of bits (in two's complement representation).

[0132] Further embodiments regarding encoding of weight parameters A further preferred embodiment shown in Fig. 5 concerns the representation of the weight parameters W 545, i.e., factoring them as a combination W→s·W' of a vector 546 and a matrix 544. W and W', i.e., the weight matrix 544, are matrices of dimension n×m, and s is the transposed vector 546 of length n. Each element of the vector s 546 is used as a row-wise scaling factor of the weight matrix W' 544. In other words, s 546 is multiplied element-wise with each column of W' 544. We refer to s 546 as the local scaling factor or local scale adaptation (LSA).

[0133] 5 illustrates a device 500 for performing inference using a NN 20. The device 500 is configured to use the NN 20 to compute an inference output 430 based on an NN input 440. The NN 20 includes a pair of NN layers 114 and 116 and an inter-neuron activation feedforward 122 from a first layer 114 of the pair of NN layers to a second layer 116 of the NN layers. The device 500 may, for example, be configured to generate an inference output 430 based on an NN input 440 using a matrix formation unit 530 of the device 500. 1 The neural network neurons 10 of the first NN layer 114 are activated by forming a matrix X 532 from the activations 520 of 1 Based on the activation 520 of the neural network neurons 10 of the second NN layer 116, 2 In addition, the device 500 is configured to calculate (542) the activations of the neural network neurons 10 of the first NN layer 114 by calculating s·W′*X. 1 Based on the activation 520 of the neural network neurons 10 of the second NN layer 116, 2 The device 500 may comprise a computation unit 540 configured to perform a computation 542 configured to compute an activation 510 of , where * represents matrix multiplication, W' is a weight matrix 544 of dimension n×m with n and m∈N, s is a transposed vector 546 of length n, and · denotes a column-wise Hadamard multiplication between a matrix on one side of · and a transposed vector on the other side of ·.

[0134] According to one embodiment, transposed vector s 546 is the result of optimizing W' 544 for higher compression and / or higher inference fidelity for encoding W' 544.

[0135] The rationale is that LSA scales the weight matrix 544, which allows the arithmetic coding method to provide higher coding gain and / or increase the neural network performance results, e.g., achieve higher accuracy. For example, after quantization of W, s 546 can be adapted with or without using input data 440, e.g., X 532, to reduce the quantization error and thereby improve the prediction performance of the quantized neural network.

[0136] Thus, s 546 and W' 544 can have different quantization parameters, i.e. different QP. This can be beneficial not only from a performance point of view, but also from a hardware efficiency point of view. For example, W' 544 may be quantized such that the dot product with the input X 532 may be performed in an 8-bit representation, but the subsequent multiplication with the scaling factor s 546 may be performed in 16-bit. The device 500 is configured, for example, to compute the matrix multiplication W'*X using n-bit fixed-point arithmetic to obtain the dot product, and to multiply the dot product with s 546 using m-bit fixed-point arithmetic, where m>n.

[0137] However, even if W' 544 and s 546 are both quantized to n-bit representations, a smaller n than that required to quantize W 545 to obtain the same inference accuracy may be sufficient. Similarly, further advantages in terms of representation efficiency may be achieved if s 546 is quantized to a representation with fewer bits than W' 544.

[0138] According to one embodiment, the device 500 comprises a NN parameterizer configured to derive W' 544 from the NN representation 110, such as the NN parameterizer 410 shown in Fig. 4. The NN parameterizer comprises an apparatus for deriving NN parameters from the NN representation 110, such as the apparatus 300 shown in Fig. 4 or Fig. 2. The weight matrix W' 544 may be the NN parameter derived by the apparatus 300. Optionally, the NN parameterizer 410 is further configured to derive s 546 from the NN representation 110 using different quantization parameters 142 compared to the NN parameters associated with W' 544.

[0139] In a preferred embodiment, the encoding of the weight matrix W 544 is as follows: First, a flag is encoded indicating whether LSA is used or not. If the flag is 1, the parameters s 546 and W' 544 are encoded using a state-of-the-art parameter encoding scheme such as DeepCABAC. If the flag is 0, W 545 is encoded instead.

[0140] In another preferred embodiment, different QP values ​​are used for W' 544 and s 546 according to the previous preferred embodiment.

[0141] Batch Norm Compression One embodiment shown in Figure 6 relates to improving batch norm compression. Figure 6 illustrates the NN parameters 610, e.g., μ, σ, of a batch norm operator 710 of a NN. 2 , γ, β, and optionally b into a NN representation 110, and from the NN representation 110, NN parameters 610, e.g., γ 722 and β 724, and parameters 732, i.e., μ, σ 2 7 shows an apparatus 700 for decoding a, and optionally a. Four embodiments are shown, the first of which describes the general case, while the others target special cases.

[0142] In general, the batch norm operator 710 1 teeth,

number

[0143] In the second embodiment, the constant ε is 0, so that the batch norm operator 710 2 teeth,

number

[0144] In the third embodiment, the bias b is 0, so that the batch norm operator 710 3 teeth,

number

[0145] In the fourth embodiment, the bias b and the constant ε are 0, so that the batch norm operator 710 4 teeth,

number

[0146] 6, some parameters of the batch norm operator 710 have an apostrophe, allowing a distinction between the original parameters 610, denoted by parameters without the apostrophe, and the modified parameters 722, 724, and 732, denoted by parameters with the apostrophe. It is clear that either the original parameters 610 or the modified parameters 722, 724, and 732 can be used as parameters of one of the batch norm operators 710 defined above.

[0147] The apparatus 600 calculates the parameters μ, γ, β, and σ 2 or σ (610 1 ~610 4 ), and optionally configured to receive b (see 610 1 and 610 2 reference).

[0148] According to a first embodiment, the device 600 comprises:

number

number

[0149] According to a second alternative embodiment, the device 600 comprises:

number

number

[0150] According to an alternative third embodiment, the device 600 comprises:

number

number

[0151] According to an alternative fourth embodiment, the device 600 comprises:

number

number

[0152] The calculated parameters β' and γ' are encoded in the NN representation 110 as NN parameters of the batch norm operator 710, e.g., such that the same (β' and γ') are also transposed vectors containing one component for each output node.

[0153] Therefore, the batch norm operator 710 of the first embodiment 1 teeth,

number

number

number

number

[0154] The predetermined parameters are 1 or 1-ε, e.g., μ′, σ′ 2 , γ', and β' are transposed vectors with one component for each output node, W is the weight matrix, X is the input matrix derived from the activations of the NN layer, and b' is a transposed vector forming the bias, e.g., a transposed vector with one component for each output node.

[0155] The apparatus 700 is configured to derive γ and β, ie γ′ and β′, from the NN representation, for example by using a γ and β derivation unit 720 that may be included in the apparatus 700 .

[0156] According to the first embodiment, the device 700 receives, via a single signaling 734 applied to all its components, σ′ 2 The method is configured to infer or derive μ′:=θ, μ′:=0, and b′:=0, where θ is a predetermined parameter.

[0157] According to the second embodiment, the device 700 receives, via a single signaling 734 applied to all its components, σ′ 2 The method is configured to infer or derive that μ′:=1, μ′:=0, and b′:=0.

[0158] According to the third embodiment, the device 700 calculates σ′ by a single signaling 734 applied to all its components. 2 The method is configured to infer or derive μ′:=θ and μ′:=0, where θ is a predetermined parameter.

[0159] According to the fourth embodiment, the device 700 calculates σ′ by a single signaling 734 applied to all its components. 2The method is configured to infer or derive that μ′:=1 and μ′:=0.

[0160] Parameter σ' 2 , μ′, and optionally b′ may be performed using a parameter inference / derivation unit 730 .

[0161] The predetermined parameters are 1 or 1-ε, e.g., μ′, σ′ 2 , γ', and β' are transposed vectors with one component for each output node, W is the weight matrix, X is the input matrix derived from the activations of the NN layer, and b' is a transposed vector forming the bias, e.g., a transposed vector with one component for each output node.

[0162] 6, the parameters derived or inferred by the device 700 are indicated by apostrophes, however, due to the fact that the device 700 never sees the original parameters 610, the parameters derived or inferred by the device 700 may be indicated without the use of apostrophes. In view of the device 700, the derived or inferred parameters are the only existing parameters.

[0163] Optionally, the apparatus 700 may be configured to use a batch norm operator with the derived or inferred parameters 722, 724, and 732, e.g., for inference. The batch norm operator computation unit may be configured to use the batch norm operator. Alternatively, an inference device, e.g., device 400 or device 500, may comprise the apparatus 700 for obtaining parameters of the batch norm operator 710.

[0164] Introducing a constant scalar value θ, i.e., a given parameter that can be equal to, for example, 1 or 1-ε, the parameters b, μ, σ 2 , γ, and β can be modified without changing BN(X), i.e., the result of the batch norm operator 710, by the following ordered steps:

number

[0165] Each operation is interpreted as an element-wise operation on the elements of the transposed vector. Further modifications that do not change BN(X) are possible, as exemplified in embodiments 2-3. For example, the bias b and the mean μ are "integrated" to β, so that b and μ are then set to 0 (see the third embodiment). Or, σ 2 may be set to 1-ε (i.e., θ=1-ε) to set the denominator of the fraction in BN(X) equal to 1, when other parameters are adjusted accordingly.

[0166] Therefore, all vector elements have the same value, b, σ 2 , μ, and b can be compressed much more efficiently.

[0167] In a preferred embodiment, a flag 734 is encoded that indicates whether all elements of the parameters have a predetermined constant value. 2 , γ, or β. The predefined value may be, for example, 0, 1, or 1-ε. If the flag is equal to 1, all vector elements of the parameter are set to the predefined value. Otherwise, the parameter is encoded using one of the state-of-the-art parameter encoding methods, for example DeepCABAC.

[0168] In another preferred embodiment, a flag is coded for each parameter indicating whether all vector elements have the same value. When all vector elements have the same value, the flag is equal to 1 and the value is coded using a state-of-the-art parameter coding method, such as DeepCABAC, or Exponential-Golomb Codes, or Fixed-Length Codes. If the flag is 0, the vector elements of the parameter are coded using one of the state-of-the-art parameter coding methods, such as DeepCABAC.

[0169] According to one embodiment, the device 600 / 700 calculates in / from the representation 110, σ′ 2 In addition, or alternatively, the apparatus 600 / 700 is configured to indicate / derive in / from representation 110 that all components of μ′ (e.g., each component is for a corresponding row of W denoting a corresponding output node) are equal to one another and to indicate / derive the value thereof. In addition, or alternatively, the apparatus 600 / 700 is configured to indicate / derive in / from representation 119 that all components of b′ (e.g., each component is for a corresponding row of W denoting a corresponding output node) are equal to one another and to indicate / derive the value thereof, if present, e.g., in the case of the first and second embodiments but not in the case of the third and fourth embodiments.

[0170] According to one embodiment, the apparatus 600 is further configured to be switchable between two batch norm coding modes, in a first batch norm coding mode the apparatus 600 is configured to perform the calculation and encoding of β′ and γ′, and in a second batch norm coding mode the apparatus is configured to perform the calculation and encoding of the received μ, σ 2or σ, γ, and β, and, if present, b. In other words, the received parameters 610 are directly encoded into the representation 110 in the second batch-norm mode. In parallel, the apparatus 700 may also be configured to be switchable between two batch-norm encoding modes, where in the first batch-norm encoding mode the apparatus 700 is configured to perform deriving and inferring or deriving, and in the second batch-norm encoding mode the apparatus 700 is configured to perform deriving and inferring or deriving. 2 or configured to decode σ, γ, and β, and, if present, b, from the representation 110. In other words, the parameters 610 are decoded directly from the representation 110 in the second batch norm mode.

[0171] According to one embodiment, the apparatus 600 comprises the apparatus 100 (see FIG. 2) for quantizing and encoding β' and γ' into the NN representation 110. For example, the apparatus 600 first performs the calculation 620 and passes the obtained parameters β' and γ' to the apparatus 100 for parameter quantization. According to one embodiment, the apparatus 700 comprises the apparatus 300 (see FIG. 2) for deriving β and γ from the NN representation 110.

[0172] For ease of understanding, a possible relationship between X and W and pairs of layers is shown in Figure 7, with fully connected layer i+1 on the left and convolutional layer i+1 on the right. The neurons of a layer are shown as circles 10. The neurons of each layer are located at array positions (x,y). Each layer i has q i Neuron 10 in column and p i In the fully connected case, X i is the component

number

number

number

number

number

number

number

[0173] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0174] The digital data, data stream, or file of the present invention containing the NN representation of the present invention can be stored on a digital storage medium or transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0175] Depending on the specific implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementations can be implemented using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective methods are executed. The digital storage medium may therefore be computer readable.

[0176] Some embodiments according to the invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system such that one of the methods described herein is performed.

[0177] Generally, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0178] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0179] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0180] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory.

[0181] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be arranged to be transmitted via a data communication connection, for example via the Internet.

[0182] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0183] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0184] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0185] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0186] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0187] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0188] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0189] Any of the methods described herein, or any of the components of the apparatus described herein, may be implemented at least in part by hardware and / or software.

[0190] The above-described embodiments merely exemplify the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein are apparent to others skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented in the description and illustration of the embodiments herein.

Claims

1. An apparatus (100) for generating a neural network representation (110), the apparatus (100) comprising: From the quantization parameter (142), a multiplier (144) based on the remainder of the division between a dividend derived by the quantization parameter (142) and a divisor derived by a precision parameter (145); A bit shift number (146) based on the rounding of the quotient of the division; configured to quantize the NN parameters (120) to quantized values ​​(130) by determining (140) the quantization parameters (142) and quantization values ​​(152) of the NN parameters (120) such that whereby the quantized value (130) of the NN parameter (120) corresponds to the product between the quantized value (152) and a factor (148) that depends on the multiplier (144), bit-shifted by a number of bits that depends on the bit shift number (146).

2. 2. The apparatus (100) of claim 1, further configured to encode the accuracy parameter (145) into the NN representation (110).

3. 3. The apparatus (100) of claim 1 or 2, wherein the NN parameters (120) are: A weighting parameter that weights the interneuron activation feedforward (12) between a pair of neurons (10); A batch norm parameter that parameterizes the affine transformation of the neural network layers, and a bias for biasing the sum of the inbound inter-neuron activation feedforward (12) of a given neural network neuron (10); The device is one of the above.

4. An apparatus (100) according to any one of claims 1 to 3, comprising: the NN parameters (120) parameterize the NN (20) with respect to a single inter-neuron activation feedforward (12) of a plurality of inter-neuron activation feedforwards (122) of the NN (20), and for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (120) is included in the NN representation (110); The apparatus (100) for each of the plurality of inter-neuron activation feedforwards (122): determining an associated quantization parameter (142) associated with each of the inter-neuron activation feedforwards (12) and an associated quantization value (152) associated with each of the inter-neuron activation feedforwards (12); configured to quantize the corresponding NN parameters (120) to corresponding quantized values ​​(130); Thereby, from the associated quantization parameter (142), an associated multiplier (144) associated with each inter-neuron activation feedforward (12) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with each inter-neuron activation feedforward (12); Based on the rounding of the quotient of the division, an associated bit shift number (146) associated with each inter-neuron activation feedforward (12) can be derived; whereby the corresponding quantized value (130) of the corresponding NN parameter (120) of each inter-neuron activation feedforward (12) corresponds to the product between the associated quantized value (152) and a factor (148) that depends on the associated multiplier (144), the product being bit-shifted by a number of bits that depends on the associated bit shift number (146).

5. An apparatus (100) according to any one of claims 1 to 3, comprising: The apparatus (100) is configured to subdivide a plurality of inter-neuron activation feedforwards (122) of a NN into subgroups (122a, 122b) of inter-neuron activation feedforwards, whereby each subgroup (122a, 122b) is associated with an associated pair of NN layers of the NN (20) and includes inter-neuron activation feedforwards (12) between the associated pair of NN layers and excludes inter-neuron activation feedforwards (12) between any further pair of NN layers other than the associated pair of layers, and two or more subgroups (122a, 122b) are associated with a given NN layer; The NN parameters (120) parameterize the NN (20) with respect to a single inter-neuron activation feedforward (12) of the plurality of inter-neuron activation feedforwards (122) of the NN, and for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (120) is included in the NN representation (110), and the apparatus (100) For each subgroup (122a, 122b) of inter-neuron activation feedforward, determining an associated quantization parameter (142) associated with each of said subgroups (122a, 122b) from said NN representation (110), whereby From the associated quantization parameters (142), an associated multiplier (144) associated with each of the subgroups (122a, 122b) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with the respective subgroup (122a, 122b); Based on the rounding of the quotient of the division, an associated bit shift number (146) associated with each of the subgroups (122a, 122b) is derivable; and For each of the plurality of inter-neuron activation feedforwards (122), determining an associated quantization value (152) associated with each inter-neuron activation feedforward (12) from said NN representation (110); quantizing the corresponding NN parameters (120) to corresponding quantized values ​​(130) by the corresponding NN parameters (120) of each inter-neuron activation feedforward (12) correspond to a product between the associated quantization value (152) associated with the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs and a factor (148) that depends on the associated multiplier (144), the product being bit-shifted by a number of bits that depends on the associated bit-shift number (146) of the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs.

6. 6. The apparatus (100) of claim 4 or 5, wherein the associated accuracy parameter (145) is evaluated equally globally across the NN (20) or within each NN layer (114, 116).

7. The apparatus (100) according to any one of claims 4 to 6, configured to encode said associated accuracy parameter (145) into said NN representation (110).

8. The apparatus (100) according to any one of claims 4 to 7, configured to encode the associated quantization parameter (142) in the form of a difference with a reference quantization parameter into the NN representation (110).

9. An apparatus (100) according to any one of claims 1 to 8, comprising: The multiplier (144) and the bit shift number (146) are obtained from the quantization parameter (142) as follows: mul=k+QP%k [0010] where mul is the multiplier (144), shift is the bit shift number (146), QP is the quantization parameter (142), and k is the precision parameter (145), [0025] is the floor operator, which produces the largest integer less than or equal to its operand, and % is the function for x % y. [0030] whereby the NN parameters (130) are (mul / k)・2 ｓｈｉｆｔ ・P where P is the quantized value (152).

10. The apparatus (100) of any one of claims 1 to 9, wherein the precision parameter (145) is a power of two.

11. An apparatus (100) according to any one of claims 1 to 10, comprising: By using context-adaptive binary arithmetic coding, or by directly writing bits representing the quantization parameter (142) into the NN representation (110); or by encoding bits representing the quantization parameter (142) into the NN representation (110) via an equal probability bypass mode of a context-adaptive binary encoder of the device (100); The apparatus is configured to encode the quantization parameter (142) into the NN representation (110).

12. The apparatus (100) of any one of claims 1 to 11, configured to encode the quantization parameters (142) into the NN representation (110) by binarizing the quantization parameters (142) into a bin string using a binarization scheme.

13. 13. The apparatus (100) of claim 12, wherein the binarization scheme is an Exponential-Golomb Code.

14. The apparatus (100) according to any one of claims 1 to 13, configured to determine the quantization parameter (142) and to encode it into the NN representation (110) in the form of a fixed-point representation.

15. 15. The apparatus (100) of claim 14, wherein the accuracy parameter (145) is: ｔ and the bit length of the fixed-point representation is set to be constant for the NN (20) or is set to be the sum of t and a basis bit length that is constant for the NN (20).

16. The apparatus (100) of any one of claims 1 to 15, configured to encode the quantization parameter (142) into the NN representation (110) as an integer-valued syntax element.

17. 17. The apparatus (100) of claim 1, configured to encode the precision parameter (145) into the NN representation (110) by directly writing a bit representative of the precision parameter (145) into the NN representation (110) or by encoding a bit representative of the precision parameter (145) into the NN representation (110) via an equal probability bypass mode of a context-adaptive binary encoder of the apparatus (100).

18. The apparatus (100) of any one of claims 1 to 17, configured to encode the quantization value (152) and the quantization parameter (142) into the NN representation (110).

19. An apparatus (100) according to any one of claims 1 to 18, comprising: providing the NN parameters (120) by training a NN using a floating point representation of the NN parameters (120); determining the quantization parameters (142) and the quantization values ​​(152) of the NN parameters (120) by an iterative optimization method aimed at reducing the quantization error; The apparatus is configured to:

20. The apparatus (100) according to any one of claims 1 to 19, configured to determine the quantization value (152) and to encode it into the NN representation (110) in the form of a fixed-point representation.

21. 21. The apparatus (100) of any one of claims 1 to 20, configured to encode the quantized values ​​(152) into the NN representation (110) by binarizing the quantized values ​​(152) into a bin string according to a binarization scheme and encoding bits of the bin string using context-adaptive arithmetic coding.

22. 22. The apparatus (100) of any one of claims 1 to 21, configured to encode the quantized values ​​(152) into the NN representation (110) by binarizing the quantized values ​​(152) into a bin string according to a binarization scheme, encoding first bits of the bin string using context-adaptive arithmetic coding, and encoding second bits of the bin string using an equal probability bypass mode.

23. digital data defining a NN representation (110), said NN representation (110) including quantization parameters (142) and quantization values ​​(152) for representing NN parameters (130); Thereby, from the quantization parameter (142), a multiplier (144) based on the remainder of the division between a dividend derived by the quantization parameter (142) and a divisor derived by a precision parameter (145); A bit shift number (146) based on the rounding of the quotient of the division; is derivable, Thereby, the NN parameter (130) corresponds to the product between the quantized value (152) and a factor (148) that depends on the multiplier (144), bit-shifted by a number of bits that depends on the bit shift number (146), digital data.

24. 24. The digital data of claim 23, wherein the accuracy parameter (145) is encoded into the NN representation (110).

25. The NN parameters (130) are A weighting parameter that weights the interneuron activation feedforward (12) between a pair of neurons (10); A batch norm parameter that parameterizes the affine transformation of the neural network layers, and a bias for biasing the sum of the inbound inter-neuron activation feedforward (12) of a given neural network neuron (10); 25. The digital data according to claim 23 or 24, wherein the digital data is one of:

26. the NN parameters (130) parameterize the NN with respect to a single inter-neuron activation feedforward (12) of a plurality of inter-neuron activation feedforwards (122) of the NN, and for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (130) is included in the NN representation (110); For each of the plurality of inter-neuron activation feedforwards (122), an associated quantization parameter (142) associated with each inter-neuron activation feedforward (12) is included in the NN representation (110); An associated quantized value (152) associated with each inter-neuron activation feedforward (12) is included in the NN representation (110), whereby From the associated quantization parameters (142), an associated multiplier (144) associated with each inter-neuron activation feedforward (12) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with each inter-neuron activation feedforward (12); Based on the rounding of the quotient of the division, an associated bit shift number (146) associated with each inter-neuron activation feedforward (12) can be derived; 26. The digital data of claim 23, whereby the corresponding quantized value (130) of the corresponding NN parameter (130) of the respective inter-neuron activation feedforward (12) corresponds to a product between the associated quantized value (152) and a factor (148) depending on the associated multiplier (144), bit-shifted by a number of bits depending on the associated bit-shift number (146).

27. A plurality of inter-neuron activation feedforwards (122) of the NN are subdivided into subgroups (122a, 122b) of inter-neuron activation feedforwards, whereby each subgroup (122a, 122b) is associated with an associated pair of NN layers of the NN and includes inter-neuron activation feedforwards between the associated pair of NN layers and excludes inter-neuron activation feedforwards between any further pair of NN layers other than the associated pair of layers, and two or more subgroups (122a, 122b) are associated with a given NN layer; the NN parameters (130) parameterize the NN with respect to a single inter-neuron activation feedforward (12) of the plurality of inter-neuron activation feedforwards (122) of the NN, and for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (130) is included in the NN representation (110); For each subgroup (122a, 122b) of inter-neuron activation feedforward, An associated quantization parameter (142) associated with each of the subgroups (122a, 122b) is included in the NN representation (110), whereby From the associated quantization parameters (142), an associated multiplier (144) associated with each of the subgroups (122a, 122b) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with the respective subgroup (122a, 122b); an associated bit shift number (146) associated with each of the subgroups (122a, 122b) based on a rounding of the quotient of the division; and For each of the plurality of inter-neuron activation feedforwards (122), an associated quantization value (152) associated with each inter-neuron activation feedforward is included in the NN representation (110); 26. The digital data of claim 23, wherein the corresponding NN parameters (130) of the respective inter-neuron activation feedforward (12) correspond to a product between the associated quantization value (152) associated with the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs and a factor (148) that depends on the associated multiplier (144), the product being bit-shifted by a number of bits that depends on the associated bit-shift number (146) of the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs.

28. 28. Digital data according to claim 26 or 27, wherein the associated accuracy parameter (145) is equally evaluated globally across the NN or within each NN layer.

29. Digital data according to any one of claims 26 to 28, wherein said associated accuracy parameter (145) is encoded into said NN representation (110).

30. Digital data according to any one of claims 26 to 29, wherein the associated quantization parameter (142) is coded in the NN representation (110) in the form of a difference with a reference quantization parameter.

31. The multiplier (144) and the bit shift number (146) are obtained from the quantization parameter (142) as follows: mul=k+QP%k [0045] where mul is the multiplier (144), shift is the bit shift number (146), QP is the quantization parameter (142), and k is the precision parameter (145), [0050] is the floor operator, which produces the largest integer less than or equal to its operand, and % is the function for x % y. [006] whereby the NN parameters (130) are (mul / k)・2 ｓｈｉｆｔ ・P 31. Digital data according to any one of claims 23 to 30, wherein P is the quantized value (152).

32. The digital data of any one of claims 23 to 31, wherein the precision parameter (145) is a power of two.

33. The quantization parameter (142) is By using context-adaptive binary arithmetic coding, or by directly writing bits representing the quantization parameter (142) into the NN representation (110); or by encoding bits representing said quantization parameter (142) into said NN representation (110) via an equal probability bypass mode of a context-adaptive binary encoder; Digital data according to any one of claims 23 to 32, encoded into said NN representation (110).

34. Digital data according to any one of claims 23 to 33, wherein the quantization parameter (142) is encoded into the NN representation (110) in the form of binarization into a bin string using a binarization scheme.

35. 35. The digital data of claim 34, wherein the binarization scheme is an Exponential-Golomb code.

36. Digital data according to any one of claims 23 to 35, wherein the quantization parameter (142) is encoded into the NN representation (110) in the form of a fixed point representation.

37. The accuracy parameter (145) is 2 ｔ 37. The digital data of claim 36, wherein the bit length of the fixed-point representation is set to be constant for the NN or is set to be the sum of a base bit length that is constant for the NN and t.

38. Digital data according to any one of claims 23 to 37, wherein the quantization parameter (142) is encoded in the NN representation (110) as an integer-valued syntax element.

39. 39. The digital data of claim 23, wherein the precision parameter (145) is encoded into the NN representation (110) by directly writing bits representative of the precision parameter (145) into the NN representation (110) or by encoding bits representative of the precision parameter (145) into the NN representation (110) via an equal probability bypass mode of a context-adaptive binary encoder.

40. Digital data according to any one of claims 23 to 39, wherein the quantised values ​​(152) are encoded into the NN representation (110) in the form of a fixed point representation.

41. 41. Digital data according to any one of claims 23 to 40, wherein the quantised values ​​(152) are encoded into the NN representation (110) in the form of the bin string, in which the quantised values ​​(152) are binarised according to a binarisation scheme, and the bits are encoded into the bitstream using context-adaptive arithmetic coding.

42. 42. The digital data of claim 23, wherein the quantized values ​​(152) are coded into the NN representation (110) in the form of the bin string in which the quantized values ​​(152) are binarized according to a binarization scheme, a first bit of which is coded into the bitstream using context-adaptive arithmetic coding and a second bit of which is coded into the bitstream using equal probability bypass mode.

43. An apparatus for deriving neural network parameters from a neural network representation (110), comprising: deriving a quantization parameter (142) from said NN representation (110); deriving a quantized value (152) from said NN representation (110); From the quantization parameter (142), a multiplier (144) based on the remainder of the division between a dividend derived by the quantization parameter (142) and a divisor derived by a precision parameter (145); A bit shift number (146) based on the rounding of the quotient of the division; and [0023] The NN parameter (130) corresponds to a product between the quantized value (152) and a factor (148) that depends on the multiplier (144), the product being bit-shifted by a number of bits that depends on the bit shift number (146).

44. The apparatus of claim 43, further configured to derive the accuracy parameter (145) from the NN representation (110).

45. The NN parameters (130) are A weighting parameter that weights the interneuron activation feedforward (12) between a pair of neurons (10); A batch norm parameter that parameterizes the affine transformation of the neural network layers, and a bias for biasing the sum of the inbound inter-neuron activation feedforward (12) of a given neural network neuron (10); 45. The device according to claim 43 or 44, wherein

46. The NN parameters (130) parameterize the NN with respect to a single inter-neuron activation feedforward (12) of a plurality of inter-neuron activation feedforwards (122) of the NN, and the apparatus derives, for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (130) from the NN representation (110): For each of the plurality of inter-neuron activation feedforwards (122), deriving an associated quantization parameter (142) associated with each inter-neuron activation feedforward (12) from the NN representation (110); deriving an associated quantization value (152) associated with each inter-neuron activation feedforward (12) from the NN representation (110); From the associated quantization parameters (142), an associated multiplier (144) associated with each inter-neuron activation feedforward (12) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with each inter-neuron activation feedforward (12); deriving an associated bit shift number (146) associated with each of the inter-neuron activation feedforwards (12) based on a rounding of the quotient of the division; and configured to derive by 46. ​​The apparatus of claim 43, wherein the corresponding NN parameters (130) of each inter-neuron activation feedforward (12) correspond to a product between the associated quantization value (152) and a factor (148) that depends on the associated multiplier (144), bit-shifted by a number of bits that depends on the associated bit-shift number (146).

47. The apparatus is configured to subdivide a plurality of inter-neuron activation feedforwards (122) of a NN into subgroups (122a, 122b) of inter-neuron activation feedforwards, whereby each subgroup (122a, 122b) is associated with an associated pair of NN layers of the NN and includes inter-neuron activation feedforwards between the associated pair of NN layers and excludes inter-neuron activation feedforwards between any further pair of NN layers other than the associated pair of layers, and two or more subgroups (122a, 122b) are associated with a given NN layer; The NN parameters (130) parameterize the NN with respect to a single inter-neuron activation feedforward (12) of the plurality of inter-neuron activation feedforwards (122) of the NN, and the apparatus derives, for each of the plurality of inter-neuron activation feedforwards (122), a corresponding NN parameter (130) from the NN representation (110): For each subgroup (122a, 122b) of inter-neuron activation feedforward, deriving associated quantization parameters (142) associated with each of the subgroups (122a, 122b) from the NN representation (110); From the associated quantization parameters (142), an associated multiplier (144) associated with each of the subgroups (122a, 122b) based on a remainder of a division between a dividend derived by the associated quantization parameter (142) and a divisor derived by an associated precision parameter (145) associated with the respective subgroup (122a, 122b); deriving an associated bit shift number (146) associated with each of the subgroups (122a, 122b) based on a rounding of the quotient of the division; For each of the plurality of inter-neuron activation feedforwards (122), deriving an associated quantization value (152) associated with each inter-neuron activation feedforward (12) from the NN representation (110); and configured to derive by 46. ​​The apparatus according to claim 43, wherein the corresponding NN parameters (130) of the respective inter-neuron activation feedforward (12) correspond to a product between the associated quantization value (152) associated with the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs and a factor (148) that depends on the associated multiplier (144), the product being bit-shifted by a number of bits that depends on the associated bit-shift number (146) of the subgroup (122a, 122b) to which the respective inter-neuron activation feedforward (12) belongs.

48. 48. Apparatus according to claim 46 or 47, wherein the associated accuracy parameter (145) is equally evaluated globally across the NN or within each NN layer.

49. An apparatus according to any one of claims 46 to 48, configured to derive the associated accuracy parameter (145) from the NN representation (110).

50. The device according to any one of claims 46 to 49, configured to derive the associated quantization parameter (142) from the NN representation (110) in the form of a difference with a reference quantization parameter.

51. From the quantization parameter (142), the multiplier (144) and the bit shift number (146) are calculated as follows: mul=k+QP%k [0070] where mul is the multiplier (144), shift is the bit shift number (146), QP is the quantization parameter (142), and k is the precision parameter (145), [0080] is the floor operator, which produces the largest integer less than or equal to its operand, and % is the function for x % y. [0090] whereby the NN parameters (130) are (mul / k)・2 ｓｈｉｆｔ ・P Apparatus according to any one of claims 43 to 50, wherein P is the quantized value (152).

52. Apparatus according to any one of claims 43 to 51, wherein the precision parameter (145) is a power of two.

53. By using context-adaptive binary arithmetic decoding, or by directly reading the bits representing the quantization parameter (142) from the NN representation (110); or deriving bits representing the quantization parameter (142) from the NN representation (110) via an equal probability bypass mode of a context-adaptive binary decoder of the device; An apparatus according to any one of claims 43 to 52, configured to derive the quantisation parameter (142) from the NN representation (110).

54. The apparatus of any one of claims 43 to 53, configured to derive the quantization parameter (142) from the NN representation (110) by de-binarizing a bin string using a binarization scheme.

55. 55. The apparatus of claim 54, wherein the binarization scheme is an Exponential-Golomb code.

56. An apparatus according to any one of claims 43 to 55, configured to derive the quantisation parameter (142) from the NN representation (110) in the form of a fixed-point representation.

57. The accuracy parameter (145) is 2 ｔ 57. The apparatus of claim 56, wherein a bit length of the fixed-point representation is set to be constant for the NN or is set to be a sum of t and a base bit length that is constant for the NN.

58. An apparatus according to any one of claims 43 to 57, configured to derive the quantisation parameter (142) from the NN representation (110) as an integer-valued syntax element.

59. 59. An apparatus according to any one of claims 43 to 58, configured to derive the precision parameter (145) from the NN representation (110) by reading bits representative of the precision parameter (145) directly from the NN representation (110) or by deriving bits representative of the precision parameter (145) from the NN representation (110) via an equal probability bypass mode of a context-adaptive binary decoder of the apparatus.

60. An apparatus according to any one of claims 43 to 59, configured to derive the quantised values ​​(152) from the NN representation (110) in the form of a fixed point representation.

61. 61. The apparatus of any one of claims 43 to 60, configured to derive the quantized value (152) from the NN representation (110) by de-binarizing the quantized value (152) from a bin string according to a binarization scheme and decoding bits of the bin string from the NN representation (110) using context-adaptive arithmetic decoding.

62. 62. The apparatus of claim 43, configured to derive the quantized value (152) from the NN representation (110) by de-binarizing the quantized value (152) from a bin string according to a binarization scheme, decoding a first bit of the bin string from the NN representation (110) using context adaptive arithmetic decoding, and decoding a second bit of the bin string using an equal probability bypass mode.

63. A device (400) for performing inference using a neural network (20), the device (400) comprising: a NN parameterizer (410) configured to parameterize the NN (20), the NN parameterizer (410) comprising an apparatus (300) for deriving NN parameters (130) from a NN representation (110) according to any one of claims 43 to 62; a computation unit (420) configured to compute an inference output (430) based on NN inputs (440) using the NN (20); A device comprising:

64. 64. The device (400) of claim 63, The NN parameterizer (410) deriving, via the apparatus (300), at least one of a first NN parameter (130) and a second NN parameter (130), whereby the first NN parameter (130) corresponds to a product of a first quantized value (152) and a first factor (148), bit-shifted by a first number of bits (146), and the second NN parameter (130) corresponds to a product of a second quantized value (152) and a second factor (148), bit-shifted by a second number of bits (146); forming a sum between a first addend formed by a first quantized value (152) of the first NN parameter (130) and weighted by the first multiplier (144) and a second addend formed by a second quantized value (152) of the first NN parameter (130), weighted by the second multiplier (144) and bit-shifted by the difference between the first number of bits (146) and the second number of bits (146); and bit-shifting the sum of the first addend and the second addend by a number of bits that depends on one of the first number of bits (146) and the second number of bits (146); A device that is configured to:

65. 65. The device (400) of claim 64, the first NN parameters (130) represent a base layer representation of the NN (20) and the second NN parameters (130) represent an enhancement layer representation of the NN; or the first NN parameters (130) represent a current representation of the NN (20) and the second NN parameters (130) represent an update to the current NN representation (110); or The device, wherein the first NN parameter (130) represents a bias that biases the sum of inbound inter-neuron activation feedforward (12) of a given neural network neuron (10), and the second NN parameter (130) represents a batch norm parameter that parameterizes an affine transformation of a neural network layer.

66. A device (400) according to any one of claims 63 to 65, comprising: The NN parameterizer (410) configured to derive, via the apparatus (300), at least one of a third NN parameter (130) and a fourth NN parameter (130), whereby the third NN parameter (130) corresponds to a product of a third quantized value (152) and a third factor (148), bit-shifted by a third number of bits (146), and the fourth NN parameter (130) corresponds to a product of a fourth quantized value (152) and a fourth factor (148), bit-shifted by a fourth number of bits (146); The calculation unit (420), in performing the calculation, 16. A device configured to generate a product by multiplying the third NN parameter (130) and the fourth NN parameter (130) to form a product of a first factor formed by a third quantized value (152) of the third NN parameter (130), a second factor formed by the third multiplier (144), a third factor formed by a fourth quantized value (152) of the fourth NN parameter (130), and a fourth factor formed by the fourth multiplier (144), the product being bit-shifted by a number of bits corresponding to a sum including a first addend formed by the third number of bits and a second addend formed by the fourth number of bits.

67. 67. The device (400) of claim 66, The third NN parameter (130) is a parameter of the first neuron (10) of the first NN layer (114). １ ) to the second neuron (10 ２ ), and the fourth NN parameter (130) represents a batch norm parameter.

68. 68. The device (400) of claim 67, wherein the batch norm parameter is a function of the first neuron (10) for the second NN layer. １ ) activating feed-forward amplification of the device.

69. A device (400) according to any one of claims 63 to 68, comprising: the device is further configured to quantize the NN input (440) by quantizing the activations to quantized values ​​(130) by determining a fifth quantization parameter (142) and a fifth quantization value (152) for the activations, and to derive from the fifth quantization parameter (142) a fifth multiplier (144) based on a remainder of a division between a dividend derived by the fifth quantization parameter (142) and a divisor derived by a precision parameter (145) associated with the activations, and a fifth bit shift number (146) based on rounding a quotient of the division, thereby obtaining the quantized value (130) corresponding to a product between the fifth quantization value (152) and a factor (148) depending on the fifth multiplier (144), the product being bit shifted by a fifth number of bits depending on the fifth bit shift number (146).

70. 70. The device (400) of claim 69, The NN parameterizer (410) deriving, via said device (300), a sixth NN parameter (130), whereby said sixth NN parameter (130) corresponds to a product of a sixth quantization value (152) and a sixth factor (148), bit-shifted by a sixth number of bits (146); generating a product by multiplying the sixth NN parameter (130) and the activations to form a product of a first factor formed by a sixth quantized value (152) of the sixth NN parameter (130), a second factor formed by the sixth multiplier (144), a third factor formed by the fifth quantized value (152), and a fourth factor formed by the fifth multiplier (144), the product being bit-shifted by a number of bits corresponding to a sum including a first addend formed by the sixth number of bits and a second addend formed by the fourth number of bits (146); A device that is configured to:

71. A device (500) for performing inference using a NN (20), the device (500) being configured to compute an inference output (430) based on an NN input (440) using the NN (20), the NN (20) including a pair of NN layers and an inter-neuron activation feedforward (12) from a first NN layer of the pair of NN layers to a second NN layer of the pair of NN layers, the device (500) comprising: forming (530) a matrix X (532) from the activations of the neural network neurons (10) of the first NN layer; Calculating (540) s·W′*X, where * denotes matrix multiplication, W′ is a weight matrix (544) of dimension n×m, with n and m∈N, s is a transposed vector (546) of length n, and · denotes column-wise Hadamard multiplication between a matrix on one side of · and a transposed vector on the other side of ·; 13. A device configured to calculate activations of the neural network neurons (10) of the second NN layer based on the activations of the neural network neurons (10) of the first NN layer by:

72. 72. The device (500) of claim 71, configured to compute (540) the matrix multiplication using n-bit fixed-point arithmetic to generate an inner product and to multiply the inner product with s (546) using m-bit fixed-point arithmetic, where m>n.

73. 73. The device (500) of claim 71 or 72, wherein s (546) is the result of optimization of W' (544) for higher compression for encoding W' (544) and / or higher inference fidelity.

74. A device (500) according to any one of claims 71 to 73, comprising a NN parameterizer (410) configured to derive W' (544) from a NN representation (110), said NN parameterizer (410) comprising an apparatus (300) for deriving NN parameters (130) from a NN representation (110) according to any one of claims 43 to 62.

75. 75. The device (500) of any one of claims 71 to 74, wherein the NN parameterizer (410) is further configured to derive s (546) from the NN representation (110) using a different quantization parameter (142) compared to the NN parameters (130) associated with W' (544).

76. An apparatus (600) for encoding NN parameters of a batch norm operator (710) of a NN (20) into a NN representation (110), the batch norm operator (710) comprising: [0089] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (600) comprises: b, μ, σ ２ or σ, γ, and β (610); ##EQU00011## Calculating (620) Encoding β' and γ' into the NN representation (110) as NN parameters of the batch norm operator (710), ##EQU00012## Define it as 3)in ２ :=θ 4) μ′:=0 5) b':=0, where θ is a predetermined parameter; and The apparatus is configured to:

77. 77. The apparatus (600) of claim 76, wherein the predetermined parameter is 1 or 1-ε.

78. An apparatus (600) for encoding NN parameters of a batch norm operator (710) of a NN into a NN representation (110), the batch norm operator (710) comprising: ##EQU00013## is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is the weight matrix X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (600) comprises: b, μ, σ ２ or σ, γ, and β (610); ##EQU14## Calculating (620) Encoding β' and γ' into the NN representation (110) as NN parameters of the batch norm operator (710), ##EQU00015## Define it as 3)in ２ :=1 4) μ′:=0 5) b':=0; and The apparatus is configured to:

79. An apparatus (600) for encoding NN parameters of a batch norm operator (710) of a NN into a NN representation (110), the batch norm operator (710) comprising: ##EQU00016## is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (600) comprises: μ, σ ２ or σ, γ, and β (610); ##EQU00017## Calculating (620) Encoding β' and γ' into the NN representation (110) as NN parameters of the batch norm operator (710), [0018] Define it as 3)in ２ :=θ 4) μ′:=0, where θ is a predetermined parameter; and The apparatus is configured to:

80. 80. The apparatus (600) of claim 79, wherein the predetermined parameter is 1 or 1-ε.

81. An apparatus (600) for encoding NN parameters of a batch norm operator (710) of a NN into a NN representation (110), the batch norm operator (710) comprising: [0019] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (600) comprises: μ, σ ２ or σ, γ, and β (610); [0020] Calculating (620) Encoding β' and γ' into the NN representation (110) as NN parameters of the batch norm operator (710), ##EQU00021## Define it as 3)in ２ :=1 4) μ′:=0; and The apparatus is configured to:

82. Apparatus (600) according to any one of claims 76 to 81, comprising: In the above expression, σ′ ２ and / or In said expression, all components of μ′ are equal to each other and their values; and / or The apparatus is further configured to indicate that in said representation, if present, all components of b' are equal to one another and their values.

83. The apparatus (600) according to any one of claims 76 to 82, further configured to be switchable between two batch-norm coding modes, in a first batch-norm coding mode the apparatus (600) is configured to perform the calculation and the encoding of β' and γ', and in a second batch-norm coding mode the apparatus (600) is configured to perform the calculation and the encoding of the received μ, σ ２ or σ, γ, and β, and, if present, b.

84. The apparatus (600) of any one of claims 76 to 83, further comprising an apparatus (100) of any one of claims 1 to 22, for quantizing and encoding β' and γ' into said NN representation (110).

85. An apparatus (700) for decoding NN parameters of a batch norm operator (710) of a NN from a NN representation (110), the batch norm operator (710) comprising: [0022] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (700) comprises: Deriving (720) γ (722) and β (724) from the NN representation (110); By one signaling (734) applied to all its components, 1) in ２ :=θ 2) μ′:=0 3) inferring or deriving (730) that b':=0, where θ is a predetermined parameter; The apparatus is configured to:

86. 86. The apparatus (700) of claim 85, wherein the predetermined parameter is 1 or 1-ε.

87. An apparatus (700) for decoding NN parameters of a batch norm operator (710) of a NN from a NN representation (110), the batch norm operator (710) comprising: [0023] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (700) comprises: Deriving (720) γ (722) and β (724) from the NN representation (110); By one signaling (734) applied to all its components, 1) s ２ :=1 2) μ:=0 3) inferring or deriving (730) that b:=0; The apparatus is configured to:

88. An apparatus (700) for decoding NN parameters of a batch norm operator (710) of a NN from a NN representation (110), the batch norm operator (710) comprising: ##EQU00024## is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (700) comprises: Deriving (720) γ (722) and β (724) from the NN representation (110); By one signaling (734) applied to all its components, 1) s ２ :=θ 2) Inferring or deriving (730) that μ:=0, where θ is a predetermined parameter; The apparatus is configured to:

89. 89. The apparatus (700) of claim 88, wherein the predetermined parameter is 1 or 1-ε.

90. An apparatus (700) for decoding NN parameters of a batch norm operator (710) of a NN from a NN representation (110), the batch norm operator (710) comprising: [0025] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The device (700) comprises: Deriving (720) γ (722) and β (724) from the NN representation (110); By one signaling (734) applied to all its components, 1) s ２ :=1 2) inferring or deriving (730) that μ:=0; The apparatus is configured to:

91. Apparatus (700) according to any one of claims 85 to 90, comprising: From the above expression, σ' ２ and derive the value of all components of From said expression, derive that all components of μ′ are equal to one another and their values; and / or If present, the apparatus is further configured to derive from said representation that all components of b' are equal to one another and their values.

92. The apparatus (700) of any one of claims 85 to 91, further configured to be switchable between two batch-norm coding modes, in a first batch-norm coding mode the apparatus (700) is configured to perform the deriving and the inference or deriving, and in a second batch-norm coding mode the apparatus (700) is configured to perform the inference or deriving and the inference or deriving. ２ or σ, γ, and β, and, if present, b, from said representation.

93. The apparatus (700) of any one of claims 85 to 92, further comprising an apparatus (700) of any one of claims 43 to 62 for deriving β and γ from the NN representation (110).

94. 1. A method for generating a neural network representation, comprising the steps of: From the quantization parameters, a multiplier based on a remainder of a division between a dividend derived by the quantization parameter and a divisor derived by a precision parameter; A bit shift number based on the rounding of the quotient of the division; quantizing the NN parameters to quantized values ​​by determining the quantization parameters and quantization values ​​of the NN parameters such that whereby the quantized value of the NN parameter corresponds to a product of the quantized value and a factor that depends on the multiplier, bit-shifted by a number of bits that depends on the bit shift number.

95. 1. A method for deriving neural network parameters from a neural network representation, comprising the steps of: Deriving a quantization parameter from the NN representation; and deriving a quantized value from the NN representation; From the quantization parameters, a multiplier based on a remainder of a division between a dividend derived by the quantization parameter and a divisor derived by a precision parameter; A bit shift number based on the rounding of the quotient of the division; and Including, A method according to claim 1, wherein the NN parameter corresponds to a product of the quantized value and a factor that depends on the multiplier, the product being bit-shifted by a number of bits that depends on the bit-shift number.

96. 1. A method of performing inference using a neural network, comprising: Parameterizing the NN using the method of claim 95 to derive NN parameters from the NN representation; using the NN to compute inference outputs based on the NN inputs; A method comprising:

97. 1. A method of performing inference using a NN, comprising: using the NN to compute an inference output based on a NN input, the NN comprising a pair of NN layers and inter-neuron activation feedforward from a first NN layer of the pair of NN layers to a second NN layer of the pair of NN layers, the method comprising: forming a matrix X from the activations of the neural network neurons of the first NN layer; Calculating s·W′*X, where * denotes matrix multiplication, W′ is a weight matrix of dimension n×m with n and m∈N, s is a transpose vector of length n, and · denotes column-wise Hadamard multiplication between a matrix on one side of · and the transpose vector on the other side of ·; and calculating activations of the neural network neurons of the second NN layer based on activations of the neural network neurons of the first NN layer by:

98. 1. A method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator comprising: [0026] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: b, μ, σ ２ or receiving σ, γ, and β; [0027] Calculating Encoding β' and γ' into the NN representation as NN parameters of the batch norm operator, [0028] Define it as 3)in ２ :=θ 4) μ′:=0 5) b':=0, where θ is a predetermined parameter; and A method comprising:

99. 1. A method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator comprising: [0029] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: b, μ, σ ２ or receiving σ, γ, and β; [0030] Calculating Encoding β' and γ' into the NN representation as NN parameters of the batch norm operator, [0031] Define it as 3)in ２ :=1 4) μ′:=0 5) b':=0; and A method comprising:

100. 1. A method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator comprising: [0032] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: μ, σ ２ or receiving σ, γ, and β; [Equation 33] Calculating Encoding β' and γ' into the NN representation as NN parameters of the batch norm operator, [0034] Define it as 3)in ２ :=θ 4) μ′:=0, where θ is a predetermined parameter; and A method comprising:

101. 1. A method for encoding NN parameters of a batch norm operator of a NN into a NN representation, the batch norm operator comprising: [Equation 35] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: μ, σ ２ or receiving σ, γ, and β; [0036] Calculating s; and Encoding β' and γ' into the NN representation as NN parameters of the batch norm operator, [Equation 37] Define it as 3)in ２ :=1 4) μ′:=0; and A method comprising:

102. 1. A method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator comprising: [Equation 38] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: Deriving γ and β from the NN representation; With one signaling that applies to all its components, 1) in ２ :=θ 2) μ′:=0 3) inferring or deriving that b':=0, where θ is a predetermined parameter; A method comprising:

103. 1. A method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator comprising: [0039] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, b is the transposed vector forming the bias, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: Deriving γ and β from the NN representation; With one signaling that applies to all its components, 1) s ２ :=1 2) μ:=0 3) inferring or deriving that b:=0; A method comprising:

104. 1. A method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator comprising: [0040] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, ε is a constant to avoid division by zero, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: Deriving γ and β from the NN representation; With one signaling that applies to all its components, 1) s ２ :=θ 2) inferring or deriving that μ:=0, where θ is a predetermined parameter; A method comprising:

105. 1. A method for decoding NN parameters of a batch norm operator of a NN from a NN representation, the batch norm operator comprising: [0041] is defined as: μ, σ ２ , γ, and β are batch norm parameters, W is a weighting matrix, X is the input matrix derived from the activations of the NN layer, denotes column-wise Hadamard multiplication between a matrix on one side of and a transposed vector on the other side of, * denotes matrix multiplication, The method comprises: Deriving γ and β from the NN representation; With one signaling that applies to all its components, 1) s ２ :=1 2) inferring or deriving that μ:=0; A method comprising:

106. A digital storage medium containing digital data according to any one of claims 23 to 42.

107. A computer program for carrying out the method according to any one of claims 94 to 105.

108. A data stream generated by an apparatus according to any one of claims 1 to 22 and 76 to 84.

Images