Decoder, encoder, method and computer program for providing decoding parameters of neural network using reordering

Abstract

Embodiments according to the present invention relate to a decoder for providing decoding parameters of a neural network based on an encoded representation wherein the decoder is configured to obtain a first multidimensional array comprising a plurality of neural network parameter values using decoding of the neural network parameters, and wherein the decoder is configured to obtain a reordered multidimensional array using the reordering, and wherein the first dimension of the first multidimensional array is reordered to another dimension of the reordered multidimensional array. Further, an encoder, a method and a computer program using reordering are disclosed.

Description

Technical Field

[0001] Embodiments according to the invention relate to a decoder, an encoder, a method and a computer program for providing decoding parameters of a neural network using reordering.

[0002] Embodiments according to the invention involve reordering tensor dimensions by tensor dimension shifting.

[0003] Embodiments according to the present invention relate to a method for reordering tensor dimensions for encoding of a neural network.

[0004] Embodiments according to the invention relate to compression of neural networks for multimedia content description and analysis. Background Art

[0005] Many frameworks provide a variety of compression tools, including quantization and / or lossless encoding methods and lossless decoding methods, for processing and / or compressing tensors.

[0006] For example, based on the successful use of artificial neural networks in multimedia analysis and processing, media coding, data analysis and many other applications, the need has increased for concepts that allow the exchange of such networks, for example in the form of respective neural network parameters. For example, these neural network parameters can be processed and / or encoded and decoded in the form of tensors.

[0007] Therefore, the processing, encoding, and decoding of such tensors may be the key aspects that provide efficiency to such frameworks.

[0008] Therefore, it is desirable to have an improved concept for encoding, decoding and processing parameters provided in the form of multidimensional arrays or tensors or structured representations, such as neural network parameters, that achieves a better trade-off between coding efficiency, flexibility and complexity.

[0009] The present application achieves this object by the subject matter of the independent claims.

[0010] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application. Summary of the invention

[0011] An embodiment according to the invention comprises a decoder for providing decoded parameters, such as "weights" or "coefficients", of a neural network based on an encoded representation, such as based on a bitstream representing (or e.g. comprising) the neural network parameters in an encoded (e.g. compressed) form.

[0012] The decoder is configured to obtain a first multidimensional array (e.g., inputTensor[idxA], where idxA is a vector of multiple index variables; e.g., A[m][n][o][p], where m, n, o, p are index variables), the first multidimensional array comprising multiple neural network parameter values ​​(e.g., neural network weights), using decoding of the neural network parameters, for example, so that the first multidimensional array comprises decoded neural network parameters.

[0013] In addition, the decoder is configured to use the reordering order to obtain a reordered multidimensional array (e.g., reorderedTensor[idxB], where idxB is a vector of multiple index variables; e.g., B[n][o][m][p], where n, o, m, p are index variables), wherein a first dimension of the first multidimensional array (e.g., the dimension of the multidimensional array specified by the leftmost array index) is reordered, e.g., moved to a different dimension, e.g., different from the first dimension in the reordered multidimensional array.

[0014] In many frameworks, a structured representation of parameters, such as a tensor, can be interpreted as a one-dimensional or two-dimensional representation, such as in the form of a one-dimensional array or a two-dimensional array. Therefore, the shape of this interpreted representation can be determined by the original dimensions of the structured representation, in particular, the respective order of the parameters in the original structured representation.

[0015] For example, according to some frameworks, in such an interpretation, the length of a first dimension of the interpreted array may be equal to the length of the first dimension of the original representation, and the length of a second dimension of the interpreted array may be equal to the product of all other dimensions of the original representation.

[0016] The inventors recognize that such shape and / or order of parameters may also affect the efficiency of compression and / or processing pipelines. In addition, conventional frameworks lack flexibility and are unable to adjust corresponding structured parameter representations (eg, tensors) between processing steps to improve efficiency.

[0017] Thus, the inventors recognized that a reordering of dimensions based on a structured representation of parameters can provide a better tradeoff between coding efficiency, flexibility, and complexity for encoding, decoding, and / or processing parameters (e.g., neural network parameters) that are provided in a structured representation (e.g., a multidimensional array or tensor).

[0018] Furthermore, for example, for encoding and / or decoding of these parameters, the reordering can exploit the correlation between the parameters, thereby improving the coding efficiency. In particular, the reordering can be adjusted to improve the correlation characteristics, thereby using context adaptive coding (such as CABAC).

[0019] Furthermore, the inventors have recognized that the reordering method of the present invention allows decoupling the requirements or constraints of the tensor shape used for encoding from the requirements or constraints of the tensor shape used for processing. Therefore, even if the goals of the tensor shapes are contradictory, they can be achieved through reordering.

[0020] Furthermore, the inventors have recognized that the reordering based on the present invention can improve the efficiency of existing coding frameworks with limited impact on complexity. For example, processing methods that have been developed require predefined parameter structures, such as tensor forms, but in order to improve processing efficiency, these methods can remain unchanged while allowing for reshaping of parameter representations. For example, the reordering of the present invention can allow for reshaping of the corresponding tensors, thereby achieving efficient block scanning.

[0021] According to an embodiment of the present invention, the decoder is configured to decode a dimension shift value (e.g., a "first_tensor_dimension_shift" value, e.g., a single scalar value). The dimension shift value describes how much the first dimension of a first multidimensional array (e.g., having dimensions [D2, D0, D1, D3]), e.g., D2 (e.g., also designated as the "first tensor dimension") should be shifted, e.g., how many array indices should be shifted, e.g., toward a larger dimension number, when performing reordering, e.g., in order to obtain a reordered multidimensional array having dimensions [D0, D1, D2, D3]. The inventors recognized that, based on the dimension shift value, the shifting of tensor dimensions can be controlled with lower signaling overhead. In addition, signaling the shifting information of the dimensions can make the reordering more efficient.

[0022] According to an embodiment of the present invention, the decoder is configured to decode a dimension shift value, such as a “first_tensor_dimension_shift” value, and the dimension shift value (e.g., a scalar value) describes a new position of a first dimension, such as D2 (e.g., also designated as a “first tensor dimension”) of a first multidimensional array (e.g., having dimensions [D2, D0, D1, D3]) in the reordered multidimensional array, such as “a third dimension in the multidimensional array having dimensions [D0, D1, D2, D3]”. Therefore, absolute dimension reordering information can be provided instead of relative displacement information. This allows for an accurate reordering of the corresponding tensor dimensions.

[0023] According to an embodiment of the present invention, the dimension shift value is Exp Golomb encoded. Optionally, the decoder is configured to perform Exp Golomb decoding on the dimension shift value. The inventors have recognized that Exp Golomb encoding can provide an efficient representation of the dimension shift value.

[0024] According to an embodiment of the present invention, the decoder is configured to perform reordering using a single scalar dimension shift value (e.g., first_tensor_dimension shift) as the only parameter describing the new order of dimensions in the reordered multidimensional array. The inventors recognized that the reordering can be controlled by a single scalar dimension shift value, so the additional data (e.g., in the form of signaling overhead) is very limited, thereby allowing to maximize the coding and / or processing efficiency gains based on the reordering.

[0025] According to an embodiment of the present invention, the decoder is configured to, for example, perform only a single shift of a single dimension to another position to obtain a reordered multidimensional array. The inventors recognize that allowing arbitrary dimensional order may result in huge signaling overhead, especially for high-dimensional tensors, because the number of permutations may increase greatly with the number of tensor dimensions. In contrast, according to one aspect of the present invention, the decoder may perform only a single shift of a single dimension to another position, thereby limiting the signaling overhead.

[0026] According to an embodiment of the present invention, the decoder is configured to derive the dimension of the auxiliary array from the first dimension of the first multidimensional array. As an example, the decoder may optionally be configured to derive the dimension of a vector or a two-dimensional array or an array with more than two dimensions from the first dimension of the first multidimensional array; for example, the dimension of a tensor or a "related tensor" or a "corresponding tensor". For example, the decoder may optionally be configured to derive the dimension of an "auxiliary" array of bias values, or the dimension of an "auxiliary" array of batch normalization parameters, or the dimension of an "auxiliary" array of scaling factors from the first dimension of the first multidimensional array. Optionally, for example, the decoder is configured to set the dimension of the auxiliary array to be equal to the first dimension of the first multidimensional array.

[0027] Some frameworks (such as MPEG-7 Part 17) specify special NDU types (such as NNR_PT_BLOCK) that allow the weight tensor and several corresponding tensors, namely biases, batch normalization parameters, and scaling factors, to be transmitted together in a single NDU. All tensors in a block can share the same header information (e.g., instead of transmitting separate header information), so the size of the bitstream can be reduced. Since other tensors may be directly related to the weight tensor, the inventors realized that their dimensions can be derived from the (weight) tensor dimensions.

[0028] Therefore, generally for multidimensional arrays (such as weight tensors), the decoder of the present invention can be configured to derive one or more auxiliary arrays (such as including biases, batch normalization parameters and scaling factors). Therefore, the dimensionality determination explained above can be performed, thereby limiting the signaling overhead.

[0029] According to an embodiment of the present invention, the decoder is configured to shift a single dimension of the first multidimensional array to a different position to obtain a reordered multidimensional array. This can provide a simple variant of the reordering of the present invention, thereby limiting the overall impact on the complexity of processing and signaling overhead, while allowing for improved processing and / or coding efficiency.

[0030] According to an embodiment of the present invention, the decoder is configured to obtain a two-dimensional matrix, for example having dimensions [D2, (D0*D1*D3)], wherein a first dimension of the two-dimensional matrix is ​​determined by a first dimension value (for example, D2), and a second dimension of the two-dimensional matrix is ​​determined based on the product of multiple further dimension values ​​represented by the encoding (for example, D0*D1*D2). In addition, the decoder is configured to obtain a first multidimensional array (e.g., having dimensions [D2, D0, D1, D3]), the dimensions of the first multidimensional array being optionally determined by a single dimension value among a first dimension value (e.g., D2) and further dimension values ​​(e.g., D0, D1, D3) (wherein, for example, each of the first dimension value and the further dimension value may be associated with (and / or defined) a dimension of the first multidimensional array; where, for example, the first dimension value may be associated with (and / or defined) a first dimension of the first multidimensional array based on a two-dimensional matrix. In addition, the decoder is configured to obtain a reordered multidimensional array, e.g., a multidimensional array having dimensions [D0, D1, D2, D3], based on the first multidimensional array, wherein the first dimension of the reordered multidimensional array is defined by one of the further dimension values.

[0031] The inventors have realized that the decoder may perform, for example as an intermediate step of the reordering, an interpretation of the parameter representation of the encoded representation as a two-dimensional matrix. This approach may enable compatibility with existing frameworks to include the reordering of the invention based on the interpreted two-dimensional matrix. In particular, the inventors have realized that "shuffling" one of the further dimension values ​​obtained by the encoded representation into the first dimension of the reordered multidimensional array may increase the flexibility of the processing, since the shape of the encoded tensor may be different from the shape of the processed tensor.

[0032] According to an embodiment of the invention, the decoder is configured to decode the encoded neural network parameters using context-based entropy decoding, for example using arithmetic decoding, wherein one or more contexts for the arithmetic decoding of a given encoded neural network parameter are determined based on previously decoded neural network parameters, for example based on previously decoded neural network parameters that directly precede the given encoded neural network parameter in decoding order. In addition, the decoder is configured to determine the position of the decoded neural network parameters in the first multidimensional array using a position mapping function such as TensorIndex(tensorDimensions[],i,scan), which maps a scalar neural network parameter index (e.g., i) to a set of dimension indices (e.g., a vector or a (preferably 1-dimensional) array, such as an array having a number of entries equal to the number of dimensions of the first multidimensional array), such as an array pointing to the i-th element in row-major scan order of the multidimensional array having the dimensions of the first multidimensional array, or to an array pointing to the i-th element in block scan order (e.g., scan sub-blocks) of a two-dimensional interpretation of the first multidimensional array, such as defined by the function TensorIndex(tensorDimensions[],I,scan), when used in conjunction with IndexToXY, scan>0. This approach can efficiently decode and reorder tensor values ​​and dimensions.

[0033] According to an embodiment of the present invention, the decoder is configured to store a plurality of decoded neural network parameters (e.g., D3 decoded neural network parameters) continuously along a given dimension (e.g., the fourth dimension or the highest dimension) of the first multidimensional array, which parameters are directly adjacent with respect to the decoding order (e.g., decoding in sequence; e.g., decoding based on the previous value taking into account context adaptation), wherein, for example, the index of the given dimension increases or decreases continuously, wherein, for example, the index of the other dimensions remains unchanged. Therefore, the storage of the decoded neural network parameters can be performed while maintaining the respective order of the parameters, for example, at least within one dimension.

[0034] According to an embodiment of the present invention, the decoder is configured to obtain the reordered multidimensional array using the following form of processing (such as calculation):

[0035]

[0036] Where i is a running variable, where Prod(inputTensorDims) is the product of the dimension lengths of the first multidimensional array, e.g., may be equal to the number of elements of the first multidimensional array. In addition, TensorIndex(.,.,.) is a function that maps an integer value (e.g., a running variable i, e.g., a scalar integer value that references an element of the first multidimensional array, e.g., in ascending order) to a set of array indices (e.g., a one-dimensional array or vector) (e.g., a set that includes a number of array indices equal to the number of dimensions of the first multidimensional array), the set specifying elements of the first multidimensional array (e.g., decoded neural network parameters). In addition, inputTensorDims is a set of values ​​(e.g., a one-dimensional array or collection) describing the lengths of dimensions of the first multidimensional array, idxA is a variable (e.g., a set representing array indices (e.g., a one-dimensional array or vector)), and ShiftArrayIndex(.,.) is a function that maps an input set of array indices (e.g., a one-dimensional array or collection) (e.g., a set of array indices equal to the number of dimensions of the first multidimensional array, e.g., idxA) to a returned set of array indices (e.g., a one-dimensional array or collection, e.g., idxB), wherein the returned set of array indices is a copy of the input set of array indices, but the element at position 0 of the input set of array indices is shifted to the position specified by firstTensorDimShift (where the positions can be counted from 0 up to the size of the input set of array indices - 1). In addition, firstTensorDimShift is a dimension shift value (e.g., describing how much the first dimension of the first multidimensional array should be shifted), idxB is a variable (e.g., representing a set of array indices (e.g., a one-dimensional array or a vector), inputTensor[.] is the first multidimensional array, reorderedTensor[.] is the reordered multidimensional array, inputTensor[idxA] is the element at the position specified by idxA in the first multidimensional array, and reorderedTensor[idxB] is the element at the position specified by idxB in the reordered multidimensional array. Optionally, firstTensorDimShift can correspond to first_tensor_dimension_shift. The inventors recognize that this method can efficiently perform the reordering process.

[0037] According to an embodiment of the present invention, the decoder is configured to obtain a reordered multidimensional array using a function (e.g., TensorIndex(tensorDimensions[], I, scan)) that maps an integer element index i that specifies (e.g., uniquely specifies) an element of a first multidimensional array to a set of array indices (e.g., a one-dimensional array or vector), wherein the returned set of array indices specifies the i-th element in a row-major scan order of the first multidimensional array (e.g., an array whose array dimensions are specified by tensorDimensions[]), or wherein the returned set of array indices specifies the i-th element of the first multidimensional array scanned in blocks, e.g., using blocks of size bs times bs, wherein the first multidimensional array is a two-dimensional array, e.g., a two-dimensional array of width w and height h, wherein the elements are represented by x and y, wherein x and y can be obtained, for example, by a function IndexToXY that maps a scalar integer index i to a set (e.g., an array) consisting of two elements x and y. The inventors recognize that the reordering of the present invention can be performed according to different scan orders of the first multidimensional array. Therefore, the method of the present invention is flexible and easily compatible with multiple scanning methods.

[0038] According to an embodiment of the present invention, the decoder is configured to decode a skip line list, such as skip line list [i], wherein, for example, the number of entries of the skip line list (such as tensor2Dheight) can be equal to the length of the first dimension of the first multidimensional array (such as dimensions [0]). In addition, the decoder is configured to input the decoded neural network coefficients (such as QuantParam [idx]) into the first multidimensional array, for example by setting QuantParam [idx] to obtain the decoded value, and input it to the corresponding position described by the corresponding array index set (such as represented by the variable idx, which is obtained, for example, using the function idx = TensorIndex (dimensions, i, scan_order)). Furthermore, the decoder is further configured to decide, based on the skip row list entry referenced (e.g., specified) by the current array index of the first dimension of the first multidimensional array (e.g., row_skip_list[idx[0]]), whether to use a default value (e.g., QuantParam[idx]=0) for a given neural network parameter (or, e.g., even a sequence of neural network parameters having the same current array index of the first dimension of the first multidimensional array), or whether to use decoding to determine the given neural network parameter (or, e.g., even a sequence of neural network parameters having the same current array index of the first dimension of the first multidimensional array), e.g., by a call of a decoding function (e.g., int_param(i, maxNumNoRemMinus1, stateId)). Thus, based on the information of the skip row, the decoding work can be reduced.

[0039] According to an embodiment of the present invention, the decoder is configured to decode an array dimension set (e.g., a one-dimensional array or a vector (e.g., describing the length of a first multidimensional array in different dimensions, such as tensor_dimensions[.]), wherein, for example, the dimensions of the array may be Exp-Golomb encoded. In addition, the decoder is configured to obtain a reordered array dimension set, such as a one-dimensional array or a vector, wherein the entries describe the length of the reordered multidimensional array in different directions, for example, by processing the decoded array dimension set using a ShiftArrayIndex function. For example, the determination of the ordering of tensor or array dimensions may therefore be performed separately from the shaping or ordering of the respective tensors and / or arrays themselves. The inventors recognize that this may improve the efficiency of the reordering.

[0040] According to an embodiment of the present invention, the decoder is configured to subsequently input the decoded neural network parameters (e.g., QuantParam[idx]) into the first multidimensional array, for example by setting QuantParam[idx] to obtain the decoded value, into the corresponding position described by the corresponding array index set, for example represented by the variable idx, for example, the variable is obtained by the function idx=TensorIndex(dimensions,i,scan_order). In addition, the decoder is configured to obtain the corresponding array index set using a mapping function (e.g., TensorIndex(tensorDimensions[],i,scan), for example, which can call IndexToXY), which maps a scalar integer parameter index (e.g., i) to a corresponding array index set and defines a block-by-block scan, wherein, for example, subsequent values ​​of the scalar integer parameter index can be associated with subsequently decoded parameter values. The inventors recognized that this approach can provide an efficient reordering method.

[0041] According to an embodiment of the present invention, a mapping function includes a mapping of a scalar integer parameter index (e.g., i) to two coordinates (e.g., x, y), wherein the two coordinates point to a position corresponding to a scan index defined by the scalar integer parameter index (e.g., i), and when scanning, a block (e.g., a two-dimensional array with a width of w and a height of h) is scanned in blocks, such as a secondary block scan; such as scanning in blocks of size bs times bs. In addition, the mapping function includes a mapping of the two coordinates to respective array index sets. The inventors recognize that this method can efficiently handle dimensions that need to be reordered.

[0042] According to an embodiment of the present invention, the decoder is configured to subsequently input the decoded neural network parameters (e.g., QuantParam[idx]) into the first multidimensional array, for example, by setting QuantParam[idx] to obtain the decoded value, into the corresponding position described by the corresponding array index set (e.g., represented by the variable idx, which is obtained, for example, using the function idx=TensorIndex(dimensions, i, scan_order)). In addition, the decoder is configured to obtain the corresponding array index set based on the scalar integer parameter i using the function TensorIndex(tensorDimensions[], i, scan), wherein the function TensorIndex(tensorDimensions[], i, scan) returns an array of array indices having the same dimensionality as the array tensorDimensions[], wherein the elements of the array of array indices are set to integer values ​​so that the array of array indices can be used as an index to point to array elements, for example, an array with dimensions tensorDimensions[], as follows:

[0043] If the variable scan is equal to 0:

[0044] The returned array index points to the i-th element in row-major scan order of the array [e.g., tensor] [e.g., the first multidimensional array] with the dimensions defined by tensorDimensions[];

[0045] If the variable scan is greater than 0:

[0046] The variable bs is set to 4< <scan_order;

[0047] The variable h is set to tensorDimensions[0]

[0048] [For example, set to the length of the first dimension of the first multidimensional array];

[0049] The variable w is set to Prod(tensorDimensions) / h

[0050] [For example, set to the product of the lengths of the first multidimensional arrays in different dimensions divided by h, or the number of elements in the first multidimensional array divided by h];

[0051] The two variables x and y are set to the first and second elements of the returned array,

[0052] By calling IndexToXY(w,h,i,bs) respectively;

[0053] The returned array is TensorIndex(tensorDimensions,y*w+x,0);

[0054] Wherein, the function IndexToXY(w,h,i,b) returns an array including two elements, the first element returned by the function IndexToXY is the x coordinate, and the second element returned by the function IndexToXY is the y coordinate, pointing to a two-dimensional array of width w and height h, where when the block is scanned in units of blocks of size bs times bs [for example, a two-dimensional array of width w and height h], x and y point to the position corresponding to the scan index i; where x and y are derived as follows:

[0055] The variable fullRowOfBlocks is set to w*bs;

[0056] The variable blockY is set to i / fullRowOfBlocks;

[0057] The variable iOff is set to i%fullRowOfBlocks;

[0058] The variable currBlockH is set to Min(bs,h-blockY*bs);

[0059] The variable fullBlocks is set to bs*currBlockH;

[0060] The variable blockX is set to iOff / fullBlocks;

[0061] The variable blockOff is set to iOff%fullBlocks;

[0062] The variable currBlockW is set to Min(bs,w-blockX*bs);

[0063] The variable posX is set to blockOff%currBlockW;

[0064] The variable posY is set to blockOff / currBlockW;

[0065] The variable x is set to blockX*bs+posX;

[0066] The variable y is set to blockY*bs+posY;

[0067] Where tensorDimensions[] is an array of values ​​describing the lengths of the first multidimensional array at different dimensions; where scan is an integer variable describing the scanning method. The inventors recognize that this method can provide an efficient reordering method.

[0068] The embodiment consisting of encoder according to the design of the present invention will be discussed below. The encoder described below can be based on the consideration identical with the above-mentioned decoder. Incidentally, these encoders can be used alone or in combination, and have all features and functions, which will also be introduced in the decoder.

[0069] According to a further embodiment of the invention, the encoder is configured to provide an encoded representation of neural network parameters (e.g., "weights" or "coefficients"), e.g., a representation of the encoded (e.g., compressed) neural network parameters in the form of a bitstream. Furthermore, the encoder is configured to obtain a reordered multidimensional array (e.g., BEnc[o][m][n][p], where o, m, n, p are index variables) using a reordering, wherein a given dimension of a given multidimensional array of neural network parameters (e.g., a given dimension of a multidimensional array specified by a given array index, e.g., a third dimension of a given multidimensional array AEnc[m][n][o][p]) is reordered, e.g., moved to a first dimension in the reordered multidimensional array, which dimension is optionally different from the given dimension. Furthermore, the encoder is configured to encode the reordered multidimensional array (Benc[o][m][n][p]), e.g., using an encoding of the entries of the reordered multidimensional array.

[0070] According to an embodiment of the present invention, the encoder is configured to encode a dimension shift value (e.g., a “first_tensor_dimension_shift” value, such as a single scalar value), and the dimension shift value describes which given dimension of a given multidimensional array is reordered as the first dimension in the reordered multidimensional array, or the dimension shift value describes how many dimensions (e.g., how many array indices) the first dimension of the encoded reordered multidimensional array (e.g., D2) should be shifted when performing decoder-side reordering, for example, to a higher dimension to obtain a decoder-side (twice) reordered multidimensional array, such as an array with dimensions [D0, D1, D2, D3].

[0071] In accordance with an embodiment of the present invention, the encoder is configured to encode a dimension shift value, such as a “first_tensor_dimension_shift” value, and the dimension shift value (e.g., a scalar value) describes the new position to which the first dimension, e.g., D2, of the encoded reordered multidimensional array (e.g., having dimensions [D2, D0, D1, D3]) should be moved in the decoder.

[0072] According to an embodiment of the present invention, the encoder is configured to encode the dimension shift value using Exp Golomb code.

[0073] According to an embodiment of the present invention, the encoder is configured to perform the reordering using a single scalar dimension shift value (eg, first_tensor_dimensionshift) as the only parameter describing the new order of dimensions in the reordered multidimensional array.

[0074] According to an embodiment of the invention, the encoder is configured to, for example, perform only a single shift of a single dimension to another position to obtain a reordered multidimensional array.

[0075] According to an embodiment of the present invention, the encoder is configured to set the dimensions of auxiliary arrays included in the encoded representation (e.g., the dimensions of a vector or a two-dimensional array or an array with more than two dimensions; for example, the dimensions of a tensor or a "related tensor" or a "corresponding tensor", such as the dimensions of an "auxiliary" array of bias values, or the dimensions of an "auxiliary" array of batch normalization parameters, or the dimensions of an "auxiliary" array of scaling factors) to be equal to the first dimension of the reordered multidimensional array.

[0076] According to an embodiment of the present invention, the encoder is configured to shift a single dimension of a given multidimensional array to a different position to obtain a reordered multidimensional array.

[0077] According to an embodiment of the present invention, the encoder is configured to obtain a reordered multidimensional array (e.g., having dimensions [D2, D0, D1, D3]), which dimensions can be optionally determined by a single dimension value of a first dimension value (e.g., D2) and further dimension values ​​(e.g., D0, D1, D3) associated with a first dimension (where, for example, each of the first dimension value and the further dimension value can be associated with (and / or defined) a dimension of the reordered multidimensional array; where, for example, the first dimension value can be associated with (and / or defined) a first dimension of the reordered multidimensional array based on a given multidimensional array. In addition, the encoder is configured to obtain a two-dimensional matrix, for example, having dimensions [D2, (D0*D1*D3)], where the first dimension is determined by the first dimension value (e.g., D2), and the second dimension of the two-dimensional matrix is ​​determined by the product of the further dimension values ​​based on the reordered multidimensional array (e.g., D0*D1*D2). In addition, the encoder is configured to encode the two-dimensional matrix.

[0078] According to an embodiment of the present invention, the encoder is configured to encode the neural network parameters using context-based entropy coding, for example using arithmetic coding, wherein one or more contexts for the arithmetic coding of a given neural network parameter are determined based on previously encoded neural network parameters, for example, based on previously encoded neural network parameters that directly precede the given neural network parameter in the coding order. In addition, the encoder is configured to determine the neural network parameters to be encoded in the reordered multidimensional array using a position mapping function (such as TensorIndex(tensorDimensions[],i,scan)) that maps a scalar parameter index (e.g., i) to a set of dimension indices (e.g., a vector or a (preferably 1-dimensional) array, such as an array of data having entry data equal to the number of dimensions of the reordered multidimensional array), for example, to an array pointing to the i-th element in a row-major scan order of the multidimensional array having the dimensions of the reordered multidimensional array, or to an array pointing to the i-th element in a block-by-block scan order (e.g., a scan sub-block) of a two-dimensional interpretation of the reordered multidimensional array, such as defined by the function TensorIndex(tensorDimensions[],i,scan), when used in conjunction with IndexToXY, scan>0.

[0079] According to an embodiment of the present invention, the encoder is configured to continuously read multiple neural network parameters to be encoded (e.g., D3 neural network parameters) along a given dimension (e.g., the fourth dimension or the highest dimension) of a reordered multidimensional array, which are directly adjacent with respect to the encoding order (e.g., encoded in sequence; for example, encoding taking into account context adaptation based on a previous value), wherein, for example, the index of the given dimension increases or decreases continuously, wherein, for example, the indexes of other dimensions remain unchanged.

[0080] According to an embodiment of the present invention, the encoder is configured to obtain a reordered multidimensional array using the following form of processing (such as calculation):

[0081]

[0082] Wherein, i is a running variable; wherein Prod Prod(inputTensorDims) is the product of the dimension lengths of a given multidimensional array, for example, may be equal to the number of elements of the given multidimensional array. In addition, TensorIndex(.,.,.) is a function (e.g., a set of array indices equal to the number of dimensions of a given multidimensional array) that maps an integer value (e.g., a running variable i, such as a scalar integer value referencing the first multidimensional array element, such as in ascending order) to an array index set (e.g., a one-dimensional array or a vector), the set specifying elements of the given multidimensional array, such as neural network parameters to be encoded. In addition, inputTensorDims is a set of values ​​(e.g., a one-dimensional array or a set) describing the lengths of the dimensions of a given multidimensional array, and idxA is a variable, for example, representing an array index set (e.g., a one-dimensional array or a vector). In addition, ShiftArrayIndexEnc(.,.) is a function that maps an input array index set (e.g., a one-dimensional array or collection) (e.g., a set of array indices equal to the number of dimensions of a given multidimensional array, e.g., idxA) to a returned set of array indices (e.g., a one-dimensional array or collection, e.g., idxB), wherein the returned array index set is a copy of the input array index set, but the elements of the input array index set at the positions specified by firstTensorDimShift are shifted to position 0, wherein as an optional feature position is counted from 0 up to the size of the input array index set - 1. In addition, firstTensorDimShift is a dimension shift value, for example, describing which given dimension of a given multidimensional array should be shifted to the first position, idxB is a variable, for example, representing an array index set (such as a one-dimensional array or a vector), inputTensor[.] is a given multidimensional array, reorderedTensor[.] is the reordered multidimensional array, inputTensor[idxA] is the element at the position specified by idxA in the given multidimensional array, and reorderedTensor[idxB] is the element at the position specified by idxB in the reordered multidimensional array.

[0083] According to an embodiment of the present invention, the encoder is configured to obtain a reordered multidimensional array using a function (e.g., TensorIndex(tensorDimensions[], I, scan)), which maps an integer element index i that specifies (e.g., uniquely specifies) an element of a given multidimensional array to an array index set (e.g., a one-dimensional array or a vector), wherein the returned array index set specifies the i-th element in a row-major scan order of a given multidimensional array (e.g., an array whose array dimensions are the dimensions of the array specified by tensorDimensions[]), or wherein the returned array index set specifies the i-th element of a given multidimensional array scanned in blocks (e.g., using blocks of size bs by bs), wherein the given multidimensional array is a two-dimensional array, e.g., a two-dimensional array with a width of w and a height of h, and the elements are, for example, represented by x and y, where x and y can be obtained using a function IndexToXY, which maps a scalar integer index i to a set (e.g., an array) consisting of two elements x and y.

[0084] According to an embodiment of the present invention, (e.g., where the encoder is configured to obtain a skip row list) the encoder is configured to encode a skip row list (e.g., skip row list [i]), wherein the number of entries of the skip row list (e.g., tensor2Dheight) may be equal to the length of the first dimension of the reordered multidimensional array (e.g., dimensions[0]). In addition, the encoder is configured to optionally encode the neural network coefficients of the reordered multidimensional array (e.g., QuantParam[idx]) at the corresponding position described by the corresponding array index set (e.g., represented by the variable idx, which is obtained using the function idx = TensorIndex(dimensions, I, scan_order)). In addition, the encoder is further configured to decide whether to encode the neural network parameters at the current position, such as the neural network parameters specified by the current array index set, or, for example, even a sequence of neural network parameters with the same current array index on the first dimension of the reordered multidimensional array, based on the entry of the skip row list specified by the current array index (e.g., row_skip_list[idx[0]]) of the first dimension of the reordered multidimensional array.

[0085] According to an embodiment of the present invention, the encoder is configured to optionally obtain a reordered array dimension set, such as a one-dimensional array or a vector, wherein the entries describe the length of the reordered multidimensional array in different directions, for example, using the ShiftArrayIndexEnc function to process the array dimension set associated with a given multidimensional array. In addition, the encoder is configured to optionally encode the reordered array dimension set (such as a one-dimensional array or a vector), for example, describing the length of the reordered multidimensional array in different dimensions, such as tensor_dimensions[.], where, for example, the array dimensions can be Exp-Golomb encoded.

[0086] According to an embodiment of the present invention, the encoder is configured to encode the neural network parameters (e.g., QuantParam[idx]) of the reordered multidimensional array at the corresponding position described by the corresponding array index set (e.g., represented by the variable idx, for example, obtained using the function idx=TensorIndex(dimensions, I, scan_order)). In addition, the encoder is configured to obtain the corresponding array index set using a mapping function (TensorIndex(tensorDimensions[], i, scan), for example, it can call IndexToXY), which maps a scalar integer parameter index (e.g., i) to a corresponding array index set and defines a block-by-block scan, where, for example, subsequent values ​​of the scalar integer parameter index can be associated with subsequently encoded parameter values.

[0087] According to an embodiment of the present invention, the mapping function includes a mapping of a scalar integer parameter index (e.g., i) to two coordinate examples (e.g., x, y), the two coordinates pointing to a position corresponding to a scan index defined by the scalar integer parameter index (e.g., i), when scanning, a block (e.g., a two-dimensional array with a width of w and a height of h) is scanned in blocks, such as a quadratic block scan; such as a block scan with a size of bs times bs. In addition, the mapping function includes a mapping of the two coordinates to respective array index sets.

[0088] According to an embodiment of the present invention, the encoder is configured to encode the neural network parameters of the reordered multidimensional array at the corresponding position described by the corresponding array index set (for example, represented by the variable idx, which is obtained, for example, using the function idx=TensorIndex(dimensions, i, scan_order)). In addition, the encoder is configured to obtain the corresponding array index set based on the scalar integer parameter index i using the function TensorIndex(tensorDimensions[], i, scan), where the function TensorIndex TensorIndex(tensorDimensions[], i, scan) returns an array of array indices with the same dimensions as the array tensorDimensions[], and the elements of the array of array indices are set to integer values ​​so that the array of array indices can be used as an index to point to array elements, for example, an array with dimensions tensorDimensions[], as shown below:

[0089] If the variable scan is equal to 0:

[0090] The returned array index points to the i-th element in row-major scan order of the array [e.g., tensor] [e.g., the first multidimensional array] with the dimensions defined by tensorDimensions[];

[0091] If the variable scan is greater than 0:

[0092] The variable bs is set to 4< <scan_order;

[0093] The variable h is set to tensorDimensions[0]

[0094] (For example, set to the length of the first dimension of the first multidimensional array);

[0095] The variable w is set to Prod(tensorDimensions) / h

[0096] (For example, set to the product of the lengths of the first multidimensional arrays in different dimensions divided by h, or the number of elements in the first multidimensional array divided by h);

[0097] The two variables x and y are set to the first and second elements of the returned array

[0098] By calling IndexToXY(w,h,i,bs) respectively;

[0099] The returned array is TensorIndex(tensorDimensions,y*w+x,0);

[0100] Wherein, the function IndexToXY(w,h,i,b) returns an array including two elements, the first element returned by the function IndexToXY is the x coordinate, and the second element returned by the function IndexToXY is the y coordinate, pointing to a two-dimensional array of width w and height h, where when the block is scanned in units of blocks of size bs times bs [for example, a two-dimensional array of width w and height h], x and y point to the position corresponding to the scan index i; where x and y are derived as follows:

[0101] The variable fullRowOfBlocks is set to w*bs;

[0102] The variable blockY is set to i / fullRowOfBlocks;

[0103] The variable iOff is set to i%fullRowOfBlocks;

[0104] The variable currBlockH is set to Min(bs,h-blockY*bs);

[0105] The variable fullBlocks is set to bs*currBlockH;

[0106] The variable blockX is set to iOff / fullBlocks,

[0107] The variable blockOff is set to iOff%fullBlocks;

[0108] The variable currBlockW is set to Min(bs,w-blockX*bs);

[0109] The variable posX is set to blockOff%currBlockW;

[0110] The variable posY is set to blockOff / currBlockW;

[0111] The variable x is set to blockX*bs+posX;

[0112] The variable y is set to blockY*bs+posY;

[0113] In addition, tensorDimensions[] is an array of values ​​describing the lengths of the first multidimensional array at different dimensions; where scan is an integer variable describing the scanning method.

[0114] The following will discuss methods according to embodiments of the present invention. The methods described below can be based on the same considerations as the above-mentioned decoder and encoder, respectively. Incidentally, these methods can also use all the features and functions of the decoder and encoder, either alone or in combination.

[0115] Embodiments of the invention include a method for providing decoded parameters, such as "weights" or "coefficients", of a neural network based on an encoded representation (e.g., based on a bitstream representing the neural network parameters in an encoded (e.g., compressed) form). In addition, the method includes obtaining a first multidimensional array (e.g., inputTensor[idxA], where idxA is a vector of multiple index variables; e.g., A[dm][dn][do][dp], where dm, dn, do, dp are index variables, e.g., A[m][n][o][p], where m, n, o, p are index variables) comprising multiple neural network parameter values, such as neural network weights, using decoding of the neural network parameters, such as causing the first multidimensional array to include decoded neural network parameters. In addition, the method includes using the reordering to obtain a reordered multidimensional array (e.g., reorderedTensor[idxB], where idxB is a vector of multiple index variables; e.g., B[n][o][m][p], where n, o, m, p are index variables), wherein a first dimension of a first multidimensional array, e.g., a dimension of the multidimensional array specified by a leftmost array index, is reordered (e.g., moved) to a different dimension in the reordered multidimensional array, which is optionally different from the first dimension of the reordered multidimensional array.

[0116] According to an embodiment of the invention, a method of providing an encoded representation of neural network parameters (e.g., "weights" or "coefficients"), such as a bitstream representing the parameters of the neural network in an encoded (e.g., compressed) form. In addition, the method includes using a reordering to obtain a reordered multidimensional array (e.g., BEnc[o][m][n][p], where o, m, n, p are index variables), wherein a given dimension of a given multidimensional array of neural network parameters (e.g., a given dimension of a multidimensional array specified by a given array index, such as a third dimension of a given multidimensional array AEnc[m][n][o][p]) is reordered, such as moved to a first dimension in the reordered multidimensional array, which dimension is optionally different from the given dimension. In addition, the method also includes encoding the reordered multidimensional array (e.g., Benc[o][m][n][p]), such as using an encoding of the entries of the reordered multidimensional array.

[0117] The embodiment includes a computer program, which is used to execute the method according to any embodiment disclosed in this disclosure when the computer program is run on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] The accompanying drawings are not necessarily drawn to scale, with emphasis generally being placed on illustrating the principles of the invention. In the following description, various embodiments of the invention will be described with reference to the following drawings, in which:

[0119] Figure 1 A schematic diagram of a decoder according to an embodiment of the present invention is shown, for providing decoding parameters of a neural network based on an encoded representation;

[0120] Figure 2 A schematic diagram of a decoder according to an embodiment of the present invention having further optional features is shown;

[0121] Figure 3 A schematic diagram of an encoder according to an embodiment of the present invention is shown for providing an encoded representation of neural network parameters;

[0122] Figure 4 A schematic block diagram of a method for providing decoding parameters of a neural network based on an encoded representation is shown;

[0123] Figure 5 A schematic block diagram of a method for providing an encoded representation of neural network parameters is shown;

[0124] Figure 6 A schematic block diagram showing the concept of tensor dimension reordering of an encoding-decoding pipeline according to an embodiment of the present invention is shown;

[0125] Figure 7 A schematic diagram of a double-layer feedforward neural network according to an embodiment of the present invention is shown;

[0126] Figure 8 A schematic diagram of a uniform reconstruction quantizer according to an embodiment of the present invention is shown;

[0127] 9(a), (b) show schematic diagrams of allowable reconstruction vector positions in a simple case with two weight parameters according to an embodiment: (a) independent scalar quantization; (b) dependent scalar quantization;

[0128] Fig.10 shows a schematic example of dividing the reconstruction level set into two subsets according to an embodiment;

[0129] Fig.11 shows a schematic example of an NNR encoding pipeline according to an embodiment;

[0130] Fig.12 shows a schematic example of an NNR unit data structure according to an embodiment;

[0131] Fig.13 shows a schematic example of an aggregated NNR unit data structure; and

[0132] Fig.14 A schematic example of an NNR bitstream data structure according to an embodiment is shown. DETAILED DESCRIPTION

[0133] The same or equivalent elements or elements having the same or equivalent functions are denoted by the same or equivalent reference numerals in the following description even if they appear in different drawings.

[0134] In the following description, a plurality of details are proposed to provide a more comprehensive explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be implemented without these specific details. In other cases, in order to avoid obscuring embodiments of the present invention, well-known structures and devices are shown in block diagram form rather than in detail. In addition, unless otherwise specifically stated, the features of the different embodiments described below may be combined with each other.

[0135] Figure 1 A schematic diagram of a decoder according to an embodiment of the present invention is shown, the decoder being used to provide decoding parameters of a neural network based on an encoded representation. Figure 1 A decoder 100 is shown consisting of a decoding unit 110 and a reordering unit 120. A bitstream 101 is provided to the decoder 100, the bitstream comprising an encoded representation (e.g. in compressed form) of neural network parameters. For example, the bitstream 101 provided to the decoder 100 may be an encoded representation of neural network parameters.

[0136] like Figure 1 As shown, the decoding unit 110 is provided with the bitstream 101 to obtain a first multidimensional array 111 comprising a plurality of neural network parameter values. Therefore, the decoding unit 100 is configured to decode the bitstream 101 or at least decode the encoded representation of the neural network parameters, which is part of the bitstream 101.

[0137] The first multidimensional array 111 is provided to a reordering unit 120. The reordering unit 120 is configured to reorder the first multidimensional array 111 to obtain a reordered multidimensional array 121. The reordering unit 120 is configured to perform a reordering in which a first dimension of the first multidimensional array 111 is reordered to a different dimension in the reordered multidimensional array 121.

[0138] The decoder 100 may provide the reordered multidimensional array 121 as the decoding parameters of the neural network 102. As shown, as an optional feature, in order to provide the decoding parameters of the neural network, the decoder 100 may be configured to further process (e.g., using multiple or at least one optional further processing unit 130) the reordered multidimensional array 121 to provide the decoding parameters 102 of the neural network.

[0139] like Figure 1 As shown, as an optional feature, the decoding unit is configured to decode a dimension shift value 112 from the bitstream 101 and provide it to the reordering unit 120. The dimension shift value 112 provides information to the reordering unit 120 about the extent to which the first dimension of the first multidimensional array 111 needs to be rearranged (e.g., shifted, such as moved).

[0140] Therefore, the dimension shift value 112 may optionally include or be a kind of information based on which the reordering unit 120 may determine a new location or position of the first dimension of the first multidimensional array 111 to be reordered (eg, changed) in the reordered multidimensional array 121 .

[0141] As an optional feature, the dimension shift value 112 is Exp Golomb-coded. Therefore, as an optional feature, the decoding unit 110 is configured to decode the Exp Golomb-coded dimension shift value 112 from the bitstream 101 .

[0142] It should be noted that the dimension shift value 112 may be, for example, a single scalar. As an optional feature, the single scalar may be the only parameter based on which the new order of the dimensions in the reordered multidimensional array 121 may be specified.

[0143] Furthermore, it is worth noting that, according to some embodiments, the reordering unit 120 may be configured to perform a single shift of a single dimension to another position to obtain the reordered multidimensional array 121 .

[0144] Figure 2 A schematic diagram of a decoder with further optional functions according to an embodiment of the present invention is shown. Figure 2 A decoder 200 is shown, comprising a decoding unit 210 and a reordering unit 220. The decoder 200 is provided with a bitstream 201 to obtain a first multidimensional array 211. The first multidimensional array 211 is provided to the reordering unit 220, which is configured to reorder the first multidimensional array 211 to obtain a reordered multidimensional array 221. The decoder 200 may provide the reordered multidimensional array 221 as a decoding parameter 202 of a neural network. As shown, as an optional feature, in order to provide the decoding parameters of the neural network, the decoder 200 may be configured to further process (e.g., using a plurality of or at least one optional further processing unit 230) the reordered multidimensional array 221 to provide the decoding parameters 202 of the neural network.

[0145] for Figure 2 For details of the above features and functions of the illustrated embodiment, for the sake of brevity, please refer to Figure 1 It should be noted that if Figure 2Similar or corresponding features shown may optionally include Figure 1 The same or corresponding functions and / or details are described. In addition, Figure 1 All features marked and / or discussed as optional are for Figure 2 The embodiments are also optional.

[0146] like Figure 2 As shown, the reordering unit 220 has a dimension shift value 212 as an optional feature.

[0147] Furthermore, as an optional feature, the first multidimensional array 211 is provided to an optional further processing unit 230a. The further processing unit 230a is configured to derive the dimension of the auxiliary array from the first dimension of the first multidimensional array 211.

[0148] For example, the first multidimensional array 211 may be a weight tensor, and the bitstream 201 may include an encoded representation of a floating-point parameter tensor, the floating-point parameter tensor including the (optionally decomposed) weight tensor and optional local scaling parameters, biases and batch normalization parameters, which form a block in the model architecture. In other words, the weight tensor and several corresponding tensors, such as biases, batch normalization parameters and scaling factors, may be included together in the bitstream 201, for example in one compressed data unit (such as an NDU). According to such an example, the auxiliary array may include the respective corresponding tensors, or at least their values, for example in an adapted shape.

[0149] For example, since the corresponding tensors may be related to the weight tensor, their dimensions may be derived from the (weight) tensor dimensions, for example using further processing unit 230a. For example, biases, batch normalization parameters and scaling factors may be one-dimensional tensors whose lengths are respectively equal to the number of output channels (e.g., fully connected layers) or the number of filters (e.g., convolutional layers) of the weight tensor. In order to derive these dimensions, the first dimension of the decoded weight tensor may be determined as the length of the one-dimensional tensor. Thus, the decoded weight tensor may be sorted such that the first dimension corresponds to the number of output channels or filters.

[0150] In other words, the first multidimensional array 211 (e.g., as such a weight tensor) may be ordered such that the first dimension corresponds to the number of output channels or filters. Thus, one or more auxiliary arrays may be determined to represent offsets, batch normalization parameters, and scaling factors. Corresponding information 213 for determining such auxiliary arrays may be provided by the decoding unit 213 to the further processing unit 230a.

[0151] Thus, the first multidimensional array 211 (e.g. in the form of a weight tensor representation) may have been reordered on the encoder side, so that the first dimension of the encoded representation of the first multidimensional array in the bitstream 201 can be derived from the first dimension of the first multidimensional array 211. After determining the dimension, the first multidimensional array 211 can be reordered to its state before encoding using a reordering unit 220. As shown, information 231a including such an auxiliary array is optionally provided for determining the decoded neural network parameters 202.

[0152] As an optional feature, the reordering unit 220 is configured to shift a single dimension of the first multidimensional array to a different position to obtain a reordered multidimensional array. For example, allowing arbitrary dimensional order may result in huge signaling overhead, so shifting a single dimension to another position can keep the signaling overhead low.

[0153] As an optional feature, the decoding unit 210 is configured to perform context-based decoding. In particular, the decoding unit 210 is configured to decode the encoded neural network parameters using context-based entropy decoding. Therefore, the context information 240 is provided to the decoding unit 210. In particular, as an optional feature, the decoding unit 210 is configured to determine the position of the decoded neural network parameters in the first multidimensional array 211 using a position mapping function that maps a scalar neural network parameter index to a set of dimension indices.

[0154] Figure 3 A schematic diagram of an encoder according to an embodiment of the present invention is shown, the encoder being used to provide an encoded representation of neural network parameters. Figure 3 An encoder 300 is shown, comprising an encoding unit 310 and a reordering unit 320 .

[0155] The reordering unit is configured to obtain a reordered multidimensional array 321 using the reordering, wherein a given dimension 301 of a given multidimensional array 302 of neural network parameters is reordered to a first dimension in the reordered multidimensional array 321. In addition, the encoding unit 310 is configured to encode the reordered multidimensional array 321, for example, as a bit stream 303 as shown.

[0156] As mentioned above, encoders according to embodiments of the present invention may be based on the same considerations as corresponding decoders. Therefore, for the sake of brevity, reference is made to the embodiments including decoders disclosed above and below for additional, optional features. Therefore, it should be noted that encoders according to embodiments, such as encoder 300, may include any or all of the features disclosed in the context of any decoder of the present invention, either alone or in combination.

[0157] Figure 4 A block diagram of a method for providing decoded parameters of a neural network based on an encoded representation is shown. The method 400 includes: 410, using decoding of neural network parameters to obtain a first multidimensional array including a plurality of neural network parameter values, and 420, using reordering to obtain 420 a reordered multidimensional array, wherein a first dimension of the first multidimensional array is reordered to a different dimension in the reordered multidimensional array.

[0158] Figure 5 A block diagram of a method for providing an encoded representation of neural network parameters is shown. The method 500 includes: 510, obtaining a reordered multidimensional array using reordering, wherein a given dimension of a given multidimensional array of neural network parameters is reordered to a first dimension in the reordered multidimensional array, and 520, encoding the reordered multidimensional array.

[0159] In the following, different embodiments and aspects of the invention will be described in the sections “Recommendation - Tensor dimension reordering by tensor dimension shifting”, “Tensor dimension reordering method for neural network encoding” and “Proposed draft update of planned international standard ISO / IEC15938-17” and their corresponding subsections.

[0160] Furthermore, further embodiments are defined by the appended claims.

[0161] It should be noted that any embodiment defined in the claims may optionally be supplemented by any of the details (features, functions and details) described in the above sections and / or subsections.

[0162] In addition, the embodiments described in the above sections and / or subsections may be used alone or supplemented by any features in other sections or subsections or any features included in the claims.

[0163] In addition, it should be noted that the various aspects described herein can be used alone or in combination. Therefore, details can be added to each of the individual aspects described without adding details to another aspect described.

[0164] It should also be noted that the present disclosure explicitly or implicitly describes features that can be used in a neural network encoder (an encoder for providing an encoded representation of neural network parameters) and a neural network decoder (a decoder for providing decoded parameters of the neural network parameters based on the encoded representation). Thus, any feature described herein can be used in the context of a neural network encoder and in the context of a neural network decoder.

[0165] In addition, the features and functions related to the method disclosed herein may also be used in a device (configured to perform such functions). In addition, any characteristics and functions related to the device disclosed herein may also be used in a corresponding method. In other words, the method disclosed herein may be optionally supplemented with any features and functions described for the device.

[0166] Furthermore, any of the features and functions described herein may be implemented in hardware or software, or using a combination of hardware and software, as will be described in the “Implementation Alternatives” section.

[0167] Recommendation - Reordering tensor dimensions by shifting them

[0168] Embodiments related to "[NNC] tensor dimension reordering via tensor dimension shifting" are discussed below.

[0169] 1. Summary (example, details are optional)

[0170] According to one aspect, the paper proposed a method for reordering tensor dimensions, for example to solve problems associated with the use of NNR_PT_BLOCK. For example, at the encoder, the tensor dimensions are reordered, while at the decoder, the operation is reversed to restore the original dimensional order. For example, for the standard-related part (i.e., at the decoder), after decoding, the tensor dimensions are reordered in the decoded weight tensor by shifting the first dimension to its original position. For example, this solves two problems. First, for example, the compressed data unit type NNR_PT_BLOCK derives dimensions such as bias and / or batch normalization and / or local scaling parameters from the first dimension of the weight tensor. For example, the reordering of tensor dimensions after decoding enables the encoder to process tensors with arbitrary dimensional order. Secondly, the order of tensor dimensions may not be the optimal order for efficient encoding.

[0171] 2. Problem Description

[0172] 2.1 Question 1: Derivation of tensor dimension of NNR_PT_BLOCK

[0173] For example, the current working draft [1] allows encoding multiple floating-point parameter tensors as blocks in a single NDU (e.g., nnr_compressed_data_unit_payload_type == NNR_PT_BLOCK). For example, a block may include a (optionally decomposed) weight tensor and optional local scaling parameters and / or offsets and / or batch normalization parameters. For example, the dimensions of the tensors encoded in the NDU are related to the weight tensor. For example, the dimensions of the other parameter tensors are derived as follows. For example, each tensor is one-dimensional and its length is equal to the first dimension of the weight tensor. For example, it is assumed here that the first dimension of the weight tensor corresponds to the number of output channels (in the case of a fully connected layer) or the number of filters (in the case of a convolutional layer). Therefore, when the dimensions of the weight tensor to be encoded are arranged in another order, the derivation of the tensor dimensions of other parameters in the block may fail, resulting in, for example, an undecodable bitstream.

[0174] 2.2 Problem 2: Low processing efficiency

[0175] For example, for the encoding process, each tensor to be compressed is (optionally) converted to a 2D matrix having, for example, a first dimension equal to the first dimension of the tensor, and a second dimension equal to the product of all other tensor dimensions. Depending on the first dimension of the tensor, this may result in a rather "slim" matrix, for example, if the first dimension represents the width or height of a 3x3 filter kernel. This may, for example, be suboptimal for subsequent encoding stages, leading to reduced encoding efficiency or inefficient processing, in particular for block scan orders (scan_order>0), where each tensor is decomposed into blocks (e.g. 4x4, 8x8, etc.).

[0176] 3. Reordering of tensor dimensions by tensor dimension shifting

[0177] In order to solve one or more problems described in Section 2, according to one aspect of the present invention, this paper proposes a method for reordering tensor dimensions by shifting a single tensor dimension to another position. To this end, the encoder reorders the tensor, for example, the tensor dimension representing the number of output channels is the first dimension after reordering. For example, from the perspective of the decoder, the decoded tensor is reordered by shifting the first dimension of the tensor to another position indicated by a single value. Figure 6 An example illustrates this concept. Figure 6A block diagram of the tensor dimension reordering concept of an encoding-decoding pipeline according to an embodiment of the present invention is shown. For example, each box corresponds to a tensor / matrix representation whose dimensions are given in square brackets. For example, the black solid arrows / lines represent the general processing flow, and the dashed lines / arrows represent the processing flow of the syntax element first_tensor_dimension_shift. It should be noted that the use of a two-dimensional matrix is ​​optional.

[0178] like Figure 6 As shown, the input tensor 610 can be a four-dimensional tensor, having D0 elements in the first dimension, D1 elements in the second dimension, D2 elements in the third dimension, and D3 elements in the third dimension. For example, the elements can be neural network parameters. As shown, for example, using Figure 3 The encoder 300 shown, such as its reordering unit 320, can reorder 612 the input tensor 610 (which can be an example of the multidimensional array 302) to obtain a reordered tensor 620 (which can be an example of the multidimensional array 620). As shown, a given dimension (D2 in this example) can be reordered to the first dimension in the reordered tensor 620. Therefore, as an optional feature, first_tensor_dimension_shift 630 is used (which can be an example of a dimension shift value).

[0179] As previously described, such a dimension shift value may describe which given dimension of a given multidimensional array has been reordered to the first dimension in the reordered multidimensional array, or describe how many dimensions the first dimension of the encoded reordered multidimensional array should be shifted by when performing a decoder-side reordering. In addition, the dimension shift value may describe a new position to which the first dimension of the encoded reordered multidimensional array should be shifted in the decoder. In addition, the corresponding encoder (e.g., 300) may be configured to optionally encode the dimension shift value into the bitstream 650 (which may be an example of the bitstream 102, 202, and / or 303) using an ExpGolomb code. In particular, the dimension shift value may optionally be a single scalar dimension shift value as the only parameter of the reordering 612. As Figure 6 As shown, the reordering 612 may include a single shift of a single dimension D2 to another position (ie, the first position) to obtain a reordered multidimensional array in the form of a reordered tensor 620 .

[0180] Based on the reordered tensor 620, as an optional feature, a 2D matrix 640 or a 2D matrix interpretation 640 may be optionally obtained using a 2D transformation 622 or a 2D interpretation 622. As shown, the first dimension of the two-dimensional matrix 640 may be determined by the first dimension of the reordered multidimensional array 620, and thus the second dimension of the two-dimensional matrix 640 is determined by the product of the further dimension values ​​of the reordered multidimensional array 620 (in the form of the reordered tensor 620).

[0181] As shown, the two-dimensional matrix 640 may be encoded 642 in the bitstream 650. It should be noted that the two-dimensional transformation 622 or the two-dimensional interpretation 622 is only optional, and thus, the reordered tensor 620 may be encoded directly as an option.

[0182] In any case, optionally, the corresponding encoder (e.g., 300) can encode the neural network parameters using context-based entropy coding.

[0183] The decoder (such as decoder 100, 200) receives a bitstream 650, which, as shown, optionally includes dimension shift value information 630. For example, the bitstream is decoded 652 using the decoding unit 110, 210 to obtain a decoded 2D matrix 660 or a decoded 2D matrix interpretation 660, and the information about the dimension shift 630 is obtained using decoding 634. Therefore, in other words, a two-dimensional matrix 660 is obtained based on the encoded representation, whose first dimension is determined by the first dimension value and whose second dimension is determined by the product of a plurality of further dimension values.

[0184] Based on this and the tensor transformation 662 or the tensor interpretation 662, the decoder obtains a reconstructed tensor 670 (which may be an example of the first multidimensional array 111, 211), which corresponds to the reordered tensor 620, for example. In short, based on the two-dimensional matrix 660, the original multiple dimensions are reconstructed, wherein the first dimension of the matrix 660 corresponds to the first dimension of the tensor 670. Afterwards, the reconstructed tensor 670 is reordered 672, for example using the reordering unit 120, 220, to obtain an input tensor 680, which corresponds to or is optionally equal to the input tensor 610. Therefore, as an example, the first_tensor_dimension_shift 630 (for example, the information 112, 212) is used.

[0185] 3.1 Suggested syntax (example, details are optional)

[0186] To this end, for example, when count_tensor_dimensions is greater than 1, a new syntax element first_tensor_dimension_shift (e.g., indicating a new position of the first dimension) is decoded from the bitstream. For example, the value of first_tensor_dimension_shift is non-negative and less than the value of count_tensor_dimensions. For example, a value equal to zero corresponds to no reordering. For example, if first_tensor_dimension_shift does not exist, it can be inferred to be zero.

[0187] 3.2 Reordering Method - Example

[0188] For example, the working draft text attached to this document (the proposed updated draft of the planned international standard ISO / IEC15938-17) provides syntax and reordering methods. For simplicity, two examples using 2D tensors and 4D tensors are used to illustrate the reordering method.

[0189] 3.2.1 Example: 2D-Tensor, first_tensor_dimension_shift == 1

[0190] For example, assuming that a 2D tensor A with tensor dimensions [D1, D0] and first_tensor_dimension_shift equal to 1 has been decoded from the bitstream, and elements of A can be accessed through A[m][n] (with m∈[0,D1-1] and n∈[0,D0-1]), the reordering process is as follows:

[0191] ● For example, initialize the reordered tensor B with dimensions [D0,D1]

[0192] ●For example, set the value of B according to the following:

[0193]

[0194] For example, for a 2D tensor, the reordered tensor B is equal to the transposed tensor A.

[0195] 3.2.2 Example: 4D-Tensor, first_tensor_dimension_shift == 2

[0196] For example, assuming that a 4D-tensor A with tensor dimensions [D2, D1, D0, D3] and first_tensor_dimension_shift equal to 2 has been decoded from the bitstream, and the elements of A can be accessed by A[m][n][o][p] (using A[m][n][o][p], n∈[0, D1-1], o∈[0, D0-1], p∈[0, D3-1]), the reordering process is as follows:

[0197] ●For example, initialize the reordered tensor B to dimensions [D0, D1, D2, D3].

[0198] ●For example, set the value of B according to the following:

[0199]

[0200] 4 Recommendations

[0201] According to one aspect of the invention, we propose to adopt the proposed technique in a working draft.

[0202] 5 References for the section "Recommendation - Reordering Tensor Dimensions by Shifting Tensor Dimensions"

[0203] [1]MPEG, "Working Draft 3on Incremental Compression of neural networks", Document of ISO / IEC JTC1 / SC29 / WG04,WG04N0178,OnLine,Jan.2022

[0204] Tensor dimension reordering method for encoding neural networks

[0205] One aspect of the present invention relates to a method for reordering the dimensions of a (N-dimensional) neural network parameter tensor, for example, in order to enable encoding and / or efficient processing of multiple tensors in a block in a neural network compression framework (e.g., the MPEG-7 Part 17 standard for multimedia content description and analysis [2]). Such a framework (e.g., MPEG-7 Part 17) may provide a variety of compression tools, including quantization and / or lossless encoding and lossless decoding methods. Typically (but not necessarily), for efficient processing, the internal interpretation of the tensor is equivalent to a one-dimensional (1D) or two-dimensional (2D) representation. The shape of this 1D / 2D representation is determined by the original tensor dimensions and their order. In addition, this shape may also affect the efficiency of the compression and processing pipeline.

[0206] Therefore, reordering of tensor dimensions, such as described in the present invention, can improve efficiency.

[0207] According to one aspect, the present invention is primarily directed to lossy and lossless coding of neural network parameter layers in neural network compression, but may also be selectively applied to other lossy and lossless coding fields.

[0208] For example, the method of the apparatus (or system) can be divided into different major parts, including the following:

[0209] 1. Quantification

[0210] 2. Lossless Coding

[0211] 3. Lossless decoding

[0212] However, it is not necessary to have all of these major parts.

[0213] In order to understand the main advantages of the present invention, we will first briefly introduce the subject of neural networks and related methods of parameter encoding.

[0214] 1 Application Area (all details are optional)

[0215] For example, the most basic form of neural networks consists of a chain of affine transformations followed by element-wise nonlinear functions. For example, they can be represented as a directed acyclic graph, as shown in the figure below. For example, each node contains a specific value, which is propagated forward to the next node by multiplying it with the respective weight value on the edge. For example, all incoming values ​​are then simply summed up.

[0216] Figure 7 A schematic diagram of a 2-layer feedforward neural network according to an embodiment of the present invention is shown. Figure 7 A diagram of an example of a feedforward neural network is shown. Specifically, this 2-layer neural network 700 is a nonlinear function that maps a 4-dimensional input vector 710 onto a real line. Therefore, the network 700 includes an input layer 720, a hidden layer 730, and an output layer 740.

[0217] For example, mathematically, the above neural network would compute the output in the following way:

[0218] Output = σ(W2·σ(W1·Input))

[0219] where, for example, W2 and W1 are neural network weight parameters (edge ​​weights) and sigma is a nonlinear function. For example, so-called convolutional layers can also be used, which are matrix-matrix products, as described in [1]. For example, incremental updates usually aim to provide updates to the weights of W1 and W2, for example, which can be the result of additional training processes. For example, the updated versions of W2 and W1 usually lead to modified outputs. From now on, we will call the process of computing the output based on a given input as inference. In addition, we call intermediate results hidden layers or hidden activations, for example, they constitute a linear transformation + element-wise nonlinearity, for example, as the calculation of the first dot product + nonlinearity mentioned above.

[0220] For example, neural networks often have millions of parameters and thus may require hundreds of MB to represent. Therefore, their inference process, for example, can require a lot of computational resources because it involves many dot product operations between large matrices. Therefore, it is very important to reduce the complexity of performing these dot products.

[0221] 2 Traditional Solutions

[0222] It should be noted that any feature functions and details in this section “conventional solutions” can be used alone or in combination in embodiments according to the present invention.

[0223] Furthermore, it should be noted that parameters (e.g., parameters of a neural network, such as weights of a neural network) are typically (but not necessarily) N-dimensional tensors (or equivalently N-dimensional arrays).

[0224] 2.1 Tensor Representation of Neural Networks (All Details Are Optional)

[0225] For example, the parameters of a neural network are often (but not necessarily) represented by N-dimensional tensors, where the dimension N depends on the model architecture and application. For example, in the case where N is equal to 1 and N is equal to 2, they correspond to vectors and matrices, respectively. For example, the size of each dimension of the tensor can be represented by an array of length N as follows [D0, D1, ..., D N-1 ]. For example, for processing in encoders and decoders of the MPEG-7 Part 17 standard, all tensors with N greater than or equal to 3 are (preferably but not necessarily) interpreted as tensors with dimensions equal to [D0, (D1·D2·…·D N-1 )]. If not available at the decoder, the standard specifies methods to transmit the tensor dimensions in the bitstream, e.g. so that the original tensor shape can be reconstructed after decoding. For the methods described below, it can be assumed, for example, that the weight parameters are represented or interpreted as vectors or 2D matrices, respectively.

[0226] 2.2 Related methods of quantization and entropy coding (all details are optional)

[0227] For example, the standard for neural network compression MPEG-7 Part 17 for multimedia content description and analysis [2] provides different neural network parameter quantization methods, such as independent scalar quantization and dependent scalar quantization (DQ, also known as trellis-coded quantization TCQ, TCQ). In addition, it also specifies an entropy quantization scheme, also known as deepCABAC [7]. For a better understanding, a brief overview is now given, which can optionally be used in embodiments according to the present invention. Details can be found, for example, in [2].

[0228] 2.2.1 Scalar Quantizer (Example)

[0229] For example, a scalar quantizer can be used to quantize the neural network parameters. For example, the result of the quantization is a set of acceptable values ​​that is reduced to the parameters. In other words, for example, the neural network parameters are mapped to a countable set (actually a finite set) of so-called reconstruction levels. For example, the reconstruction level set represents a proper subset of the set of possible neural network parameter values. For example, in order to simplify the following entropy coding, a quantization index can be used to represent the acceptable reconstruction level, and the quantization index is transmitted as part of the bitstream. On the decoder side, the quantization index is mapped to the reconstructed neural network parameters. For example, the possible values ​​of the reconstructed neural network parameters correspond to the reconstruction level set. For example, on the encoder side, the result of the scalar quantization is a set of (integer) quantization indices.

[0230] For example, a uniform reconstruction quantizer (URQ) is used in this application. Figure 8 Its basic design is illustrated. Figure 8 A schematic diagram of a uniform reconstruction quantizer according to an embodiment of the present invention is shown. For example, URQ has the property that the reconstruction levels are equidistant. The distance Δ between two adjacent reconstruction levels is called the quantization step size. For example, one of the reconstruction levels is equal to 0. Therefore, the complete set of available reconstruction levels is uniquely specified by the quantization step size Δ. For example, the decoder mapping from the quantization index q to the reconstruction weight parameter t' is in principle given by the following simple formula

[0231] t′=q·Δ.

[0232] In this context, the term "independent scalar quantization" means, for example, that given a quantization index q of any weight parameter, the associated reconstruction weight parameter t' can be determined independently of all quantization indices of other weight parameters.

[0233] 2.2.2 Dependency Scalar Quantization (Example, all details are optional)

[0234] In dependent scalar quantization (DQ), the allowable reconstruction level of a neural network parameter depends on the quantization index selection of the previous neural network parameter in the reconstruction order. The concept of dependent scalar quantization is combined with improved entropy coding, where the probability model selection (or codeword table selection) of the neural network parameters depends on the set of allowable reconstruction levels. The advantage of neural network parameter dependent quantization is that the allowable reconstruction vectors are more densely packed in the N-dimensional signal space (where N represents the number of samples or neural network parameters in the sample set to be processed, such as one layer). The reconstruction vector of a neural network parameter set refers to the ordered reconstructed neural network parameters (or alternatively, the ordered reconstructed samples) of the neural network parameter set. Figure 9 illustrates the effect of dependent scalar quantization using the simplest two neural network parameters as an example. Figure 9 shows a schematic illustration of the position of the allowable reconstruction vector for the simple case of two weight parameters according to an embodiment: (a) independent scalar quantization; (b) dependent scalar quantization. Figure 9a The allowable reconstruction vectors for independent scalar quantization are shown (represented as 910 on the 2d plane). It can be seen that the set of allowable values ​​for the second neural network parameter t'1 is independent of the selected values ​​for the first reconstruction neural network parameter t'0. Figure 9b An example of dependent scalar quantization is shown. Note that in contrast to independent scalar quantization, the selectable reconstruction value of the second neural network parameter t′1 depends on the reconstruction level selected for the first neural network parameter t′0. Figure 9b In the example of , there are two different sets of available reconstruction levels for the second neural network parameter t′1 (indicated by different colors, e.g., set 920 and set 930). If the quantization index of the first neural network parameter t′0 is an even number (…, -2, 0, 2, …), the second neural network parameter t′1 can select any reconstruction level of the first set (blue dots, 920). If the quantization index of the first neural network parameter t′0 is an odd number (…, -3, -1, 1, 3, …), the second neural network parameter t′1 can select any reconstruction level of the second set (red dots, 930). In this example, the reconstruction levels of the first set and the second set are shifted by half the quantization step size (any reconstruction level of the second set is between two reconstruction levels of the first set).

[0235] The effect of dependent scalar quantization of neural network parameters is that the expected value of the distance between a given input vector of neural network parameters and the nearest available reconstruction vector is reduced for a given average number of reconstruction vectors per N-dimensional unit volume. Therefore, for a given average number of bits, the average distortion between the input vector of neural network parameters and the reconstructed neural network parameter vector can be reduced. In vector quantization, this effect is called space filling gain. Using dependent scalar quantization for a set of neural network parameters, most of the potential space filling gain of high-dimensional vector quantization can be utilized. Moreover, compared to vector quantization, the implementation complexity of the reconstruction process (or decoding process) is comparable to the complexity of encoding the dependent neural network parameters using independent scalar quantizers.

[0236] Due to the factors mentioned above, DQ usually achieves the same distortion level at lower bit rates.

[0237] 2.2.3 DQ in MPEG-7 Part 17 (example, all details are optional)

[0238] The MPEG-7 Part 17 standard for neural network compression for multimedia content description and analysis uses two quantizers Q1 and Q2 with different reconstruction level sets. Both sets contain integer multiples of the quantization step size Δ. Q1 contains all even multiples of the quantization step size and 0, and Q2 contains all odd multiples of the quantization step size and 0. Fig.10 This splitting of the reconstruction set is shown. Fig.10 An illustrative example of splitting a set of reconstruction levels into two subsets according to an embodiment is shown. The two subsets of quantization set 0 are labeled "A" and "B", and the two subsets of quantization set 1 are labeled "C" and "D".

[0239] The process of switching between sets determines the quantizer to be applied based on the quantization index selected for the previous neural network parameters in the reconstruction order, or more precisely, based on the parity of the previously encoded quantization index. The switching process is implemented by a finite state machine with 8 states (as shown in Table 16), where each state is associated with one of the quantizers Q1 or Q2.

[0240] Table 16: Preferred example of a state transition table for a configuration with 8 states.

[0241]

[0242] Using the concept of state transition, the current state and the current quantization set are uniquely determined by the previous state (in reconstruction order) and the previous quantization index.

[0243] 2.2.4 Entropy Coding (Example, all details are optional)

[0244] As a result of the quantization applied in the previous step, the weight parameters are mapped to a finite set of so-called reconstruction levels. These can be represented by a (integer) quantizer index (also called parameter level or weight level) and a quantization step size, which can be fixed, for example, for the entire layer. In order to restore all quantized weight parameters of a layer, the decoder can know the step size and the dimension of the layer. For example, they can be transmitted separately.

[0245] In this section, scanning may be described. For example, a block coding (NNR_PT_BLOCK) portion may be described herein.

[0246] 2.2.4.1 Quantized Index Scan (Example, all details are optional)

[0247] The quantization index (integer representation) is then transmitted using entropy coding techniques. Therefore, a layer of weights is mapped to a sequence of quantized weight levels using scanning. To this end, five different scanning orders are specified in the standard, represented by the syntax element scan_order. The first (scan_order==0) represents a row scan, starting from the top row of the matrix and encoding the contained values ​​from left to right. In this way, all rows are encoded from top to bottom. All other scanning orders (scan_order>0) correspond to block scanning orders. Here, the matrix is ​​decomposed into blocks of size 4x4 (scan_order==1), 8x8 (scan_order==2), 16x16 (scan_order==3) or 32x32 (scan_order==4), and these blocks are processed in block row order. That is, starting from the top data block row, the data block is processed from left to right. Processing a block means scanning its values ​​row by row (scanning rows from top to bottom and scanning each row from left to right).

[0248] 2.2.4.2 Quantization Index Coding Using Context Adaptive Binary Arithmetic Coding (CABAC) (Example, all details are optional)

[0249] For level coding, CABAC (Context-Adaptive Binary Arithmetic Coding) is used. For details, please refer to [2]. Therefore, the quantized weight level q is decomposed into a series of binary symbols or syntax elements, which can then be handed over to the binary arithmetic coder (CABAC).

[0250] In the first step, a binary syntax element sig_flag is derived for the quantized weight level to specify whether the corresponding weight level is equal to zero. If sig_flag is equal to 1, another binary syntax element sign_flag is derived. The binary (bin) indicates whether the current weight level is positive (e.g., bin = 0) or negative (e.g., bin = 1).

[0251] Next, the binary unary sequence is encoded, followed by a fixed-length sequence, as follows:

[0252] The variable k is initialized to a non-negative integer, and X is initialized to 1 << k.

[0253] One or more syntax elements abs_level_greater_X are encoded, indicating that the absolute value of the quantized weight level is greater than X. If abs_level_greater_X is equal to 1, the variable k is updated (e.g., incremented by 1), then 1 << k is added to X, and abs_level_greater_X is further encoded. This process continues until abs_level_greater_X is equal to 0. Then, a fixed-length code of length k is sufficient to complete the encoding of the quantizer index. For example, the variable rem = X - |q| can be encoded using k bits. Alternatively, the variable rem' can be defined as |rem' = (1 << k) - rem - 1 rem' = (1 << k) - rem - 1 and encoded using k bits. Alternatively, any other mapping of the variable rem to a fixed-length code of k bits can be used.

[0254] When k is incremented by 1 after each abs_level_greater_X, this method is equivalent to applying exponential Golomb coding (if sign_flag is not considered).

[0255] Furthermore, if the maximum absolute value abs_max is known at both the encoder and decoder sides, when X >= abs_max for the next abs_level_greater_X to be transmitted, then the encoding of the abs_level_greater_X syntax element can be terminated.

[0256] 2.2.4.3 Quantization Index Decoding Using Context-Adaptive Binary Arithmetic Coding (CABAC) (Example, all details are optional)

[0257] The decoding of the quantized weight levels (integer representation) works similarly to the encoding. The decoder first decodes sig_flag. If it is equal to 1, it is followed by a unary sequence of sign_flag and abs_level_great_x, where the update of k (and thus the increment of X) must follow the same rules as in the encoder. Finally, the k-bit fixed-length code is decoded and interpreted as an integer (e.g. rem or rem′ depending on which of the two is encoded). The absolute value of the decoded quantized weight level |q| is then reconstructed from X and forms the fixed-length part. For example, using rem as the fixed-length part, |q|=X-rem. Or alternatively, if rem′ is encoded, then |q|=X+1+rem′-(1<<k). As the last step, the sign needs to be applied to |q| according to the decoded sign_flag to obtain the quantized weight level q. Finally, the quantized weight w is reconstructed by multiplying the quantized weight level q with the step size Δ.

[0258] In the implementation, k is initialized to 0 and updated as follows. Whenever abs_level_greater_X is equal to 1, the necessary update of k is performed according to the following rule: if X>X', k is incremented by 1, where X' is a constant that depends on the application. For example, X' is a number (e.g., between 0 and 100) derived by the encoder and signaled to the decoder.

[0259] 2.2.4.4 Context Modeling (Example, all details are optional)

[0260] In CABAC entropy coding, most syntax elements at the quantization weight level are coded using binary probability modeling. Each binary decision (bin) is associated with a context. The context represents a probability model for a class of coded bins. For each context, the probability of one of the two possible binary values ​​is estimated based on the value of the bin that has been coded with the corresponding context. Depending on the application, different context modeling methods can be used. Typically, for several bins associated with quantization weight coding, the context for coding is selected based on the syntax elements that have already been transmitted. Depending on the actual application, different probability estimators can be selected, such as SBMP[4], HEVC[5] or VTM-4.0[6]. This choice affects compression efficiency and complexity, among other things.

[0261] A context modeling scheme applicable to various neural networks is described below. To decode the quantized weight level q at a specific position (x, y) in the weight matrix (layer), a local template is applied to the current position. The template contains many other (ordered) positions, such as (x-1, y), (x, y-1), (x-1, y-1), etc. For each position, a state identifier is derived.

[0262] In one implementation variant (denoted as Sil), the state identifier s at position (x, y) x，y The derivation is as follows: If the point at position (x, y) is outside the matrix, or the quantization weight level q at position (x, y) x，y has not been decoded or is equal to zero, then the state identifier s x，y = 0. Otherwise, the state identifier is s x，y =q x,y <0?1:2.

[0263] For a particular template, a sequence of state identifiers is derived, with each possible combination of state identifier values ​​mapped to a context index, identifying the context to be used. The templates and mappings may be different for different syntax elements. For example, from a template containing the (ordered) positions (x-1, y), (x, y-1), (x-1, y-1), an ordered sequence of state identifiers s may be derived. x-1，y ,s x，y-1 ,s x-1，y-1 For example, the sequence can be mapped to a context index C=s x-1，y +3*s x，y-1 +9*s x-1，y-1 . For example, context index C can be used to identify multiple contexts of sig_flag.

[0264] In one embodiment variant (denoted as Method 1), the quantized weight level 1 q at position (x, y) x，y The local template of sig_flag or sign_flag contains only one position (x-1, y) (i.e. the left neighbor). The associated state identifier s x-1y It is derived from the implementation variant Sil.

[0265] For sig_flag, according to s x-1，y The value of s is selected from one of three contexts, or, for sign_flag, according to s x-1，y The value of selects one from the other three contexts.

[0266] In another embodiment variant (denoted as method 2), the local template of the sig flag contains three ordered positions (x-1, y), (x-2, y), (x-3, y). The associated state identifier sequence s x-1，y ,s x-2，y ,s x-3，y Derived according to embodiment variant Si2.

[0267] For sig_flag, the context index C is derived as follows:

[0268] if s x-1，y ≠0, then C=0. Otherwise, if s x-2，y ≠0 Then C=1.

[0269] Otherwise, if s x-3，y ≠0 Then C = 2. Otherwise, C = 3.

[0270] This can also be expressed as the following formula:

[0271] C=(s x-1，y ≠0)? 0:((s x-2，y ≠0)? 1:((s x-3，y ≠0)? 2:3))

[0272] In the same way, the number of neighbors to the left can be increased or decreased so that the context index C is equal to the distance to the next non-zero weight to the left (not exceeding the template size).

[0273] For example, each abs_level_greater_X flag can apply its own set of two contexts. One of the two contexts is then selected based on the value of sign_flag.

[0274] In an implementation variant, for the abs_level_great_X flag where X is less than a predefined number X', different contexts are distinguished depending on the value of X and / or sign_flag.

[0275] In one implementation variant, for the abs_level_greater_X flag where X is greater than or equal to a predefined number X′, different contexts are differentiated only according to X.

[0276] In another implementation variant, the abs_level_great_X flag, where X is greater than or equal to a predefined number X', is encoded using a fixed code length of 1 (eg using the bypass mode of the arithmetic encoder).

[0277] In addition, the coding of some or all of the syntax elements can also be context - independent. Instead, their coding length is fixed at 1 bit. For example, the so - called CABAC bypass binary (bin) is used.

[0278] In another implementation variant, a bypass mode is used to code the fixed - length remainder rem.

[0279] In another implementation variant, the encoder determines a predefined number X’, differentiates two contexts for each syntax element abs_level_greater_X with X < X’ according to the symbol, and uses one context for each abs_level_greater_X with X < X’.

[0280] 2.2.4.5 Context Modeling for Dependent Scalar Quantization (Example, all details are optional)

[0281] The main aspect of dependent scalar quantization is that the neural network parameters have different sets of admissible reconstruction levels (also called quantization sets). The quantization set of the current neural network parameters is determined based on the value of the quantization index of the previous neural network parameters. If we consider Fig.10 the preferred example in [reference] and compare the two quantization sets, it is obvious that the distance between the reconstruction level equal to 0 in set 0 and the adjacent reconstruction levels is greater than that in set 1. Therefore, if set 0 is used, the probability that the quantization index is equal to 0 is greater, and if set 1 is used, the probability is smaller. In an implementation variant, entropy coding switches the codeword table or probability model according to the quantization set (or state) used by the current quantization index, thus taking advantage of this effect.

[0282] It should be noted that when entropy - decoding the current quantization index (or the corresponding binary decision of the current quantization index), if an appropriate switch of the codeword table or probability model is to be made, the path of all previous quantization indices (associated with a subset of the quantization sets used) must be known. Therefore, it is necessary to code the neural network parameters in the reconstruction order. Thus, in an implementation variant, the coding order of the neural network parameters is equal to their reconstruction order. In addition, any coding / reconstruction order of the quantization index is possible, such as the order specified in Section 2.2.4.2, or any other uniquely - defined order.

[0283] At least a part of the binary of the absolute level is usually coded using an adaptive probability model (also called context). In an implementation variant, the probability model of one or more binaries is selected based on the quantization set of the corresponding neural network parameters (or more generally, the corresponding state variable). The selected probability model can depend on multiple parameters or attributes of the transmitted quantization index, but one of the parameters is the quantization set or state applicable to the quantization index being coded.

[0284] In another embodiment variant, the syntax for transmitting a layer quantization index comprises a binary for specifying whether the quantization index is equal to zero or not equal to 0. The probability model used to encode the binary is selected from a set of two or more probability models. The selection of the probability model used depends on the quantization set (i.e. the reconstruction level set) applicable to the corresponding quantization index. In another embodiment variant, the probability model used depends on the current state variable (the state variable means the quantization set used).

[0285] In a further embodiment variant, the syntax for the transport layer quantization index comprises a binary specifying whether the quantization index is greater than zero or less than zero. In other words, the binary represents the sign of the quantization index. The choice of the used probability model depends on the quantization set (i.e. the reconstruction level set) applicable to the corresponding quantization index. In another embodiment variant, the used probability model depends on the current state variable (the state variable means the used quantization set).

[0286] In a further implementation variant, the syntax for transmitting the quantization index includes a binary for specifying whether the absolute value of the quantization index (neural network parameter level) is greater than X (see Section 2.2.4.2 for details). The probability model used to encode this binary is selected from a set of two or more probability models. The choice of the probability model used depends on the quantization set (i.e. the set of reconstruction levels) applicable to the corresponding quantization index. In another implementation variant, the probability model used depends on the current state variable (the state variable means the quantization set used).

[0287] One aspect is dependent quantization of neural network parameters combined with entropy coding, where the choice of one or more binary probability models of the binary representation of the quantization index (also called quantization level) depends on the quantization set (the set of admissible reconstruction levels) or the corresponding state variable of the current quantization index. The quantization set (or state variable) is given by the quantization index (or a subset of the binary representation of the quantization index) of the previous neural network parameters in the encoding and reconstruction order.

[0288] In an implementation variant, the selection of the probability model is combined with one or more of the following entropy coding aspects:

[0289] The absolute value of the quantization index is transmitted using a binarization scheme consisting of a number of bins coded using an adaptive probability model, and if the adaptively coded bins do not yet fully specify the absolute value, the suffix part is coded using the bypass mode of the arithmetic coding engine (a non-adaptive probability model using pmf (0.5, 0.5) for all bins). In an implementation variant, the binarization of the suffix part depends on the value of the quantization index that has been transmitted.

[0290] ●The binarization of the absolute value of the quantization index includes an adaptively coded binary that specifies whether the quantization index is not equal to 0. The probability model used to encode this binary (called context) is selected from a set of candidate probability models. The selected candidate probability model is determined not only by the quantization set (admissible reconstruction level set) or state variables of the current quantization index, but also by the quantization index transmitted for this layer. In an implementation variant, the quantization set (or state variable) determines a subset of available probability models (also called context set), and the value of the coded quantization index determines the probability model used in this subset (context set).

[0291] In an implementation variant, the probability model used in the context set is determined based on the values ​​of the quantization indices encoded in the local neighborhood of the current neural network parameters. Listed below are some example measures that can be derived based on the values ​​of the quantization indices in the local neighborhood and then used to select the probability model of the predetermined context set:

[0292] ○The sign of the quantization index within the local neighborhood is not equal to 0.

[0293] ○ The number of quantization indices in the local neighborhood that are not equal to 0. This number may be clipped to a maximum value.

[0294] ○ The sum of the absolute values ​​of the quantization indices in the local neighborhood. This number may be clipped to a maximum value.

[0295] ○ The difference between the sum of the absolute values ​​of the quantization indices in the local neighborhood and the number of quantization indices in the local neighborhood that are not equal to 0. This number may be clipped to a maximum value.

[0296] The binarization of the absolute value of the quantization index includes an adaptively coded binary that specifies whether the absolute value of the quantization index is greater than X. The probability model used to encode these bins (called context) is selected from a set of candidate probability models. The selected probability model is determined not only by the quantization set (the set of admissible reconstruction levels) or state variables of the current quantization index, but also by the quantization index transmitted for this layer. In an implementation variant, the quantization set and (or state variables) determine a subset of available probability models (also called a context set), and the data of the encoded quantization index determines the probability model used in this subset (context set). In order to select a probability model, any of the above methods (for specifying a binary that is not equal to 0) can be used.

[0297] 2.3 Related methods for compressing incremental updates of neural networks (examples, all details are optional)

[0298] This section describes a method for encoding incremental updates of neural networks, where the reconstructed network layers are composed of an existing base layer (the base layer of the base model) and one or more incremental update layers, which can be encoded and transmitted separately.

[0299] 2.3.1 Concept of base model and update model (example, all details are optional)

[0300] The concept introduces the neural network model according to Section 1, which can be considered a complete model in the sense that the output can be calculated given the input. The model is denoted as the base model N B Each base model consists of layers, represented by the base layer L B1 , L B2 , ..., L BJ The base layer contains the base values, which can be chosen, for example, to be efficiently represented or compressed / transmitted in the first step. In addition, the concept introduces an update model (N U1 , N U2 , ..., N UK ), these models may have similar or even identical architectures to the base model. For example, the update model may not be a complete model in the sense described above. Instead, it may be combined with the base model through a combination method to form a new complete model N B1 The model itself can be used as a base model for further updating models. Update model N Uk It consists of layers, denoted as update layer L Uk,1 , L Uk,2 , ..., L Uk，J The update layer contains the underlying values, which can be selected, for example, to be efficiently represented or compressed / transmitted separately.

[0301] The update model can be the result of applying an (additional) training process to the base model on the encoder side. Depending on the type of update provided by the update model, several combination methods can be applied. Please note that the methods described in this invention are not limited to any specific type of update / combination method, but are applicable to any architecture using the base model / update model method.

[0302] In a preferred embodiment, the kth update model N Uk Contains layer L with differential values Uk,j (also called incremental updates), these layers are added to the corresponding layers L of the base model Bj In order to form a new model layer L according to the following Nk,j :

[0303] L Nkj =L Bi +L Uk,j , for all j

[0304] The new model layers form the (updated) new model, which is then transferred separately as the base model for the next incremental update.

[0305] In a further preferred embodiment, the kth update model comprises a layer L with a scaling factor value Uk，j , these layers are multiplied by the corresponding base layer L Bj value, a new model L is formed according to the following Nk,j :

[0306] L Nk，j =L Bj ·L Uk,j , for all j

[0307] The new model layers form the (updated) new model, which is then transferred separately as the base model for the next incremental update.

[0308] Note that in some cases, the updated model can also contain new layers that replace one or more existing layers (i.e. for all layers k:L Nk,j =L Uk,j , for all j, instead of updating the layer as above.

[0309] 2.3.2 Neural Network Parameter Encoding for Incremental Updates (Example, all details are optional)

[0310] During the entropy coding phase, the concept of a base model and one or more incremental updates can be exploited to improve coding efficiency. The parameters of a layer are typically represented as multidimensional tensors. For the encoding process, all tensors are typically mapped to a 2D matrix, entities such as rows and columns. This 2D matrix is ​​then scanned according to a predefined order and the parameters are encoded / transmitted. Note that the methods described below are not limited to 2D matrices. These methods are applicable to all representations of neural network parameters that provide known size parameter entities (such as rows, columns, blocks, etc. and / or their combinations). In order to better understand these methods, a 2D matrix representation will be used below.

[0311] In a preferred embodiment, the parameters of a layer are represented as a 2D matrix providing numerical entities as rows and columns.

[0312] 2.3.2.1 Row or channel skip mode (example, all details are optional)

[0313] Typically, the magnitude of the values ​​in the update model is smaller than in the full (base) model. There are usually a significant number of values ​​that are zero, which is further amplified by the quantization process. Therefore, the layers to be transferred may contain long sequences of zeros, meaning that some rows of the 2D matrix are completely zero.

[0314] This can be exploited by introducing a flag (skip_row_flag) for each row, which specifies whether all parameters in a row are equal to zero. If this flag is equal to 1 (skip_row_flag == 1), no other parameters of this row are encoded. On the decoder side, if this flag is equal to 1, no parameters of this row are decoded. Instead, these parameters are assumed to be 0.

[0315] A variation here is to arrange all skip_row_flags into a flag array skip_row_flag[N], where N is the number of rows. Also, in a variation, N may be signaled before the array.

[0316] Otherwise, if this flag is equal to 0, the parameters of this line are encoded and decoded normally.

[0317] Each skip_row_flag is associated with a probability model or a context model. A context model is selected from a set of context models based on previously encoded symbols (such as previously encoded parameters or skip_row_flag).

[0318] In a preferred embodiment, a single context model is applied to all skip_row_flags of a layer.

[0319] In another preferred embodiment, a context model is selected from a set of two context models based on the previously encoded value of skip_row_flag: if the previous skip_row_flag value is equal to 0, the first context model is selected; if the value is equal to 1, the second context model is selected.

[0320] In a further preferred embodiment, a context model is selected from a set of two context models based on the value of the co-located skip_row_flag in the corresponding layer of the previously encoded update or base model. If the previous skip_row_flag value is equal to 0, the first context model is selected; if the value is equal to 1, the second context model is selected.

[0321] In another preferred embodiment, such as in the previous embodiment, the given number of context models is doubled to form two sets of context models. Then, a set of context models is selected based on the value of the co-located skip_row_flag in the corresponding layer of the previously encoded specific update or base model. This means that if the previous skip_row_flag value is equal to 0, the first set is selected, and if the value is equal to 1, the second set is selected.

[0322] A further preferred embodiment is identical to the previous embodiment, but if the corresponding layer does not exist in a previously encoded specific update or base model, the first context model is selected. Therefore, if the corresponding layer exists in a previously encoded specific update or base model, the second set is selected.

[0323] Note that the specific mechanism described for skipping rows also applies to columns of two-dimensional matrices, and to generalized tensors with N parameter dimensions, where K (K < N) smaller sub-blocks or sub-rows of dimension can be skipped using the described skip_flag or skip_flag_array mechanism.

[0324] 2.3.2.2 Improved Base Model Update Contextual Modeling of Model Structure (Example, all details are optional)

[0325] The entropy coding stage can utilize the concept of a base model and one or more update models. The methods described here are applicable to any entropy coding scheme that uses a context model, such as the method described in Section 2.2.4.

[0326] Typically, independent update models (and base models) are related and can be used at both the encoder and decoder sides. This can be used in the context modeling phase to improve coding efficiency by providing new context models and context model selection methods.

[0327] In a preferred embodiment, the binarization (sig_flag, sign_flag, etc.), context modeling and encoding scheme according to Section 2.2.4.2 is adopted.

[0328] In another preferred embodiment, a given number of context models (context sets) for the symbol to be encoded are replicated to form two or more context model sets. Then, the context model set is selected based on the value of the co-location parameter in the corresponding layer of the previously encoded specific update or base model. This means that if the co-location parameter is below the first threshold T1, the first set is selected, if the value is greater than or equal to the threshold T1, the second set is selected, if the value is greater than or equal to the threshold T2, the third set is selected, and so on. This process can be applied to more or fewer thresholds.

[0329] In a preferred embodiment identical to the previous one, a single threshold value T1=0 is used.

[0330] In another preferred embodiment, a given number of context models (context sets) for a symbol to be encoded are replicated to form two or more context model sets. The context model set is then selected based on a set of values ​​consisting of co-location parameters and neighborhood values ​​(e.g., one or more spatial neighborhoods of co-location parameters) in the corresponding layer of a particular update or base model previously encoded.

[0331] In the same preferred embodiment as the previous one, if the sum of the values ​​(or absolute values) within the template is below a first threshold T1, if the sum is greater than or equal to threshold T1, the first set is selected, if the sum is greater than or equal to threshold T2, the second set is selected, etc. This process can be applied to more or fewer thresholds.

[0332] In a particularly preferred embodiment, which is the same as the previous embodiment, the template includes the co-parameter and the left neighbor of the co-parameter, and a single threshold value T1=0 is used.

[0333] In another preferred embodiment, a context model is selected from a set of context models for a syntax element based on a set of values ​​consisting of co-located parameters and neighboring values ​​(e.g. one or several spatial neighboring values ​​of the co-located parameters) in the corresponding layer of a specific update model or base model previously encoded.

[0334] In the same preferred embodiment as the previous one, if the sum of the values ​​(or absolute values) within the template is below a first threshold value T1, if the sum is greater than or equal to threshold value T1, the first context model is selected, if the value is greater than or equal to threshold value T1, the second context model is selected, if the value is greater than or equal to threshold value T2, the third context model is selected, and so on. This process can be applied to more or fewer thresholds.

[0335] In a particularly preferred embodiment, which is equivalent to the previous one, the template comprises the co-parameter and the left neighbour of the co-parameter, and a single threshold T1=0 is used.

[0336] In a further preferred embodiment, a given number of context models (context sets) for the symbol to be encoded are copied to form two or more context model sets. Then, the context model set is selected based on the absolute value of the co-location parameter in the corresponding layer of the previously encoded specific update or basic model. This means that if the absolute value of the co-location parameter is less than a first threshold T1, the first set is selected, if the absolute value is greater than or equal to another threshold T1, the second set is selected, if the absolute value is greater than or equal to threshold T2, the third set is selected, and so on. This process can be applied to more or fewer thresholds.

[0337] In the same preferred embodiment as the previous one, sig_flag is encoded using a context model set to indicate whether the current value to be encoded is equal to zero. This embodiment uses a single threshold value T1=1.

[0338] Another preferred embodiment is the same as the previous embodiment, but instead of encoding sig_flag, it is sign_flag that indicates the sign of the current value to be encoded.

[0339] Another preferred embodiment is the same as the previous embodiment, but if sig_flag is used, abs_level_greater_X is encoded, which indicates whether the current value to be encoded is greater than X.

[0340] In a further preferred embodiment, a given number of context models (context sets) for a symbol to be encoded is doubled to form two context model sets. The context model set is then selected based on whether there is a corresponding previously encoded update (or base) model. If there is no corresponding previously encoded update (or base) model, the first context model set is selected, otherwise the second set is selected.

[0341] In another preferred embodiment, a context model is selected from a set of context models for a syntax element based on the corresponding co-location parameter value in a previously encoded specific update (or base) model. This means that if the co-location parameter is below a threshold T1, a first model is selected, if the value is greater than or equal to the threshold T1, a second model is selected, and if the value is greater than or equal to another threshold T2, a third model is selected. This process can be applied to more or fewer thresholds.

[0342] In the same preferred embodiment as the previous one, a sign_flag is encoded, which indicates the sign of the current value to be encoded. The first threshold of the context model selection process is T1=0 and the second threshold is T2=1.

[0343] In another preferred embodiment, a context model is selected from a set of context models for a syntax element based on the absolute value of a particular corresponding co-location parameter in a previously encoded update (or base) model. This means that if the absolute value of the co-location parameter is less than a threshold T1, a first model is selected, if the value is greater than or equal to the threshold T1, a second model is selected, if the value is greater than or equal to the threshold T2, a third model is selected, and so on. This process can be applied to more or fewer thresholds.

[0344] In the same preferred embodiment as the previous one, sig_flag is encoded, which indicates whether the current value to be encoded is equal to zero. It sets the first threshold to T1=1 and the second threshold to T2=2.

[0345] In another preferred embodiment that is the same as the previous embodiment, sig_flag is not used, but an abs_level_greater_X flag is used for encoding, which indicates whether the current value to be encoded is greater than X. In addition, only one threshold is used, which is set to T1=X.

[0346] Please note that any one of the above embodiments may be combined with one or more other embodiments.

[0347] 2.4 Compressing multiple tensors in a block (example, all details are optional)

[0348] Typically, each tensor is encoded separately and transmitted in a so-called compressed data unit (NDU), which also contains (header) information about the compressed tensor, such as the tensor type, encoding parameters or the original dimensions of the tensor (as described in Section 2.1). However, MPEG-7 Part 17 specifies a special NDU type (NNR_PT_BLOCK) that allows to transmit the weight tensor and several corresponding tensors, namely biases, batch normalization parameters and scaling factors, in a single NDU. All tensors in a block share the same header information (instead of transmitting separate headers), thus reducing the bitstream size. Since the other tensors are directly related to the weight tensor, their dimensions can be derived from the (weight) tensor dimensions in the following way. Biases, batch normalization parameters and scaling factors are 1D tensors with a length equal to the number of output channels (fully connected layers) or the number of filters (convolutional layers) of the weight tensor, respectively. To derive these dimensions, the standard specifies to use the first dimension of the decoded weight tensor as the length of the 1D tensor. This means that the decoded weight tensor is expected to be ordered so that the first dimension always corresponds to the number of output channels or filters.

[0349] 3 Invention

[0350] Various aspects of the present invention and specific implementation methods of the present invention will be introduced below.

[0351] It should be noted that any features, functions and details described in this section can be used alone or in combination. In addition, it should be noted that any features, functions and details described in this section can be selectively introduced into any traditional encoding concept and decoding concept.

[0352] According to one aspect, the present invention describes a tensor dimension reordering scheme to enable efficient processing of tensors in a neural network compression framework, such as the MPEG-7 Part 17 standard for multimedia content description and analysis [2]. For example, before encoding the tensor, the tensor dimensions are reordered at the encoder side. For example, at the decoder side, the reordering order is reversed to restore the original dimensions. For example, information about how the original dimensions were derived is signaled in the bitstream.

[0353] For example, allowing arbitrary dimensional ordering may result in significant signaling overhead, especially for high-dimensional tensors, since the number of permutations increases greatly with the number of tensor dimensions N. In contrast, according to one aspect of the present invention, for example, the method of the present invention only performs a shift of a single dimension to another position.

[0354] The next section will briefly explain the necessity of reordering tensor dimensions. Then, in Section 3.2, the method will be illustrated with examples. After that, in Sections 3.3 and 3.4, the method and related syntax will be introduced with examples.

[0355] 3.1 Motivation

[0356] As mentioned in Section 2.1, tensors are often (but not necessarily) (e.g. in MPEG-7 Part 17) represented or interpreted as a one-dimensional vector or a two-dimensional matrix, respectively, e.g., with the length of the first dimension equal to the first dimension of the tensor and the length of the second dimension equal to the product of all other dimensions.

[0357] Therefore, the shape of a two-dimensional matrix may essentially depend on the first dimension of the tensor. For example, if the ordering of the tensor dimensions is such that the length of the first dimension is small (e.g. the width or height of a 3x3 filter kernel), a "slim" matrix is ​​output. For example, this may lead to inefficient processing of the two-dimensional matrix to be encoded (interpretation), especially for the block scanning order described in Section 2.2.4.1, where the matrix is ​​decomposed into blocks. For example, the MPEG-7 Part 17 standard specifies 4 block scanning orders with block sizes of 4x4, 8x8, 16x16 and 32x32. For example, suppose the first dimension of the matrix is ​​less than 4 (or 8, 16, 32), this will produce a row of clipped blocks, which is not optimal for further processing.

[0358] According to one aspect, the invention is further motivated, for example, by the method described in Section 2.4, wherein, for example, multiple related tensors are compressed in a block, for example using the method in MPEG-7 Part 17. For example, here the dimensions of the corresponding tensors are derived from the dimensions of the weight tensor, or, for example, more specifically, from the first dimension of the weight tensor. In some cases, this requires, for example, to order the tensors so that the first dimension is related to the number of output channels (fully connected layers) or filters (convolutional layers), otherwise the derivation process may fail in some cases.

[0359] For example, the above aspects demonstrate the advantages of an efficient tensor dimension reordering method, which is described in the following sections.

[0360] 3.2 Description of the tensor dimension reordering scheme (example, details are optional)

[0361] As mentioned above, according to one aspect of the present invention, the method reorders the tensor, for example, shifting one dimension to another position. Figure 6 The 4-dimensional example shown here illustrates the concept. As mentioned earlier, Figure 6Schematic diagram showing the concept of tensor dimension reordering for the encoding-decoding pipeline. For example, each box corresponds to a tensor / matrix representation / interpretation, with the dimensions given in square brackets. For example, the solid black arrows / lines indicate the general process flow, and the dashed lines / arrows indicate the process flow of the syntax element first_tensor_dimension_shift. (Example; encoding and decoding can be used separately; 2-dimensional matrix interpretation is optional).

[0362] For example, from the encoder point of view, the value first_tensor_dimension_shift (which can be set to 2 here) represents the tensor dimension, which is shifted to the first position of the tensor dimensions during the reordering process (see D2 in the example). So, for example, in a first step, the tensor is reordered, and in a second step, this tensor is, for example, (optionally) interpreted as a 2D matrix and then encoded. On the decoder side, the (optional) 2D matrix (interpretation) is, for example, decoded and interpreted as a tensor. Then, reordering is performed, for example, in reverse order, for example, by moving dimension D2 back to position 2. For example, the value first_tensor_dimension_shift here specifies the new position of the first dimension of the decoded and reconstructed tensor after reordering.

[0363] For the given example, the reordering process of the encoder and decoder is as follows:

[0364] Encoder (example; details are optional):

[0365] For example, suppose we are given a 4D tensor A with tensor dimensions [D0,D1,D2,D3] Enc and first_tensor_dimension_shift is equal to 2, and A Enc The elements of A Enc [m][n][o][p] visits (m∈[0,D0-1],n∈[0,D1-1],o∈[0,D2-1],and p∈[0,D3-1])), then

[0366] ● For example (optionally) initialize the reordered tensor B Enc The dimensions of [D2,D0,D1,D3]

[0367] ●For example, according to the following setting B Enc Values:

[0368]

[0369]

[0370] Decoder: (example; details are optional)

[0371] For example, suppose a 4D tensor B with tensor dimensions [D2, D0, D1, D3] Dec and first_tensor_dimension_shift equal to 2 have been decoded from the bitstream and can be passed to B Dec [o][m][n][p] Visit B Dec elements (m∈[0,D0-1],n∈[0,D1-1],o∈[0,D2-1],and p∈[0,D3-1]), then:

[0372] For example, initialize the reordered tensor A Dec The dimensions are [D0, D1, D2, D3]

[0373] ●For example, according to the following settings A Dec Values:

[0374]

[0375] Therefore, in general, a decoder (such as Figure 1 100 as shown, for example Figure 2 200, such as the corresponding decoding unit 110, 210) is configured to obtain a two-dimensional matrix 660 based on the encoded representation in the bit stream 650, for example, whose first dimension is determined by the first dimension value, such as Figure 6 D2 is shown, and its second dimension is determined by the product of multiple further dimension values, such as Figure 6 In addition, such a decoder can be configured to obtain a first multidimensional array (e.g., 111, e.g., 211, e.g., Figure 6 670 shown), whose dimension is determined by a single dimension value in the first dimension value and the further dimension value, and such a decoder can be configured to obtain a reordered multidimensional array (e.g., 111, e.g., 221, e.g., based on the first multidimensional array) Figure 6 680), wherein a first dimension of the reordered multidimensional array is defined by one of the further dimension values.

[0376] 3.3 Methods for reordering tensor dimensions (examples, details are optional)

[0377] This section introduces the invention using arbitrary tensor dimensions as an example, and its working principle is similar to the example in Section 3.2. For example, the encoder reorders the tensors so that a tensor dimension, such as specified by the value first_tensor_dimension_shift, is shifted to the first position. For example, the decoder completes this shift in reverse order (e.g., shifting the first dimension to the position indicated by first_tensor_dimension_shift). The following example illustrates this method.

[0378] For example, given an N-dimensional tensor Its dimensions are written as array dims T =[D0, D1, ..., D N-1 ], where, for example, D0 is associated with the first dimension of the tensor, D1 is associated with the second dimension of the tensor, and so on. k The value of (k∈[0, 1, ..., N-1]) represents, for example, the length of the associated dimension (e.g., D0 is the length of the first dimension). For example, the position of the weight in the tensor T can be represented by To indicate, for example, through another array Its length is equal to dims T Here, d k ∈[0, 1, ..., D k -1] For example, it indicates dimension D k The position in the scalar i∈[0,1,...,(D0·D1·...·D N-1 )] For example, specify the position in the tensor. i to The mapping is for example determined by scanning, for example in a row-major scan (see below for details).

[0379] In addition, there is for example the arrayIndexShift(inputArray[], shiftPos, reverseOrder) function, which accepts as input the array inputArray[], the value shiftPos and the boolean reverseOrder, and the output is for example the array outputArray[], as shown below:

[0380] ●For example, initialize outputArray[] with a copy of inputArray[].

[0381] ●For example, if reverseOrder == False:

[0382] ○The element at position shiftPos will be deleted from outputArray[], for example.

[0383] ○The element at position shiftPos of inputArray[] is inserted, for example, at the first position of outputArray[].

[0384] ●For example, otherwise:

[0385] ○If shiftPos>0:

[0386] ■The element at the first position is deleted from outputArray[]

[0387] ■The first element of inputArray[] is inserted into outputArray[], for example before the element at position shiftPos and after the element at position shiftPos-1

[0388] Then, for example, at the encoder one can obtain the reordered tensor R (from tensor T) as follows:

[0389] ● For example, to initialize the dimensions of a tensor

[0390] dims R =arraylndexShift(dims T , first_tensor_dimension_shift, False)

[0391] ●For example, for (i=0;i<(D0·D1·...·D N-1 ); i++)

[0392] ○

[0393] ○

[0394] For example, the decoder can obtain the reordered tensor R (from tensor T) as follows:

[0395] ●For example, initialize the dimensions of the tensor R

[0396] dims R =arraylndexShift(dims T , first_tensor_dimension_shift, True)

[0397] ●For example, for (i=0;i<(D0·D1·...·D N-1 ); i++)

[0398] ○

[0399] ○

[0400] In a preferred embodiment, the decoder process of the method is implemented as follows:

[0401] The inputs to this process are, for example:

[0402] ● variable inputTensor , e.g. the tensor whose dimensions should be reordered

[0403] ●Variable inputTensorDims, for example, specifies the dimensions of inputTensor

[0404] ●The variable firstTensorDimShift, for example, represents the shift of the first dimension of inputTensor

[0405] For example, the output of this process is the variable reorderedTensor, for example, whose dimensions are equal to ShiftArrayIndex(inputTensorDims,firstTensorDimShift). For example, the elements of the variable reorderedTensor are set as follows:

[0406]

[0407] For example, the functions Size, Prod, TensorIndex, and ShiftArrayIndex are defined as follows:

[0408] Prod(arrayName[]) for example returns the product of all elements of the array arrayName[].

[0409] Size(arrayName[]) returns the number of elements contained in the array or tensor named arrayName. If arrayName[] is a tensor, then the returned value is the product of all dimensions of the tensor.

[0410] TensorIndex(tensorDimensions[], i, scan) returns an array with the same dimensions as tensorDimensions[], where the elements of the array are set to integer values, such that the array can be used as an index to point to the elements of a tensor with dimensions tensorDimensions[], as shown below:

[0411] For example, if the variable scan is equal to 0:

[0412] For example, the returned array points to the i-th element of the tensor in row-major scan order.

[0413] The dimensions of a tensor, tensorDimensions[], are derived as follows:

[0414] For example, the variable outputArray[] is initialized with size set to Size(tensorDimensions[]).

[0415] For example, the variable idx is set to i

[0416] For example, the elements of outputArray[] are set as follows:

[0417] for(a=Size(tensorDimensions)-1;a>=0;a--){

[0418] outputArray[a]=idx%tensorDimensions[a]

[0419] idx = idx / tensorDimensions[a]

[0420] For example, the returned array is outputArray[].

[0421] For example, if the variable scan is greater than 0:

[0422] For example, the variable bs is set to 4<<scan_order.

[0423] For example, the variable h is set to tensorDimensions[0].

[0424] For example, the variable w is set to Prod(tensorDimensions) / h

[0425] For example, by calling IndexToXY(w, h, i, bs), the variables x and y are set to the first and second elements of the returned array, respectively.

[0426] For example, the returned array is TensorIndex(tensorDimensions, y*w+x, 0).

[0427] Note: The " / " operator is defined as integer division, with the result truncated toward zero. For example, 7 / 4 and -7 / -4 are truncated to 1, and -7 / 4 and 7 / -4 are truncated to -1.

[0428] Note: The operator "++" is defined as increment, that is, x++ is equivalent to x=x+1; when used in array indexing, its value is the value of the variable before the increment operation.

[0429] Note: The operator "--" is defined as decrement, that is, x-- is equivalent to x=x-1; when used in array indexing, its value is the value of the variable before the decrement operation.

[0430] Note: The operator "x % y" is defined as the modulus. The remainder of x divided by y is defined only for integers x and y where x ≥ 0 and y > 0.

[0431] Note: The function TensorIndex, for example, provides a mapping from "i" to positions in the tensor, which is equivalent to scanning a 2D matrix with dimensions equal to [tensorDimensions[0], Prod(tensorDimensions) / tensorDimensions[0]] according to "scan" (see 2.2.4.1).

[0432] For example, ShiftArrayIndex(inputArray[],TensoshiftIndexPosition) returns the array outputArray[], which is a copy of inputArray[], but with the element at position 0 in inputArray shifted to shiftIndexPosition, such as the following:

[0433] For example, the variable outputArray[] is initialized with a copy of inputArray[].

[0434] For example, if shiftIndexPosition is greater than 0:

[0435] For example, the first element of outputArray[] is deleted from outputArray[].

[0436] For example, the first element of inputArray[] is inserted into outputArray[] before the element at position shiftIndexPosition and after the element at position shiftIndexPosition-1.

[0437] Generally speaking, according to the decoder of the embodiment, for example Figure 1 The decoder 100 shown, for example, Figure 2The decoder 200 shown, such as its corresponding decoding units 110, 210, can be configured to determine the position of the decoded neural network parameters in the first multidimensional array (e.g., 111, e.g., 211) using a position mapping function (e.g., TensorIndex(tensorDimensions[], i, scan)), which maps a scalar parameter index (e.g., i) to a set of dimension indices. Such a decoder can be particularly configured to decode the encoded neural network parameters using context-based entropy decoding, such as Figure 2 shown.

[0438] Furthermore, in general, a decoder according to an embodiment, such as Figure 1 The decoder 100 shown, for example, Figure 2 The decoder 200 shown, such as its corresponding decoding units 120, 220, can be configured to obtain a reordered multidimensional array (e.g., 121, e.g., 221) using a function (e.g., TensorIndex(tensorDimensions[], i, scan)), which maps an integer element index i specifying an element of a first multidimensional array (e.g., 111, 211) to an array index set, wherein the returned array index set specifies the i-th element in a row-major scan order of the first multidimensional array, or wherein the returned array index set specifies the i-th element of a block scan of the first multidimensional array, wherein the first multidimensional array is considered to be a two-dimensional array.

[0439] Furthermore, in general, a decoder according to an embodiment, such as Figure 1 The decoder 100 shown, for example Figure 2 The decoder 200 shown, such as its corresponding reordering unit 120, 220, for example, can be configured to decode an array dimension set, such as tensor_dimensions[], and obtain a reordered array dimension set. Figure 6 , the reordered array dimension set may be a vector [D0, D1, D2, D3], for example representing a reordered tensor 680 , which is obtained based on a decoded array dimension set of, for example, tensor 670 or optionally tensor 660 .

[0440] Furthermore, in general, a decoder according to an embodiment, such as Figure 1 The decoder 100 shown, for example Figure 2The decoder 200 shown, such as its corresponding decoding units 110, 210, can be configured to input the decoded neural network parameters into a first multidimensional array (e.g., 111, 211, 670) at corresponding positions described by corresponding array index sets, wherein the decoder is configured to obtain the corresponding array index set using a mapping function, which maps a scalar integer parameter index to a corresponding array index set and defines a block scan.

[0441] In addition, in general, the mapping function optionally includes a mapping of a scalar integer parameter index to two coordinates, where the two coordinates point to locations corresponding to scan indices defined by the scalar integer parameter index when the blocks are scanned block by block, and the mapping function may include a mapping of the two coordinates to a corresponding set of array indices.

[0442] 3.4 Syntax for reordering tensor dimensions (example, details are optional)

[0443] For example, the syntax element first_tensor_dimension_shift is written to the bitstream. For example, from the decoder's point of view, it can specify the new position of the first tensor dimension decoded from the bitstream after reordering. Therefore, the value of first_tensor_dimension_shift is a non-negative value and is less than the number of tensor dimensions. For example, a value equal to zero means that the tensors are not reordered (for example, this is equivalent to shifting the first dimension to the first position).

[0444] For example, in a preferred embodiment, count_tensor_dimensions represents the number of tensor dimensions, general_profile_idc is equal to 1, and the first tensor dimension is encoded as follows:

[0445]

[0446] For example, if irst_tensor_dimension_shift does not exist, it is inferred to be 0.

[0447] For example, in another preferred embodiment, first_tensor_dimension_shift is encoded using an Exponential Golomb code. Here, the number of bits required to encode smaller values ​​is generally lower. Thus, for example, the number of bits required for a value equal to 1 is less than or equal to the number of bits required for a value equal to 2. For example, the number of bits required for a value equal to 2 is less than or equal to the number of bits required for a value equal to 3, and so on.

[0448] References for the section "Tensor Dimension Reordering Methods for Neural Network Encoding"

[0449] [1]S.Chetlur et al.,"cuDNN:Efficient Primitives for Deep Learning,"arXiv:1410.0759,2014

[0450] [2]MPEG,“Text of ISO / IEC DIS15938-17Compression of Neural Networksfor Multimedia Content Description and Analysis”,Document of ISO / IEC JTC1 / SC29 / WG11,w19764,OnLine,Oct.2020

[0451] [3]D.Marpe,H.Schwarz und T.Wiegand,,,Context-Based Adaptive BinaryArithmetic Coding in the H.264 / AVC Video Compression Standard,“IEEEtransactions on circuits and systems for video technology,Vol.13,No.7,pp.620-636,July 2003.

[0452] [4]H.Kirchhoffer,J.Stegemann,D.Marpe,H.Schwarz und T.Wiegand,,JVET-K0430-v3-CE5-related:State-based probalility estimator,“in JVET,Ljubljana,2018.

[0453] [5]ITU-International Telecommunication Union,,,ITU-T H.265Highefficiency video coding,“Series H:Audiovisual and multimedia systems-Infrastructure of audiovisual services-Coding of moving video,April 2015.

[0454] [6] B.Bross, J.Chen und S.Liu,,,JVET-M1001-v6-Versatile Video Coding(Draft 4),” in JVET,Marrakech,2019.

[0455] [7] S.Wiedemann et al., "DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks," in IEEE Journal of Selected Topics in SignalProcessing, vol.14, no.4, pp.700-714, May 2020, doi:10.1109 / JSTSP.2020.2969554.

[0456] Proposed draft update of the planned international standard ISO / IEC 15938-17

[0457] (e.g. ISO / IEC DIS15938-17: xxxx(E))

[0458] Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis (2nd edition)

[0459] General Notes: All details are optional

[0460] The following sections discuss various aspects of the invention: Preface, Introduction, 1 Scope, 2 Normative References, 3 Terms and Definitions, 4 Abbreviated Terms, Conventions, and Symbols, 4.1 General, 4.2 Abbreviated Terms, 4.3 List of Symbols, 4.4 Numerical Formats and Calculation Conventions, 4.5 Arithmetic Operators, 4.6 Logical Operators, 4.7 Relational Operators, 4.8 Bitwise Operators, 4.9 Assignment Operators, 4.10 Range Representation, 4.11 Mathematical Functions, 4.12 Array Functions, 4.13 Order of Operation Precedence, 4.14 Variables, Syntax Elements, and Tables, 5 Overview, 5.1 General, 5.2 Compression Tools, 5.3 Creating an Encoding Pipeline, 6 Syntax and Semantics, 6.1 Syntax and Semantics Specification, 6.1.1 Methods for Specifying Syntax in Tabular Form, 6 .1.2 Bit Ordering, 6.1.3 Specification of Syntax Functions and Data Types, 6.1.4 Semantics, 6.2 General Bitstream Syntax Elements, 6.2.1 NNR Unit, 6.2.2 Aggregated NNR Units, 6.2.3 Composition of NNR Bitstream, 6.3 NNR Bitstream Syntax, 6.3.1 NNR Unit Syntax, 6.3.2 NNR Unit Size Syntax, 6.3.3 NNR Unit Header Syntax, 6.3.4 NNR Unit Payload Syntax, 6.3.5 Byte Alignment Syntax, 6.4 Semantics, 6.4.1 General, 6.4.2 NNR Unit Size Semantics, 6.4.3 NNR Unit Header Semantics, 6.4.4 NNR Unit Payload Semantics, 7 Decoding Flow, 7.1 General, 7.2 NNR Decompression 7.3 Decoding Methods 7.3.1 General 7.3.2 Decoding Methods for NNR Compressed Payloads of Type NNR_PT_INT 7.3.3 Decoding Methods for NNR Compressed Payloads of Type NNR_PT_FLOAT 7.3.4 Decoding Methods for NNR Compressed Payloads of Type NNR_PT_RAW_FLOAT 7.3.5 Decoding Methods for NNR Compressed Payloads of Type NNR_PT_BLOCK 7.3.6 Decoding Process for Integer Weight Tensors 8 Parameter Reduction 8.1 General (Informative) 8.2 Methods (Informative) 8.2.1 Sparsification Using Compressibility Loss 8.2.2 Sparsification Using Microstructure Pruning 8.2.3 Combined Pruning and Sparsification ,8.2.4 Structured Sparsification,8.2.5 Parameter Unification,8.2.6 Low Rank / Low Shift Rank for Convolutional and Fully Connected Layers,8.2.7 Batch Normalization Folding,8.2.8 Local Scaling Adaptation,8.3 Syntax and Semantics,8.3.1 Sparsification Using Compressibility Loss,8.3.2 Sparsification Using Microstructured Pruning,8.3.3 Combined Pruning and Sparsification,8.3.4 Structured Sparsification,8.3.5 Weight Unification,8.3.6 Low Rank / Low Shift Rank for Convolutional and Fully Connected Layers,8.3.7 Batch Normalization Folding,8.3.8 Local Scaling,9 Parameter Quantization,9.1 Methods,9.1.1 Uniform Quantization Methods,9.1.2 Codebook-Based Methods,9.1.3 Dependent Scalar Quantization Methods,9.1.4 Iterative QP Optimization (Informative), 9.2 Syntax and Semantics, 9.2.1 Uniform Quantization Method, 9.2.2 Codebook-Based Method, 9.2.3 Dependent Scalar Quantization Method, 10 Entropy Coding, 10.1 Method, 10.1.1 DeepCABAC, 10.2 Syntax and Semantics, 10.2.1 DeepCABAC Syntax, 10.3 Entropy Decoding Process, 10.3.1 Overview, 10.3.2 Initialization Process, 10.3.3 Binarization Process, 10.3.4 Decoding Process Stream, Annex A (Normative) Implementation of NNEF, A.1 Overview, A.2 Identifiers, A.3 Definitions Used in NNR Bitstreams, A.4 Carrying of NNEF Data in NNR Bitstreams, . Annex B (Informative) implementation of, B.1 General, B.2 Identifiers, B.3 Definitions used in NNR bitstreams, B.4 Carrying of ONNX data in NNR bitstreams, Annex C (Informative) implementation of, C.1 General, C.2 Identifiers, C.3 Definitions used in NNR bitstreams, C.4 Carrying of PyTorch data in NNR bitstreams, Annex D (Informative) Implementation of, D.1 General, D.2 Identifiers, D.3 Definitions used in NNR bitstreams, D.4 Carrying TensorFlow data in NNR bitstreams, Annex E (Informative) Recommendations for carrying NNR bitstreams in other containers, E.1 Recommendations for carrying NNR bitstreams in NNEF container organization (specified in the NNEF format specification), E.2 Recommendations for carrying NNR encoded bitstreams in ONNX (specified in the ONNX format specification), References to "Information Technology - Multimedia Content Description Interface"

[0461] In addition, it should be noted that, for the sake of brevity, some of the above chapters (e.g., Annexes A to E) may not be included in full herein, for example, because the features disclosed therein are optional for embodiments of the present invention. However, it should be noted that embodiments according to the present invention may include any or all of the features disclosed in the chapters, for example, the features disclosed in the complete version of the chapters, for example, the features disclosed in the (e.g., planned) draft of the international standard ISO / IEC15938-17 (e.g., ISO / IECDIS15938-17: xxxx (E)).

[0462] Preface

[0463] This proposed second edition cancels and replaces the technically revised first edition (ISO / IEC 15938-17).

[0464] Compared with the previous version, the main changes are as follows:

[0465] —xxx xxxxxxx xxx xxxx

[0466] Introduction (any details are optional)

[0467] Artificial neural networks have been widely used in multimedia analysis and processing, media coding, data analytics, and many other fields. Artificial neural networks (deep neural networks, DNNs) have recently achieved great success due to their larger and more complex processing power than in the past and the availability of large-scale training datasets. As a result, trained neural networks contain a large number of parameters and weights, resulting in a considerable size (e.g., hundreds of MB). Many applications may require the deployment of a particular trained network instance to a large number of devices, which may be limited in terms of processing power and memory (e.g., panning devices or smart cameras) as well as communication bandwidth. Any use case that requires the deployment of a trained neural network (or its updates) to a large number of devices can benefit from a standard for the compressed representation of neural networks.

[0468] Given that compression of neural networks may have both hardware-dependent and hardware-independent components, this document is designed as a toolbox of compression techniques. Some of these techniques require feature representation in an interchange format (e.g. sparse representation, adaptive quantization), so standard specifications for representing the output of these techniques are defined. Other techniques do not require a serialized representation at all (e.g. pruning), however, required metadata is specified for the latter as well. This document is agnostic to a specific neural network interchange format, interoperability with common formats is described in the annexes.

[0469] Therefore, this document defines a high-level syntax, specifies the required metadata elements and the associated semantics. Where it is necessary to specify the structure of the binary data (e.g. a factorization matrix), this document also specifies the actual bitstream syntax of the corresponding blocks. The annexes to this document specify the requirements and constraints for the compressed neural network representation as defined in this document; and how it should be applied.

[0470] - Annex A, as an example, specifies the implementation of this document using the Neural Network Exchange Format (NNEF), defining the use of NNEF to represent network topology in compressed neural network bitstreams.

[0471] - Annex B provides examples of using the Open Neural Network Exchange Format (ONNX) Implements the proposal of this document, defining the use of ONNX to represent network topology in compressed neural network bitstreams.

[0472] - Annex C provides examples of the use of The format implements the proposal of this document and defines references to PyTorch elements in network topology descriptions of compressed neural network bitstreams.

[0473] - Annex D provides examples of the use of The format implements the proposal of this document and defines references to Tensorflow elements in network topology descriptions of compressed neural network bitstreams.

[0474] - Annex E provides, as an example, suggestions for carrying tensors compressed according to this document in third-party container formats.

[0475] The compression tools described in this paper were selected and evaluated for neural networks used in multimedia description, analysis, and processing applications. However, these tools can also be used to compress neural networks and other types of data in other applications.

[0476] Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis (2nd edition)

[0477] 1 Scope

[0478] This document specifies, as an example, a compressed representation of the parameters / weights of a trained neural network, and a decoding process for the compressed representation.

[0479] ________________________

[0480] 1 ONNX is a trademark of a product owned by LF PROJECTS, LLC. This information is provided for the convenience of users of this document and does not constitute an ISO / IEC endorsement of the product named.

[0481] 2 PyTorch is a trademark of a product provided by Facebook, Inc. This information is provided for the convenience of users of this document and does not constitute an ISO / IEC endorsement of the named product.

[0482] 3 TensorFlow is a trademark of a product provided by Google LLC. This information is provided for the convenience of users of this document and does not constitute an endorsement by ISO / IEC of the product described.

[0483] It complements the description of network topology in the existing Neural Network (Interchange) Format. It builds on the toolbox of compression methods, specifying (where applicable) the resulting elements of the compressed bitstream. For example, all of these tools can be applied to the compression of an entire neural network, and some of them can also be applied to the differential updates of a compressed neural network with respect to a base network. Such differential updates are useful, for example, when a model is re-distributed after fine-tuning or transfer learning, or when providing versions of a neural network with different compression ratios.

[0484] For example, this document specifies compressed representations of neural networks that can be further incorporated into other standards. Only the syntactic format, semantics, related decoding process requirements, parameter sparsification, parameter conversion methods, parameter quantization, entropy coding methods, and integration / signaling in existing interchange formats are specified, while other matters such as pre-processing, system signaling and multiplexing, data loss recovery, and post-processing are considered to be outside the scope of this document. In addition, the internal processing steps performed within the decoder are also considered to be outside the scope of this document; only the externally observable output behavior needs to conform to the specifications of this document.

[0485] 2 Normative reference documents

[0486] The following documents cited in this document, part or all of which constitute the requirements of this document. For dated references, only the referenced version applies. For undated references, the latest version of the referenced document (including any revisions) applies.

[0487] ISO / IEC10646, Information technology - Universal coded character set (UCS)

[0488] ISO / IEC 60559, Information technology - Microprocessor systems - Floating-point operations

[0489] IETF RFC1950, ZLIB Compressed Data Format Specification Version 3.3, 1996

[0490] NNEF-v1.0.3, Neural Network Interchange Format, Khronos NNEF Working Group, version 1.0.3, 2020-06-12 ( https: / / www.khronos.org / registry / NNEF / specs / 1.0 / nnef-1.0.3.pdf )

[0491] 3 Terms and definitions

[0492] For the purposes of this document, the following terms and definitions apply.

[0493] ISO and IEC maintain databases of terminology used in their standardization work at the following addresses:

[0494] -ISO online browsing platform: available at https: / / www.iso.org / obp

[0495] -IEC Electronics Encyclopedia: http: / / www.electropedia.org / 3.1

[0497] Polymerized NNR units

[0498] NNR unit, carrying multiple NNR units in the payload of the NNR unit 3.2

[0500] Basic Neural Network

[0501] Neural network used as reference for differential updates 3.3

[0503] Compressed Neural Network Representation

[0504] Encode neural network representations with model parameters using compression tools 3.4

[0506] break down

[0507] Transformation that represents a tensor as the product of two tensors 3.5

[0509] Hyperparameters

[0510] Parameters, whose values ​​are used to control the learning process 3.6

[0512] layer

[0513] A collection of nodes that operate together at a specific depth in a neural network 3.7

[0515] Model parameters

[0516] The coefficients of the neural network model, such as weights and biases 3.8

[0518] NNR Unit

[0519] Data structure used to hold (compressed or uncompressed) neural network data and associated metadata 3.9

[0521] prune

[0522] Reduce the parameters of (part of) the neural network 3.10

[0524] Sparse

[0525] Increase the number of zero-valued entries of a tensor 3.11

[0527] Tensor

[0528] Multidimensional structural grouping related model parameters 3.12

[0530] Updated Neural Network

[0531] Neural network obtained by modifying the basic neural network

[0532] Note: The updated neural network is reconstructed by applying differential updates to the base neural network.

[0533] 4 Abbreviations, conventions and symbols

[0534] 4.1 General

[0535] This subclause contains definitions of operators, symbols, functions, textual conventions, and procedures used in this document.

[0536] The mathematical operators used in this document are similar to those used in the C programming language. However, this document specifies the results of integer division and arithmetic shift operations more precisely, and specifies additional operations such as exponentiation and real-valued division. Numbering and counting conventions generally start at 0, for example, "first" is equivalent to the 0th, "second" is equivalent to the 1st, and so on.

[0537] 4.2 Abbreviations

[0538] DeepCABAC Deep Neural Network Context-Adaptive Binary Arithmetic Coding

[0539] LR Low-rank

[0540] LDR Low displacement rank

[0541] LPS Layer parameter set

[0542] LSB Least significant bit

[0543] MSB Most significant bit

[0544] MPS Model parameter set

[0545] NN Neural network

[0546] NNEF Neural network exchange format

[0547] NNR Compressed neural network representation

[0548] SVD Singular value decomposition

[0549] 4.3 Symbol List

[0550] This document defines the following symbols:

[0551] A input tensor

[0552] B output tensor

[0553] Blocks in superblock j at level k

[0554] b Bias parameter

[0555] C in Number of input channels of the convolutional layer

[0556] C out The number of output channels of the convolutional layer

[0557] The number of channels of the tensor in layer k

[0558] The number of channels of the tensor in the inferred layer k

[0559] The depth dimension of the tensor at layer k

[0560] ef circulant matrix Z e Parameters

[0561] F is the parameter tensor of the convolutional layer

[0562] ff Parameters of the circulant matrix Zf

[0563] G k Matrix W k The left-hand side matrix of the low-rank decomposition representation

[0564] H k Matrix W k The right-hand side matrix of the low-rank decomposition representation

[0565] The height dimension of the tensor at layer k

[0566] K is the dimension of the convolution kernel

[0567] L Network

[0568] L compressibility Compression loss

[0569] L aiversity Diversity loss

[0570] L task Mission Loss

[0571] L train Training loss

[0572] M feature matrix

[0573] M k Pruning Mask

[0574] m is the sparsification hyperparameter

[0575] m i The i-th row of the feature matrix M

[0576] The size of the kernel of the tensor at layer k

[0577] n k The dimension produced by the product of

[0578] P random transition matrix

[0579] p pruning rate hyperparameter

[0580] p i，j The elements of the transformation matrix P

[0581] q Sparsity hyperparameter

[0582] Importance of S-pruning parameter

[0583] Super Block

[0584] s local scaling factor

[0585] Superblock size

[0586] u uniform ratio hyperparameter

[0587] W parameter tensor

[0588] W l The weight tensor of layer l

[0589] W k Parameter tensor for layer k

[0590] W k Low-rank approximation of

[0591] w parameter vector

[0592] The width dimension of the tensor at layer k

[0593] w l，i The weight vector of the i-th filter in the l-th layer

[0594] w′ l，i The normalized weight vector of the ith filter in layer l

[0595] X is the input to the batch normalization layer.

[0596] Z e f Circular Matrix

[0597] Z f f Circular Matrix

[0598] Batch Normalization parameter for α fold

[0599] α′ is the combined value of the batch normalization parameter and the local scaling factor for the fold

[0600] β Batch Normalization Parameter

[0601] γ c Compression loss multiplier

[0602] γ Batch Normalization parameter

[0603] δ is the batch normalization parameter for the fold

[0604] ∈ scalar close to zero to avoid division by zero in batch normalization

[0605] λ eigenvector

[0606] λ c Compression loss weight

[0607] λ d Diversity loss weight

[0608] μ Batch Normalization parameter

[0609] The equilibrium probability of π P

[0610] σ Batch Normalization parameter

[0611] τ: sparse pruning threshold

[0612] Smoothing Factor

[0613] 4.4 Number Formats and Calculation Conventions

[0614] This document defines the following number formats:

[0615] integer An integer, which can be of any size. Integers are also called signed integers.

[0616] unsigned integer An unsigned integer that can be 0 or arbitrarily large.

[0617] Float Floating point number, compliant with ISO / IEC60559.

[0618] The results of all operators and mathematical functions are mathematically exact unless otherwise specified. Whenever a result is a floating point number, it is explicitly specified.

[0619] 4.5 Arithmetic operators

[0620] The following arithmetic operators are defined:

[0621] + Addition

[0622] - Subtraction (as a two-argument operator) or negation (as a unary prefix operator)

[0623] * Multiplication, including matrix multiplication

[0624] . Element-wise multiplication of two transposed vectors, or element-wise multiplication of a transposed vector and a matrix row, or Hadamard product of two matrices of the same dimension

[0625] x y Exponentiation. Specifies x raised to the power of y. In other contexts, this notation is used for superscripts and is not interpreted as a power.

[0626] / Integer division truncating the result towards zero. For example, 7 / 4 and -7 / -4 are truncated to 1, and -7 / 4 and 7 / -4 are truncated to -1.

[0627] ÷ is used to represent division in mathematical equations where truncation or rounding is not required.

[0628] Used to represent division in mathematical equations without intended truncation or rounding, including element-wise division of two transposed vectors or element-wise division of a transposed vector and a row of a matrix.

[0629] The sum of f(i), where i takes all integer values ​​from x to y (inclusive).

[0630] The product of f(i) and i, where i is all integers from x to y (including y).

[0631] The remainder of x%yx divided by y is valid only for integers x and y where x≥0 and y>0.

[0632] 4.6 Logical Operators

[0633] The following logical operators are defined:

[0634] x&&y Boolean logical "AND" of x and y

[0635] x‖y Boolean logical "OR" of x and y

[0636] ! Boolean logic "not"

[0637] x? y:z If x is true or not equal to 0, it evaluates to the value of y; otherwise, it evaluates to the value of z.

[0638] 4.7 Relational Operators

[0639] The following relational operators are defined as follows:

[0640] > Greater than

[0641] ≥ greater than or equal to

[0642] < Less than

[0643] ≤ Less than or equal to

[0644] == is equal to

[0645] ! = Not equal to

[0646] When a relational operator is applied to a syntax element or variable that is assigned the value "na" (not applicable), the value "na" is considered to be an independent value for that syntax element or variable. The value "na" is considered not equal to any other value.

[0647] 4.8 Bitwise Operators

[0648] The following bitwise operators are defined as follows:

[0649] & Bitwise AND. When operating on integer arguments, the operation is performed on the binary representation of the integer value. When operating on binary arguments, if a binary argument contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0650] | Bitwise OR. When operating on integer arguments, the operation is performed on the binary representation of the integer value. When operating on binary arguments, if a binary argument contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0651] ^ Bitwise exclusive OR. When operating on integer arguments, the operation is performed on the binary representation of the integer value. When operating on binary arguments, if a binary argument contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0652] x>>y Performs an arithmetic right shift of the binary integer representation of x by y binary bits. This function works only for non-negative integer values ​​of y. The value of the bit shifted into the MSB after the right shift is equal to the MSB of x before the shift operation.

[0653] x<<y Performs an arithmetic left shift of the binary integer representation of x by y binary digits. This function works only for non-negative integer values ​​of y. The value of the bit shifted into the LSB after the left shift is equal to 0.

[0654] The bitwise NOT operator returns 1 when it is zero and returns 0 when it is 1.

[0655] 4.9 Assignment Operators

[0656] The following arithmetic operators are defined as follows:

[0657] = Assignment Operator

[0658] ++ Increment, that is, x++ is equivalent to x=x+1; when used in array indexing, it evaluates to the value of the variable before the increment operation.

[0659] -- Decrement, that is, x- is equivalent to x=x-1; when used in array indexing, it evaluates to the value of the variable before the decrement operation.

[0660] += specifies the increment, i.e. x+=3 is equivalent to x=x+3, and x+=(-3) is equivalent to x=x+(-3).

[0661] -= decrements by the specified amount, i.e. x-=3 is equivalent to x=x-3, and x-=(-3) is equivalent to x=x-(-3).

[0662] 4.10 Range Representation

[0663] The following symbols are used to specify ranges of values:

[0664] x=y..zx takes integer values ​​from y to z (including z), where x, y and z are integers and z is greater than y.

[0665] array[x, y] A subarray consisting of the elements between positions x and y in array. If x is greater than y, the subarray is empty.

[0666] 4.11 Mathematical functions

[0667] The following mathematical functions are defined:

[0668] Ceil(x) The smallest integer greater than or equal to x

[0669] Floor(x) The largest integer less than or equal to x

[0670] Log2(x) Logarithm of x to base 2

[0671]

[0672] 4.12 Array Functions

[0673] Size(arrayName[]) returns the number of elements in the array or tensor named arrayName. If arrayName[] is a tensor, it returns the product of all dimensions of the tensor. (Optional)

[0674] Prod(arrayName[]) returns the product of all elements in the array arrayName[]. (optional)

[0675] Tensor(arrayName[],tensorDimension[]) uses the tensor array_name[] reshaped to the specified tensorDimension[], but does not change its data. (optional)

[0676] For example, IndexToXY(w,h,i,bs) returns an array containing two elements. For example, the first element is the x coordinate and the second element is the y coordinate, such as pointing to a two-dimensional array with width w and height h. For example, when , x and y point to the position corresponding to the scan index i when scanning in blocks (e.g., size bs times bs). For example, the derivation of x and y is as follows:

[0677] The variable fullRowOfBlocks is set to w*bs

[0678] The variable blockY is set to i / fullRowOfBlocks

[0679] The variable iOff is set to i%fullRowOfBlocks

[0680] The variable currBlockH is set to Min(bs,h-blockY*bs)

[0681] The variable fullBlocks is set to bs*currBlockH

[0682] The variable blockX is set to iOff / fullBlocks

[0683] The variable blockOff is set to iOff % fullBlocks

[0684] The variable currBlockW is set to Min(bs,w-blockX*bs)

[0685] The variable posX is set to blockOff%currBlockW

[0686] The variable posY is set to blockOff / currBlockW

[0687] The variable x is set to blockX*bs+posX

[0688] The variable y is set to blockY*bs+posY

[0689] For example, TensorIndex(tensorDimensions[],i,scan) returns an array with the same dimensions as tensorDimensions[], where the elements of the array are set to integer values ​​so that the array can be used as an index into an element of a tensor with dimensions tensorDimensions[], for example:

[0690] If the variable scan is equal to 0:

[0691] For example, the returned array points to the i-th element of the tensor in row-major scan order.

[0692] Dimensions tensorDimensions[] .

[0693] If the variable scan is greater than 0: (optional) (example)

[0694] For example, the variable bs is set to 4< <scan_order。

[0695] For example, the variable h is set to tensorDimensions[0].

[0696] For example, the variable w is set to Prod(tensorDimensions) / h

[0697] For example, the variables x and y are set to the first and second elements of the array, respectively.

[0698] Returns, for example:

[0699] By calling IndexToXY(w,h,i,bs) respectively.

[0700] For example, the returned array is TensorIndex(tensorDimensions, y*w+x, 0).

[0701] Note: The variable scan usually (but not necessarily) corresponds to the syntax element scan_order.

[0702] GetEntryPointIdx(tensorDimensions[], i, scan (optional), for example, if the index i does not point to the first position of the entry point, return -1. If the index i points to the first position of the entry point, return, for example, the entry point index in the tensor. The method for determining the position and index of the entry point is as follows: (example)

[0703] The variable w is set to Prod(tensorDimensions) / tensorDimensions[0]

[0704] The variable epIdx is set to i / (w*(4<<scan))-1

[0705] If i>0 and i % (w*(4<<scan)) equals 0, the index i points to the first position of the entry point, and the entry point index is equal to epIdx.

[0706] Otherwise, the index i does not point to the first position of the entry point.

[0707] For example, ShiftArrayIndex(inputArray[], shiftIndexPosition) returns the array outputArray[], which is a copy of inputArray[], but moves the element at position 0 in inputArray to shiftIndexPosition, for example as follows:

[0708] For example, the variable outputArray[] is initialized with a copy of inputArray[].[[]END]]

[0709] If shiftIndexPosition is greater than 0:

[0710] For example, the first element of outputArray[] is removed from outputArray[].[[]END]]

[0711] For example, the first element of inputArray[] is inserted into outputArray[] before the element at position shiftIndexPosition,

[0712] after the element at position shiftIndexPosition-1.

[0713] AxisSwap(inputTensor[], tensorDimensions[], numberOfDimensions, axis0, axis1) (optional) Returns a tensor derived from inputTensor (with dimensions tensorDimensions and numberOfDimensions) with axis indices axis0 and axis1 of inputTensor swapped.

[0714] TensorSplit(inputTensor[], splitIndices, splitAxis) (optional) returns a tensor array subTensors that is obtained by splitting the tensor inputTensor into N = Size(splitIndices) + 1 tensors along the provided axis splitAxis using the provided index array splitIndices, as shown below:

[0715] The array inputDims is set to the dimensions of the tensor inputTensor.

[0716] Elements with a value of 0 are inserted into splitIndices before the first element, and elements with a value of inputDims[splitAxis] are inserted into splitIndices after the last element.

[0717] The tensor subTensors[X] (where X is an integer from 0 to N) is derived as follows:

[0718] The array subTensorDims is set to inputDims.

[0719] The element subTensorDims[splitAxis] is replaced by the value splitIndices[X+1]-splitIndices[X].

[0720] The elements of subTensors[X] are set as follows:

[0721] for(i=0;i <Prod(subTensorDims);i++){

[0722] subIdx=TensorIndex(subTensorDims,i,0)

[0723] inputIdx=TensorIndex(inputDims,i,0)

[0724] inputIdx[splitAxis]+=splitIndices[X]

[0725] subTensors[subIdx]=inputTensor[inputIdx]

[0726] 4.13 Operation precedence

[0727] When the order of precedence in an expression is not explicitly indicated by the use of parentheses, the following rules apply:

[0728] ●Operations with higher priority are evaluated before operations with lower priority.

[0729] ●Operations of the same priority are evaluated from left to right.

[0730] Table 1 specifies the priority of operations, from highest to lowest. Higher positions in the table indicate higher priority.

[0731] Note—For operators that are also used in the C programming language, the order of precedence used in this document is the same as that used in the C programming language.

[0732] Table 1. Operation priority, from highest (top of table) to lowest (bottom of table).

[0733]

[0734] 4.14 Variables, syntax elements, and tables

[0735] Syntax elements in the bitstream are in bold. Each syntax element is described by its name (all lowercase letters with underscore characters) and a data type for its coded representation. The decoding process operates on the value of the syntax element and the values ​​of previously decoded syntax elements. When the value of a syntax element is used in a syntax table or text, the value is shown in normal font (i.e., not bold).

[0736] In some cases, syntax tables can use the values ​​of other variables derived from the values ​​of syntax elements. Such variables appear in syntax tables or texts with a mixture of lowercase and uppercase letters, without any underscore characters (camel case). Variables beginning with an uppercase letter are used in the decoding of the current syntax structure and all related syntax structures. Variables beginning with an uppercase letter can be used in later syntax structures during decoding without referring to the original syntax structure of the variable. Variables beginning with a lowercase letter are used only in the (sub)clauses from which they are derived.

[0737] In some cases, "mnemonic" names for syntax element values ​​or variable values ​​are used interchangeably with their numeric values. Sometimes "mnemonic" names are used without any associated numeric value. The association between value and name is specified in the text. Names consist of one or more groups of letters separated by underscores. Each group begins with an uppercase letter and can contain more uppercase letters.

[0738] Note - The way the syntax is described is very close to the grammatical structure of C language.

[0739] Functions that specify properties of the current position in the bitstream are called syntax functions. These functions are specified in subclause 6.3 and assume the presence of a bitstream pointer that indicates the position of the next bit to be read from the bitstream by the decoding process. A syntax function is described by its name, which is constructed in the form of a syntax element name, terminated by left and right parentheses, and consisting of zero or more variable names (for definitions) or variable values ​​(for uses), separated by commas (if there is more than one variable).

[0740] Functions that are not syntax functions (including mathematical functions specified in Subclause 4.11 and array functions specified in Subclause 4.12) are described by a name that begins with an uppercase letter, contains both lowercase and uppercase letters, does not contain underscores, and ends with an opening and closing parenthesis followed by zero or more variable names (for definitions) or variable values ​​(for uses), separated by commas (if there is more than one variable).

[0741] A one-dimensional array is called a list. A two-dimensional array is called a matrix. Arrays can be either syntax elements or variables. Subscripts or square brackets are used to index into arrays. Referring to the intuitive description of matrices, the first subscript is used as the row (vertical) index and the second subscript is used as the column (horizontal) index. When square brackets are used instead of subscripts for indexing, the indexing order is reversed. Thus, the element at horizontal position x and vertical position y in matrix s can be represented as s[x][y] or as s yx A single column of a matrix can be called a list, omitting the row index. Thus, the column of the matrix s at horizontal position x can be called the list s[x].

[0742] A multidimensional array is a variable with multiple dimensions. The elements of a multidimensional array can be indexed either by specifying all the required indices (such as variable[x][y][z]) or by a single index variable (which is itself a one-dimensional array of the specified indices). For example, variable[i], where i is a one-dimensional array with elements [x, y, z]. Multidimensional arrays are used to specify tensors, for example.

[0743] The values ​​of the entries in the rows and columns of an array can be specified using {{...}{...}}, where each pair of square brackets specifies the values ​​of the elements in the rows in increasing order of the columns, and the rows in increasing order of the rows. Thus, setting the matrix s to {{16}{49}} specifies that s[0][0] equals 1, s[1][0] equals 6, s[0][1] equals 4, and s[1][1] equals 9.

[0744] Binary notation is represented by enclosing a bit value string in single quotes. For example, '01000001' represents an 8-bit string where only the second and last bits (counting from the most significant bit to the least significant bit) are equal to 1.

[0745] When the number of bits is an integer multiple of 4, you can use hexadecimal notation instead of binary notation, that is, add "0x" before the hexadecimal number. For example, 0x41 represents an 8-bit string where only the second and last bits (counting from the most significant bit to the least significant bit) are equal to 1.

[0746] Numeric values ​​that are not enclosed in single quotes and not prefixed with "0x" are decimal values.

[0747] In a test statement, a value equal to 0 indicates a FALSE condition. Any value different from 0 indicates TRUE.

[0748] 5 Overview

[0749] 5.1 General

[0750] This clause provides an overview of the compression tools defined in this document and describes how to combine these tools with encodings.

[0751] 5.2 Compression tool (example; all details are optional)

[0752] This file contains the following groups of compression tools.

[0753] Parameter reduction methods process the model to obtain a compact representation. Examples of these methods include parameter sparsification, parameter pruning, weight unification, and factorization methods.

[0754] Sparsification processes parameters or groups of parameters to produce a sparse representation of the model, for example, by replacing some weight values ​​with zeros. Sparsification may generate additional metadata (e.g., masks). Sparsification may be structured or unstructured. This document includes unstructured sparsification methods with compressibility loss (e.g., methods disclosed in the corresponding subclauses), structured sparsification methods using microstructured sparsification (e.g., methods disclosed in the corresponding subclauses), unstructured statistical adaptive sparsification methods (e.g., methods disclosed in the corresponding subclauses), and structured sparsification methods (e.g., methods disclosed in the corresponding subclauses).

[0755] Unification processes parameters to produce a set of similar parameters. Unification does not eliminate or constrain weights to zero, but rather reduces the entropy of model parameters by making them similar to each other. This document includes methods for weight unification (such as those disclosed in the corresponding subclauses).

[0756] Pruning reduces the number of parameters by removing parameters or groups of parameters. For example, by removing some redundant convolutional filters from a layer, the process can result in a dense representation with fewer parameters than the original model. This document includes methods that combine pruning and sparsification (such as those disclosed in the corresponding subclauses).

[0757] Decomposition performs matrix factorization operations to change the structure of the model weights. This document includes low-rank / low-shifted-rank methods for convolutional and fully-connected layers (such as those disclosed in the corresponding subclauses).

[0758] In addition to the above reduction methods, this document also includes decomposition methods introduced and tested as part of parameter quantization techniques. Examples of these methods include batch normalization folding (such as disclosed in the corresponding subclause) and local scaling adaptation (such as disclosed in the corresponding subclause).

[0759] Parameter reduction methods can be used in combination or sequentially to obtain a compact model.

[0760] Parameter quantization methods reduce the precision of parameter representation. If supported by the inference engine, the quantized representation can be used for more efficient inference. This document includes unified quantization methods (such as disclosed in the corresponding subclause), codebook-based quantization methods (such as disclosed in the corresponding subclause), dependent scalar quantization methods (such as disclosed in the corresponding subclause), and iterative QP optimization methods (such as disclosed in the corresponding subclause).

[0761] The entropy coding method encodes the result of the parameter quantization method. This document includes DeepCABAC (subclause 9.1.1), and the entropy coding method supports extensions to DeepCABAC including row skipping and temporal context modeling.

[0762] Row skipping reduces the number of bits to be decoded and the size of the bitstream by skipping the decoding of matrix rows that are completely zero. To do this, row skipping signals a flag for each matrix row. The method is described in subclause 9.1.1.2.

[0763] Temporal context modeling improves DeepCABAC's context modeling by leveraging previously decoded incremental update information, thereby improving coding efficiency. This method is described in subclause 9.1.1.4.

[0764] 5.3 Create an encoding pipeline (example, all details are optional)

[0765] The compression tools in this article can be combined into different encoding pipelines. Some of these tools are alternatives for tackling neural network models with different types of features, while others are designed to work sequentially.

[0766] Fig.11 Shown are schematic overviews of encoding pipelines according to embodiments, which may optionally be assembled using the compression tools in this document. Fig.11 A schematic example of an NNR encoding pipeline according to an embodiment is shown. From the parameter conversion tool set, multiple tools can be applied in sequence. Parameter quantization can be applied to the source model or to the output of the conversion performed by the parameter reduction method. Entropy coding is usually applied to the quantized output. If necessary, the original output of the first few steps without entropy coding can be serialized.

[0767] The following encoding pipeline is a typical example of using this file:

[0768] 1. Dependency Scalar Quantization (such as disclosed in the corresponding subclause) - DeepCABAC (subclause 9.1.1)

[0769] 2. Sparsification (such as disclosed in the corresponding subclause) - Dependency Scalar Quantization (such as disclosed in the corresponding subclause) - DeepCABAC (subclause 9.1.1)

[0770] 3. Low-rank decomposition (such as disclosed in the corresponding subclause) - Dependency scalar quantization (such as disclosed in the corresponding subclause) - DeepCABAC (subclause 9.1.1)

[0771] 4. Codebook-based quantization (such as disclosed in the corresponding subclause) - DeepCABAC (subclause 9.1.1)

[0772] 5. Unified (e.g. as disclosed in the corresponding subclause) - DeepCABAC (subclause 9.1.1)

[0773] This list is not exhaustive.

[0774] 6 Syntax and Semantics (all details are optional)

[0775] [Editor's note (HT) after MPEG135: Changes regarding the newly added incremental weight updates in Section 8 may be introduced later]

[0776] 6.1 Syntax and Semantics Specifications

[0777] 6.1.1 Methods for specifying grammar in tabular form

[0778] The syntax table specifies a superset of all allowed bitstream syntaxes. Other clauses may directly or indirectly specify additional restrictions on the syntax.

[0779] Table 2 lists an example of the syntax specification format. When syntax_element appears, it specifies that the syntax element is parsed from the bitstream and the bitstream pointer is shifted forward to the next position outside the syntax element during the bitstream parsing process.

[0780] Table 2. Examples of syntax specification formats

[0781]

[0782]

[0783] 6.1.2 Bit sorting (example)

[0784] For bit-oriented transfer, the bit order of the syntax fields in the syntax table is specified starting from the MSB to the LSB.

[0785] 6.1.3 Specification of Syntax Functions and Data Types (Examples)

[0786] The functions given here are used for grammatical description. These functions are expressed with the value of a bitstream pointer, which indicates the location of the next bit to be read from the bitstream by the decoding process.

[0787] byte_aligned() is specified as follows:

[0788] ●If the current position in the bitstream is on a byte boundary, that is, the next bit in the bitstream is the first bit in the byte, the return value of byte_aligned() is equal to TRUE.

[0789] ●Otherwise, the return value of byte_aligned() is FALSE.

[0790] read_bits(n) reads the next n bits from the bitstream and shifts the bitstream pointer forward by n bits. When n is equal to 0, read_bits(n) is specified to return a value equal to 0 and not shift the bitstream pointer forward.

[0791] get_bit_pointer() returns the unsigned integer value of the bitstream pointer relative to the start of the current NNR unit. get_bit_pointer()>>3 points to the current byte of the bitstream pointer. get_bit_pointer()&7 points to the current bit of the current byte of the bitstream pointer, where 0 indicates the most significant bit.

[0792] set_bit_pointer(pos) sets the position of the bitstream pointer so that get_bit_pointer() is equal to pos.

[0793] The following data types specify the parsing process for each syntax element.

[0794] ae(v): Context-adaptive arithmetic entropy coded syntax element. The parsing process for this data type is specified in subclause 9.3.4.3.2.

[0795] at(v): The termination syntax for arithmetic entropy coding. The parsing process for data types is specified in subclause 9.3.4.3.5.

[0796] iae(n): a signed integer of n arithmetic entropy coded bits using the DeepCABAC bypass mode as specified in subclause 9.3.4.3.4. The read bypass partition is interpreted as a binary integer representation with the most significant bit written first.

[0797] uae(n): unsigned integer of n arithmetic entropy coded bits using DeepCABAC bypass mode as specified in subclause 9.3.4.3.4. The read bypass partition is interpreted as the binary representation of the unsigned integer, with the most significant bit written first. When n=0, uae(n) does not decode any partition and returns 0.

[0798] f(n): A fixed-pattern bit string that uses n bits (from left to right), writing the left bit first. The parsing process for this data type is specified by the return value of the function read_bits(n).

[0799] i(n): A signed integer using n bits. When n is "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. The parsing of this data type is specified by the return value of the function read_bits(n), which is interpreted as a binary integer representation with the most significant bit written first.

[0800] u(n): An unsigned integer using n bits. When n is "v" in the syntax table, the number of bits changes depending on the values ​​of other syntax elements. The parsing process of this data type is specified by the return value of the function read_bits(n), which is interpreted as the binary representation of the unsigned integer, where the most significant bit is written first.

[0801] ue(k): unsigned integer k-order Golomb coding syntax element. The parsing process of the gai descriptor is shown in the following pseudo code, and the result is x:

[0802]

[0803] ie(k): signed integer k-order Golomb coding syntax element. The parsing process of this descriptor is shown in the following pseudo code, and the result is x:

[0804]

[0805] flt(n) : Floating-point value using n bits, where n can be 32, 64, or 128, in little-endian order as specified in ISO / IEC 60559, binary32, binary64, or binary128, respectively.

[0806] st(v): A null-terminated string that must be encoded as UTF-8 characters according to the ISO / IEC10646 standard. The parsing process is as follows: st(v) starts at a byte-aligned position in the bit stream, reads and returns a series of bytes from the bit stream, starting from the current position until but not including the next byte-aligned byte equal to 0x00, and shifts the bit stream pointer forward by (stringLength+1)*8 bit positions, where stringLength is equal to the number of bytes returned.

[0807] Note that the st(v) and flt(n) syntax descriptors are used in this document only when the current position in the bitstream is a byte-aligned position.

[0808] bs(v): Byte-sequence specifies a variable-length byte sequence, starting at a byte-aligned position. The length of the sequence is determined by the size of the NNR unit containing the byte sequence.

[0809] The definition of more_data_in_nnr_unit() is as follows:

[0810] - If there is more data in the current nnr_unit, that is, the data decoded so far in the current nnr_unit is less than numBytesInNNRUnit, the return value of more_data_in_nnr_unit() is equal to TRUE.

[0811] Otherwise, the return value of more_data_in_nnr_unit() is equal to FALSE.

[0812] 6.1.4 Semantics (Example)

[0813] The semantics associated with grammatical structures and the grammatical elements within each structure are specified in subclauses following the subclause containing the grammatical structure.

[0814] The following definitions apply to the semantic specification.

[0815] unspecified is used to designate certain values ​​of a particular syntax element to indicate that those values ​​have no special meaning in this document and will not have a special meaning in the future as part of a future version of this document.

[0816] reserved is used to designate certain values ​​of particular syntax elements for future use by ISO / IEC and shall not be used in bitstreams conforming to this version of this document, but may be used in bitstreams conforming to future extensions of this document by ISO / IEC.

[0817] nnr_reserved_zero_0bit shall be an element of length 0. The decoder shall ignore the value of nnr_reserved_zero_0bit.

[0818] The nnr_reserved_zero_1 bit, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of the nnr_reserved_zero_1 bit are reserved for future use by ISO / IEC. A decoder shall ignore the value of the nnr_reserved_zero_1 bit.

[0819] nnr_reserved_zero_2bits, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_2bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of nnr_reserved_zero_2bits.

[0820] nnr_reserved_zero_3bits, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_3bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of nnr_reserved_zero_3bits.

[0821] nnr_reserved_zero_5bits, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_5bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of nnr_reserved_zero_5bits.

[0822] nnr_reserved_zero_7bits, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_7bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of nnr_reserved_zero_7bits.

[0823] 6.2 General bitstream syntax elements (examples, all details are optional)

[0824] 6.2.1NNR Unit

[0825] For example, an NNR unit may be a data structure for carrying neural network data and associated metadata that are compressed or represented using this file.

[0826] For example, an NNR unit carries compressed or uncompressed information, such as information about neural network metadata, topology information, complete or partial layer data, filters, kernels, biases, quantized weights, tensors, or the like.

[0827] For example, the NNR unit consists of the following data elements (such as Fig.12 shown). Fig.12 Schematic diagram example showing a NNR unit data structure according to an embodiment):

[0828] ●NNR unit size (optional): This data element indicates, for example, the total byte size of the NNR unit, including the NNR unit size.

[0829] ●NNR unit header (optional): This data element contains information such as the NNR unit type and related metadata.

[0830] NNR Unit Payload: This data element contains compressed or uncompressed data related to the neural network.

[0831] 6.2.2 Polymerization of NNR units (optional, example)

[0832] For example, an aggregate NNR unit is an NNR unit that carries multiple NNR units in its payload. For example, an aggregate NNR unit provides a grouping mechanism for multiple mutually related NNR units that can benefit from being aggregated into one NNR unit (e.g. Fig.13 shown. Fig.13 Schematic example of aggregated NNR unit data structure is shown).

[0833] 6.2.3 Composition of NNR bitstream (optional; example)

[0834] For example, an NNR bitstream consists of a series of NNR units (such as Fig.14 shown). Fig.14 Schematic example of an NNR bitstream data structure according to an embodiment is shown).

[0835] For example, in an NNR bitstream, the following restrictions apply unless otherwise specified in this document or defined by the NNR profile:

[0836] (For example, NNR_STR, NNR_MPS, NNR_NDU, NNR_LPS, NNR_TPL, and NNR_QNT are NNR unit types as specified, for example, in Table 3 of subclause 6.4.3)

[0837] For example, an NNR bitstream shall start with an NNR start unit (NNR_STR) (subclause 6.4.3).

[0838] For example, there shall be a single NNR model parameter set (NNR_MPS) (subclause 6.4.3) in the NNR bitstream, which shall precede any NNR_NDU (subclause 6.4.3) in the NNR bitstream.

[0839] For example, an NNR layer parameter set (NNR_LPS) shall remain valid until the next NNR layer parameter set in the NNR bitstream or until the boundary of an aggregated NNR unit is reached.

[0840] For example, topology_elem_id and topology_elem_id_index (subclause 6.4.3.7) values ​​shall be unique in an NNR bitstream.

[0841] The NNR_TPL or NNR_QNT unit; if present in the NNR bitstream; shall precede any NNR_NDU that references its data structure (e.g. topology_elem_id).

[0842] 6.3 NNR bitstream syntax (example, all details are optional)

[0843] 6.3.1 NNR unit syntax (example)

[0844]

[0845] 6.3.2 NNR Unit Size Syntax (Details are optional)

[0846] 6.3.3 NNR unit header syntax (details are optional)

[0847] 6.3.3.1 General

[0848]

[0849] 6.3.3.2 NNR Start Unit Header Syntax (Details are Optional)

[0850]

[0851]

[0852] 6.3.3.3 NNR Model Parameter Set Unit Header Syntax (Details are optional)

[0853] 6.3.3.4 NNR layer parameter setting unit header syntax (details are optional)

[0854] 6.3.3.5 NNR Topology Unit Header Syntax (Details are optional)

[0855] Descriptors u(8) u(8)

[0856] 6.3.3.6 NNR Quantization Unit Header Syntax (Details are optional)

[0857] Descriptors u(8) u(8)

[0858] 6.3.3.7 NNR Compressed Data Unit Header Syntax (Example, all details are optional)

[0859]

[0860]

[0861]

[0862] For example, integer_codebook() is defined as follows (all details are optional):

[0863]

[0864] For example, tensor_dimension_list() is defined as follows (all details are optional):

[0865]

[0866] For example, topology_elements_ids_list(topologyIndexedFlag) is defined as follows (all details are optional):

[0867]

[0868] For example, topology_tensor_dimension_mapping() is defined as follows (all details are optional):

[0869]

[0870] 6.3.3.8 NNR aggregation unit header syntax (details are optional)

[0871]

[0872]

[0873] 6.3.4 NNR unit payload syntax (example, all details are optional)

[0874] 6.3.4.1 General (ie, all details are optional)

[0875]

[0876] 6.3.4.2 NNR Start Unit Payload Syntax (all details are optional)

[0877] Descriptors u(0)

[0878] 6.3.4.3 NNR Model Parameter Set Unit Payload Syntax (all details are optional)

[0879] 6.3.4.4 NNR layer parameter set unit payload syntax (all details are optional)

[0880] 6.3.4.5 NNR Topology Unit Payload Syntax (all details are optional)

[0881] 6.3.4.6 NNR Quantization Unit Payload Syntax (all details are optional)

[0882]

[0883] 6.3.4.7 NNR Compressed Data Unit Payload Syntax (Example, details are optional)

[0884]

[0885]

[0886] For example, decode_compressed_data_unit_payload() calls a decoding process such as specified in subclause 7.3.

[0887] 6.3.4.8 NNR Aggregation Unit Payload Syntax (all details are optional)

[0888] 6.3.5 Byte Alignment Syntax (all details are optional)

[0889] 6.4 Semantics

[0890] 6.4.1 General

[0891] For example, semantics associated with syntax structures and elements within those structures are specified in this subclause. For example, when the semantics of a syntax element is specified by means of a table or a set of tables, any value not specified in the table shall not appear in the bitstream unless otherwise specified in this document.

[0892] 6.4.2 NNR unit size semantics (all details are optional)

[0893] 6.4.3 NNR unit header semantics (example, all details are optional)

[0894] 6.4.3.1 General (Example: All details are optional)

[0895] For example, nnr_unit_type specifies the type of the NNR unit, as shown in Table 3.

[0896] Table 3: NNR unit types (example)

[0897]

[0898] For example, values ​​in the range NNR_RSVD are reserved for use in future versions of this specification or related specifications. Encoders MUST NOT use these values. For example, decoders conforming to this version of this specification MAY ignore NNR units that use these values. Values ​​in the range NNR_UNSP are unspecified and their use is outside the scope of this specification. For example, decoders conforming to this version of this specification MAY ignore NNR units that use these values.

[0899] For example, independently_decodable_flag specifies whether the compressed data unit is independently decodable. For example, a value of 1 indicates an independently decodable NNR unit. For example, a value of 0 indicates that the NNR unit is not independently decodable and its payload should be combined with other NNR units for successful decoding / decompression. For example, the value of independently_decodable_flag should be the same for all NNR units that reference the same topology_elem_id or topology_elem_id_index value or the same topology_elem_id_list.

[0900] For example, partial_data_counter_present_flag equal to 1 indicates that the syntax element partial_data_counter is present in the NNR unit header. For example, partial_data_counter_present_flag equal to 0 indicates that the syntax element partial_data_counter is not present in the NNR unit header.

[0901] For example, partial_data_counter specifies the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a topological element. For example, a value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topological element and is complete), and for example, a value greater than 0 indicates the index of partial information (i.e., the data in the NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 1). For example, the counter counts backward, initially indicating the total number of partitions. For example, if not present, the value of partial_data_counter is inferred to be 0. For example, if the value of independently_decodable_flag is equal to 0, the value of partial_data_counter_present_flag should be equal to 1, and the value of partial_data_counter should be greater than 0. For example, in this version of this document, if the value of independently_decodable_flag is equal to 1, then the values ​​of partial_data_counter_present_flag and partial_data_counter are undefined.

[0902] NOTE - In future versions of this document, if the value of independently_decodable_flag is equal to 1 and partial_data_counter_present_flag is equal to 1, partial_data_counter may, for example, have a non-zero value, based on the assumption that multiple independently decodable NNR units are combined to build one model.

[0903] 6.4.3.2 NNR Start Unit Header Semantics (Details are Optional)

[0904] 6.4.3.3 NNR Model Parameter Set Unit Header Semantics (Details are Optional)

[0905] 6.4.3.4 NNR layer parameter setting unit header semantics (details are optional)

[0906] 6.4.3.5 NNR Topology Unit Header Semantics (Details are Optional)

[0907] 6.4.3.6 NNR Quantization Unit Header Semantics (Details are Optional)

[0908] 6.4.3.7 NNR Compressed Data Unit Header Semantics (Example, Details Optional)

[0909] For example, nnr_compressed_data_unit_payload_type (optional) is defined in Table 7 of Subclause 7.3.

[0910] For example, nnr_multiple_topology_elements_present_flag (optional) specifies whether there are multiple topology units in the bitstream. If there are multiple topology units, a list of their IDs is included. When nnr_compressed_data_unit_payload_type is set to NNR_PT_BLOCK, this flag is set to 1 and the topology_elements_ids_list() in the NNR compressed data unit header shall list the topology elements or topology element indices of RecWeight, RecWeightG, RecWeightH, RecLS, RecBeta, RecGamma, RecMean, RecVar and RecBias in the given order according to the presence indicated by the value of compressed_parameter_type in the NNR compressed data unit header.

[0911] For example, nnr_decompressed_data_format_present_flag (optional) specifies whether the data format obtained after decompression is present in the bitstream.

[0912] For example, input_parameters_present_flag (optional) specifies whether a group of elements including tensor dimensions, DeepCABAC unit length, and compression parameter type is present in the bitstream.

[0913] For example, topology_elem_id (optional) specifies the unique identifier of the topology element to which the NNR compressed data unit refers. The semantic interpretation of this field depends on the context.

[0914] For example, topology_elem_id_index (optional) specifies the unique index value of the topology element that is signaled in the topology information with a payload type of NNR_TPL_REFLIST. The first index should be 0 (ie, 0 index).

[0915] For example, node_id_present_flag (optional) equal to 1 indicates that the syntax elements device_id, parameter_id, and put_node_depth are present.

[0916] For example: device_id (optional) uniquely identifies the device that generated the current NDU.

[0917] For example, parameter_id (optional) uniquely identifies the parameter of the model to which the tensor stored in the NDU is associated. If parent_node_id_type = ICNN_NDU_ID, parameter_id should be equal to the parameter_id of the corresponding parent NDU.

[0918] For example, put_node_depth (optional) is the depth of the tree where the current NDU is located. Depth 0 corresponds to the root node. If parent_node_id_type = ICNN_NDU_ID, then put_node_depth-1 must be equal to the put_node_depth of the associated parent NDU.

[0919] For example, parent_node_id_present_flag (optional) equal to 1 indicates that the syntax element parent_node_id_type is present.

[0920] For example, parent_node_id_type (optional) specifies the parent node id type. It indicates which syntax elements are used to uniquely identify the parent node. The allowed values ​​of parent_node_id_type are defined in Table 4.

[0921] Table 4: Parent node id type identifier. (Example)

[0922]

[0923] For example, temporal_context_modeling_flag (optional) specifies whether temporal context modeling is enabled. If temporal_context_modeling_flag is equal to 1, it indicates that temporal context modeling is enabled. If temporal_context_modeling_flag does not exist, it is inferred to be 0.

[0924] For example, parent_device_id (optional) is equal to the syntax element device_id of the parent NDU.

[0925] For example, parent_node_payload_sha256 (optional) is the SHA256 hash of the parent NDU's nnr_compressed_data_unit_payload.

[0926] For example, parent_node_payload_sha512 (optional) is the SHA512 hash of the parent NDU's nnr_compressed_data_unit_payload.

[0927] For example, count_topology_elements_minus2+2 (optional) specifies the number of topology elements that carry data in the payload of this NNR compressed data unit.

[0928] For example, codebook_present_flag (optional) specifies whether a codebook is used. If codebook_present_flag is not present, it is inferred to be 0.

[0929] For example, dq_flag (optional) specifies whether the quantization method is uniform quantization according to subclause Error! Reference source not found. dq_flag equal to 0 indicates that the uniform quantization method is used. dq_flag equal to 1 indicates that the dependent scalar quantization method is used. If dq_flag is not present, it is inferred to be 0.

[0930] For example, nnr_decompressed_data_format (optional) is defined in Table 6 of Subclause 7.2.

[0931] For example, tensor_dimensions_flag (optional) specifies whether the tensor dimensions are defined in the bitstream. If not, they should be obtained from the model topology description.

[0932] For example, cabac_unary_length_flag (optional) specifies whether the unary part length in DeepCABAC binarization is included in the bitstream.

[0933] For example, compressed_parameter_types (optional) specifies the compressed parameter types present in the current topology element referenced by the NNR compressed data unit. If multiple compressed parameter types are specified, they are combined using OR. The compressed parameter types are defined in Table 5.

[0934] Table 5: Compressed parameter type identifiers. (Example)

[0935]

[0936] For example, when there is a decomposition, the tensors G and H represent the results of the decomposition of the original tensor. If (compressed_parameter_types & NNR_CPT_DC)! = 0, the variables TensorDimensionsG and TensorDimensionsH are derived, as shown below:

[0937] - The variable TensorDimensionsG is set to [g_number_of_rows, decomposition_rank].

[0938] -The variable TensorDimensionsH is set to [decomposition_rank, hNumberOfColumns], where hNumberOfColumns is defined as

[0939]

[0940] For example, if (compressed_parameter_types & NNR_CPT_DC) != 0, and if nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, then the NNR unit contains the decomposed tensor G, and the next NNR unit in the bitstream contains the corresponding decomposed tensor H.

[0941] For example, the variable TensorDimensions is derived as follows:

[0942] - If the NNR unit contains a decomposed tensor G and nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, then TensorDimensions is set to TensorDimensionsG.

[0943] Otherwise, if the NNR unit contains a decomposed tensor H and nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, then TensorDimensions is set to TensorDimensionsH.

[0944] Otherwise, set TensorDimensions to tensor_dimensions.

[0945] For example, the variable NumBlockRowsMinus1 is defined as follows:

[0946] -If scan_order=0, NumBlockRowsMinus1 is set to 0.

[0947] - Otherwise, if nnr_compressed_data_unit_payload_type == NNR_PT_BLOCK and (compressed_parameter_types & NNR_CPT_DC)! = 0, then NumBlockRowsMinus1 is set to ((TensorDimensionsG[0] + (4 <<scan_order)-1)> >(2+scan_order))+((TensorDimensionsH[0]+(4<<scan_order)-1)> >(2+scan_order))-2.

[0948] Otherwise, NumBlockRowsMinus1 is set to ((TensorDimensions[0]+(4<<scan_order)-1)> >(2+scan_order))-1.

[0949] For example, decomposition_rank (optional) specifies the rank of the low-rank decomposition weight tensor components relative to tensor_dimensions.

[0950] For example, in the case of reconstructing a tensor of NNR unit decomposition of type NNR_PT_BLOCK, g_number_of_rows (optional) specifies the number of rows of the matrix G.

[0951] For example, cabac_unary_length_minus1 (optional) specifies the length of the unary part in DeepCABAC binarization minus 1.

[0952] For example, first_tensor_dimension_shift (optional, present in some embodiments) specifies the shift of the first tensor dimension for tensor dimension reordering and should be less than the value of count_tensor_dimensions. If first_tensor_dimension_shift is not present, it is inferred to be 0.

[0953] For example, scan_order (optionally reset in some embodiments) specifies the block scan order for parameters with more than 1 dimension, for example, according to the following table:

[0954] -0: Do not scan blocks

[0955] -1: 8x8 block

[0956] -2: 16x16 fast

[0957] -3: 32x32 blocks

[0958] -4: 64x64 blocks

[0959] For example, cabac_offset_list (optional) specifies a list of values ​​used to initialize the variable IvlOffset at the beginning of the entry point.

[0960] For example, dq_state_list (optional) specifies a list of values ​​used to initialize the variable stateId at the beginning of the entry point.

[0961] For example, bit_offset_delta1 (optional) specifies the first element of the list BitOffsetList.

[0962] For example, bit_offset_delta2 (optional) specifies the difference between the elements in the list BitOffsetList except the first element and the previous element in the list BitOffsetList.

[0963] For example, the variable BitOffsetList (optional) is a list of bit offsets used to set the position of the bitstream pointer at the beginning of the entry point.

[0964] For example, codebook_egk (optional) specifies the Exp-Golomb parameter k used for decoding the syntax elements codebook_delta_left and codebook_delta_right.

[0965] For example, codebook_size (optional) specifies the number of elements in the codebook.

[0966] For example, codebook_centre_offset (optional) specifies the offset of accessing elements in the codebook relative to the center of the codebook. It is used to calculate the variable CbZeroOffset.

[0967] For example, codebook_zero_value (optional) specifies the value of the codebook at the CbZeroOffset position. It participates in creating the variable Codebook (an array representing the codebook).

[0968] For example, codebook_delta_left (optional) specifies the difference between a codebook value and its right neighbor minus 1 (for codebook values ​​to the left of the center position). It contributes to the creation of the variable Codebook (an array representing the codebook).

[0969] For example, codebook_delta_right (optional) specifies the difference between a codebook value and its left neighbor minus 1 (for codebook values ​​to the right of the center position). It contributes to the creation of the variable Codebook (an array representing the codebook).

[0970] For example, count_tensor_dimensions (optional) specifies a counter for a number of dimensions. For example, for a 4D tensor, count_tensor_dimensions is 4. If not included in the bitstream, it needs to be obtained from the model topology description.

[0971] For example, tensor_dimensions (optional) specifies an array or list of dimension values. For example, for a convolutional layer, tensor_dimensions is an array or list of length 4. For the NNR unit of the tensor element G or H of the shoelace decomposition, tensor_dimensions is set to the dimensions of the original tensor. The actual tensor dimensions of G and H in the decoding method are derived from tensor_dimensions, decomposition_rank, and g_number_of_rows. If not included in the bitstream, they should be obtained from the model topology description. For example, refer to Figure 6, where tensor_dimensions of tensor 680 can be the vector [D0, D1, D2, D3].

[0972] For example, topology_elem_id_list (optional) specifies a list of unique identifiers associated with the topology elements referenced by the NNR compressed data unit. The elements of topology_elem_id_list are semantically equivalent to the syntax elements topology_elem_id or their indexes listed in the topology payload of type NNR_TPL_REFLIST. The semantic interpretation of this field depends on the context.

[0973] For example, topology_elem_id_index_list (optional) specifies a list of unique indices associated with the topology elements listed in the topology information with a payload type of NNR_TPL_REFLIST. The index value of the first element in the topology should be 0.

[0974] For example, concatentation_axis_index (optional) indicates the 0-based concatenation axis.

[0975] For example, split_index[] (optional) indicates the tensor split index along the connection axis indicated by concatentation_axis_index, so as to generate each connected independent tensor.

[0976] For example, number_of_shifts[] (optional) indicates how many left shift operations to perform.

[0977] For example, shift_index[k][i] (optional) indicates the axis index of the k-th topology element to be shifted left.

[0978] For example, shift_value[k][i] (optional) indicates the magnitude of the left shift on the axis of index[k][i].

[0979] 6.4.3.8 NNR aggregation unit header semantics (details are optional)

[0980] 6.4.4 NNR unit payload semantics

[0981] 6.4.4.1 General

[0982] For example, specifying the following NNR unit payload type:

[0983] 6.4.4.2 NNR Start Unit Payload Semantics (Details are Optional)

[0984] 6.4.4.3 NNR model parameter set unit payload semantics (details are optional)

[0985] 6.4.4.4 NNR layer parameter setting unit payload semantics (details are optional)

[0986] 6.4.4.5 NNR topology unit payload semantics (optional)

[0987] 6.4.4.6 NNR Quantization Unit Payload Semantics (Details are Optional)

[0988] 6.4.4.7 NNR Compressed Data Unit Payload Semantics (Example, details are optional)

[0989] Raw_float32_parameter is a floating point parameter tensor.

[0990] 6.4.4.8 NNR aggregation unit payload semantics (details are optional)

[0991] 7Decoding process

[0992] 7.1 General (examples, details are optional)

[0993] For example, a decoder conforming to this document shall take as input the NNR bitstream specified in subclause 6.3, and

[0994] Generate decompressed data, for example, conforming to the NNR decompressed data format (as defined in Table 6), or

[0995] ● Generate ASCII or compressed data output (optional) such as via NNR_TPL and NNR_QNTNNR unit payloads (as described in subclause 6.3.3)

[0996] For example, for the decoding process, the following conditions should be met:

[0997] ● For example, any information required for decoding the NNR unit of the NNR bitstream should be signaled as part of the NNR bitstream. If this information is not part of the NNR bitstream, it should be provided to the decoding process in other ways (for example, out-of-band topology information or parameters required for decoding but not signaled or carried in the NNR bitstream).

[0998] ● For example, the decoding process shall be initiated by an NNR unit of type NNR_STR. Upon receiving the NNR_STR unit, the decoder shall reset its internal state and be ready to receive the NNR bitstream. The presence and cardinality of the above NNR units shall be specified in the relevant clauses and annexes of this document.

[0999] NOTE: For example, the decoder can be further initialized by an NNR unit of type NNR_MPS to set the global neural network model parameters.

[1000] For example, a decoder that complies with the requirements of this document outputs a data structure that complies with the decompressed NNR data format immediately after decompression. Thus, for example, the delay between inputting an NNR compressed data unit and accessing the decompressed data structure from its output is short. How to establish the relationship between the input NNR unit and the NNR decompressed output data is beyond the scope of this document and is left to the implementation.

[1001] 7.2 NNR decompressed data format (example, details are optional)

[1002] For example, depending on the compression method used to create a particular bitstream, an NNR decoder is expected to output different decompressed data formats after decoding an NNR data unit. For example, Table 6 specifies these NNR decompressed data formats, such as those produced after decompressing an NNR compressed data unit.

[1003] Table 6 NNR decompression data format (example)

[1004]

[1005] 7.3 Decoding method (example, details are optional)

[1006] 7.3.1 General (details are optional)

[1007] This clause specifies, for example, a decoding method for this file. For example, depending on the value of nnr_compressed_data_unit_payload_type, the subclause specified in Table 7 is called.

[1008] Table 7 NNR Compressed Data Payload Type (Example, Details are Optional)

[1009]

[1010] For example, if the payload identifier is NNR_PT_INT, NNR_PT_FLOAT, or NNR_PT_FLOAT_RAW and multiple topology elements are combined (e.g., as signaled in the NNR compressed data unit header, e.g., via nnr_multiple_topology_elements_present_flag), then the NNR decompressed tensor should be further split into multiple tensors after the decoding process, e.g., as shown below:

[1011] ● Split the tensor RecParam into multiple tensors, for example, by calling TensorSplit(RecParam, split_index, concatenation_axis_index).

[1012] • For example, the output of the function TensorSplit is a list of split output tensors associated with the topology elements specified, for example, by topology_elem_id_list.

[1013] ●For example, in topology_tensor_dimension_mapping(), the output tensor is further processed by calling AxisSwap() to swap the axes of the output tensor.

[1014] 7.3.1.1 Tensor Dimension Reordering Using First Tensor Dimension Shift (Example, Details Optional)

[1015] For example, the input to this process is:

[1016] ● variable inputTensor , i.e. the tensor whose dimensions should be reordered

[1017] ●Variable inputTensorDims, for example, specifies the dimensions of inputTensor

[1018] The output of this process is, for example, a variable reorderedTensor whose dimensions are equal to ShiftArrayIndex(inputTensorDims, first_tensor_dimension_shift). For example, the elements of the variable reorderedTensor are set as follows:

[1019]

[1020] Therefore, in general, decoders such as Figure 1 As shown in 100, Figure 2 The illustrated 200 , for example, its corresponding reordering units 120 , 220 , may be configured to obtain a reordered multi-dimensional array, for example 121 , 221 , using the above-described processing.

[1021] 7.3.2 Decoding method for NNR compressed payload of type NNR_PT_INT (example, details are optional)

[1022] For example, the input to this process is:

[1023] • One or more NNR compressed data units marked as being decompressed together, for example by the partial_data_counter and nnr_compressed_data_unit_payload_type fields, are set to NNR_PT_INT, for example.

[1024] For example, the output of this process is a variable RecParam of type TENSOR_INT, as shown in Table 6. For example, the dimension of RecParam is equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift). For example, when decoding a bitstream conforming to method NNR_PT_INT, only values ​​that can be represented as 32-bit binary integer values ​​can be generated for RecParam.

[1025] For example, optionally, the arithmetic coding engine and context model are initialized as specified in subclause 9.3.2.

[1026] For example, optionally, the syntax structure shift_parameter_ids(cabac_unary_length_minus1) is decoded from the bitstream according to subclause 9.2.1.6, and the probability estimation parameter initialization procedure specified in subclause 9.3.2.2 is called.

[1027] For example, optionally, the syntax structure quant_tensor(TensorDimensions, cabac_unary_length_minus1, 0) is decoded from the bitstream according to subclause 9.2.1.4, for example, with RecParam being set equal to QuantParam.

[1028] For example, optionally, the syntax structure terminate_cabac() is decoded from the bitstream according to subclause 9.2.1.2.

[1029] For example, optionally, call Subclause 7.3.1.1 with RecParam and TensorDimensions as input, and assign the output to RecParam.

[1030] 7.3.3 Decoding method for NNR compressed payload of type NNR_PT_FLOAT (example, details are optional)

[1031] For example, the input to this process is:

[1032] • One or more NNR compressed data units marked as being decompressed together by partial_data_counter and, for example, their nnr_compressed_data_unit_payload_type field are set to NNR_PT_FLOAT.

[1033] For example, the output of the process is a variable RecParam of type TENSOR_FLOAT, as shown in Table 6. For example, the dimension of RecParam is equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift).

[1034] For example, the arithmetic coding engine and context model are initialized as specified in subclause 9.3.2.

[1035] For example, optionally, call subclause 7.3.6, for example with TensorDimensions, 0 and (codebook_present_flag?0:1) as input, for example, and assign the output to RecParam.

[1036] For example, optionally, the syntax structure terminate_cabac() is decoded from the bitstream according to subclause 9.2.1.2.

[1037] For example, optionally, invoke subclause 7.3.1.1, for example with RecParam and TensorDimensions as input, and assigning the output to RecParam.

[1038] For example, when decoding a bitstream conforming to method NNR_PT_FLOAT, only values ​​that can be represented as floating point values ​​without loss of precision may be generated for RecParam.

[1039] 7.3.4 Decoding method for NNR compressed payload of type NNR_PT_RAW_FLOAT (example, details are optional)

[1040] For example, the output of the process is a variable RecParam of type TENSOR_FLOAT, such as shown in Table 6. For example, the dimension of RecParam is equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift).

[1041] For example, RecParam is set equal to raw_float32_parameter.

[1042] For example, optionally, invoke subclause 7.3.1.1, for example with RecParam and TensorDimensions as input, and assigning the output to RecParam.

[1043] 7.3.5 Decoding method for NNR compressed payload of type NNR_PT_BLOCK (example, details are optional)

[1044] For example, the input to this process is:

[1045] • One or more NNR compressed data units marked to be decompressed together, for example by partial_data_counter and their nnr_compressed_data_unit_payload_type fields are set to NNR_PT_BLOCK.

[1046] For example, the output of the process is one or more variables of type TENSOR_FLOAT (as shown in Table 6), for example, according to the value of compressed_parameter_types, as follows:

[1047] If (compressed_parameter_types & NNR_CPT_DC) == 0: RecWeight (optional, example)

[1048] If (compressed_parameter_types & NNR_CPT_DC)! = 0: RecWeightG, RecWeightH (optional, example)

[1049] If (compressed_parameter_types & NNR_CPT_LS) != 0: RecLS (optional, example)

[1050] If (compressed_parameter_types & NNR_CPT_BN) != 0: RecBeta, RecGamma, RecMean, RecVar (optional, example)

[1051] If (compressed_parameter_types & NNR_CPT_BI) != 0: RecBias (optional, example)

[1052] If present, the dimensions of RecWeight are equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift).

[1053] (optional, example)

[1054] If present, the dimensions of RecWeightG are equal to TensorDimensionsG (optional, example).

[1055] If present, the dimensions of RecWeightH are equal to TensorDimensionsH (optional, example).

[1056] If present, the variables RecLS, RecBeta, RecGamma, RecMean, RecVar, and RecBias are one-dimensional with length equal to the first dimension of TensorDimensions (optional, for example).

[1057] For example, optionally, the arithmetic coding engine and context model are initialized as specified in subclause 9.3.2.

[1058] If (compressed_parameter_types & NNR_CPT_LS) != 0, then subclause 7.3.6 is called with dimensions of RecLS, -1 and 1 as input, and the output is assigned to RecLS (optional, example).

[1059] If (compressed_parameter_types & NNR_CPT_BI) != 0, then subclause 7.3.6 is called with dimensions of RecBias, -1 and -1 as input, and the output is assigned to RecBias (optional, example).

[1060] If (compressed_parameter_types & NNR_CPT_BN) != 0, then subclause 7.3.6 is called with RecBeta, -1 and -1 dimensions as input, and the output is assigned to RecBeta (optional, example).

[1061] If (compressed_parameter_types & NNR_CPT_BN) != 0, then subclause 7.3.6 is called with RecGamma, 1 and -1 dimensions as input, and the output is assigned to RecGamma (optional, example).

[1062] If (compressed_parameter_types & NNR_CPT_BN) != 0, then subclause 7.3.6 is called with RecMean, 1 and -1 dimensions as input, and the output is assigned to RecMean (optional, example).

[1063] If (compressed_parameter_types & NNR_CPT_BN) != 0, then subclause 7.3.6 is called with RecVar, 1 and -1 dimensions as input, and the output is assigned to RecVar (optional, example).

[1064] If (compressed_parameter_types & NNR_CPT_DC) == 0, then subclause 7.3.6 is called with RecWeight, 0 and (codebook_present_flag?0:1) dimensions as input, and the output is assigned to RecWeight (optional, example).

[1065] If (compressed_parameter_types & NNR_CPT_DC) != 0, then execute the following: (optional, example)

[1066] For example, calling subclause 7.3.6, for example with TensorDimensionsG, 0 and (codebook_present_flag?0:-1) as input, the output is assigned to RecWeightG.

[1067] For example, calling subclause 7.3.6, for example with TensorDimensionsH, (TensorDimensionsG[0]+(4<<scan_order)-1)> >(2+scan_order))-1 and (codebook_present_flag?1:-1) are taken as input, for example, the output is assigned to RecWeightH.

[1068] Note—For example, from the decoded RecWeightG and RecWeightH, the variable RecWeight can be derived as follows:

[1069] RecWeight=TensorReshape(RecWeightG*RecWeightH,TensorDimensions)

[1070] For example, optionally, invoke subclause 7.3.1.1, for example with RecWeight and TensorDimensions as input, and assign the output to RecWeight.

[1071] For example, optionally, the syntax structure terminate_cabac() is decoded from the bitstream according to subclause 9.2.1.2.

[1072] For example, optionally, if (compressed_parameter_types & NNR_CPT_DC) == 0, then invoke subclause 7.3.1.1, for example with RecWeight and TensorDimensions as input, and assign the output to RecWeight.

[1073] 7.3.6 Decoding process of integer weight tensors (eg, details are optional)

[1074] For example, the input to this process is:

[1075] ●Variable tensorDims, specifies the dimensions of the tensor to decode (optional, example).

[1076] ●Variable entryPointOffset, indicating whether there is an entry point for decoding, and if so, the entry point offset (optional, example).

[1077] - A variable codebookld indicating whether a codebook is applied, and if so, which codebook should be used (optional, example).

[1078] For example, the output of this process is a variable recParam of type TENSOR_FLOAT, as shown in Table 6, whose dimension is equal to tensorDims.

[1079] For example, the syntax structure quant_param(QpDensity) according to subclause 9.2.1.3 is optionally decoded from the bitstream.

[1080] For example, optionally, the syntax structure shift_parameter_ids(cabac_unary_length_minus1) according to subclause 9.2.1.6 is decoded from the bitstream and the initialization procedure of the probability estimation parameters specified in subclause 9.3.2.2 is called, for example.

[1081] For example, optionally, the syntax structure quant_tensor(tensorDims, cabac_unary_length_minus1, entryPointOffset) according to subclause 9.2.1.4 is decoded from the bitstream, and recParam is set as follows (example):

[1082]

[1083] For example, optionally, the variable stepSize is derived as follows:

[1084] mul=(1< <QpDensity)+((qp_value+QuantizationParameter)&((1<<QpDensity)-1))

[1085] shift=(qp_value+QuantizationParameter)>>QpDensity

[1086] stepSize = mul * 2 shift-QpDensity

[1087] For example, optionally, the variable recParam is updated as follows:

[1088] recParam = recParam * stepSize

[1089] Note - recParam can always be represented as a binary fraction, for example, based on the above calculations.

[1090] 8 parameter reduction (details are optional)

[1091] 9 Entropy Coding (Example, details are optional)

[1092] 9.1 Methods

[1093] 9.1.1 DeepCABAC (example, details are optional)

[1094] 9.1.1.1 Binarization (example, details are optional)

[1095] For example, the encoding method scans the parameter tensor in a manner defined, for example, by the function TensorIndex(). For example, each quantization parameter level with an integer parameter 'maxNumNoRemMinus1' is encoded according to the following process:

[1096] For example, optionally, in a first step, a binary syntax element sig_flag is encoded for the quantization parameter level, which is used to specify whether the corresponding level is equal to 0. For example, if sig_flag is equal to 1, another binary syntax element sign_flag is encoded. For example, the binary (bin) indicates whether the level of the current parameter is positive or negative. For example, the unary binary sequence is encoded next, followed by a fixed length sequence, as shown below:

[1097] For example, the variable k is initialized to 0, and X is initialized to 1<<k. For example, encoding the syntax element abs_level_greater_x / x2, for example, indicating that the absolute value of the quantization parameter level is greater than x. For example, if abs_level_greater_x / x2 is equal to 1 and x is greater than maxNumNoRemMinus1, the variable k is incremented by 1, for example. For example, afterwards, 1<<k is added to x, for example, and another abs_level_greater_x / x2 is encoded. For example, this process continues until abs_level_greater_x / x2 is equal to 0. For example, it is now obvious that X must be one of the values in (x, x-1, … X-(1<<k)+1). For example, encoding a code of length k, which points to a value in the list of absolute quantization parameter levels, for example.

[1098] 9.1.1.2 Skipping Rows (Examples, Details are Optional)

[1099] For example, if the row_skip_flag_enabled_flag is enabled, the row skipping technique can emit a row_skip_list[i] flag signal for each value i on the first axis of the parameter tensor. For example, if the flag row_skip_list[i] is 1, all elements with the index of the first axis of the parameter tensor equal to i are set to 0. For example, if the flag row_skip_list[i] is 0, all elements with the index of the first axis of the parameter tensor equal to i are encoded individually.

[1100] Generally, according to an embodiment of the decoder, for example Figure 1 the decoder 100 shown, for example Figure 2 the decoder 200 shown, for example, and its corresponding decoding units 110, 210, for example, can be configured to decode the skip row list to input the decoded neural network coefficients into the first multi-dimensional array (such as 111, 211) at the corresponding positions described by the corresponding array index sets, and determine whether to use the default value of a given neural network parameter or whether to use the decoded value to determine a given neural network parameter according to the entry of the skip row list referenced by the current array index of the first dimension of the first multi-dimensional array.

[1101] 9.1.1.3 Context Modeling (Examples, Details are Optional)

[1102] For example, context modeling corresponds to associating three types of flags sig_flag, sign_flag, and abs_level_greater_x / x2 with a context model. In this way, flags with similar statistical behavior should be associated with the same context model so that a probability estimator (e.g., a probability estimator inside a context model) can adapt to the underlying statistics.

[1103] For example, the context modeling of the proposed method is as follows:

[1104] For example, sig_flag can distinguish twenty-four context models depending on whether the state value and the quantization parameter level adjacent to the left are zero, less than zero, or greater than zero.

[1105] For example, if dq_flag is 0, only the first three context models are used.

[1106] For example, sign_flag can distinguish the other three context models according to whether the left adjacent quantization parameter level is zero, less than zero, or greater than zero.

[1107] For example, for the abs_level_greater_x / x2 flag, one or two separate context models are used for each x. For example, if x<=maxNumNoRemMinus1, two context models are distinguished according to sign_flag. For example, if x>maxNumNoRemMinus1, only one context model is used.

[1108] 9.1.1.4 Temporal context modeling (example, details are optional)

[1109] For example, if the temporal_context_modeling_flag flag is enabled, an additional context modeling set of flags sig_flag, sign_flag, and abs_level_great_x is available. For example, the derivation of ctxIdx is also based on the value of the quantized co-location parameter level in the previously encoded parameter update tensor, which can be uniquely identified by the parameter update tree, for example. For example, if the co-location parameter level is not available or equal to zero, context modeling is applied according to subclause 9.1.1.3. Otherwise, for example, if the co-location parameter level is not equal to zero, the temporal context modeling of the proposed method is, for example, as follows:

[1110] For example, sig_flag can distinguish 16 context models according to whether the absolute value of the state value and the quantized co-location parameter level is greater than 1.

[1111] For example, if dq_flag is 0, only the first two additional context models are used.

[1112] For example, whether the quantized parity parameter level is less than zero or greater than zero, sign_flag can distinguish between two other context models.

[1113] For example, for the abs_level_greater_x flag, two separate context models are used for each x. For example, the distinction between the two context models depends on whether the absolute value of the quantized co-location parameter level is greater than or equal to x-1.

[1114] 9.2 Syntax and Semantics (Examples, details are optional)

[1115] 9.2.1 DeepCABAC Syntax (Example, Details are Optional)

[1116] 9.2.1.1 General

[1117] This subclause specifies the entropy coding syntax, for example for use in the decoding process of subclause 7.

[1118] 9.2.1.2 DeepCABAC Termination Syntax (Details are Optional)

[1119] 9.2.1.3 Quantization parameter syntax (details are optional)

[1120] 9.2.1.4 Quantized Tensor Syntax (Example, details are optional)

[1121]

[1122]

[1123] For example, row_skip_enabled_flag specifies whether row skipping is enabled. For example, row_skip_enabled_flag equal to 1 indicates that row skipping is enabled.

[1124] For example, row_skip_list specifies a list of flags, where the i-th flag row_skip_list[i] indicates whether all tensor elements whose first dimension index in QuantParam is equal to i are zero. For example, if row_skip_list[i] is equal to 1, then all tensor elements whose first dimension index in QuantParam is equal to i are zero.

[1125] For example, init_prob_est_param() calls the initialization procedure specified in, for example, Subclause 9.3.2.2.

[1126] For example, the two-dimensional integer array StateTransTab[][] specifies a state transition table for dependent scalar quantization, such as the following: StateTransTab[][] = {{0,2},{7,5},{1,3},{6,4},{2,0},{5,7},{3,1},{4,6}}

[1127] 9.2.1.5 Quantization parameter syntax (details are optional)

[1128] 9.2.1.6 Shift parameter index syntax (details are optional)

[1129] 9.2.1.7 Shift parameter syntax (details are optional)

[1130] 9.3 Entropy Decoding Process (Details are Optional)

[1131] 9.3.1 General (details are optional)

[1132] 9.3.2 Initialization process (details are optional)

[1133] 9.3.2.1 General (details are optional)

[1134] 9.3.2.2 Initialization process of probability estimation parameters (details are optional)

[1135] 9.3.2.3 Context variable initialization process (details are optional)

[1136] 9.3.2.4 Arithmetic Decoding Engine Initialization Process (Details are Optional)

[1137] 9.3.3 Binarization (Details are optional)

[1138] 9.3.3.1 General (details are optional)

[1139] 9.3.3.2 Fixed-length binarization (details are optional)

[1140] 9.3.4 Decoding Process Flow (Details are Optional)

[1141] 9.3.4.1 General (details are optional)

[1142] 9.3.4.2Derivation of ctxIdx and bypassFlag (details are optional)

[1143] 9.3.4.2.1 General (details are optional)

[1144] 9.3.4.2.2 Derivation of ctxInc for the syntax element sig_flag (details are optional)

[1145] 9.3.4.2.3 Derivation of ctxInc for the syntax element sign_flag (details are optional)

[1146] 9.3.4.2.4 Derivation of ctxInc for the syntax element abs_level_greater_x[j] (details optional)

[1147] 9.3.4.3 Arithmetic decoding process (details are optional)

[1148] 9.3.4.3.1 General (details are optional)

[1149] 9.3.4.3.2 Arithmetic decoding process for binary decisions (details are optional)

[1150] 9.3.4.3.2.1 General (details are optional)

[1151] 9.3.4.3.2.2 State Transition Process (Optional)

[1152] 9.3.4.3.3 Renormalization process in the arithmetic decoding engine (details are optional)

[1153] 9.3.4.3.4 Bypass decoding process for binary decisions (details are optional)

[1154] 9.3.4.3.5 Decoding process for binary decisions before termination (details are optional)

[1155] Another Annex A may be related to or define the implementation of NNEF.

[1156] For the sake of brevity, Annexes A to E or the entire contents of Annexes A to E are not included. It is worth noting that for the embodiments of the present invention, the respective details of the functions and features in these annexes are optional for the implementation of the present invention and can be used alone or in combination.

[1157] References for "Information Technology - Multimedia Content Description Interface"

[1158] [1]Open neural network Exchange,VERSION 7,2020-05-09(https: / / github.com / onnx / onnx / blob / master / onnx / onnx.proto)

[1159] [2]PyTorch,VERSION 1.5.1,2020-10-22(https: / / github.com / pytorch / pytorch / tree / v1.5.1)

[1160] [3]TensorFlow,VERSION 2.2.0,2020-10-22(https: / / github.com / tensorflow / tensorflow / tree / v2.2.0)

[1161] Alternative implementation:

[1162] Although certain aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a module or device corresponds to a method step or a feature of a method step. Similarly, various aspects described in the context of a method step also represent a description of a corresponding module or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[1163] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may use a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which can cooperate (or can cooperate) with a programmable computer system in order to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[1164] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that the system performs one of the methods described herein.

[1165] Generally speaking, the embodiments of the present invention can be implemented as a computer program product with a program code, which is operable to perform one of the above methods when the computer program product runs on a computer.For example, the program code can be stored on a machine-readable carrier.

[1166] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[1167] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[1168] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium, or the recorded medium are typically tangible and / or non-transitory.

[1169] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection, for example via the Internet.

[1170] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[1171] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[1172] A further embodiment according to the invention comprises an apparatus or system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described in the invention to a receiver. The receiver may be, for example, a computer, a shifting device, a storage device, etc. For example, the apparatus or system may comprise a file server for transmitting the computer program to the receiver.

[1173] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, these methods are preferably performed by any hardware device.

[1174] The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

[1175] The apparatus described herein, or any component of the apparatus described herein, may be implemented at least partially in hardware and / or software.

[1176] The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[1177] The methods described herein, or any component of the apparatus described herein, may be implemented at least in part by hardware and / or software.

[1178] The embodiments are merely illustrative of the principles of the invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, the present invention is intended to be limited only by the scope of the claims to be filed, and not by the specific details presented herein through the description and explanation of the embodiments.

Claims

1. A decoder (100, 200) for providing decoding parameters of a neural network based on an encoded representation, in, The decoder is configured to obtain a first multidimensional array (111, 211, 670) including a plurality of neural network parameter values ​​using the decoding (652) of the neural network parameters; The decoder is configured to obtain a reordered multidimensional array (121, 221, 680) using the reordering (672), wherein a first dimension of the first multidimensional array is reordered to a different dimension in the reordered multidimensional array.

2. The decoder (100, 200) according to claim 1, in, The decoder is configured to decode the dimension shift value (112, 212, 630), and The dimension shift value describes how much the first dimension of the first multidimensional array (111, 211, 670) should be shifted when performing the reordering.

3. The decoder (100, 200) according to claim 1 or 2, in, The decoder is configured to decode the dimension shift value (112, 212, 630), and The dimension shift value describes the new position of the first dimension of the first multidimensional array (111, 211, 670) in the reordered multidimensional array (121, 221, 680).

4. The decoder (100, 200) according to claim 3, in, The dimension shift values ​​are Exp-Golomb encoded.

5. The decoder (100, 200) according to any one of claims 1 to 4, in, The decoder is configured to perform the reordering using a single scalar dimension shift value (112, 212, 630) as the only parameter describing a new order of dimensions in the reordered multidimensional array (121, 221, 680).

6. The decoder (100, 200) according to any one of claims 1 to 5, in, The decoder is configured to perform a single shift of a single dimension to another position to obtain a reordered multidimensional array (121, 221, 680).

7. The decoder (100, 200) according to any one of claims 1 to 6, in, The decoder is configured to derive a dimension of the auxiliary array from a first dimension of the first multidimensional array (111, 211, 670).

8. The decoder (100, 200) according to any one of claims 1 to 7, in, The decoder is configured to shift a single dimension of the first multidimensional array (111, 211, 670) to a different position to obtain a reordered multidimensional array (121, 221, 680).

9. The decoder (100, 200) according to any one of claims 1 to 8, in, The decoder is configured to obtain a two-dimensional matrix (660) based on the encoded representation, a first dimension of the two-dimensional matrix being determined by the first dimension value and a second dimension of the two-dimensional matrix being determined by the product of a plurality of further dimension values; as well as wherein the decoder is configured to obtain a first multidimensional array (111, 211, 670) based on the two-dimensional matrix (660), wherein the dimension of the first multidimensional array is determined by each dimension value in the first dimension value and the further dimension value; and The decoder is configured to obtain a reordered multidimensional array (121, 221, 680) based on the first multidimensional array, wherein a first dimension of the reordered multidimensional array is defined by one of the further dimension values.

10. The decoder (100, 200) according to any one of claims 1 to 9, in, The decoder is configured to decode the encoded neural network parameters using context-based entropy decoding, and The decoder is configured to determine the position of the decoded neural network parameters in the first multidimensional array (111, 211, 670) using a position mapping function, wherein the position mapping function maps a scalar neural network parameter index to a set of dimension indices.

11. The decoder (100, 200) according to any one of claims 1 to 10, in, The decoder is configured to store a plurality of decoded neural network parameters consecutively along a given dimension of a first multidimensional array (111, 211, 670), the plurality of decoded neural network parameters being directly adjacent in decoding order.

12. The decoder (100, 200) according to any one of claims 1 to 11, in, The decoder is configured to obtain the reordered multidimensional array (121, 221, 680) by processing of the following form: Among them, i is the running variable; Where Prod(inputTensorDims) is the product of the lengths of the dimensions of the first multidimensional array (111, 211, 670); Wherein, TensorIndex(.,.,.) is a function that maps integer values ​​to a set of array indices that specify elements of the first multidimensional array; Where inputTensorDims is a collection of values ​​describing the lengths of the dimensions of the first multidimensional array; Among them, idxA is a variable; Among them, ShiftArrayIndex(.,.) is a function that maps the input array index set to the returned array index set, where the returned array index set is a copy of the input array index set, but shifts the element at position 0 in the input array index set to the position specified by firstTensorDimShift; Among them, firstTensorDimShift is the dimension shift value (112, 212, 630); Among them, idxB is a variable; Among them, inputTensor[.] is the first multidimensional array; Among them, reorderedTensor[.] is the reordered multidimensional array; Where inputTensor[idxA] is the element at the position specified by idxA in the first multidimensional array; where reorderedTensor[idxB] is the element at the position specified by idxB in the reordered multidimensional array.

13. The decoder (100, 200) according to any one of claims 1 to 12, in, The decoder is configured to obtain the reordered multidimensional array (121, 221, 680) using a function that maps an integer element index i specifying an element of the first multidimensional array (111, 211, 670) to a set of array indices, where the returned array index set specifies the i-th element in row-major scan order of the first multidimensional array (111, 211, 670), or The returned array index set specifies the i-th element of the block scan of the first multidimensional array, where the first multidimensional array is treated as a two-dimensional array.

14. The decoder (100, 200) according to any one of claims 1 to 13, in, The decoder is configured to decode the skip line list; wherein the decoder is configured to input the decoded neural network coefficients into corresponding positions described by corresponding array index sets in the first multidimensional array (111, 211, 670); The decoder is configured to decide whether to use default values ​​of given neural network parameters or whether to use decoding to determine the given neural network parameters based on an entry of a skip row list referenced by a current array index of a first dimension of a first multidimensional array.

15. The decoder (100, 200) according to any one of claims 1 to 14, in, The decoder is configured to decode the array dimension set; and The decoder is configured to obtain a reordered set of array dimensions.

16. The decoder (100, 200) according to any one of claims 1 to 15, in, The decoder is configured to input the decoded neural network parameters into corresponding positions described by corresponding array index sets in the first multidimensional array (111, 211, 670); The decoder is configured to obtain a corresponding array index set using a mapping function, wherein the mapping function maps a scalar integer parameter index to a corresponding array index set and defines a block scan.

17. The decoder (100, 200) according to claim 16, in, The mapping function includes a mapping of a scalar integer parameter index to two coordinates, the coordinates pointing to positions corresponding to a scan index defined by the scalar integer parameter index when performing a block-by-block scan; and The mapping function includes mapping two coordinates to corresponding array index sets.

18. The decoder (100, 200) according to any one of claims 1 to 17, in, The decoder is configured to input the decoded neural network parameters into corresponding positions of the first multidimensional array (111, 211, 670) described by corresponding array index sets; The decoder is configured to obtain the corresponding array index set based on the scalar integer parameter index i using the function TensorIndex(tensorDimensions[],i,scan); Among them, the function TensorIndex(tensorDimensions[],i,scan) returns an array of array indices with the same dimensions as the array tensorDimensions[], Here, the elements of the array of array indices are set to integer values ​​so that the array of array indices can be used as indices pointing to elements of the array with dimensions tensorDimensions[], as shown below: If the variable scan is equal to 0: The returned array of array indices points to the i-th element in row-major scan order of the array with dimensions defined by tensorDimensions[]; If the variable scan is greater than 0: The variable bs is set to 4< <scan_order; The variable h is set to tensorDimensions[0]; The variable w is set to Prod(tensorDimensions) / h; The two variables x and y are set to the first and second elements of the returned array, By calling IndexToXY(w,h,i,bs) respectively; The returned array is TensorIndex(tensorDimensions,y*w+x,0); Among them, the function IndexToXY(w,h,i,bs) returns an array containing two elements; Among them, the first element returned by the function IndexToXY is the x coordinate, and the second element returned by the function IndexToXY is the y coordinate, pointing to a two-dimensional array with a width of w and a height of h; Among them, when the block scan is performed in blocks of size bs times bs, x and y point to the position corresponding to the scan index i; The derivation of x and y is as follows: The variable fullRowOfBlocks is set to w*bs; The variable blockY is set to i / fullRowOfBlocks; The variable iOff is set to i%fullRowOfBlocks; The variable currBlockH is set to Min(bs,h-blockY*bs); The variable fullBlocks is set to bs*currBlockH; The variable blockX is set to iOff / fullBlocks; The variable blockOff is set to iOff%fullBlocks; The variable currBlockW is set to Min(bs,w-blockX*bs) The variable posX is set to blockOff%currBlockW; The variable posY is set to blockOff / currBlockW; The variable x is set to blockX*bs+posX; The variable y is set to blockY*bs+posY; where tensorDimensions[] is an array of values ​​describing the lengths of the different dimensions of the first multidimensional array; and Where scan is an integer variable describing the scan mode.

19. An encoder (300) for providing an encoded representation of a neural network parameter, in, The encoder is configured to obtain a reordered multidimensional array (321, 620) using the reordering, wherein a given dimension (301) of a given multidimensional array (302, 610) of neural network parameters is reordered to a first dimension in the reordered multidimensional array; The encoder is configured to encode the reordered multidimensional array.

20. The encoder (300) according to claim 19, in, The encoder is configured to encode the dimension shift value (112, 212, 630); and wherein the dimension shift value describes which given dimension of the given multidimensional array (302, 610) is to be reordered to the first dimension in the reordered multidimensional array (321, 620), or Therein, the dimension shift value describes how much the first dimension of the encoded reordered multidimensional array should be shifted when performing decoder-side reordering (672).

21. The encoder (300) according to claim 19 or 20, in, The encoder is configured to encode (632) the dimension shift value (112, 212, 630), and Therein, the dimension shift value describes the new position to which the first dimension of the encoded reordered multidimensional array should be moved in the decoder.

22. The encoder (300) according to claim 21, in, The encoder is configured to encode the dimension shift values ​​using Exp Golomb code.

23. The encoder (300) according to any one of claims 19 to 22, in, The encoder is configured to perform the reordering using a single scalar dimension shift value (112, 212, 630) as the only parameter describing a new order of dimensions in the reordered multidimensional array (321, 620).

24. The encoder (300) according to any one of claims 19 to 23, The encoder is configured to perform a single shift of a single dimension to another position to obtain a reordered multidimensional array (321, 620).

25. The encoder (300) according to any one of claims 19 to 24, in, The encoder is configured to set the dimension of the auxiliary array included in the encoded representation to be equal to the first dimension of the reordered multidimensional array (321, 620).

26. The encoder (300) according to any one of claims 19 to 25, in, The encoder is configured to shift a single dimension of a given multidimensional array (302, 610) to a different position to obtain a reordered multidimensional array (321, 620).

27. The encoder (300) according to any one of claims 19 to 26, in, The encoder is configured to obtain a reordered multidimensional array (321, 620) based on a given multidimensional array (302, 610), the dimensions of the reordered multidimensional array being determined by respective dimension values ​​of a first dimension value associated with the first dimension and further dimension values ​​associated with further dimensions other than the first dimension; and wherein the encoder is configured to obtain a two-dimensional matrix (640) based on the reordered multidimensional array (321, 620), wherein a first dimension of the two-dimensional matrix is ​​determined by a first dimension value and a second dimension is determined by a product of a further dimension value; and The encoder is configured to encode a two-dimensional matrix (642).

28. The encoder (300) according to any one of claims 19 to 27, in, The encoder is configured to encode the neural network parameters using context-based entropy coding, and The encoder is configured to determine the neural network parameters to be encoded in the reordered multidimensional array (321, 620) using a position mapping function, wherein the position mapping function maps a scalar neural network parameter index to a set of dimension indices.

29. The encoder (300) according to any one of claims 1 to 28, in, The encoder is configured to continuously read multiple neural network parameters to be encoded along a given dimension of the reordered multidimensional array (321, 620), and the multiple neural network parameters to be encoded are directly adjacent in the encoding order.

30. The encoder (300) according to any one of claims 19 to 29, in, The encoder is configured to obtain the reordered multidimensional array (321, 620) by processing in the following form: Among them, i is the running variable; Where Prod(inputTensorDims) is the product of the lengths of the dimensions of the given multidimensional array (302, 610); Among them, TensorIndex(.,.,.) is a function that maps integer values ​​to a set of array indices that specify elements of a given multidimensional array; Among them, inputTensorDims is a collection of values ​​describing the length of the dimensions of a given multidimensional array; Among them, idxA is a variable; Among them, ShiftArrayIndexEnc(.,.) is a function that maps the input array index set to the returned array index set, where the returned array index set is a copy of the input array index set, but the elements in the input array index set at the position specified by firstTensorDimShift are shifted to position 0; Among them, firstTensorDimShift is the dimension shift value (112, 212, 630); Among them, idxB is a variable; Among them, inputTensor[.] is a given multidimensional array; Among them, reorderedTensor[.] is the reordered multidimensional array; Where inputTensor[idxA] is the element at the position specified by idxA in the given multidimensional array; where reorderedTensor[idxB] is the element at the position specified by idxB in the reordered multidimensional array.

31. The encoder (300) according to any one of claims 19 to 30, in, The encoder is configured to obtain a reordered multidimensional array (321, 620) using a function that maps an integer element index i specifying an element in a given multidimensional array (302, 620) to a set of array indices, where the returned set of array indices specifies the i-th element in row-major scan order of a given multidimensional array, or The returned set of array indices specifies the i-th element of a block-wise scan of a given multidimensional array, where the given multidimensional array is treated as a two-dimensional array.

32. The encoder (300) according to any one of claims 19 to 31, in, The encoder is configured to encode the skip line list; wherein the encoder is configured to encode the neural network coefficients of the reordered multidimensional array (321, 620) at corresponding positions described by corresponding array index sets; The encoder is configured to decide whether to encode the encoded neural network parameter at the current position based on an entry in a skip row list referenced by a current array index of a first dimension of the reordered multidimensional array.

33. The encoder (300) according to any one of claims 19 to 32, in, The encoder is configured to obtain a set of array dimensions; The encoder is configured to encode the array dimension set.

34. The encoder (300) according to any one of claims 19 to 33, in, The encoder is configured to encode the neural network parameters of the reordered multidimensional array (321, 620) at corresponding positions described by the corresponding array index sets; The encoder is configured to obtain a corresponding array index set using a mapping function, wherein the mapping function maps a scalar integer parameter index to a corresponding array index set and defines a block-by-block scan.

35. The encoder (300) according to claim 34, in, The mapping function includes a mapping of a scalar integer parameter index to two coordinates, the coordinates pointing to a position corresponding to a scan index defined by the scalar integer parameter index when performing a block-by-block scan; and The mapping function includes mapping two coordinates to corresponding array index sets.

36. The encoder (300) according to any one of claims 19 to 35, in, The encoder is configured to encode the neural network parameters of the reordered multidimensional array (321, 620) at corresponding positions described by the corresponding array index sets; The encoder is configured to use the function TensorIndex(tensorDimensions[],i,scan) to obtain the corresponding array index set based on the scalar integer parameter index i; Among them, the function TensorIndex(tensorDimensions[],i,scan) returns an array of array indices with the same dimensions as the array tensorDimensions[]; Here, the elements of the array of array indices are set to integer values ​​so that the array of array indices can be used as indices pointing to elements of the array with dimensions tensorDimensions[], as shown below: If the variable scan is equal to 0: The returned array of array indices points to the i-th element in row-major scan order of the array with dimensions defined by tensorDimensions[]; If the variable scan is greater than 0: The variable bs is set to 4< <scan_order; The variable h is set to tensorDimensions[0]; The variable w is set to Prod(tensorDimensions) / h; The two variables x and y are set to the first and second elements of the returned array, By calling IndexToXY(w,h,i,bs) respectively; The returned array is TensorIndex(tensorDimensions,y*w+x,0); Among them, the function IndexToXY(w,h,i,bs) returns an array containing two elements. Among them, the first element returned by the function IndexToXY is the x coordinate, and the second element returned by the function IndexToXY is the y coordinate, pointing to a two-dimensional array with a width of w and a height of h. Among them, when the block scan is performed in blocks of size bs times bs, x and y point to the position corresponding to the scan index i; The derivation of x and y is as follows: The variable fullRowOfBlocks is set to w*bs; The variable blockY is set to i / fullRowOfBlocks; The variable iOff is set to i%fullRowOfBlocks; The variable currBlockH is set to Min(s,h-blockY*bs); The variable fullBlocks is set to bs*currBlockH; The variable blockX is set to iOff / fullBlocks; The variable blockOff is set to iOff%fullBlocks; The variable currBlockW is set to Min(bs,w-blockX*bs) The variable posX is set to blockOff%currBlockW; The variable posY is set to blockOff / currBlockW; The variable x is set to blockX*bs+posX; The variable y is set to blockY*bs+posY; where tensorDimensions[] is an array of values ​​describing the lengths of the different dimensions of the first multidimensional array; and Where scan is an integer variable describing the scan mode.

37. A method (400) for providing decoding parameters of a neural network based on an encoded representation, in, The method includes obtaining (410) a first multidimensional array (111, 211, 670) including a plurality of neural network parameter values ​​using decoding (652) of neural network parameters; Therein, the method includes obtaining (420) a reordered multidimensional array (121, 221, 680) using a reordering (672), wherein a first dimension of a first multidimensional array is reordered to a different dimension in the reordered multidimensional array.

38. A method (500) for providing an encoded representation of parameters of a neural network, wherein the method comprises obtaining (510) a reordered multidimensional array (321, 620) using a reordering (612), wherein a given dimension (301) of a given multidimensional array (302, 620) of neural network parameters is reordered to a first dimension in the reordered multidimensional array; in, The method includes encoding a reordered multidimensional array (520).

39. A computer program for performing the method of one of claims 37 to 38 when the computer program is run on a computer.

Images