DECODER, ENCODER, METHOD FOR PROVIDING DECODING PARAMETERS FOR NEURAL NETWORKS AND COMPUTER PROGRAM USING PERMISSION - Patent application

Abstract

An embodiment according to the invention relates to a decoder for providing decoded parameters of a neural network based on an encoded representation, the decoder configured to use decoding of neural network parameters to obtain a first multidimensional array comprising a plurality of neural network parameter values, the decoder configured to use reordering to obtain a reordered multidimensional array, wherein a first dimension of the first multidimensional array is rearranged to a different dimension within the reordered multidimensional array. Further disclosed is an encoder, a method and a computer program using reordering.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments according to the invention relate to a computer program using a decoder, an encoder, a method and a reordering for providing decoding parameters for a neural network.

[0002] Embodiments in accordance with the present invention relate to tensor dimension reordering by tensor dimension shifting.

[0003] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment in accordance with the present invention relates to a method for tensor dimension reordering for neural network coding.

[0004] SUMMARY OF THE PRESENT EMBODIMENTS Embodiments in accordance with the present invention relate to the compression of neural networks for multimedia content description and analysis. [Background technology]

[0005] Many frameworks provide several compression tools, including quantization and / or lossless encoding and decoding methods, for processing and / or compressing tensors.

[0006] As an example, based on the success of artificial neural networks in multimedia analysis and processing, media coding, data analysis, and many other applications, there is an increasing demand for concepts that allow the exchange of such networks, e.g., in the form of their respective neural network parameters, which may be processed and / or encoded and decoded, e.g., in the form of tensors.

[0007] Therefore, the processing, encoding, and decoding of such tensors can represent a key aspect for the efficiency of such a framework. Summary of the Invention [Problem to be solved by the invention]

[0008] It is therefore desirable to have improved concepts for encoding, decoding, and / or processing parameters, such as neural network parameters, provided in a structured representation, such as a multidimensional array or tensor, which would provide a better compromise between coding efficiency, flexibility, and complexity.

[0009] This is achieved by the subject matter of the independent claims of the present application.

[0010] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application. [Means for solving the problem]

[0011] An embodiment in accordance with the invention includes a decoder for providing decoded parameters, e.g. "weights" or "coefficients", of the neural network based on the encoded representation, e.g. based on a bitstream representing (or e.g. including) the parameters of the neural network in encoded, e.g. compressed, form.

[0012] The decoder is configured to obtain a first multidimensional array including a plurality of neural network parameter values, e.g., neural network weights (e.g., inputTensor[idxA], where idxA is a vector of a plurality of index variables, e.g., A[m][n][o][p], where m, n, o, p are index variables), using decoding of the neural network parameters, e.g., such that the first multidimensional array includes the decoded neural network parameters.

[0013] The decoder is further configured to obtain a reordered multidimensional array (e.g., reorderedTensor[idxB], where idxB is a vector of multiple index variables, e.g., B[n][o][m]][p], where n, o, m, p are index variables) using reordering, in which a first dimension of the first multidimensional array, e.g., the dimension of the multidimensional array specified by the leftmost array index, is reordered, e.g., moved, to a different dimension within the reordered multidimensional array, e.g., different from the first dimension.

[0014] In many frameworks, a structured representation of a parameter, e.g., a tensor, may be interpreted as a one- or two-dimensional representation, e.g., in the form of a 1D or 2D array, and the shape of such an interpreted representation may thus be determined by the original dimensionality of the structured representation, and in particular the respective ordering of the parameters within the original structured representation.

[0015] As an example, according to some frameworks, in such an interpretation, the length of the first dimension of the interpreted array may be equal to the length of the first dimension of the original representation, and the length of the second dimension of the interpreted array may be equal to the product of all other dimensions of the original representation.

[0016] The inventors have recognized that such shape and / or parameter ordering also impacts the efficiency of the compression and / or processing pipeline. Furthermore, conventional frameworks lack the flexibility to adapt respective structured parameter representations, e.g., tensors, between processing steps to increase efficiency.

[0017] Thus, the inventors have recognised that a better compromise between coding efficiency, flexibility and complexity may be provided for encoding, decoding and / or processing parameters, such as neural network parameters, provided in a structured representation such as a multidimensional array or tensor, based on a reordering of the dimensionality of the structured representation of the parameters.

[0018] Further, by way of example, for encoding and / or decoding of such parameters, the reordering may make it possible to exploit correlation between the parameters, which may make it possible to increase coding efficiency. In particular, the reordering may be adapted to improve correlation properties for using context-adaptive coding, e.g. CABAC.

[0019] Furthermore, the inventors have recognized that the reordering technique of the present invention allows for the decoupling of tensor shape requirements or constraints for encoding from tensor shape requirements or constraints for processing, such that even conflicting objectives of tensor shape may be achieved by reordering.

[0020] Furthermore, the inventors have recognized that based on the inventive reordering, the efficiency of existing coding frameworks can be increased with only limited impact on complexity. As an example, already developed processing techniques that expect a predefined parameter structure, e.g., tensor form, can be left intact while allowing reshaping of the parameter representation, e.g., to increase processing efficiency. As an example, the inventive reordering can allow reshaping of respective tensors to allow efficient block traversal.

[0021] According to an embodiment of the present invention, the decoder is configured to decode a dimension shift value, e.g., a "first_tensor_dimension_shift" value, e.g., a single scalar value. The dimension shift value describes by how many dimensions, e.g., by how many array indices, a first dimension, e.g., D2 (e.g., also referred to as "first tensor dimension") of a first multidimensional array (e.g., having dimensions [D2,D0,D1,D3]), should be shifted, e.g., toward a higher dimensionality, when performing the reordering, to obtain a reordered multidimensional array, e.g., having dimensions [D0,D1,D2,D3]. The inventors have recognized that based on the dimension shift value, the shift of tensor dimensions can be controlled with low signaling overhead. Furthermore, signaling shift information regarding dimensions may enable performing the reordering in an efficient manner.

[0022] According to an embodiment of the present invention, the decoder is configured to decode a dimension shift value, e.g., a "first_tensor_dimension_shift" value, where the dimension shift value, e.g., a scalar value, describes a new position (e.g., "third dimension in a multidimensional array with dimensions [D0, D1, D2, D3]") of a first multidimensional array (e.g., with dimensions [D2, D0, D1, D3]) in the reordered multidimensional array, e.g., D2 (e.g., also called "first tensor dimension"). Thus, instead of, e.g., relative shift information, absolute dimension reordering information may be provided. This may enable accurate reordering of the respective tensor dimensions.

[0023] According to an embodiment of the present invention, the dimension shift value is Exp-Golomb coded. Optionally, the decoder is configured to perform Exp-Golomb decoding of the dimension shift value. The inventors have realized that Exp-Golomb coding can provide an efficient representation of the dimension shift value.

[0024] According to an embodiment of the present invention, the decoder is configured to perform the reordering using a single scalar dimension shift value, e.g., first_tensor_dimension shift, as the only parameter describing the new order of dimensions in the reordered multidimensional array. The inventors have realised that the reordering can be controlled with a single scalar dimension shift value and thus with very limited additional data, e.g., in the form of signalling overhead, thus maximising coding and / or processing efficiency gains based on the reordering.

[0025] According to an embodiment of the present invention, the decoder is configured to perform a single shift, e.g., only a single shift, of a single dimension to another position to obtain a permuted multidimensional array. The inventors recognized that since the number of permutations can increase significantly with the number of tensor dimensions, allowing arbitrary ordering of dimensions can induce significant signaling overhead, especially for high-dimensional tensors. Instead, according to one aspect of the present invention, the decoder can only perform a shift of a single dimension to another position, thus limiting the signaling overhead.

[0026] According to an embodiment of the present invention, the decoder is configured to derive the dimension of the auxiliary array from the first dimension of the first multidimensional array. As an example, optionally, the decoder may be configured to derive the dimension of a vector or the dimension of a two-dimensional array or the dimension of an array having three or more dimensions. For example, a tensor or a "related tensor" or a "corresponding tensor" from the first dimension of the first multidimensional array. As an example, optionally, the decoder may be configured to derive the dimension of an "auxiliary" array of bias values, or an "auxiliary" array of batch norm parameters, or an "auxiliary" array of scaling factors from the first dimension of the first multidimensional array. Optionally, as an example, the decoder may be configured to set the dimension of the auxiliary array to be equal to the first dimension of the first multidimensional array.

[0027] Some frameworks, e.g., MPEG-7 Part 17, specify a special NDU type (e.g., NNR_PT_BLOCK) that allows a weight tensor and several corresponding tensors, i.e., biases, batch norm parameters, and scaling factors, to be transmitted together in a single NDU. All tensors in the block can share the same header information (e.g., instead of transmitting individual headers), thus reducing the bitstream size. Since other tensors may be directly related to the weight tensor, the inventors recognized that their dimensions can be derived from the (weight) tensor dimensions.

[0028] Thus, in general, for multi-dimensional arrays (e.g., weight tensors), the decoder of the present invention may be configured to derive one or more auxiliary arrays (e.g., including biases, batch norm parameters, and scaling factors) so that the dimensionality determination described above can be performed and signaling overhead can be limited.

[0029] According to an embodiment of the present invention, the decoder is configured to shift a single dimension of the first multidimensional array to a different position to obtain a permuted multidimensional array. This can provide a simple variant of the inventive permutation, thus limiting the overall impact on processing complexity and signaling overhead while allowing to increase processing and / or coding efficiency.

[0030] According to an embodiment of the present invention, the decoder is configured to obtain a two-dimensional matrix, for example with dimensions [D2, (D0*D1*D3)], where a first dimension of the two-dimensional matrix is ​​determined by a first dimension value, for example D2, and a second dimension of the two-dimensional matrix is ​​determined by a product of a number of further dimension values, for example D0*D1*D2, based on the encoded representation. Furthermore, the decoder is configured to obtain a first multidimensional array (for example with dimensions [D2,D0,D1,D3]), the dimensions of which are determined, optionally separately, by the first dimension value (for example D2) and the further dimension values ​​(for example D0,D1,D3) based on the two-dimensional matrix (for example, each of the first dimension value and the further dimension value may be associated with (and / or define) one dimension of the first multidimensional array; for example, the first dimension value may be associated with (and / or define) the first dimension of the first multidimensional array). Further, the decoder is configured to obtain a permuted multidimensional array based on the first multidimensional array, for example having dimensions [D0, D1, D2, D3], where a first dimension of the permuted multidimensional array is defined by one of the further dimension values.

[0031] The inventors have recognized that a decoder may perform an interpretation of the parameter representation of the encoded representation as a two-dimensional matrix, for example as an intermediate step for reordering. This approach may allow achieving compliance with existing frameworks to incorporate the inventive reordering based on the interpreted two-dimensional matrix. In particular, the inventors have recognized that a "shuffle" of one of the further dimension values ​​obtained via the encoded representation into the first dimension of the reordered multidimensional array allows for increased processing flexibility, since the tensor shape for encoding may be implemented differently from the tensor shape for processing.

[0032] According to an embodiment of the present invention, the decoder is configured to decode the coded neural network parameters using context-based entropy decoding, e.g., using arithmetic decoding in which one or more contexts for the arithmetic decoding of a given coded neural network parameter are determined depending on previously decoded neural network parameters, e.g., depending on previously decoded neural network parameters that directly precede the given coded neural network parameter in the decoding order. In addition, the decoder is configured to map the scalar neural network parameter index, e.g., i, to a set of dimension indices, e.g., a vector or (preferably one-dimensional) array, e.g., an array with a number of entries equal to the number of dimensions of the first multidimensional array, e.g., mapping onto an array pointing to the i-th element in a row-major scan order of the multidimensional array with the dimensions of the first multidimensional array, or mapping onto an array pointing to the i-th element in a block-wise scan order (e.g., scan sub-blocks) of a two-dimensional interpretation of the first multidimensional array, and to determine the location of the decoded neural network parameter within the first multidimensional array, e.g., as defined by the function TensorIndex(tensorDimensions[],I,scan) for scan>0 when taken with IndexToXY. This approach can enable efficient decoding and reordering of tensor values ​​and dimensions.

[0033] According to an embodiment of the present invention, the decoder is configured to store adjacently a number of decoded neural network parameters (e.g., the decoded neural network parameters of D3) that are directly adjacent with respect to the decoding order (e.g., decoded in sequence; e.g., decoded taking into account context adaptation depending on the values ​​of predecessors) along a given dimension (e.g., the fourth dimension or the largest dimension), e.g., the index of the given dimension increases or decreases successively, e.g., the index of the other dimension remains unchanged. Thus, the storage of the decoded neural network parameters may be performed while maintaining the respective order of the parameters, e.g., at least within the dimension.

[0034] According to an embodiment of the present invention, the decoder comprises: for( i = 0; i < Prod( inputTensorDims ); i++ ){ idxA = TensorIndex( inputTensorDims, i, 0 ) idxB = ShiftArrayIndex( idxA, firstTensorDimShift ) reorderedTensor[idxB] = inputTensor[idxA] } where i is a running variable and Prod(inputTensorDims) is a product of the lengths of the dimensions of the first multidimensional array, which may be equal to the number of elements of the first multidimensional array. Furthermore, TensorIndex(.,.,.) is a function that maps an integer value (e.g., running variable i; e.g., a scalar integer value referencing the elements of the first multidimensional array, e.g., in ascending order) to a set (e.g., one-dimensional array or vector) of array indices (e.g., a set including a number of array indices equal to the number of dimensions of the first multidimensional array) that specify the elements (e.g., the decoded neural network parameters of the first multidimensional array). Furthermore, inputTensorDims is a set of values ​​(e.g., a one-dimensional array or set) describing the lengths of the dimensions of the first multidimensional array, idxA is a variable (e.g., representing a set of array indexes (e.g., a one-dimensional array or vector)), and ShiftArrayIndex(.,.) is a function that maps an input set of array indexes (e.g., a one-dimensional array or set) (e.g., a collection of sets containing a number of array indexes equal to the number of dimensions of the first multidimensional array, e.g., idxA) to a returned set of array indexes (e.g., a one-dimensional array or set, e.g., idxB), where the returned set of array indexes is a copy of the input set of array indexes, except that the element of the input set of array indexes at position 0 has been shifted to the position specified by firstTensorDimShift (where positions are optionally counted from 0 to the size of the input set of array indexes minus 1).In addition, firstTensorDimShift is a dimension shift value (e.g., describing by how many dimensions a first dimension of the first multidimensional array should be shifted), idxB is a variable (e.g., representing a set of array indices (e.g., a one-dimensional array or vector)), inputTensor[.] is the first multidimensional array, reorderedTensor[.] is the reordered multidimensional array, inputTensor[idxA] is the element of the first multidimensional array at the position specified by idxA, and reorderedTensor[idxB] is the element of the reordered multidimensional array at the position specified by idxB. Optionally, firstTensorDimShift may correspond to first_tensor_dimension_shift. The inventors have recognized that such an approach may enable providing an efficient implementation of the reordering procedure.

[0035] According to an embodiment of the present invention, the decoder is configured to obtain the reordered multidimensional array using a function, e.g., TensorIndex(tensorDimensions[],i,scan), that maps an integer element index i, e.g., uniquely designating an element of the first multidimensional array, to a set of array indices, e.g., a one-dimensional array or vector, where the returned set of array indices designates the i-th element in a row-major scan order of the first multidimensional array (e.g., an array with array dimensions designated by the set tensorDimensions[]), or the returned set of array indices designates the i-th element of a block-wise scan of the first multidimensional array, e.g., using blocks of size bs by bs), where the first multidimensional array is considered as a two-dimensional array, e.g., of width w and height h, e.g., whose elements are referenced by x and y, where x and y may be obtained, e.g., using a function IndexToXY that maps a scalar integer index i to a set (e.g., an array) of two elements x and y. The inventors have recognized that the reordering of the present invention may be performed based on different scan orders of the first multidimensional array. Thus, the present technique is flexible and easily adapted to multiple scanning techniques.

[0036] According to an embodiment of the present invention, the decoder is configured to decode a list of skipped rows, e.g., row skiplist[i], e.g., the number of entries of the list of skipped rows, e.g., tensor2Dheight, may be equal to the length of a first dimension of the first multidimensional array, e.g., dimensions[0]. Furthermore, the decoder is configured to input the decoded neural network coefficients, e.g., QuantParam[idx], into the first multidimensional array, e.g., at the respective positions described by a respective set of array indices represented by a variable idx, e.g., obtained using the function idx=TensorIndex(dimensions, i, scan_order), e.g., by setting QuantParam[idx] to take the decoded values. In addition, the decoder is configured to determine whether to use a default value for the given neural network parameter, e.g., QuantParam[idx]=0 (or even for a sequence of neural network parameters with the same current array index of the first dimension of the first multidimensional array), or to use decoding, e.g., a call to a decoding function, e.g., int_param(i,maxNumNoRemMinus1,stateId), depending on an entry of a list of skipped rows, e.g., specified, e.g., row_skip_list[idx[0]], referenced by the current array index of the first dimension of the first multidimensional array. Thus, the decoding effort can be reduced based on the information about the skipped rows.

[0037] According to an embodiment of the present invention, the decoder is configured to decode a set of array dimensions, e.g. a one-dimensional array or a vector (e.g. describing the length of a first multidimensional array in different dimensions, e.g. tensor_dimensions[.], where the array dimensions may be Exp-Golomb coded, for example). The decoder is further configured to obtain a set of permuted array dimensions, e.g. a one-dimensional array or vector, whose entries describe the lengths of the permuted multidimensional array in different directions, e.g. using a function ShiftArrayIndex to the set of decoded array dimensions. Thus, by way of example, the determination of the ordering of tensor or array dimensions may be performed separately from the shaping or ordering of the respective tensors and / or arrays themselves. Thus, the inventors have recognised that the permutation efficiency can be increased.

[0038] According to an embodiment of the present invention, the decoder is configured to input, for example, the next decoded neural network parameter, for example QuantParam[idx], into a first multidimensional array at a respective position described by a respective set of array indexes represented by a variable idx, for example obtained using a function idx=TensorIndex(dimensions, i, scan_order), for example by setting QuantParam[idx] to take a decoded value. In addition, the decoder is configured to obtain the respective set of array indexes using a mapping function (for example TensorIndex(tensorDimensions[], i, scan) which may call, for example, IndexToXY) that maps a scalar integer parameter index, for example i, to a respective set of array indexes and defines a block-wise scan, for example the next value of the scalar integer parameter index may be associated with the next decoded parameter value. The inventors have realized that in this way an efficient reordering technique may be provided.

[0039] According to an embodiment of the present invention, the mapping function includes mapping a scalar integer parameter index, for example i, to two coordinates, for example x, y, that indicate a position corresponding to a scan index defined by the scalar integer parameter index when a block, for example a two-dimensional array of width w and height h, is scanned within the block, for example within a block of size bs×bs. Further, the mapping function includes mapping the two coordinates to respective sets of array indices. The inventors have recognized that in this way, the dimensions to be reordered can be efficiently addressed.

[0040] According to an embodiment of the present invention, the decoder is configured to input, for example, the following decoded neural network parameters, for example QuantParam[idx], by setting QuantParam[idx] to take a decoded value, for example, at each position described by each set of array indices represented by a variable idx obtained using, for example, the function idx = TensorIndex(dimensions, i, scan_order), in a first multi-dimensional array. Further, the decoder is configured to obtain each set of array indices based on a scalar integer parameter index i using the function TensorIndex(tensorDimensions[], i, scan), where the function TensorIndex(tensorDimensions[], i, scan) returns an array of array indices having the same number of dimensions as the array tensorDimensions[], and where the elements of the array of array indices are such that when the variable scan is equal to 0: When the variable scan is equal to 0: the returned array of array indices points to the i-th element [for example, a tensor] [for example, the first multi-dimensional array] in row-major traversal order of an array having the dimensions defined by tensorDimensions[], When the variable scan is greater than 0: the variable bs is set to 4 << scan_order, The variable h is set to tensorDimensions[0].

[0041] [e.g., the length of the first dimension of the first multidimensional array], The variable w is set to Prod(tensorDimensions) / h. [For example, the product of the lengths of the first multidimensional array of different dimensions divided by h, or the number of elements in the first multidimensional array divided by h] Two variables x and y are set to the first and second elements of the array returned by calling IndexToXY(w,h,i,bs), respectively. The returned array is TensorIndex(tensorDimensions,y*w+x,0) and the function IndexToXY(w,h,i,bs) returns an array with two elements, the first element returned by the function IndexToXY is the x coordinate and the second element returned by the function IndexToXY is the y coordinate that points to a 2D array of width w and height h, where x and y point to a position corresponding to scan index i when a block [e.g. a 2D array of width w and height h] is scanned in blocks of size bs×bs, and x and y are The variable fullRowOfBlocks is set to w*bs, The variable blockY is set to i / fullRowOfBlocks, The variable iOff is set to i%fullRowOfBlocks, The variable currBlockH is set to Min(bs,h-blockY*bs), The variable fullBlocks is set to bs*currBlockH, The variable blockX is set to iOff / fullBlocks, The variable blockOff is set to iOff%fullBlocks, The variable currBlockW is set to Min(bs,w-blockX*bs), The variable posX is set to blockOff%currBlockW, The variable posY is set to blockOff / currBlockW, The variable x is set to blockX*bs+posX, The variable y is set to blockY*bs+posY where tensorDimensions[] is an array of values ​​describing the lengths of the first multidimensional array in different dimensions, and scan is an integer variable describing the scanning mode. The inventors have recognized that in this manner, an efficient sorting technique can be provided.

[0042] In the following, embodiments according to the invention comprising encoders are described. Each of the encoders described below can be based on the same considerations as the decoders described above. However, the encoders can be completed by combining all the features and functions described for the decoders individually.

[0043] A further embodiment according to the invention comprises an encoder for providing an encoded representation of parameters of a neural network, e.g., "weights" or "coefficients", e.g., in the form of a bitstream representing the parameters of the neural network in an encoded (e.g., compressed) form. The encoder is further configured to obtain a permuted multidimensional array (e.g., BEnc[o][m][n][p], where o, m, n, p are index variables) using permutation, where a given dimension of a given multidimensional array of neural network parameters (e.g., a given dimension of a multidimensional array specified by a given array index, e.g., the third dimension of a given multidimensional array AEnc[m][n][o][p]) is rearranged, e.g., moved, in the permuted multidimensional array, to a first dimension, optionally different from the given dimension. In addition, the encoder is configured to encode the permuted multidimensional array, e.g., Benc[o][m][n][p], e.g., using encoding of the entries of the permuted multidimensional array.

[0044] According to an embodiment of the present invention, the encoder is configured to encode a dimension shift value (e.g., a “first_tensor_dimension_shift” value, e.g., a single scalar value), where the dimension shift value describes which given dimension of a given multidimensional array has been rearranged to a first dimension in the permuted multidimensional array, or the dimension shift value describes how many dimensions, e.g., by how many array indexes, e.g., to a higher dimensionality, a first dimension, e.g., D2, of the encoded permuted multidimensional array should be shifted when performing decoder-side permutation to obtain a decoder-side (twice) permuted multidimensional array having dimensions, e.g., [D0,D1,D2,D3].

[0045] According to an embodiment of the present invention, the encoder is configured to encode a dimension shift value, e.g., a “first_tensor_dimension_shift” value, where the dimension shift value, e.g., a scalar value, describes a new position to which a first dimension, e.g., D2, of the encoded permuted multidimensional array, e.g., one having dimensions [D2, D0, D1, D3], should be moved in the decoder.

[0046] According to an embodiment of the present invention, the encoder is configured to encode the dimension shift values ​​using Exp-Golomb codes.

[0047] According to an embodiment of the present invention, the encoder is configured to perform the reordering using a single scalar dimension shift value, eg, first_tensor_dimension shift, as the only parameter that describes the new order of the dimensions in the reordered multidimensional array.

[0048] According to an embodiment of the present invention, the encoder is configured to perform a single shift, e.g., only a single shift, to another position of a single dimension to obtain a permuted multi-dimensional array.

[0049] According to an embodiment of the invention, the encoder is configured to set the dimensions of an auxiliary array (e.g., a vector, or a two-dimensional array, or an array of three or more dimensions; e.g., a tensor, or an "associated tensor" or "corresponding tensor", e.g. an "auxiliary" array of bias values, or an "auxiliary" array of batch norm parameters, or an "auxiliary" array of scaling factors) included in the encoded representation to be equal to the first dimension of the reordered multidimensional array.

[0050] According to an embodiment of the present invention, the encoder is configured to shift a single dimension of a given multi-dimensional array to a different position to obtain a permuted multi-dimensional array.

[0051] According to an embodiment of the present invention, the encoder is configured to obtain a permuted multidimensional array (e.g., having dimensions [D2, D0, D1, D3]), the dimensions of which are determined, optionally individually, based on a given multidimensional array, by a first dimension value, e.g., D2, associated with a first dimension, and further dimension values, e.g., D0, D1, D3, associated with further dimensions beyond the first dimension (e.g., each of the first dimension value and the further dimension values ​​may be associated with (and / or define) one dimension of the permuted multidimensional array; e.g., the first dimension value may be associated with (and / or define) a first dimension of the permuted multidimensional array). Additionally, the encoder is configured to obtain a two-dimensional matrix, where a first dimension of the two-dimensional matrix is ​​determined by a first dimension value, e.g., D2, and a second dimension of the two-dimensional matrix is ​​determined by a product of a further dimension value, e.g., D0*D1*D2, based on the permuted multidimensional array, e.g., a matrix having dimensions [D2, (D0*D1*D3)]. Further, the encoder is configured to encode the two-dimensional matrix.

[0052] According to an embodiment of the present invention, the encoder is configured to encode the neural network parameters using context-based entropy coding, e.g., arithmetic decoding, in which one or more contexts for the arithmetic coding of a given neural network parameter are determined depending on previously coded neural network parameters, e.g., depending on previously coded neural network parameters that directly precede the given neural network parameter in the coding order. The encoder is further configured to map the scalar neural network parameter index, e.g., i, onto a set of dimensional indices, e.g., a vector or (preferably one-dimensional) array, e.g., an array with a number of entries equal to the number of dimensions of the reordered multidimensional array, e.g., mapping onto an array pointing to the i-th element in a row-major scan order of the multidimensional array having the dimensions of the reordered multidimensional array, or mapping onto an array pointing to the i-th element in a block-wise scan order (e.g., scan sub-block) of a two-dimensional interpretation of the reordered multidimensional array, using a position mapping function, e.g., TensorIndex(tensorDimensions[],I,scan), e.g., as defined by the function TensorIndex(tensorDimensions[],I,scan), for scan>0, when taken together with IndexToXY, to determine the neural network parameters in the reordered multidimensional array to be encoded.

[0053] According to an embodiment of the present invention, the encoder is configured to contiguously read multiple directly adjacent encoded neural network parameters (e.g., D3 neural network parameters) along a given dimension (e.g., the fourth dimension or the largest dimension) of the permuted multidimensional array with respect to the encoding order (e.g., encoded in sequence; e.g., encoded taking into account context adaptation depending on the values ​​of predecessors), e.g., the index of the given dimension increases or decreases successively while, e.g., the index of the other dimensions remains unchanged.

[0054] According to an embodiment of the present invention, an encoder includes: for( i = 0; i < Prod( inputTensorDims ); i++ ){ idxA = TensorIndex( inputTensorDims, i, 0 ) idxB = ShiftArrayIndexEnc( idxA, firstTensorDimShift ) reorderedTensor[idxB] = inputTensor[idxA] } where i is an implementation variable and Prod(inputTensorDims) is a product of the lengths of the dimensions of the given multidimensional array, which may be equal to the number of elements of the given multidimensional array. Furthermore, TensorIndex(.,.,.) is a function that maps an integer value (e.g., implementation variable i; e.g., a scalar integer value that references the elements of the first multidimensional array, e.g., in ascending order) to a set, e.g., a one-dimensional array or vector, of array indices (e.g., a set including a number of array indices equal to the number of dimensions of the given multidimensional array) that specify the elements, e.g., the encoded neural network parameters of the given multidimensional array. Additionally, inputTensorDims is a set, e.g., a one-dimensional array or set, of values ​​that describe the lengths of the dimensions of the given multidimensional array, and idxA is a variable, e.g., a set of array indices (e.g., a one-dimensional array or vector). Furthermore, ShiftArrayIndexEnc(.,.) is a function that maps an input set of array indexes, e.g., a one-dimensional array or set, (e.g., a collection of sets containing a number of array indexes equal to the number of dimensions of a given multidimensional array, e.g., idxA) to a returned set of array indexes (e.g., a one-dimensional array or set, e.g., idxB), where the returned set of array indexes is a copy of the input set of array indexes, except that elements of the input set of array indexes at positions specified by firstTensorDimShift have been shifted to position 0, and, as an optional feature, positions are counted from 0 to the size of the input set of array indexes minus 1.Further, firstTensorDimShift is a dimension shift value, e.g., describing which given dimension of a given multidimensional array should be shifted to a first position, idxB is a variable (e.g., representing a set of array indices (e.g., a one-dimensional array or vector)), inputTensor[.] is the given multidimensional array, reorderedTensor[.] is the reordered multidimensional array, inputTensor[idxA] is the element of the given multidimensional array at the position specified by idxA, and reorderedTensor[idxB] is the element of the reordered multidimensional array at the position specified by idxB.

[0055] According to an embodiment of the present invention, the encoder is configured to obtain the reordered multidimensional array using a function, e.g., TensorIndex(tensorDimensions[], i, scan), that maps an integer element index i, e.g., uniquely specifying, an element of a given multidimensional array, to a set of array indices, e.g., a one-dimensional array or vector, where the returned set of array indices specifies the i-th element in a row-major scan order of the given multidimensional array, e.g., an array having array dimensions specified by the set tensorDimensions[], or the returned set of array indices specifies the i-th element of a block-wise scan of the given multidimensional array, e.g., using blocks of size bs by bs, where the given multidimensional array is considered as a two-dimensional array, e.g., with width w and height, e.g., its elements are referenced by, e.g., x and y, where x and y may be obtained, e.g., using a function IndexToXY that maps a scalar integer index i to a set (e.g., an array) of two elements x and y.

[0056] According to an embodiment of the invention, the encoder (e.g., the encoder is configured to obtain a list of skipped rows) is configured to encode the list of skipped rows, e.g., rowskiplist[i], e.g., the number of entries in the list of skipped rows, e.g., tensor2Dheight, may be equal to the length of a first dimension of the sorted multidimensional array, e.g., dimensions[0]. Further, the encoder is optionally configured to encode the neural network coefficients, e.g., QuantParam[idx], of the sorted multidimensional array at each position described by a respective set of array indices represented by a variable idx, e.g., obtained using a function idx=TensorIndex(dimensions, i, scan_order). Further, the encoder may be configured to determine, depending on an entry in a list of skipped rows referenced, e.g., specified, by a current array index of the first dimension of the sorted multidimensional array, e.g., row_skip_list[idx[0]], whether to encode the neural network parameters at, e.g., the current position specified by the current set of array indexes, or even a set of neural network parameters having, e.g., the same current array index of the first dimension of the sorted multidimensional array.

[0057] According to an embodiment of the invention, the encoder is configured to obtain an optionally permuted set of array dimensions, e.g., a one-dimensional array or a vector, whose entries describe the lengths of the permuted multidimensional array in different directions, e.g., using the function ShiftArrayIndexEnc applied to the set of array dimensions associated with a given multidimensional array. Furthermore, the encoder is configured to encode the optionally permuted set of array dimensions, e.g., a one-dimensional array or a vector, e.g., describing the lengths of the permuted multidimensional array in different dimensions, e.g., tensor_dimensions[.], where e.g., the array dimensions may be Exp-Golomb coded.

[0058] According to an embodiment of the present invention, the encoder is configured to encode a permuted multidimensional array of neural network parameters, e.g., QuantParam[idx], at respective positions described by a respective set of array indices represented by a variable idx, e.g., obtained using a function idx=TensorIndex(dimensions, i, scan_order). Furthermore, the encoder is configured to obtain the respective set of array indices using a mapping function (e.g., TensorIndex(tensorDimensions[], i, scan), which may, e.g., call IndexToXY) that maps a scalar integer parameter index, e.g., i, to a respective set of array indices and defines a block-wise scan, e.g., the next value of the scalar integer parameter index may be associated with the next encoded parameter value.

[0059] According to an embodiment of the present invention, the mapping function includes a mapping of a scalar integer parameter index, e.g., i, to two coordinates, e.g., x, y, that point to a location corresponding to a scan index defined by the scalar integer parameter index, e.g., i, when a block, e.g., a two-dimensional array of width w and height h, is scanned within the block, e.g., a block of size bs×bs. Further, the mapping function includes mapping the two coordinates to a respective set of array indices.

[0060] According to an embodiment of the present invention, the encoder is configured to encode neural network parameters of a multi-dimensional array rearranged at each position described by each set of array indices represented by a variable idx obtained, for example, using a function idx = TensorIndex(dimensions, i, scan_order). In addition, the encoder is configured to obtain each set of array indices based on a scalar integer parameter index i using a function TensorIndex(tensorDimensions[], i, scan), and the function TensorIndex(tensorDimensions[], i, scan) returns an array of array indices having the same number of dimensions as the array tensorDimensions[], where the elements of the array of array indices are such that the array of array indices is when the variable scan is equal to 0: the returned array of array indices points to the i-th element [e.g., tensor] [e.g., the first multi-dimensional array] in row-major traversal order of the array having the dimensions defined by tensorDimensions[], when the variable scan is greater than 0: a variable bs is set to 4 << scan_order, a variable h is set to tensorDimensions[0]

[0061] (e.g., the length of the first dimension of the first multi-dimensional array), a variable w is set to Prod(tensorDimensions) / h (e.g., the product of the lengths of the first multi-dimensional array of different dimensions divided by h, or the number of elements of the first multi-dimensional array divided by h) two variables x and y are set to the first and second elements of the array returned by calling IndexToXY(w, h, i, bs) respectively, the returned array is TensorIndex(tensorDimensions, y * w + x, 0) In addition, the function IndexToXY(w,h,i,bs) returns an array with two elements, where the first element returned by the function IndexToXY is the x coordinate and the second element returned by the function IndexToXY is the y coordinate that points into a 2D array of width w and height h, where x and y point to a block, e.g., a position corresponding to scan index i when a 2D array of width w and height h is scanned in blocks of size bs×bs, and x and y are The variable fullRowOfBlocks is set to w*bs, The variable blockY is set to i / fullRowOfBlocks, The variable iOff is set to i%fullRowOfBlocks, The variable currBlockH is set to Min(bs,h-blockY*bs), The variable fullBlocks is set to bs*currBlockH, The variable blockX is set to iOff / fullBlocks, The variable blockOff is set to iOff%fullBlocks, The variable currBlockW is set to Min(bs,w-blockX*bs), The variable posX is set to blockOff%currBlockW, The variable posY is set to blockOff / currBlockW, The variable x is set to blockX*bs+posX, The variable y is derived to be set to blockY*bs+posY.

[0062] Furthermore, tensorDimensions[] is an array of values ​​describing the lengths of the first multidimensional array in different dimensions, and scan is an integer variable describing the scanning mode.

[0063] In the following, a method according to an embodiment of the present invention will be discussed. The method described below can be based on the same considerations as the above-mentioned decoder and encoder, respectively. However, the method can achieve all the features and functions described for the decoder and encoder, respectively, either individually or in combination.

[0064] An embodiment according to the invention includes a method for providing decoded parameters, e.g., "weights" or "coefficients", of a neural network based on an encoded representation, e.g., based on a bitstream representing the parameters of the neural network in encoded (e.g., compressed) form. In addition, the method includes obtaining a first multidimensional array (e.g., inputTensor[idxA], where idxA is a vector of multiple index variables, e.g., A[dm][dn][do][dp], where dm, dn, do, dp are index variables, e.g., A[m][n][o][p], where m, n, o, p are index variables) containing a plurality of neural network parameter values, e.g., neural network weights, using decoding of the neural network parameters, e.g., such that the first multidimensional array contains the decoded neural network parameters. The method further includes obtaining a reordered multidimensional array (e.g., reorderedTensor[idxB], where idxB is a vector of multiple index variables, e.g., B[n][o][m]][p], where n, o, m, p are index variables) using reordering, in which a first dimension of the first multidimensional array, e.g., the dimension of the multidimensional array specified by the leftmost array index, is reordered, e.g., moved, to a different dimension, optionally different from the first dimension in the reordered multidimensional array.

[0065] An embodiment according to the invention includes a method for providing an encoded representation of parameters, e.g., "weights" or "coefficients", of a neural network, e.g., in the form of a bitstream representing the parameters of the neural network in an encoded (e.g., compressed) form. The method further includes obtaining a permuted multidimensional array (e.g., BEnc[o][m][n][p], where o, m, n, p are index variables) using permutation, where a given dimension of a given multidimensional array of neural network parameters (e.g., a given dimension of a multidimensional array specified by a given array index, e.g., a third dimension of a given multidimensional array AEnc[m][n][o][p]) is rearranged, e.g., moved, in the permuted multidimensional array, to a first dimension, optionally different from the given dimension. The method further includes encoding the permuted multidimensional array, e.g., Benc[m][n][o][p], using, e.g., encoding the entries of the permuted multidimensional array.

[0066] The embodiments include a computer program for performing the method according to any of the embodiments disclosed herein when the computer program runs on a computer.

[0067] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: [Brief description of the drawings]

[0068] [Figure 1] FIG. 2 is a schematic diagram of a decoder for providing decoding parameters for a neural network based on an encoded representation according to an embodiment of the present invention; [Diagram 2] FIG. 4 is a schematic diagram of a decoder according to an embodiment of the present invention, with further optional features; [Diagram 3] FIG. 2 is a schematic diagram of an encoder for providing an encoded representation of parameters of a neural network according to an embodiment of the present invention; [Figure 4] FIG. 1 is a schematic block diagram of a method for providing decoding parameters for a neural network based on an encoded representation. [Diagram 5] FIG. 1 is a schematic block diagram of a method for providing a coded representation of parameters of a neural network. [Figure 6] FIG. 1 is a schematic block diagram of a tensor dimension reordering concept for an encoding-decoding pipeline according to an embodiment of the present invention. [Figure 7] FIG. 2 is a schematic diagram of a two-layer feedforward neural network according to an embodiment of the present invention. [Figure 8] FIG. 2 is a schematic diagram of a uniform reconstruction quantizer according to an embodiment of the present invention; [Figure 9(a)] FIG. 13 shows a schematic example of allowable reconstruction vector positions for a simple case of two weighting parameters according to an embodiment, illustrating independent scalar quantization. [Figure 9(b)] FIG. 13 shows a schematic example of allowable reconstruction vector positions for a simple case of two weighting parameters according to an embodiment, illustrating dependent scalar quantization. [Figure 10] FIG. 2 illustrates a schematic example for splitting a set of reconstruction levels into two subsets according to an embodiment. [Figure 11] FIG. 2 illustrates a schematic example of an NNR encoding pipeline according to an embodiment. [Figure 12] A diagram showing a schematic example of an NNR unit data structure according to an embodiment. [Figure 13] FIG. 1 is a schematic example of an aggregate NNR unit data structure; [Figure 14] A diagram showing a schematic example of an NNR bitstream data structure according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0069] Equivalent or equivalent elements, or elements having equivalent or equivalent functionality, are designated in the following description with the same or equivalent reference numbers, even if they occur in different figures.

[0070] In the following description, a number of details are set forth to provide a more complete description of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention. In addition, the features of different embodiments described herein below may be combined with each other unless otherwise specified.

[0071] Fig. 1 is a schematic diagram of a decoder for providing decoded parameters of a neural network based on an encoded representation according to an embodiment of the present invention. Fig. 1 shows a decoder 100 comprising a decoding unit 110 and a reordering unit 120. The decoder 100 is provided with a bitstream 101, for example in compressed form, which comprises an encoded representation of the neural network parameters. As an example, the bitstream 101 provided to the decoder 100 may be an encoded representation of the neural network parameters.

[0072] 1, a decoding unit 110 is provided with a bitstream 101 to obtain a first multidimensional array 111 comprising a plurality of neural network parameter values. The decoding unit 100 is thus configured to decode the bitstream 101 or at least an encoded representation of the neural network parameters that are part of the bitstream 101.

[0073] The first multidimensional array 111 is provided to a reordering unit 120. The reordering unit 120 is configured to reorder the first multidimensional array 111 to obtain a reordered multidimensional array 121. The reordering unit 120 is configured to perform a reordering in which a first dimension of the first multidimensional array 111 is reordered into a different dimension in the reordered multidimensional array 121.

[0074] The decoder 100 may provide the reordered multidimensional array 121 as the decoding parameters 102 of the neural network. As indicated as an optional feature, in order to provide said decoding parameters of the neural network, the decoder 100 may be configured to further process the reordered multidimensional array 121 (e.g., using multiple, or at least one optional further processing unit 130) in order to provide the decoding parameters 102 of the neural network.

[0075] 1, the decoding unit is configured to decode dimension shift values ​​112 from the bitstream 101 and provide them to the reordering unit 120. The dimension shift values ​​112 provide information to the reordering unit 120 about how much a first dimension of the first multidimensional array 111 should be rearranged, e.g. shifted, e.g. moved.

[0076] Thus, the dimension shift value 112 optionally includes or may be information based on which the sorting unit 120 can determine a new location or position in the sorted multidimensional array 111 where the first dimension of the first multidimensional array 121 is sorted, e.g., switched.

[0077] As an optional feature, the dimension shift values ​​112 are Exp-Golomb coded. Thus, as an optional feature, the decoding unit 110 is configured to decode the Exp-Golomb coded dimension shift values ​​112 from the bitstream 101.

[0078] Note that dimension shift value 112 may be, for example, a single scalar. As an optional feature, the single scalar may be the only parameter based on which the new order of dimensions in permuted multidimensional array 121 is specified.

[0079] Furthermore, it should be noted that according to some embodiments, the permutation unit 120 may be configured to perform a single shift to another position of a single dimension to obtain the permuted multidimensional array 121.

[0080] Fig. 2 is a schematic diagram of a decoder with further optional features according to an embodiment of the present invention. Fig. 2 shows a decoder 200 comprising a decoding unit 210 and a reordering unit 220. The decoder 200 is provided with a bitstream 201 to obtain a first multidimensional array 211. The first multidimensional array 211 is provided to a reordering unit 220 configured to reorder the first multidimensional array 211 to obtain a reordered multidimensional array 221. The decoder 200 can provide the reordered multidimensional array 221 as a decoding parameter 202 of the neural network. As shown as an optional feature, in order to provide said decoding parameters of the neural network, the decoder 200 can be configured to further process the reordered multidimensional array 221 (e.g. using multiple or at least one optional further processing unit 230) to provide the decoding parameters 202 of the neural network.

[0081] For details of the above-mentioned features and functions of the embodiment shown in Fig. 2, for the sake of brevity, reference is made to the features and functions as described in the context of Fig. 1. It is noted that similar or corresponding features shown in Fig. 2 may optionally include the same or corresponding functions and / or details as described in the context of Fig. 1. In addition, all features marked and / or described as optional in Fig. 1 are also optional in the embodiment according to Fig. 2.

[0082] As shown in FIG. 2, the reordering unit 220 is provided with a dimension shift value 212 as an optional feature.

[0083] Furthermore, as an optional feature, the first multidimensional array 211 is provided to an optional further processing unit 230a configured to derive a dimension of the auxiliary array from the first dimension of the first multidimensional array 211.

[0084] As an example, the first multidimensional array 211 may be a weight tensor, and the bitstream 201 may include an encoded representation of a floating parameter tensor including the (optionally decomposed) weight tensor and, optionally, local scaling parameters, biases, and batch norm parameters that form blocks in the model architecture. In other words, the weight tensor and several corresponding tensors, e.g., biases, batch norm parameters, and scaling coefficients, may be included in the bitstream 201 together, e.g., in a single compressed data unit, e.g., NDU. According to such an example, the auxiliary array may include, e.g., the corresponding tensors of each of the fitted shapes, or at least their values.

[0085] As an example, corresponding tensors may be related to the weight tensor, so that their dimensions may be derived from the (weight) tensor dimensions, for example using the further processing unit 230a. As an example, the bias, batch norm parameter, and scaling factor may be, for example, a 1D tensor with a length preferably equal to the number of output channels (e.g., in the case of a fully connected layer) of the weight tensor or the number of filters (e.g., a convolutional layer), respectively. To derive these dimensions, the first dimension of the decoded weight tensor may be determined as the length of the 1D tensor. Thus, the decoded weight tensor may be ordered such that the first dimension corresponds to the number of output channels or filters.

[0086] In other words, the first multidimensional array 211, e.g. such a weight tensor, may be ordered such that the first dimension corresponds to the number of output channels or filters. Thus, one or more auxiliary arrays may be determined to represent biases, batch norm parameters, and scaling factors. Respective information 213 for determining such auxiliary arrays may be provided by the decoding unit 213 to the further processing unit 230a.

[0087] Thus, the first multidimensional array 211, for example in the form of a weight tensor representation, may be rearranged at the encoder side, so that a first dimension of the encoded representation of the first multidimensional array in the bitstream 201 allows to derive said dimension of the auxiliary array from the first dimension of the first multidimensional array 211. After determining said dimension, the first multidimensional array 211 may be rearranged to a pre-encoding state using the rearrangement unit 220. As optionally shown, information 231a including such auxiliary array may be provided for the determination of the decoded neural network parameters 202.

[0088] As an optional feature, the reordering unit 220 is configured to shift a single dimension of the first multidimensional array to a different position to obtain a reordered multidimensional array, for example, allowing arbitrary dimension ordering may induce significant signaling overhead, and thus shifting a single dimension to a different position can keep the signaling overhead low.

[0089] As an optional feature, the decoding unit 210 is configured to perform context-based decoding. In particular, the decoding unit 210 is configured to decode the encoded neural network parameters using context-based entropy decoding. Thus, the context information 240 is provided to the decoding unit 210. In particular, as an optional feature, the decoding unit 210 is configured to determine the positions of the decoded neural network parameters in the first multidimensional array 211 using a position mapping function that maps a scalar neural network parameter index to a set of dimension indexes.

[0090] 3 is a schematic diagram of an encoder for providing a coded representation of parameters of a neural network according to an embodiment of the invention.

[0091] The reordering unit is configured to obtain a reordered multidimensional array 321 using a reordering where a given dimension 301 of a given multidimensional array 302 of neural network parameters is reordered to a first dimension in the reordered multidimensional array 321. Further, the encoding unit 310 is configured to encode the reordered multidimensional array 321 into a bitstream 303, e.g., as shown.

[0092] As explained before, an encoder according to an embodiment of the present invention may be based on the same considerations as a corresponding decoder. For the sake of brevity, therefore, reference is made to the embodiment comprising a decoder as disclosed above and below for additional optional features. It is therefore noted that an encoder according to an embodiment, for example the encoder 300, may include any or all of the features disclosed in the context of any of the decoders of the present invention, either individually or in combination.

[0093] 4 is a schematic block diagram of a method for providing decoded parameters of a neural network based on an encoded representation. The method 400 includes obtaining 410 a first multidimensional array including a plurality of neural network parameter values ​​using neural network parameter decoding, and obtaining 420 a sorted multidimensional array using reordering, where a first dimension of the first multidimensional array is rearranged to a different dimension within the sorted multidimensional array.

[0094] 5 is a schematic block diagram of a method for providing an encoded representation of parameters of a neural network. The method 500 includes obtaining 510 a permuted multidimensional array using permutation, where a given dimension of a given multidimensional array of neural network parameters is rearranged to a first dimension in the permuted multidimensional array, and encoding 520 the permuted multidimensional array.

[0095] In the following, different embodiments and aspects of the present invention are described in the chapters "Proposal - Tensor Dimension Reordering by Tensor Dimension Shift", "Method for Tensor Dimension Reordering for Coding of Neural Networks", and "Updated Draft of Proposed Planned International Standard ISO / IEC 15938-17", and their respective subchapters.

[0096] Further embodiments are defined by the appended claims.

[0097] It is to be noted that any embodiment defined by the claims can optionally be supplemented by any of the details (features and functions and details) described in the above chapters and / or subchapters.

[0098] Furthermore, the embodiments described in the above chapters and / or subchapters can be used individually or can be supplemented by any of the features of another chapter or subchapter or by any features included in the claims.

[0099] It should also be noted that the individual aspects described herein may be used individually or in combination, and thus details may be added to each of the individual aspects described above without adding the details to another one of the aspects described above.

[0100] It should also be noted that this disclosure explicitly or implicitly describes features that can be used in a neural network encoder (an encoder for providing an encoded representation of parameters of a neural network) and a neural network decoder (a decoder for providing decoded parameters of the neural network parameters based on the encoded representation). Thus, any of the features described herein can be used in the context of a neural network encoder as well as in the context of a neural network decoder.

[0101] Moreover, features and functions disclosed herein in relation to a method may also be used in an apparatus (configured to perform such functions). Moreover, any features and functions disclosed herein in relation to an apparatus may also be used in a corresponding method. In other words, the methods disclosed herein may optionally be supplemented by any of the features and functions described in relation to the apparatus.

[0102] Additionally, any of the features and functions described herein may be implemented in hardware or software, or using a combination of hardware and software, as described in the "Alternative Implementations" section.

[0103] Proposal - Tensor dimension reordering by tensor dimension shift In the following, an embodiment relating to “[NNC] Tensor dimension rearrangement by tensor dimension shift” will be described.

[0104] 1 Summary (example, details optional) According to one aspect, this input contribution proposes a method for reordering tensor dimensions, for example to address issues related to the use of NNR_PT_BLOCK. For example, in the encoder, the tensor dimensions are rearranged, but for example, in the decoder, this operation is reversed to restore the original dimension order. For example, in the case of the standard relevant part (i.e., in the decoder), the tensor dimensions are reordered in the decoded weight tensor by shifting the first dimension to its original position after decoding. For example, this addresses two issues. First, for example, the compressed data unit type NNR_PT_BLOCK derives dimensions of, for example, bias, and / or batch norm, and / or local scaling parameters from the first dimension of the weight tensor. For example, reordering of tensor dimensions after decoding allows the encoder to process tensors with any dimension order. Second, the order of tensor dimensions may not be optimal for efficient encoding.

[0105] 2. Problem Description 2.1 Problem 1: Deriving the tensor dimensions of NNR_PT_BLOCK For example, the current working draft [1] allows coding several floating parameter tensors in a single NDU as a block (e.g., nnr_compressed_data_unit_payload_type==NNR_PT_BLOCK). A block may, for example, include a weight tensor (optionally decomposed), and optionally a local scaling parameter, and / or a bias and / or batch norm parameter. For example, the tensor dimensions coded in the NDU relate to the weight tensor. For example, the dimensions of the other parameter tensors are then derived as follows: For example, each tensor is 1D, with its length, for example, equal to the first dimension of the weight tensor. Here, for example, we assume that the first dimension of the weight tensor corresponds to the number of output channels (for fully connected layers) or the number of filters (for convolutional layers). Therefore, whenever the dimensions of the weight tensor to be coded are arranged in another order, the tensor dimension derivation of the other parameters in the block may fail, resulting, for example, in an undecodable bitstream.

[0106] 2.2 Problem 2: Inefficient Processing For example, in the coding process, each tensor to be compressed is (optionally) converted into a 2D matrix, e.g., with a first dimension equal to the first dimension of the tensor and a second dimension equal to the product of all other tensor dimensions. Depending on the first dimension of the tensor, this can produce a fairly "slim" matrix, e.g., if the first dimension indicates the width or height of a 3x3 filter kernel. This may not be optimal for the subsequent coding stage, for example, resulting in reduced coding efficiency or inefficient processing, especially in the case of block scan order (scan_order>0) where each tensor is decomposed into blocks (e.g., 4x4, 8x8, etc.).

[0107] 3. Tensor dimension reordering by tensor dimension shift To address one or more of the problems described in section 2, according to one aspect of the invention, this contribution proposes a method for tensor dimension reordering by shifting a single tensor dimension to another position. For this, for example, the encoder reorders the tensors such that the tensor dimension representing, for example, the number of output channels is the first dimension after reordering. For example, from the decoder's point of view, the decoded tensor is reordered by shifting the first dimension of the tensor to another position indicated by a single value. As an example, the concept is illustrated in Fig. 6. Fig. 6 is a schematic block diagram of the concept of tensor dimension reordering for an encoding-decoding pipeline according to an embodiment of the invention. For example, each box corresponds to a tensor / matrix representation with dimensions given in square brackets. For example, the solid black arrows / lines indicate the general processing flow, while the dashed lines / arrows indicate the processing flow of the syntax element first_tensor_dimension_shift. It should be noted that the use of 2D matrices is optional.

[0108] As shown in FIG. 6, the input tensor 610 may be a four-dimensional tensor having, as an example, a D0 element in the first dimension, a D1 element in the second dimension, a D2 element in the third dimension, and a D3 element in the third dimension. The elements may be, for example, neural network parameters. As shown, using, for example, an encoder 300 as shown in FIG. 3, e.g., its reordering unit 320, the input tensor 610 (which may be an example of a multidimensional array 302) may be reordered 612 to obtain a reordered tensor 620 (which may be an example of a multidimensional array 620). As shown, a given dimension, in this case D2, may be reordered to the first dimension in the reordered tensor 620. Therefore, as an optional feature, a first_tensor_dimension_shift 630 (which may be an example of a dimension shift value) is used.

[0109] As already explained, such dimension shift value may describe which given dimension of a given multidimensional array is permuted to the first dimension in the permuted multidimensional array, or by how many dimensions the first dimension of the encoded permuted multidimensional array should be shifted when performing the decoder-side permutation. Furthermore, the dimension shift value may describe a new position to which the first dimension of the encoded permuted multidimensional array should be moved in the decoder. Furthermore, each encoder, e.g., 300, may be configured to encode 632 the dimension shift value into the bitstream 650 (which may be an example of the bitstream 102, 202, and / or 303), optionally using an Exp-Golomb code. In particular, the dimension shift value may optionally be a single scalar dimension shift value as the only parameter of the permutation 612. As shown in FIG. 6, the permutation 612 may include a single shift of a single dimension D2 to another position, i.e., the first position, to obtain a permuted multidimensional array in the form of a permuted tensor 620.

[0110] Based on the permuted tensor 620, as an optional feature, a 2D matrix 640 or a 2D matrix interpretation 640 may optionally be obtained using a 2D transformation 622 or a 2D interpretation 622. As shown, a first dimension of the two-dimensional matrix 640 may be determined by a first dimension of the permuted multidimensional array 620, such that a second dimension of the two-dimensional matrix 640 is determined by a product of further dimension values ​​of the permuted multidimensional array 620 (in the form of the permuted tensor 620).

[0111] As shown, the two-dimensional matrix 640 may then be encoded 642 in a bitstream 650. Note that the 2D transform 622 or 2D interpretation 622 is only optional, whereby the reordered tensor 620 may optionally be directly encoded.

[0112] In either case, optionally, each encoder, for example 300, may use context-based entropy coding for encoding the neural network parameters.

[0113] A decoder, e.g., decoder 100, 200, receives a bitstream 650 including dimension shift value information 630, as optionally shown. The bitstream is decoded 652, e.g., using a decoding unit 110, 210, to obtain 634 a decoded 2D-matrix 660 or a decoded 2D-matrix interpretation 660, and using the decoding to obtain information regarding the dimension shift 630. Thus, in other words, based on the coded representation, a two-dimensional matrix 660 is obtained whose first dimension is determined by a first dimension value and whose second dimension is determined by a product of multiple further dimension values.

[0114] Based thereon and based on the tensor transformation 662 or the tensor interpretation 662, the decoder obtains, as an example, a reconstructed tensor 670 (which may be an example of the first multidimensional array 111, 211) corresponding to the reconstructed tensor 620. In simple terms, based on the two-dimensional matrix 660, the original number of dimensions is reconstructed, the first dimension of the matrix 660 corresponding to the first dimension of the tensor 670. The reconstructed tensor 670 is then reconstructed 672, for example using the reconstructing unit 120, 220, to obtain an input tensor 680 corresponding to or optionally equal to the input tensor 610. Thus, as an example, the first_tensor_dimension_shift 630 (e.g., information 112, 212) is used.

[0115] 3.1 Suggested syntax (e.g., details are optional) To this end, e.g., whenever count_tensor_dimensions is greater than 1, a new syntax element first_tensor_dimension_shift is decoded from the bitstream (e.g., indicating a new position of the first dimension). For example, the value of first_tensor_dimension_shift is non-negative and less than the value of count_tensor_dimensions. For example, a value equal to 0 is equivalent to no reordering. For example, if first_tensor_dimension_shift is not present, it is inferred to be 0.

[0116] 3.2 Sorting method - Example For example, the syntax and reordering methods are provided in the working draft text attached to this document (an updated draft of the proposed, planned International Standard ISO / IEC 15938-17). For simplicity, the reordering methods are illustrated by two examples using 2D and 4D tensors, respectively.

[0117] 3.2.1 Example: 2D-tensor, first_tensor_dimension_shift==1 For example, tensor dimension 2D tensor with TIFF2025513886000002.tif516 and first_tensor_dimension_shift equal to 1 TIFF2025513886000003.tif43 is decoded from the bitstream, TIFF2025513886000004.tif43 element TIFF2025513886000005.tif516( TIFF2025513886000006.tif532 and Assuming the image can be accessed by a filename (TIFF2025513886000007.tif530), the sorting process can be, for example, as follows:

[0118] For example, the dimension Sorted tensor with TIFF2025513886000008.tif516 Initialize TIFF2025513886000009.tif43. For example, according to Set the value of TIFF2025513886000010.tif43. for( TIFF2025513886000011.tif412; TIFF2025513886000012.tif415; TIFF2025513886000013.tif34++ ) { for( TIFF2025513886000014.tif411; TIFF2025513886000015.tif414; TIFF2025513886000016.tif33++ ) { TIFF2025513886000017.tif517= TIFF2025513886000018.tif517} } For example, a 2D tensor For TIFF2025513886000019.tif43, the sorted tensor is the transposed tensor Equivalent to TIFF2025513886000020.tif43.

[0119] 3.2.2 Example: 4D-tensor, first_tensor_dimension_shift==2 For example, tensor dimension TIFF2025513886000021.tif531 is a 4D tensor with first_tensor_dimension_shift equal to 2 TIFF2025513886000022.tif43 is decoded from the bitstream, TIFF2025513886000023.tif43 element TIFF2025513886000024.tif528( TIFF2025513886000025.tif532, TIFF2025513886000026.tif530, TIFF2025513886000027.tif530, and Assuming the image can be accessed by a filename (TIFF2025513886000028.tif530), the sorting process is, for example, as follows:

[0120] For example, the dimension Sorted tensor with TIFF2025513886000029.tif531 Initialize TIFF2025513886000030.tif43. For example, according to Set the value of TIFF2025513886000031.tif43. for( TIFF2025513886000032.tif412; TIFF2025513886000033.tif415; TIFF2025513886000034.tif34++ ) { for( TIFF2025513886000035.tif411; TIFF2025513886000036.tif414; TIFF2025513886000037.tif33++ ) { for( TIFF2025513886000038.tif411; TIFF2025513886000039.tif414; TIFF2025513886000040.tif33++ ) { for( TIFF2025513886000041.tif411; TIFF2025513886000042.tif414; TIFF2025513886000043.tif43++ ) { TIFF2025513886000044.tif527= TIFF2025513886000045.tif527} } } }

[0121] 4 Recommendations In accordance with one aspect of the present invention, the inventors recommend that the proposed technique be adopted into a working draft.

[0122] 5 Reference to the chapter "Proposal - Tensor dimension reordering by tensor dimension shift" [1] MPEG, “Working Draft 3 on Incremental Compression of Neural Networks”, Document of ISO / IEC JTC1 / SC29 / WG04, WG04N0178, OnLine, Jan. 2022

[0123] A method for tensor dimension reordering for neural network coding. One aspect of the present invention is to provide a method for coding multiple tensors in a block and / or for efficient processing in neural network compression frameworks, such as the MPEG-7 Part 17 standard for the compression of neural networks for multimedia content description and analysis [2]. This paper describes a method for rearranging the dimensions of a 4-dimensional (13-D) neural network parameter tensor. Such frameworks (e.g., MPEG-7 Part 17) can provide several compression tools, including, for example, quantization and / or lossless encoding and decoding methods. Typically (but not necessarily), for efficient processing, tensors are internally interpreted as equivalent to a one-dimensional (1D) or two-dimensional (2D) representation. The shape of this 1D / 2D representation is determined, for example, by the original tensor dimensions and their order. Furthermore, this shape can also affect, for example, the efficiency of the compression and processing pipeline.

[0124] Thus, for example, reordering of tensor dimensions, as described in this invention, can improve efficiency.

[0125] According to one aspect, the present invention is primarily directed to lossy and lossless coding of layers of neural network parameters in neural network compression, but can optionally be applied to other areas of lossy and lossless coding as well.

[0126] The methodology of the apparatus (or system) can be divided into different main parts, consisting, for example, of: 1. Quantization 2. Lossless encoding 3. Lossless Decoding However, it is not necessary to have all of these main parts.

[0127] In order to understand the main advantages of the present invention, we first provide a brief introduction to the topic of neural networks and related methods for parameter coding.

[0128] 1 Applicable area (all details are optional) In their most basic form, neural networks construct, for example, a chain of affine transformations followed by element-wise nonlinear functions. They can be represented, for example, as directed acyclic graphs, as shown in the following figure. For example, each node is accompanied by a certain value that is, for example, forward propagated to the next node, for example by multiplication with the respective weight value of the edge. For example, all input values ​​are then simply summed.

[0129] Figure 7 is a schematic diagram of a two-layer feedforward neural network, according to an embodiment of the present invention. Figure 7 shows a graphical representation of an example of a feedforward neural network. Specifically, this two-layer neural network 700 is a nonlinear function that maps a four-dimensional input vector 710 to a real line. Thus, the network 700 comprises an input layer 720, a hidden layer 730, and an output layer 740.

[0130] Mathematically, the neural network described above calculates its output in, for example, the following manner: TIFF2025513886000047.tif8105In the formula, for example, W2 and W1 are weight parameters (edge ​​weights) of the neural network, and sigma is, for example, some nonlinear function. For example, so-called convolutional layers can also be used by casting them as a matrix-matrix product as described in [1]. For example, the incremental update is usually intended to provide an update of the weights of W1 and W2, and can be, for example, the result of an additional training process. For example, the updated version of W2 and W1 usually results in a modified output. In the following, the procedure of calculating the output from a given input is called inference. Also, the intermediate results are called hidden layers or hidden activation values, which constitute, for example, a linear transformation + element-wise nonlinearity, such as the calculation of the first inner product + nonlinearity described above.

[0131] Typically, neural networks may require hundreds of MB to represent, for example, since they are typically equipped with millions of parameters. As a result, they require high computational resources to be performed, for example, since their inference procedures involve the calculation of many dot product operations between large matrices. It is therefore very important to reduce the complexity of performing these dot products.

[0132] 2. Traditional Solutions It should be noted that, optionally, any of the feature functions and details from this section "Conventional Solutions" may be used in embodiments according to the present invention, either individually or in combination.

[0133] Furthermore, note that typically (but not necessarily) parameters (e.g., parameters of a neural network, such as the weights of the neural network) are in N-dimensional tensors (or, equivalently, in N-dimensional arrays).

[0134] 2.1 Tensor representation of neural networks (all details optional) For example, the parameters of a neural network are usually (but not necessarily) TIFF2025513886000048.tif4 is represented by a three-dimensional tensor, TIFF2025513886000049.tif43 Dimensions depend on, for example, the model architecture and application. For example, TIFF2025513886000050.tif43 is equal to 1, If TIFF2025513886000051.tif43 is equal to 2, this corresponds to a vector and a matrix, respectively. The size of a tensor in each dimension is, for example, of length: It can be represented using an array of TIFF2025513886000052.tif43. For example, in the case of processing in an encoder and decoder conforming to the MPEG-7 part 17 standard, 3 or more All tensors with TIFF2025513886000054.tif43 will be stored in a .tif file, for example (preferably, but not necessarily) if they The tensor dimensions are interpreted as if they were a 2D matrix of dimensions equal to TIFF2025513886000055.tif549. If not available at the decoder, the standard specifies how to transmit the tensor dimensions in the bitstream, e.g. so that the original tensor shape can be reconstructed after decoding. For the methods described below, it can be assumed, for example, that the weight parameters are represented or interpreted as vectors or 2D matrices, respectively.

[0135] 2.2 Related methods for quantization and entropy coding (all details are optional) For example, the MPEG-7 Part 17 standard for compression of neural networks for multimedia content description and analysis [2] provides different methods for the quantization of neural network parameters, such as independent scalar quantization and dependent scalar quantization (DQ or also trellis coded quantization TCQ). Furthermore, it specifies an entropy quantization scheme, also known as deepCABAC [7]. These methods, which can be optionally used in embodiments according to the invention, are briefly summarized for a better understanding. Details can be found, for example, in [2].

[0136] 2.2.1 Scalar quantizer (example) For example, the neural network parameters can be quantized using a scalar quantizer. For example, as a result of the quantization, the set of allowable values ​​of the parameters is reduced. In other words, the neural network parameters are, for example, mapped to a countable set (in practice, a finite set) of so-called reconstruction levels. The set of reconstruction levels represents, for example, a proper subset of the set of possible neural network parameter values. For example, to simplify the entropy coding below, the allowable reconstruction levels are represented, for example, by quantization indexes, which are transmitted, for example, as part of the bitstream. On the decoder side, the quantization indexes are mapped to the reconstructed neural network parameters. For example, the possible values ​​of the reconstructed neural network parameters correspond to the set of reconstruction levels. On the encoder side, the result of the scalar quantization is, for example, a set of (integer) quantization indexes.

[0137] In this application, for example, a uniform reconstruction quantizer (URQ) is used. As an example, its basic design is shown in FIG. 8. FIG. 8 is a schematic diagram of a uniform reconstruction quantizer according to an embodiment of the present invention. For example, URQ has the property that the reconstruction levels are equally spaced. The distance Δ between two adjacent reconstruction levels is called the quantization step size. For example, one of the reconstruction levels is equal to 0. Thus, the complete set of available reconstruction levels is uniquely specified, for example, by the quantization step size Δ. For example, the decoder mapping of the quantization index q to the reconstructed weight parameter t' is, in principle, given by the following simple formula: TIFF2025513886000056.tif621 In this context, the term "independent scalar quantization" refers to the property that, for example, given a quantization index q of any weighting parameter, the associated reconstructed weighting parameter t' can be determined independently of all quantization indexes of the other weighting parameters.

[0138] 2.2.2 Dependent Scalar Quantization (e.g., all details are optional) In dependent scalar quantization (DQ), the acceptable reconstruction level of a neural network parameter depends on the selected quantization index of the preceding neural network parameter in the reconstruction order. The concept of dependent scalar quantization is combined with modified entropy coding, where the probability model selection (or alternatively, the codeword table selection) of the neural network parameter depends on the set of acceptable reconstruction levels. The advantage of dependent quantization of neural network parameters is that the acceptable reconstruction vectors are packed more densely in the N-dimensional signal space (where N indicates the set of samples to be processed, e.g., the number of samples or neural network parameters in a layer). The reconstruction vector of a set of neural network parameters refers to the ordered reconstructed neural network parameters (or alternatively, the ordered reconstructed samples) of the set of neural network parameters. The effect of dependent scalar quantization is illustrated in FIG. 9 for the simplest case of two neural network parameters. 9 shows schematic examples of positions of allowable reconstruction vectors for the simple case of two weight parameters according to an embodiment, where (a) shows independent scalar quantization and (b) shows dependent scalar quantization. FIG. 9a shows allowable reconstruction vectors for independent scalar quantization (which represents a point 910 in the 2d plane). As can be seen, the second neural network parameter The set of acceptable values ​​in TIFF2025513886000057.tif84 is the first reconstructed neural network parameter In contrast to independent scalar quantization, the second neural network parameter The selectable reconstruction values ​​for TIFF2025513886000059.tif84 are the first neural network parameters Note that the reconstruction level of TIFF2025513886000060.tif84 depends on the selected reconstruction level. In the example of Fig. 9b, the second neural network parameter There are two different sets of available reconstruction levels for TIFF2025513886000061.tif84 (e.g., shown in different colors as set 920 and set 930). First Neural Network Parameter If the quantization index of TIFF2025513886000062.tif84 is even (..., -2, 0, 2, ...), we can use any reconstruction level of the first set (blue point, 920) as the second neural network parameter You can select for TIFF2025513886000063.tif84. And the first neural network parameters If the quantization index of TIFF2025513886000064.tif84 is odd (..., -3, -1, 1, 3, ...), we can use any reconstruction level of the second set (red point, 930) with the second neural network parameters For TIFF2025513886000065.tif84, in this example the reconstruction levels of the first and second sets are shifted by half the quantization step size (any reconstruction level of the second set lies between two reconstruction levels of the first set).

[0139] The dependent scalar quantization of neural network parameters has the effect that for a given average number of reconstructed vectors per N-dimensional unit volume, the expected value of the distance between a given input vector of neural network parameters and the closest available reconstructed vector is reduced. As a result, the average distortion between the input vector of neural network parameters and the vector reconstructed neural network parameters can be reduced for a given average number of bits. In vector quantization, this effect is called space-filling gain. Using dependent scalar quantization of a set of neural network parameters, most of the potential space-filling gain of high-dimensional vector quantization can be utilized. Also, in contrast to vector quantization, the complexity of the implementation of the reconstruction process (or the decoding process) is comparable to the complexity of the implementation of the related neural network parameter coding with an independent scalar quantizer.

[0140] As a result of the above aspects, DQ typically achieves the same distortion level at a lower bit rate.

[0141] 2.2.3 DQ for MPEG-7 part 17 (e.g., all details are optional) The MPEG-7 part 17 standard for compression of neural networks for multimedia content description and analysis employs two quantizers Q1 and Q2 with different sets of reconstruction levels. Both sets include integer multiples of the quantization step size Δ. Q1 includes all even multiples of the quantization step size and 0, and Q2 includes all odd multiples of the quantization step size and 0. This division of reconstruction sets is illustrated in FIG. 10, which shows a schematic example for dividing a set of reconstruction levels into two subsets according to an embodiment. The two subsets of quantization set 0 are labeled using "A" and "B", and the two subsets of quantization set 1 are labeled using "C" and "D".

[0142] The process for switching between sets determines which quantizer to apply based on the quantization index selected for the preceding neural network parameter in the reconstruction order, or more precisely, based on the parity of the previously coded quantization index. This switching process is realized by a finite state machine with eight states (shown in Table 16), each state associated with either quantizer Q1 or Q2.

[0143] [Table 1] Using the concept of state transitions, the current state, and therefore the current quantization set, is uniquely determined by the previous state (in reconstruction order) and the previous quantization index.

[0144] 2.2.4 Entropy Coding (e.g., all details optional) As a result of the quantization applied in the previous step, the weight parameters are mapped to a finite set of so-called reconstruction levels. These can be represented by (integer) quantizer indices (also called parameter levels or weight levels) and quantization step sizes, which may for example be fixed for the entire layer. To recover all quantized weight parameters of a layer, the step sizes and dimensions of the layer can be known by the decoder. They may for example be transmitted separately.

[0145] In this section, the scan can be described. For example, a section for block coding (NNR_PT_BLOCK) can be described here.

[0146] 2.2.4.1 Traversing Quantization Indexes (i.e., all details are optional) The quantization indexes (integer representation) are then transmitted using entropy coding techniques. Thus, the weight layers are mapped to a sequence of quantized weight levels using a scan. For this purpose, five different scan orders, indicated by the syntax element scan_order, are specified in the standard. The first one (scan_order==0) represents a row-first operation that starts from the top row of the matrix and encodes the contained values ​​from left to right. In this way, all rows are encoded from top to bottom. All other scan orders (scan_order>0) correspond to a block scan order. Here, the matrix is ​​decomposed into blocks of size 4x4 (scan_order==1), 8x8 (scan_order==2), 16x16 (scan_order==3) or 32x32 (scan_order==4) and the blocks are processed block-row by block, i.e. starting from the top block row and processing the blocks from left to right. Processing a block means scanning its values ​​in a row-major fashion (rows from top to bottom, left to right within each row).

[0147] 2.2.4.2 Coding of quantization indexes using context-adaptive binary arithmetic coding (CABAC) (e.g., all details optional) The level coding is done using CABAC (Context-Adaptive Binary Arithmetic Coding). For details, see [2]. Therefore, the quantized weight levels TIFF2025513886000067.tif43 can be decomposed into a series of binary symbols or syntax elements and then passed to a binary arithmetic coder (CABAC).

[0148] In the first step, a binary syntax element sig_flag of the quantized weight level is derived that specifies whether the corresponding level is equal to 0. If sig_flag is equal to 1, a further binary syntax element sign_flag is derived. The bin indicates whether the current weight level is positive (e.g., bin = 0) or negative (e.g., bin = 1).

[0149] Next, the single-term sequence of the bin is encoded, followed by encoding of the fixed-length sequence as follows. The variable k is initialized with a non-negative integer, and X is initialized with 1 << k.

[0150] One or more syntax elements abs_level_greater_X indicating that the absolute value of the quantized weight level is greater than X are encoded. If abs_level_greater_X is equal to 1, the variable k is updated (e.g., incremented by 1), then 1 << k is added to X, and further abs_level_greater_X is encoded. This procedure continues until abs_level_greater_X is equal to 0. Thereafter, the fixed-length code of length k is sufficient to complete the encoding of the quantizer index. For example, the variable TIFF2025513886000068.tif528 can be encoded using k bits. Alternatively, the variable TIFF2025513886000069.tif410 is encoded using k bits and can be defined as TIFF2025513886000070.tif6109. Alternatively, any other mapping of the variable to the k-bit fixed-length code TIFF2025513886000071.tif38 may be used.

[0151] When k is incremented by 1 after each abs_level_greater_X, this approach is identical to applying exponential Golomb coding (when sign_flag is not considered).

[0152] Furthermore, if the maximum absolute value abs_max is known at the encoder and decoder sides, the encoding of the abs_level_greater_X syntax element may be terminated, and when the next abs_Level_greater_X is transmitted, X>=abs_max holds.

[0153] 2.2.4.3 Decoding Quantization Indexes using Context-Adaptive Binary Arithmetic Coding (CABAC) (e.g., all details optional) Decoding the quantized weight levels (integer representation) works similarly to encoding. The decoder first decodes sig_flag. If this is equal to 1, then a single term sequence of sign_flag and abs_level_greater_X follows, where the update of k (and therefore the increment of X) must follow the same rules as the encoder. Finally, it decodes the k-bit fixed length code and returns an integer (e.g., TIFF2025513886000072.tif38 and Depending on whether TIFF2025513886000073.tif410 or TIFF2025513886000073.tif410 was encoded, TIFF2025513886000074.tif38 or TIFF2025513886000075.tif410). Then the decoded quantized weight levels The absolute value of TIFF2025513886000076.tif55 can be reconstructed from X to form the fixed length part. For example, If TIFF2025513886000077.tif38 is used as the fixed length part, TIFF2025513886000078.tif528. Or, When TIFF2025513886000079.tif410 is encoded, TIFF2025513886000080.tif6119. As a final step, the sign is Quantized weighting levels applied to TIFF2025513886000081.tif55 We need to generate TIFF2025513886000082.tif43. Finally, the quantized weight levels Step size to TIFF2025513886000083.tif43 Quantized weights by multiplying TIFF2025513886000084.tif43 TIFF2025513886000085.tif34 is reconstructed.

[0154] In one implementation variant, k is initialized at 0 and updated as follows: After each abs_level_greater_X equals 1, the necessary updates of k are made according to the following rules: if X>X', k is incremented by 1, where X' is a constant depending on the application. For example, X' is a number (e.g., between 0 and 100) derived by the encoder and signaled to the decoder.

[0155] 2.2.4.4 Context Modeling (e.g., all details are optional) In CABAC entropy coding, most syntax elements of the quantized weight levels are coded using binary probability modeling. Each binary decision (bin) is associated with a context. The context represents a probability model of the class of coded bins. The probability of one of the two possible bin values ​​is estimated for each context based on the values ​​of the bins already coded in the corresponding context. Depending on the application, different context modeling techniques can be applied. Typically, for some bins related to the coding of quantized weights, the context used for coding is selected based on the syntax elements already transmitted. Depending on the actual application, different probability estimators can be selected, for example, those of SBMP [4], or HEVC [5] or VTM-4.0 [6]. The choice affects, for example, compression efficiency and complexity.

[0156] A context modeling scheme that is suitable for a wide range of neural networks is described below: The quantized weight level at a particular location (x,y) in the weight matrix (layer). To decode TIFF2025513886000086.tif43, a local template is applied to the current position. This template contains several other (ordered) positions, e.g. (x-1, y), (x, y-1), (x-1, y-1), etc. For each position, a state identifier is derived.

[0157] In an implementation variant (denoted as Si1), the state identifier for the position (x,y) TIFF2025513886000087.tif47 is derived as follows: if position (x,y) points outside the matrix, or if the quantized weight level at position (x,y) If TIFF2025513886000088.tif48 has not yet been decoded or is equal to 0, the state identifier TIFF2025513886000089.tif515. Otherwise, the status identifier is Let it be TIFF2025513886000090.tif542.

[0158] For a particular template, a sequence of state identifiers is derived and each possible constellation of state identifier values ​​is mapped to a context index to identify the context in which it will be used. The templates and mappings may be different for different syntax elements. For example, an ordered sequence of state identifiers may be derived from a template containing (ordered) positions (x-1,y), (x,y-1), (x-1,y-1). TIFF2025513886000091.tif412, TIFF2025513886000092.tif411, For example, this sequence is derived from the context index For example, the context index TIFF2025513886000095.tif43 may be used to identify the number of contexts for sig_flag.

[0159] In one implementation variant (shown in Method 1), the quantized weight level at position (x,y) The local template for sig_flag or sign_flag of TIFF2025513886000096.tif48 consists of only one position (x-1,y) (i.e., the left neighbor). Associated state identifier TIFF2025513886000097.tif412 is derived according to the implementation variant Si1.

[0160] For sig_flag, Depending on the value of TIFF2025513886000098.tif412, one of three contexts is selected, or for sign_flag, Depending on the value of TIFF2025513886000099.tif412, one of the other three contexts is selected.

[0161] In another implementation variant (shown in technique 2), the local template of the sig flag includes three ordered positions: (x-1, y), (x-2, y), (x-3, y). TIFF2025513886000100.tif412, The associated sequence of TIFF2025513886000101.tif424 is derived according to the implementation variant Si2.

[0162] For sig_flag, the context index TIFF2025513886000102.tif43 is derived as follows: In the case of TIFF2025513886000103.tif722TIFF2025513886000104.tif520, TIFF2025513886000105.tif411. Otherwise, In the case of TIFF2025513886000106.tif520TIFF2025513886000107.tif722, TIFF2025513886000108.tif411. Otherwise, In the case of TIFF2025513886000109.tif520TIFF2025513886000110.tif722, TIFF2025513886000111.tif411. Otherwise, The file is TIFF2025513886000112.tif411.

[0163] This can also be expressed by the following formula: Similarly, the number of neighbors to the left can be increased or decreased so that the context index C is equal to the distance to the next non-zero weight (not exceeding the template size) to the left.

[0164] Each abs_level_greater_X flag can, for example, apply its own set of two contexts, after which one of the two contexts is selected depending on the value of sign_flag.

[0165] In one implementation variant, for abs_level_greater_X flags where X is less than a predefined number X', different contexts are distinguished depending on the values ​​of X and / or sign_flag.

[0166] In one implementation variant, for abs_level_greater_X flags where X is greater than or equal to a predefined number X', different contexts are differentiated depending on X only.

[0167] In another embodiment, the abs_level_greater_X flag where X is greater than or equal to a predetermined number X' is encoded using a fixed code length of 1 (e.g., using the bypass mode of an arithmetic coder).

[0168] Furthermore, some or all of the syntax elements may also be encoded without using a context. Instead, they are encoded with a fixed length of 1 bit. For example, the so-called bypass bins of CABAC are used.

[0169] In another embodiment, the remainder of the fixed length TIFF2025513886000114.tif38 is encoded using the bypass mode.

[0170] In another embodiment, the encoder determines a predetermined number X', distinguishes two contexts according to the sign for each syntax element abs_level_greater_X where X < X', and uses one context for each abs_level_greater_X where X >= X'.

[0171] 2.2.4.5 Context Modeling for Dependent Scalar Quantization (Example, all details are optional) The main aspect of dependent scalar quantization is that there are different sets of permissible reconstruction levels (also called quantization sets) of neural network parameters. The quantization set of the current neural network parameter is determined based on the value of the quantization index of the preceding neural network parameter. Considering the preferred example of FIG. 10 and comparing the two quantization sets, it is clear that the distance between the reconstruction level equal to 0 and the adjacent reconstruction level is larger in set 0 than in set 1. Thus, the probability of the quantization index being equal to 0 is larger when set 0 is used and smaller when set 1 is used. In one implementation variant, this effect is exploited in entropy coding by switching the codeword table or probability model based on the quantization set (or state) used for the current quantization index.

[0172] It should be noted that for proper switching of codeword tables or probability models, when entropy decoding the current quantization index (or the corresponding binary decision of the current quantization index), the path (association with the subset of the quantization set used) of all previous quantization indexes needs to be known. Therefore, the neural network parameters need to be coded in reconstruction order. Therefore, in one implementation variant, the coding order of the neural network parameters is equal to their reconstruction order. In addition to that aspect, any coding / reconstruction order of the quantization indexes is possible, such as the one specified in section 2.2.4.2, or any other uniquely defined order.

[0173] At least some of the bins of absolute levels are typically coded using adaptive probability models (also called contexts). In one implementation variant, the probability model of one or more bins is selected based on the quantization set of the corresponding neural network parameters (or, more generally, the corresponding state variables). The selected probability model can depend on several parameters or characteristics of the quantization indexes already transmitted, one of the parameters being the quantization set or state applied to the quantization index being coded.

[0174] In another implementation variant, the syntax for transmitting the quantization index of a layer includes a bin that specifies whether the quantization index is equal to 0 or not. The probability model used to code this bin is selected from a set of two or more probability models. The choice of the probability model used depends on the quantization set (i.e., the set of reconstruction levels) applied to the corresponding quantization index. In another implementation variant, the probability model used depends on the current state variables (which imply the quantization set used).

[0175] In a further implementation variant, the syntax for transmitting the quantization index of a layer includes a bin that specifies whether the quantization index is greater than or less than 0. In other words, the bin indicates the sign of the quantization index. The choice of the probability model used depends on the quantization set (i.e., the set of reconstruction levels) applied to the corresponding quantization index. In another implementation variant, the probability model used depends on the current state variables (which imply the quantization set used).

[0176] In a further implementation variant, the syntax for transmitting the quantization index includes a bin that specifies whether the absolute value of the quantization index (neural network parameter level) is greater than X (see section 2.2.4.2 for details). The probability model used to code this bin is selected from a set of two or more probability models. The choice of the probability model used depends on the quantization set (i.e. the set of reconstruction levels) applied to the corresponding quantization index. In another implementation variant, the probability model used depends on the current state variables (which imply the quantization set used).

[0177] In one embodiment, dependent quantization of neural network parameters is combined with entropy coding, in which the selection of a probability model for one or more bins of the binary representation of a quantization index (also called a quantization level) depends on a quantization set (a set of allowable reconstruction levels) or a corresponding state variable of the current quantization index, where the quantization set (or state variable) is given by the quantization index (or a subset of bins representing the quantization index) of the preceding neural network parameter in the coding and reconstruction order.

[0178] In one implementation variant, the described choice of probability model is combined with one or more of the following entropy coding aspects:

[0179] The absolute values ​​of the quantization indexes are transmitted using a binarization scheme consisting of some bins coded using an adaptive probability model and a suffix part coded in bypass mode of the arithmetic coding engine (non-adaptive probability model with pmf(0.5,0.5) for all bins) if the adaptive coding bins do not yet fully specify the absolute values. In one implementation variant, the binarization used for the suffix part depends on the values ​​of the quantization indexes already transmitted.

[0180] The binarization of the absolute value of the quantization index includes an adaptively coded bin that specifies if the quantization index is not equal to 0. The probability model (called context) used to code this bin is selected from among a set of candidate probability models. The selected candidate probability model is determined not only by the quantization set (set of acceptable reconstruction levels) or state variables of the current quantization index, but also by the already transmitted quantization indexes of the layer. In one implementation variant, the quantization set (or state variables) determines a subset of the available probability models (also called context set) and the values ​​of the already coded quantization indexes determine the probability model to be used within this subset (context set).

[0181] In one implementation variant, the probability model to be used within a context set is determined based on the values ​​of the already coded quantization indexes in the local neighborhood of the current neural network parameters. Below we list some exemplary measures that can be derived based on the values ​​of the quantization indexes in the local neighborhood and then used to select a probability model for a given context set.

[0182] The sign of the quantization index that is not equal to 0 in the local neighborhood. The number of quantization indices in the local neighborhood that are not equal to 0. This number can optionally be clipped to a maximum value. The sum of the absolute values ​​of the quantization indexes in a local neighborhood. This number can be clipped to a maximum value. The difference between the sum of the absolute values ​​of the quantization indexes in a local neighborhood and the number of quantization indexes in the local neighborhood that are not equal to 0. This number can be clipped to a maximum value.

[0183] The binarization of the absolute value of the quantization index includes adaptively coded bins that specify whether the absolute value of the quantization index is greater than X. The probability model (called context) used to code these bins is selected from a set of candidate probability models. The selected probability model is determined not only by the quantization set (set of acceptable reconstruction levels) or state variables of the current quantization index, but also by the already transmitted quantization indexes of the layer. In one embodiment variant, the quantization set (or state variables) determines a subset of the available probability models (also called context set), and the data of the already coded quantization indexes determines the probability model to be used within this subset (context set). The selection of the probability model can use any of the methods described above (for bins that specify the quantization index is not equal to 0).

[0184] 2.3 Related methods for compressing incremental updates of neural networks (e.g., all details optional) This section describes a method for encoding incremental updates of a neural network, where the reconstructed network layers are a decoding of the existing base layer (of the base model) and one or more incremental update layers, which may be coded and transmitted separately.

[0185] 2.3.1 Base model and updated model concept (i.e. all details are optional) This concept introduces a neural network model according to Section 1 that can be considered a complete model in the sense that it is possible to compute an output for a given input. We call this model the base model. Each base model is represented as a base layer. The base layer contains base values ​​that can be selected, for example, so that they can be efficiently represented or compressed / transmitted in a first step. Furthermore, this concept is applied to the update model ( TIFF2025513886000117.tif535, which may have an architecture similar to or even identical to the base model. The updated models do not have to be, for example, complete models in the sense described above. Instead, the updated models are combined with the base model using composition methods, so that they form a new complete model. TIFF2025513886000118.tif58 can be formed. This model itself can serve as a base model for further updated models. Updated Model TIFF2025513886000119.tif58 is the update layer It consists of a layer called TIFF2025513886000120.tif538. The update layer contains, for example, base values ​​that can be selected such that they can be efficiently represented or compressed / transmitted separately.

[0186] The updated model may be the result of an (additional) training process applied to the base model at the encoder side. Depending on the type of updates provided by the updated model, several synthesis methods can be applied. It should be noted that the methods described within this invention are not limited to any particular type of update / synthesis method, but are applicable to any architecture that uses the base model / updated model approach.

[0187] In a preferred embodiment, TIFF2025513886000121.tif 43rd updated model TIFF2025513886000122.tif58 is a new model layer according to the following: Base model to form TIFF2025513886000123.tif59 A layer with difference values ​​(also called incremental updates) to be added to the corresponding layer in TIFF2025513886000124.tif56 Includes TIFF2025513886000125.tif59. TIFF2025513886000126.tif553The new model layer forms the (updated) new model, which then serves as the base model for the next incremental update, which is sent separately.

[0188] In a further preferred embodiment, TIFF2025513886000127.tif4The third update model is the new model according to the following The corresponding base layer to form TIFF2025513886000128.tif59 A layer with a scaling factor value multiplied by the value in TIFF2025513886000129.tif56 Includes TIFF2025513886000130.tif59. TIFF2025513886000131.tif552The new model layer forms the (updated) new model, which then serves as the base model for the next incremental update, which is sent separately.

[0189] In some cases, the update model may also update one or more existing layers (i.e., for layer k: Note that you can include a new layer that replaces the old one (TIFF2025513886000132.tif644).

[0190] 2.3.2 Coding Neural Network Parameters for Incremental Updates (e.g., all details optional) The concept of base model and one or more incremental updates can be utilized in the entropy coding stage to improve coding efficiency. The parameters of the layers are usually represented by multidimensional tensors. For the coding process, all tensors are usually mapped into a 2D matrix, such as entities like rows and columns. Then, this 2D matrix is ​​scanned in a predefined order and the parameters are coded / transmitted. It should be noted that the method described below is not limited to 2D matrices. The method is applicable to all representations of neural network parameters that provide parameter entities of known size, such as rows, columns, blocks, etc., and / or combinations thereof. The 2D matrix representation is used below to better understand the method.

[0191] In a preferred embodiment, the parameters of the layers are represented as a 2D matrix providing value entities such as rows and columns.

[0192] 2.3.2.1 Line or Channel Skip Mode (i.e., all details are optional) Typically, the magnitude of the values ​​in the updated model is small compared to the full (base) model. Often, a significant number of values ​​are zero, which is also further amplified by the quantization process. As a result, the transmitted layers can contain long zero sequences, which means that some rows of the 2D matrix are entirely zero.

[0193] This can be exploited by introducing a flag (skip_row_flag) for each row. This flag specifies whether all parameters of the row are equal to 0 or not. If the flag is equal to 1 (skip_row_flag==1), no further parameters are encoded for that row. On the decoder side, if the flag is equal to 1, no parameters are decoded for this row; instead, they are assumed to be 0.

[0194] A variant here is to place all the skip_row_flags in a flag array skip_row_flag[N], where N is the number of rows. Also, in a variant, N could be signaled before the array.

[0195] Otherwise, if the flag is equal to 0, the parameters are encoded and decoded regularly for this row.

[0196] Each of the skip_row_flags is associated with a probability model or context model, which is selected from a set of context models based on previously coded symbols (e.g., previous coding parameters or skip_row_flags).

[0197] In a preferred embodiment, a single context model applies to all skip_row_flags of a layer.

[0198] In another preferred embodiment, a context model is selected from a set of two context models based on the previously coded value of skip_row_flag: a first context model if the preceding skip_row_flag value is equal to 0, and a second context model if the value is equal to 1.

[0199] In a further preferred embodiment, a context model is selected from a set of two context models based on the value of skip_row_flag at the same location in the corresponding layer of a previously coded update or base model: a first context model if the preceding skip_row_flag value is equal to 0, and a second context model if the value is equal to 1.

[0200] In another preferred embodiment, for example as in the foregoing embodiment, a given number of context models are doubled to form two sets of context models. Next, a set of context models is selected based on the value of a skip_row_flag placed at the same location within the corresponding layer of a particular previously encoded update or base model. This means that if the value of the preceding skip_row_flag is equal to 0, the first set is selected, and if the value is equal to 1, the second set is selected.

[0201] A further preferred embodiment is equal to the foregoing embodiment, except that if there is no layer corresponding to a particular previously encoded update or base model, the first set of context models is selected. As a result, if there is a layer corresponding to a particular previously encoded update or base model, the second set is selected.

[0202] It should be noted that the particular described mechanism for skipping rows applies equally to columns in the case of a 2D matrix and in the case of a generalized tensor having N parameter dimensions, and that the described mechanism of skip_flag or skip_flag_array can be used to skip sub-blocks or sub-rows of a smaller dimension K (K < N).

[0203] 2.3.2.2 Improved Context Modeling for the Base Model Update Model Structure (Example, all details are optional) The concept of a base model and one or more update models can be utilized during the entropy coding stage. The methods described herein are applicable to any entropy coding scheme that uses context models, such as those described, for example, in Section 2.2.4.

[0204] Typically, separate update models (and base models) are available correlated at the encoder and decoder sides, which can be used in the context modeling stage to improve coding efficiency by providing new context models and methods for context model selection.

[0205] In the preferred embodiment, binarization (sig_flag, sign_flag, etc.), context modeling and encoding schemes according to section 2.2.4.2 are applied.

[0206] In another preferred embodiment, the context models (context sets) of a given number of symbols to be coded are replicated to form a set of two or more context models. A set of context models is then selected based on the values ​​of co-located parameters in a corresponding layer of a particular previously coded update or base model. This is done by selecting the set of context models based on the values ​​of the co-located parameters within a corresponding layer of a particular previously coded update or base model. The first set is selected if the value is lower than the threshold The second set is selected if the value is greater than or equal to the threshold TIFF2025513886000135.tif55 or greater, a third set is selected, etc. This procedure can be applied with more or fewer thresholds.

[0207] In a preferred embodiment equivalent to the previous embodiment, a single threshold TIFF2025513886000136.tif513 is used.

[0208] In another preferred embodiment, a given number of context models (context sets) of a symbol to be coded are replicated to form a set of two or more context models, and then a set of context models is selected based on a set of values ​​consisting of co-located parameters and adjacent values ​​(e.g., one or more spatial neighbors of the co-located parameters) in a corresponding layer of a particular previously coded update or base model.

[0209] In a preferred embodiment equivalent to the previous embodiment, the sum of the values ​​(or absolute values) in the template is equal to a first threshold value. The first set is selected if the sum is lower than the threshold The second set is selected if the sum is greater than or equal to the threshold. TIFF2025513886000139.tif55 or greater, a third set is selected, etc. This procedure can be applied with more or fewer thresholds.

[0210] In a particularly preferred embodiment equivalent to the previous embodiment, the template includes a co-located parameter and a left neighbor of the co-located parameter, and is scaled by a single threshold TIFF2025513886000140.tif513 is used.

[0211] In another preferred embodiment, a context model from the set of context models for a syntax element is selected based on a set of values ​​consisting of co-located parameters and adjacent values ​​(e.g., one or more spatial neighborhoods of the co-located parameters) within a corresponding layer of a particular previously coded update or base model.

[0212] In a preferred embodiment equivalent to the previous embodiment, the sum of the values ​​(or absolute values) in the template is equal to a first threshold value. The first context model is selected if the sum is lower than the threshold The second context model is selected if the value is greater than or equal to the threshold. TIFF2025513886000143.tif55 or greater then the third context model is selected etc. This procedure can be applied with more or less thresholds.

[0213] In a particularly preferred embodiment equivalent to the previous embodiment, the template includes a co-located parameter and a left neighbor of the co-located parameter, and is scaled by a single threshold TIFF2025513886000144.tif513 is used.

[0214] In a further preferred embodiment, the context models (context sets) of a given number of symbols to be coded are replicated to form a set of two or more context models. A set of context models is then selected based on the absolute values ​​of the co-located parameters in the corresponding layer of a particular previously coded update or base model, which is determined when the absolute values ​​of the co-located parameters exceed a first threshold. The first set is selected if its absolute value is lower than another threshold The second set is selected if the absolute value is greater than or equal to the threshold. TIFF2025513886000147.tif55 or greater, a third set is selected, etc. This procedure can be applied with more or fewer thresholds.

[0215] In a preferred embodiment, which is equivalent to the previous embodiment, a sig_flag is encoded which indicates whether the current value being encoded is equal to 0 or not, which employs a set of context models. Use TIFF2025513886000148.tif513.

[0216] Another preferred embodiment is equal to the previous embodiment, but instead of sig_flag a sign_flag is encoded which indicates the sign of the current value being encoded.

[0217] A further preferred embodiment is equal to the previous embodiment, but instead of sig_flag, abs_level_greater_X is coded which indicates whether the current value being coded is greater than X.

[0218] In a further preferred embodiment, the context models (context sets) of a given number of symbols to be coded are multiplied by two to form two sets of context models. Then, depending on whether there is a corresponding previously coded update (or base) model, the set of context models is selected. If there is no corresponding previously coded update (or base) model, the first set of context models is selected, otherwise the second set is selected.

[0219] In another preferred embodiment, a context model from the set of context models for a syntax element is selected based on the value of a co-located parameter in a particular corresponding previously encoded updated (or base) model, where the co-located parameter is greater than or equal to a threshold. The first model is selected if the value is lower than the threshold The second model is selected if the value is greater than or equal to TIFF2025513886000150.tif55 and is set to another threshold. TIFF2025513886000151.tif55 or greater, a third set is selected, etc. This procedure can be applied with more or fewer thresholds.

[0220] In a preferred embodiment equivalent to the previous embodiment, a sign_flag is encoded which indicates the sign of the current value being encoded. The first threshold for the context model selection process is TIFF2025513886000152.tif513, and the second threshold is The file is TIFF2025513886000153.tif513.

[0221] In another preferred embodiment, a context model from the set of context models for a syntax element is selected based on the absolute value of a co-located parameter in a particular corresponding previously coded updated (or base) model, where the absolute value of the co-located parameter is greater than or equal to a threshold. The first model is selected if the value is lower than the threshold The second model is selected if the value is greater than or equal to the threshold. TIFF2025513886000156.tif55 or greater, a third model is selected, etc. This procedure can be applied with more or fewer thresholds.

[0222] In a preferred embodiment, which is equivalent to the previous embodiment, a sig_flag is encoded which indicates whether the current value being encoded is equal to 0 or not. The first threshold value set in TIFF2025513886000157.tif513 and The second threshold set in TIFF2025513886000158.tif513 is adopted.

[0223] In another preferred embodiment, which is equivalent to the previous embodiment, instead of sig_flag, a flag abs_level_greater_X is coded, which indicates whether the current value being coded is greater than X. Furthermore, Only one threshold is used, which is set in TIFF2025513886000159.tif514.

[0224] It should be noted that any of the above embodiments can be combined with one or more of the other embodiments.

[0225] 2.4 Compressing multiple tensors in a block (e.g., all details optional) Usually, each tensor is coded separately and transmitted in a so-called Compressed Data Unit (NDU), which also contains (header) information about the compressed tensor, e.g., tensor type, coding parameters, or original dimensions of the tensor (explained in section 2.1). However, MPEG-7 Part 17 specifies a special NDU type (NNR_PT_BLOCK) that allows to transmit a weight tensor and several corresponding tensors, namely biases, batch norm parameters, and scaling factors, together in a single NDU. All tensors in a block share the same header information (instead of transmitting individual headers), thus reducing the bitstream size. Since the other tensors are directly related to the weight tensor, their dimensions can be derived from the (weight) tensor dimensions in the following way: The biases, batch norm parameters, and scaling factors are 1D tensors with length equal to the number of output channels (fully connected layers) or the number of filters (convolutional layers) of the weight tensor, respectively. To derive these dimensions, the standard specifies to use the first dimension of the decoded weight tensor as the length of the 1D tensor. This implies that the decoded weight tensor is expected to be ordered such that the first dimension always corresponds to the number of output channels or filters.

[0226] 3. Invention Aspects of the invention and embodiments according to the invention are described below.

[0227] It should be noted that any of the features, functions and details described in this section can be used individually and in combination. Furthermore, it should be noted that, optionally, any of the features, functions and details described in this section can be optionally introduced into any of the conventional encoding and decoding concepts.

[0228] According to one aspect, the present invention describes a tensor dimension reordering scheme to enable efficient processing of tensors in neural network compression frameworks, such as the MPEG-7 Part 17 standard for the compression of neural networks for multimedia content description and analysis [2]. For example, tensor dimensions are reordered at the encoder side before encoding of the tensor. For example, at the decoder side, this reordering is then applied in reverse order to restore the original dimensions. For example, information on how to derive the original dimensions is signaled in the bitstream.

[0229] For example, the number of permutations is the tensor dimension Since the number of tensors can increase significantly with the number of TIFF2025513886000160.tif43, allowing arbitrary ordering of dimensions would induce significant signaling overhead, especially for high-dimensional tensors. Instead, according to one aspect of the invention, the method described herein performs only a shift of a single dimension, e.g., to another position.

[0230] In the next section, the need for tensor dimension reordering is quickly motivated. Then, in Section 3.2, the method is illustrated with an example. After that, in Sections 3.3 and 3.4, the method and associated syntax are explained by way of examples.

[0231] 3.1 Motivation As already mentioned in section 2.1, tensors are usually (though not always) represented or interpreted (e.g. in MPEG-7 part 17) as 1D vectors or 2D matrices, respectively, such that, for example, the length of the first dimension is equal to the first dimension of the tensor, and the length of the second dimension is equal to the product of all the other dimensions.

[0232] As a result, the shape of the 2D matrix may, for example, essentially depend on the first dimension of the tensor. For example, if the tensor dimensions are ordered such that the length of the first dimension is small (e.g., the width or height of a 3x3 filter kernel), the output is, for example, a fairly "slim" matrix. This may result in inefficient processing of the 2D matrix (interpretation) to be encoded, for example, especially in the case of a block scan order such as that described in section 2.2.4.1, where the matrix is ​​decomposed into blocks. For example, the MPEG-7 Part 17 standard specifies four block scan orders with block sizes of 4x4, 8x8, 16x16, and 32x32. For example, assuming the first dimension of the matrix is ​​smaller than 4 (or 8, 16, 32), this will generate a single block row of truncated blocks, which is not optimal for further processing.

[0233] According to one aspect, the invention is further motivated by the method described in section 2.4, where, for example, multiple related tensors are compressed in a block, for example, as used in MPEG-7 part 17. Here, for example, the dimension of the corresponding tensor is derived from the dimension of the weight tensor, or, more specifically, from the first dimension of the weight tensor. In some cases, this requires, for example, that the tensors be ordered such that the first dimension is related to, for example, the number of output channels (fully connected layers) or filters (convolutional layers), otherwise the derivation process may fail in some cases.

[0234] The above aspects illustrate, for example, the advantages of efficient tensor dimension reordering methods described in the following sections.

[0235] 3.2 Diagram of tensor dimension reordering scheme (example, details optional) As mentioned above, according to one aspect of the present invention, the method reorders tensors, for example, by shifting a single dimension to another position. First, the concept is illustrated using a 4-dimensional example, i.e., as shown in FIG. 6. As discussed above, FIG. 6 illustrates a schematic example of the concept of tensor dimension reordering, for example, for an encoding-decoding pipeline. For example, each box corresponds to a representation / interpretation of a tensor / matrix with dimensions given in square brackets. For example, the solid black arrows / lines indicate the general processing flow, and the dashed lines / arrows indicate the processing flow of the syntax element first_tensor_dimension_shift. (Example; encoding and decoding can be used separately; interpretation of 2D matrices is optional).

[0236] For example, from the encoder's point of view, the value first_tensor_dimension_shift (here optionally set to 2) indicates the tensor dimension that is shifted to the first position of the tensor dimension in the reordering process (see D2 in the example). Thus, for example, in a first step, a tensor is reordered, and in a second step, this tensor is interpreted, for example, as a (optional) 2D matrix and then encoded. For example, on the decoder side, the (optional) 2D matrix (interpretation) is decoded, for example, and interpreted, for example, as a tensor. Then, for example, the reordering is done in the reverse order, i.e., for example, shifting the dimension D2 to position 2 again. Now, for example, the value first_tensor_dimension_shift specifies the new position of the first dimension of the decoded and reconstructed tensor after reordering.

[0237] In the given example, the reordering process in the encoder and decoder is, for example, as follows:

[0238] Encoder (example; details optional): For example, tensor dimension 4D tensor with tif2025513886000161.tif529 and first_tensor_dimension_shift equal to 2 Given TIFF2025513886000162.tif49, TIFF2025513886000163.tif49 element TIFF2025513886000164.tif532( TIFF2025513886000165.tif530, TIFF2025513886000166.tif529, TIFF2025513886000167.tif529, and TIFF2025513886000168.tif529), For example, (optionally) the dimension Sorted tensor with TIFF2025513886000169.tif531 Initialize TIFF2025513886000170.tif59.

[0239] For example, according to Set the value of TIFF2025513886000171.tif59. for( TIFF2025513886000172.tif412; TIFF2025513886000173.tif415; TIFF2025513886000174.tif34++ ) { for( TIFF2025513886000175.tif411; TIFF2025513886000176.tif414; TIFF2025513886000177.tif33++ ) { for( TIFF2025513886000178.tif411; TIFF2025513886000179.tif414; TIFF2025513886000180.tif33++ ) { for( TIFF2025513886000181.tif411; TIFF2025513886000182.tif414; TIFF2025513886000183.tif43++ ) { TIFF2025513886000184.tif533= TIFF2025513886000185.tif533} } } }

[0240] Decoders (examples; details optional): For example, tensor dimension TIFF2025513886000186.tif531 is a 4D tensor with first_tensor_dimension_shift equal to 2 TIFF2025513886000187.tif59 is decoded from the bitstream, The elements of TIFF2025513886000188.tif59 are TIFF2025513886000189.tif534( TIFF2025513886000190.tif532, TIFF2025513886000191.tif530, TIFF2025513886000192.tif530, and TIFF2025513886000193.tif530), For example, the dimension Sorted tensor with TIFF2025513886000194.tif531 Initialize TIFF2025513886000195.tif510. For example, according to Set the value of TIFF2025513886000196.tif510. for( TIFF2025513886000197.tif412; TIFF2025513886000198.tif415; TIFF2025513886000199.tif34++ ) { for( TIFF2025513886000200.tif411; TIFF2025513886000201.tif414; TIFF2025513886000202.tif33++ ) { for( TIFF2025513886000203.tif411; TIFF2025513886000204.tif414; TIFF2025513886000205.tif33++ ) { for( TIFF2025513886000206.tif411; TIFF2025513886000207.tif414; TIFF2025513886000208.tif43++ ) { TIFF2025513886000209.tif533= TIFF2025513886000210.tif533} } } }

[0241] Thus, in general, a decoder, e.g. 100 as shown in FIG. 1 , e.g. 200 as shown in FIG. 2 , e.g. its respective decoding unit 110, 210, may be configured to obtain, e.g. a two-dimensional matrix 660, whose first dimension is determined by a first dimension value, e.g. D2 as shown in FIG. 6 , and whose second dimension is determined by a product of multiple further dimension values, e.g. D0*D1*D3 as shown in FIG. 6 , e.g. based on the encoded representation in the bitstream 650. Further, such a decoder may be configured to obtain a first multidimensional array, e.g. 111, e.g. 211, e.g. 670 as shown in FIG. 6, where the dimension of the first multidimensional array is determined by the respective dimension values ​​of the first dimension value and the further dimension value based on the two-dimensional matrix 660, and such a decoder may be configured to obtain a reordered multidimensional array, e.g. 111, e.g. 221, e.g. 680 as shown in FIG. 6, based on the first multidimensional array, where the first dimension of the reordered multidimensional array is defined by one of the further dimension values.

[0242] 3.3 Methods for tensor dimension reordering (e.g., details optional) As an example, this section describes the invention for any tensor dimension, working similarly to the example given in section 3.2. For example, in the encoder, the tensors are reordered such that a particular tensor dimension, e.g., specified by the value first_tensor_dimension_shift, is shifted to the first position. For example, in the decoder, this shifting is done in the reverse order (e.g., shifting the first dimension to the position indicated by first_tensor_dimension_shift). This method is described below as an example.

[0243] For example, an array TIFF2025513886000211.tif552 has dimensions written as TIFF2025513886000212.tif4 3-dimensional tensor TIFF2025513886000213.tif43 (where Given TIFF2025513886000214.tif515, where, for example, TIFF2025513886000215.tif55 is associated with the first dimension of the tensor, TIFF2025513886000216.tif55 is associated with the second dimension of the tensor, etc. TIFF2025513886000217.tif56( The value of TIFF2025513886000218.tif539 indicates, for example, the length of the associated dimension (for example, TIFF2025513886000219.tif55 is the length of the first dimension). For example, the tensor The position of the weights in TIFF2025513886000220.tif43 is, for example, Another array of length equal to the length of TIFF2025513886000221.tif512 Using TIFF2025513886000222.tif647, TIFF2025513886000223.tif612, where TIFF2025513886000224.tif542, for example, has dimensions It indicates the position in TIFF2025513886000225.tif56 and is a scalar. TIFF2025513886000226.tif562, for example, specifies a position within a tensor. From TIFF2025513886000227.tif42 The mapping of TIFF2025513886000228.tif69 is determined, for example, by scanning, for example row-major scanning (details of which can be found below). TIFF2025513886000229.tif528, value TIFF2025513886000230.tif418 and boolean value Take TIFF2025513886000231.tif429 as input, and convert it to an array, for example: A function that outputs TIFF2025513886000232.tif530 TIFF2025513886000233.tif5119 exists.

[0244] ·for example, Copy of TIFF2025513886000234.tif526 Initialize TIFF2025513886000235.tif529.

[0245] ·for example, For TIFF2025513886000236.tif449: ○Position The elements of TIFF2025513886000237.tif418 are, for example: It will be erased from TIFF2025513886000238.tif530.

[0246] ○ Location of TIFF2025513886000239.tif528 For example, the elements in TIFF2025513886000240.tif418 are: It is inserted at the first position of TIFF2025513886000241.tif530.

[0247] For example, if not: ○ For TIFF2025513886000242.tif425: ■ The element in the first position is, for example, It will be erased from TIFF2025513886000243.tif530.

[0248] ■ The first element of TIFF2025513886000244.tif528 is, for example, an element with position Element with front and position TIFF2025513886000245.tif418 After TIFF2025513886000246.tif426, for example Inserted into TIFF2025513886000247.tif530.

[0249] Then, for example, in the encoder (tensor Sorted tensor (from TIFF2025513886000248.tif43) TIFF2025513886000249.tif43 can be obtained as follows.

[0250] For example, the dimension Tensor with TIFF2025513886000250.tif5153 Initialize TIFF2025513886000251.tif43.

[0251] ·for example, Regarding TIFF2025513886000252.tif574, 〇 TIFF2025513886000253.tif59= TIFF2025513886000254.tif6133〇 TIFF2025513886000255.tif638 For example, in the decoder (tensor Sorted tensor (from TIFF2025513886000256.tif43) TIFF2025513886000257.tif43 can be obtained as follows.

[0252] For example, the dimension Tensor with TIFF2025513886000258.tif5150 Initialize TIFF2025513886000259.tif43.

[0253] ·for example, Regarding TIFF2025513886000260.tif574, 〇 TIFF2025513886000261.tif59= TIFF2025513886000262.tif6130〇 In a preferred embodiment, the process in the decoder for the TIFF2025513886000263.tif638 method is implemented, for example, as follows:

[0254] The inputs to this process are, for example:

[0255] A variable, for example inputTensor, representing the tensor whose dimensions are to be permuted. For example, the variable inputTensorDims specifies the dimensions of the inputTensor. For example, a variable called firstTensorDimShift that indicates the shift of the first dimension of inputTensor. For example, the output of this process is a variable reorderedTensor with dimensions equal to, for example, ShiftArrayIndex(inputTensorDims, firstTensorDimShift). The elements of the variable reorderedTensor are set, for example, as follows: for( i = 0; i < Prod( inputTensorDims ); i++ ){ idxA = TensorIndex( inputTensorDims, i, 0 ) idxB = ShiftArrayIndex( idxA, firstTensorDimShift ) reorderedTensor[idxB] = inputTensor[idxA] } For example, the functions Size, Prod, TensorIndex, and ShiftArrayIndex are defined as follows:

[0256] Prod(arrayName[]), for example, returns the product of all elements of the array arrayName[].

[0257] Size(arrayName[]) returns, for example, the number of elements contained in the array or tensor named arrayName. If arrayName[] is a tensor, this corresponds, for example, to the product of all the dimensions of the tensor.

[0258] TensorIndex(tensorDimensions[],i,scan) returns an array with the same number of dimensions as tensorDimensions[], where, for example, the elements of the array are set to integer values, so that the array can be used as an index to specify elements of a tensor with dimensions tensorDimensions[], e.g., as follows:

[0259] For example, if variable scan is equal to 0: The returned array might look something like this: It points to the i-th element in row-major traversal order of the tensor with dimensions tensorDimensions[] and is derived, for example, as follows:

[0260] For example, the variable outputArray[] is initialized with its size set to Size(tensorDimensions[]). For example, set the variable idx to i.

[0261] For example, the elements of outputArray[] are set as follows: for( a = Size(tensorDimensions)-1; a >= 0; a-- ){ outputArray[a] = idx % tensorDimensions[a] idx = idx / tensorDimensions[a] For example, the returned array is outputArray[].

[0262] For example, when the variable scan is greater than 0: The variable bs is set to, for example, 4 << scan_order, and the variable h is set to, for example, tensorDimensions[0].

[0263] The variable w is set to, for example, Prod(tensorDimensions) / h.

[0264] Two variables x and y are set to the first and second elements of an array returned, for example, by calling IndexToXY(w, h, i, bs) respectively.

[0265] For example, the returned array is TensorIndex(tensorDimensions, y * w + x, 0).

[0266] Note: The operator " / " is defined as integer division that truncates the result to 0. For example, 7 / 4 and 7 / -4 are truncated to 1, and -7 / 4 and 7 / -4 are truncated to -1.

[0267] Note: The operator "++" is defined as an increment, i.e., x++ is equivalent to x = x + 1. When used in an array index, the value of the variable is evaluated before the increment operation.

[0268] Note: The operator "--" is defined as a decrement, i.e., x-- is equivalent to x = x - 1. When used in an array index, the value of the variable is evaluated before the decrement operation.

[0269] Note: The operator "x % y" is defined as a modulus. It is defined only for integers x and y where x >= 0 and y > 0, and is the remainder when x is divided by y.

[0270] Note: The function TensorIndex provides a mapping from 'i' to a position in a tensor (see 2.2.4.1) that is equivalent to scanning a 2D matrix with dimensions equal to [tensorDimensions[0],Prod(tensorDimensions) / tensorDimensions[0]] with 'scan'.

[0271] ShiftArrayIndex(inputArray[],TensoshiftIndexPosition) returns an array outputArray[] that is a copy of inputArray[] but with the element at position 0 of inputArray shifted to shiftIndexPosition, e.g.:

[0272] For example, the variable outputArray[] is initialized with a copy of inputArray[].

[0273] For example, if shiftIndexPosition is greater than 0: For example, the first element of outputArray[] is removed from outputArray[].

[0274] For example, the first element of inputArray[] is inserted into outputArray[] before the element with position shiftIndexPosition and after the element with position shiftIndexPosition-1.

[0275] In general, a decoder according to an embodiment, e.g., decoder 100 as shown in Fig. 1, e.g., decoder 200 as shown in Fig. 2, e.g., its respective decoding unit 110, 210, may be configured to determine the position of the decoded neural network parameters in the first multidimensional array, e.g., 111, e.g., 211, using, e.g., a position mapping function that maps a scalar neural network parameter index, e.g., i, to a set of dimension indices, e.g., TensorIndex(tensorDimensions[],i,scan). Such a decoder may in particular be configured to decode the encoded neural network parameters using context-based entropy decoding, e.g., as shown in Fig. 2.

[0276] Further, in general, a decoder according to an embodiment, e.g., decoder 100 as shown in FIG. 1 , e.g., decoder 200 as shown in FIG. 2 , e.g. its respective reordering unit 120, 220, may be configured to obtain a reordered multidimensional array, e.g., 121, 221, using a function (e.g., TensorIndex(tensorDimensions[], i, scan)) that maps an integer element index i specifying an element of a first multidimensional array, e.g., 111, 211, to a set of array indexes, where the returned set of array indexes specifies the i-th element in a row-major scan order of the first multidimensional array, or where the returned set of array indexes specifies the i-th element of a block-wise scan of the first multidimensional array, where the first multidimensional array is regarded as a two-dimensional array.

[0277] Further, in general, a decoder according to an embodiment, e.g., decoder 100 as shown in Fig. 1, e.g., decoder 200 as shown in Fig. 2, e.g., its respective reordering unit 120, 220, may be configured to, e.g., decode a set of array dimensions, e.g., tensor_dimensions[], and to obtain a set of reordered array dimensions. As an example, referring to Fig. 6, the set of reordered array dimensions may be, e.g., a vector [D0, D1, D2, D3] characterizing a reordered tensor 680, obtained based on the set of decoded array dimensions, e.g., of tensor 670 or, optionally, of tensor 660.

[0278] Further, a decoder according to an embodiment, e.g. decoder 100 as shown in FIG. 1, e.g. decoder 200 as shown in FIG. 2, e.g. its respective decoding units 110, 210, may be configured to, e.g. input the decoded neural network parameters into a first multidimensional array, e.g. 111, 211, 670, at respective positions described by respective sets of array indices, the decoder being configured to obtain the respective sets of array indices using a mapping function that maps scalar integer parameter indices to the respective sets of array indices and defines a block-wise scan.

[0279] Additionally, in general, the mapping function may optionally include a mapping of a scalar integer parameter index to two coordinates that indicate a position corresponding to a scan index defined by the scalar integer parameter index as the block is scanned within the block, and the mapping function may include mapping the two coordinates to a respective set of array indexes.

[0280] 3.4 Syntax for tensor dimension reordering (example, details optional) For example, the syntax element first_tensor_dimension_shift is written into the bitstream. From the decoder's point of view, it can specify the new position of the first tensor dimension decoded from the bitstream after reordering. Thus, the value of first_tensor_dimension_shift is, for example, non-negative and less than the number of tensor dimensions. For example, a value equal to 0 means that there is no reordering of the tensor (which is, for example, the same as shifting the first dimension to the first position).

[0281] In a preferred embodiment, the number of tensor dimensions is represented, for example, by count_tensor_dimensions, general_profile_idc is, for example, equal to 1, then first_tensor_dimension is, for example, coded as follows:

[0282] [Table 2] For example, if irst_tensor_dimension_shift is not present, it is inferred to be 0.

[0283] In a further preferred embodiment, first_tensor_dimension_shift is coded, for example using an exponential-Golomb code, where the number of bits required to code a value is typically less for smaller values. Thus, for example, a value equal to 1 requires no more bits than a value of 2. For example, a value equal to 2 requires no more bits than a value of 3, etc.

[0284] See the chapter "Methods for Tensor Dimension Reordering for Neural Network Coding". [1] S. Chetlur et al.,"cuDNN:Efficient Primitives for Deep Learning," arXiv:1410.0759,2014 [2] MPEG,"Text of ISO / IEC DIS 15938-17 Compression of Neural Networks for Multimedia Content Description and Analysis",Document of ISO / IEC JTC1 / SC29 / WG11,w19764,OnLine,Oct.2020 [3] D.Marpe,H.Schwarz und T.Wiegand,"Context-Based Adaptive Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard," IEEE transactions on circuits and systems for video technology,Vol.13,No.7,pp.620-636,July 2003. [4] H.Kirchhoffer,J.Stegemann,D.Marpe,H.Schwarz und T.Wiegand, "JVET-K0430-v3-CE5-related:State-based probalility estimator," in JVET,Ljubljana,2018. [5] ITU-International Telecommunication Union,"ITU-T H.265 High efficiency video coding," Series H:Audiovisual and multimedia systems-Infrastructure of audiovisual services-Coding of moving video,April 2015. [6] B.Bross,J.Chen und S.Liu,"JVET-M1001-v6-Versatile Video Coding(Draft 4)," in JVET,Marrakech,2019. [7] S. Wiedemann et al., "DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 4, pp. 700-714, May 2020, doi:10.1109 / JSTSP.2020.2969554.

[0285] Updated drafts of proposed and planned International Standard ISO / IEC 15938-17 (e.g. ISO / IEC DIS 15938-17:xxxx(E)) Information technology-Multimedia content description interface-Part 17:Compression of neural networks for multimedia content description and analysis(2 nd edition)

[0286] General Note: All details are optional. The following aspects according to the invention are described in the following sections: Preamble, Introduction, 1 Scope, 2 Normative References, 3 Terms and Definitions, 4 Abbreviations, Conventions, and Symbols, 4.1 General, 4.2 Abbreviations, 4.3 List of Symbols, 4.4 Numeric Format and Computation Rules, 4.5 Arithmetic Operators, 4.6 Logical Operators, 4.7 Relational Operators, 4.8 Bitwise Operators, 4.9 Assignment Operators, 4.10 Range Notation, 4.11 Mathematical Functions, 4.12 Array Functions, 4.13 Operation Precedence, 4.14 Variables, Syntax Elements, and Tables, 5 Overview, 5.1 General, 5.2 Compression Tools, 5.3 Creating an Encoding Pipeline, 6 Syntax and Semantics, 6.1 Specification of Syntax and Semantics, 6.1.1 How to Specify Tabular Syntax, 6.1.2 Bit Ordering, 6.1.3 Specification of Syntax Functions and Data Types, 6.1.4 6.2 Semantics, 6.2 General bitstream syntax elements, 6.2.1 NNR unit, 6.2.2 Aggregated NNR unit, 6.2.3 Composition of an NNR bitstream, 6.3 NNR bitstream syntax, 6.3.1 NNR unit syntax, 6.3.2 NNR unit size syntax, 6.3.3 NNR unit header syntax, 6.3.4 NNR unit payload syntax, 6.3.5 Byte alignment syntax, 6.4 Semantics, 6.4.1 General, 6.4.2 NNR unit size semantics, 6.4.3 NNR unit header semantics, 6.4.4 NNR unit payload semantics, 7 Decoding process, 7.1 General, 7.2 NNR decompressed data format, 7.3 Decoding method, 7.3.1 General, 7.3.2 Decoding method for NNR compressed payload of type NNR_PT_INT, 7.3.3 How to decode an NNR compressed payload of type NNR_PT_FLOAT, 7.3.4 How to decode an NNR compressed payload of type NNR_PT_RAW_FLOAT, 7.3.5 How to decode an NNR compressed payload of type NNR_PT_BLOCK, 7.3.6 Decoding process for integer weight tensors, 8 Parameter reduction, 8.1 General (Informative), 8.2 Methods (Informative), 8.2.1 Sparsifying using compression loss, 8.2.2 Sparsifying using microstructure pruning, 8.2.3 Decoded pruning and sparsifying, 8.2.4 Structured sparsifying, 8.2.5 Parameter integration, 8.2.6 Low rank / low displaced rank for convolutional and fully connected layers, 8.2.7 Batch norm folding, 8.2.8 Local scaling adaptation, 8.3 Syntax and semantics, 8.3.1 Sparsifying using compression loss, 8.3.2 Sparsifying using micro-structure pruning, 8.3.3 Decoded pruning and sparsifying, 8.3.4 Structured sparsifying, 8.3.5 Weight integration, 8.3.6 Low rank / low displaced rank for convolutional and fully connected layers, 8.3.7 Batch norm folding, 8.3.8 Local scaling, 9 Parameter quantization, 9.1 Methods, 9.1.1 Uniform quantization methods, 9.1.2 Codebook-based methods, 9.1.3 9.1.4 Dependent Scalar Quantization Methods, 9.2 Syntax and Semantics, 9.2.1 Uniform Quantization Methods, 9.2.2 Codebook-Based Methods, 9.2.3 Dependent Scalar Quantization Methods, 10 Entropy Coding 10.1 Methods, 10.1.1 DeepCABAC, 10.2 Syntax and Semantics, 10.2.1 DeepCABAC Syntax, 10.3 Entropy Decoding Process, 10.3.1 General, 10.3.2 Initialization Process, 10.3.3 Binarization Process, 10.3.4 Decoding Process Flow, Appendix A (Normative) Implementation for NNEF, A.1 General, A.2 Identifiers, A.3 Definitions for Use in NNR Bitstreams, A.4 Carriage of NNEF Data in NNR Bitstreams, Appendix B (Informative) Implementation for ONNX, B.1 B.1 General, B.2 Identifiers, B.3 Definitions for use with NNR bitstreams, B.4 Carrying ONNX data in NNR bitstreams, Appendix C (Informative) Implementations for PyTorch®, C.1 General, C.2 Identifiers, C.3 Definitions for use with NNR bitstreams, C.4 Carrying PyTorch data in NNR bitstreams, Appendix D (Informative) Implementations for TensorFlow®, D.1 General, D.2 Identifiers, D.3 Definition for use in NNR bitstreams, D.4 Carriage of TensorFlow data in NNR bitstreams, Appendix E (Informative) Recommendations for the carriage of NNR bitstreams in other containers, E.1 Recommendations for the carriage of NNR bitstreams in NNEF container organizations (specified in the NNEF format specification), E.2 Recommendations for the carriage of NNR coded bitstreams inside ONNX (specified in the ONNX format specification) are discussed in the bibliography "Information technology - Multimedia content description interface".

[0287] Further, it should be noted that some of the above sections (e.g., Appendices A-E) may not be included in their entirety herein, e.g., for the sake of brevity, because features as disclosed in those sections are considered optional for embodiments of the present invention. However, it should be noted that embodiments in accordance with the present invention may include any or all of the features disclosed in the respective sections, e.g., as disclosed in a draft of (e.g., proposed) International Standard ISO / IEC 15938-17 (e.g., ISO / IEC DIS 15938-17:xxxx(E)), e.g., as disclosed in the complete version of the respective sections.

[0288] Introduction This proposed second edition cancels and replaces the technically revised first edition (ISO / IEC 15938-17).

[0289] The main changes compared to the previous version are: -xxx xxxxxxx xxx xxxx Introduction (details optional) Artificial neural networks have been employed for a wide range of tasks in multimedia analysis and processing, media coding, data analysis, and many other fields. These recent successes are based on the feasibility of processing much larger and more complex neural networks (deep neural networks, DNNs) than in the past, and on the availability of large training datasets. As a result, trained neural networks contain a large number of parameters and weights, resulting in very large sizes (e.g., hundreds of MB). Many applications require that a particular trained network instance be deployed to potentially many devices, which may have limitations in terms of processing power and memory (e.g., mobile devices or smart cameras), as well as in terms of communication bandwidth. Any use case that requires the deployment of a trained neural network (or its updates) to several devices would therefore benefit from a standard for a compressed representation of neural networks.

[0290] Considering the fact that compression of neural networks is likely to have hardware-dependent and hardware-independent components, this document is designed as a toolbox of compression techniques. Some of these techniques require specific representations in the exchange format (i.e., sparse representations, adaptive quantization), and therefore standard specifications for representing the output of these techniques are defined. Others are not realized at all in the serialized representation (e.g., pruning), but for the latter the necessary metadata is specified. This document is independent of any specific neural network exchange format, and interoperability with common formats is documented in the appendix.

[0291] This document therefore defines a high-level syntax that specifies the required metadata elements and associated semantics. Where the structure of the binary data is specified (e.g., a decomposed matrix), this document also specifies the actual bitstream syntax of each block. The appendix to the document specifies the requirements and constraints for the compressed neural network representation, as defined in this document, and how it is applied.

[0292] - Appendix A specifies, as an example, an implementation of this document using the Neural Network Interchange Format (NNEF), which defines the use of NNEF to represent network topology in compressed neural network bitstreams.

[0293] - Appendix B defines, as an example, the Open Neural Network Interchange Format (ONNX) (registered trademark), which defines the use of ONNX for representing network topology in compressed neural network bitstreams. 1 This document provides recommendations for the implementation of this document using

[0294] Appendix C defines, as an example, references to PyTorch elements in the network topology description of a compressed neural network bitstream, PyTorch® 2 This document provides recommendations on the implementation of this document using formats.

[0295] - Appendix D provides an example of a compressed neural network bitstream Defines references to Tensorflow elements in network topology descriptions, and 3 This document provides recommendations on the implementation of this document using formats.

[0296] - Appendix E provides, as an example, recommendations for transporting tensors compressed according to this document in third-party container formats.

[0297] The compression tools described in this document have been selected and evaluated for neural networks used in applications for multimedia description, analysis, and processing, but they may be useful for compressing neural networks used in other applications and applied to other types of data.

[0298] Information technology-Multimedia content description interface-Part 17:Compression of neural networks for multimedia content description and analysis(2nd ed)

[0299] 1 ONNX is a trademark of a product owned by LF PROJECTS, LLC. This information is provided as a convenience to users of this document and does not constitute an endorsement by ISO / IEC of the products named. 2 PyTorch is a trademark of a product provided by Facebook, Inc. This information is provided for the convenience of users of this document and does not constitute an endorsement by ISO / IEC of the product named. 3 TensorFlow is a trademark of a product provided by Google LLC. This information is provided as a convenience to users of this document and does not constitute an endorsement by ISO / IEC of the product named.

[0300] 1 Range This document specifies, as an example, the compressed representation of the parameters / weights of a trained neural network and the decoding process of the compressed representation, complementing the description of the network topology in existing (exchange) formats for neural networks. It establishes a toolbox of compression methods that specify (where applicable) the resulting elements of the compressed bitstream. All of these tools can be applied, for example, to compress an entire neural network, and some of them can also be applied, for example, to compress differential updates of a neural network with respect to a base network. Such differential updates are useful, for example, when a model is redeployed after fine-tuning or transfer learning, or when providing versions of a neural network with different compression ratios.

[0301] This document specifies a compressed representation of neural networks that can be further incorporated into other standards, for example. Only the syntax format, semantics, associated decoding process requirements, parameter sparsification, parameter transformation methods, parameter quantization, entropy coding methods, and integration / signaling within existing exchange formats are specified, while other matters such as pre-processing, system signaling and multiplexing, data loss recovery, and post-processing are considered to be outside the scope of this document. Furthermore, internal processing steps performed within the decoder are also considered to be outside the scope of this document. Only the externally observable output behavior needs to conform to the specifications of this document.

[0302] 2 Normative References The following documents are referred to in the text, the contents of which, in part or in whole, constitute requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO / IEC 10646, Information technology - Universal coded character set (UCS) ISO / IEC 60559, Information technology - Microprocessor Systems - Floating-Point arithmetic IETF RFC 1950, ZLIB Compressed Data Format Specification version 3.3, 1996 NNEF-v1.0.3, Neural Network Exchange Format, The Khronos NNEF Working Group, Version 1.0.3, 2020-06-12 (https: / / www.khronos.org / registry / NNEF / specs / 1.0 / nnef-1.0.3.pdf)

[0303] 3. Terms and definitions For the purposes of this document, the following terms and definitions apply:

[0304] ISO and IEC maintain a database of terms for use in standardization at the following address:

[0305] - ISO Online Browsing Platform: Available at https: / / www.iso.org / obp -IEC Electropedia: Available at http: / / www.electropedia.org /

[0306] 3.1 Aggregated NNR Unit An NNR unit carrying multiple NNR units in its payload 3.2 Base Neural Network Neural network as basis for differential updates 3.3 Compressed Neural Network Representation Representation of a neural network with model parameters encoded using a compression tool 3.4 Decomposition A transformation that represents a tensor as the product of two tensors

[0307] 3.5 Hyperparameters Parameters whose values ​​are used to control the learning process 3.6 layer A set of nodes working together at a particular depth in a neural network 3.7 Model parameters Coefficients of the neural network model, such as weights and biases 3.8 NNR Unit Data structures for carrying neural network data (compressed or uncompressed) and associated metadata

[0308] 3.9 Pruning Parameter reduction in (part of) neural networks 3.10 Sparsification Increasing the number of zero-valued entries in a tensor 3.11 tensor Multidimensional structural grouping related model parameters 3.12 Updated Neural Network The resulting neural network after transforming the base neural network Note: The updated neural network is reconstructed by applying the differential updates to the base neural network.

[0309] 4 Abbreviations, Conventions and Symbols 4.1 General This subclause contains definitions of the operators, notations, functions, textual conventions, and processes used throughout this document.

[0310] The mathematical operators used in this document are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are more precisely specified, and additional operations such as exponentiation and real division are specified. The numbering and counting conventions generally start from zero, e.g., "first" corresponds to the 0th, "second" corresponds to the 1st, etc.

[0311] 4.2 Abbreviations (Table 3) DeepCABAC: Context-Adaptive Binary Arithmetic Coding for Deep Neural Networks LR Low Rank LDR Low Displacement Rank LPS Layer Parameter Set LSB Least Significant Bit MSB Most significant bit MPS model parameter set NN Neural Network NNEF Neural Network Exchange Format NNR Compressed Neural Network Representation SVD Singular Value Decomposition

[0312] 4.3 List of symbols This document defines the following symbols: TIFF2025513886000265.tif43 Input tensor TIFF2025513886000266.tif43 Output tensor TIFF2025513886000267.tif76 Blocks of superblock j in layer k. TIFF2025513886000268.tif43 Bias parameters TIFF2025513886000269.tif56 Number of input channels of convolutional layer TIFF2025513886000270.tif59 Number of output channels of convolutional layer TIFF2025513886000271.tif75 The number of channels of the tensor at layer k. TIFF2025513886000272.tif95 The derived number of channels of the tensor at layer k. TIFF2025513886000273.tif75Depth dimension of the tensor at layer k. TIFF2025513886000274.tif33f - Circular matrix Z e Parameters. TIFF2025513886000275.tif43 Parameter tensor of convolutional layer TIFF2025513886000276.tif42f Parameters of the circulant matrix Zf. TIFF2025513886000277.tif55 matrix Left matrix of low-rank decomposition of TIFF2025513886000278.tif57 TIFF2025513886000279.tif56 matrix Right-hand matrix of low-rank decomposition of TIFF2025513886000280.tif57 TIFF2025513886000281.tif75 Height dimension of tensor at layer k. TIFF2025513886000282.tif43 Convolution kernel dimensions TIFF2025513886000283.tif43 Network TIFF2025513886000284.tif527 lossy compression TIFF2025513886000285.tif517 Diversity Loss TIFF2025513886000286.tif49 Task Loss TIFF2025513886000287.tif511 Training loss TIFF2025513886000288.tif44 Feature matrix TIFF2025513886000289.tif56 Pruning Mask TIFF2025513886000290.tif34 Sparsification hyperparameters TIFF2025513886000291.tif45Feature matrix i-th row of TIFF2025513886000292.tif44 TIFF2025513886000293.tif75 The kernel size of the tensor at layer k. Dimensions that result in the product of TIFF2025513886000294.tif614 TIFF2025513886000295.tif43 Probability transition matrix TIFF2025513886000296.tif43 Pruning ratio hyperparameter TIFF2025513886000297.tif46 transition matrix Elements of TIFF2025513886000298.tif43 TIFF2025513886000299.tif43 Sparsification rate hyperparameter TIFF2025513886000300.tif43 Importance of pruning parameters TIFF2025513886000301.tif75 Superblock TIFF2025513886000302.tif32 Local Scaling Factor TIFF2025513886000303.tif75 Superblock size TIFF2025513886000304.tif33 Integration rate hyperparameter TIFF2025513886000305.tif45 Parameter Tensor TIFF2025513886000306.tif55Weight tensor of the l-th layer TIFF2025513886000307.tif57 layer Parameter tensor for TIFF2025513886000308.tif43 Low-rank approximation of TIFF2025513886000309.tif517 TIFF2025513886000310.tif34 Parameter Vector TIFF2025513886000311.tif75The width dimension of the tensor at layer k. TIFF2025513886000312.tif47Weight vector of the ith filter in the lth layer TIFF2025513886000313.tif712 Vector of normalized weights for the ith filter in the lth layer TIFF2025513886000314.tif43 Input to batch normalization layer TIFF2025513886000315.tif55f circular matrix TIFF2025513886000316.tif54f circular matrix TIFF2025513886000317.tif33 Folded batch normalization parameters TIFF2025513886000318.tif56 Folded batch normalization parameters and local scaling coefficients TIFF2025513886000319.tif43 Batch normalization parameters TIFF2025513886000320.tif56 compression lossy multiplier TIFF2025513886000321.tif33 Batch normalization parameters TIFF2025513886000322.tif42 Folded Batch Normalization parameters TIFF2025513886000323.tif32 Scalar close to zero to avoid division by zero in batch normalization TIFF2025513886000324.tif43 Eigenvector TIFF2025513886000325.tif66 Compression Loss Weight TIFF2025513886000326.tif67 Diversity Loss Weight TIFF2025513886000327.tif33 Batch normalization parameters Equilibrium probability of TIFF2025513886000328.tif412 TIFF2025513886000329.tif33 Batch normalization parameters TIFF2025513886000330.tif32 Sparsification Pruning Threshold TIFF2025513886000331.tif43 Smoothing factor 4.4 Number Formats and Calculation Rules This document defines the following number formats:

[0313] Integer An integer that can be arbitrarily small or large. Integers are also called signed integers. unsigned integer 0 or an arbitrarily large unsigned integer. Floating point numbers according to ISO / IEC 60559.

[0314] Unless otherwise specified, the results of all operators and mathematical functions are mathematically exact. Whenever a result can be a float, it is explicitly specified.

[0315] 4.5 Arithmetic Operators The following arithmetic operators are defined: + Add - subtraction (as a two-argument operator) or negation (as a unary prefix operator) * Multiplication, including matrix multiplication TIFF2025513886000332.tif22 Element-wise multiplication of two transposed vectors or element-wise multiplication of a transposed vector and a row of a matrix or Hadamard product of two matrices with the same dimensions x y Exponentiation. Specifies x to the power of y. In other contexts, such notation is used as a superscript that is not intended to be interpreted as a power. / Integer division with result truncated to 0. For example, 7 / 4 and 7 / -4 round down to 1, and -7 / 4 and 7 / -4 round down to -1. ÷ Used to indicate division in mathematical equations where truncation or rounding is not intended. TIFF2025513886000333.tif82 Used to indicate division in mathematical equations where no truncation or rounding is intended, including element-wise division of two transposed vectors, or element-wise division of a transposed vector by a row of a matrix. The sum of TIFF2025513886000334.tif717f(i), for all integer values ​​of i from x to y inclusive. The product of TIFF2025513886000335.tif617f(i), for all integer values ​​of i from x to y inclusive. x%y ratio. The remainder when x is divided by y, defined only for integers x and y, x ≥ 0 and y > 0.

[0316] 4.6 Logical Operators The following logical operators are defined: x&&y The Boolean logic "and" of x and y x||y The Boolean "or" of x and y ! Boolean logic "not" x?y:z Evaluates to the value of y if x is TRUE or not equal to 0, otherwise it evaluates to the value of z.

[0317] 4.7 Relational Operators The following relational operators are defined as follows: > Greater than ≧ Greater than or equal to < Less than <= Equal to == Equal != Not equal When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is treated as a separate value of the syntax element or variable. The value "na" is considered not equal to any other value.

[0318] 4.8 Bitwise Operators The following bitwise operators are defined as follows: & Bitwise "and". When operating on integer arguments, it operates on the two's complement representation of the integer values. When operating on a binary argument that contains fewer bits than another argument, the shorter argument is extended by appending most significant bits equal to 0. | Bitwise "or". When operating on integer arguments, it operates on the two's complement representation of the integer values. When operating on a binary argument that contains fewer bits than the other argument, the shorter argument is extended by adding leading bits equal to 0. ^ Bitwise "exclusive or". When operating on integer arguments, it operates on the two's complement representation of the integer values. When operating on a binary argument that contains fewer bits than the other argument, the shorter argument is extended by adding leading bits equal to 0. x>>y Arithmetically right-shift the two's complement integer representation of x by the number of bits in the binary representation of y. This function is defined only for non-negative integer values of y. The bit shifted into the MSB as a result of the right shift has a value equal to the MSB of x before the shift operation. x<<y Arithmetically left-shift the two's complement integer representation of x by the number of bits in the binary representation of y. This function is defined only for non-negative integer values of y. The bit shifted into the LSB as a result of the left shift has a value equal to 0. ! Bitwise not operator, which returns 1 when applied to 0 and returns 0 when applied to 1.

[0319] 4.9 Assignment Operators Define the following arithmetic operators as follows. = Assignment operator ++ Increment, i.e., x++ is equivalent to x=x+1. When used as an array index, the value of the variable is evaluated before the increment operation. -- Decrement, i.e., x-- is equivalent to x=x-1. When used as an array index, the value of the variable is evaluated before the decrement operation. += Increment by the specified amount, i.e., x+=3 is equivalent to x=x+3 and x+=(-3) is equivalent to x=x+(-3). -= Decrement by the specified amount, i.e., x-=3 is equivalent to x=x-3 and x-=(-3) is equivalent to x=x-(-3).

[0320] 4.10 Range Notation The following notation is used to specify ranges of values: x=y..zx takes integer values ​​greater than or equal to y and less than or equal to z, where x, y, and z are integers and z is greater than y. Array[x,y] A subarray containing the elements of array contained between included positions x and y. If x is greater than y, the resulting subarray is empty.

[0321] 4.11 Mathematical Functions The following mathematical functions are defined:

[0322] Ceil(x) The smallest integer greater than or equal to x Floor(x) The largest integer less than or equal to x Log2(x) The base 2 logarithm of x TIFF2025513886000336.tif951TIFF2025513886000337.tif9524.12 Array Functions Size(arrayName[]) returns the number of elements in the array or tensor named arrayName. If arrayName[] is a tensor, this corresponds to the product of all the dimensions of the tensor. (Optional) Prod(arrayName[]) returns the product of all elements of the array arrayName[]. (Optional) TensorReshape(arrayName[],tensorDimension[]) returns the reshaped tensor array_name[] with the specified tensorDimension[] without modifying its data. (Optional) IndexToXY(w,h,i,bs) returns an array with two elements, e.g. the first element is the x coordinate and the second element is the y coordinate, pointing to a 2D array of e.g. width w and height h. For example, x and y point to the position corresponding to the scan index i when the block is scanned in blocks of e.g. size bs×bs. For example, x and y are derived as follows:

[0323] The variable fullRowOfBlocks is set to w*bs. The variable blockY is set to i / fullRowOfBlocks. The variable iOff is set to i%fullRowOfBlocks. The variable currBlockH is set to Min(bs,h-blockY*bs).

[0324] The variable fullBlocks is set to bs*currBlockH. The variable blockX is set to iOff / fullBlocks. The variable blockOff is set to iOff%fullBlocks. The variable currBlockW is set to Min(bs,w-blockX*bs).

[0325] The variable posX is set to blockOff%currBlockW. The variable posY is set to blockOff / currBlockW. The variable x is set to blockX*bs+posX. The variable y is set to blockY*bs+posY.

[0326] TensorIndex(tensorDimensions[],i,scan) returns an array with the same number of dimensions as tensorDimensions[], where, for example, the elements of the array are set to integer values, so that the array can be used as an index to specify elements of a tensor with, for example, dimensions tensorDimensions[], as follows:

[0327] If variable scan is equal to 0: The returned array might look something like this: Points to the i-th element in row-major traversal order of the tensor with dimensions tensorDimensions[].

[0328] When the variable scan is greater than 0: (Optional) (Example) The variable bs is set to, for example, 4 << scan_order.

[0329] The variable h is set to, for example, tensorDimensions[0].

[0330] The variable w is set to, for example, Prod(tensorDimensions) / h.

[0331] Two variables x and y are, for example,[[]] each set to the first element and the second element of the array returned by calling IndexToXY(w, h, i, bs) respectively.

[0332] The returned array is, for example, TensorIndex(tensorDimensions, y * w + x, 0).

[0333] Note - The variable scan usually (but not always) corresponds to the syntax element scan_order.

[0334] GetEntryPointIdx(tensorDimensions[], i, scan) (Optional) returns, for example, -1 if the index i does not point to the first position of the entry point. If the index i points to the first position of the entry point, it returns, for example, the entry point index within the tensor. To determine the position and index of the entry point, the following is applied: (Example) The variable w is set to Prod(tensorDimensions) / tensorDimensions[0].

[0335] The variable epIdx is set to i / (w * (4 << scan)) - 1.

[0336] If i>0 and i%(w*(4<<scan)) is equal to 0, index i points to the first position of the entry point, and the entry point index is equal to epIdx.

[0337] Otherwise, index i does not point to the first position of the entry point.

[0338] ShiftArrayIndex(inputArray[], shiftIndexPosition) returns, for example, an array outputArray[] that is a copy of inputArray[] but with the element at position 0 of inputArray[] shifted to shiftIndexPosition.

[0339] The variable outputArray[] is initialized as a copy of inputArray[], for example.

[0340] If shiftIndexPosition is greater than 0: The first element of outputArray[] is removed from outputArray[], for example.

[0341] The first element of inputArray[] is inserted into outputArray[] before the element with position shiftIndexPosition and after the element with position shiftIndexPosition - 1, for example.

[0342] AxisSwap(inputTensor[], tensorDimensions[], numberOfDimensions, axis0, axis1) (optional) returns a tensor derived from inputTensor (with dimensions tensorDimensions and number of dimensions numberOfDimensions) where the values of the axis indices axis0 and axis1 of inputTensor are swapped.

[0343] TensorSplit(inputTensor[],splitIndices,splitAxis) (optional) returns an array of tensors subTensors derived by splitting the tensor inputTensor into N=Size(splitIndices)+1 tensors using the provided array of indices splitIndices along the provided axis splitAxis as follows:

[0344] The array inputDims is set to the dimensions of the tensor inputTensor.

[0345] An element with value 0 is inserted into splitIndices before the first element, and an element with value inputDims[splitAxis] is inserted into splitIndices after the last element.

[0346] The tensor subTensors[X], where X is an integer between 0 and N, is derived as follows: The array subTensorDims is set to inputDims.

[0347] The element subTensorDims[splitAxis] is replaced by the value splitIndices[X+1]-splitIndices[X].

[0348] The elements of subTensors[X] are set as follows: for( i = 0; i < Prod( subTensorDims ); i++ ) { subIdx = TensorIndex( subTensorDims, i, 0 ) inputIdx = TensorIndex( inputDims, i, 0 ) inputIdx[splitAxis] += splitIndices[X] subTensors[subIdx] = inputTensor[inputIdx] 4.13 Operation Precedence When precedence in an expression is not explicitly indicated through the use of parentheses, the following rules apply:

[0349] · Operations with higher precedence are evaluated before any operations with lower precedence. Operations of equal precedence are evaluated sequentially from left to right.

[0350] Table 1 specifies the precedence of operations from highest to lowest. A higher position in the table indicates a higher precedence.

[0351] NOTE - For operators that are also used in the C programming language, the precedence used in this document is the same as that used in the C programming language.

[0352] [Table 4]

[0353] 4.14 Variables, Syntax Elements, and Tables Syntax elements in the bitstream are represented in bold. Each syntax element is described by its name (all lowercase including the underscore character) and one data type for the method of coded representation. The decoding process operates according to the value of the syntax element and the values ​​of previously decoded syntax elements. When a syntax element's value is used in a syntax table or text, it appears in normal (i.e., not bold) type.

[0354] In some cases, syntax tables may use the values ​​of other variables derived from syntax element values. Such variables are named with a mixture of lower and upper case letters and appear in the syntax table or text without underscore characters (camel case notation). Variables beginning with an upper case letter are derived for the decoding of the current syntax structure and all dependent syntax structures. Variables beginning with an upper case letter can be used in the decoding process of later syntax structures without reference to the syntax structure from which the variable originates. Variables beginning with a lower case letter are used only within the (sub)clause from which they are derived.

[0355] In some cases, "mnemonic" names for syntax element or variable values ​​are used interchangeably with their numeric values. "Mnemonic" names may be used without their associated numeric values. The association between value and name is specified textually. Names are constructed from one or more groups of characters separated by underscore characters. Each group may start with an uppercase letter and may contain more uppercase letters. NOTE - The syntax is written to closely follow the syntax structure of the C language.

[0356] Functions that specify properties of the current position in the bitstream are called syntax functions. These functions are specified in subclause 6.3 and assume the existence of a bitstream pointer with an indication of the position of the next bit to be read by the decoding process from the bitstream. Syntax functions are constructed as syntax element names, with their names terminated by left and right parentheses containing zero or more variable names (in the case of more than one variable) or values ​​(in the case of use), separated by commas.

[0357] Functions that are not syntax functions (including the mathematical functions specified in sub-clause 4.11 and the array functions specified in sub-clause 4.12) are written with their names beginning with a capital letter, containing a mixture of lower and upper case letters without any underscore characters, and ending with left and right parentheses containing zero or more variable names (in the case of a definition) or values ​​(in the case of a use), separated by commas (in the case of more than one variable).

[0358] A one-dimensional array is called a list. A two-dimensional array is called a matrix. Arrays can be either syntax elements or variables. Subscripts or square brackets are used to index into arrays. Referring to a visual depiction of a matrix, the first subscript is used as the row (vertical) index and the second subscript is used as the column (horizontal) index. The order of indexing is reversed when using square brackets rather than subscripts for indexing. Thus, an element of a matrix s at horizontal position x and vertical position y is expressed as s[x][y] or s yx A single column of a matrix is ​​called a list and may be represented by omitting the row index. Thus, the column of a matrix s at horizontal position x may be referred to as the list s[x].

[0359] A multidimensional array is a variable that has a number of dimensions. The elements of a multidimensional array are indexed by specifying all the necessary indices, e.g. variables [x][y][z], or by a single index variable that is itself a one-dimensional array that specifies the indices, e.g. variable [i], where i is a one-dimensional array with elements [x, y, z]. Multidimensional arrays are used, for example, to specify tensors.

[0360] Specifying the values ​​of entries in rows and columns of an array can be expressed as {{...}{...}}, where each inner pair of brackets specifies the value of an element in a row in ascending column order, and the rows are ordered in ascending row order. Thus, setting a matrix s equal to {{1 6}{4 9}} specifies that s[0][0] is set equal to 1, s[1][0] is set equal to 6, s[0][1] is set equal to 4, and s[1][1] is set equal to 9.

[0361] Binary representation is indicated by enclosing a string of bit values ​​in single quotation marks. For example, "01000001" represents an 8-bit string with only its second and its last bits (counting from most significant to least significant) equal to 1.

[0362] Note that, instead of binary notation, hexadecimal notation may be used by preceding the hexadecimal digit with "0x" when the number of bits is an integer multiple of 4. For example, 0x41 represents an 8-bit string with only its second and its last bit (counting from the most significant bit to the least significant bit) equal to 1.

[0363] Numbers that are not enclosed in single quotes and not preceded by "0x" are decimal values.

[0364] A value equal to 0 represents a FALSE condition in the test statement. A TRUE value is represented by any value different from 0.

[0365] 5. Overview 5.1 General This section provides an overview of the compression tools defined in this document and explains, for example, how they can be combined into an encoding.

[0366] 5.2 Compression Tools (example; all details optional) This document includes the following compression tools:

[0367] Parameter reduction methods process the model to obtain a compact representation. Examples of such methods include parameter sparsification, parameter pruning, weight merging, and decomposition methods.

[0368] Sparsification processes a parameter or group of parameters to generate a sparse representation of the model, for example, by replacing some weight values ​​with zero. Sparsification can generate additional metadata (e.g., a mask). Sparsification can be structured or unstructured. This document includes methods for unstructured sparsification with compressibility loss (e.g., as disclosed in each subclause), structured sparsification using fine-structured sparsification (e.g., as disclosed in each subclause), unstructured statistics-adaptive sparsification (e.g., as disclosed in each subclause), and structured sparsification (e.g., as disclosed in each subclause).

[0369] Fusion processes parameters to generate groups of similar parameters. Fusion reduces the entropy of model parameters by making them more similar to each other, rather than eliminating or constraining weights to be zero. This document includes methods for weight fusion (e.g., as disclosed in the respective subsections).

[0370] Pruning reduces the number of parameters by eliminating parameters or groups of parameters. This procedure results in a dense representation with fewer parameters compared to the original model, for example by removing some redundant convolution filters from a layer. This document includes methods for combined pruning and sparsification (e.g., as disclosed in the respective subsections).

[0371] The decomposition performs a matrix decomposition operation to change the structure of the model weights. This document includes methods for low rank / low variance rank for convolutional layers and fully connected layers (e.g., as disclosed in the respective subsections).

[0372] In addition to the reduction methods mentioned above, this document includes decomposition methods that are introduced and tested as part of the parameter quantization technique. Examples of such methods are batch norm folding (e.g., as disclosed in the respective subsections) and local scaling adaptation (e.g., as disclosed in the respective subsections).

[0373] Parameter reduction methods can be combined and applied to sequences to create compact models.

[0374] Parameter quantization methods reduce the precision of the representation of the parameters. If supported by the inference engine, the quantized representation can be used for more efficient inference. This document includes methods for uniform quantization (e.g., as disclosed in each subclause), codebook-based quantization (e.g., as disclosed in each subclause), dependent scalar quantization (e.g., as disclosed in each subclause), and iterative QP optimization (e.g., as disclosed in each subclause).

[0375] The entropy coding method encodes the result of the parameter quantization method. This document includes DeepCABAC (subclause 9.1.1) as an entropy coding method. Supported extensions of DeepCABAC include row skipping and temporal context modeling.

[0376] Row skipping reduces the number of decoded bins and thus the bitstream size by skipping the decoding of matrix rows that are entirely zero. To this end, row skipping signals one flag per matrix row. This method is described in subclause 9.1.1.2.

[0377] Temporal context modeling uses information from previously decoded incremental updates to improve the context modeling of DeepCABAC and therefore improve coding efficiency. This method is described in subclause 9.1.1.4.

[0378] 5.3 Creating an encoding pipeline (e.g., all details optional) The compression tools in this document can be combined to form different encoding pipelines: some tools are alternatives to deal with neural network models with different types of characteristics, while others are designed to work sequentially.

[0379] Fig. 11 shows a schematic diagram of an entire encoding pipeline according to an embodiment that can be optionally assembled using the compression tools of this document. Fig. 11 shows a schematic example of an NNR encoding pipeline according to an embodiment. From a group of parameter transformation tools, multiple tools can be applied in sequence. Parameter quantization can be applied to the source model and the output of the transformation by the parameter reduction method. Entropy coding is typically applied to the output of the quantization. Row outputs of previous steps that do not apply entropy coding can be serialized if necessary.

[0380] The following encoding pipeline is considered a typical example for using this document. 1. Dependent Scalar Quantization (e.g., as disclosed in the respective sub-clauses) - DeepCABAC (Sub-clauses 9.1.1) 2. Sparsification (e.g., as disclosed in the respective subclauses) - Dependent Scalar Quantization (e.g., as disclosed in the respective subclauses) - DeepCABAC (Subclauses 9.1.1)

[0381] 3. Low Rank Decomposition (e.g., as disclosed in the respective sub-clauses) - Dependent Scalar Quantization (e.g., as disclosed in the respective sub-clauses) - DeepCABAC (Sub-clauses 9.1.1) 4. Codebook-Based Quantization (e.g., as disclosed in the respective sub-clauses) - DeepCABAC (Sub-clauses 9.1.1)

[0382] 5. Integration (e.g., as disclosed in each sub-clause) - DeepCABAC (Sub-clause 9.1.1) This list is not exhaustive. 6 Syntax and semantics (all details optional)

[0383] [Ed. Note (HT) MPEG 135 and later: Changes in Section 8 New additions for incremental weight updates may be introduced later.] 6.1 Syntax and Semantics Specification 6.1.1 How to specify tabular syntax The syntax table specifies a superset of all permitted bitstream syntax. Additional constraints on the syntax may be specified directly or indirectly in other clauses.

[0384] Table 2 lists examples of syntax specification formats. When syntax_element appears, it specifies that the syntax element is to be parsed from the bitstream and the bitstream pointer is to be advanced past the syntax element to the next location in the bitstream parsing process.

[0385] [Table 5]

[0386] 6.1.2 Bit Order (Example) For bit-oriented distribution, the bit order of the syntax fields in the syntax table is specified starting with the MSB and proceeding to the LSB.

[0387] 6.1.3 Syntax Function and Data Type Specification (Example) The functions presented here are used in the syntax description and are represented by the value of a bitstream pointer that indicates the location of the next bit to be read by the decoding process from the bitstream.

[0388] byte_aligned() is specified as follows: If the current position in the bitstream is on a byte boundary, i.e. the next bit in the bitstream is the first bit in a byte, the return value of byte_aligned() is equal to TRUE.

[0389] Otherwise, the return value of byte_aligned() is equal to FALSE.

[0390] read_bits(n) reads the next n bits from the bitstream and advances the bitstream pointer by n bit positions. If n is equal to 0, read_bits(n) is specified to return a value equal to 0 and not advance the bitstream pointer.

[0391] get_bit_pointer() returns the position of the bitstream pointer relative to the beginning of the current NNR unit as an unsigned integer value. get_bit_pointer()>>3 points to the current byte of the bitstream pointer. get_bit_pointer()&7 points to the current bit within the current byte of the bitstream pointer, with a value of 0 indicating the most significant bit.

[0392] set_bit_pointer(pos) sets the position of the bitstream pointer so that get_bit_pointer() is equal to pos.

[0393] The following data types specify the parsing process for each syntax element.

[0394] ae(v): Context-Adaptive Arithmetic Entropy Coding syntax element. The parsing process for this data type is specified in subclause 9.3.4.3.2.

[0395] at(v): Arithmetic entropy coded terminal syntax. The parsing process for this data type is specified in subclause 9.3.4.3.5.

[0396] iae(n): A signed integer using n arithmetic entropy coded bits using the bypass mode of DeepCABAC as specified in subclause 9.3.4.3.4. Read bypass bins are interpreted as a two's complement integer representation written most significant bit first.

[0397] uae(n): An unsigned integer using n arithmetic entropy coded bits using the bypass mode of DeepCABAC as specified in subclause 9.3.4.3.4. Read bypass bins are interpreted as a binary representation of an unsigned integer written most significant bit first. For n=0, uae(n) does not decode any bins and returns 0.

[0398] f(n): a fixed-pattern bit string of n bits written left bit first (left to right). The parsing process for this data type is specified by the return value of the function read_bits(n).

[0399] i(n): A signed integer using n bits. If n is "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. The parsing process for this data type is specified by the return value of the function read_bits(n), which is interpreted as a two's complement integer representation written most significant bit first.

[0400] u(n): An unsigned integer using n bits. If n is "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. The parsing process for this data type is specified by the return value of the function read_bits(n), which is interpreted as a binary representation of an unsigned integer written with the most significant bit first.

[0401] ue(k): unsigned integer k-th Exp-Golomb coded syntax element. The parsing process of this descriptor follows the following pseudocode with x as the result: x = 0 bit = 1 while( bit ) { bit = 1 - u(1) x += bit << k k += 1 } k -= 1 if( k > 0 ) x += u(k) ie(k): signed integer k-th Exp-Golomb coded syntax element. The parsing process of this descriptor follows the following pseudocode with x as the result: val = ue( k ) if( (val & 1) != 0 ) x = ((val+1)>>1) else x = - (val>>1) flt(n): a floating-point value using n bits, where n can be 32, 64, or 128 in little-endian byte order, specified in ISO / IEC 60559 as binary32, binary64, or binary128, respectively.

[0402] st(v): A null-terminated string, shall be encoded as UTF-8 characters according to ISO / IEC 10646. The parsing process is specified as follows: st(v) reads and returns a sequence of bytes from the bitstream, starting at a byte-aligned position in the bitstream, starting at the current position and continuing up to, but not including, the next byte-aligned byte equal to 0x00, and advances the bitstream pointer by (stringLength+1)*8 bit positions, where stringLength is equal to the number of bytes returned.

[0403] NOTE The st(v) and flt(n) syntax descriptors are used in this document only when the current position in the bitstream is a byte-aligned position.

[0404] bs(v): A byte sequence specifies a variable-length sequence of bytes starting at a byte-aligned position. The length of the sequence is determined from the size of the NNR unit that contains the byte sequence.

[0405] more_data_in_nnr_unit() is specified as follows:

[0406] If more than -numBytesInNNRUnit data follows the current nnr_unit, i.e., if the decoded data so far in the current nnr_unit is less than numBytesInNNRUnit, the return value of more_data_in_nnr_unit() is equal to TRUE.

[0407] - Otherwise, the return value of more_data_in_nnr_unit() is equal to FALSE.

[0408] 6.1.4 Semantics (Example) The semantics associated with the syntax structures and the syntactic elements within each structure are specified in subclauses following the subclauses containing the syntax structures.

[0409] The following definitions apply to the semantics specification:

[0410] unspecified is used to designate some values ​​of certain syntax elements to indicate that these values ​​have no specified meaning in this document and may not be specified in the future as an integral part of a future version of this document.

[0411] reserved is used to specify that some values ​​of a particular syntax element are for future use by ISO / IEC and shall not be used in bitstreams conforming to this version of this document, but may be used in bitstreams conforming to future extensions of this document by ISO / IEC.] nnr_reserved_zero_0bit is an element of length 0. Decoders shall ignore the value of nnr_reserved_zero_0bit.

[0412] nnr_reserved_zero_1bit, if present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_1bit are reserved for future use by ISO / IEC. Decoders shall ignore values ​​of nnr_reserved_zero_1bit.

[0413] nnr_reserved_zero_2bits, if present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_2bits are reserved for future use by ISO / IEC. Decoders shall ignore values ​​of nnr_reserved_zero_2bits.

[0414] nnr_reserved_zero_3bits, if present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_3bits are reserved for future use by ISO / IEC. Decoders shall ignore values ​​of nnr_reserved_zero_3bits.

[0415] nnr_reserved_zero_5bits, if present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_5bits are reserved for future use by ISO / IEC. Decoders shall ignore values ​​of nnr_reserved_zero_5bits.

[0416] nnr_reserved_zero_7bits, if present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of nnr_reserved_zero_7bits are reserved for future use by ISO / IEC. Decoders shall ignore values ​​of nnr_reserved_zero_7bits.

[0417] 6.2 General bitstream syntax elements (e.g., all details are optional) 6.2.1 NNR unit For example, an NNR unit may be a data structure for carrying neural network data and associated metadata that is compressed or represented using this document.

[0418] The NNR unit carries, for example, compressed or uncompressed information, such as information regarding neural network metadata, topology information, full or partial layer data, filters, kernels, biases, quantized weights, tensors, etc.

[0419] An NNR unit consists, for example, of the following data elements (e.g., as shown by way of example in FIG. 12, which is a diagram illustrating a schematic example of an NNR unit data structure according to an embodiment): · NNR unit size (optional): This data element signals the total size in bytes of the NNR unit, including, for example, the NNR unit size.

[0420] · NNR unit header (optional): This data element contains, for example, information about the NNR unit type and associated metadata.

[0421] NNR unit payload: this data element contains, for example, compressed or uncompressed data related to a neural network.

[0422] 6.2.2 Aggregated NNR Units (Optional, Example) For example, an aggregated NNR unit is an NNR unit that carries multiple NNR units in its payload. An aggregated NNR unit provides, for example, a grouping mechanism for several NNR units that are related to each other and benefit from aggregation under a single NNR unit (as shown in FIG. 13, which illustrates a schematic example of an aggregated NNR unit data structure).

[0423] 6.2.3 Composition of NNR bitstream (optional; example) For example, an NNR bitstream is composed of a series of NNR units (as shown in FIG. 14, which illustrates a schematic example of an NNR bitstream data structure according to an embodiment).

[0424] For NNR bitstreams; unless otherwise specified in this document or defined by an NNR profile, the following constraints apply:

[0425] (For example, NNR_STR, NNR_MPS, NNR_NDU, NNR_LPS, NNR_TPL, and NNR_QNT are NNR unit types as specified, for example, in Table 3 of subclause 6.4.3.) An NNR bitstream shall begin, for example, with an NNR start unit (NNR_STR) (sub-clause 6.4.3).

[0426] For example, there shall be a single NNR model parameter set (NNR_MPS) (sub-clause 6.4.3) in the NNR bitstream that shall precede any NNR_NDU (sub-clause 6.4.3) in the NNR bitstream.

[0427] For example, an NNR layer parameter set (NNR_LPS) should be active until the next NNR layer parameter set in the NNR bitstream or until an aggregated NNR unit boundary is reached.

[0428] For example, the values ​​of topology_elem_id and topology_elem_id_index (sub-clause 6.4.3.7) must be unique within an NNR bitstream.

[0429] An NNR_TPL or NNR_QNT unit; if present in an NNR bitstream; shall precede any NNR_NDU that references a data structure (eg, topology_elem_id).

[0430] 6.3 NNR bitstream syntax (example, all details are optional) 6.3.1 NNR unit syntax (example)

[0431] [Table 6] 6.3.2 NNR unit size syntax (details optional) 6.3.3 NNR unit header syntax (details optional) 6.3.3.1 General

[0432] [Table 7] 6.3.3.2 NNR Start Unit Header Syntax (Details Optional)

[0433] [Table 8] 6.3.3.3 NNR model parameter set unit header syntax (details optional) 6.3.3.4 NNR layer parameter set unit header syntax (details optional) 6.3.3.5 NNR topology unit header syntax (details optional)

[0434] [Table 9] 6.3.3.6 NNR quantization unit header syntax (details optional)

[0435] [Table 10] 6.3.3.7 NNR Compressed Data Unit Header Syntax (e.g., all details are optional)

[0436] [Table 11] TIFF2025513886000346.tif234157TIFF2025513886000347.tif85158For example, integer_codebook() is defined as follows (all details are optional):

[0437] [Table 12] For example, tensor_dimension_list() is defined as follows (all details are optional):

[0438] [Table 13] For example, topology_elements_ids_list(topologyIndexedFlag) is defined as follows (all details are optional):

[0439] [Table 14] For example, topology_tensor_dimension_mapping() is defined as follows (all details are optional):

[0440] [Table 15] 6.3.3.8 NNR Aggregation Unit Header Syntax (Details Optional)

[0441] [Table 16] 6.3.4 NNR unit payload syntax (i.e., all details are optional) 6.3.4.1 General (i.e. all details are optional)

[0442] [Table 17] 6.3.4.2 NNR Start Unit Payload Syntax (all details are optional)

[0443] [Table 18] 6.3.4.3 NNR model parameter set unit payload syntax (all details are optional) 6.3.4.4 NNR layer parameter set unit payload syntax (all details are optional) 6.3.4.5 NNR Topology Unit Payload Syntax (all details are optional) 6.3.4.6 NNR quantization unit payload syntax (all details are optional)

[0444] [Table 19] 6.3.4.7 NNR Compressed Data Unit Payload Syntax (e.g., details are optional)

[0445] [Table 20] For example, decode_compressed_data_unit_payload() invokes the decryption process, for example as specified in subclause 7.3.

[0446] 6.3.4.8 NNR Aggregation Unit Payload Syntax (all details optional) 6.3.5 Byte-Aligned Syntax (All Details are Optional) 6.4 Semantics 6.4.1 General As an example, the semantics associated with syntax structures and elements within those structures are specified in this subclause. For example, if the semantics of a syntax element are specified using a table or set of tables, values ​​not specified in the table(s) shall not be present in the bitstream, for example, unless otherwise specified in this document.

[0447] 6.4.2 NNR Unit Size Semantics (all details are optional) 6.4.3 NNR unit header semantics (e.g., all details are optional) 6.4.3.1 General (i.e. all details are optional) For example, nnr_unit_type specifies the type of NNR unit, e.g., as specified in Table 3.

[0448] [Table 21] For example, values ​​in the range NNR_RSVD are reserved for use in future versions of this or a related specification. Encoders MUST NOT use these values. Decoders conforming to this version of the specification MAY, for example, ignore NNR units that use these values. Values ​​in the range NNR_UNSP are unspecified and their use is outside the scope of this specification. Decoders conforming to this version of the specification MAY, for example, ignore NNR units that use these values.

[0449] For example, independently_decodable_flag specifies whether this compressed data unit is independently decodable or not. A value of 1, for example, indicates an independently decodable NNR unit. A value of 0, for example, indicates that this NNR unit is not independently decodable and should be combined with other NNR units in order for its payload to be successfully decodable / decodable. The value of independently_decodable_flag should be the same for all NNR units that refer to the same topology_elem_id or topology_elem_id_index value or the same topology_elem_id_list, for example.

[0450] For example, partial_data_counter_present_flag equal to 1 specifies that the syntax element partial_data_counter is present in the NNR unit header. For example, partial_data_counter_present_flag equal to 0 specifies that the syntax element partial_data_counter is not present in the NNR unit header.

[0451] For example, partial_data_counter specifies the index of the partial data carried in the payload of this NNR Data Unit relative to the total data for a topology element. For example, a value of 0 indicates no partial information (i.e., the data in this NNR unit is all the data associated with the topology element and it is complete), and for example, a value greater than 0 indicates an index of partial information (i.e., the data in this NNR unit should be concatenated with the data in the accompanying NNR units until the partial_data_counter of the NNR unit reaches 1). For example, this counter counts backwards and indicates the total number of partitions first. For example, if not present, the value of partial_data_counter is inferred to be equal to 0. For example, if the value of independently_decodable_flag is equal to 0, the value of partial_data_counter_present_flag shall be equal to 1 and the value of partial_data_counter shall be greater than 0. For example, if the value of independently_decodable_flag is equal to 1, then the values ​​of partial_data_counter_present_flag and partial_data_counter are unspecified in this version of this document.

[0452] NOTE - In future versions of this document, when the value of independently_decodable_flag is equal to 1 and partial_data_counter_present_flag is equal to 1, partial_data_counter may have a non-zero value, for example, based on the assumption that multiple independently decodable NNR units are combined to build a model.

[0453] 6.4.3.2 NNR Start Unit Header Semantics (Details are Optional) 6.4.3.3 NNR model parameter set unit header semantics (details are optional) 6.4.3.4 NNR Layer Parameter Set Unit Header Semantics (Details Optional) 6.4.3.5 NNR Topology Unit Header Semantics (Details Optional) 6.4.3.6 NNR quantization unit header semantics (details optional) 6.4.3.7 NNR Compressed Data Unit Header Semantics (e.g., details are optional) For example, nnr_compressed_data_unit_payload_type (optional) is as defined in Table 7 of subclause 7.3.

[0454] For example, nnr_multiple_topology_elements_present_flag (optional) specifies whether multiple topology units are present in the bitstream. If multiple units are present, a list of their IDs is included. If nnr_compressed_data_unit_payload_type is set to NNR_PT_BLOCK, this flag is set to 1 and topology_elements_ids_list() in the NNR compressed data unit header lists, in the given order, the topology elements or topology element indices of RecWeight, RecWeightG, RecWeightH, RecLS, RecBeta, RecGamma, RecMean, RecVar, and RecBias based on their presence as indicated by the value of compressed_parameter_type in the NNR compressed data unit header.

[0455] For example, nnr_decompressed_data_format_present_flag (optional) specifies whether the data format obtained after decompression is present in the bitstream.

[0456] For example, input_parameters_present_flag (optional) specifies whether a group of elements including tensor dimensions, DeepCABAC unary lengths, and compressed parameter types are present in the bitstream.

[0457] For example, topology_elem_id (optional) specifies a unique identifier for the topology element to which the NNR compressed data unit refers. The semantic interpretation of this field is context dependent.

[0458] For example, topology_elem_id_index (optional) specifies the unique index value of a topology element signaled in the topology information of payload type NNR_TPL_REFLIST. The first index shall be 0 (i.e., 0-indexed).

[0459] For example, node_id_present_flag (optional) equal to 1 indicates that the syntax elements device_id, parameter_id, and put_node_depth are present.

[0460] For example, the device_id (optional) uniquely identifies the device that generated the current NDU.

[0461] For example, parameter_id (optional) uniquely identifies a parameter of the model to which the tensor stored in the NDU pertains. If parent_node_id_type is equal to ICNN_NDU_ID, parameter_id shall be equal to the parameter_id of the associated parent NDU.

[0462] For example, put_node_depth (optional) is the tree depth where the current NDU is located. Depth 0 corresponds to the root node. If parent_node_id_type is equal to ICNN_NDU_ID, put_node_depth-1 must be equal to the put_node_depth of the associated parent NDU.

[0463] For example, parent_node_id_present_flag (optional) equal to 1 indicates that the syntax element parent_node_id_type is present.

[0464] For example, parent_node_id_type (optional) specifies the parent node id type, which indicates what further syntax elements are present to uniquely identify the parent node. Allowed values ​​for parent_node_id_type are defined in Table 4.

[0465] [Table 22] For example, temporal_context_modeling_flag (optional) specifies whether to enable temporal context modeling. A temporal_context_modeling_flag equal to 1 indicates that temporal context modeling is enabled. If temporal_context_modeling_flag is not present, it is inferred to be 0.

[0466] For example, parent_device_id (optional) is equal to the syntax element device_id of the parent NDU.

[0467] For example, parent_node_payload_sha256 (optional) is the SHA256 hash of the nnr_compressed_data_unit_payload of the parent NDU.

[0468] For example, parent_node_payload_sha512 (optional) is the SHA512 hash of the nnr_compressed_data_unit_payload of the parent NDU.

[0469] For example, count_topology_elements_minus2+2 (optional) specifies the number of topology elements for which this NNR compressed data unit carries data in the payload.

[0470] For example, codebook_present_flag (optional) specifies whether a codebook is used. If codebook_present_flag is not present, it is inferred to be 0.

[0471] For example, dq_flag (optional) specifies whether the quantization method is dependent scalar quantization or, according to the subclause, uniform quantization. A dq_flag of 0 indicates that the uniform quantization method is used. A dq_flag of 1 indicates that the dependent scalar quantization method is used. If dq_flag is not present, it is inferred to be 0.

[0472] For example, nnr_decompressed_data_format (optional) is as defined in Table 6 of subclause 7.2.

[0473] For example, tensor_dimensions_flag (optional) specifies whether tensor dimensions are defined in the bitstream. If they are not included in the bitstream, they shall be obtained from the model topology description.

[0474] For example, cabac_unary_length_flag (optional) specifies whether the length of the unary part in DeepCABAC binarization is included in the bitstream.

[0475] For example, compressed_parameter_types (optional) specifies the compressed parameter types present in the current topology element to which the NNR compressed data unit refers. If multiple compressed parameter types are specified, they are combined by OR. The compressed parameter types are defined in Table 5.

[0476] [Table 23] For example, if a decomposition exists, then the tensors G and H represent the results of decomposing the original tensor. If (compressed_parameter_types&NNR_CPT_DC) != 0, then the variables TensorDimensionsG and TensorDimensionsH are derived, for example, as follows:

[0477] The variable TensorDimensionsG is set to [g_number_of_rows,decomposition_rank].

[0478] The variable TensorDimensionsH is set to [decomposition_rank, hNumberOfColumns], where hNumberOfColumns is defined as follows:

[0479] hNumberOfColumns = TIFF2025513886000360.tif1073For example, if (compressed_parameter_types&NNR_CPT_DC) != 0 and nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, then the NNR unit contains a decomposed tensor G and the next NNR unit in the bitstream contains, for example, the corresponding decomposed tensor H.

[0480] For example, the variable TensorDimensions is derived as follows:

[0481] -If the NNR unit contains a decomposed tensor G and nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, TensorDimensions is set to TensorDimensionsG.

[0482] - Else, if the NNR unit contains a decomposed tensor H and nnr_compressed_data_unit_payload_type != NNR_PT_BLOCK, TensorDimensions is set to TensorDimensionsH.

[0483] - Otherwise, TensorDimensions is set to tensor_dimensions.

[0484] For example, the variable NumBlockRowsMinus1 is defined as follows:

[0485] If scan_order is equal to 0, NumBlockRowsMinus1 is set to 0.

[0486] - Otherwise, if nnr_compressed_data_unit_payload_type == NNR_PT_BLOCK and (compressed_parameter_types & NNR_CPT_DC) != 0, then NumBlockRowsMinus1 is (TensorDimensionsG[0] + (4 <<scan_order)-1)> >(2+scan_order))+((TensorDimensionsH[0]+(4<<scan_order)-1)> > (2+scan_order) - 2.

[0487] - Otherwise, NumBlockRowsMinus1 is less than or equal to ((TensorDimensions[0]+(4<<scan_order)-1)> > (2+scan_order) - 1.

[0488] For example, decomposition_rank (optional) specifies the rank of the low-rank decomposed weight tensor components with respect to tensor_dimensions.

[0489] For example, g_number_of_rows (optional) specifies the number of rows of matrix G when reconstruction is performed on a decomposed tensor in an NNR unit of type NNR_PT_BLOCK.

[0490] For example, cabac_unary_length_minus1 (optional) specifies the length of the unary part in DeepCABAC binarization minus 1.

[0491] For example, first_tensor_dimension_shift (optional and present in some embodiments) specifies the shift of the first tensor dimension for tensor dimension reordering and shall be less than the value of count_tensor_dimensions. If first_tensor_dimension_shift is not present, it is inferred to be 0.

[0492] For example, scan_order (optional, and in some embodiments resent) specifies the block scan order for parameters with two or more dimensions, for example according to the table below.

[0493] -0: No block scanning -1:8×8 blocks -2: 16×16 blocks -3: 32×32 blocks -4: 64x64 blocks For example, cabac_offset_list (optional) specifies a list of values ​​that are used to initialize the variable IvlOffset at the beginning of the entry point.

[0494] For example, dq_state_list (optional) specifies a list of values ​​that are used to initialize the variable stateId at the beginning of the entry point.

[0495] For example, bit_offset_delta1 (optional) specifies the first element of the list BitOffsetList.

[0496] For example, bit_offset_delta2 (optional) specifies an element of the list BitOffsetList, excluding the first element, as the difference from the previous element of the list BitOffsetList.

[0497] For example, the variable BitOffsetList (optional) is a list of bit offsets that are used to set the bitstream pointer position at the beginning of the entry point.

[0498] For example, codebook_egk (optional) specifies the Exp-Golomb parameter k for decoding the syntax elements codebook_delta_left and codebook_delta_right.

[0499] For example, codebook_size (optional) specifies the number of elements in the codebook.

[0500] For example, codebook_centre_offset (optional) specifies the offset for accessing elements in the codebook relative to the center of the codebook. This is used to calculate the variable CbZeroOffset.

[0501] For example, codebook_zero_value (optional) specifies the value of the codebook at position CbZeroOffset, which participates in the creation of the variable Codebook (an array representing the codebook).

[0502] For example, codebook_delta_left (optional) specifies the difference between a codebook value and its right neighbor minus 1 for the left value relative to the center position. It is responsible for the creation of the variable Codebook (an array representing the codebook).

[0503] For example, codebook_delta_right (optional) specifies the difference between a codebook value and its left neighbor minus 1 for values ​​to the right of the center position. It participates in the creation of the variable Codebook (an array representing the codebook).

[0504] For example, count_tensor_dimensions (optional) specifies a counter for how many dimensions are specified. For example, for a 4-dimensional tensor, count_tensor_dimensions is 4. If not included in the bitstream, it shall be obtained from the model topology description.

[0505] For example, tensor_dimensions (optional) specifies an array or list of dimension values. For example, for a convolutional layer, tensor_dimensions is an array or list of length 4. For an NNR unit carrying elements G or H of a decomposed tensor, tensor_dimensions is set to the dimensions of the original tensor. The actual tensor dimensions of G and H for the decoding method are derived from tensor_dimensions, decomposition_rank, and g_number_of_rows. If not included in the bitstream, they shall be obtained from the model topology description. As an example, referring to FIG. 6, tensor_dimensions of tensor 680 may be, for example, a vector [D0, D1, D2, D3].

[0506] For example, topology_elem_id_list (optional) specifies a list of unique identifiers for the topology elements to which the NNR compressed data unit refers. The elements of topology_elem_id_list are semantically equivalent to the syntax element topology_elem_id or its index when listed in a topology payload of type NNR_TPL_REFLIST. The semantic interpretation of this field is context dependent.

[0507] For example, topology_elem_id_index_list (optional) specifies a list of unique indexes for the topology elements listed in the topology information of payload type NNR_TPL_REFLIST. The first element in a topology shall have an index value of 0.

[0508] For example, concatenation_axis_index (optional) indicates a 0-based concatenation axis.

[0509] For example, split_index[] (optional) indicates the tensor split index along the concatenation axis indicated by concatenation_axis_index to generate each of the individual concatenated tensors.

[0510] For example, number_of_shifts[] (optional) indicates how many left shift operations are performed.

[0511] For example, shift_index[k][i] (optional) denotes the axis index of the kth topology element to be shifted left.

[0512] For example, shift_value[k][i] (optional) indicates the amount of left shift on the axis by index index[k][i].

[0513] 6.4.3.8 NNR Aggregation Unit Header Semantics (Details Optional) 6.4.4 NNR unit payload semantics 6.4.4.1 General For example, the following NNR unit payload types are specified:

[0514] 6.4.4.2 NNR Start Unit Payload Semantics (Details are Optional) 6.4.4.3 NNR model parameter set unit payload semantics (details are optional) 6.4.4.4 NNR layer parameter set unit payload semantics (details are optional) 6.4.4.5 NNR Topology Unit Payload Semantics (details are optional) 6.4.4.6 NNR quantization unit payload semantics (details are optional) 6.4.4.7 NNR Compressed Data Unit Payload Semantics (e.g., details are optional) raw_float32_parameter is a float parameter tensor.

[0515] 6.4.4.8 NNR Aggregation Unit Payload Semantics (Details Optional) 7 Decryption Process 7.1 General (e.g., details optional) A decoder conforming to this document may, for example, take as input an NNR bitstream, as specified in subclause 6.3, and Generate decompressed data that conforms, for example, to the NNR decompressed data format (as defined, for example, in Table 6); or · Optionally generate, for example, ASCII or compressed data output, as indicated by using, for example, the NNR unit payloads of NNR_TPL and NNR_QNT (described in subclause 6.3.3).

[0516] As an example, for the decryption process, the following conditions may apply:

[0517] Any information required for decoding, for example, NNR units of an NNR bitstream SHOULD be signaled, for example, as part of the NNR bitstream. If such information is not part of the NNR bitstream, it SHALL be provided to the decoding process, for example, by other means (e.g., out-of-band topology information or parameters that are required for decoding but are not signaled or carried in the NNR bitstream).

[0518] For example, the decoding process shall start with an NNR unit of type NNR_STR. Upon receiving an NNR_STR unit, the decoder shall reset its internal state and prepare to receive an NNR bitstream. The presence and cardinality of the preceding NNR unit shall be as specified in the relevant clauses and appendices of this document.

[0519] Note: For example, the decoder may be further initialized via an NNR unit of type NNR_MPS to set global neural network model parameters.

[0520] For example, a decoder conforming to this document shall output data structures conforming to, e.g., the decompressed NNR data format as soon as it decompresses them. This allows, e.g., low latency between inputting NNR compressed data units and accessing the decompressed data structures from the output. How to establish the relationship between the input NNR units and the NNR decompressed output data is outside the scope of this document and is left to the implementation.

[0521] 7.2 NNR decompressed data format (example, details optional) For example, depending on the compression method used to create a particular bitstream, an NNR decoder is expected to output different decompressed data formats as a result of decoding an NNR data unit. For example, Table 6 specifies these NNR decompressed data formats that result after decompressing, for example, an NNR compressed data unit.

[0522] [Table 24] 7.3 Decryption methods (e.g., details optional) 7.3.1 General (details optional) This subclause specifies, by way of example, how to decode this document. For example, depending on the value of nnr_compressed_data_unit_payload_type, one of the subclause as specified in Table 7 is invoked.

[0523] [Table 25] For example, if the payload identifier is NNR_PT_INT, NNR_PT_FLOAT, or NNR_PT_FLOAT_RAW and multiple topology elements are combined (e.g., as signaled in the NNR compressed data unit header, e.g., via nnr_multiple_topology_elements_present_flag), the NNR decompressed tensor shall be further split into multiple tensors after the decoding process, for example, as follows:

[0524] For example, a Tensor RecParam is split into multiple tensors by calling, for example, TensorSplit(RecParam,split_index,concatenation_axis_index).

[0525] For example, the output of the function TensorSplit is a list of split output tensors associated with the topology elements, as specified by the array topology_elem_id_list, for example.

[0526] The output tensors are further processed by swapping their axes, as signaled in topology_tensor_dimension_mapping(), for example by calling AxisSwap().

[0527] 7.3.1.1 Tensor dimension reordering using the first tensor dimension shift (example, details optional) The inputs to this process are, for example:

[0528] A variable, for example inputTensor, representing the tensor whose dimensions are to be permuted. For example, the variable inputTensorDims specifies the dimensions of the inputTensor. For example, the output of this process is a variable reorderedTensor with dimensions equal to, for example, ShiftArrayIndex(inputTensorDims, first_tensor_dimension_shift). The elements of the variable reorderedTensor are set, for example, as follows:

[0529] for(i=0;i <Prod(inputTensorDims);i++){ idxA=TensorIndex(inputTensorDims,i,0) idxB=ShiftArrayIndex(idxA,first_tensor_dimension_shift) reorderedTensor[idxB]=inputTensor[idxA] } Thus, in general, a decoder, e.g. 100 as shown in FIG. 1 , e.g. 200 as shown in FIG. 2 , e.g. its respective permutation units 120, 220, may be configured to obtain a permuted multidimensional array, e.g. 121, 221, e.g. using the above process.

[0530] 7.3.2 How to decode an NNR compressed payload of type NNR_PT_INT (example, details optional) The inputs to this process are, for example:

[0531] For example, one or more NNR compressed data units marked to be decompressed together, for example by the partial_data_counter and nnr_compressed_data_unit_payload_type fields, are set as, for example, NNR_PT_INT.

[0532] The output of this process is a variable RecParam of type TENSOR_INT, e.g., as specified in Table 6. The dimensions of RecParam are, e.g., equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift). Decoding of a bitstream conforming to method NNR_PT_INT shall, e.g., only produce values ​​of RecParam that can be represented as 32-bit integer values, e.g., in two's complement representation.

[0533] For example, optionally, the arithmetic coding engine and the context model are initialized, for example, as specified in subclause 9.3.2.

[0534] For example, optionally, a syntax structure shift_parameter_ids(cabac_unary_length_minus1), eg according to sub-clause 9.2.1.6, is decoded from the bitstream and an initialization process of the probability estimation parameters, eg as specified in sub-clause 9.3.2.2, is invoked.

[0535] For example, optionally, a syntax structure quant_tensor(TensorDimensions, cabac_unary_length_minus1,0) according to subclause 9.2.1.4 is decoded from the bitstream, and, for example, RecParam is set equal to QuantParam.

[0536] For example, optionally, a syntax structure terminate_cabac(), for example according to subclause 9.2.1.2, is decoded from the bitstream.

[0537] For example, optionally, sub-clause 7.3.1.1 is called with RecParam and TensorDimensions as inputs, and the output is assigned to RecParam.

[0538] 7.3.3 How to decode an NNR compressed payload of type NNR_PT_FLOAT (example, details optional) For example, the inputs to this process are:

[0539] · One or more NNR compressed data units marked to be decompressed together, e.g. by their partial_data_counter and, e.g., their nnr_compressed_data_unit_payload_type fields, are set as NNR_PT_FLOAT.

[0540] For example, the output of this process is a variable RecParam of type TENSOR_FLOAT as specified in Table 6. For example, the dimension of RecParam is equal to ShiftArrayIndex(tensor dimension, first_tensor_dimension_shift).

[0541] For example, the arithmetic coding engine and the context model are initialized, for example, as specified in subclause 9.3.2.

[0542] For example, optionally, sub-clause 7.3.6 is called with, for example, TensorDimensions, 0 and (codebook_present_flag? 0:-1) as input, and the output is assigned to, for example, RecParam.

[0543] For example, optionally, a syntax structure terminate_cabac(), for example according to subclause 9.2.1.2, is decoded from the bitstream.

[0544] For example, optionally, sub-clause 7.3.1.1 is called with, for example, RecParam and TensorDimensions as input, and the output is assigned to RecParam.

[0545] For example, decoding of a bitstream conforming to the method NNR_PT_FLOAT shall, for example, only produce values ​​of RecParam that can be represented as float values ​​without loss of precision.

[0546] 7.3.4 How to decode an NNR compressed payload of type NNR_PT_RAW_FLOAT (example, details optional) For example, the output of this process is a variable RecParam of type TENSOR_FLOAT, for example as specified in Table 6. For example, the dimensions of RecParam are equal to ShiftArrayIndex(TensorDimensions, first_tensor_dimension_shift).

[0547] For example, RecParam is set equal to raw_float32_parameter.

[0548] For example, optionally, sub-clause 7.3.1.1 is called with, for example, RecParam and TensorDimensions as input, and the output is assigned to, for example, RecParam.

[0549] 7.3.5 How to decode an NNR compressed payload of type NNR_PT_BLOCK (example, details optional) For example, the inputs to this process are:

[0550] · One or more NNR compressed data units marked to be decompressed together, eg by partial_data_counter and their nnr_compressed_data_unit_payload_type fields, are set as eg NNR_PT_BLOCK.

[0551] For example, the output of this process is one or more variables, eg, of type TENSOR_FLOAT, depending on the value of compressed_parameter_types, eg, as specified in Table 6, eg, as follows:

[0552] If (compressed_parameter_types & NNR_CPT_DC) == 0: RecWeight (optional, example) If(compressed_parameter_types&NNR_CPT_DC)!=0:RecWeightG,RecWeightH(optional, example) if(compressed_parameter_types&NNR_CPT_LS)!=0:RecLS(optional, example) If (compressed_parameter_types & NNR_CPT_BN) != 0: RecBeta, RecGamma, RecMean, RecVar (any, example) If (compressed_parameter_types & NNR_CPT_BI) != 0: RecBias (optional, example) If present, the dimensions of RecWeight are equal to ShiftArrayIndex(TensorDimensions,first_tensor_dimension_shift) (optional, example).

[0553] If present, the dimensions of RecWeightG are equal to TensorDimensionsG (optional, e.g.).

[0554] If present, the dimensions of RecWeightH are equal to TensorDimensionsH (optional, example).

[0555] If present, the variables RecLS, RecBeta, RecGamma, RecMean, RecVar, and RecBias are 1D and their length is equal to the first dimension of TensorDimensions (optional, example).

[0556] For example, optionally, the arithmetic coding engine and the context model are initialized, for example, as specified in subclause 9.3.2.

[0557] If (compressed_parameter_types&NNR_CPT_LS) !=0, then subclause 7.3.6 is called with dimensions RecLS, -1, and 1 as inputs, and the output is assigned to RecLS (any, example).

[0558] If (compressed_parameter_types&NNR_CPT_BI) !=0, then subclause 7.3.6 is called with the dimensions RecBias, -1, and 1 as inputs, and the output is assigned to RecBias (any, example).

[0559] If (compressed_parameter_types&NNR_CPT_BN) !=0, then sub-clause 7.3.6 is called with dimensions RecBeta, -1, and 1 as inputs, and the output is assigned to RecBeta (any, example).

[0560] If (compressed_parameter_types&NNR_CPT_BN) !=0, then subclause 7.3.6 is called with dimensions RecGamma, -1, and 1 as inputs, and the output is assigned to RecGamma (any, example).

[0561] If (compressed_parameter_types&NNR_CPT_BN) !=0, then subclause 7.3.6 is called with dimensions RecMean, -1, and 1 as inputs, and the output is assigned to RecMean (any, example).

[0562] If (compressed_parameter_types&NNR_CPT_BN) !=0, then subclause 7.3.6 is called with RecVar, -1, and dimensions 1 as inputs, and the output is assigned to RecVar (any, example).

[0563] If (compressed_parameter_types&NNR_CPT_DC)==0, then subclause 7.3.6 is called with inputs RecWeight, 0, and the dimensions of (codebook_present_flag?0:-1), and the output is assigned to RecWeight (any, example).

[0564] If (compressed_parameter_types&NNR_CPT_DC) != 0 then the following applies: (Optional, Example) For example, subclause 7.3.6 is called with, for example, TensorDimensionsG,0 and (codebook_present_flag?0:-1) as inputs, and the output is assigned to, for example, RecWeightG.

[0565] For example, subclause 7.3.6 states that, for example, TensorDimensionsH, (TensorDimensionsG[0]+(4<<scan_order)-1)> >(2+scan_order))-1, and (codebook_present_flag?1:-1) as inputs, and the output is assigned to, for example, RecWeightH.

[0566] NOTE - From the decoded RecWeightG and RecWeightH, the variable RecWeight can be derived, for example, as follows: RecWeight=TensorReshape(RecWeightG*RecWeightH,TensorDimensions) For example, optionally, sub-clause 7.3.1.1 is called with, for example, RecWeight and TensorDimensions as input, and the output is assigned to, for example, RecWeight.

[0567] For example, optionally, a syntax structure terminate_cabac(), for example according to subclause 9.2.1.2, is decoded from the bitstream.

[0568] For example, optionally, if (compressed_parameter_types&NNR_CPT_DC)==0, sub-clause 7.3.1.1 is called with, for example, RecWeight and TensorDimensions as inputs, and the output is assigned to, for example, RecWeight.

[0569] 7.3.6 Decoding process of integer weight tensors (example, details optional) For example, the inputs to this process are:

[0570] The variable tensorDims (optional, example) that specifies the dimensions of the tensor to be decoded.

[0571] A variable entryPointOffset (optional, example) indicating whether an entry point exists for decryption, and if so, the entry point offset.

[0572] · A variable codebookId (optional, example) that indicates whether a codebook is applied and, if so, which codebook should be used.

[0573] For example, the output of this process is a variable recParam of type TENSOR_FLOAT with dimensions equal to tensorDims, for example, as specified in Table 6.

[0574] For example, optionally, the syntax structure quant_param(QpDensity) according to subclause 9.2.1.3 is decoded from the bitstream.

[0575] For example, optionally, the syntax structure shift_parameter_ids(cabac_unary_length_minus1) according to sub-clause 9.2.1.6 is decoded from the bitstream, and an initialization process of the probability estimation parameters, for example as specified in sub-clause 9.3.2.2, is invoked.

[0576] For example, optionally, a syntax structure quant_tensor(tensorDims, cabac_unary_length_minus1, entryPointOffset) according to subclause 9.2.1.4 is decoded from the bitstream and recParam is set as follows: if( codebookId == -1 ) recParam = QuantParam else { for( i = 0; i < Prod( tensorDims ); i++ ) { idx = TensorIndex( tensorDims, i ) if( codebookId == 0 ) recParam[idx] = Codebook[ QuantParam[idx] + CbZeroOffset ] else recParam[idx] = CodebookDC[ QuantParam[idx] + CbZeroOffsetDC ] } } For example, optionally, the variable stepSize is derived as follows: mul = (1 << QpDensity) + ( (qp_value + QuantizationParameter) & ( ( 1 << QpDensity ) - 1 ) ) shift = (qp_value + QuantizationParameter) >> QpDensity stepSize = mul * 2 shift - QpDensity For example, optionally, the variable recParam is updated as follows: recParam = recParam * stepSize NOTE - Following the above calculations, recParam can, for example, always be expressed as a dyadic fraction.

[0577] 8 Parameter reduction (details optional) 9 Entropy coding (example, details optional) 9.1 Method 9.1.1 DeepCABAC (example, details optional) 9.1.1.1 Thresholding (e.g., details optional) For example, the encoding method scans the parameter tensor in a manner defined, for example, by the function TensorIndex(). For example, each quantization parameter level is encoded according to the following procedure, employing, for example, an integer parameter "maxNumNoRemMinus1".

[0578] For example, optionally, in a first step, a binary syntax element sig_flag is coded for a quantization parameter level, for example specifying whether the corresponding level is equal to 0. If sig_flag is equal to 1, a further binary syntax element sign_flag is coded. For example, a bin indicates whether the level of the current parameter is positive or negative. Then, for example, a single term sequence of bins is coded, followed by a fixed length sequence, for example as follows:

[0579] For example, the variable k is initialized to 0, and for example, X is initialized to 1<<k. For example, the syntax element abs_level_greater_x / x2 is encoded, which indicates that, for example, the absolute value of the quantization parameter level is greater than x. For example, if abs_level_greater_x / x2 is equal to 1 and x is greater than maxNumNoRemMinus1, the variable k increases by, for example, 1 only. For example, then 1<<k is added to x, and for example, a further abs_level_greater_x / x2 is encoded. This procedure continues, for example, until abs_level_greater_x / x2 becomes equal to 0. Here, it is clear that, for example, X must be one of the values (x, x-1,... X-(1<<k)+1). For example, a code of length k is encoded, which refers to the values within a list that are, for example, the absolute quantization parameter levels.

[0580] 9.1.1.2 Row Skip (Example, Details Optional) For example, when enabled by the flag row_skip_flag_enabled_flag, the row skip technique can signal, for example, one flag row_skip_list[i] for each value i along the first axis of the parameter tensor. For example, if the flag row_skip_list[i] is 1, all elements of the parameter tensor whose index of the first axis is equal to i are set to 0. For example, if the flag row_skip_list[i] is 0, all elements of the parameter tensor whose index of the first axis is equal to i are encoded individually.

[0581] In general, a decoder according to an embodiment, e.g., decoder 100 as shown in FIG. 1, e.g., decoder 200 as shown in FIG. 2, e.g., its respective decoding unit 110, 210, may be configured, e.g., to decode a list of skipped rows, input the decoded neural network coefficients at respective positions described by a respective set of array indexes into a first multidimensional array, e.g., 111, 211, and determine whether to use default values ​​for given neural network parameters or to use the decoding to determine given neural network parameters depending on an entry of the list of skipped rows referenced by a current array index of a first dimension of the first multidimensional array.

[0582] 9.1.1.3 Context modelling (e.g., details optional) For example, context modeling corresponds to associating three types of flags, sig_flag, sign_flag, and abs_level_greater_x / x2, with a context model. In this way, for example, flags that have similar statistical behavior should be associated with the same context model so that, for example, a probability estimator (e.g., inside the context model) can, for example, adapt to the underlying statistics.

[0583] For example, the context modeling of the presented approach is as follows:

[0584] For example, 24 context models are distinguished for sig_flag depending on, for example, the state value and, for example, whether the left adjacent quantization parameter level is 0, less than 0, or greater than 0.

[0585] For example, if dq_flag is 0, then, for example, only the first three context models are used.

[0586] For example, the other three context models are differentiated for sign_flag depending on, for example, whether the left adjacent quantization parameter level is 0, less than 0, or greater than 0.

[0587] For example, for abs_level_greater_x / x2 flag, each x uses, for example, either one or two separate context models. For example, if x<=maxNumNoRemMinus1, two context models are distinguished depending on sign_flag. For example, if x>maxNumNoRemMinus1, only one context model is used.

[0588] 9.1.1.4 Temporal Context Modeling (e.g., details are optional) For example, if enabled by the flag temporal_context_modeling_flag, an additional set of context models for flags sig_flag, sign_flag, and abs_level_greater_x are available. For example, the derivation of ctxIdx is therefore also based on the value of the quantized co-located parameter level in the previously coded parameter update tensor, which can be uniquely identified, for example, by the parameter update tree. If, for example, the co-located parameter level is not available or is equal to 0, then the context modeling according to subclause 9.1.1.3 is applied. Otherwise, for example, if the co-located parameter level is not equal to 0, then the temporal context modeling of the presented approach is, for example, as follows:

[0589] For example, 16 context models are differentiated for sig_flag depending on, for example, the state value and whether the absolute value of the quantized co-located parameter level is greater than 1 or not.

[0590] For example, if dq_flag is 0, then only the first two additional context models are used.

[0591] For example, more than two further context models may be differentiated for sign_flag, eg depending on whether the quantized co-located parameter level is less than or greater than zero.

[0592] For example, for the abs_level_greater_x flag, each x uses two separate context models, where the two context models are differentiated depending on whether the absolute value of the quantized co-located parameter level is greater than or equal to x-1.

[0593] 9.2 Syntax and semantics (e.g., details are optional) 9.2.1 DeepCABAC syntax (example, details are optional) 9.2.1.1 General This subclause specifies the entropy coding syntax used, for example, by the decoding process of clause 7.

[0594] 9.2.1.2 DeepCABAC termination syntax (details optional) 9.2.1.3 Quantization parameter syntax (details optional) 9.2.1.4 Quantized tensor syntax (example, details optional)

[0595] [Table 26] TIFF2025513886000364.tif156158For example, row_skip_enabled_flag specifies whether to enable row skipping. For example, row_skip_enabled_flag being 1 indicates that row skipping is enabled.

[0596] For example, row_skip_list specifies a list of flags, e.g., the i-th flag row_skip_lsit[i] indicates whether all tensor elements of QuantParam whose first dimension index equals i are, e.g., 0. For example, if row_skip_list[i] is equal to 1, then all tensor elements of QuantParam whose first dimension index equals i are, e.g., 0.

[0597] For example, init_prob_est_param() invokes the initialization process specified, for example, in sub-clause 9.3.2.2.

[0598] For example, the 2D integer array StateTransTab[][] specifies the state transition table for dependent scalar quantization, e.g.: StateTransTab[][] = { {0, 2}, {7, 5}, {1, 3}, {6, 4}, {2, 0}, {5, 7}, {3, 1}, {4, 6}} 9.2.1.5 Quantization parameter syntax (details optional) 9.2.1.6 Shift Parameter Index Syntax (Details are Optional) 9.2.1.7 Shift Parameter Syntax (Details Optional) 9.3 Entropy Decoding Process (Details Optional) 9.3.1 General (details optional) 9.3.2 Initialization process (details optional) 9.3.2.1 General (details optional) 9.3.2.2 Initialization process for probability estimation parameters (details optional) 9.3.2.3 Context Variable Initialization Process (Details are Optional) 9.3.2.4 Arithmetic Decoding Engine Initialization Process (Details Optional) 9.3.3 Threshold process (details optional) 9.3.3.1 General (details optional) 9.3.3.2 Fixed-Length Threshold Process (Details Optional) 9.3.4 Decryption Process Flow (Details Optional) 9.3.4.1 General (details optional) 9.3.4.2 Derivation process of ctxIdx and bypassFlag (details are optional) 9.3.4.2.1 General (details optional) 9.3.4.2.2 Derivation process of ctxInc for syntax element sig_flag (details optional) 9.3.4.2.3 Derivation process of ctxInc for syntax element sign_flag (details optional) 9.3.4.2.4 Derivation process of ctxInc for syntax element abs_level_greater_x[j] (details are optional) 9.3.4.3 Arithmetic Decoding Process (Details Optional) 9.3.4.3.1 General (details optional) 9.3.4.3.2 Arithmetic Decoding Process for Binary Decision (Details Optional) 9.3.4.3.2.1 General (details optional) 9.3.4.3.2.2 State Transition Process (Details Optional) 9.3.4.3.3 Renormalization Process in the Arithmetic Decoding Engine (Details are Optional) 9.3.4.3.4 Bypass Decoding Process for Binary Decisions (Details Optional) 9.3.4.3.5 Decoding process for binary decision before termination (details optional) A further appendix A may relate to or define an implementation of NNEF.

[0599] Appendices A-E have not been included or have not been included in their entirety for the sake of brevity. It should be noted that the detailed features and characteristics of each of these appendices are optional both individually and in combination with respect to embodiments of the present invention.

[0600] This is discussed in the bibliography "Information technology-Multimedia content description interface". [1] Open Neural Network Exchange,VERSION 7,2020-05-09 (https: / / github.com / onnx / onnx / blob / master / onnx / onnx.proto) [2] PyTorch,VERSION 1.5.1,2020-10-22 (https: / / github.com / pytorch / pytorch / tree / v1.5.1) [3] TensorFlow,VERSION 2.2.0,2020-10-22 (https: / / github.com / tensorflow / tensorflow / tree / v2.2.0) Alternative implementations: Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0601] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system so that the respective methods are executed. The digital storage medium can therefore be computer readable.

[0602] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0603] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer. The program code can, for example, be stored on a machine readable carrier.

[0604] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0605] In other words, therefore, an embodiment of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0606] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0607] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0608] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0609] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0610] Further embodiments according to the invention comprise an apparatus or a system arranged to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0611] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0612] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0613] Apparatus The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0614] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0615] Any of the methods described herein, or elements of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0616] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and explanations of the embodiments herein.

Claims

1. A decoder (100, 200) for providing decoding parameters for a neural network based on an encoded representation, comprising: the decoder is configured to use the decoding (652) of the neural network parameters to obtain a first multidimensional array (111, 211, 670) including a plurality of neural network parameter values; the decoder is configured to obtain a permuted multidimensional array (121, 221, 680) using permutation (672), a first dimension of the first multidimensional array being rearranged to a different dimension within the permuted multidimensional array; Decoder (100, 200).

2. the decoder is configured to decode the dimension shift values ​​(112, 212, 630); the dimension shift value describes how much the first dimension of the first multidimensional array (111, 211, 670) should be shifted when performing the permutation; A decoder (100, 200) according to claim 1.

3. the decoder is configured to decode the dimension shift values ​​(112, 212, 630); the dimension shift value describes a new position of the first dimension of the first multidimensional array (111, 211, 670) in the permuted multidimensional array (121, 221, 680); A decoder (100, 200) according to claim 1 or 2.

4. The dimension shift values ​​are Exp-Golomb coded. A decoder (100, 200) according to claim 3.

5. the decoder is configured to perform the reordering using a single scalar dimension shift value (112, 212, 630) as the only parameter describing the new order of the dimensions in the reordered multidimensional array (121, 221, 680). A decoder (100, 200) according to any one of claims 1 to 4.

6. the decoder is configured to perform a single shift of a single dimension to another position to obtain the permuted multidimensional array (121, 221, 680). A decoder (100, 200) according to any one of claims 1 to 5.

7. the decoder is configured to derive a dimension of an auxiliary array from a first dimension of the first multidimensional array (111, 211, 670); A decoder (100, 200) according to any one of claims 1 to 6.

8. the decoder is configured to shift a single dimension of the first multidimensional array (111, 211, 670) to a different position to obtain the permuted multidimensional array (121, 221, 680). A decoder (100, 200) according to any one of claims 1 to 7.

9. the decoder is configured to obtain a two-dimensional matrix (660), a first dimension of the two-dimensional matrix (660) being determined by a first dimension value and a second dimension of the two-dimensional matrix (660) being determined by a product of a plurality of further dimension values ​​based on the encoded representation; the decoder is configured to obtain the first multidimensional array (111, 211, 670), the dimensions of which are determined by respective dimension values ​​of the first dimension value and the further dimension value based on the two-dimensional matrix (660), the decoder is configured to obtain the permuted multidimensional array (121, 221, 680) based on the first multidimensional array, a first dimension of the permuted multidimensional array being defined by one of the further dimension values. A decoder (100, 200) according to any one of claims 1 to 8.

10. the decoder is configured to decode the encoded neural network parameters using context-based entropy decoding; the decoder is configured to determine a location of the decoded neural network parameters in the first multidimensional array (111, 211, 670) using a location mapping function that maps a scalar neural network parameter index to a set of dimension indices. A decoder (100, 200) according to any one of claims 1 to 9.

11. the decoder is configured to consecutively store, along a given dimension of the first multidimensional array (111, 211, 670), a plurality of decoded neural network parameters that are immediately adjacent in terms of decoding order; A decoder (100, 200) according to any one of claims 1 to 10.

12. The decoder comprises: for( i = 0; i < Prod( inputTensorDims ); i++ ){ idxA = TensorIndex( inputTensorDims, i, 0 ) idxB = ShiftArrayIndex( idxA, firstTensorDimShift ) reorderedTensor[idxB] = inputTensor[idxA] } to obtain the permuted multidimensional array (121, 221, 680) using a process of the form where i is the execution variable, Prod(inputTensorDims) is the product of the lengths of the dimensions of the first multidimensional array (111, 211, 670); TensorIndex(.,.,.) is a function that maps integer values ​​to a set of array indices that specify elements of the first multidimensional array; inputTensorDims is a set of values ​​describing the lengths of the dimensions of the first multidimensional array; idxA is a variable, ShiftArrayIndex(.,.) is a function that maps an input set of array indexes to a returned set of array indexes that is a copy of the input set of array indexes, except that the element of the input set of array indexes at position 0 has been shifted to the position specified by firstTensorDimShift; firstTensorDimShift is the dimension shift value (112, 212, 630), idxB is a variable, inputTensor[. ] is the first multidimensional array, reorderedTensor[. ] is the reordered multidimensional array, inputTensor[idxA] is the element of the first multidimensional array at the position specified by idxA; reorderedTensor[idxB] is the element of the reordered multidimensional array at the position specified by idxB. A decoder (100, 200) according to any one of claims 1 to 11.

13. the decoder is configured to obtain the sorted multidimensional array (121, 221, 680) using a function that maps integer element indices i that specify elements of the first multidimensional array (111, 211, 670) to a set of array indices; the returned set of array indices designates the i-th element in row-major scan order of the first multidimensional array (111, 211, 670); or the set of returned array indices designates the i-th element of a block-wise scan of the first multidimensional array, the first multidimensional array being considered as a two-dimensional array; A decoder (100, 200) according to any one of claims 1 to 12.

14. the decoder is configured to decode the list of skipped rows; the decoder is configured to input decoded neural network coefficients into the first multidimensional array (111, 211, 670) at respective positions described by respective sets of array indices; the decoder is configured to determine whether to use a default value for a given neural network parameter or to determine the given neural network parameter using decoding in response to an entry in the list of skipped rows referenced by a current array index of the first dimension of the first multidimensional array. A decoder (100, 200) according to any one of claims 1 to 13.

15. the decoder is configured to decode a set of array dimensions; the decoder configured to obtain a set of permuted array dimensions; A decoder (100, 200) according to any one of claims 1 to 14.

16. the decoder is configured to input decoded neural network parameters into the first multidimensional array (111, 211, 670) at respective positions described by respective sets of array indices; the decoder is configured to map a scalar integer parameter index to the respective set of array indexes to obtain the respective set of array indexes using a mapping function that defines a block-wise scan. A decoder (100, 200) according to any one of claims 1 to 15.

17. the mapping function includes a mapping of the scalar integer parameter index to two coordinates that indicate a position corresponding to a scan index defined by the scalar integer parameter index when the block is scanned within the block; the mapping function includes mapping the two coordinates to a respective set of array indices; A decoder (100, 200) according to claim 16.

18. the decoder is configured to input decoded neural network parameters into the first multidimensional array (111, 211, 670) at respective positions described by respective sets of array indices; the decoder is configured to obtain a set of respective array indices based on a scalar integer parameter index i using a function TensorIndex(tensorDimensions[], i, scan); The function TensorIndex(tensorDimensions[], i, scan) returns an array of array indices with the same number of dimensions as the array tensorDimensions[]; wherein the elements of the array of array indices are If variable scan is equal to 0: the returned array of array indices points to the i-th element in row-major scan order of an array having dimensions defined by tensorDimensions[ ]; If variable scan is greater than 0: The variable bs is set to 4<<scan_order, The variable h is set to tensorDimensions[0], The variable w is set to Prod(tensorDimensions) / h, Two variables x and y are set to the first and second elements of the array returned by calling IndexToXY(w,h,i,bs), respectively; The returned array is TensorIndex(tensorDimensions, y*w+x, 0). The tensorDimensions[] elements are set to integer values ​​so that they can be used as indices to point to elements of arrays, e.g. tensors, with dimensions tensorDimensions[], such that The function IndexToXY(w,h,i,bs) returns an array with two elements, the first element returned by the function IndexToXY is an x ​​coordinate and the second element returned by the function IndexToXY is a y coordinate that points into a 2D array of width w and height h; x and y indicate a position corresponding to scan index i when the block is scanned in blocks of size bs×bs; x and y are The variable fullRowOfBlocks is set to w*bs, The variable blockY is set to i / fullRowOfBlocks, The variable iOff is set to i%fullRowOfBlocks, The variable currBlockH is set to Min(bs, h-blockY*bs), The variable fullBlocks is set to bs*currBlockH, The variable blockX is set to iOff / fullBlocks, The variable blockOff is set to iOff%fullBlocks, The variable currBlockW is set to Min(bs, w-blockX*bs), The variable posX is set to blockOff%currBlockW, The variable posY is set to blockOff / currBlockW, The variable x is set to blockX*bs+posX, The variable y is set to blockY*bs+posY It is derived as follows: tensorDimensions[ ] is an array of values ​​describing the lengths of the first multidimensional array in different dimensions; scan is an integer variable describing the scanning mode; A decoder (100, 200) according to any one of claims 1 to 17.

19. An encoder (300) for providing an encoded representation of parameters of a neural network, comprising: the encoder is configured to obtain a permuted multidimensional array (321, 620) using permutation, where a given dimension (301) of a given multidimensional array (302, 610) of neural network parameters is rearranged to a first dimension in the permuted multidimensional array; the encoder configured to encode the permuted multi-dimensional array. Encoder (300).

20. the encoder is configured to encode a dimension shift value (112, 212, 630); the dimension shift value describes which given dimension of the given multidimensional array (302, 610) is rearranged to the first dimension in the permuted multidimensional array (321, 620); or the dimension shift value describes how much the first dimension of the encoded permuted multidimensional array should be shifted when performing decoder-side permutation (672); 20. The encoder (300) of claim 19.

21. the encoder configured to encode (632) the dimension shift values ​​(112, 212, 630); the dimension shift value describes a new position to which the first dimension of the encoded permuted multidimensional array should be moved within a decoder. An encoder (300) according to claim 19 or 20.

22. the encoder is configured to encode the dimension shift values ​​using an Exp-Golomb code. The encoder (300) of claim 21.

23. the encoder is configured to perform the permutation using a single scalar dimension shift value (112, 212, 630) as the only parameter describing a new order of the dimensions in the permuted multidimensional array (321, 620). An encoder (300) according to any one of claims 19 to 22.

24. the encoder is configured to perform a single shift of a single dimension to another position to obtain the permuted multidimensional array (321, 620). An encoder (300) according to any one of claims 19 to 23.

25. the encoder is configured to set a dimension of an auxiliary array included in the encoded representation to be equal to the first dimension of the permuted multidimensional array (321, 620); An encoder (300) according to any one of claims 19 to 24.

26. the encoder is configured to shift a single dimension of the given multidimensional array (302, 610) to a different position to obtain the permuted multidimensional array (321, 620). An encoder (300) according to any one of claims 19 to 25.

27. The encoder is configured to obtain the permuted multidimensional array (321, 620), and a dimension of the permuted multidimensional array (321, 620) is determined by respective dimension values ​​of a first dimension value associated with the first dimension and further dimension values ​​associated with further dimensions beyond the first dimension based on the given multidimensional array (302, 610); the encoder is configured to obtain a two-dimensional matrix (640), a first dimension of which is determined by a first dimension value and a second dimension of which is determined by a product of the further dimension values ​​based on the permuted multidimensional array (321, 620); the encoder configured to encode (642) the two-dimensional matrix; An encoder (300) according to any one of claims 19 to 26.

28. the encoder is configured to encode the neural network parameters using context-based entropy coding; the encoder is configured to determine neural network parameters in the permuted multidimensional array (321, 620) to be encoded using a position mapping function that maps scalar neural network parameter indices to a set of dimension indices. An encoder (300) according to any one of claims 19 to 27.

29. the encoder is configured to read consecutively, along a given dimension of the permuted multidimensional array (321, 620), a number of neural network parameters to be encoded that are immediately adjacent in terms of encoding order; An encoder (300) according to any one of claims 1 to 28.

30. The encoder is configured as follows: for (i=0; i<Prod(inputTensorDims); i++), idxA=TensorIndex(inputTensorDims,i,0) idxB=ShiftArrayIndexEnc(idxA, firstTensorDimShift) reorderedTensor[idxB]=inputTensor[idxA] ｝ to obtain the permuted multidimensional array (321, 620) using a process of the form where i is the execution variable, Prod(inputTensorDims) is the product of the lengths of the dimensions of the given multidimensional array (302, 610); TensorIndex(.,.,.) is a function that maps integer values ​​to a set of array indices that specify elements of the given multidimensional array; inputTensorDims is a set of values ​​describing the lengths of the dimensions of the given multidimensional array; idxA is a variable, ShiftArrayIndexEnc(.,.) is a function that maps an input set of array indexes to a returned set of array indexes that is a copy of the input set of array indexes, except that the elements of the input set of array indexes at the position specified by firstTensorDimShift have been shifted to position 0; firstTensorDimShift is the dimension shift value (112, 212, 630), idxB is a variable, inputTensor[. ] is the given multidimensional array, reorderedTensor[. ] is the reordered multidimensional array, inputTensor[idxA] is the element of the given multidimensional array at the position specified by idxA; reorderedTensor[idxB] is the element of the reordered multidimensional array at the position specified by idxB. An encoder (300) according to any one of claims 19 to 29.

31. the encoder is configured to obtain the permuted multidimensional array (321, 620) using a function that maps integer element indices i that specify elements of the given multidimensional array (302, 620) to a set of array indices; the returned set of array indices specifies the i-th element in row-major scan order of the given multidimensional array; or the returned set of array indices designates the i-th element of a block-wise scan of said given multi-dimensional array, said given multi-dimensional array being considered as a two-dimensional array; An encoder (300) according to any one of claims 19 to 30.

32. the encoder configured to encode a list of skipped rows; the encoder is configured to encode neural network coefficients of the permuted multidimensional array (321, 620) at respective positions described by respective sets of array indices; the encoder is configured to determine whether to encode neural network parameters at a current position in response to an entry in the list of skipped rows referenced by a current array index of the first dimension of the permuted multidimensional array. An encoder (300) according to any one of claims 19 to 31.

33. the encoder configured to obtain a set of array dimensions; the encoder configured to encode the set of array dimensions; An encoder (300) according to any one of claims 19 to 32.

34. the encoder is configured to encode neural network parameters of the permuted multidimensional array (321, 620) at respective positions described by respective sets of array indices; the encoder is configured to map a scalar integer parameter index to the respective set of array indexes to obtain the respective set of array indexes using a mapping function that defines a block-wise scan. An encoder (300) according to any one of claims 19 to 33.

35. the mapping function includes a mapping of the scalar integer parameter index to two coordinates that indicate a position corresponding to a scan index defined by the scalar integer parameter index when the block is scanned within the block; the mapping function includes mapping the two coordinates to a respective set of array indices; The encoder (300) of claim 34.

36. the encoder is configured to encode neural network parameters of the permuted multidimensional array (321, 620) at respective positions described by respective sets of array indices; the encoder is configured to obtain a set of respective array indices based on a scalar integer parameter index i using a function TensorIndex(tensorDimensions[], i, scan); said function TensorIndex(tensorDimensions[], i, scan) returns an array of array indices having the same number of dimensions as the array tensorDimensions[]; wherein the elements of the array of array indices are If variable scan is equal to 0: the returned array of array indices points to the i-th element in row-major scan order of an array having dimensions defined by tensorDimensions[ ]; If variable scan is greater than 0: The variable bs is set to 4<<scan_order, The variable h is set to tensorDimensions[0], The variable w is set to Prod(tensorDimensions) / h, Two variables x and y are set to the first and second elements of the array returned by calling IndexToXY(w,h,i,bs), respectively; The returned array is TensorIndex(tensorDimensions, y*w+x, 0). The tensorDimensions[] elements are set to integer values ​​so that they can be used as indices to point to elements of arrays, e.g. tensors, with dimensions tensorDimensions[], such that The function IndexToXY(w,h,i,bs) returns an array with two elements, the first element returned by the function IndexToXY is an x ​​coordinate and the second element returned by the function IndexToXY is a y coordinate that points into a 2D array of width w and height h; x and y indicate a position corresponding to scan index i when the block is scanned in blocks of size bs×bs; x and y are The variable fullRowOfBlocks is set to w*bs, The variable blockY is set to i / fullRowOfBlocks, The variable iOff is set to i%fullRowOfBlocks, The variable currBlockH is set to Min(bs, h-blockY*bs), The variable fullBlocks is set to bs*currBlockH, The variable blockX is set to iOff / fullBlocks, The variable blockOff is set to iOff%fullBlocks, The variable currBlockW is set to Min(bs, w-blockX*bs), The variable posX is set to blockOff%currBlockW, The variable posY is set to blockOff / currBlockW, The variable x is set to blockX*bs+posX, The variable y is set to blockY*bs+posY It is derived as follows: tensorDimensions[ ] is an array of values ​​describing the lengths of the first multidimensional array in different dimensions; 36. An encoder (300) according to any one of claims 19 to 35, wherein scan is an integer variable describing a scanning mode.

37. A method (400) for providing decoding parameters for a neural network based on an encoded representation, comprising: The method includes obtaining (410) a first multidimensional array (111, 211, 670) including a plurality of neural network parameter values ​​using a decoding (652) of the neural network parameters; The method includes obtaining (420) a permuted multidimensional array (121, 221, 680) using permutation (672), where a first dimension of the first multidimensional array is rearranged to a different dimension within the permuted multidimensional array. Method (400).

38. A method (500) for providing an encoded representation of parameters of a neural network, comprising: The method includes obtaining (510) a permuted multidimensional array (321, 620) using permutation (612), where a given dimension (301) of a given multidimensional array (302, 620) of neural network parameters is rearranged to a first dimension in the permuted multidimensional array; The method includes encoding (520) the permuted multi-dimensional array. Method (500).

39. A computer program adapted to perform the method according to claim 37 or 38 when said computer program is run on a computer.

Images