Representation formats of neural networks

Abstract

To provide a concept for the efficient utilization of a neural network and / or, efficient transmission and / or update of a neural network.SOLUTION: A data stream (45) is an encoded representation of a neural network (NN). The data stream (45) includes serialization parameters (102) indicating an encoding order (104) in which neural network parameters (32) defining neural network (NN) neuron interconnections are encoded in the data stream (45).SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to concepts related to the representation form of neural networks.

Background Art

[0002] Neural networks (NNs) are currently achieving breakthroughs in many applications. - Detection and classification of objects in image / video data - Speech recognition and keyword recognition of speech - Speech synthesis - Optical character recognition - Language translation - etc.

[0003] However, due to the huge amount of data required for NN representation, it may be difficult to apply depending on the usage scenario. In many cases, this data is composed of two types of parameters, "weights" and "biases", which represent the connections between neurons. Weights are usually parameters that perform some linear transformation (e.g., inner product or convolution) on the input values, in other words, parameters that assign weights to the inputs of neurons, and biases are offsets added after linear calculation, in other words, something that cancels out the aggregation of weighted messages received by neurons. More specifically, these weights, biases, and additional parameters characterizing each connection between potentially very many neurons (up to tens of millions) in each layer (up to hundreds) of the NN occupy the main part of the data related to a specific NN. Also, these parameters are typically composed of large floating-point date types. These parameters are usually represented as large tensors containing all the parameters of each layer. In an application, when frequent transmission or update of the related NN is required, the required data rate becomes a serious bottleneck. Therefore, performing non-invertible compression of these matrices is a promising approach to reduce the encoding size of NN representation.

[0004] Typically, parameter tensors are stored in a container format (such as ONNX (Open Neural Network Exchange), PyTorch, or TensorFlow) that carries all the data necessary to completely reconstruct and run the neural network (such as the parameter matrix mentioned above) and further properties (such as the dimension of the parameter tensor, the type of layers, and the operations).

[0005] It would be advantageous to have at hand a concept that makes sending / updating machine learning predictors, or alternatively, machine learning models such as neural networks, more efficient in terms of preserving inference quality while simultaneously reducing the coded size of the NN representation, the complexity of computational inference, and the complexity of describing or storing the NN representation; or to enable more frequent sending / updating of NNs than currently; or to improve the inference quality for specific tasks and / or specific local input data statistics at present. Furthermore, it would be advantageous to provide neural network representations for making predictions based on neural networks, derivations of such neural network representations, and methods for using such neural network representations, so that the use of neural networks becomes more effective than currently. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Accordingly, an object of the present invention is to provide a concept for the efficient use of neural networks and / or for the efficient transmission and / or updating of neural networks. This object is achieved by the subject matter of the independent claims of this application. [Means for solving the problem]

[0007] Further embodiments of the present invention are defined by the subject matter of the dependent claims of this application.

[0008] The basic idea of ​​the first aspect of the present invention is that the representation of the NN is serialized into the encoded data stream. (serial)The idea is that encoding / decoding serialization parameters improves the efficiency of using a neural network (NN). Serialization parameters indicate the encoding order in which NN parameters, which define the neuronal interconnections of the NN, are encoded into the data stream. Neuronal interconnections may represent the connections between neurons in different layers of the NN. In other words, NN parameters may define the connection between a first neuron associated with the first layer of the NN and a second neuron associated with the second layer of the NN. Decoding allows the NN parameters, decoded sequentially from the data stream, to be assigned to neuronal interconnections using the encoding order.

[0009] In particular, it has been found that using serialization parameters allows for the efficient partitioning of bit sequences into meaningful, consecutive subsets of NN parameters. Serialization parameters may indicate a grouping of NN parameters that enables efficient execution of the NN. This may depend on the application scenario of the NN. For different application scenarios, encoding may involve scanning the NN parameters using different encoding orders. Therefore, NN parameters can be encoded using individual encoding orders dependent on the NN's application scenario, and decoding can reconstruct the NN parameters accordingly during decoding based on the information provided by the serialization parameters. NN parameters may represent entries in one or more parameter matrices or tensors, which may be used in inference procedures. It has been found that one or more parameter matrices or tensors of an NN can be efficiently reconstructed by a decoder based on the decoded NN parameters and serialization parameters.

[0010] Thus, serialization parameters can use different application-specific coding orders, enabling flexible coding and decoding with improved efficiency. For example, coding parameters along different dimensions can benefit compression performance because the entropy encoder can better capture the dependencies between them. In another example, it may be desirable to group parameters according to some application-specific criterion, namely which part of the input data they relate to or whether they are co-executable, so that they can be coded / input in parallel. A further example is coding parameters according to a GEMM (General Matrix Matrix) product scan order, which supports efficient memory allocation of coded parameters when performing a dot product (Andrew Kerr, 2017).

[0011] Further embodiments are directed towards selective sorting on the data encoder side to achieve energy compression of the NN parameters to be encoded, and then to process / serialize / encode the resulting sorted data according to the order of the results. Thus, the sorting can be done so that the parameters are sorted so that they steadily increase or decrease along the encoding order.

[0012] According to a second aspect of the present application, the inventors of the present application encode / decode numerical computation representation parameters into / from an encoded data stream, thereby enabling the neural network We noticed an increase in the efficiency of using the NN (Neural Network). The numerical computation representation parameters indicate the numerical representation, e.g., floating-point or fixed-point representation, and bit size, of the NN parameters encoded in the data stream when the NN is used for inference. The encoder is configured to encode the NN parameters. The decoder is configured to decode the NN parameters and may be configured to use the numerical representation and bit size to represent the NN parameters decoded from the data stream, DS.

[0013] This embodiment is based on the idea that it is advantageous to represent NN parameters and activation values ​​obtained as a result of using NN parameters in inference using the NN with the same numerical representation and bit size. Based on the numerical computation representation parameter, it is possible to efficiently compare the possible numerical representations and bit sizes of the NN parameters and the activation values. This is considered particularly advantageous when the numerical computation representation parameter shows a fixed-point representation as its numerical representation. This is because if both the NN parameters and activation values ​​can be represented in a fixed-point representation, inference can be performed efficiently using fixed-point arithmetic.

[0014] According to a third aspect of the present application, the inventors of the present application have achieved increased efficiency in using a neural network by encoding / decoding NN layer type parameters to / from an encoded data stream in which the representation of the NN is encoded. The NN layer type parameters indicate the NN layer type (e.g., convolutional layer type or fully connected layer type) of a given NN layer of the NN. The data stream is structured into one or more individually accessible parts, each individually accessible part representing a corresponding NN layer of the NN. A given NN layer represents one of the NN layers of a neural network. Optionally, for each of two or more given NN layers of the NN, the NN layer type parameters are encoded / decoded to / from the data stream, and the NN layer type parameters can differ among at least several given NN layers.

[0015] This embodiment is based on the idea that, for example, it may be useful for the data stream to contain the NN layer type parameters of the NN layer in order to understand the meaning of the dimensions of the parameter tensor / matrix. Furthermore, different layers may be treated differently when decoding encodes important information that the decoding should know before decoding, for example by using different sets or modes of the context model, in order to better capture data dependencies and lead to higher encoding efficiency.

[0016] Similarly, it may be advantageous to encode / decode type parameters into the data stream, indicating the parameter types of NN parameters. Type parameters can indicate whether NN parameters represent weights or biases. The data stream is structured into one or more individually accessible parts, each representing a corresponding NN layer of the NN. An individually accessible part representing a given NN layer may be further structured into individually accessible subparts. Each individually accessible subpart is fully traversed by the encoding order before subsequent individually accessible subparts are traversed by the encoding order. Each individually accessible subpart can, for example, encode and decode NN parameters and type parameters. The NN parameters in the first individually accessible subpart may be of the same parameter type as those in the second individually accessible subpart, or they may be of different parameter types. Different types of NN parameters associated with the same NN layer may be encoded / decoded into different individually accessible subparts associated with the same individually accessible part. Parameter type distinctions may be beneficial for encoding / decoding, for example, when different type dependencies can be used for each parameter type, or when parallel decoding is desired. For example, the same N It is possible to encode / decode different types of NN parameters associated in parallel across the N layers. This improves the efficiency of encoding / decoding NN parameters and allows the entropy encoder to better capture the dependencies between NN parameters, which can result in advantages in compression performance.

[0017] According to a fourth aspect of the present application, the inventors of the present application have found that transmitting / updating a neural network becomes highly efficient when a pointer is encoded in a data stream in which the representation of the NN is encoded, and the pointer is decoded from the data stream. This is because the data stream is composed of individually accessible portions, and for each of one or more predetermined individually accessible portions, a pointer points to the beginning of each predetermined individually accessible portion. Not all individually accessible portions need to be predetermined individually accessible portions, but all individually accessible portions may represent predetermined individually accessible portions. One or more predetermined individually accessible portions may be set by default, or they may be set depending on the application of the NN encoded in the data stream. The pointer indicates the beginning of each predetermined individually accessible portion, for example, as a data stream position in bytes, or as an offset, for example, relative to the beginning of the data stream, or as an offset relative to the beginning of the portion corresponding to the NN layer to which each predetermined individually accessible portion belongs. This pointer may be encoded in the header portion of the data stream and decoded from the header portion. According to the embodiment, for each of one or more predetermined individually accessible portions, the pointer is encoded / decoded into the header portion of the data stream if each predetermined individually accessible portion represents the corresponding NN layer of the neural network, and the pointer is encoded / decoded into the parameter set portion of the corresponding part of the NN layer if each predetermined individually accessible portion represents the NN layer of the NN. The NN portion of the NN layer of the NN may represent the baseline portion of each NN layer, or it may represent the advanced section of each layer. By using pointers, predetermined individually accessible portions of the data stream can be accessed efficiently, making it possible, for example, to parallelize layer processing or package the data stream into its respective container format.By using pointers, it becomes easier, faster, and more appropriate to access specific, individually accessible parts, thus facilitating applications that require parallel or partial decoding and execution of neural networks.

[0018] According to a fifth aspect of the present application, the inventors of the present application have achieved highly efficient transmission / update of a neural network when a start code, a pointer, and / or a data stream length parameter are encoded into an individually accessible sub-part of the data stream in which the representation of the NN is encoded / the sub-parts are decoded from the data stream. The data stream is structured into one or more individually accessible parts, each individually accessible part representing a corresponding NN layer of the neural network. Furthermore, within one or more predetermined individually accessible parts, the data stream is further structured into individually accessible sub-parts, each individually accessible sub-part representing a corresponding NN part of each NN layer of the neural network. The device is configured to encode into and / or decode from the data stream, for each of the one or more predetermined individually accessible sub-parts, a start code in which each predetermined individually accessible sub-part begins, and / or a pointer pointing to the beginning of each predetermined individually accessible sub-part, and / or a data stream length parameter indicating the data stream length of each predetermined individually accessible sub-part in order to skip each predetermined individually accessible sub-part when parsing the DS. The start code, pointer, and / or data stream length parameter enable efficient access to predetermined, individually accessible sub-parts. This allows for partial access to the NN parameters. Because it is beneficial to decode / process / input in parallel, this is particularly useful for applications that may rely on grouping NN parameters within NN layers in a specific configurable way. Thus, access to individually accessible sub-parts of individually accessible parts helps to access necessary data in parallel and to filter out unnecessary data parts. It has also been found that it is sufficient to indicate this using a start code. This is based on the observation that the amount of data per NN layer, i.e., the individually accessible part, is usually less than the amount of data in the entire data stream that an NN layer is detected by a start code. Nevertheless, it is also advantageous to use pointers and / or data stream length parameters to improve access to individually accessible parts. According to the embodiment, one or more individually accessible sub-parts within an individually accessible part of a data stream are indicated by a pointer that indicates the data stream position in bytes within the parameter set part of the individually accessible part. A data stream length parameter may indicate the run length of the individually accessible sub-part. The data stream length parameter may be encoded / decoded to / from the header part of the data stream, or encoded / decoded to / from the parameter set part of the individually accessible part. The data stream length parameter may be used to facilitate the extraction of each individually accessible sub-part for the purpose of packaging one or more individually accessible sub-parts into appropriate containers. According to embodiments, a device for decoding a data stream is configured to use a start code and / or pointer and / or data stream length parameter to access the data stream for one or more predetermined individually accessible sub-parts.

[0019] According to a sixth aspect of the present application, the inventors of the present application have noticed that the efficiency of using a neural network is increased when processing option parameters are encoded in a data stream in which a representation of the NN is encoded / decoded from the data stream. The data stream is structured into individually accessible parts, and for each of one or more predetermined individually accessible parts, the processing option parameters indicate one or more processing options that must be used, or can be optionally used, when using the neural network for inference. The processing option parameters may indicate one of several processing options that determine whether and how a client accesses an individually accessible part (P) and / or an individually accessible subpart (SP). For example, for each of P and / or SP, such as the parallel processing capability of each P or SP and / or the sample-level parallel processing capability of each P or SP and / or the channel-level parallel processing capability of each P or SP and / or the classification-level parallel processing capability of each P or SP and / or other processing options. The processing option parameters enable the client to make appropriate decisions, which in turn enables the highly efficient use of the NN.

[0020] According to a seventh aspect of the present application, the inventors of the present application have discovered that when the reconstruction rule for inverse quantization of the NN parameters depends on the NN part to which the NN parameters belong, the transmission / update of the neural network can be made highly efficient. The NN parameters representing the neural network are quantized into quantization indices and encoded into a data stream. The apparatus for decoding is configured to inverse-quantize the quantization indices to reconstruct the NN parameters, for example, using a reconstruction rule. The NN parameters are encoded into the data stream such that the NN parameters of different NN parts of the NN are different, and the data stream indicates a reconstruction rule for inverse-quantizing the NN parameters associated with each of the NN parts. The apparatus for decoding is configured to inverse-quantize the NN parameters in each of the NN parts using the reconstruction rule indicated by the data stream for each of the NN parts. The NN part includes, for example, one or more NN layers of the NN and / or a part of the NN layer in which the predetermined NN layer is subdivided.

[0021] According to an embodiment, the first reconstruction rule for inverse-quantizing the NN parameters associated with the first NN part is delta-encoded with respect to the second reconstruction rule for inverse-quantizing the NN parameters associated with the second NN part and encoded into the data stream. The first NN part may consist of a first NN layer, the second NN part may consist of a second layer, and the first NN layer is different from the second NN layer. Alternatively, the first NN part may consist of a first NN layer, and the second NN part may consist of a part of one of the first NN layers. In this alternative, the reconstruction rule associated with the NN parameters in a part of a predetermined NN layer, for example, the second reconstruction rule, is delta-encoded with respect to the reconstruction rule associated with the predetermined NN layer, for example, the first reconstruction rule. This special delta-encoding of the reconstruction rule may make it possible to use only a small number of bits to signal the reconstruction rule and may result in an efficient transmission / update of the neural network.

[0022] According to an eighth aspect of the present application, the inventors of the present application have found that the transmission / update of a neural network is greatly streamlined if the reconstruction rule for dequantizing NN parameters depends on the size of the quantization index associated with the NN parameters. The NN parameters representing the neural network are quantized into quantization indices and encoded into a data stream. The decoding device is configured, for example, to dequantize the quantization indices to reconstruct the NN parameters using a reconstruction rule. The data stream comprises a quantization step size parameter indicating the quantization step size and a set of parameters defining a quantization index-reconstruction level mapping to indicate the reconstruction rule for dequantizing the NN parameters. The reconstruction rule for NN parameters in a given NN portion is defined by the quantization step size for quantization indices within a given index interval and by a quantization index-reconstruction level mapping for quantization indices outside the given index interval. For each NN parameter, for example, each NN parameter corresponding to a quantization index within a given index interval is reconstructed by multiplying each quantization index by the quantization step size, and for example, each NN parameter corresponding to a quantization index outside a given index interval is reconstructed by mapping each quantization index onto a reconstruction level using a quantization index-reconstruction level mapping. The decoder may be configured to determine the quantization index-reconstruction level mapping based on a parameter set in the data stream. According to one embodiment, the parameter set defines the quantization index-reconstruction level mapping by pointing to one of the set of quantization index-reconstruction level mappings, and the set of quantization index-reconstruction level mappings may not be part of the data stream and may be stored, for example, on the encoder side and the decoder side.Defining a reconstruction rule based on the size of the quantization index enables signaling of the reconstruction rule with a small number of bits.

[0023] According to a ninth aspect of the present application, the inventors of the present application have noticed that if the representation of the NN is encoded with identification parameters in individually accessible parts of the encoded data stream / the identification parameters are decoded from the individually accessible parts of the encoded data stream, the transmission / update of the neural network becomes very efficient. The data stream is structured into individually accessible parts, and for each of one or more predetermined individually accessible parts, identification parameters for identifying each of the predetermined individually accessible parts are encoded in the data stream / decoded from the data stream. The identification parameter may indicate the version of a predetermined individually accessible part. This is particularly advantageous in scenarios such as distributed learning where many clients individually further train the NN and send back relative NN updates to a central entity. Yes. The identification parameter can be used to identify the NNs of individual clients by a versioning (version management) scheme. Thereby, the central entity can identify the NN for which the NN update is constructed. Additionally, or alternatively, the identification parameter may indicate whether a predetermined individually accessible part is associated with a baseline part of the NN or an advanced / enhanced / complete part of the NN. This is advantageous in use cases such as, for example, scalable NNs, and the baseline part of the NN can be executed, for example, to generate preliminary results before the complete or enhanced NN is executed to receive complete results. Further, transmission errors or involuntary changes in the parameter tensor that can be reconstructed based on the NN parameters representing the NN are easily recognizable using the identification parameter. The identification parameter can make the operation more error-robust when each pre-determined individually accessible part can be checked for consistency and verified based on the NN characteristics.

[0024] According to a tenth aspect of the present application, the inventors of the present application have found that transmitting / updating a neural network can be greatly improved by encoding / decoding different versions of the NN into / from a data stream using delta coding or a compensation scheme. The data stream is encoded in a layered manner so that different versions of the NN are encoded into the data stream. The data stream is structured into one or more individually accessible parts, each individually accessible part relating to a corresponding version of the NN. For example, the data stream is delta coded for a second version of the NN encoded in a second part, where a first version of the NN encoded in a first part is delta coded for a second version of the NN encoded in a second part. Additionally or alternatively, the data stream may have, for example, one or more compensated NN parts of a first version of the NN encoded in a first part, each of which is executed in addition to the execution of the corresponding NN part of the second version of the NN encoded in a second part in order to perform inference based on the first version of the NN, and the outputs of each compensated NN part and the corresponding NN part are summed. These encoded versions of the NN in the data stream allow a client, such as a decoder, to match its processing capabilities, or to perform inference on the first version, such as a baseline, before processing a second version, such as a more complex advanced NN. Furthermore, by applying / using delta coding and / or compensation schemes, different versions of the NN can be encoded into DS with fewer bits.

[0025] According to an eleventh aspect of the present application, the inventors of the present application have found that the utilization efficiency of a neural network is increased if supplemental data is encoded / decoded from a data stream in which a representation of a neural network is encoded into individually accessible portions of the data stream. The data stream is structured into individually accessible portions, and the data stream contains supplemental data to supplement the representation of the neural network for each of one or more predetermined individually accessible portions. This supplemental data is not usually necessary for decoding / reconstructing / inferring the neural network, but may be essential from an application standpoint. Therefore, it is advantageous to mark this supplemental data as irrelevant to decoding the neural network for the sole purpose of inference, so that clients that do not require the supplemental data, such as a decoder, can skip this portion of the data.

[0026] According to a twelfth aspect of the present application, the inventors of the present application have found that the use of neural networks becomes very efficient if hierarchical control data is encoded / decoded into a data stream in which the representation of the NN is encoded. The data stream consists of hierarchical control data structured into a sequence of control data portions, where the control data portions provide information about the NN with increasing detail along the sequence of control data portions. The control data is structured hierarchically. This approach is advantageous because the decoder may only require control data up to a certain level, thus allowing it to skip control data that provides further levels. In this way, different levels of control data may be required depending on the use case and environment, and the aforementioned method of presenting such control data enables efficient access to the control data needed for different use cases.

[0027] While several embodiments have been described in the context of apparatus, it is clear that these embodiments also represent a description of a method in which a block or apparatus corresponds to a method step or a feature of a method step. Embodiments relate to computer programs having program code for performing such methods when executed on a computer.

[0028] Embodiments of the present invention are the subject of the dependent claims. Preferred embodiments of the present invention are described below with reference to the drawings. The drawings are not necessarily to scale and instead focus on illustrating the general principles of the present invention. Various embodiments of the present invention are described below with reference to the following drawings. [Brief explanation of the drawing]

[0029] [Figure 1] Figure 1 shows an example of an encoding / decoding pipeline for encoding / decoding a neural network. [Figure 2] Figure 2 shows a neural network that can be encoded / decoded according to one embodiment. [Figure 3] Figure 3 shows the serialization of the parameter tensors of the layers of a neural network according to an embodiment. [Figure 4] Figure 4 shows the use of serialization parameters to illustrate how neural network parameters are serialized according to an embodiment. [Figure 5] Figure 5 shows an example of a single-output channel convolutional layer. [Figure 6] Figure 6 shows an example of a fully connected layer. [Figure 7a] Figure 7a shows a set of n coding orders in which neural network parameters can be coded, according to an embodiment. [Figure 7b] Figure 7b shows a set of n coding sequences in which neural network parameters can be coded, according to an embodiment. [Figure 7c] Figure 7c shows a set of n coding orders in which neural network parameters can be coded, according to an embodiment. [Figure 7d] Figure 7d shows a set of n coding sequences in which neural network parameters can be coded, according to an embodiment. [Figure 8] Figure 8 shows a context-adaptive arithmetic coding of individually accessible parts or subparts according to an embodiment. [Figure 9] Figure 9 shows how numerical calculation representation parameters are used according to an embodiment. [Figure 10] Figure 10 shows how to use the neural network layer type parameter to indicate the neural network layer type of the neural network layer of a neural network according to an embodiment. [Figure 11] Figure 11 shows a typical embodiment of a data stream having pointers that point to the beginning of individually accessible portions, according to the embodiment. [Figure 12] Figure 12 shows a detailed embodiment of a data stream having pointers that point to the beginning of individually accessible portions, according to the embodiment. [Figure 13] Figure 13 shows the use of start code and / or pointers and / or data stream length parameters to enable access to individually accessible sub-parts, according to an embodiment. [Figure 14a] Figure 14a shows a sub-layer access using a pointer according to an embodiment. [Figure 14b] Figure 14b shows a sub-tier access using a start code according to an embodiment. [Figure 15] Figure 15 shows an exemplary type of random access as a possible processing option for individually accessible parts, according to the embodiment. [Figure 16] Figure 16 shows how to use processing option parameters according to an embodiment. [Figure 17] Figure 17 shows how to use the neural network partial dependency reconstruction rule according to an embodiment. [Figure 18] Figure 18 shows the determination of a reconstruction rule based on a quantization index that represents quantized neural network parameters, according to an embodiment. [Figure 19] Figure 19 shows how to use the identification parameter according to an embodiment. [Figure 20] Figure 20 shows different versions of neural network encoding / decoding according to the embodiment. [Figure 21] Figure 21 shows two versions of delta coding of a neural network according to an embodiment, where the two versions differ in their weights and / or biases. [Figure 22] Figure 22 shows two versions of alternative delta coding for neural networks according to an embodiment, the two versions differing in the number of neurons or interconnections of neurons. [Figure 23] Figure 23 shows the coding of different versions of the neural network that use the compensating neural network portion according to the embodiment. [Figure 24a] Figure 24a shows one embodiment of a data stream having supplementary data according to an embodiment. [Figure 24b] Figure 24b shows an alternative embodiment of a data stream with supplementary data according to the embodiment. [Figure 25] Figure 25 shows one embodiment of a data stream having a sequence of control data portions. [Modes for carrying out the invention]

[0030] Furthermore, in the following explanation, elements with equivalent or equivalent functions are given equivalent or equivalent reference numbers, even if they are depicted in different diagrams.

[0031] The following description includes several details to provide a more overall description of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention can be carried out without these specific details. In other examples, well-known structures and devices are shown in block diagrams rather than in detail, so as not to obscure embodiments of the present invention. Furthermore, features of different embodiments described herein may be combined with each other unless otherwise specified.

[0032] The following description of embodiments of the present invention begins with a brief introduction and overview of the embodiments to explain their advantages and how they achieve these advantages.

[0033] In current work on coded representations of neural networks (NNs), such as those developed in ongoing MPEG activities for NN compression, it has been found beneficial to split a model bitstream representing a multi-layered parameter tensor into smaller sub-bitstreams, or layer bitstreams, containing coded representations of the parameter tensors of each individual layer. This can be generally useful when such model bitstreams need to be stored / loaded in a containerized context, or in application scenarios featuring parallel decoding / execution of NN layers.

[0034] The following describes various examples that can help achieve effective compression of neural networks (NNs) and / or improve access to the data representing the NN, thus leading to effective transmission / updating of the NN.

[0035] To facilitate understanding of the following embodiments of the present application, the description begins with the presentation of possible encoders and decoders that can be adapted thereto, into which subsequent schematic embodiments of the present application may be incorporated.

[0036] Figure 1 shows a simple sketch example of an encoding / decoding pipeline using DeepCABAC, illustrating the internal workings of such a compression scheme. First, between neurons 14, 20 and / or 18, for example, between preceding neurons 141-143 and intermediate neuron 201, The weights 32 of the connections 22 to 202, for example, connections 221-226, for example, weights 321-326 However, in this example, it is formed into a tensor shown as matrix 30 (step 1 in Figure 1). In step 1 in Figure 1, for example, the weights 32 associated with the first layer of the neural network 10 NN are formed in matrix 30. According to the embodiment shown in Figure 1, the columns of matrix 30 are associated with preceding neurons 141-143, and the rows of matrix 30 are associated with intermediate neurons 201, 202, but it is clear that the formed matrix can alternatively represent an inverted version of the illustrated matrix 30.

[0037] Next, each NN parameter, for example, weight 32, is encoded, e.g., quantized and entropy encoded, according to a specific scan order, e.g., row-major (left to right, top to bottom), using context-adaptive arithmetic coding 600, as shown in steps 2 and 3. It is also possible to use a different scan order, i.e., a coding order, as outlined in more detail below. Steps 2 and 3 are performed by the encoder 40, i.e., the device for coding. The decoder 50, i.e., the device for decoding, follows the same process in the reverse order of steps. That is, first, as shown in step 4, it decodes a list of integer representations of the encoded values, and then, as shown in step 5, it reshapes that list into its tensor representation 30'. Finally, as shown in step 6, the tensor 30' is loaded into the network architecture 10', i.e., the reconstructed NN. The reconstructed tensor 30' constitutes the reconstructed NN parameters, i.e., the decoded NN parameters 32'.

[0038] The NN10 shown in Figure 1 is merely a simple neural network with a small number of neurons: 14, 20, and 18. Neurons are also understood below as nodes, elements, model elements, or dimensions. Furthermore, reference numeral 10 indicates a machine learning (ML) predictor, in other words, a machine learning model such as a neural network.

[0039] The neural network will be described in more detail with reference to Figure 2. In particular, Figure 2 shows an ML predictor 10 comprising an input interface 12 having input nodes or elements 14 and an output interface 16 having output nodes or elements 18. The input nodes / elements 14 receive input data. In other words, input data is applied thereto. For example, they can receive a picture in which each element 14 is associated with a pixel of the picture. Alternatively, the input data applied on the elements 14 may be a signal such as an audio signal or a sensor signal, which is a one-dimensional signal. Further alternatively, the input data may represent a specific dataset, such as medical file data. The number of input elements 14 may be any number, depending, for example, on the type of input data. The number of output nodes 18 may be one, as shown in Figure 1, or more than one, as shown in Figure 2. Each output node or element 18 may be associated with a specific inference or prediction task. In particular, when the ML predictor 10 is applied to an input applied on the input interface 12 of the ML predictor 10, the ML predictor 10 generates at each output node 18 The activation, i.e., the activation value, is output at the output interface 16 as an inference or prediction result that can indicate an answer to a question about the input data, such as whether or not the input image contains certain objects such as cars, people, or faces, or to what extent such features are likely to exist.

[0040] In this regard, the inputs applied to the input interface may also be interpreted as activations, i.e., activations applied to each input node or element 14.

[0041] Between the input node 14 and the output node(s) 18, the ML predictor 10 includes further elements or nodes 20, which are connected to the preceding nodes via connections 22 to receive activations from these preceding nodes, and to the successor nodes via one or more further connections 24 to transfer the activations, i.e., activation values, of the node 20 to the successor nodes.

[0042] The preceding node may be another internal node 20 of the ML predictor 10, through which the intermediate node 20 illustrated in Figure 2 may be indirectly connected to the input node 14, or it may be directly connected to the input node 14, as shown in Figure 1. The succeeding node may be another intermediate node of the ML predictor 10, through which the intermediate node 20 illustrated in Figure 2 may be connected to an output interface or output node, or it may be directly connected to an output node 28, as shown in Figure 1.

[0043] The input node 14, output node 18, and internal node 20 of the ML predictor 10 may be associated with or assigned to a specific layer of the ML predictor 10, but the layered structure of the ML predictor 10 is arbitrary, and the ML predictor to which embodiments of the present application apply is not limited to such a layered network. With respect to the exemplary intermediate node 20 of the ML predictor 10, it contributes to the inference or prediction task of the ML predictor 10 by transferring activations, i.e., activation values, from the preceding node received from the input interface 12 via connection 22 to the successor node via connection 24 toward the output interface 16. In doing so, the node or element 20 calculates its activation, i.e., activation value, which is transferred toward the successor node via connection 24, based on the activation, i.e., activation value at the input node 22, and this calculation includes calculating a sum having a weighted sum, i.e., addition, for each connection 22 which is the product of the input received from each preceding node, i.e., its activation, and the weight associated with the connection 22 connecting each preceding node and the intermediate node 20. Alternatively or more generally, activation x is performed via a node or element i, 20 through connection 24, and a mapping function m ij Note that (x) is forwarded to the successor node j. Therefore, like connection 24, each connection 22 is also associated with some kind of weight, or alternatively, the mapping function m ijThe results may be obtained. Optionally, additional parameters may be involved in the calculation of the activation that node 20 outputs toward a particular successor node. To determine the relevance score for a portion of the ML predictor 10, an activation that occurs at the output node 18 when a certain prediction or inference task is completed for a certain input at the input interface 12, or a predefined or interesting output activation of interest, may be used. This activation at each output node 18 is used as a starting point for determining the relevance score, and the relevance is backpropagated toward the input interface 12. In particular, at each node of the ML predictor 10, such as node 20, the relevance score is distributed toward the preceding node, such as via connection 22 in the case of node 20, in a manner proportional to the aforementioned product associated with each preceding node, and contributes to the activation of the current node, which is the backpropagated activation such as node 20, via a weighted sum. In other words, the correlation rate backpropagated from a node such as node 20 to a certain preceding node is calculated by multiplying the correlation of that node by a coefficient corresponding to the ratio of the activation received from that preceding node and the weight to which that activation contributed to the sum of each node mentioned above, and the activation of the preceding node and these activations This can be calculated by dividing by a value corresponding to the sum of all products of the weights that contributed to the weighted sum of the current node whose error should be backpropagated.

[0044] In the method described above, the relevance score for a part of the ML predictor 10 is determined based on the activation of these parts, for example, such that they appear in one or more inferences performed by the ML predictor. The “parts” on which such relevance scores are determined may be nodes or elements of the predictor 10, as described above, and again, it should be noted that the ML predictor 10 is not limited to any layered ML network, for example, element 20 may be any calculation of intermediate values, for example, that is calculated during inference or prediction performed by the predictor 10. For example, in the method discussed above, the relevance score of an element or node 20 is calculated by aggregating or summing the inbound relevance messages that this node or element 20 receives from its successor nodes / elements, which then, in turn, distribute their relevance scores with respect to node 20 in the manner outlined above.

[0045] The ML predictor 10, i.e., the neural network (NN), as described in Figure 2, is encoded into a data stream 45 using the encoder 40 described in Figure 1, and then reconstructed / decoded from the data stream 45 using the decoder 50 described in Figure 1.

[0046] The features and / or functions described below can be implemented using the compression scheme described with respect to Figure 1, and are related to the neural networks described with respect to Figures 1 and 2.

[0047] 1. Serialization of a parameter tensor There are applications where bitstream sub-layer processing is effective. For example, there are neural networks (NNs) that adapt to the available computing power of the client, where the layers consist of independent subsets, such as separately trained baseline and advanced portions, and the client can decide whether to run only the baseline layer subset or additionally the advanced layer subset (Tao, 2018). Another example is a NN featuring data channel-specific operations, such as layers in an image processing NN that can perform operations separately and in parallel for each color channel (Chollet, 2016).

[0048] For the purposes described above, referring to Figure 3, serialization 1001 or 1002 of the layer parameter tensor 30 requires bit strings 421 or 422 that can be easily divided into a contiguous subset 431 to 433, or 441 and 442, which are meaningful from an application perspective, before entropy coding, for example. This is per channel 1001 or All NN parameters for each of the 1002 samples, for example, grouping by weight 32, or This may include grouping of neurons into baseline versus advanced portions. Such bit strings can then be entropy-encoded to form sublayer bitstreams with functional relationships.

[0049] As shown in Figure 4, the serialization parameter 102 can be encoded into / decoded from the data stream 45. The serialization parameter can indicate how the NN parameters 32 are grouped before or during encoding. The serialization parameter 102 shows how the NN parameters 32 of the parameter tensor 30 are serialized into a bitstream to enable encoding of the NN parameters into the data stream 45.

[0050] In one embodiment, serialization information, i.e., serialization parameter 102, is a bitstream That is, the parameter set portion 110 of the data stream 45 is shown in layers; see, for example, Figures 12, 14a, 14b, or 24b.

[0051] Another embodiment involves a parameter tensor 30 (see coding order 1061 in Figures 1 and 7). The dimensions 341 and 342 of ) are signaled as serialization parameters 102. This information may be useful, for example, in an exemplary image processing NN where there is a clear relationship between the entries of the parameter matrix, i.e., the parameter tensor 30, i.e., the weights 32, and the samples 1002 and the color channels 1001, as shown in Figure 3, when the decoded list of parameters should be grouped / organized in their respective ways, for example in memory, to enable efficient execution. Figure 3 shows two different serialization modes 1001 and 1 This is an illustrative diagram of 002 and the resulting sublayers 43 and 44.

[0052] In a further embodiment, as shown in Figure 4, the bitstream, or data stream 45, specifies the order in which the encoder 40 scanned the NN parameters 32, e.g., layers, neurons, and tensors, so that the decoder 50 can reconstruct the NN parameters 32 accordingly during decoding. See Figure 1 for a description of the encoder 40 and decoder 50. That is, different scanning orders 301 and 302 of the NN parameters 32 can be applied in different application scenarios.

[0053] For example, encoding parameters along different dimensions may benefit the resulting compression performance because the entropy encoder might be able to better capture the dependencies between them. In another example, it may be desirable to group parameters according to application-specific criteria, namely which parts of the input data they relate to or whether they can be performed jointly, so that they are decoded / input in parallel. A further example is encoding parameters according to a GEMM (General Matrix Matrix) product scan order that supports efficient memory allocation of decoded parameters when performing a dot product (Andrew Kerr, 2017).

[0054] A further example is, for instance, achieving energy compression of the NN parameters 32 to be encoded, and then processing / serializing / encoding the resulting reordered data according to the resulting order 104, for example, the encoding order 1064 shown in Figure 7. This relates to the selection and sorting of the data on the encoder side. The sorting may therefore also involve rearranging the NN parameters 32 so that they steadily increase or decrease in the same way along the coding order 104.

[0055] Figure 5 shows an example of a single-output channel convolutional layer, for example, for image and / or video analysis applications. A color image has multiple channels, typically one for each color channel such as red, green, and blue. From a data perspective, this means that one image provided as input to the model is actually three images.

[0056] Tensor 30a is applied to the input data 12 and can scan the input like a window with a constant step size. Tensor 30a is understood as a filter. Tensor 30a moves through the input data 12 from left to right, jumping to the next lower row after passing through each. Optional so-called padding determines how Tensor 30a should behave when it hits an edge of the input matrix. For each point in its field of view, Tensor 30a has NN parameters 32, such as fixed weights, and calculates a result matrix from, for example, the pixel value in the current field of view and these weights. The size of this result matrix is ​​determined by the size of Tensor 30a (kernel size), padding, and especially the step size. It depends on the input image. The input image has 3 channels (e.g., depth 3), and the tensor 30a applied to that image also has 3 channels (e.g., depth 3). Regardless of the depth of the input 12 and the depth of the tensor 30a, the tensor 30a is applied to the input 12 using a dot product operation that yields a single value.

[0057] By default, DeepCABAC converts any given tensor 30a into the form of its respective matrix 30b, sorted by row length 1041, as shown in Figure 5, that is, Next, the NN parameters 32 are encoded into the data stream 45 from left to right and top to bottom. However, as explained with respect to Figure 7, other encoding orders 104 / 106 may be advantageous for achieving higher compression.

[0058] Figure 6 shows an example of a fully connected layer. A fully connected layer, or dense layer, is a typical neural network structure where all neurons are connected to all inputs 12, i.e., preceding nodes, and all outputs 16', i.e., succeeding nodes. Tensor 30 represents the corresponding NN layer, and tensor 30 contains NN parameters 32. The NN parameters 32 are encoded into a data stream according to coding order 104. As illustrated with respect to Figure 7, a particular coding order 104 / 106 would be advantageous for achieving high compression.

[0059] Now, we return to Figure 4 to allow for a general explanation of the serialization of NN parameters 32. The concepts described in relation to Figure 4 are applicable to both single-output channel convolutional layers, which refer to Figure 5, and fully connected layers, which refer to Figure 6.

[0060] As shown in Figure 4, Embodiment A1 of the present invention relates to a data stream 45 (DS) on which a representation of a neural network (NN) is encoded. The data stream includes serialization parameters 102 that indicate an encoding order 104 in which NN parameters 32 defining the neuronal interconnections of the neural network are encoded in the data stream 45.

[0061] According to Embodiment ZA1, the device for encoding a representation of a neural network into a data stream 45 is configured to provide the data stream 45 with serialization parameters 102 indicating an encoding order 104 in which NN parameters 32 defining the neuronal interconnections of the neural network are encoded into the data stream 45.

[0062] According to Embodiment XA1, the device for decoding a representation of a neural network from DS45 is configured to decode from the data stream 45 a serialization parameter 102 indicating an encoding order 104 in which NN parameters 32 defining the neuronal interconnections of the neural network are encoded in the data stream 45, and to assign the NN parameters 32 decoded in series from DS45 using the encoding order 104 to the neuronal interconnections.

[0063] Figure 4 shows different representations of an NN layer having NN parameters 32 associated with the NN layer. According to the embodiment, a 2D tensor 301, i.e., a matrix, or 3D tensor 301. A 302-dimensional tensor can represent the corresponding NN layer.

[0064] In the following, different features and / or functionalities are described in the context of data stream 45, but features and / or functionalities may also be features and / or functionalities of the apparatus according to embodiment ZA1 or the apparatus according to embodiment XA1.

[0065] According to Embodiment A2, in the DS45 of the previous Embodiment A1, the NN parameter 32 is coded using context-adaptive arithmetic coding 600, as shown in Figures 1 and 8, for example. Therefore, the device according to embodiment ZA1 can be configured to encode the NN parameters 32 using context-adaptive arithmetic coding 600, and the device according to embodiment XA1 can be configured to decode the NN parameters 32 using context-adaptive arithmetic decoding.

[0066] According to Embodiment A3, in Embodiment A1 or A2, the data stream 45 of the DS45 is structured into one or more individually accessible parts 200, as shown in Figure 8 or one of the following figures, where each individually accessible part 200 represents a corresponding NN layer 210 of a neural network, and the serialization parameter 102 indicates an encoding order 104 in which NN parameters 32 defining the neuronal interconnections of the neural network within a given NN layer 210 are encoded in the data stream 45.

[0067] According to Embodiment A4, in any of the above embodiments A1 to A3, the serialization parameter 102 in the DS45 is an n-ary parameter that indicates coding order 104 from a set of n coding orders 108, as shown in Figure 7, for example.

[0068] According to Embodiment A4a, the set of n coding sequences 108 in the DS45 of Embodiment A4 includes the following: A predetermined coding instruction 104 has a first predetermined coding sequence 1061 that is different in the order in which it traverses the dimensions of the tensor 30 describing a predetermined NN layer of the NN, for example, the x dimension, the y dimension and / or the z dimension; and / or For scalable coding of NN, a second predetermined coding order 1062 in which a predetermined coding order 104 scans a predetermined NN layer of the NN a different number of times 107; and / or, A third predetermined coding order 1063 in which a predetermined coding order 104 scans the NN layer 210 of the NN in a different order; and / or, A fourth predetermined coding order 1064 in which the order in which the neurons 20 of the NN layer of the NN are scanned is different.

[0069] The first predetermined coding order 1061 is, for example, a tensor when coding the NN parameter 32. The way each of the 30 individual dimensions is scanned differs from one another. For example, coding order 1041 is different from coding order 1042. That is, coding order 1041 is row length order, that is, In other words, the tensor 30 is scanned row by row from left to right and top to bottom, and the predetermined coding order 1042 is column length order, that is, the tensor 30 is scanned column by column from top to bottom and left to right. Similarly, the first predetermined coding order 1061 is a predetermined coding order 104 that is a 3D type. The order in which the dimensions of nsor 30 are scanned can differ.

[0070] The second predetermined coding order 1062 can be represented, for example, by a tensor / matrix 30. The frequency at which the NN layers are scanned differs. For example, an NN layer can be scanned twice through a predetermined coding sequence 104, thereby encoding the baseline and advanced portions of the NN layer into / decoding them from the data stream 45. The number of times the NN layer is scanned through the predetermined coding sequence 107 defines the number of versions of the NN layer encoded into the data stream. Therefore, in the case of a serialization parameter 102 indicating a coding sequence that scans the NN layer at least twice, the decoder is configured to determine, based on its processing capacity, which versions of the NN layer it can decode and to decode the NN parameter 32 corresponding to the selected NN layer version.

[0071] The third predetermined coding order 1063 relates to different NN layers 2101 and 2102 of NN10. This defines whether the connected NN parameters are encoded into the data stream 45 using a predetermined encoding order that is different from or the same as one or more other NN layers 210 of NN10.

[0072] The fourth predetermined coding order 1064 is a tensor / matrix representing the corresponding NN layer. 30 is the NN parameter 321 in the upper left and the NN parameter 32 in the lower right. 12 Running diagonally in a zigzag pattern It includes a predetermined coding sequence 1043 to be examined.

[0073] According to Embodiment A4a, in any of the preceding Embodiments A1 to A4a, the serialization parameter 102 of DS45 indicates a permutation used by the coding order 104 to permutate the neurons of the NN layer relative to the default order. In other words, the serialization parameter 102 indicates a permutation, and when using the permutation, the coding order 104 permutates the neurons of the NN layer relative to the default order. For a fourth predetermined coding order 1064, as shown in Figure 7, the data The row length order shown for Stream 450 represents the default order. The data stream 45 contains the NN parameters encoded within it using permutations relative to the default order.

[0074] According to Embodiment A4b, among the DS45 of Embodiment A4a, the permutation orders the neurons of the NN layer 210 such that the NN parameter 32 increases monotonically along the coding order 104, or decreases monotonically along the coding order 104.

[0075] According to Embodiment A4c, among the DS45 of Embodiment A4a, the permutation is such that the neurons of the NN layer 210 are permutated among a predetermined coding sequence 104 that can be signaled by the serialization parameter 102, such that the bit rate for encoding the NN parameter 32 into the data stream 45 is the lowest for the permutation indicated by the serialization parameter 102.

[0076] According to Embodiment A5, in any of the previous embodiments A1 to A4c, the NN parameters 32 include weights and biases.

[0077] According to Embodiment A6, in any of the earlier embodiments A1 to A5, the data stream 45 of the DS45 is structured into individually accessible subparts 43 / 44, each subpart 43 / 44 representing a corresponding NN portion of the neural network 10, for example, a portion of the NN layer 210, so that each subpart 43 / 44 is fully scanned by the coding order 104 before subsequent subparts 43 / 44 are scanned by the coding order 104. Rows, columns, or channels of the tensor 30 representing the NN layer may be encoded into individually accessible subparts 43 / 44. Different individually accessible subparts 43 / 44 associated with the same NN layer include different neurons 14 / 18 / 20 or neuron interconnections 22 / 24 associated with the same NN layer. The individually accessible subparts 43 / 44 represent rows, columns, or channels of the tensor 30. Individually accessible subparts 43 / 44 are shown, for example, in Figure 3. Alternatively, as shown in Figures 21 to 23, individually accessible sub-parts 43 / 44 represent different versions of the NN layer, such as the baseline and advanced parts of the NN layer.

[0078] According to Embodiment A7, in any of the DS45s of Embodiments A3 and A6, the NN parameters 32 are coded into the DS45 using context-adaptive arithmetic coding 600 and context initialization at the beginning 202 of any individually accessible portion 200 or sub-part 43 / 44. See, for example, Figure 8.

[0079] According to Embodiment A8, in any of the DS45s of Embodiments A3 and A6, the data stream 45 includes a start code 242 in which each individually accessible portion 200 or sub-portion 240 begins, and / or a pointer 220 / 244 pointing to the beginning of each individually accessible portion 200 or sub-portion 240, and / or a pointer data stream length, i.e., when parsing the DS45 as shown in Figures 11 to 14. This parameter indicates the data stream length 246 of each individually accessible portion 200 or sub-portion 240, for skipping each individually accessible portion 200 or sub-portion 240.

[0080] Another embodiment specifies the bit size and numerical representation of the decoded parameter 32' in the bitstream, i.e., the data stream 45. For example, an embodiment may specify that the decoded parameter 32' can be represented in an 8-bit signed fixed-point format. This specification is very useful, for example, in applications where the activation value can also be represented in an 8-bit fixed-point representation, for example, as it allows for more efficient inference using fixed-point arithmetic.

[0081] Embodiment A9 further includes a numerical computation representation parameter 120 that indicates the numerical representation and bit size of the NN parameters 32 when the NN is used for inference, as in any of the previous embodiments A1 to A8. See, for example, Figure 9.

[0082] Figure 9 shows an embodiment B1 of a data stream 45 in which a representation of a neural network is encoded, and the data stream 45 includes numerical computation representation parameters 120 that indicate the numerical representations to be represented when the NN parameters 32 of the NN encoded in DS45 are used for inference, such as floating-point, fixed-point, and bit sizes.

[0083] The corresponding embodiment ZB1 relates to a device for encoding a representation of a neural network into a DS45, the device being configured such that the NN parameters 32 of the NN encoded into the DS45 provide a data stream 45 with numerical computation representation parameters 120 indicating the numerical representation to be represented, such as floating-point, fixed-point, and bit size, when the NN is used for inference.

[0084] A corresponding embodiment XB1 relates to a device for decoding a neural network representation from DS45. The device is configured to decode numerical computation representation parameters 120 from the data stream 45, which indicate the floating-point, fixed-point, and bit sizes of the numerical representation, for example, the NN parameters 32 of the NN encoded in DS45, when the NN is used for inference, and is configured to optionally use the numerical representation and bit size to represent the NN parameters 32 decoded from DS45.

[0085] In the following, different features and / or functionalities are described in the context of data stream 45, but features and / or functionalities may also be features and / or functionalities of the apparatus according to embodiment ZB1 or the apparatus according to embodiment XB1, in the same or similar manner.

[0086] Further embodiments signal the parameter types within a layer. In most cases, a layer consists of two types of parameters: weights and biases. Distinguishing between these two types of parameters can be useful before decoding, for example, when different types of dependencies are used during encoding, or when parallel decoding is desired.

[0087] According to Embodiment A10, in any of the DS45s of the previous embodiments A1 to B1, the data stream 45 is structured into individually accessible sub-parts 43 / 44. Each sub-part 43 / 44 represents the corresponding NN part of the neural network, for example, a part of the NN layer, and each sub-part 43 / 44 is fully scanned by the coding sequence 104 before subsequent sub-parts 43 / 44 are scanned by the coding sequence 104, and the data... The trim 45 includes a type parameter that indicates the parameter type of the NN parameter 32 encoded in a given subpart for that given subpart.

[0088] According to Embodiment A10a, the type parameter in the DS of Embodiment A10 identifies at least the NN weights and the NN bias.

[0089] Finally, in further embodiments, the type of layer 210 containing the NN parameters 32 is signaled, for example, convolutional or fully connected. This information may be useful, for example, to understand the meaning of the dimensions of the parameter tensor 30. For example, the weight parameters of a 2d convolutional layer may be expressed as a 4d tensor 30, where the first dimension is the number of filters, the second dimension is the number of channels, and the remainder specifies the 2d spatial dimension of the filters. Furthermore, different layers 210 may be treated differently during encoding (for example, by using different sets or modes of the context model) to better capture dependencies in the data and lead to higher encoding efficiency, and this information may be important for the decoder to know before decoding.

[0090] According to Embodiment A11, in any of the DS45s of the earlier embodiments A1 to A10a, the data stream 45 is structured into one or more individually accessible parts 200, each part 200 representing a corresponding NN layer 210 of the neural network 10, and the data stream 45 further includes an NN layer type parameter 130 indicating the NN layer type of a given NN layer for a given NN layer. See, for example, Figure 10.

[0091] Figure 10 shows an embodiment C1 of a data stream 45 in which a representation of a neural network is encoded. The data stream 45 is structured into one or more individually accessible parts 200, each part representing a corresponding NN layer 210 of the neural network, and the data stream 45 further includes an NN layer type parameter 130 indicating the NN layer type of a given NN layer of the NN for a given NN layer.

[0092] A corresponding embodiment ZC1 relates to a device for encoding a representation of a neural network into a DS45 such that the data stream 45 is structured into one or more individually accessible parts 200. Each part 200 represents a corresponding NN layer 210 of the neural network, and the device is configured to provide the data stream 45 with an NN layer type parameter 130 indicating the NN layer type of the given NN layer 210 of the NN for a given NN layer 210.

[0093] A corresponding embodiment XC1 relates to a device for decoding a representation of a neural network from DS45. The data stream 45 is structured into one or more individually accessible parts 200, each part 200 representing a corresponding NN layer 210 of a neural network. The device is configured to decode from the data stream 45, for a given NN layer 210, an NN layer type parameter indicating the NN layer type of a given NN layer 210 of the NN.

[0094] According to Embodiment A12, in any of the DS45s of Embodiments A11 and C1, the NN layer type parameter 130 is at least fully connected, NN layer 2101 reference, and Convolutional layer type, NN layer 210 N Reference, identify. Therefore, according to Embodiment ZC1 The device can encode the NN layer type parameter 130 to identify two layer types, and the device according to embodiment XB1 can decode the NN layer type parameter 130 to identify two layer types.

[0095] 2-bit stream random access 2.1 Layer Bitstream Random Access Accessing subsets of a bitstream is crucial for many applications, such as parallelizing layer processing or packaging bitstreams in various container formats. One state-of-the-art method for enabling such access is, for example, to break the coding dependencies after the parameter tensor 30 of each layer 210 and insert start codes into the model bitstream, i.e., the data stream 45, before each layer bitstream, e.g., individually accessible portions 200. In particular, inserting start codes within the model bitstream is not a good way to isolate layer bitstreams, as start code detection would potentially require parsing the entire model bitstream from the beginning across a very large number of start codes.

[0096] This aspect of the present invention relates to structuring the encoded model bitstream of the parameter tensor 30 in a state-of-the-art and superior manner, and to further techniques that enable easier, faster, and more appropriate access to bitstream portions, such as layer bitstreams, to facilitate applications requiring parallel or partial decoding and execution of neural networks.

[0097] In one embodiment of the present invention, the model bitstream, i.e., individual layer bitstreams within the data stream 45, for example, individually accessible portions 200, are indicated through the bitstream position in bytes or offsets (e.g., byte offsets relative to the beginning of the encoding unit) within the parameter set / header portion 47 of the bitstream having a model range. Figures 11 and 12 illustrate this embodiment. Figure 12 shows layer access via the bitstream position or offset indicated by the pointer 220. Furthermore, each individually accessible portion 200 can optionally constitute a layer parameter set 110, to which one or more of the aforementioned parameters can be encoded and decoded.

[0098] According to Embodiment A13, the data stream 45 of any of the DS45 in the previous embodiments A1 to A12 is structured into individually accessible portions 200. Each portion 200 represents a corresponding NN portion, for example, one or more NN layers or a portion of an NN layer of a neural network. The data stream 45 includes, for each of the one or more predetermined individually accessible portions 200, a pointer 220 that points to, for example, the beginning of each individually accessible portion 200. Refer to Figure 11 or Figure 12 when the individually accessible portion represents a corresponding NN layer. Refer to Figures 13 to 15 when the individually accessible portion represents a portion of a predetermined NN layer, for example, an individually accessible sub-portion 240. Hereafter, the pointer 220 may also be denoted by reference numeral 244.

[0099] For each NN layer, the individually accessible portion 200 associated with each NN layer represents the corresponding NN portion of that NN layer. In this case, as described herein and below, such individually accessible portion 200 is understood as an individually accessible sub-part 240.

[0100] Figure 11 shows a more general embodiment D1, in which a data stream 45 on which a representation of a neural network is encoded is shown, and the data stream 45 is structured into individually accessible portions 200. Each individually accessible portion 200 represents a corresponding NN portion of the neural network, for example, one or more NN layers or portions of NN layers. For each of the one or more predetermined individually accessible portions 200, the data stream 45 includes a pointer 220 that points to the beginning of each predetermined individually accessible portion 200.

[0101] According to the embodiment, the pointer 220 is the beginning of the first individually accessible portion 2001. Indicates the offset relative to the mari. The first individually accessible portion 2001 points to the first Pointer 2201 indicates no offset. Therefore, the first pointer 2201 can be omitted. Alternatively, pointer 220 indicates an offset relative to the end of the parameter set to which pointer 220 is encoded, for example.

[0102] A corresponding embodiment ZD1 relates to a device for encoding a representation of a neural network into a DS45 such that the data stream 45 is structured into one or more individually accessible portions 200. Each portion 200 represents a corresponding NN portion of the neural network, for example, one or more NN layers or a portion of NN layers. The device is configured to provide the data stream 45 with a pointer 220 that points to the beginning of each of the one or more predetermined individually accessible portions 200.

[0103] A corresponding embodiment XD1 relates to a device for decoding a representation of a neural network from DS45. The data stream 45 is structured into one or more individually accessible portions 200, each portion 200 representing a corresponding NN portion of a neural network, for example, one or more NN layers or a portion of NN layers. For each of the one or more predetermined individually accessible portions 200, the device decodes from the data stream 45 a pointer 220 pointing to the beginning of each predetermined individually accessible portion 200, and is configured to use one or more of the pointers 220 to access, for example, DS45.

[0104] According to Embodiment A14, in either Embodiment A13 or D1, the individually accessible portion 200 of the DS45 represents the following: The corresponding NN layer 210 of the neural network, or This is the neural network portion of the NN layer 210 of the NN. For example, please refer to Figure 3 or Figures 21-23.

[0105] 2.2 Sublayer Bitstream Random Access As discussed in Section 1, there are applications that rely on grouping the parameter tensor 30 within layer 210 in a specific configurable way, since it is beneficial to decode / process / infer partially or in parallel. Thus, sub-layer unit access to the layer bitstream, e.g., individually accessible portions 200, can be useful for accessing desired data in parallel or for omitting unnecessary data portions.

[0106] In one embodiment, the coding dependencies within the layer bitstream are reset with sublayer precision, i.e., the DeepCABAC probabilistic state is reset.

[0107] In another embodiment of the present invention, individual sub-layer bitstreams within a layer bitstream, i.e., an individually accessible portion 200, i.e., an individually accessible sub-portion 240, are indicated together with the range of a layer or model through a byte-level bitstream position, e.g., pointer 244, or offset, e.g., pointer 244, within the parameter set portion 110 of the bitstream, i.e., the data stream 45. Figures 13, 14a, and 15 illustrate embodiments. Figure 14a shows sub-layer access by relative bitstream position or offset, i.e., access to an individually accessible sub-portion 240. Furthermore, for example, an individually accessible portion 200 can also be accessed by a layer-level pointer 220. The layer-level pointer 220 is encoded, for example, in the model parameter set 47 of DS45, i.e., the header. The pointer 220 points to an individually accessible portion 200 that represents the corresponding NN portion, including the NN layers of the NN. The sub-layer-level pointer 244 points to the corresponding portion, including the NN layers of the NN. The layer parameter set 110 is encoded into an individually accessible portion 200 that represents the NN portion. Pointer 244 points to the beginning of an individually accessible sub-portion 240 that represents the corresponding NN portion, which includes a portion of the NN layers of the NN.

[0108] According to one embodiment, the layer-level pointer 220 indicates an offset relative to the beginning of the first individually accessible portion 2001. The sub-layer-level pointer 244 indicates a This indicates the offset of the individually accessible portion 200 relative to the beginning of the first individually accessible sub-portion 240.

[0109] According to one embodiment, pointers 220 / 244 indicate a byte offset relative to a set unit containing multiple units. Pointers 220 / 244 indicate a byte offset from the beginning of the set unit to the beginning of a unit within the payload of the set unit.

[0110] In another embodiment of the present invention, individual sub-layer bitstreams, i.e., individually accessible sub-parts 240, within a layer bitstream, i.e., individually accessible sub-parts 200, are indicated via a discoverable start code 242 within a bitstream, i.e., a data stream 45, which would be sufficient since the amount of data per layer is typically less than if the layer were discovered by the start code 242 within the entire model bitstream, i.e., the data stream 45. Figures 13 and 14b illustrate this embodiment. Figure 14b illustrates the use of the start code 242 at the sub-layer level, i.e., for each individually accessible sub-part 240, and the bitstream location, i.e., pointer 220, at the layer level, i.e., for each individually accessible sub-part 200.

[0111] In another embodiment, the run length of the (sub)layer bitstream portion, the individually accessible sub-portion 240, i.e., the data stream length 246, is indicated in the parameter set / header portion 47 of the bitstream 45 or the parameter set portion 110 of the individually accessible portion 200, in order to facilitate the extraction of the portion, i.e., the individually accessible sub-portion 240, for the purpose of packaging it into a suitable container. As shown in Figure 13, the data stream length 246 of the individually accessible sub-portion 240 is indicated by the data stream length parameter.

[0112] Figure 13 shows Embodiment E1 of the data stream 45 in which the representation of the neural network is encoded. The data stream 45 is structured into one or more individually accessible portions 200, each individually accessible portion 200 representing a corresponding NN layer of the neural network. Within a given portion, for example within an individually accessible portion 200, the data stream 45 is further structured into individually accessible sub-parts 240, each sub-part 240 representing a corresponding NN portion of each NN layer of the neural network. For each of the one or more predetermined individually accessible sub-parts 240, the data stream 45 includes the following: A start code 242 in which each predetermined individually accessible sub-part 240 begins, and / or A pointer 244 that points to the beginning of each predetermined individually accessible sub-part 240, and / or A data stream length parameter indicating the data stream length 246 of each predetermined individually accessible sub-part 240, for skipping each predetermined individually accessible sub-part 240 when analyzing DS45.

[0113] The individually accessible sub-part 240 described herein has the same or similar features and / or functionality as those described with respect to the individually accessible sub-parts 43 / 44. It is possible.

[0114] All individually accessible sub-parts 240 within the same predetermined portion have the same data stream length 246, so that a data stream length parameter indicates a single data stream length 246, and that data stream length 246 is applicable to each individually accessible sub-part 240 within the same predetermined portion. The data stream length parameter indicates the data stream length 246 of all individually accessible sub-parts 240 of the entire data stream 45, or the data stream length parameter indicates the data stream length 246 of all individually accessible sub-parts 240 of each individually accessible portion 200. One or more data stream length parameters are encoded in the header portion 47 of the data stream 45, or in the parameter set portion 110 of each individually accessible portion 200.

[0115] The corresponding embodiment ZE1 relates to a device for encoding a representation of a neural network into DS45. The data stream 45 is structured into one or more individually accessible portions 200. Each individually accessible portion 200 represents a corresponding NN layer of the neural network. The data stream 45 is further structured into individually accessible sub-parts 240 within a predetermined portion, for example, an individually accessible portion 200, where each sub-part 240 represents a corresponding NN portion of each NN layer of the neural network. The device is configured to supply the data stream 45 with the following for each of the one or more predetermined individually accessible sub-parts 240: A start code 242 in which each predetermined individually accessible sub-part 240 begins, and / or A pointer 244 that points to the beginning of each predetermined individually accessible sub-part 240, and / or A data stream length parameter indicating the data stream length 246 of each predetermined individually accessible sub-part 240, for skipping each predetermined individually accessible sub-part 240 when analyzing DS45.

[0116] Another corresponding embodiment, XE1, relates to a device for decoding a representation of a neural network from DS45. The data stream 45 is structured into one or more individually accessible portions 200. Each individually accessible portion 200 represents a corresponding NN layer of the neural network. The data stream 45 is further structured into individually accessible sub-parts 240 within a given portion, for example, an individually accessible portion 200, where each sub-part 240 represents a corresponding NN portion of each NN layer of the neural network. The device is configured to decode from the data stream 45, for each of the one or more predetermined individually accessible sub-parts 240, the following: A start code 242 in which each predetermined individually accessible sub-part 240 begins, and / or A pointer 244 that points to the beginning of each predetermined individually accessible sub-part 240, and / or A data stream length parameter indicating the data stream length 246 of each predetermined individually accessible sub-part 240, for skipping each predetermined individually accessible sub-part 240 when analyzing DS45. Then, for example, one or more predetermined individually accessible sub-parts 240 are configured to use this information, such as a start code 242, a pointer 244, and / or a data stream length parameter, to access the DS45.

[0117] According to Embodiment E2, in the DS45 of Embodiment E1, the data stream 45 encodes the representation of the neural network using context-adaptive arithmetic coding and context initialization at the starting point of each individually accessible portion 200 and each individually accessible sub-portion 240. See, for example, Figure 8.

[0118] According to Embodiment E3, the data stream 45 of Embodiment E1 or Embodiment E2 is by any other embodiment described herein. It is also clear that the apparatus of Embodiments ZE1 and XE1 can be completed by any other features and / or functions described herein.

[0119] 2.3 Types of Bitstream Random Access Depending on the selected serialization type, for example, the type of (sub)layer 240 resulting from serialization types 1001 and 1002 shown in Figure 3, various processing options are also available to determine whether and how the client will access the (sub)layer bitstream 240. For example, if the selected serialization 1001 is such that the sublayer 240 is image color channel specific, If this enables parallelization of the data channel for decoding / inference, this should be shown in bitstream 45 to the client. Another example is the derivation of preliminary results from a baseline NN subset that can be decoded / inferred independently of an advanced NN subset of a particular layer / model, as described with respect to Figures 20 to 23.

[0120] In one embodiment, the parameter set / header 47 within the bitstream 45, which has a range of one or more layers across the entire model, indicates the type of (sub)layer random access to enable the client to make appropriate decisions. Figure 15 shows two exemplary types 2521 and 2522 of random access determined by serialization. The exemplary types 2521 and 2522 of random access represent possible processing options for individually accessible portions 200 that represent the corresponding NN layers. The first processing option 2521 indicates data channel-level access to NN parameters within the individually accessible portion 200, and the second processing option 2522 indicates individually accessible This demonstrates sample-level access to NN parameters within section 2002.

[0121] Figure 16 shows a typical embodiment F1 of a data stream 45 in which a representation of a neural network is encoded. The data stream 45 is structured into individually accessible portions 200, each of which represents a corresponding NN portion, for example, containing one or more NN layers or a portion of the NN layers of a neural network. For each of the one or more predetermined individually accessible portions 200, the data stream 45 includes processing option parameters 250 indicating one or more processing options 252 that must be used or optionally used when using the NN for inference.

[0122] The corresponding embodiment ZF1 relates to a device for encoding a representation of a neural network into a DS45. The data stream 45 is structured into individually accessible portions 200. Each individually accessible portion 200 represents a corresponding NN portion, for example, containing one or more NN layers or containing a portion of the NN layers of a neural network. The device is configured to supply the data stream 45 with processing option parameters 250 indicating one or more processing options 252 that must be used when using an NN for inference, or that can be optionally used, for each of the one or more predetermined individually accessible portions 200.

[0123] Another corresponding embodiment XF1 decodes the neural network representation from DS45 The present invention relates to a device for performing the following. A data stream 45 is structured into individually accessible portions 200. Each individually accessible portion 200 represents a corresponding NN portion, for example, containing one or more NN layers of a neural network, or containing a portion of an NN layer. The device is configured to decode from the data stream 45, for each of the one or more predetermined individually accessible portions 200, a processing option parameter 250 indicating one or more processing options 252 that must be used or can be optionally used when using an NN for inference. For example, the device is configured to decode, based on the latter, which portions of the one or more predetermined individually accessible portions to access, skip, and / or decode. Based on one or more processing options 252, the device is configured to determine how and / or which individually accessible portions or individually accessible sub-parts can be accessed, skipped, and / or decoded.

[0124] According to Embodiment F2 of Embodiment F1 of DS45, the processing option parameter 250 indicates one or more available processing options 252 from a predetermined set of processing options, which include the following: The parallel processing capability of each predetermined individually accessible portion 200, and / or Each predetermined individually accessible portion 200 has a sample-level parallel processing capacity of 2521. , and / or, Each predetermined individually accessible portion 200 has a channel-based parallel processing capability 2522, and / or Each predetermined individually accessible portion 200 has a classification category unit parallel processing capability, and / or As shown in Figures 20 to 23, the dependency of a neural network portion, such as an NN layer, on computation results obtained from another individually accessible portion of DS belonging to a different version of the NN that is encoded in a layered manner in DS, although related to the same neural network portion.

[0125] The device according to embodiment ZF1 is configured to encode the processing option parameter 250 such that the processing option parameter 250 indicates one or more processing options from a predetermined set of processing options. The device according to embodiment XF1 is configured to decode the processing option parameter 250 which indicates one or more processing options from a predetermined set of processing options.

[0126] 3. Signaling of Quantization Parameters The layer payload, for example, NN parameters 32 encoded in individual accessible parts 200, or the sublayer payload, for example, NN parameters 32 encoded in individual accessible subparts 240, may include different types of parameters 32 that represent rational numbers, such as weights and biases.

[0127] In a preferred embodiment shown in Figure 18, one such type of parameter is signaled as an integer value in the bitstream so that the reconstructed value, i.e., the reconstructed NN parameter 32', is derived by applying a reconstruction rule 270 to these values, i.e., the quantization index 32'', which includes the reconstruction parameter. For example, such a reconstruction rule 270 may consist of multiplying each integer value, i.e., the quantization index 32'', by the associated quantization step size 263, where the quantization step size 263 is the reconstruction parameter in this case.

[0128] In a preferred embodiment, the reconstruction parameters are signaled either in the model parameter set 47, in the layer parameter set 110, or in the sublayer header 300.

[0129] In another preferred embodiment, a first set of reconstruction parameters is signaled within a model parameter set, optionally a second set of reconstruction parameters is signaled within a layer parameter set, and optionally a third set of reconstruction parameters is signaled within a sublayer header. If present, the second set of reconstruction parameters depends on the first set of reconstruction parameters. If present, the third set of reconstruction parameters may depend on the first and / or second set of reconstruction parameters. This embodiment is described in more detail with respect to Figure 17.

[0130] TIFF0007835841000001.tif45170

[0131] In a preferred embodiment, s=2 -0.5 That is the case.

[0132] A rational number s may be encoded, for example, as a floating-point value. The first and second integer values ​​x1 and x2 may be signaled using a fixed or variable number of bits to minimize the overall signal cost. For example, if the quantization step sizes of the sublayers of the layer are similar, the related values ​​x2 are rather small integers, and only a few bits are needed to signal them. In some cases, it may be more efficient to deny permission.

[0133] In a preferred embodiment, as shown in Figure 18, the reconstruction parameters consist of a codebook, i.e., a quantization index-reconstruction level mapping which is a list of mappings between integers and rational numbers. The integer value W is entered into bitstream 45. n Encoded as, layer Alternatively, the relevant parameters of the sublayer payload are reconstructed using the following reconstruction rule 270. Each integer value W n This is searched in the codebook. The related integer is W n One matching mapping is selected, and the associated rational number becomes the reconstructed value, i.e., the reconstructed NN parameter 32'.

[0134] In another preferred embodiment, each of the first and / or second and / or third sets of reconfiguration parameters consists of a codebook according to the previously preferred embodiment. However, in order to apply the reconfiguration rules, a single joint codebook is derived by creating a set union of the codebook mappings for the first and / or second and / or third sets of reconfiguration parameters. Where mappings with the same integer exist, the codebook mapping for the third set of reconfiguration parameters takes precedence over the codebook mapping for the second set of reconfiguration parameters, and the codebook mapping for the second set of reconfiguration parameters takes precedence over the codebook mapping for the first set of reconfiguration parameters.

[0135] Figure 17 shows an embodiment G1 of a data stream 45 having encoded NN parameters 32 within it, representing a neural network 10. The NN parameters 32 are encoded in DS45 in a method 260 which is quantized to a quantization index. The NN parameters 32 of different NN parts of NN10 are encoded in DS45 so that they are quantized differently 260. For each NN part, DS45 is encoded in a way that is relevant to each NN part. Reconstruction rule 270 for inverse quantization of parameters is shown.

[0136] For example, each part of a neural network (NN) includes interconnections between nodes of the NN, while different parts of the NN include different interconnections between nodes of the NN.

[0137] According to the embodiment, the NN portion includes the NN layer 210 of NN10 and / or a sub-layer portion 43 of a predetermined NN layer of NN. As shown in Figure 17, all NN parameters 32 in one layer 210 of NN are the NN parameters 32 in the first layer 210 of NN10. The NN parameter 32 in the second layer 2102 of NN10 is quantized 260. This represents the NN portion of the NN. Also, the NN parameters 32 within the NN layer 2101 are different. It is also possible to group the layers into sub-parts 43, i.e., individually accessible sub-parts, where each group may represent a part of the NN layer. Thus, different layer sub-parts 43 of the NN layer 2101 are quantized differently.

[0138] A corresponding embodiment ZG1 relates to a device for encoding NN parameters 32 representing a neural network 10 into a DS45. The NN parameters 32 are encoded into the DS45 in a method 260 quantized to a quantization index. The NN parameters 32 are encoded into the DS45 such that the NN parameters 32 of different NN parts of NN10 are quantized differently 260. The device is configured to provide, for each NN part, a DS45 indicating a reconstruction rule for dequantizing the NN parameters 32 associated with each NN part. Optionally, the device may also perform the quantization 260.

[0139] Another corresponding embodiment XG1 relates to a device for decoding NN parameters 32 representing a neural network 10 from a data stream 45. The NN parameters 32 are encoded in the DS45 in a method 260 quantized to a quantization index. The NN parameters 32 are encoded in the DS45 such that the NN parameters 32 of different NN parts of NN10 are quantized differently 260. The device is configured to decode a reconstruction rule 270 from the data stream 45 for dequantizing the NN parameters 32 associated with each NN part for each NN part. Optionally, the device may also perform dequantization using the reconstruction rule 270 associated with the NN part to which the currently dequantized NN parameters 32 belong. The device may also be configured to dequantize the NN parameters of each NN part for each NN part using the decoded reconstruction rule 270 associated with each NN part.

[0140] The following describes different features and / or functionalities in the context of data stream 45, but features and / or functionalities can also be features and / or functionalities of the apparatus according to embodiment ZG1 or embodiment XG1, in the same or similar manner.

[0141] As already described above, in the DS45 of embodiment G1, according to embodiment G2, the NN portion includes the NN layer 210 of NN10 and / or a subdivided layer portion of a predetermined NN layer 210 of NN10.

[0142] According to Embodiment G3, in Embodiment G1 or G2, the DS45 is delta-coded with respect to a second reconstruction rule 2702 for inverse quantization 260 of the NN parameters 32 related to the second NN portion, and the NN parameters 32 related to the first NN portion It contains therein is the first reconstruction rule 2701 for inverse quantization. As shown in Figure 17, the first NN portion, i.e., the NN associated with layer sub-part 431 The first reconstruction rule 270a1 for inverse quantizing parameter 32 is the second NN portion , that is, it is delta-coded with respect to the second reconstruction rule 270a2 related to layer subpart 432 and encoded into DS45. Also, the first NN part, i.e., layer subpart 43 The first reconstruction rule 270a1 for inverse quantizing the NN parameter 32 related to 1 is It is also possible to delta encode the second NN portion, i.e., the second reconstruction rule 2702 associated with the NN layer 2102, into DS45.

[0143] In the following embodiments, to avoid ambiguity of the embodiments, the first reconstruction rule will be denoted as 2701 and the second reconstruction rule as 2702. However, in the following embodiments as well, it is clear that the first reconstruction rule and / or the second reconstruction rule can correspond to the NN portion representing the layer sub-part 43 of the NN layer 210, as described above.

[0144] According to Embodiment G4, in Embodiment G3, DS45 includes a first index value for representing the first reconstruction rule 2701 and a second index value for representing the second reconstruction rule 2702. The first reconstruction rule 2701 is defined by a given basis exponent and a first exponent value. Defined by the first quantization step size defined by the first exponent, and The second reconstruction rule 2702 is the sum of the first and second exponential values ​​with a given base exponent. It is defined by the second exponent that is defined and the second quantization step size that is defined by the second exponent.

[0145] According to Embodiment G4a, among the DS of Embodiment G4, DS45 further represents a predetermined base.

[0146] According to Embodiment G4', among the DS of any of the above embodiments G1 to G3, DS45 is a first exponential value for indicating a first reconstruction rule 2701 for inverse quantizing the NN parameter 32 related to the first NN portion, and the NN parameter 3 related to the second NN portion. This includes a second exponential value to show a second reconstruction rule 2702 for inverse quantization of 2. . The first reconstruction rule 2701 is a predetermined base exponent, a first exponent value and a predetermined exponent value Defined by the first exponent defined by the sum of and the first quantization step size defined by, The second reconstruction rule is defined by a predetermined basis exponent and a second quantization step size defined by the sum of the second exponent value and the predetermined exponent value.

[0147] According to embodiment G4'a of embodiment G4' of the DS, the DS further represents a predetermined base.

[0148] According to embodiment G4'b, among the DS of embodiment G4'a, DS represents a predetermined basis related to the range of NN, i.e., the entire NN.

[0149] According to embodiment G4'c, among the DS of any of the earlier embodiments from embodiment G4' to G4'b, DS45 further exhibits a predetermined index value.

[0150] According to Embodiment G4'd, in Embodiment G4'c, DS45 shows a predetermined index value for a predetermined NN layer 210 to which the NN layer range, i.e., the first NN portion 431 and the second NN portion 432, belong.

[0151] According to embodiment G4'e, in any of the earlier embodiments G4'c and G4'd, DS45 further indicates a predetermined base, and DS45 indicates a predetermined exponential value in a range finer than the range indicated by DS45.

[0152] According to embodiment G4f, among the DS45s of the preceding embodiments G4 to G4a or G4' to G4'e, the DS45 has a predetermined base encoded therein in a non-integer format, such as a floating-point, rational, or fixed-point number, and first and second exponent values ​​in an integer format, such as signed integers. Optionally, the predetermined exponent values ​​may also be encoded in the DS45 in integer format.

[0153] According to Embodiment G5, among the DSs of any of Embodiments G3 to G4f, DS45 represents the first quantization index-reconstruction level to demonstrate the first reconstruction rule 2701. This shows the first parameter set that defines the mapping and the second reconstruction rule 2702. Therefore, it includes a second set of parameters that define a second quantization index-reconstruction level mapping. The first reconstruction rule 2701 is the first quantization index-reconstruction level mapping. Defined by: The second reconstruction rule 2702 is the first quantization index-reconstruction level mapping. This is defined by extending it in a predetermined manner with a second quantization index-reconstruction level mapping.

[0154] According to Embodiment G5', the DS45 described in any of Embodiments G3 to G5, the DS45, in order to demonstrate the first reconstruction rule 2701, the first quantization index-reconstruction rule The first parameter set defines the bell mapping, and the second reconstruction rule 2702 is specified. This includes a second quantization index for demonstration purposes and a second set of parameters that define a reconstruction level mapping. The first reconstruction rule 2701 is a given quantization index-reconstruction level mapping. This is defined by extending it in a predetermined manner by a first quantization index-reconstruction level mapping, and The second reconstruction rule 2702 is a given quantization index-reconstruction level mapping. This is defined by extending it in a predetermined manner by a second quantization index-reconstruction level mapping.

[0155] According to Embodiment G5'a, in Embodiment G5', DS45 further indicates a predetermined quantization index-reconstruction level mapping.

[0156] According to Embodiment G5'b, among the DS45 of Embodiment G5'a, DS45 is in the NN range, that is, relating to the entire NN, or in the NN layer range, that is, the first NN portion 431 and a predetermined quantization index relating to a predetermined NN layer 210 to which the second NN portion 432 belongs - This indicates a reconstruction level mapping. A given quantization index-reconstruction level mapping may be shown in the NN scope if it is an NN portion representing an NN layer. For example, for each NN portion, each NN portion represents a corresponding NN layer, and for example, the first NN portion represents a different NN layer from the second NN portion. However, if at least some of the NN portions represent a layer sub-part 43, it is also possible to show a given quantization index-reconstruction level mapping in the NN range. Furthermore, or alternatively, if it is an NN portion representing a layer sub-part 43, the given quantization index-reconstruction level mapping may also be shown in the NN layer range.

[0157] According to embodiment G5c, according to a predetermined configuration of the DS45 in any of the previous embodiments G5 or G5' to G5'b, The mapping of each index value, i.e., quantization index 32'', to the first reconstruction level, according to the extended quantization index-reconstruction level mapping, is the mapping of each index value, i.e., quantization index 32'', to the second reconstruction level, according to the quantization index-reconstruction level mapping that extends the extended quantization index-reconstruction level mapping. If a mapping to Bell exists, it takes precedence, and / or, According to the extended quantization index-reconstruction level mapping, no reconstruction level is defined to which each index value should be mapped; according to the quantization index-reconstruction level mapping that extends the extended quantization index-reconstruction level mapping, for any index value that maps to the corresponding reconstruction level, the mapping from each index value to the corresponding reconstruction level is adopted, and / or According to the extended quantization index-reconstruction level mapping, the reconstruction level to which each index value should be mapped is not defined, and according to the extended quantization index-reconstruction level mapping, for any index value that is mapped on the corresponding reconstruction level, the mapping from each index value to the corresponding reconstruction level is adopted.

[0158] According to Embodiment G6 shown in Figure 18, among the DS45s of any of the above embodiments G1 to G5c, the DS45 includes, for example, the following to represent an NN layer or a layer sub-part of an NN layer in order to show a predetermined NN portion reconstruction rule 270. The quantization step size parameter 262 indicates a quantization step size of 263, and A parameter set 264 defines the quantization index-reconstruction level mapping 265. Here, the reconstruction rule 270 for a given NN portion is: A quantization step size 263 for a quantization index 32'' within a predetermined index interval 268, and Quantization index-reconstruction level mapping 265 for quantization index 32'' outside a predetermined index interval 268, Defined by:

[0159] Figure 18 shows an embodiment H1 of a data stream 45 in which the NN parameters 32 representing the neural network are encoded. Here, the NN parameter 32 is encoded into DS45 in a quantized form on the quantization index 32''. Here, DS45 includes the following to show the reconstruction rule 270 for dequantizing the NN parameter 280, i.e., the quantization index 32'': Quantization step size parameter 262, which indicates a quantization step size of 263, A parameter set 264 defines the quantization index-reconstruction level mapping 265. Here, the reconstruction rule 270 for a given NN portion is: A quantization step size 263 for a quantization index 32'' within a predetermined index interval 268, Defined by a quantization index-reconstruction level mapping 265 for quantization indices 32'' outside a predetermined index interval 268.

[0160] A corresponding embodiment ZH1 relates to a device for encoding NN parameters 32 representing a neural network into a DS45. The NN parameters 32 are quantized to a quantization index 32'' and encoded into the DS45. The device is configured to provide the DS45 with the following to instruct a reconstruction rule 270 for dequantizing the NN parameters 32: Quantization step size parameter 262, which indicates a quantization step size of 263, A parameter set 264 defines the quantization index-reconstruction level mapping 265. Here, the reconstruction rule 270 for a given NN portion is: A quantization step size 263 for a quantization index 32'' within a predetermined index interval 268, This is defined by a quantization index-reconstruction level mapping 265 for quantization indices 32'' outside a predetermined index interval 268.

[0161] Another corresponding embodiment XH1 relates to a device for decoding NN parameters 32 representing a neural network from DS45. The NN parameters 32 are quantized to a quantization index 32'' and encoded into DS45. The device, Quantization step size parameter 262, which indicates a quantization step size of 263, The configuration allows for the derivation of a reconstruction rule 270 for inverse quantization of the NN parameters, i.e., the quantization index 32'', by decoding a parameter set 264 that defines the quantization index-reconstruction level mapping 265 from DS45. Here, the reconstruction rule 270 for a given NN portion is: A quantization step size 263 for a quantization index 32'' within a predetermined index interval 268, This is defined by a quantization index-reconstruction level mapping 265 for quantization indices 32'' outside a predetermined index interval 268.

[0162] The following describes different features and / or functionalities in the context of data stream 45, but these features and / or functionalities may be the same or similar features and / or functionalities of the apparatus according to embodiment ZH1 or embodiment XH1.

[0163] According to Embodiment G7, the predetermined index interval 268 of the DS45 in either Embodiment G6 or H1 includes zero.

[0164] According to Embodiment G8, in the DS45 of Embodiment G7, a predetermined index interval 268 extends up to a predetermined size threshold y, and quantization indices 32'' exceeding the predetermined size threshold y represent an escape code that signals that the quantization index-reconstruction level mapping 265 is used for inverse quantization 280.

[0165] According to Embodiment G9, the parameter set 264 of the DS45 in any of the previous embodiments G6 to G8 defines a quantization index-reconstruction level mapping 265 by a list of reconstruction levels associated with quantization indices 32'' outside a predetermined index interval 268.

[0166] According to Embodiment G10, in any of the DS45s from Embodiments G1 to G9, the NN portion includes one or more sub-parts of the NN layers of the NN and / or one or more NN layers of the NN. Figure 18 shows an example of an NN portion including one NN layer of the NN. The NN parameter tensor 30, which includes the NN parameters 32, represents the corresponding NN layer.

[0167] According to Embodiment G11, in any of the DS45s of the previous embodiments G1 to G10, the data stream 45 is structured into individually accessible parts, and each individually accessible part has NN parameters 32 encoded therein for the corresponding NN part. See, for example, Figure 8 or one of Figures 10 to 17.

[0168] According to embodiment G12, the individually accessible portions of DS45 in G11 are encoded at the starting point of each individually accessible portion using context-adaptive arithmetic coding and context initialization, as shown in Figure 8, for example.

[0169] According to Embodiment G13, among the DS45 of any previous Embodiment G11 or G12, The datastream 45, for example, as shown in one of Figures 11 to 15, has each individually accessible portion, Each individually accessible section begins with start code 242, and / or Pointers 220 / 244 that point to the beginning of each individually accessible portion, and / or This includes a data stream length parameter 246 that indicates the data stream length of each individually accessible portion, for skipping each individually accessible portion when analyzing DS45.

[0170] According to Embodiment G14, in any of the DS45s of Embodiments G11 to G13 above, the data stream 45 has a reconstruction rule 270 for inverse quantization 280 of the NN parameters 32 associated with each NN portion for each NN portion. 47, the main header section of DS45 concerning the entire NN, The NN layer-related header portion 110 of DS45 related to the NN layer 210 to which each NN portion belongs, or The NN portion specific header portion 300 of DS45 related to each NN portion, for example, if the NN portion represents a layer sub-part of the NN layer 210, i.e., individually accessible sub-parts 43 / 44 / 240, then it is a part of that. It will be shown inside.

[0171] According to Embodiment G15, among the DS45s of any prior embodiments G11 to G14, the DS45 is from any prior embodiment A1 to F2.

[0172] 4. Identifiers that depend on the parameter hash. In scenarios like distributed learning, where many clients individually further train a network and send relative NN updates back to a central entity, it is crucial to identify the network using a versioning scheme. This allows the central entity to identify the NN on which the NN update is being built.

[0173] In other use cases, such as scalable neural networks (NNs), a baseline portion of the NN can be run to generate preliminary results, for example, before running the full or enhanced NN to obtain the complete results. The enhanced NN may use a slightly different version of the baseline NN, for example, an updated parameter tensor. If such an updated parameter tensor is differentially encoded, i.e., the parameter tensor on which the differentially encoded update is constructed as an update to a previously encoded parameter tensor needs to be identified using the identification parameter 310, for example, as shown in Figure 19.

[0174] Furthermore, there are use cases where the integrity of the NN is paramount, namely, use cases where transmission errors or unintended changes in parameter tensors must be easily detectable. Identifiers, i.e., identification parameters 310, make the operation more error-robust if they can be verified based on the properties of the NN.

[0175] However, state-of-the-art version control is done via checksums or hashes of the entire container's data format, making it difficult to match equivalent neural networks (NNs) across different containers. Furthermore, the clients involved may be using different frameworks / containers. Additionally, it's impossible to identify / verify only a subset (layers, sublayers) of an NN without completely reconstructing it.

[0176] Therefore, as part of the present invention, in one embodiment, an identifier, i.e., an identification parameter 310 is carried along with each entity, namely the model, layer, and sublayer, enabling each entity to do the following: - Identity verification, and / or, -to refer to, or be referred to, and / or, - Verify consistency.

[0177] In another embodiment, the identifier is derived from a parameter tensor using a hash algorithm such as MD5 or SHA5, or an error detection code such as CRC or checksum.

[0178] In another embodiment, such identifiers for an entity are derived using identifiers of subordinate entities. For example, layer identifiers may be derived from the identifiers of the constituent sub-layers, and model identifiers from the identifiers of the constituent layers.

[0179] Figure 19 shows an embodiment I1 of a data stream 45 in which a representation of a neural network is encoded. The data stream 45 is structured into individually accessible portions 200. Each portion 200 represents a corresponding NN portion of the neural network, for example, containing one or more NN layers or a portion of an NN layer. For each of the one or more predetermined individually accessible portions 200, the data stream 45 includes an identification parameter 310 for identifying each predetermined individually accessible portion 200.

[0180] The corresponding embodiment ZI1 relates to a device for encoding a representation of a neural network into a DS45. The data stream 45 is structured into individually accessible portions 200. Each portion 200 includes, for example, one or more NN layers or a portion of an NN layer, representing a corresponding NN portion of the neural network. The device is configured to provide the data stream 45 with identification parameters 310 for each of the one or more predetermined individually accessible portions 200, for each predetermined individually accessible portion 200.

[0181] Another corresponding embodiment XI1 relates to a device for decoding a representation of a neural network from DS45. The data stream 45 is structured into individually accessible portions 200. Each portion 200 represents a corresponding NN portion of a neural network, for example, containing one or more NN layers or a portion of NN layers. The device is configured to decode from the data stream 45, for each of one or more predetermined individually accessible portions 200, an identification parameter 310 for identifying each predetermined individually accessible portion 200.

[0182] The following describes different features and / or functionalities in the context of data stream 45, but these features and / or functionalities can also be the features and / or functionalities of the apparatus according to embodiment ZI1 or embodiment XI1, respectively.

[0183] According to Embodiment I2, in the DS45 of Embodiment I1, the identification parameter 310 is associated with each predetermined individually accessible portion 200 via a hash function or an error detection code or an error correction code.

[0184] Embodiment I3 further includes a higher-level identification parameter for identifying one or more predetermined sets of individually accessible portions 200 of the DS45 from any of the previous embodiments I1 and I2.

[0185] According to Embodiment I4, the higher-level identification parameter of the DS45 in Embodiment I3 is associated with one or more predetermined individually accessible parts 200 identification parameters 310 via a hash function or an error detection code or an error correction code.

[0186] According to Embodiment I5, in any of the DS45s from Embodiments I1 to I4, the individually accessible portion 200 is encoded at the starting point of each individually accessible portion using context-adaptive arithmetic coding and context initialization, as shown in Figure 8, for example.

[0187] According to Embodiment I6, in any of the DS45s of the previous embodiments I1 to I5, the data stream 45 is, for example, as shown in any of Figures 11 to 15, individually accessible portions 200, Each individually accessible portion 200 begins with starting code 242, and / or Pointers 220 / 244 that point to the beginning of each individually accessible portion 200, and / or Includes a data stream length parameter 246 that indicates the data stream length of each individually accessible portion 200, for skipping each individually accessible portion 200 when analyzing DS45.

[0188] According to Embodiment I7, in any of the DS45s of the previous embodiments I1 to I6, the NN portion includes one or more sub-parts of the NN layer and / or one or more NN layers of the NN.

[0189] According to Embodiment I8, among the DS45s in any of the previous embodiments I1 to I7, the DS45 is one of the previous embodiments A1 to G15.

[0190] 5. Scalable NN Bitstreams As mentioned above, some applications further structure NN10 by dividing layer 210 or a group of layers, i.e., sublayers 43 / 44 / 240, into a baseline, e.g., a second version 3301 of NN10, and an advanced part 3302, e.g., a first version 3302 of NN10, as shown in Figures 20 to 23. It depends. The client may be able to match its processing power or make inferences on a baseline before processing more complex advanced neural networks. In such cases, it is beneficial to sort, encode, and make accessible in an informed manner the parameter tensor 30 of each sub-part of the NN layer independently, as described in Sections 1 to 4.

[0191] Furthermore, in some cases, NN10, - As shown in Figure 22, reducing the number of neurons in a layer, for example, requiring fewer operations, and / or, - As shown in Figure 21, making the weight quantization coarser, for example, to enable faster reconstruction, and / or, - As shown in Figure 23, different training (learning) methods are used, such as a general baseline NN and a personalized advanced NN. - These can be used to divide the data into baseline and advanced variants.

[0192] Figure 21 shows a modified version of the NN and the difference delta signal 342. A baseline version, e.g., a second version 3301 of the NN, and an advanced version, e.g., The first version of NN, version 3302, is illustrated. Figure 21 shows two quantization settings. One example of the above case is to create two variants of the original NN from a single layer, for example, a parameter tensor 30 representing the corresponding layer, and create the respective delta signals 342. Baseline version 3301 is associated with coarse quantization, and the advanced version Version 3302 is associated with finer quantization. Advanced version 3302 can be delta coded relative to baseline version 3301.

[0193] Figure 22 shows a further variation of the original NN separation. In Figure 22, a further variation of the NN separation is shown. For example, on the left side, the separation of a layer, e.g., a parameter tensor 30 representing a corresponding layer, into a baseline portion 30a and an advanced portion 30b is shown, i.e., the advanced portion 30b extends the baseline portion 30a. Inference of the advanced portion 30b requires inference of the baseline portion 30a. On the right side of Figure 22, it is shown that the central portion of the advanced portion 30b consists of an update of the baseline portion 30a, which can also be delta coded as shown in Figure 21.

[0194] In these cases, the NN parameters 32, e.g., weights, of the baseline 3301 and advanced 3302 NN versions have a clear dependency, and / or the baseline version 3301 of the NN is in some way part of the advanced version 3302 of the NN.

[0195] Therefore, the advanced NN portion, i.e., the first version of NN 3302, is parametric Encoding the meter tensor 30b as the delta of the parameter tensor 30b of the baseline NN version, i.e., the second part 3301 of the NN, at the NN scale, layer scale, or sub-layer scale is beneficial in terms of coding efficiency, processing overhead, and parallelization.

[0196] Further variations are shown in Figure 23, where an advanced version of the NN is created to compensate for the compression effect on the original NN by training in the presence of a lossy compressed baseline NN variant. The advanced NN is inferred in parallel with the baseline NN, and its NN parameters, e.g., weights, are connected to the same neurons as the baseline NN. Figure 23 shows, for example, the training of an augmented NN based on a lossy encoded baseline NN variant.

[0197] In one embodiment, a (sub)layer bitstream, i.e., individually accessible portion 200 or individually accessible sub-parts 34 / 44 / 220, is divided into two or more (sub)layer bitstreams, the first of which is the baseline version 3301 of the (sub)layer. The second one is expressed as an advanced version 3302 of the first (sub) layer. The baseline version 3301 is the advanced version in bitstream order. It precedes n3302.

[0198] In another embodiment, the (sub)layer bitstream is shown to include incremental updates of another (sub)layer parameter tensor 30 in the bitstream, for example, an incremental update including a delta parameter tensor, i.e., a delta signal 342, and / or a parameter tensor.

[0199] In another embodiment, the (sub)layer bitstream carries a reference identifier that refers to the (sub)layer bitstream having a matching identifier that includes incremental updates of the parameter tensor 30.

[0200] Figure 20 shows an embodiment J1 of the data stream 45 in which the representation of the neural network 10 is encoded in a layered manner so that different versions 330 of NN10 are encoded in the data stream 45. The data stream 45 contains one or more individually accessible It is structured into parts 200. Each part 200 is related to a corresponding version 330 of the neural network 10. Data stream 45 is version 3302 of NN10, Delta-coded 340 for the second version 3301 of NN10 encoded in the second part 2001, and / or To perform inference based on the first version 3302 of the neural network Each of these is in the form of one or more compensating NN parts 332 that are performed in addition to the execution of the corresponding NN part 334 of the second version 3301 of NN10 encoded in the second part 2001, and In the form of a 338-compensated NN portion 332, the outputs 336 of each compensated NN portion 332 and the corresponding NN portion 334 are summed together. The first part is encoded in 2002.

[0201] According to the embodiment, the compensating NN portion 332 may include a delta signal 342 as shown in Figure 21, or an additional tensor and delta signal as shown in Figure 22, or NN parameters learned to be different from the NN parameters in the corresponding NN portion 334 as shown in Figure 23, for example.

[0202] As shown in Figure 23, according to the embodiment, the compensated NN portion 332 includes the quantized NN parameters of the NN portion of the second neural network, and the NN portion of the second neural network is associated with NN10, i.e., the corresponding NN portion 334 of the first NN. The second neural network may be trained so that the compensated NN portion 332 can be used to compensate for compression effects, such as quantization errors, on the corresponding NN portion 334 of the first NN. The output of each compensated NN portion 332 and the corresponding NN portion 334 is re-engineered with the NN parameters corresponding to the first version 3302 of NN10. It is configured to enable inference based on the first version 3302 of NN10. It can be done.

[0203] While the embodiments described above primarily focus on providing different versions 330 of NN10 in a single data stream, it is also possible to provide multiple different versions 330 in multiple different data streams. Different versions 330 are delta-coded into different data streams, for example, compared to a simpler version. Therefore, separate data streams (DS) may be used. For example, a DS containing the initial NN data might be sent first, followed later by a DS containing the updated NN data.

[0204] The corresponding embodiment ZJ1 relates to an apparatus for hierarchically encoding a representation of a neural network into a DS45 such that different versions 330 of NN10 are encoded into a data stream 45, and the data stream 45 is structured into one or more individually accessible parts 200. Each part 200 corresponds to a corresponding version 330 of the neural network 10. The apparatus, The first version 3302 of NN10, Delta-coded 340 for the second version 3301 of NN10 encoded in the second part 2001, and / or To perform inference based on the first version 3302 of NN10, each of them is second A form of one or more compensating NN parts 332 that are performed in addition to the execution of the corresponding NN part 334 of the second version 3301 of NN10 encoded in part 2001, and In the form of a 338-compensated NN portion 332, the outputs 336 of each compensated NN portion 332 and the corresponding NN portion 334 are summed together. It is configured to encode into the first part 2002.

[0205] Another corresponding embodiment XJ1 relates to an apparatus for decoding a representation of a neural network 10 from a layered encoded DS45. Different versions 330 of NN10 are encoded in a data stream 45, the data stream 45 is structured into one or more individually accessible parts 200, each part 200 relating to a corresponding version 330 of the neural network 10, and the apparatus, By using delta decoding 340 for the second version 3301 of NN10 encoded in the second part 2001, and / or, Each of the DS45s performs inference based on the first version 3302 of NN10. To do this, by decoding one or more compensation NN portions 332, It is configured to decode the first version 3302 of NN10 encoded from the first part 2002, The second part 2001 is encoded in addition to the execution of the corresponding NN part 334 of the second version 3301 of NN10, The outputs 336 of each compensated NN section 332 and the corresponding NN section 334 are summed up to 338.

[0206] The following describes different features and / or functionalities in the context of data stream 45, but the features and / or functionalities may be the same or similar features and functionalities of the apparatus according to embodiment ZJ1 or embodiment XJ1.

[0207] According to Embodiment J2, among the data streams 45 of Embodiment J1, data stream 45 is the first version 3301 of NN10, The difference in weights and / or biases, i.e., the difference between the NN parameters associated with the first version 3301 of NN10 and the NN parameters associated with the second version 3302 of NN10, as shown in Figure 21, and / or For example, as shown in Figure 22, additional neurons or interconnections of neurons, The second version 3302 of NN10 encoded in the second part 2002 is delta encoded 340 with respect to the first part 2001 which is encoded.

[0208] According to Embodiment J3, among the DSs of any previous Embodiments J1 and J2, the individually accessible parts 200 are encoded using context adaptive arithmetic coding with context initialization at the start point of each individually accessible part 200, as shown, for example, in FIG. 8.

[0209] According to Embodiment J4, among the DSs of any previous Embodiments J1 - J3, the data stream 45, as shown, for example, in one of FIGS. 11 - 15, each individually accessible part 200 includes a start code 242 at which each individually accessible part 200 starts, and / or a pointer 220 / 244 pointing to the start of each individually accessible part 200, and / or when analyzing the DS45, a data stream length parameter indicating the data stream length 246 of each individually accessible part 200 for skipping each individually accessible part 200.

[0210] According to Embodiment J5, among the DS45s of any previous Embodiments J1 - J4, the data stream 45 includes an identification parameter 310 for identifying each of one or more predetermined individually accessible parts 200, as shown, for example, in FIG. 19.

[0211] According to Embodiment J6, among the DS45s of any of the previous Embodiments J1 - J5, the DS45 is according to any of the previous Embodiments A1 - I8.

[0212] 6 Reinforcement Data As shown in Figures 24a and 24b, there are application scenarios in which the parameter tensor 30 is accompanied by additional augmentation (or supplementary / auxiliary) data 350. This supplementary data 350 is not usually necessary for decoding / reconstructing / inference of the neural network, but can be essential from an application perspective. For example, it may be information about the relationships between each parameter 32 (Sebastian Lapuschkin, 2019), or information about sufficient statistics of the parameters 32, such as intervals or variances that show the robustness of each parameter 32 to perturbations (Christos Louizos, 2017).

[0213] Such augmentation information, i.e., supplemental data 350, can introduce a considerable amount of data with respect to the NN parameter tensor 30, so that it is desirable to encode the augmentation data 350 similarly using a scheme such as DeepCABAC. However, it is important to mark this data as irrelevant to the NN's decoding, solely for inference purposes, so that clients that do not require augmentation can skip this part of the data.

[0214] In one embodiment, the augmented data 350 is encoded independently of the (sub)layer bitstream data, for example, independently of the individually accessible portions 200 and / or individually accessible sub-portions 240, but is carried in additional (sub)layer augmented bitstreams, i.e., further individually accessible portions 352, which are scattered with each (sub)layer bitstream to form a model bitstream, i.e., a data stream 45. Figures 24a and 24b illustrate this embodiment. Figure 24b is a diagram illustrating the augmented bitstream 352.

[0215] Figures 24a and 24b show an embodiment K1, which illustrates a data stream 45 on which a representation of a neural network is encoded. The data stream 45 is structured into individually accessible portions 200, each portion 200 representing a corresponding NN portion of the neural network. For each of one or more predetermined individually accessible portions 200, the data stream 45 includes augmentation data 350 to supplement the representation of the NN. Alternatively, as shown in Figure 24b, the data stream 45 includes supplemental data 350 to supplement the representation of the NN for one or more predetermined individually accessible portions 200.

[0216] Corresponding embodiment ZK1 relates to a device for encoding a representation of a neural network into DS45. The data stream 45 is structured into individually accessible portions 200. Each portion 200 represents a corresponding NN portion of the neural network. The device is configured to provide the data stream 45 with supplemental data 350 to supplement the representation of the NN for each of one or more predetermined individually accessible portions 200. Alternatively, the device is configured to provide the data stream 45 with supplemental data 350 to supplement the representation of the NN for one or more predetermined individually accessible portions 200.

[0217] Another corresponding embodiment XK1 relates to a device for decoding a representation of a neural network from DS45. The data stream 45 is structured into individually accessible portions 200, each portion 200 representing a corresponding NN portion of a neural network. The device is configured to decode from the data stream 45 supplemental data 350 to supplement the representation of the NN for each of one or more predetermined individually accessible portions 200. Alternatively, the device decodes from the data stream 45 one or more predetermined For each individually accessible portion 200, it is configured to decode supplemental data 350 to supplement the representation of the neural network.

[0218] In the following, different features and / or functionalities are described in the context of data stream 45, but features and / or functionalities may also be features and / or functionalities of the apparatus according to embodiment ZK1 or embodiment XK1, in the same or similar manner.

[0219] According to Embodiment K2, in the data stream 45 of Embodiment K1, DS45 indicates that the supplemental data 350 is not essential for NN-based inference.

[0220] According to Embodiment K3, in any of the data streams 45 of the previous embodiments K1 and K2, the data stream 45 has supplemental data 350 to supplement the representation of the NN for one or more predetermined individually accessible portions 200 encoded into further individually accessible portions 352, as shown in Figure 24b. The DS45 includes, for one or more predetermined individually accessible portions 200, for each of the one or more predetermined individually accessible portions 200, a corresponding further predetermined individually accessible portion 352 related to the NN portion that each predetermined individually accessible portion 200 corresponds to.

[0221] According to Embodiment K4, in any of the earlier embodiments K1 to K3, the NN portion of the DS45 includes one or more NN layers of the NN and / or a subdivided layer portion of a predetermined NN layer of the NN. According to Figure 24b, for example, an individually accessible portion 2002 and a corresponding further predetermined The individually accessible portion 352 relates to an NN portion that includes one or more NN layers.

[0222] According to Embodiment K5, any individually accessible portion 200 of the DS45 from any previous Embodiments K1 to K4 is encoded using context-adaptive arithmetic coding and context initialization at the starting point of each individually accessible portion 200, for example, as shown in Figure 8.

[0223] According to Embodiment K6, among the DS45 of any of the above embodiments K1 to K5, the data stream 45 is, for example, as shown in any of Figures 11 to 15, for each individually accessible portion 200, Each individually accessible portion 200 begins with starting code 242, and / or Pointers 220 / 244 that point to the beginning of each individually accessible portion 200, and / or The DS45 includes a data stream length parameter that indicates the data stream length 246 of each individually accessible portion 200, for skipping each individually accessible portion 200 when analyzing the DS45.

[0224] According to Embodiment K7, among the DS45 of any of the above embodiments K1 to K6, the supplementary data 350 is: Relevance score of NN parameters, and / or, This concerns the perturbation robustness of NN parameters.

[0225] According to Embodiment K8, among the DS45s of any of the above embodiments K1 to K7, the DS45 is one of the above embodiments A1 to J6.

[0226] 7. Extended control data In addition to the described functionality of different access functions, there is an extended hierarchical control data structure, That is, the sequence 410 of the control data part 420 may be required for different applications and usage scenarios. On the one hand, the compressed NN representation (or bitstream) may be used from within a specific framework such as TensorFlow or Pytorch, in which case only a minimal amount of control data 400 is required, for example, to decode the deepCABAC coding parameter tensor. On the other hand, a specific type of framework may not be known to the decoder, in which case additional control data 400 is required. Thus, depending on the knowledge of the use case and environment, different levels of control data 400 may be required, as shown in Figure 25.

[0227] Figure 25 shows the hierarchical control data (CD) structure for a compressed neural network, that is, the sequence 410 of the control data part 420. Depending on the usage environment, different CD levels, that is, whether the control data part 420, for example, the dotted box, exists or not. In Figure 25, for example, the compressed bitstream containing the neural network representation 500 may be of any of the above model bitstream types, for example, including all the compressed data of the network with or without subdivision into sub-bitstreams.

[0228] Therefore, when a specific network (e.g., TensorFlow, Pytorch, Keras, etc.) with a type and architecture known to the decoder and encoder includes compressed NN technology, only the compressed NN bitstream is required. However, when the decoder does not know any encoder settings, in addition to enabling a complete reconstruction of the network, a full set of control data, that is, the complete sequence 410 of the control data part 420 is required.

[0229] Examples of different hierarchical control data layers, that is, the control data section 420, are as follows. - CD level 1: Compressed data decoder control information. -CD Level 2: Specific syntax elements from each framework (Tensor Flow, PyTorch, Keras) -CD Level 3: Inter-framework formatting elements, such as ONNX (Open Neural Network Exchange) for use in different frameworks. -CD Level 4: Information about the network topology -CD Level 5: Complete network parameter information (for complete reconstruction without knowledge of network topology)

[0230] Therefore, this embodiment describes a hierarchical control data structure of N levels, i.e., N control data portions 420. Here, levels 0 to N may exist to enable different usage modes, from the use of core data with specific compression only to fully self-contained network reconstruction. The levels, i.e., control data portions 420, may even include syntax from existing network architectures and frameworks.

[0231] In another embodiment, different levels, i.e., control data portions 420, may contain information about the neural network at different levels of precision. For example, the level structure may be configured as follows: -CD Level 1: Contains information about network parameters, such as type and dimensions. -CD Level 2: Includes information about the network layer. Examples: type, identification, etc. -CD Level 3: Includes information about the network topology. Example: connectivity between layers. -CD Level 4: Contains information about the neural network model. Example: Version Learning parameters, performance, etc. -CD Level 5: Includes information about the trained and validated dataset. Example: Input natural image with 227x227 resolution and 1000 labels for categories.

[0232] Figure 25 shows Embodiment L1. The data stream 45 encodes a representation 500 of the neural network. The data stream 45 includes hierarchical control data 400 structured into a sequence 410 of control data portions 420. The control data portions 420 provide information about the NN with increasing detail along the sequence 410 of control data portions 420. The second hierarchical control data 4002 of the second control data portion 4202 contains more detailed information than the first hierarchical control data 4001 of the first control data portion 4201.

[0233] According to one embodiment, the control data portion 420 may represent different units that may include additional topology information.

[0234] The corresponding embodiment ZL1 relates to a device for encoding a representation of a neural network 500 into DS45. The device is configured to provide a data stream 45 with hierarchical control data 400 structured into a sequence 410 of control data portions 420. The control data portions 420 provide information about the NN with increasing detail along the sequence 410 of control data portions 420.

[0235] Another corresponding embodiment, XL1, relates to a device for decoding a representation of a neural network 500 from DS45. The device is configured to decode hierarchical control data 400 structured into a sequence 410 of control data portions 420 from the data stream 45. The control data portions 420 provide information about the NN with increasing detail along the sequence 410 of control data portions 420.

[0236] In the following, different features and / or functionalities are described in the context of data stream 45, but features and / or functionalities may also be features and functionalities of the device according to embodiment ZL1 or embodiment XL1, or similarly.

[0237] According to Embodiment L2, at least some of the control data portion 420 of the data stream 45 in Embodiment L1 provides partially redundant information about the NN.

[0238] According to Embodiment L3, in the data stream 45 of Embodiment L1 or L2, the first control data portion 4201 indicates a default NN type that signifies a default setting. The second control data section 4202 provides information about NN, and the default setting is that of NN. It includes parameters that indicate this.

[0239] According to Embodiment L4, among the DS45s of any of the previous embodiments L1 to L3, the DS45 is one of the previous embodiments A1 to K8.

[0240] Embodiment X1 relates to a device for decoding a data stream 45 according to any previous embodiment. The device is configured to derive an NN10 from the data stream 45, for example, according to any of the embodiments XA1 to XL1 described above. For example, the device is further configured to encode / decode DS45 according to any previous embodiment.

[0241] This device is, for example, Search for start code 242, and / or, Using the data stream length parameter 45, individually accessible portions 200 are used. Ticket, and / or, Using pointers 220 / 244, restart the parsing of data stream 45 at the beginning of the individually accessible portion 200, and / or The decoded NN parameter 32' is associated with neurons 14, 18, 20 or neuron interconnections 22 / 24 according to the coding order 104, and / or Perform context-adaptive arithmetic decoding and context initialization, and / or Perform inverse quantization / value reconstruction 280, and / or To calculate the quantization step size of 263, perform an exponential sum and / or, In response to a quantized index 32'' that deviates from a predetermined index interval 268, such as estimating an escape code, a lookup of the quantized index-reconstruction level mapping 265 is performed, and / or A hash is performed on a certain individually accessible portion 200, or an error detection / correction code is applied, and the result is compared with the corresponding identification parameter 310 to check the correctness of the individually accessible portion 200, and / or Reconstructing a version 330 of NN10 by adding weights and / or bias differences to the underlying NN version 330, and / or adding additional neurons 14, 18, 20 or neuron interconnections 22 / 24 to the underlying NN version 330, or by performing joint execution of one or more compensated NN parts and corresponding NN parts and summing their outputs, and / or When the currently read control data unit 420 assumes a known parameter state for the device, reading stops, and the control data unit 420 is sequentially read while providing information, i.e., hierarchical control data 400, in a manner that conforms to a predetermined degree.

[0242] Embodiment Y1 relates to an apparatus for performing inference using NN10, and includes an apparatus for decoding a data stream 45 and deriving NN10 from the data stream 45 according to Embodiment X1, and a processor configured to perform inference based on NN10.

[0243] Embodiment Z1 relates to a device for encoding a data stream 45 according to any previous embodiment, for example, a device according to any of the above embodiments ZA1 to ZL1, further configured to encode / decode DS45 according to any previous embodiment.

[0244] This device, for example, selects encoding order 104 to find the optimal one for optimal compression efficiency.

[0245] Embodiment U relates to a method that is carried out by any of the apparatuses of Embodiments XA1 to XL1 or ZA1 to ZL1.

[0246] Embodiment W relates to a computer program that, when executed by a computer, causes the computer to perform the method of Embodiment U.

[0247] Modified examples of the embodiment.

[0248] While some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where the block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of the corresponding block or item or the feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such apparatus.

[0249] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, which has electronically readable control signals stored thereon and cooperates (or can cooperate) with a computer system programmable to perform each method. Therefore, the digital storage medium may be computer-readable.

[0250] Some embodiments of the present invention consist of a data carrier having an electronically readable control signal, which is capable of cooperating with a programmable computer system so that one of the methods described herein is performed.

[0251] Generally, embodiments of the present invention can be implemented as a computer program product having program code, which is operable to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0252] Other embodiments consist of storing a computer program for performing one of the methods described herein on a machine-readable carrier.

[0253] In other words, embodiments of the present invention are, therefore, computer programs having program code for performing one of the methods described herein, when the computer program is executed on a computer.

[0254] A further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) having a computer program for performing one of the methods described herein recorded thereon. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitional.

[0255] Therefore, a further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0256] Further embodiments include processing means configured or adapted to perform one of the methods described herein, such as a computer or a programmable logic device.

[0257] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0258] Further embodiments of the present invention comprise an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0259] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functionality of the method described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0260] The devices described in this book can be implemented using hardware devices, computers, or a combination of hardware devices and computers.

[0261] The apparatus described herein, or any component of the apparatus described herein, may be implemented at least partially in hardware and / or software.

[0262] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0263] Any component of the methods or apparatus described herein may be performed at least partially by hardware and / or software.

[0264] The embodiments described above are for illustrative purposes only. Modifications and variations of the arrangements and details described herein will be obvious to those skilled in the art. Therefore, it is intended that the invention is limited only by the scope of the imminent claims and not by any specific details presented in the description and commentary of the embodiments herein.

[0265] 8. See reference Andrew Kerr, DM (2017, 5). Retrieved from https: / / devblogs.nvidia.com / cutlass-linear-algebra-cuda / Chollet, F. (2016). Xception: Deep Learning with Depthwise Separable Convolutions. Retrieved from https: / / arxiv.org / abs / 1610.02357 Christos Louizos, KU (2017). Bayesian Compression for Deep Learning. NI PSSebastian Lapuschkin, SW-R. (2019). Unmasking Clever Hans predictors and asse ssing what machines really learn. Nature Communications. Tao, KC (2018). Once for All: A Two-Flow Convolutional Neural Network for Vis ual Tracking. IEEE Transactions on Circuits and Systems for Video Techno logy, 3377-3386

Claims

1. A device for encoding a representation of a neural network (10) into a data stream (45), The apparatus is configured to provide the data stream (45) with serialization parameters (102) indicating the encoding order (104) in which neural network parameters (32) defining the neuron interconnections (22, 24) of the neural network are encoded in the data stream (45).

2. A device for decoding a representation of a neural network (10) from a data stream (45), The apparatus is configured to decode from the data stream (45) a serialization parameter (102) indicating the encoding order (104) in which the neural network parameters (32) defining the neuron interconnections (22, 24) of the neural network are encoded in the data stream (45).

3. The apparatus according to claim 2, wherein the apparatus is configured to decode the neural network parameters (32) from the data stream (45) using context-adaptive arithmetic decoding.

4. The data stream is structured into one or more individually accessible portions (200), each of which is structured into an individually accessible portion that represents a corresponding neural network layer (210, 30) of the neural network. The device serially decodes the neural network parameters that define the neuronal interconnections (22, 24) of the neural network within a predetermined neural network layer from the data stream (45). Using the coding order (104), the neural network parameters decoded serially from the data stream (45) are assigned to the neuron interconnects (22, 24). The apparatus according to claim 2 or 3, configured as described above.

5. The apparatus according to claim 4, wherein the serialization parameter (102) is an n-value parameter that indicates the coding order (104) from a set (108) of n coding orders (104).

6. The set (108) of the n coding order (104) is, The first default coding order (106) is different in the order in which the traverse of the dimensions (34) of the tensor (30) describing the default neural network layers (210, 30) of the neural network (10) is performed. 1 ), and / or, A second default coding order (106) differs from the first default coding order in that it scans the default neural network layers (210, 30) of the neural network (107) for scalable coding of the neural network. 2 ), and / or, A third default coding order (106) in which the order in which the neural network layers of the neural network are scanned is different from the default coding order. 3 ), and / or, A fourth default coding order (106) in which the neurons (14, 18, 20) of the neural network layer of the aforementioned neural network are scanned in a different order. 4 ) The apparatus according to claim 5, including the apparatus described in claim 5.

7. The apparatus according to any one of claims 2 to 6, wherein the serialization parameter (102) indicates a permutation used by the coding order (104) to permutate the neurons (14, 18, 20) of the neural network layer (210, 30) relative to the default order.

8. The apparatus according to claim 7, wherein the permutation orders the neurons (14, 18, 20) of the neural network layer (210, 30) such that the neural network parameter (32) increases monotonically along the coding order (104) or decreases monotonically along the coding order (104).

9. The apparatus according to claim 7, wherein the permutation is a set of predetermined coding sequences that can be signaled by the serialization parameter (102), and the neurons (14, 18, 20) of the neural network layer (210, 30) are ordered such that the bit rate for encoding the neural network parameter (32) into the data stream (45) is minimized for the permutation indicated by the serialization parameter (102).

10. The apparatus according to any one of claims 2 to 9, wherein the neural network parameters (32) include weights and biases.

11. The apparatus according to any one of claims 2 to 10, wherein the apparatus comprises individually accessible sub-parts (43, 44, 240) in which the data stream is structured into individually accessible parts (200), each sub-part (43, 44, 240) representing a corresponding neural network part of the neural network, and is configured to decode the individually accessible sub-parts from the data stream so that each sub-part (43, 44, 240) is fully scanned by the coding sequence (104) before subsequent sub-parts are scanned by the coding sequence (104).

12. The apparatus according to any one of claims 4 to 11, wherein the neural network parameters (32) are decoded from the data stream using context-adaptive arithmetic decoding and using context initialization at the starting point of any individually accessible portion (200) or sub-portions (43, 44, 240).

13. The apparatus according to any one of claims 4 to 12, wherein the apparatus is configured to decode from the data stream a start code (242) in which each individually accessible portion (200) or sub-portion (43, 44, 240) begins, and / or pointers (220, 244) pointing to the beginning of each individually accessible portion or sub-portion, and / or pointers pointing to the data stream length (246) of each individually accessible portion or sub-portion in order to skip each individually accessible portion or sub-portion when parsing the data stream.

14. The apparatus according to any one of claims 2 to 13, wherein the apparatus is configured to decode numerical computational representation parameters (120) indicating numerical representations and bit sizes of the neural network parameters (32) when the neural network (10) is used for inference from the data stream.

15. A method for encoding a representation of a neural network into a data stream (45), A method comprising the step of providing the data stream with serialization parameters indicating the encoding order in which neural network parameters defining the interconnections of neurons in the neural network are encoded in the data stream.

16. A method for decoding a representation of a neural network from a data stream, A method comprising the step of decoding serialization parameters from the data stream that indicate the encoding order in which neural network parameters defining the interconnections of neurons in the neural network are encoded in the data stream.

17. A computer program, when executed by a computer, that causes the computer to perform the method according to claim 15 or 16.

Images