Decoder, Encoder, Controller, Method, and Computer Program for Updating Neural Network Parameters Using Node Information

Abstract

An embodiment according to the invention includes a decoder for decoding parameters of a neural network, the decoder being configured to obtain a plurality of neural network parameters of the neural network based on an encoded bitstream. Further, the decoder is configured to obtain, e.g., receive, e.g., extract, from the encoded bitstream, node information describing nodes of a parameter update tree, the node information including a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and the node information including parameter update information, e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter and a newer version thereof, e.g., corresponding to a child node of the update tree. Further, the decoder is configured to derive one or more neural network parameters using parameter information of a parent node identified by the parent node identifier (the parameter information, e.g., including node information of the parent node, e.g., including parameter update information and a parent node identifier of the parent node, and / or e.g., including node parameters of the parent node, e.g., neural network parameters associated with the parent node, e.g., neural network parameters implicitly defined by the node information of the parent node, e.g., for recursive reconstruction or recursive determination or recursive calculation or recursive derivation of one or more neural network parameters) and using the parameter update information, which may be included in the node information.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments in accordance with the present invention relate to decoders, encoders, controllers, methods, and computer programs that use node information to update neural network parameters.

[0002] A further embodiment according to the invention relates to efficient signaling of neural network updates in a distributed scenario. [Background technology]

[0003] Neural networks (NNs), e.g., neural nets, are used in a variety of application fields. To provide good performance, for example, complex and computationally expensive training techniques have been developed. To perform such training processes on dedicated training devices, neural network parameters may need to be transmitted, for example, from an end-user device to the training device and vice versa. Furthermore, efficient neural network parameter representation and parameter transmission techniques may become even more important in distributed learning scenarios, where multiple devices train a neural network and updated parameters for each training process may be aggregated using a central server. Summary of the Invention [Problem to be solved by the invention]

[0004] It is therefore desirable to have a concept that allows for efficient representation and transmission of neural network parameters for a training or updating process. [Means for solving the problem]

[0005] This is achieved by the subject matter of the independent claims of the present application.

[0006] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application.

[0007] An embodiment in accordance with the present invention includes a decoder for decoding parameters of a neural network, the decoder being configured to obtain a plurality of neural network parameters of the neural network based on the encoded bitstream. The decoder is further configured to obtain, e.g., receive, e.g., extract, from the encoded bitstream, node information describing nodes of a parameter update tree, the node information including, e.g., a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and parameter update information, e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter and a newer version thereof, e.g., corresponding to a child node of the update tree.

[0008] Further, the decoder is configured to derive one or more neural network parameters using parameter information of the parent node identified by the parent node identifier (the parameter information, e.g., includes node information of the parent node, which node information includes, e.g., parameter update information and a parent node identifier of the parent node, and / or includes, e.g., node parameters of the parent node, e.g., neural network parameters associated with the parent node, e.g., neural network parameters implicitly defined by the node information of the parent node, e.g., for recursive reconstruction or recursive determination or recursive calculation or recursive derivation of one or more neural network parameters), and using, e.g., the parameter update information that may be included in the node information.

[0009] Embodiments according to the present invention are based on the main idea of ​​providing an efficient representation of neural network parameters based on parameter update trees.

[0010] The inventors have recognized that it may not be necessary to transmit / receive all parameters of a neural network to update or modify the neural network. Instead, only update information, i.e., (neural network) parameter update information, can be encoded / decoded and transmitted / received. To utilize such, e.g., difference information, or, e.g., difference information, update information including, e.g., changed values, can be used to provide information about a set of reference parameters to be adapted. The inventors have recognized that such information can be represented using node information, which includes the parameter update information and, e.g., a parent node identifier that can serve as a pointer to the set of reference parameters to be adjusted.

[0011] As an example, a decoder of the present invention may include information regarding a parameter update tree, which includes one or more nodes, for example, in a hierarchical order.

[0012] Thus, to determine one or more neural network parameters of an updated or trained neural network, a decoder can receive such node information or can extract the node information from an encoded bitstream provided to the decoder.

[0013] Using the parent node identifier, the decoder can select a particular node in the parameter update tree.

[0014] As an example, neural network parameters associated with a selected node, e.g., associated with a particular version of a neural network represented by the node, may be stored within the node. The decoder may then use the parameter update information to adapt, adjust, or update these stored neural network parameters to determine updated neural network parameters and, therefore, e.g., an updated version of the neural network represented by the selected node. As an example, a new node may be added to the update tree using the parent node identifier and parameter update information of the received or extracted node information. The new node may, for example, include or represent updated neural network parameters.

[0015] As another example, the selected node may include its parent node identifier and its parameter update information. Thus, the decoder may, for example, recursively derive the predecessor nodes of the selected node until it reaches a node for which neural network parameters are available (e.g., instead of reference information and a pointer to an update value), e.g., a root or source node. These neural network parameters may then be updated based on the parameter update information of the derived node, the selected node, and finally, the parameter update information of the received or extracted node information.

[0016] Further embodiments according to the present invention are discussed below. Optional examples and explanations regarding particular nodes can relate to any two nodes, for example, nodes U2 and U3, or the nodes shown in Figures 5 and / or 15, which are described in more detail below.

[0017] According to a further embodiment of the present invention, the decoder is configured to modify one or more neural network parameters, e.g., node parameters, defined by a parent node identified by a parent node identifier (e.g., defined implicitly or recursively by the parameter update information and parent node information of the parent node) using parameter update information, which may include instructions on how to update the parameters associated with the parent node.

[0018] One or more neural network parameters, e.g., recursively determined by a parent node identifier, can be modified, e.g., using difference information provided in parameter update information. Thus, the parameter update information can include, e.g., an update value (e.g., a delta value) and an update instruction (simply put, an instruction on what to do with the update value (e.g., add, subtract, multiply, divide, etc.)). The inventors have recognized that such inventive modifications can enable efficient determination of updated neural network parameters.

[0019] According to a further embodiment of the present invention, the decoder is configured to set up a parameter update tree, where multiple child nodes containing different parameter update information (and optionally containing the same parent node identifier) ​​are associated with a common parent node, e.g., a root node R, e.g., each node in the tree can represent a version of the neural network parameters associated with the root node of the tree.

[0020] As an example, a decoder can be configured not only to obtain node information describing the nodes of a parameter update tree, but also to configure the respective update tree. Thus, the decoder can manipulate or update, e.g., adjust, the update tree based on the received node information. As an example, multiple decoders in multiple different devices can update their respective parameter update trees such that only node information, e.g., difference information and reference information, need be transmitted between them to update their respective, e.g., common, update trees to obtain updated neural network parameters.

[0021] According to a further embodiment of the present invention, the decoder is configured to use parameter update information associated with the currently considered node, e.g., node U3, and use parameter information, e.g., tree parameters, e.g., neural network parameters of the base model, e.g., default or pre-trained or initial neural network parameters of the neural network, associated with a root node, e.g., node R, and parameter update information, e.g., update rules, associated with one or more intermediate nodes (e.g., intermediate nodes), e.g., node U2, that are between the root node, e.g., node R, and the currently considered node, e.g., node U3, in the update tree to obtain one or more neural network parameters associated with the currently considered node.

[0022] Thus, the parameter update can be performed recursively via intermediate nodes, which may be, for example, associated with intermediate neural network parameters from a previous training session on the basis of which the updated neural network parameters are obtained, and which may be located, for example, along one path in the parameter update tree from the root node to the currently considered node.

[0023] According to a further embodiment of the present invention, the decoder is configured to traverse the parameter update tree from a root node, e.g., node R, to a currently considered node, e.g., node U3, and the decoder is configured to apply the update instructions of visited nodes (e.g., update parameters of nodes U2 and U3, e.g., of nodes between the root node and the currently considered node, and of the currently considered node, e.g., of all visited nodes) to one or more initial neural network parameters (e.g., one or more "tree parameters" which may, e.g., be associated with the root node, or which may, e.g., be defined by the root node, or which may, e.g., be defined by a basic mode (which basic mode may, e.g., be associated with the parameter update tree)), to obtain one or more neural network parameters associated with the currently considered node.

[0024] The inventors have recognized that an updated version of the neural network parameters may be provided based on the set of neural network parameters of a root node by starting from the root node and applying parameter update information for each child node in a path of the parameter update tree leading to the currently considered node. The inventors have recognized that this may allow for efficient coding of the neural network parameters, since in some cases many neural network parameters may not change between different updated versions of the neural network, and therefore only a limited amount of difference information is stored and applied to modify, for example, the reference set, e.g., basic or initial set, of neural network parameters of the root node.

[0025] According to a further embodiment of the invention, the decoder is configured to aggregate multiple consecutive nodes of the parameter update tree (e.g., aggregating nodes U2 and U3 into a new single node U23. For example, aggregating multiple consecutive nodes may include determining an update rule or update instruction that is equivalent or at least approximately equivalent to the successively implemented update rules or update instructions of the aggregated nodes). Alternatively, or in addition, the decoder is configured to aggregate one or more consecutive nodes of the parameter update tree with the parameter update information.

[0026] The inventors have recognized that in this way, simply put, a shortcut may be provided to a particular set of neural network parameters represented by an aggregated node. Thus, rather than having to modify the neural network parameters of a parent node using multiple parameter updates of successive child nodes, it may be necessary to modify the neural network parameters of the parent node using just one aggregated parameter update.

[0027] According to a further embodiment of the present invention, the decoder is configured to update the parameter update tree based on the node information, for example by adding a child node associated with the parameter update information of the node information to a node of the parameter update tree associated with a parent node identifier of the node information.

[0028] The inventors have recognized that in this manner, information from multiple neural network trainings can be incorporated into each parameter update tree, and further, as another example, that in this manner, multiple parameter update trees can be kept up to date to prevent situations where a parent node identifier may refer to a node that is not known to each update tree, to enable efficient communication between devices that include the update trees.

[0029] According to a further embodiment of the invention, the decoder is configured to decide to select a neural network parameter, e.g., a tree tensor, associated with the root node, or to select a neural network parameter, e.g., a node tensor, associated with one of the descendant nodes of the root node, e.g., a child node.

[0030] Thus, the decoder can choose which version of the neural network whose parameters are represented in the parameter update tree to run or provide for further processing. The inventors have recognized that, as an example, based on information received in the bitstream, e.g., node information, the decoder may be able to choose the neural network parameters that are most suitable for a particular task. For example, in a simple implementation, the decoder could always choose the most recent node with the corresponding neural network parameters.

[0031] According to a further embodiment of the present invention, the parameter update information includes or is an update instruction defining a scaling of one or more parameter values ​​associated with a parent node of the currently considered node. Additionally, the decoder is configured to apply the scaling defined by the update instruction to, for example, one or more parameter values ​​associated with a parent node of the currently considered node, in order to obtain one or more neural network parameters associated with the currently considered node. The inventors have recognized that the scaling information can be used to efficiently update neural network parameters.

[0032] According to a further embodiment of the present invention, a plurality of neural network parameters associated with the currently considered node are represented by a parameter tensor, and the decoder is configured to apply the product tensor to the parameter tensor to obtain the parameter tensor associated with the currently considered node, e.g., by forming an element-wise product between the input parameter tensor elements and the product tensor elements. The inventors have recognized that NN parameters can be efficiently represented and coded using tensors. Furthermore, product tensors can enable computationally efficient manipulation of parameter tensors to represent multiplicative modifications between NN parameters.

[0033] According to a further embodiment of the present invention, the plurality of neural network parameters associated with a parent node are represented by a parameter tensor (e.g., a parent node tensor, e.g., a multidimensional array of values ​​of neural network parameter values), the parameter update information e.g. includes or is a product tensor of the same shape as the parent node tensor, and the decoder is configured to apply the product tensor to the parameter tensor of the parent node to obtain the parameter tensor associated with the currently considered node, e.g., by forming an element-wise product between the parent node tensor elements and the product tensor elements.

[0034] The inventors have recognized that neural network parameters can be efficiently represented using parameter tensors, and that neural network parameter updates can be efficiently represented using product tensors.

[0035] According to a further embodiment of the present invention, the parameter update information comprises or is an update instruction defining the addition of one or more modification values ​​to and / or the subtraction of one or more modification values ​​from one or more parameter values ​​associated with a parent node of the currently considered node. Further, the decoder is configured to apply the addition or subtraction of the modification values ​​defined by the update instruction, e.g., addition to or subtraction from one or more parameter values ​​associated with a parent node of the currently considered node, in order to obtain one or more neural network parameters associated with the currently considered node.

[0036] The inventors have realised that using addition or subtraction, neural network parameter updates can be performed with low computational effort.

[0037] According to a further embodiment of the present invention, the parameter update information comprises or is an update instruction defining a weighted combination of one or more parameter values ​​and one or more modification values ​​associated with parent nodes of the currently considered node, e.g. in the form of a sum tensor, a scalar node tensor weight value, and a scalar sum tensor weight value.

[0038] Further, the decoder may be configured to apply a weighted combination of one or more parameter values ​​associated with a parent node of the currently considered node, e.g., elements of the "node tensor" associated with the parent node of the currently considered node, and one or more modification values, e.g., elements of the "sum tensor", to obtain one or more neural network parameters associated with the currently considered node, e.g., elements of a "node tensor" associated with the currently considered node, wherein the weighted combination may, for example, comprise an element-wise weighted sum of the parameter values ​​and the respective modification values ​​associated with the parent node of the currently considered node.

[0039] The parameter values ​​may be, for example, neural network parameter values ​​of a particular version of the neural network associated with the parent node. The inventors have recognized that a weighted combination may be used to efficiently provide the neural network parameters associated with the currently considered node.

[0040] According to a further embodiment of the present invention, a plurality of neural network parameters associated with a parent node of the currently considered node are represented by a parameter tensor, and a plurality of neural network parameters associated with the currently considered node are represented by a parameter tensor.

[0041] Further, the multiple change values ​​may be represented by, for example, a node tensor of a parent node, e.g., a sum tensor having the same shape as the parent node tensor, and the decoder may be configured to multiply elements of the parameter tensor associated with a parent node of the currently considered node by node tensor weight values ​​to obtain a scaled parameter tensor, multiply elements of the sum tensor by sum tensor weight values ​​to obtain a scaled sum tensor, and form an element-wise sum of the scaled parameter tensor and the scaled sum tensor to obtain a parameter tensor associated with the currently considered node, e.g., the node tensor, and the parameter update information may include at least one of the node tensor weight values, sum tensor weight values, sum tensor, and / or change values.

[0042] Optionally, both weights, e.g., node tensor weight value and sum tensor weight value, can also be set to 1, which corresponds to unweighted sum as a special case of weighted sum. Alternatively, both weights can be set to 0.5, which corresponds to averaging as a special case of weighted sum.

[0043] The inventors have recognized that individual scaling of the parent node information in the form of a parameter tensor and the update information in the form of a sum tensor can allow for efficient determination of the parameter tensor of the currently considered node. Furthermore, the scaling and summation can be performed at low computational cost.

[0044] According to a further embodiment of the present invention, the parameter update information includes or is an update instruction defining the replacement of one or more parameter values ​​associated with a parent node of the currently considered node with one or more modified values, e.g., in the form of a replacement tensor. Furthermore, the decoder is configured to replace one or more parameter values ​​associated with a parent node of the currently considered node, e.g., elements of a "node tensor" associated with the parent node of the currently considered node, with one or more replacement values, e.g., elements of a "replacement tensor", to obtain one or more neural network parameters associated with the currently considered node, e.g., elements of a "node tensor" associated with the currently considered node. The inventors have recognized that replacing values ​​can, in some cases, be performed with less computational cost than using differential update information and arithmetic operations.

[0045] According to a further embodiment of the present invention, the neural network parameters associated with a parent node of the currently considered node are represented by parameter tensors, and the parameter update information comprises or is update instructions in the form of update tensors, e.g., permutation tensors, sum tensors, and / or product tensors, which may be represented by Compressed Data Units (NDUs). Furthermore, the decoder is configured to, e.g., implicitly transform the shape of the update tensor according to the shape of the parameter tensor of the parent node, e.g., such that for a parameter tensor that is a 2D tensor [[a,b,c],[d,e,f]] (dimensions are [2,3]), e.g., the node tensor of the parent node, an update tensor given as [[x],[y]] (dimensions are [2,1]) is expanded to, e.g., implicitly, [[x,x,x],[y,y,y]] (dimensions are [2,3]). The inventors have recognized that in this manner, neural network parameters may be updated, for example, in an approximate manner, but that the shapes of tensors representing, for example, base or initial neural network parameters, e.g., parameters of parent nodes, may not match the shapes of the updated tensors.

[0046] Furthermore, changes in tensor shape can be associated with topology changes of the updated neural network or its layers. Thus, embodiments in accordance with the present invention can enable incorporating topology changes of the neural network in the training and / or update process. Thus, communication and, for example, distributed training can be provided with great flexibility.

[0047] According to a further embodiment of the present invention, the tensor elements of the parameter tensor arranged along a first direction (e.g., along the rows of the tensor) are associated with the contributions of output signals of multiple neurons of a previous layer of the neural network to the input signals of a given neuron of the currently considered layer of the neural network, and the tensor elements of the parameter tensor arranged along a second direction (e.g., along the columns of the tensor) are associated with the contributions of output signals of a given neuron of the previous layer of the neural network to the input signals of multiple neurons of the currently considered layer of the neural network.

[0048] Further, the decoder is configured to expand the update tensor in a first direction (e.g., row direction) or expand the dimension of the update tensor in a first direction if, for example, the dimension is smaller than the dimension of the parameter tensor in the first direction, and / or the decoder is configured to expand the update tensor in a second direction (e.g., column direction) or expand the dimension of the update tensor in a second direction if, for example, the dimension is smaller than the dimension of the parameter tensor in the second direction.

[0049] According to further embodiments of the present invention, the decoder is configured to copy row entries of the update tensor to obtain one or more augmented row entries of the reshaped update tensor if the number of rows of the update tensor is smaller than the number of rows of the parameter tensor. Alternatively, or in addition, the decoder is configured to copy column entries of the update tensor to obtain one or more augmented column entries of the reshaped update tensor if the number of columns of the update tensor is smaller than the number of columns of the parameter tensor.

[0050] The inventors have recognized that copying or duplicating rows or columns can be a computationally inexpensive way to expand or extrapolate information. Furthermore, copies of the relevant parameters can be good approximations of the neural network parameters associated with the expanded row or column.

[0051] According to a further embodiment of the present invention, the decoder is configured to copy one or more entries of the update tensor in the row and column directions, for example a single entry of an update tensor having dimensions of 1 in all directions, or a group of two or more entries of the update tensor, to obtain the transformed, e.g., expanded, entry of the update tensor. The inventors have recognized that such reconstruction of parameters can be performed in a computationally inexpensive manner.

[0052] According to a further embodiment of the invention, the decoder is configured to determine the need to transform the shape of the update tensor and / or the degree of transformation of the shape of the update tensor depending on information about the extension of the update tensor, and preferably also depending on information about the extension of the parameter tensor to which the shape-transformed update tensor is applied.

[0053] The node information, e.g., parameter update information of the node information, may include, for example, information regarding an extension of the update tensor. The inventors have recognized that in this manner, such extension information may be transmitted or received with limited resource requirements.

[0054] According to further embodiments of the present invention, the decoder is configured to determine whether a parent node identifier is present, e.g., in the currently considered data block, e.g., by evaluating whether there is signaling indicating the presence of a parent node identifier or by parsing the syntax of the parent node identifier. Furthermore, the decoder is configured to derive one or more neural network parameters according to any of the embodiments disclosed herein, e.g., using parameter update information, e.g., if the parent node identifier is present, and in addition, depending on the value of the signaling, e.g., "parent_node_id_present_flag", a parent node identifier that uniquely identifies another NDU containing the parent node of the current PUT node is transmitted, e.g., in the form of a further new syntax element "parent_node_id".

[0055] Furthermore, the decoder is configured to make the currently considered node the root node if a parent node identifier is not present, in which case the decoder can apply independent decoding of the neural network parameters that does not rely on parameter update information.

[0056] Thus, the decoder can adjust the parameter update tree structure, for example, when some tree sections are removed, e.g., when the respective neural network parameters become outdated. A new root tree may be selected, simply by using the corresponding neural network parameters instead of the parameter update information, e.g., to establish a new "starting point" in the tree.

[0057] According to a further embodiment of the invention, the decoder is configured to compare a parent node identifier, e.g., parent_node_id, which is optionally a cryptographic hash value, with e.g., cryptographic hash values ​​associated with one or more nodes, e.g., previously determined nodes, in order to identify a parent node of the currently considered node.

[0058] This can allow for efficient and fast iteration through the parameter update tree, for example, to find even multiple predecessor nodes that modify neural network parameters. Furthermore, it can allow for secure transmission of update information, such that parent node identifiers can only be used with the correct hash value.

[0059] According to further embodiments of the present invention, the hash value may be a hash value of a full compressed data unit (NDU), including, for example, data size information, header information, and payload information, where the payload information may include, for example, arithmetically coded neural network parameters associated with one or more previously decoded nodes. The inventors have recognized that the hash value may, for example, be efficiently transmitted and / or received within the NDU.

[0060] According to a further embodiment of the present invention, the hash value is a hash value of a payload portion (e.g., including data size information, header information, and payload information, which may include, for example, arithmetically coded neural network parameters) of a compressed data unit (NDU) associated with one or more previously decoded nodes while leaving the data size information and header information unaccounted for. The inventors have recognized that the hash value can be efficiently extracted from the payload portion of the NDU.

[0061] According to a further embodiment of the present invention, the parent node identifier is a combined value representing a device identifier and a serial number, both associated with the parent node, e.g., the parent node represented as an NDU. The inventors have recognized that using such a device identifier and serial number, the parent node, and therefore the neural network parameters to be modified, can be robustly identified.

[0062] According to further embodiments of the present invention, the parent node identifier identifies an update tree, e.g., including an explicit update tree identifier or implicitly identifying an update tree, and / or a layer of a neural net, e.g., using an explicit layer identifier or implicitly identifying a layer, and e.g., the decoder may be configured to evaluate the parent node identifier to assign the node information to the appropriate update tree and / or, e.g., the decoder may be configured to evaluate the parent node identifier to assign the node information to the appropriate layer of the neural net.

[0063] In general, according to embodiments of the present invention, multiple update trees, e.g., for parameters of different neural networks, or for parameters, e.g., weights, of different layers of the same neural network, may be used to store different versions, e.g., updated versions, of the parameters.

[0064] The inventors have recognised that parent node identifiers according to embodiments of the present invention may enable such multiple update trees, and therefore parameters, to be selected and / or organised and / or managed.

[0065] According to a further embodiment of the present invention, the node information comprises, for example, a node identifier capable of identifying the node, e.g., a syntax element "node_id". The inventors have recognized that such a node identifier may enable robust identification of each node of the update tree.

[0066] According to a further embodiment of the present invention, the decoder is configured to store the node identifier, for example together with other node information or in a manner linked or referenced to other node information. The inventors have recognised that storing the node identifier may enable time-delayed processing of the corresponding node information.

[0067] According to further embodiments of the present invention, the decoder is configured to compare one or more stored node identifiers with parent node identifiers in the node information of a new node when adding a new node, for example when extending the update tree structure in response to detecting a new node, or when identifying a path through the update tree structure to the root node, to identify a parent node of the new node, thereby enabling efficient extension of the update tree.

[0068] According to further embodiments of the present invention, the node identifier identifies the update tree with which the node information is associated (e.g., includes an explicit update tree identifier or implicitly identifies the update tree), and / or the node identifier identifies a layer of the neural net, e.g., using an explicit layer identifier or implicitly identifying the layer with which the node information is associated.

[0069] Optionally, the decoder may be configured to identify, for example based on the node identifier, the update tree (or update tree structure) with which the node is associated, or the layer of the neural net with which the node is associated.

[0070] The inventors have recognized that in this way an efficient neural network parameter update scheme can be provided with low computational effort and good information compression.

[0071] According to a further embodiment of the invention, the node identifier, e.g., the syntax element node_id in the bitstream encoding the node information, comprises or consists of a device identifier and / or parameter update tree depth information, e.g., information about the number of nodes to be visited when walking the tree from the current node to the root node, and / or a parameter update tree identifier.

[0072] Optionally, the decoder may be configured to identify the update tree (or update tree structure) with which the node is associated, or the position within the update tree (or update tree structure) with which the node is associated, for example depending on the node identifier.

[0073] The inventors have recognized that based on or using such node identifiers, neural network parameters to be modified, addressed, or provided may be efficiently selected, even within organizational structures such as parameter update trees having a large number of nodes, or for different devices including, for example, multiple update trees. The depth information may enable quickly finding the tree level or layer at which the selected node is located.

[0074] According to a further embodiment of the invention, the node information comprises signalling, eg a flag, indicating whether a node identifier is present or not.

[0075] Optionally, the decoder is configured to selectively evaluate the node identifier information (eg, by parsing the bitstream) depending on signaling indicating whether the node identifier is present or not.

[0076] Thus, for example, a bitstream may be provided that includes node information with a flag that indicates or indicates that no node identifier is present in the bitstream, so that transmission resources can be provided for different information.

[0077] According to a further embodiment of the invention, the decoder is configured to obtain signaling, e.g., signaling encoded in the coded bitstream, e.g., in a header of the coded bitstream, containing information about the type of parent node identifier, e.g., parent_node_id_type, and further configured to evaluate the signaling in order to take into account the respective type of parent node identifier.

[0078] This can allow for efficient extraction of information regarding the type of identifier, e.g., whether the parent node identifier is, for example, a cryptographic hash, or a combined value representing a device identifier and serial number, or another information representation as disclosed herein.

[0079] According to a further embodiment of the present invention, the decoder is configured to selectively evaluate a syntax element indicating the type of parent node identifier, e.g., parent_node_id_type, depending on a syntax element indicating the presence of a parent node identifier, e.g., parent_node_id_present_flag, thus preserving the indication and evaluation of the type of parent node identifier when the parent node identifier is not present.

[0080] According to a further embodiment of the present invention, the decoder is configured to obtain topology change signaling in node information, e.g., in the form of a compressed data unit (NDU) containing information about a topology change of the neural network, and the decoder is configured to modify parameter information, e.g., a parameter tensor, of a parent node according to the topology change to derive one or more neural network parameters of the neural network having the modified topology.

[0081] Note that the decoder can be configured to implicitly change the network topology, for example, upon receiving an update tensor, e.g., a product or sum tensor, that has a shape that does not match the tensors of the parameter tensors of the corresponding parent node. Note that, optionally, dedicated topology change signaling can be received (or, for example, transmitted by the corresponding encoder) to robustly adapt the neural network structure.

[0082] According to a further embodiment of the present invention, the decoder is configured to modify the shape of one or two tensors (e.g., capable of describing the derivation of input signals for neurons in a given layer of the neural network based on the output signals of neurons in a neural network layer preceding the given layer, and e.g., capable of describing the derivation of input signals for a neural network layer following the given layer based on the output signals of neurons in the given layer) in response to the topology modification information, e.g., the size of the tensor describing the derivation of input signals for neurons in the given layer and the size of the tensor describing the derivation of input signals for a neural network layer following the given layer can be changed in a coordinated manner, e.g., typically the dimensions of the two tensors can be changed in the same way or in a coordinated manner.

[0083] The inventors have recognized that topology changes in a neural network can be efficiently represented, for example, by adjusting the tensors associated with an updated version of the neural network.

[0084] According to further embodiments of the present invention, the decoder is configured to modify the number of neurons in a given layer in response to the topology modification information. Thus, embodiments according to the present invention may allow, for example, to incorporate reshaping or topology adaptation of the neural network structure.

[0085] According to a further embodiment of the present invention, the decoder is configured to replace one or more tensor values ​​of one or more tensors whose shape is to be changed and associated with a parent node of the currently considered node, e.g., elements of a "node tensor" associated with the parent node of the currently considered node, with one or more replacement values, e.g., elements of a "replacement tensor", in order to obtain one or more tensors having a modified size.

[0086] Alternatively, the decoder may be configured to replace elements of one or more tensors associated with a parent node of the currently considered node whose shape is to be changed, e.g., a "node tensor" associated with the parent node of the currently considered node, with elements of one or more permutation tensors, e.g., "permutation tensors", the entries of which may be defined in the node information, e.g., using permutation instructions, to obtain one or more tensors with modified sizes.

[0087] The inventors have recognized that permuting, exchanging, and / or swapping values ​​or entire tensors or combinations thereof can allow for efficient updating of neural network parameters, for example, particularly when the shape of the respective tensors is changed or altered.

[0088] According to a further embodiment of the invention, the decoder is configured to, in response to a topology change signaling, synchronously change the shapes, e.g. sizes, of two tensors in two update trees associated with adjacent layers of the neural network, e.g. so that the number of input signals of a given layer of the neural network, the calculation of which is defined in a first update tree, is changed in the same way as the number of output signals of a given layer, the use of which for the calculation of input signals of a subsequent layer is defined in a second update tree.

[0089] A topology change in one layer of a neural network can affect preceding and / or subsequent layers (e.g., with respect to information flow through the neuron layers of the neural network). Accordingly, the inventors have recognized that, e.g., directly, cross-correlated layers, or e.g., parameters thereof, e.g., weight parameters, can be adapted together, e.g., in a synchronous manner.

[0090] A further embodiment according to the invention comprises an encoder for encoding parameters of a neural network to obtain an encoded bitstream, the encoder being configured to provide node information describing nodes of the parameter update tree, e.g. in the form of an encoded bitstream, the node information comprising a parent node identifier, e.g. a unique parent node identifier, e.g. an integer, a string and / or a cryptographic hash.

[0091] Additionally, the node information includes parameter update information (e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter (e.g., associated with a parent node) and a newer (current) version thereof, e.g., corresponding to a child node in the update tree), where the parameter update information describes a difference between the neural network parameter associated with the parent node defined by the parent node identifier and the current neural network parameter.

[0092] The encoder as described above may be based on the same considerations as the decoder as described above, and the encoder may include all (e.g., all corresponding or all similar) features and functionality that are also described with respect to the decoder.

[0093] According to a further embodiment of the present invention, the encoder is configured to determine a difference between one or more neural network parameters, e.g., node parameters, defined by a parent node identified by a parent node identifier and one or more current neural network parameters, to obtain parameter update information, which may include instructions on how to update parameters associated with the parent node.

[0094] According to a further embodiment of the present invention, the encoder is configured to set up a parameter update tree, where multiple child nodes containing different parameter update information, and optionally containing the same parent node identifier, are associated with a common parent node, e.g., a root node R, e.g., each node in the tree can represent a version of the neural network parameters associated with the root node of the tree.

[0095] According to a further embodiment of the present invention, the encoder is configured to provide the node information such that it is possible to obtain one or more neural network parameters associated with the currently considered node using parameter update information associated with the currently considered node, e.g., node U3, using parameter information, e.g., tree parameters, e.g., neural network parameters of the base model, e.g., default or pre-trained or initial neural network parameters of the neural network, associated with the root node, e.g., node R, and using parameter update information, e.g., update rules, associated with one or more intermediate nodes, e.g., node U2, between the root node, e.g., node R, and the currently considered node, e.g., node U3, in the update tree.

[0096] According to a further embodiment of the present invention, the encoder is configured to provide a plurality of node information blocks, wherein a parent node identifier of a first node information block points to a root node, and the parameter update information of the first node describes a difference between neural network parameters associated with the root node defined by the parent node identifier of the first node information block and the neural network parameters of the first node, and further wherein a parent node identifier of an Nth node information block points to an N-1th node, and the parameter update information of the Nth node describes a difference between neural network parameters associated with the N-1th node defined by the parent node identifier of the Nth node information block and the neural network parameters of the Nth node.

[0097] The inventors have recognised that in this way a recursively usable parameter update tree can be provided, so that update or difference information can be coded, transmitted and therefore efficiently evaluated.

[0098] According to a further embodiment of the present invention, the encoder is configured to provide signaling to the decoder to selectively choose neural network parameters, e.g., a tree tensor, associated with the root node or neural network parameters, e.g., a node tensor, associated with one of the descendant nodes of the root node, e.g., a child node.

[0099] According to a further embodiment of the invention, the parameter update information comprises or is an update instruction defining a scaling of one or more parameter values ​​associated with a parent node of the currently considered node, and the encoder is configured to determine the scaling based on the one or more parameter values ​​associated with the parent node of the currently considered node and the parameter values ​​of the currently considered node.

[0100] According to a further embodiment of the present invention, the plurality of neural network parameters associated with the currently considered node are represented by a parameter tensor, and the encoder is further configured to provide a product tensor to apply to the parameter tensor to obtain the parameter tensor associated with the currently considered node, for example by forming an element-wise product between the input parameter tensor elements and the product tensor elements.

[0101] According to a further embodiment of the present invention, the plurality of neural network parameters associated with the parent node are represented by a parameter tensor, e.g., a parent node tensor, e.g., a multidimensional array of values, e.g., a multidimensional array of neural network parameter values. Furthermore, the parameter update information comprises or is, e.g., a product tensor of the same shape as the parent node tensor. Furthermore, the encoder is configured to provide the product tensor such that application of the product tensor to the parameter tensor of the parent node, e.g., by forming an element-wise product between the parent node tensor elements and the product tensor elements, results in the parameter tensor associated with the currently considered node.

[0102] According to a further embodiment of the present invention, the parameter update information comprises or is an update instruction defining the addition of one or more modification values ​​to and / or subtraction of one or more modification values ​​from one or more parameter values ​​associated with a parent node of the currently considered node. Optionally, as an example, the encoder is configured to provide the modification values ​​such that applying the addition or subtraction of the modification values ​​defined by the update instruction, e.g., to or from one or more parameter values ​​associated with a parent node of the currently considered node, results in one or more neural network parameters associated with the currently considered node.

[0103] According to a further embodiment of the present invention, the parameter update information comprises or is an update instruction defining a weighted combination of one or more parameter values ​​and one or more modification values ​​associated with parent nodes of the currently considered node, e.g. in the form of a sum tensor, a scalar node tensor weight value, and a scalar sum tensor weight value.

[0104] Optionally, as an example, the encoder is configured to provide update instructions such that application of a weighted combination of one or more parameter values ​​associated with a parent node of the currently considered node, e.g., elements of a “node tensor” associated with the parent node of the currently considered node, and one or more modification values, e.g., elements of a “sum tensor”, results in one or more neural network parameters associated with the currently considered node, e.g., elements of the “node tensor” associated with the currently considered node, where the weighted combination may, for example, comprise an element-wise weighted sum of the parameter values ​​associated with the parent node of the currently considered node and the respective modification values.

[0105] According to a further embodiment of the present invention, the neural network parameters associated with the parent node of the currently considered node are represented by a parameter tensor, the neural network parameters associated with the currently considered node are represented by a parameter tensor, and the modification values ​​are represented by a sum tensor, e.g., of the same shape as the node tensor of the parent node, e.g., a parent node tensor.

[0106] Optionally, by way of example, the encoder is configured to: obtain a scaled parameter tensor; multiply elements of the parameter tensor associated with a parent node of the currently considered node with a node tensor weight value; obtain a scaled sum tensor; multiply elements of the sum tensor with the sum tensor weight value; and provide a modification value such that forming an element-wise sum of the scaled parameter tensor and the scaled sum tensor results in a parameter tensor associated with the currently considered node, e.g., the node tensor. Optionally, by way of example, the parameter update information can include at least one of a node tensor weight value, a sum tensor weight value, a sum tensor, and / or a modification value. Optionally, both weights, e.g., the node tensor weight value and the sum tensor weight value, can also be set to 1, which corresponds to a non-weighted sum as a special case of a weighted sum.

[0107] According to a further embodiment of the invention, the parameter update information comprises or is an update instruction defining the replacement of one or more parameter values ​​associated with a parent node of the currently considered node with one or more modified values, e.g. in the form of a replacement tensor.

[0108] Optionally, as an example, the encoder is configured to update the instructions such that replacement of one or more parameter values ​​associated with a parent node of the currently considered node, e.g., elements of a "node tensor" associated with the parent node of the currently considered node, by one or more replacement values, e.g., elements of a "replacement tensor", results in one or more neural network parameters associated with the currently considered node, e.g., elements of the "node tensor" associated with the currently considered node.

[0109] According to a further embodiment of the present invention, the neural network parameters associated with a parent node of the currently considered node are represented by parameter tensors, and the parameter update information comprises or is update instructions in the form of update tensors, e.g., permutation tensors, sum tensors, and / or product tensors, which may be represented, e.g., by Compressed Data Units (NDUs). Further, the encoder is configured to provide the update tensors such that the shape of the update tensors differs from the shape of the parameter tensors of the parent node, e.g., such that for a parameter tensor that is a 2D tensor [[a,b,c],[d,e,f]] (dimensions are [2,3]), e.g., the node tensor of the parent node, an update tensor given as [[x],[y]] (dimensions are [2,1]) is implicitly expanded to [[x,x,x],[y,y,y]] (dimensions are [2,3]).

[0110] According to a further embodiment of the present invention, tensor elements of the parameter tensor arranged along a first direction (e.g., along the rows of the tensor) are associated with contributions of output signals of multiple neurons of a previous layer of the neural network to input signals of a given neuron of a currently considered layer of the neural network. Furthermore, tensor elements of the parameter tensor arranged along a second direction (e.g., along the columns of the tensor) are associated with contributions of output signals of a given neuron of a previous layer of the neural network to input signals of multiple neurons of the currently considered layer of the neural network, and the encoder is configured to provide the update tensor such that the expansion of the update tensor in the first direction (e.g., row direction) is smaller than the dimension of the parameter tensor in the first direction. Alternatively or additionally, the encoder is configured to provide the update tensor such that the expansion of the update tensor in a second direction (e.g., column direction) is smaller than the dimension of the parameter tensor in the second direction.

[0111] According to further embodiments of the present invention, the encoder is configured to provide the update tensor such that the number of rows of the update tensor is smaller than the number of rows of the parameter tensor, and / or the encoder is configured to provide the update tensor such that the number of columns of the update tensor is smaller than the number of columns of the parameter tensor.

[0112] According to a further embodiment of the invention, the encoder is configured to provide information regarding the extension of the update tensor.

[0113] According to a further embodiment of the present invention, the encoder is configured to provide signaling, e.g., signaling encoded in the coded bitstream, e.g., in a header of the coded bitstream, e.g., a flag, e.g., "parent_node_id_present_flag", containing information whether a parent node identifier is present or not.

[0114] Optionally, the encoder is configured to omit signaling that a parent node exists, for example, when encoding the neural network parameters of the root node, or as another optional feature, the encoder is configured to provide signaling indicating that a parent node does not exist, for example, when encoding the neural network parameters of the root node.

[0115] According to a further embodiment of the invention, the encoder is configured to provide as parent node identifier, e.g. a cryptographic hash value associated with a node, e.g. a previously determined node, in order to identify the parent node of the currently considered node.

[0116] According to a further embodiment of the present invention, the hash value is a hash value of a full compressed data unit (e.g., an NDU, e.g., including data size information, header information, and payload information, which may include, e.g., arithmetically coded neural network parameters) associated with one or more previously encoded nodes.

[0117] According to a further embodiment of the present invention, the hash value is a hash value of a payload portion of a compressed data unit (e.g., an NDU, e.g., including data size information, header information, and payload information, which may include, e.g., arithmetically coded neural network parameters) associated with one or more previously encoded nodes, while leaving out, e.g., data size information of the compressed data unit and, e.g., header information of the compressed data unit.

[0118] According to a further embodiment of the present invention, the parent node identifier is a combined value representing a device identifier and a serial number both associated with the parent node, for example the parent node represented as an NDU.

[0119] According to further embodiments of the present invention, the parent node identifier identifies an update tree (e.g., including an explicit update tree identifier or implicitly identifying an update tree) and / or a layer of the neural net, e.g., using an explicit layer identifier or implicitly identifying a layer, e.g., the encoder is configured to provide the parent node identifier for assigning the node information to the appropriate update tree and / or e.g., the encoder is configured to provide the parent node identifier for assigning the node information to the appropriate layer of the neural net.

[0120] According to a further embodiment of the invention, the node information comprises, for example, a node identifier, for example a syntax element "node_id", by which the node can be identified.

[0121] According to a further embodiment of the present invention, the encoder is configured to store the node identifier, for example together with other node information or in a manner linked or referenced to other node information.

[0122] According to further embodiments of the present invention, the encoder is configured to compare one or more stored node identifiers with parent node identifiers in the node information of a new node when adding a new node to identify the parent node of the new node, e.g., when extending the update tree structure in response to detecting a new node, or when identifying a path through the update tree structure to the root node.

[0123] According to further embodiments of the present invention, the node identifier identifies an update tree, e.g., includes an explicit update tree identifier or implicitly identifies the update tree with which the node information is associated, and / or the node identifier identifies a layer of the neural net, e.g., using an explicit layer identifier or implicitly identifying the layer with which the node information is associated.

[0124] Optionally, the encoder may be configured to use, for example, the node identifier to identify the update tree (or update tree structure) with which the node is associated, or the layer of the neural net with which the node is associated.

[0125] According to a further embodiment of the invention, the node identifier, e.g., the syntax element node_id in the bitstream encoding the node information, comprises or consists of a device identifier and / or parameter update tree depth information, e.g., information about the number of nodes to be visited when walking the tree from the current node to the root node, and / or a parameter update tree identifier.

[0126] Optionally, the encoder may be configured to identify, for example, using the node identifier, the update tree (or update tree structure) with which the node is associated, or the position within the update tree (or update tree structure) with which the node is associated.

[0127] According to a further embodiment of the invention, the node information comprises signalling, eg a flag, indicating whether a node identifier is present or not.

[0128] Optionally, the encoder is configured to provide signaling indicating whether the node identifier is present, and / or the encoder is configured to selectively encode the node identifier information (e.g., by parsing the bitstream) in dependence on the signaling indicating whether the node identifier is present.

[0129] According to a further embodiment of the present invention, the parent node identifier is a combined value representing a device identifier and a serial number both associated with the parent node, for example the parent node represented as an NDU.

[0130] According to a further embodiment of the invention, the encoder is configured to provide signaling, e.g. a syntax element, e.g. signaling that is encoded in the encoded bitstream, e.g. in a header of the encoded bitstream, comprising information about the type of parent node identifier.

[0131] According to a further embodiment of the present invention, the encoder is configured to selectively provide a syntax element indicating the type of the parent node identifier, e.g., parent_node_id_type, when a syntax element describing the parent node identifier is present, e.g., in a bitstream block.

[0132] According to a further embodiment of the invention, the encoder is configured to provide topology change signaling within node information, for example within node information in the form of a compressed data unit (NDU) containing information about a topology change of the neural network.

[0133] According to further embodiments of the present invention, the encoder is configured to signal a change in the shape of one or two tensors (e.g., capable of describing the derivation of input signals for neurons in a given layer of the neural net based on the output signals of neurons in a neural net layer preceding the given layer, and e.g., capable of describing the derivation of input signals for a neural net layer following the given layer based on the output signals of neurons in the given layer) along with the signaling of the topology change; for example, the size of the tensor describing the derivation of input signals for neurons in the given layer and the size of the tensor describing the derivation of input signals for a neural net layer following the given layer can be changed in a coordinated manner, and typically the dimensions of the two tensors change in the same way or in a coordinated manner.

[0134] According to a further embodiment of the present invention, the encoder is configured to signal a change in the number of neurons in a given layer (or to change the number of neurons in a given layer) using the topology change information.

[0135] According to a further embodiment of the present invention, the encoder is configured to signal a replacement of one or more tensor values ​​of one or more tensors whose shape is to be changed, associated with a parent node of the currently considered node, e.g., elements of a "node tensor" associated with the parent node of the currently considered node, with one or more replacement values, e.g., elements of a "replacement tensor", e.g., to enable the decoder to obtain one or more tensors having a modified size.

[0136] Alternatively, the encoder may be configured to signal the replacement of elements of one or more tensors whose shape is to be changed, e.g., “node tensors” associated with the parent node of the currently considered node, with elements of one or more permutation tensors, e.g., “permutation tensors”, the entries of which may be defined, e.g., in the node information, e.g., using permutation instructions, to enable the decoder to obtain one or more tensors with modified sizes.

[0137] According to a further embodiment of the present invention, the encoder is configured to use topology change signaling to synchronously signal changes in the shape, e.g., size, of two tensors in two update trees associated with adjacent layers of the neural network, such that, e.g., the number of input signals of a given layer of the neural network, the computation of which is defined in a first update tree, is changed in the same way as the number of output signals of a given layer, the use of which for the computation of input signals of a subsequent layer is defined in a second update tree.

[0138] A further embodiment according to the invention includes a neural network controller configured to train the neural network, e.g., by performing training, to obtain updated, e.g., improved, neural network parameters based on the initial neural network parameters.

[0139] Further, the neural network controller is configured to determine parameter update information based on reference neural network parameters, which may be equal to the initial neural network parameters, for example, and the updated, e.g., improved, neural network parameters, and the parameter update information includes one or more update instructions that describe how to at least approximately derive the updated neural network parameters from the initial neural network parameters or, for example, from the reference neural network parameters.

[0140] Further, the neural network controller may be configured to provide node information including, for example, a unique parent node identifier, e.g., a parent node identifier that is an integer, a string, and / or a cryptographic hash, and parameter update information, where the parent node identifier defines a parent node that is parameter information that serves as or is to be used as a starting point for application of the parameter update information, such that, for example, the parent node identifier can specify a parent node whose parameter information was used as a reference neural network parameter for determining the parameter update information.

[0141] In this way, not only can neural networks be trained, but training or learning procedures can also be efficiently represented or organized. The neural network controller of the present invention can provide incremental or differential training updates in the form of parameter updates to provide different neural network parameter versions of the training process. Thus, results from different learning stages may be combined, or reversion to a previous parameter set may simply be possible. Additionally, the neural network controller can be configured to set up or update a parameter update tree, for example, by providing node information, which can enable efficient neural network parameter versioning even across different devices.

[0142] Furthermore, providing node information may require only a small number of bits in the bitstream, as opposed to a complete transmission of neural network parameters.

[0143] Additionally, it should be noted that a neural network controller as described above may be based on the same considerations as the decoder and / or encoder described above, and may include all (e.g., all corresponding or all similar) features and functionality also described with respect to the decoder and / or encoder.

[0144] According to further embodiments of the present invention, the neural network controller comprises an encoder according to any embodiment as disclosed herein, or the neural network controller comprises any feature or combination of features of an encoder according to any embodiment as disclosed herein.

[0145] A further embodiment according to the present invention includes a neural network federation learning controller configured to receive node information for multiple neural networks (e.g., multiple neural networks having identical structures but somewhat different parameters, e.g., multiple neural networks trained based on the same initial neural network parameters, e.g., using different training data and / or using different training algorithms), where the node information includes, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash.

[0146] Further, the node information includes parameter update information, e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter and a newer version thereof, e.g., corresponding to a child node of the update tree, and the neural network federated learning controller is configured to combine the parameter update information of multiple corresponding nodes of different neural networks, e.g., nodes having equal parent node identifiers, to obtain combined parameter update information.

[0147] Furthermore, the neural network federated learning controller is configured to distribute the combined parameter update information, eg, in coded form, to a plurality of decoders, eg, as defined above.

[0148] Thus, a distributed learning and update structure can be provided, with the neural network associative learning controller being the central junction in the exchange of learning or update information. The inventors recognized that this can enable training structurally equivalent or even structurally different neural networks (e.g., via corresponding tensors, e.g., using implicit or explicit shape adaptation) or evaluating parameter sets of these neural networks on multiple devices, with the neural network associative learning controller being able to process the training results, e.g., by updating and / or distributing parameter update trees representing different versions of the neural network parameter sets. This can include combining, discarding, or evaluating parameters or corresponding parameter nodes, and / or determining which parameter update information associated with a neural network parameter set is provided to which device, e.g., for further training, use, or evaluation.

[0149] Additionally, it should be noted that the neural network associative learning controller as described above may be based on the same considerations as the decoder, encoder, and / or neural network controller described above, and may include all (e.g., all corresponding or all similar) features and functionality described with respect to the decoder, encoder, and / or neural network controller.

[0150] According to a further embodiment of the present invention, the neural network federated learning controller is configured to combine parameter update information of multiple corresponding nodes with equal parent node identifiers of different neural networks to obtain combined parameter update information.

[0151] Thus, the update or training results (or, for example, rather the corresponding updates) can be arithmetically combined. As a simple example, an average of the parameter updates may be provided as the combined parameter update. However, more complex combinations may also be performed, including, for example, weighting the parameters according to a figure of merit, for example, based on performance evaluations of the neural network parameters associated with the corresponding nodes.

[0152] According to a further embodiment of the present invention, the neural network federated learning controller is configured to distribute parameter information of parent nodes associated with parent node identifiers to a plurality of decoders, e.g., as defined above, and the neural network federated learning controller is configured to receive, from the decoders, node information (e.g., node information of multiple neural networks, e.g., node information of multiple neural networks having identical structures but somewhat different parameters, e.g., multiple neural networks trained using different training data and / or using different training algorithms, e.g., based on the same initial neural network parameters), e.g., including unique parent node identifiers, e.g., parent node identifiers that are integers, strings, and / or cryptographic hashes. Furthermore, the neural network federated learning controller is configured to combine parameter update information of multiple corresponding nodes having parent node identifiers.

[0153] Thus, the neural network associative learning controller can provide, for example, reference NN parameters or initial NN parameters of the parent node for modification in each decoder. Upon receiving node information from the decoders, the neural network associative learning controller can combine different training results to improve the NN training progress and / or to provide common parameters that may be more robust (e.g., resulting from training in different devices including different decoders with different data sets, or in different, e.g., real-world applications).

[0154] According to a further embodiment of the present invention, the neural network federated learning controller is configured to provide node information, e.g., in the form of an encoded bitstream, describing combined node information of the parameter update tree, the combined node information including, e.g., a parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, that is a unique parent node identifier. Furthermore, the combined node information includes combined parameter update information, e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter and a combined version thereof, e.g., obtained by combining parameter update information obtained from multiple neural network controllers, corresponding to child nodes of the update tree.

[0155] According to further embodiments of the present invention, the neural network associative learning controller includes an encoder according to any of the embodiments disclosed herein, or the neural network associative learning controller includes any feature or combination of features of an encoder according to any of the embodiments disclosed herein.

[0156] A further embodiment according to the invention includes a method for decoding parameters of a neural network, the method including obtaining a plurality of neural network parameters of the neural network based on an encoded bitstream, and obtaining, e.g., receiving, e.g., extracting, from the encoded bitstream, node information describing nodes of a parameter update tree, the node information including a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and the node information including parameter update information, e.g., one or more update instructions, e.g., a difference signal between an initial neural network parameter and a newer version thereof, e.g., corresponding to a child node of the update tree.

[0157] Further, the method includes deriving one or more neural network parameters using parameter information of the parent node identified by the parent node identifier (the parameter information, e.g., including node information of the parent node, e.g., including parameter update information and a parent node identifier of the parent node, and / or including, e.g., node parameters of the parent node, e.g., neural network parameters associated with the parent node, e.g., neural network parameters implicitly defined by the node information of the parent node, e.g., for recursive reconstruction or recursive determination or recursive calculation or recursive derivation of one or more neural network parameters), and using, e.g., the parameter update information that may be included in the node information.

[0158] A further embodiment according to the invention comprises a method for encoding parameters of a neural network to obtain an encoded bitstream, the method comprising providing node information, e.g., in the form of an encoded bitstream, describing nodes of a parameter update tree, the node information comprising, e.g., a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and the node information comprising, parameter update information, e.g., one or more update instructions, e.g., a difference signal between initial neural network parameters (e.g., associated with the parent node) and newer (current) versions thereof, e.g., corresponding to child nodes of the update tree.

[0159] Additionally, the parameter update information describes the difference between the neural network parameters associated with the parent node defined by the parent node identifier and the current neural network parameters.

[0160] A further embodiment according to the invention includes a method of controlling a neural network, the method including training the neural network to obtain updated, e.g., improved, neural network parameters based on initial neural network parameters, e.g., by performing training, and determining parameter update information based on reference neural network parameters, which may, e.g., be equal to the initial neural network parameters, and the updated, e.g., improved neural network parameters, wherein the parameter update information includes one or more update instructions that describe how to at least approximately derive the updated neural network parameters from the initial neural network parameters.

[0161] Further, the method includes providing node information including a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and parameter update information, wherein the parent node identifier defines a parent node, e.g., parameter information that serves as or is to be used as a starting point for application of the parameter update information, such that the parent node identifier can specify a parent node having parameter information used as a reference neural network parameter for determining the parameter update information.

[0162] A further embodiment according to the present invention includes a method for controlling neural network federated learning, the method including receiving node information for multiple neural networks (e.g., multiple neural networks having identical structures but somewhat different parameters, e.g., multiple neural networks trained based on identical initial neural network parameters, e.g., using different training data and / or using different training algorithms), the node information including, e.g., a parent node identifier, e.g., a unique parent node identifier, e.g., an integer, a string, and / or a cryptographic hash, and parameter update information, e.g., one or more update instructions, e.g., a difference signal between the initial neural network parameters and a newer version thereof, e.g., corresponding to a child node in the update tree.

[0163] Furthermore, the method includes combining parameter update information of multiple corresponding nodes of different neural networks, e.g., nodes having equal parent node identifiers, to obtain combined parameter update information, and distributing the combined parameter update information, e.g., in coded form, to multiple decoders, e.g., as defined above.

[0164] It should be noted that the above-described method may be based on the same considerations as the decoder, encoder, neural network controller, and / or neural network associative learning controller described above, and the method may include all (e.g., all corresponding or all similar) features and functionality described with respect to the decoder, encoder, neural network controller, and / or neural network associative learning controller.

[0165] A further embodiment according to the invention comprises a computer program for performing a method according to any of the embodiments disclosed herein when the computer program runs on a computer.

[0166] A further embodiment according to the invention includes an encoded representation of neural network parameters, the encoded representation including node information describing nodes of a parameter update tree, the node information including parent node identifiers and parameter update information.

[0167] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings: [Brief explanation of the drawings]

[0168] [Figure 1a] FIG. 2 is a schematic diagram of a decoder according to one embodiment of the present invention; [Figure 1b] FIG. 2 is a schematic diagram of a decoder with generalized node information according to one embodiment of the present invention; [Figure 2] FIG. 2 is a schematic diagram of an encoder according to one embodiment of the present invention; [Figure 3a] FIG. 4 is a schematic diagram of another encoder according to an embodiment of the present invention. [Figure 3b] FIG. 2 is a schematic diagram of an encoder with generalized node information according to an embodiment of the present invention; [Figure 4] FIG. 4 is a schematic diagram of a further encoder according to an embodiment of the present invention; [Figure 5] FIG. 10 illustrates an example of a parameter update tree PUT according to an embodiment of the present invention. [Figure 6] FIG. 1 is a schematic diagram of tensor shape transformation according to an embodiment of the present invention; [Figure 7] FIG. 1 illustrates an example of a topology change of a neural network according to an embodiment of the present invention. [Figure 8] FIG. 1 is a schematic diagram of a neural network controller according to an embodiment of the present invention. [Figure 9] FIG. 1 is a schematic diagram of a neural network federated learning controller according to an embodiment of the present invention. [Figure 10]FIG. 2 is a schematic block diagram of a method for decoding parameters of a neural network according to an embodiment of the present invention; [Figure 11] 1 is a schematic block diagram of a method for encoding parameters of a neural network to obtain an encoded bitstream according to an embodiment of the present invention; [Figure 12] FIG. 1 is a schematic block diagram of a method for controlling a neural network according to an embodiment of the present invention. [Figure 13] 1 is a schematic block diagram of a method for controlling neural network federated learning according to an embodiment of the present invention; [Figure 14] FIG. 1 is a schematic diagram of an example of a federated learning scenario according to an embodiment of the present invention. [Figure 15] FIG. 2 is a schematic diagram of an example parameter update tree, e.g., an exemplary parameter update tree, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0169] Identical or equivalent elements, or elements with identical or equivalent functionality, are designated by the same or equivalent reference numerals in the following description, even if they appear in different figures.

[0170] In the following description, numerous details are set forth to provide a more thorough explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, to avoid obscuring embodiments of the present invention. In addition, features of different embodiments described hereinafter may be combined with each other unless specifically stated otherwise.

[0171] 1a shows a schematic diagram of a decoder according to one embodiment of the present invention. Figure 1 shows a decoder 100 comprising an acquisition unit 110, a parameter update tree (PUT) information unit 120 and a derivation unit 130.

[0172] The decoder 100 may receive the encoded bitstream 102, based on which, as an example, the obtaining unit 110 may determine node information 112. The node information 112 may describe a node of the parameter update tree PUT. The node information 112 includes parent node identifier information 114, which optionally includes or is, for example, a parent node identifier, and parameter update information 116.

[0173] Using the parent node identifier information 114 , the PUT information unit 120 can be configured to determine parameter information 122 for the parent node identified by the parent node identifier information 114 .

[0174] Using the parameter update information 116 and the parameter information 122 , the derivation unit 130 can derive one or more neural network parameters 104 .

[0175] In short, information about the neural network parameters may be provided using node information 112, including parent node identifier information 114 and parameter update information 116, encoded in the encoded bitstream 102. The parent node identifier and parameter update information 116 may then be extracted by the obtaining unit 110. For example, instead of encoding the neural network parameters themselves, reference information in the form of parent node identifier information 114 and update information in the form of parameter update information 116 may be provided.

[0176] The reference information can be identified and / or extracted, for example, based on the parameter update tree using the PUT information unit 120. Here, the idea according to an embodiment can be to extract the neural network parameters of the parent node in the form of parameter information 122 and update this information using the parameter update information 116 in the derivation unit 130.

[0177] In other words, the neural network parameter(s) 104 of the current node can be derived using the derivation unit 130 using the neural network parameters of the parent node of the current node as modified by the parameter update information 116.

[0178] As an optional feature, the parameter update information 116 may be provided in a PUT information unit 120, for example.

[0179] As another optional feature, the PUT information unit 120 may be configured to provide PUT information 124 to the derivation unit 130. The PUT information 124 may optionally include information about the parameter update tree.

[0180] 1b shows a decoder 100b comprising an obtaining unit 110b, a PUT information unit 120b, and a derivation unit 130b. The obtaining unit 110b may be configured to obtain, for example, generalized node information 112b. The information 112b may optionally be equal to or similar to the node information 112, including, for example, parent node identifier information and parameter update information. However, the generalized node information 112b may optionally include additional information, such as, for example, a node identifier and / or signaling whether a node identifier is present and / or signaling including information on the type of parent node identifier, for example in the form of a syntax element, topology change information, and / or topology change signaling, as will be described in more detail below.

[0181] Furthermore, the PUT information unit 120b can provide PUT information 132 to the derivation unit 130b. The PUT information 132 may be, for example, the parameter information 122 of the parent node as shown in Figure 1. Optionally, the update tree information 132 may include optional PUT information, for example, 124 (see Figure 1).

[0182] Thus, the acquisition unit 110b may be configured to acquire or provide such generalized node information 112b from the encoded bitstream 102, and the PUT information unit 120b may be configured to use the generalized node information 112b to provide or determine the update tree information 132, for example, based on the PUT.

[0183] Thus, the derivation unit 130b may be configured to use the generalized node information 112b and the PUT information 132 to derive one or more neural network parameters 104b.

[0184] 2 shows a schematic diagram of an encoder according to one embodiment of the present invention. Figure 2 shows an encoder 200 comprising a node information unit 210 and a bitstream unit 220.

[0185] 2, the encoder 200 may optionally receive neural network parameter(s) 204 and / or parameter update tree (PUT) information 206. Based on the neural network parameter(s) 204 and the PUT information 206, the node information unit 210 may provide node information 212 describing nodes of the parameter update tree. The node information 212 may include parent node identifier information 214, which optionally includes a parent node identifier, and parameter update information 216, which describes differences between the neural network parameters associated with the parent node defined by the parent node identifier information 214 and the current neural network parameters 204.

[0186] Using the node information 212, the bitstream unit 220 can provide an encoded bitstream 202, which includes encoded neural network parameters 204.

[0187] 2, the concept of encoder 200 may be to encode neural network parameter(s) 204 using a reference and difference or variance information with respect to the reference, rather than simply using their respective values. The reference may be a set of neural network parameters identified by a parent node of the parameter update tree, as indicated by parent node identifier information 214. The difference or variance information of the neural network parameters 204 relative to the neural network parameters of the parent node may be parameter update information 216.

[0188] Thus, the encoded bitstream 202 can enable a corresponding decoder, which may have information about the parameter update tree available to identify the corresponding parent node and therefore its neural network parameters, to adapt or modify them to determine the neural network parameter(s) 204.

[0189] Figure 3a shows a schematic diagram of another encoder according to an embodiment of the present invention. Figure 3 shows an encoder 300 comprising a parameter update tree (PUT) unit 310 and a bitstream unit 320.

[0190] In this example, encoder 300 may receive neural network parameter(s) 304. To encode parameter(s) 304, encoder 300 may, for example, use PUT unit 310 to determine node information 312 including parent node identifier information 314, optionally including a parent node identifier, and parameter update information 316, such that parameter(s) 304 may be represented as neural network parameters (represented as parent node identifier information 314) and modification or update information (represented as parameter update information 316) of a parent node in a parameter update tree.

[0191] 2, the bitstream unit 320 can encode the node information 312 in the encoded bitstream 302. As an example, the encoder 300, in contrast to the encoder 200, can be configured to provide the node information 312 based on the parameters 304 as an input signal, by way of example only. Accordingly, the PUT unit 310 can include parameter update tree information to provide an alternative representation of the parameters 304 in the form of update information 316 and an identifier.

[0192] Figure 3b shows a schematic diagram of an encoder with generalized node information according to one embodiment of the present invention. Figure 3b shows an encoder 300b comprising a parameter update tree (PUT) unit 310b and a bitstream unit 320b. The PUT unit 310b can be configured to provide generalized node information 312b based on or using the neural network parameter(s) 304, which information 312b can optionally be equal to or similar to the node information 312, including, for example, parent node identifier information and parameter update information.

[0193] However, the generalized node information 312b may optionally include additional information, such as, for example, a node identifier and / or signaling whether a node identifier is present and / or signaling including information about the type of parent node identifier, for example in the form of a syntax element, topology change information, and / or topology change signaling, as will be described in more detail below.

[0194] Using the generalized node information 312b, the bitstream unit 320b can provide the encoded bitstream 302b.

[0195] Figure 4 shows a schematic diagram of a further encoder according to an embodiment of the invention. Figure 4 shows an encoder 400 comprising a PUT unit 410 and a bitstream unit 420.

[0196] As an optional example, the PUT unit 410 may include a parameter update tree, e.g., as described in the context of FIG. 5. The PUT unit 410 may provide node information 412 including parent node identifier information 414, which optionally includes a parent node identifier, and parameter update information 416, which is provided to the bitstream unit 420 to obtain the encoded bitstream 402. Thus, for example, current neural network parameters associated with a particular node of the parameter update tree may be encoded in the form of parent node identifier information 414, which provides information about reference parameters, and parameter update information 416, which provides information about how to modify the reference parameters, to represent the current neural network parameters associated with the node. Thus, the parameter update information 416 may describe differences between the neural network parameters associated with the parent node defined by the parent node identifier and the current neural network parameters.

[0197] As an optional feature, the encoder 400, e.g., the PUT unit 410, may be configured to provide a plurality of node information blocks 418, where a parent node identifier of a first node information block refers to a root node, and the parameter update information of the first node describes a difference between the neural network parameters of the first node and the neural network parameters of the root node defined by the parent node identifier of the first node information block, and a parent node identifier of an Nth node information block refers to an N-1th node, and the parameter update information of the Nth node describes a difference between the neural network parameters of the N-1th node defined by the parent node identifier of the Nth node information block and the neural network parameters of the Nth node.

[0198] Briefly, the encoder 400 can be configured to provide information about each parameter update tree, for example, via the encoded bitstream 402 .

[0199] Figure 5 shows an example of a parameter update tree PUT according to an embodiment of the present invention. Figure 5 shows a PUT 500 that includes a root node R 510 and multiple child nodes, some of which are examples 520, 530, 540, 550, and 560.

[0200] As a simplified example, the root node R 510 is configured with four neural network parameters a, represented by tensor 512, for example. 11 =a 12 =a 21 =a 22 = 0. 11 , a 12 , a 21 , a 22 may be, for example, a parameter value associated with node R.

[0201] Child node U1 530 thus stores, by way of example, a set of four neural network parameters represented by tensor 532, e.g., parameter values ​​a 11 =a 12 =a 22 =0 and a 21 =1.

[0202] The concept according to the embodiment can be as follows: four neural network parameters a 11 =a 12 =a 22 =0 and a 21Instead of transmitting all of the n = 1s, only reference information in the form of parent node identifier 534 and parameter update information 536 may be transmitted. In the simplified example of Figure 5, parameter update information 536 may be four change values, represented by additive tensor 536 as an example. In many applications, many NN parameters may not change dramatically between training cycles, and therefore many change values ​​may be zero or quantized to zero, as shown in tensor 536. Such update information can therefore be efficiently coded (e.g., zero-compressed).

[0203] Thus, if both the encoder and the corresponding decoder contain information about the neural network parameters associated with the root node R 510, e.g., the initial or default neural network parameters, it is not necessary to fully encode the neural network parameters 532 associated with node U1 530, e.g., the neural network parameters after the first training step of the neural network, but only the reference information 534 and the difference information 536.

[0204] Thus, an encoder can transmit node information in an encoded bitstream including parameter update information 536 and parent node identifier 534, and a corresponding decoder can then determine neural network parameters 532 as the sum, e.g., element-wise sum, of neural network parameters 512 associated with root node R 510 identified by parent node identifier 534 and parameter update information 536, so that, e.g., tensor 512 + tensor 536 = tensor 532.

[0205] Thus, node U2 520 may be associated with neural network parameters 522, and the parent node of U2 is node R 510, and therefore parent node identifier 524 may be a pointer to root node R 510.

[0206] The node 540 associated with the neural network parameters 542 may be a child node of the node R 510, and the parent node identifier 544 may be a pointer to the root node R 510.

[0207] In contrast, nodes 550, 560 may be child nodes of node U2 520, and therefore their parent node identifiers 554, 564 may identify U2 520. U3 550 may be associated with neural network parameters 552 and may include parameter update information 556 for its parent node U2 520. Conversely, U4 560 may be associated with neural network parameters 562 and may include parameter update information 566 for its parent node U2 520.

[0208] Further elements of Figure 5 will be explained in more detail step by step with respect to further embodiments of the invention.In the following, further embodiments according to the invention will be explained with reference to Figures 1 to 5.

[0209] According to the above description, as an optional feature, an encoder, e.g., 200, 300, 300b and / or 400, may be configured to determine, e.g., using node information unit 210 or PUT unit 310, 310b, 410, a difference between one or more neural network parameters, e.g., 512, defined by a parent node, e.g., R510, identified by a parent node identifier, e.g., 534, and one or more current neural network parameters, e.g., 532, to obtain parameter update information, e.g., 536.

[0210] As another optional feature, inventive decoders, e.g., 100, 100b, e.g., PUT information units 120, 120b, and / or inventive encoders, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT units 310, 310b, 410, can optionally be configured to set up a parameter update tree 500, in which multiple child nodes (e.g., 520, 530, 540, 550, 560, to name but a few) containing different parameter update information (e.g., 526, 536, 546, 556, 566) are associated with a common parent node, e.g., R 510. Thus, as shown in FIG. 5, some nodes, e.g., U3 550 and U4 560, may be associated with a common parent node R via different, e.g., intermediate, nodes, e.g., U2 520.

[0211] As an example, a PUT information unit (e.g., 120, 120b) of the present invention can include a parameter update tree and / or can be configured to, for example, set up a parameter update tree. Thus, the PUT information unit can optionally receive parameter update information 116 to set up or update a corresponding PUT, as previously described.

[0212] An encoder of the present invention, e.g., 200, 300, 300b and / or 400, may be configured to set a PUT using, e.g., node information unit 210 or PUT unit 310, 310b and / or 410, respectively. These units may be configured to set, store and / or update a PUT.

[0213] As another optional feature, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide node information 212, 312, 312b and / or 412 (e.g., node information including parameter update information 556 and parent node identifier 554), such that one or more neural network parameters, e.g., 552, associated with a currently considered node, e.g., U3 550, may be obtained using parameter information, e.g., 512, associated with a root node, e.g., R 510, and using parameter update information, e.g., 526, associated with one or more intermediate nodes, e.g., U2 520, between the root node and the currently considered node in the update tree, e.g., U2 520, using parameter update information, e.g., 556, associated with the currently considered node.

[0214] Thus, as an example, to obtain the neural network parameters associated with node U3 550, the neural network parameters 512 of root node R 510 may be modified along tree path R-U2-U3 using parameter updates 526 and 556 such that tensor 512 + tensor 526 + tensor 556 = tensor 552 (with element-wise sum, as an example) representing the neural network parameters of node U3.

[0215] Thus, as an optional feature, a decoder, e.g., 100, 100b, e.g., derivation unit 130, 130b, of the present invention may be configured to use parameter information, e.g., 512, associated with a root node, e.g., R 510, to use parameter update information, e.g., 526, associated with one or more intermediate nodes, e.g., U2 520, between the root node and the currently considered node in the update tree, to use parameter update information, e.g., 556, associated with the currently considered node, e.g., U3 550, to obtain one or more neural network parameters, e.g., 104, 104b, corresponding to, e.g., 552, associated with the currently considered node, e.g., U3 550.

[0216] Thus, for example, when PUT information unit 120, 120b includes information about a PUT, PUT information unit 120, 120b can provide PUT information 132 (or optional information 124) to a derivation unit, e.g., 130, 130b, including parameter update information, e.g., 526, and optionally a parent node identifier, e.g., 524, of an intermediate node, e.g., U2 520, to obtain one or more neural network parameters, e.g., 104, 104b, corresponding to, e.g., 552. In this case, as an example, encoded bitstream 102 may include only the parameter update information, e.g., 556, of the currently considered node and the parent node identifier, e.g., 554, of the currently considered node (which can be used together with the PUT to determine a path through the PUT to determine the neural network parameters, e.g., 552, of the currently considered node, e.g., U3 550).

[0217] As another, e.g., alternative, optional feature, the encoded bitstream 102 may include information about the PUT, e.g., 500, and / or information about the path of the PUT, e.g., R-U2-U3, such that the parameter update information, e.g., 116, provided by the acquisition unit, e.g., 110, 110b, may include parameter update information, e.g., 556 and 526, of the currently considered node, e.g., U3 560, and of the intermediate node, e.g., U2 520. Thus, parent node identifiers, e.g., 554 and 524, may be provided.

[0218] Therefore, an inventive encoder, e.g., 200, 300, 300b and / or 400, can be configured to provide such an encoded bitstream 202, 302, 302b and / or 403 that includes parameter update information and parent node identifiers of the currently considered node and intermediate nodes.

[0219] Furthermore, as another optional feature, for example, a PUT information unit 120, 120b containing a PUT, for example 500, may be configured to traverse (e.g., using the PUT information unit 120, 120b) a parameter update tree, for example 500, from a root node, for example R 510 to a currently considered node, for example U3 550, and apply an update instruction, for example 526, of a visited node, for example U2 520, to one or more initial neural network parameters, for example 512, to obtain one or more neural network parameters, for example 552, associated with the currently considered node, for example U3 550.

[0220] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., a PUT information unit 120, 120b including a PUT, e.g., 500, may be configured to aggregate, e.g., multiple consecutive nodes of a parameter update tree, e.g., 500, e.g., nodes U2 520 and U3 550, and / or one or more consecutive nodes of the parameter update tree, e.g., nodes U2 520 and U3 550, with parameter update information, e.g., 526, 556.

[0221] 5, nodes U2 520 and U3 550 may be aggregated or merged into a new node U23 540. The neural network parameters associated with U3, i.e., parameters 552, may therefore be equal to the neural network parameters associated with U23, i.e., parameters 542. Parameter updates 546 for node U23 may therefore be a combination (element-wise sum, in the simple example of FIG. 5) of parameter updates 526 and 556. Parent node identifier 544 may therefore refer to the same node or may be equal to parent node identifier 524.

[0222] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b, can optionally be configured to update a parameter update tree, e.g., 500, based on node information, e.g., 112, 112b. As an example, PUT information unit 120, 120b can optionally include information about the parameter update tree, e.g., 500. The parameter update tree may be adapted, for example, by adding a new node, e.g., node U4 560, to the parameter update tree. This may include adding a corresponding parent node identifier, e.g., 564. Optionally, parameter update information 116 may be attested to PUT information unit 120, 120b, such that parameter update information 116 corresponding to, e.g., tensor 566, may also be added to the PUT. Thus, information about a new node, e.g., U4 560, i.e., parameter update information, e.g., 566, and parent node identifier, e.g., 564, may be provided in the bitstream, e.g., 102, optionally together with signaling indicating that a new node is being added. Alternatively, a decoder of the present invention, e.g., 100, may be configured to add such a node autonomously.

[0223] As another optional feature, the decoder 100, 100b of the present invention may be configured to decide to select the neural network parameters associated with the root node, e.g., R510, or to select the neural network parameters associated with one of the descendant nodes of the root node.

[0224] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., bitstream unit 220, 320, 320b, 420, may optionally be configured to provide signaling, e.g., coded bitstream 202, 302, 302b and / or 402, or a signal encoded in the coded bitstream, to a decoder, e.g., 100, 100b, for selectively choosing neural network parameters associated with the root node, e.g., R510, or neural network parameters associated with one of the descendant nodes of the root node.

[0225] As previously mentioned, the element-wise sum of tensors may be just one, eg, simple, example of processing parameter updates according to embodiments of the present invention, as explained above.

[0226] 5, the parameter update tree 500 further includes a node U5 570 associated with a parameter value, optionally in the form of a tensor, e.g., a neural network parameter 572. Node U5 570 is a child node of node U1 530, as indicated by parent node information 574. As another optional feature, the parameter update information 576 of node U5 includes scaling.

[0227] Thus, as another optional feature, the parameter update information 116, 216, 316, and / or 416 may include update instructions that define, for example, scaling of one or more parameter values ​​associated with a parent node of the currently considered node.

[0228] Furthermore, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b, may optionally be configured to apply scaling defined by update instructions, e.g., 576, to obtain one or more neural network parameters, e.g., 104, 104b, corresponding to tensor 572, associated with a currently considered node, e.g., U5 570; correspondingly, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 or PUT unit 310, 310b, 410, may be configured to determine scaling based on one or more parameter values ​​associated with a parent node of the currently considered node, e.g., U1 530, and a parameter value, e.g., 572, of the currently considered node, e.g., U5.

[0229] For purposes of explanation, scaling 576 may simply refer to doubling the parameter value 532 of parent node U1 to obtain the parameter value 572 of node U5.

[0230] 5, the parameter update tree 500 further includes a node U0 580 associated with a parameter value, optionally in the form of a tensor, e.g., a neural network parameter 582. The node U0 580 is a child node of the node R 510, as indicated by the parent node information 584. As another optional feature, the parameter update information 586 of the node U0 includes an additive change value, e.g., +3.

[0231] Therefore, as an optional feature, the parameter update information 116, 216, 316 and / or 416 (and thus the generalized node information 112b, 312b) may include one or more parameter values, e.g., a, of a tensor 512 associated with a parent node, e.g., R 510, of the currently considered node, e.g., U0 580. 12The node may include update instructions that define adding one or more modification values ​​to, for example, a modification value of 3, and / or subtracting one or more modification values ​​from one or more parameter values ​​associated with a parent node of the node currently under consideration.

[0232] Thus, a decoder of the present invention, e.g., 100, 100b, e.g., a derivation unit 130, 130b, may optionally be configured to apply an addition or subtraction of a modification value defined by an update instruction to obtain one or more neural network parameters associated with the currently considered node.

[0233] As an optional example, a tensor subtraction with parameter update information 566 of node U4 560 is shown in Figure 5. Thus, embodiments may include addition or subtraction, e.g., element-wise addition or subtraction, e.g., in tensor or matrix form.

[0234] 5, the parameter update tree 500 further includes a node U8 590 associated with a parameter value, optionally in the form of a tensor, e.g., a neural network parameter 592. Node U8 590 is a child node of node U23 540, as indicated by parent node information 594. As another optional feature, the parameter update information 596 of node U8 is a product tensor.

[0235] In other words, optionally, a plurality of neural network parameters, e.g., 592, associated with the currently considered node, e.g., U8, may be represented by a parameter tensor, and a decoder, e.g., 100, e.g., derivation unit 130, 130b of the present invention may optionally be configured to apply a product tensor, e.g., 596, to a parameter tensor, e.g., 542, to obtain a parameter tensor, e.g., 592, associated with the currently considered node, e.g., 590, using, implementing, or based on, element-wise multiplication of tensor elements, e.g., as shown as a simple variant.

[0236] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 or PUT unit 310, 310b, 410, may be configured to provide a product tensor, e.g., 596, to apply to a parameter tensor, e.g., 542, to obtain a parameter tensor, e.g., 592, associated with the currently considered node, e.g., U8 590.

[0237] Additionally, as another optional feature, multiple neural network parameters associated with a parent node may be represented by parameter tensors, and the parameter update information 116, 216, 316 and / or 416 (and thus the generalized node information 112b, 312b) may optionally include a product tensor, e.g., 596.

[0238] Furthermore, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b, may optionally be configured to apply a product tensor, e.g., 596, to a parameter tensor, e.g., 542, of a parent node, e.g., 540, to obtain a parameter tensor, e.g., 592, associated with the currently considered node, e.g., U8 590.

[0239] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may be configured to provide a product tensor, e.g., 596, in such a way that application of the product tensor to a parameter tensor, e.g., 542, of a parent node, e.g., 540, results in a parameter tensor, e.g., 592, associated with the currently considered node, e.g., U8 590.

[0240] 5, the parameter update tree 500 further includes a node U7 600 associated with parameter values, e.g., neural network parameters 602, optionally in the form of tensors. Node U7 600 is a child node of node U0 580, as indicated by parent node information 604. As another optional feature, the parameter update information 606 for node U8 may include one or more parameter values, e.g., a tensor of 582, associated with the parent node, e.g., U0 580, of the currently considered node, e.g., U7 600. 12 and one or more modification values.

[0241] Thus, optionally, parameter update information, e.g., 116, 216, 316 and / or 416 (and thus generalized node information 112b, 312b) may include one or more parameter values, e.g., a of 582, associated with a parent node, e.g., U0 580, of the currently considered node, e.g., U7 600. 12 and one or more modification values.

[0242] Thus, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b, may optionally be configured to apply a weighted combination of one or more parameter values ​​associated with a parent node of the currently considered node and one or more modification values ​​to obtain one or more neural network parameters associated with the currently considered node.

[0243] Again, it should be noted that any additional information required for processing, optional PUT information 124, may be provided by PUT information unit 120 to derivation unit 130. Thus, PUI information 132 may include such additional information.

[0244] 5, the parameter update tree 500 further includes a node U6 610, which is optionally associated with a parameter value in the form of a tensor, e.g., a neural network parameter 612. Node U6 610 is a child node of node U1 530, as indicated by parent node information 614. As another optional feature, the parameter update information 616 for node U6 may include one or more parameter values ​​associated with U6's parent node, in this case, parameter values ​​a of tensor 532. 12 with one or more modification values, in this case one modification value 5.

[0245] Thus, the parameter update information 116, 216, 316, and / or 416 (and thus the generalized node information 112b, 312b) of the present invention may optionally include update instructions, e.g., 616, that define the replacement of one or more parameter values ​​associated with a parent node, e.g., U1 530, of the currently considered node, e.g., U6 610, with one or more modified values.

[0246] Therefore, a decoder of the present invention, e.g., 100, 100b, e.g., a derivation unit 130, 130b, may optionally be configured to replace one or more parameter values ​​associated with a parent node of the currently considered node with one or more replacement values ​​in order to obtain one or more neural network parameters associated with the currently considered node.

[0247] 5, the parameter update tree 500 further includes a node U9 620 associated with a parameter value, optionally in the form of a tensor, e.g., a neural network parameter 612. The node U9 620 is a child node of the node U6 610, as indicated by the parent node information 624.

[0248] As an optional feature, as shown in the example of U9 620, neural network parameters associated with the currently considered node, e.g., a parent node of U9, may be represented by a parameter tensor, e.g., 612, neural network parameters associated with the currently considered node may be represented by a parameter tensor, e.g., 622, and change values ​​may be represented by a sum tensor, e.g., a sum tensor of parameter updates 626.

number

[0249] Further, as an example, an inventive encoder, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may multiply an element, e.g., 612, of a parameter tensor associated with a parent node, e.g., U6 610, of a currently considered node, e.g., U9 620, by a node tensor weight value, e.g., as indicated by a factor of ·2 in parameter update information 626, to obtain a scaled parameter tensor, and a sum tensor, e.g., the sum tensor in parameter update information 626.

number

[0250] It should be noted that the embodiment according to FIG. 5 may be a simple embodiment for illustrative purposes, as significantly more complex parameter update information may be used to represent, for example, a particular version of neural network parameters.

[0251] As an optional feature, for example, as previously described, multiple neural network parameters associated with a parent node of the currently considered node may be represented by a parameter tensor, and the parameter update information 116, 216, 316 and / or 416 (and thus the generalized node information 112b, 312b) may optionally include update instructions in the form of an update tensor.

[0252] Furthermore, the decoders of the present invention, eg, 100, 100b, eg, the derivation units 130, 130b, can optionally be configured to convert the shape of the update tensor according to the shape of the parameter tensor of the parent node.

[0253] Therefore, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide an update tensor such that the shape of the update tensor differs from the shape of the parameter tensor of the parent node.

[0254] For further explanation, reference is made to Figure 6, which shows a schematic example of tensor shape transformation according to an embodiment of the present invention. Figure 6 shows an example of a currently considered layer 630 of neurons of a neural network and a previous layer 640 of the neural network.

[0255] As an example, tensor 650 may contain neural network parameters, e.g., weights, associated with the parent node of the currently considered node, with tensor elements of parameter tensor 650 arranged along a first direction (e.g., along rows 652 of the tensor), e.g., a 11 , a 12 , a 13can be associated with the contribution of the output signals of multiple neurons 632, 634, 636 of the previous layer 630 of the neural network to the input signal of a given neuron, e.g., 642, of the currently considered layer 640 of the neural network, and are represented by tensor elements of the parameter tensor arranged along a second direction (e.g., along the columns 654 of the tensor), e.g., a 12 , a 22 , a 32 is associated with the contribution of the output signal of a given neuron, e.g., 634, of a previous layer 630 of the neural network to the input signals of multiple neurons 642, 644, 646 of the currently considered layer 640 of the neural network. In the example of Figure 6, for the sake of simplicity, not all weights between layers are shown (placeholder *).

[0256] Optionally, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b or acquisition unit 110, 110b, may be optionally configured to expand the dimension of the update tensor 660 in a first direction 652 if the expansion or dimension of the update tensor in the first direction (e.g., row direction) is smaller than the dimension of the parameter tensor 650 in the first direction. Alternatively, or in addition, the decoder may be configured to expand the dimension of the update tensor 660 in a second direction 654 if the expansion or dimension of the update tensor in the second direction (e.g., column direction) is smaller than the dimension of the parameter tensor 650 in the second direction.

[0257] Thus, an augmented update tensor 670 may be provided such that the augmented update tensor 670 may be combined with the parameter tensor 650 to modify the neural network parameters of the parent node to determine the neural network parameters of the current node.

[0258] It should be noted that in general, the nodes and corresponding tensors may represent neural network parameters for a layer of the neural network, and / or for the entire neural network, and thus for multiple layers.

[0259] Thus, an encoder of the present invention, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, can optionally be configured to provide update tensor 660 such that the extension or dimension of update tensor 660 in a first direction 652 (e.g., row direction) is smaller than the dimension or extension of parameter tensor 650 in the first direction. Alternatively, or in addition, the encoder can be configured to provide update tensor 660 such that the extension or dimension of update tensor 660 in a second direction 654 (e.g., column direction) is smaller than the dimension or extension of parameter tensor 650 in the second direction.

[0260] As shown in FIG. 6, the update tensor 660 may be, for example, a change value u 11 , u 12 , u 21 and u 22 As shown with the augmented update tensor 670, optionally, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b, may be optionally configured to copy row entries of the update tensor 660 to obtain one or more augmented row entries of the reshaped update tensor 670 when the number of rows of the update tensor 660 is smaller than the number of rows of the parameter tensor 650.

[0261] Alternatively, or in addition, the decoder may be configured to copy column entries of update tensor 660 to obtain one or more expanded column entries of reshaped update tensor 670 if the number of columns of the update tensor is smaller than the number of columns of parameter tensor 650.

[0262] For example, as shown in FIG. 6, as a simplified example, the first row of update tensor 660 may be duplicated to provide the third row of augmented update tensor 670, and the first column of update tensor 660 may be duplicated to provide the third column of augmented update tensor 670.

[0263] As optionally shown, a decoder of the present invention, e.g., 100, e.g., derivation unit 130, 130b, may optionally be configured to copy one or more entries of the update tensor row-wise and column-wise to obtain entries of the reshaped update tensor 670.

[0264] However, it should be noted that only rows or columns may be copied, or only certain entries in rows and / or columns may be copied.

[0265] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, can optionally be configured to provide update tensor 660 such that the number of rows of the update tensor is smaller than the number of rows of parameter tensor 650. Alternatively, or in addition, the encoder can be configured to provide update tensor 660 such that the number of columns of the update tensor is smaller than the number of columns of parameter tensor 650.

[0266] As another optional feature, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide information regarding an extension of the update tensor. This information may, for example, be additionally encoded in the bitstream, e.g., 202, 302 and / or 402. The node information 112, 112b, 212, 312, 312b and / or 412 of the present invention may optionally include such extension information.

[0267] Therefore, a decoder of the present invention, e.g., 100, 100b, e.g., derivation unit 130, 130b, may optionally be configured to determine the need to transform the shape of the update tensor and / or the degree of transformation of the shape of the update tensor depending on information about the expansion of the update tensor.

[0268] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., acquisition unit 110, 110b, may optionally be configured to determine whether parent node identifier information, e.g., 114, is present, e.g., in the encoded bitstream 102; if the parent node identifier is present, the decoder may be configured to derive one or more neural network parameters, e.g., 104, 104b, according to any embodiment disclosed herein; and further, if the parent node identifier is not present, the decoder may be configured to make the currently considered node the root node.

[0269] 1 and 1b, parent node identifier information 114 (or generalized node information 112b, respectively) may include, for example, information on the presence or absence of a parent node identifier instead of or in addition to a parent node identifier (e.g., if present), so that the parameter update tree established and stored by PUT information unit 120, 120b can be adapted accordingly. This may make it possible to discard portions of a PUT that may no longer be needed, for example, because the corresponding parameter set may be outdated or inferior (e.g., worse) than a newer parameter set. Thus, in simple terms, the upper portion of the PUT may be discarded so that the new root node of the PUT is the currently considered node. On the other hand, in this way, a new PUT may be established starting from the currently considered node.

[0270] Thus, an encoder of the present invention, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide signaling including information on whether a parent node identifier is present. Thus, with reference to Figures 2-4, such signaling may be included instead of parent node identifier information 214, 314, and / or 414, e.g., instead of or in addition to the parent node identifier.

[0271] As another optional feature, the nodes of the PUT (e.g., node R, U0-U9 510-620) may be associated with respective hash values. A decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b, may optionally be configured to compare parent node identifiers (parent node identifier information 114 (or generalized node information 112b, respectively) may include parent node identifiers) with hash values ​​associated with one or more nodes to identify the parent node of the currently considered node.

[0272] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide a hash value associated with the node as a parent node identifier (e.g., within parent node identifier information 214, 314 and / or 414, or within generalized node information 112b, 312b, e.g., encoded in bitstream 202, 302, 302b and / or 402, respectively) to identify the parent node of the node currently under consideration.

[0273] Optionally, the hash value may be a hash value of a full compressed data unit NDU associated with one or more previously decoded nodes.

[0274] As another optional feature, the hash value may be a hash value of the payload portion of the compressed data unit associated with one or more previously encoded nodes, while leaving data size information and header information unaccounted for.

[0275] As another optional feature, the parent node identifier may be a combined value representing a device identifier and a serial number, both associated with the parent node.

[0276] As another optional feature, the parent node identifier can identify an update tree, e.g., 500, and / or a layer of a neural network. A PUT can, for example, represent a portion of the neural network parameters of a neural network. Thus, in the case of a neural network, multiple update trees may be established, and thus distinguishing between the trees can be advantageous. As an example, a PUT can represent one layer of a neural network.

[0277] As another optional feature, the node information, e.g., 112, 112b, 212, 312, 312b, and / or 412, may include a node identifier. A decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b, may optionally be configured to store the node identifier. Accordingly, an encoder of the present invention, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to store and / or provide the node identifier.

[0278] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b, may optionally be configured to compare one or more stored node identifiers with parent node identifiers in the node information of the new node when adding the new node, in order to identify the parent node of the new node.

[0279] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to compare one or more stored node identifiers with parent node identifiers in the node information of a new node when adding the new node, in order to identify the parent node of the new node.

[0280] As another optional feature, the node identifier can identify the update tree, e.g., 500, with which the node information, e.g., 112, 112b, 212, 312, 312b, and / or 412, is associated and / or the layer of the neural net to which the node information pertains. Because neural networks can contain millions of parameters, only the parameter selection can be organized into one single parameter tree. Also, by using information about which neural network layer the neural network parameter being searched for is associated with as the parameter associated with the parent node, search time in the encoder or decoder can be reduced.

[0281] As another optional feature, for example, the node identifiers in the node identifier information 114, 214, 314, 414 and generalized node information 112b, 312b, respectively, can include device identifiers and / or parameter update tree depth information and / or parameter update tree identifiers. A single neural network can be trained on different devices, thereby making available different sets of parameters, e.g., even different iterations of such sets of parameters. Thus, information about device identifiers can enable efficient indication of specific sets of neural network parameters. The PUT depth information can reduce the time required, e.g., to find the corresponding parent node, to determine the neural network parameters of the currently considered node, since it may not be necessary to search through all layers of the PUT.

[0282] As another optional feature, the node information, eg, 112, 112b, 212, 312, 312b and / or 412, may include signaling indicating whether a node identifier is present.

[0283] As another optional feature, the parent node identifier (e.g., encoded in bitstream 202, 302 and / or 402, e.g., in parent node identifier information 114, 214, 314 and / or 414, and generalized node information 112b, 312b, respectively) is a combined value representing a device identifier and a serial number, both associated with the parent node.

[0284] As another optional feature, the parent node identifier information 114, 214, 314 and / or 414 and the generalized node information 112b, 312b, respectively, may optionally include information regarding the type of parent node identifier. Thus, a decoder of the present invention, e.g., 100, 100b, e.g., an acquiring unit 110, 110b, may be configured to acquire signaling including information regarding the type of parent node identifier, and the decoder may optionally be configured to evaluate the signaling to take into account the respective type of parent node identifier.

[0285] Therefore, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide signaling including information about the type of parent node identifier.

[0286] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., a PUT information unit 120, 120b, may optionally be configured to selectively evaluate syntax elements indicating the type of parent node identifier depending on syntax elements indicating the presence of a parent node identifier.

[0287] Therefore, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to selectively provide a syntax element indicating the type of parent node identifier when a syntax element describing the parent node identifier is present.

[0288] 7 shows an example of a topology change of a neural network according to an embodiment of the present invention. FIG. 7 shows a first topology 710 of a neural network section having neurons 712, and a neural network parameter a 11 , a 12 , a 21 , and a 22 , e.g., weights, can be represented by parameter tensor 720. As an example, the topology of a neural network section can change to a second topology 730 with a new node 732 and additional parameters b1 and b2. The neural network parameters of the neural network with topology 730 can be represented by tensor 740.

[0289] As an optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., acquisition unit 110, 110b, may optionally be configured to acquire topology change signaling in node information, including information about a topology change of the neural network. As an example, parent node identifier information, e.g., 114, may optionally include the topology change signaling. Furthermore, the decoder may be configured to modify parameter information of the parent node according to the topology change to derive one or more neural network parameters, e.g., as represented by tensor 740, of the neural network having the modified topology.

[0290] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to provide topology change signaling that includes information about a topology change of the neural network within the node information, e.g., within parent node identifier information 214, 314 and / or 414.

[0291] As another optional feature, the decoders of the present invention, e.g., 100, 100b, e.g., PUT information units 120, 120b and / or e.g., derivation units 130, 130b, may optionally be configured to modify the shape of one or two tensors in response to the topology modification information. As an example, the tensors containing the modification values ​​may be adapted to the new shape of the parent node according to the new neural network topology.

[0292] Therefore, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, can optionally be configured to signal changes in the shape of one or two tensors along with signaling topology changes.

[0293] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b, may optionally be configured to change the number of neurons in a given layer in response to topology change information. As an example, a decoder on a device implementing a neural network may receive topology change information and thus adapt the structure of the neural network, e.g., in addition to adapting neural network parameters, e.g., weight values.

[0294] Thus, an encoder of the present invention, e.g., 200, 300, 300b, and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to signal a change in the number of neurons in a given layer using topology change information, which may be included in generalized node information, e.g., 312b.

[0295] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., PUT information unit 120, 120b and / or e.g., derivation unit 130, 130b, may optionally be configured to replace one or more tensor values ​​of one or more tensors whose shape is to be changed and associated with a parent node of the currently considered node with one or more replacement tensors to obtain one or more tensors having a modified size, or the decoder may be configured to replace one or more tensors whose shape is to be changed and associated with a parent node of the currently considered node with one or more replacement tensors to obtain one or more tensors having a modified size.

[0296] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to signal replacement of one or more tensor values ​​of one or more tensors whose shape is to be changed and associated with a parent node of the currently considered node, with one or more replacement values, or the encoder may be configured to signal replacement of one or more tensors whose shape is to be changed and associated with a parent node of the currently considered node, with one or more replacement tensors.

[0297] As another optional feature, a decoder of the present invention, e.g., 100, 100b, e.g., a derivation unit 130, 130b, may optionally be configured to change the shapes of two tensors in two update trees associated with adjacent layers of the neural net in a synchronized manner in response to topology change signaling.

[0298] Thus, an encoder of the present invention, e.g., 200, 300, 300b and / or 400, e.g., node information unit 210 and / or PUT unit 310, 310b, 410, may optionally be configured to use topology change signaling to synchronize and signal changes to the shape of two tensors in two update trees associated with adjacent layers of a neural net.

[0299] 8 shows a schematic diagram of a neural network controller according to an embodiment of the present invention. FIG. 8 shows a neural network controller 800 comprising a training unit 810, a reference unit 820, a parameter update information (PUI) unit 830, and a node information providing unit 840.

[0300] The training unit 810 may be configured to train the neural network to obtain updated neural network parameters 812 based on initial neural network parameters. As an example, the initial neural network parameters may be default parameters or, for example, a first set of neural network parameters from which the neural network may be trained. As an optional feature, the initial neural network parameters may be provided to the training unit 810 using or by a reference unit 820. The initial neural network parameters may be stored in the reference unit 820 or may be initially provided to the neural network controller 800, for example.

[0301] As an example, a second training can be performed based on the updated neural network parameters. Thus, the updated neural network parameters can be, for example, reference parameters or starting parameters for the second training. Such reference parameters can be stored in the reference unit 820. Thus, in the first step, the reference parameters, for example, the parameters on which the training is performed, can be equal to the initial neural network parameters.

[0302] Furthermore, the PUI unit 830 may be configured to determine parameter update information (PUI) 832 based on the reference neural network parameters 822 and the updated neural network parameters 812. Accordingly, the reference unit 820 may provide the reference neural network parameters 822, e.g., the parameters on which training was performed to obtain updated neural network (NN) parameters, to the PUI unit 830.

[0303] PUI 832 may also include, for example, one or more update instructions that describe how to derive, at least approximately, updated neural network parameters from the initial neural network parameters.

[0304] The reference NN parameters 882 may be, for example, initial neural network parameters (e.g., in a second training step, a third training step, a fourth training step, or even in a further training step), such that the parameter update information 832 may include information on how to modify the initial neural network parameters to calculate or determine the updated NN parameters 812.

[0305] However, such information may be included in the PUI 832 for recursive reconstruction. Thus, the PUI may include information on how to modify the reference NN parameters 822, e.g., the starting parameters of one training cycle, to obtain updated NN parameters 812 that differ from the initial NN parameters.

[0306] 5, the PUI may thus contain information about the entire path from the root node R 510 associated with the initial neural network parameters, e.g., 512, to the currently considered node, e.g., U3 550, associated with the updated NN parameters, e.g., 552, or only about a section of such path, e.g., via one or more nodes, e.g., from U2 520 associated with the reference parameters, e.g., 522, to the node U4 associated with the updated NN parameters, e.g., 562. It should again be noted here that, briefly, one or more trainings, and thus one or more parameter updates, may be performed between the reference NN parameters 822 and the updated NN parameters.

[0307] In other words, the PUI information may optionally include information on how to modify the reference NN parameters 822, which are the initial or any or intermediate starting point for NN training, to obtain updated NN parameters 812.

[0308] Further, the node information providing unit 840 may be configured to provide node information 802 including parent node identifier information (e.g., as previously described) and parameter update information PUI (e.g., as previously described), where the parent node identifier defines the parent node and the parameter information serves as a starting point for applying the parameter update information.

[0309] Therefore, the parent node identifier can be used to provide a reference to the PUI information to identify the reference NN parameters 822 that are modified by the PUI information to obtain the updated NN parameters 832 .

[0310] As another optional feature, neural network controller 800 may include an encoder according to any of the embodiments as disclosed herein and / or any functionality or combination of functionality of any inventive encoder as disclosed herein.

[0311] 9 shows a schematic diagram of a neural network federated learning controller according to an embodiment of the present invention. FIG. 9 shows a neural network federated learning controller 900 comprising a processing unit 910 and a distribution unit 920.

[0312] The neural network federated learning controller 900 is configured to receive node information 902 of a plurality of neural networks, where the node information includes parent node identifiers (or parent node identifier information including, for example, parent node identifiers) and parameter update information.

[0313] Furthermore, the processing unit 910 is configured to combine the parameter update information of multiple corresponding nodes of different neural networks to obtain combined parameter update information. The processed information 912 may include or be the combined parameter update information.

[0314] Furthermore, the distribution unit 920 is configured to distribute the processed information, for example, the combined parameter update information.

[0315] Therefore, the neural network federated learning controller 900 can act as a collaborative unit to combine multiple training results (e.g., parameter update information) of multiple corresponding nodes of different neural networks, so that robust neural network parameters can be extracted and provided in the form of processed information.

[0316] As an optional feature, the neural network federated learning controller 900, e.g., the processing unit 910, may be configured to combine parameter update information of multiple corresponding nodes with the same parent node identifier of different neural networks to obtain combined parameter update information. Thus, the processed information 912 may include or be the combined parameter update information.

[0317] Thus, as an example, NN training results based on equal starting parameters can be combined to provide a robust set of NN parameters.

[0318] As an optional feature, the neural network federated learning controller 900, e.g., the distribution unit 920, can be configured to distribute parameter information of parent nodes associated with parent node identifiers to multiple decoders, and the neural network federated learning controller 900, e.g., the processing unit 912, can be configured to receive node information including the parent node identifiers from the decoders. Further, the neural network federated learning controller 900, e.g., the processing unit 910, can be configured to combine parameter update information of multiple corresponding nodes having parent node identifiers.

[0319] As another optional feature, the neural network federated learning controller 900, e.g., the distribution unit 920, can be configured to provide node information describing, e.g., combined node information of the parameter update tree, e.g., in or as the processed information 912, where the combined node information includes parent node identifiers and where the combined node information includes combined parameter update information.

[0320] As another optional feature, the neural network federated learning controller 900 may optionally include an encoder according to any embodiment disclosed herein, or the neural network federated learning controller 900 may optionally include any functionality or combination of functionality of the inventive encoders disclosed herein.

[0321] 10 shows a schematic block diagram of a method for decoding parameters of a neural network according to an embodiment of the present invention. The method 1000 includes obtaining (1010) a plurality of neural network parameters of a neural network based on an encoded bitstream, obtaining (1020) node information describing nodes of a parameter update tree, the node information including a parent node identifier and the node information including parameter update information, and deriving (1030) one or more neural network parameters using the parameter information of the parent node identified by the parent node identifier and using the parameter update information.

[0322] Figure 11 shows a schematic block diagram of a method for encoding parameters of a neural network to obtain an encoded bitstream, according to an embodiment of the present invention. Figure 11 shows a method 1100 that includes providing 1110 node information describing nodes of a parameter update tree, the node information including a parent node identifier, the node information including parameter update information, the parameter update information describing differences between neural network parameters associated with the parent node defined by the parent node identifier and current neural network parameters.

[0323] Figure 12 shows a schematic block diagram of a method for controlling a neural network according to an embodiment of the present invention. Figure 12 illustrates a method 1200 that includes training 1210 a neural network to obtain updated neural network parameters based on initial neural network parameters, determining 1220 parameter update information based on reference neural network parameters and the updated neural network parameters, where the parameter update information includes one or more update instructions that describe how to derive the updated neural network parameters, at least approximately, from the initial neural network parameters, and providing 1230 node information including a parent node identifier and the parameter update information, where the parent node identifier defines the parent node and the parameter information of the parent node serves as a starting point for application of the parameter update information.

[0324] 13 shows a schematic block diagram of a method for controlling neural network federated learning according to an embodiment of the present invention. FIG. 13 illustrates a method 1300 that includes receiving (1310) node information for a plurality of neural networks, where the node information includes a parent node identifier and the node information includes parameter update information. The method further includes combining (1320) the parameter update information for a plurality of corresponding nodes of different neural networks to obtain combined parameter update information, and distributing (1330) the combined parameter update information.

[0325] Further embodiments in accordance with the present invention may relate to, be used for, or address HLS (e.g., HTTP (e.g., hypertext transfer protocol) live streaming) update signaling.

[0326] Furthermore, it should be noted that embodiments may be applied to compressing entire neural networks, and some of them may also be applied to compressing differential updates of a neural network relative to a base network. Such differential updates are useful, for example, when a model is redistributed after fine-tuning or transfer learning, or when providing versions of a neural network with different compression ratios.

[0327] Embodiments may further address the use, eg, manipulation or modification, of a base neural network, eg, a neural network on which differential updates are based.

[0328] Embodiments may further address, include, or provide an updated neural network, e.g., a neural network resulting from modifying a base neural network. Note: An updated neural network may be reconstructed, for example, by applying differential updates to a base neural network.

[0329] Further embodiments in accordance with the present invention may include syntax elements in the form of NNR units, which may be, for example, data structures carrying neural network data and / or associated metadata that may be compressed or represented in accordance with embodiments of the present invention.

[0330] The NNR unit may carry at least one of compressed information about neural network metadata, uncompressed information about neural network metadata, topology information, full or partial layer data, filters, kernels, biases, quantized weights, tensors, etc.

[0331] An NNR unit may, for example, include or consist of the following data elements:

[0332] NNR unit size (optional): This data element may signal the total byte size of the NNR unit, including the NNR unit size.

[0333] NNR unit header: This data element may contain information about the NNR unit type and / or associated metadata.

[0334] NNR Unit Payload: This data element may contain or encompass compressed or uncompressed data related to a neural network.

[0335] As an example, an embodiment may include (or use) the following bitstream syntax:

[0336] [Table 1]

[0337] [Table 2]

[0338] [Table 3]

[0339] The parent node identifier may include, for example, one or more of the syntax elements listed above, such as device_id, parameter_id, and / or put_node_depth.

[0340] [Table 4]

[0341] [Table 5]

[0342] decode_compressed_data_unit_payload() can be used to modify the parameters of the neural network's base model to obtain an updated model. Thus, using the above nnr_compressed_data_unit_payload(), one can derive one or more neural network parameters using the parameter information of the parent node identified by the parent node identifier and using the parameter update information.

[0343] A node_id_present_flag equal to 1 may indicate that the syntax elements device_id, parameter_id, and / or put_node_depth are present.

[0344] The device_id may, for example, uniquely identify the device that generated the current NDU.

[0345] The parameter_id may, for example, uniquely identify a parameter of a model to which a tensor stored in the NDU pertains. If the parent_node_id_type is equal to ICNN_NDU_ID, the parameter_id may, for example, be or shall be equal to the parameter_id of the associated parent NDU.

[0346] The put_node_depth may be, for example, the tree depth where the current NDU is located. A depth of 0 may correspond to the root node. If parent_node_id_type is equal to ICNN_NDU_ID, then put_node_depth-1 may be, for example, or must even be equal to the put_node_depth of the associated parent NDU.

[0347] A parent_node_id_present_flag equal to 1 may indicate, for example, that the syntax element parent_node_id_type is present.

[0348] parent_node_id_type can, for example, specify the parent node id type, which may indicate the presence of further syntax elements that uniquely identify the parent node. Example allowed values ​​for parent_node_id_type are defined in the following table:

[0349] [Table 6]

[0350] The temporal_context_modeling_flag may, for example, specify whether temporal context modeling is enabled. A temporal_context_modeling_flag equal to 1 may indicate that temporal context modeling is enabled. If temporal_context_modeling_flag is not present, it is inferred to be 0.

[0351] The parent_device_id may be equal to the syntax element device_id of the parent NDU, for example.

[0352] The parent_node_payload_sha256 may be, for example, the SHA256 hash of the nnr_compressed_data_unit_payload of the parent NDU.

[0353] The parent_node_payload_sha512 may be, for example, the SHA512 hash of the nnr_compressed_data_unit_payload of the parent NDU.

[0354] Additionally, embodiments in accordance with the present invention may include a row skipping feature. As an example, when enabled by the flag row_skip_flag_enabled_flag, the row skipping technique signals one flag row_skip_list[i] for each value i along the first axis of the parameter tensor. When the flag row_skip_list[i] is 1, all elements of the parameter tensor whose index on the first axis is equal to i are set to zero. When the flag row_skip_list[i] is 0, all elements of the parameter tensor whose index on the first axis is equal to i are individually encoded.

[0355] Furthermore, embodiments in accordance with the present invention may include context modeling. As an example, context modeling may correspond to associating three types of flags, sig_flag, sign_flag, and abs_level_greater_x / x2, with a context model. In this way, flags with similar statistical behavior may or should be associated with the same context model so that a probability estimator (inside the context model) can, for example, adapt to the underlying statistics.

[0356] The context modeling of the presented approach may be, for example, as follows:

[0357] For example, 24 context models can be distinguished for sig_flag depending on the state value and whether the left adjacent quantized parameter level is zero, less than zero, or greater than zero.

[0358] If dq_flag is 0, for example, only the first three context models can be used.

[0359] Three other context models can be distinguished for sign_flag, for example, depending on whether the left adjacent quantized parameter level is zero, less than zero, or greater than zero.

[0360] For the abs_level_greater_x / x2 flag, each x can use, for example, either one or two distinct context models. If x≦maxNumNoRemMinus1, two context models are distinguished depending on sign_flag. If x>maxNumNoRemMinus1, for example, only one context model can be used.

[0361] Furthermore, embodiments in accordance with the present invention may include temporal context modeling. As an example, if enabled by the flag temporal_context_modeling_flag, an additional context model set of flags sig_flag, sign_flag, and abs_level_greater_x may be available. The derivation of ctxIdx may then also be based on the value of the quantized collocated parameter level in a previously encoded parameter update tensor, which may be uniquely identified, for example, by a parameter update tree. If the collocated parameter level is not available or is equal to zero, for example, context modeling as previously described may be applied. Otherwise, if the collocated parameter level is not equal to zero, the temporal context modeling of the presented approach may be as follows:

[0362] The 16 context models can be distinguished for sig_flag, for example, depending on the state value and whether the absolute value of the quantized collocated parameter level is greater than 1 or not.

[0363] If dq_flag is 0, only the first two additional context models can be used.

[0364] Furthermore, two context models can be distinguished for sign_flag, for example, depending on whether the quantized collocated parameter level is less than or greater than zero.

[0365] For the abs_level_greater_x flag, each x can use two distinct context models, which can be distinguished, for example, depending on whether the absolute value of the quantized collocated parameter level is greater than or equal to x-1.

[0366] Embodiments in accordance with the present invention may optionally include the following tensor syntax, for example, quantized tensor syntax:

[0367] [Table 7]

[0368] The skip information may include, for example, any or all of the row skip information described above, eg, row_skip_enabled_flag and / or row_skip_list.

[0369] As an example, row_skip_enabled_flag may specify whether row skipping is enabled: row_skip_enabled_flag equal to 1 may indicate that row skipping is enabled.

[0370] row_skip_list can specify a list of flags, and the i-th flag row_skip_lsit[i] can indicate whether all tensor elements of QuantParam whose first dimension index is equal to i are zero. If row_skip_list[i] is equal to 1, all tensor elements of QuantParam whose first dimension index is equal to i may be zero.

[0371] Embodiments in accordance with the present invention may further include, for example, a quantized parameter syntax, such as the syntax defined below, where all elements may be considered optional.

[0372] [Table 8]

[0373] sig_flag may, for example, specify whether the quantized weight QuantParam[i] is non-zero. sig_flag equal to 0 may, for example, indicate that QuantParam[i] is 0. sign_flag may, for example, specify whether the quantized weight QuantParam[i] is positive or negative. sign_flag equal to 1 may, for example, indicate that QuantParam[i] is negative. abs_level_greater_x[j] may, for example, indicate whether the absolute level of QuantParam[i] is greater than j+1.

[0374] abs_level_greater_x2[j] may, for example, contain the unary part of the exponential-Golomb residue.

[0375] abs_remainder may indicate, for example, a fixed length remainder.

[0376] Further embodiments in accordance with the present invention may include, for example, the following shift parameter index syntax: All elements may be considered optional.

[0377] [Table 9]

[0378] Further embodiments in accordance with the present invention include an entropy decoding process, as described below.

[0379] In general, the input to this process can be, for example, a request for the value of a syntax element and the value of a previously parsed syntax element.

[0380] The output of this process can be, for example, the value of a syntax element.

[0381] Parsing of syntax elements can proceed, for example, as follows:

[0382] For each requested value of a syntax element, for example, a binarization can be derived.

[0383] The binarization of the syntax elements and the sequence of parsed bins can determine, for example, the decoding process flow.

[0384] An example of the initialization process:

[0385] In general, the output of this process can be, for example, initialized DeepCABAC internal variables.

[0386] The context variables of the arithmetic decoding engine can be initialized, for example, as follows:

[0387] The decoding engine may, for example, register IvlCurrRange and IvlOffset, both with 16-bit register precision, and may be initialized, for example, by invoking an arithmetic decoding engine initialization process.

[0388] Embodiments in accordance with the present invention may include an initialization process for probability estimation parameters, for example as described below.

[0389] The output of this process may be, for example, the initialized probability estimation parameters shift0, shift1, pStateIdx0, and pStateIdx1 for each context model of the syntax elements sig_flag, sign_flag, abs_level_greater_x, and abs_level_greater_x2.

[0390] The 2D array CtxParameterList[][] can be initialized, for example, as follows:

[0391] CtxParameterList[][]={{1,4,0,0},{1,4,-41,-654},{1,4,95,1519},{0,5,0, 0},{2,6,30,482},{2,6,95,1519},{2,6,-21,-337},{3,5,0,0},{3,5,30,482}}

[0392] If dq_flag is equal to 1 and temporal_context_modeling_flag is equal to 1, for example, for each of the 40 context models of the syntax element sig_flag, the associated context parameter shift0 may be set, for example, to CtxParameterList[setId][0], shift1 may be set, for example, to CtxParameterList[setId][1], pStateIdx0 may be set, for example, to CtxParameterList[setId][2], and pStateIdx1 may be set, for example, to CtxParameterList[setId][3], where i may be, for example, an index of the context model and setId may be equal, for example, to ShiftParameterIdsSigFlag[i].

[0393] If dq_flag== is equal to 1 and temporal_context_modeling_flag is equal to 0, then for each of the first, e.g., 24, context models of the syntax element sig_flag, the associated context parameter shift0 may be set, e.g., to CtxParameterList[setId][0], shift1 may be set, e.g., to CtxParameterList[setId][1], pStateIdx0 may be set, e.g., to CtxParameterList[setId][2], and pStateIdx1 may be set, e.g., to CtxParameterList[setId][3], where i may be, e.g., the index of the context model and setId may be equal, e.g., to ShiftParameterIdsSigFlag[i].

[0394] If dq_flag is equal to 0 and temporal_context_modeling_flag is equal to 1, then for each of, for example, the first three context models of the syntax element sig_flag and, for example, context models 24 and 25, the associated context parameter shift0 may be set, for example, to CtxParameterList[setId][0], shift1 may be set, for example, to CtxParameterList[setId][1], pStateIdx0 may be set, for example, to CtxParameterList[setId][2], and pStateIdx1 may be set, for example, to CtxParameterList[setId][3], where i may be, for example, an index of the context model and setId may, for example, be equal to ShiftParameterIdsSigFlag[i].

[0395] If temporal_context_modeling_flag is equal to 1, then for each of the, e.g., five context models of the syntax element sign_flag, the associated context parameter shift0 may be set, e.g., to CtxParameterList[setId][0], shift1 may be set, e.g., to CtxParameterList[setId][1], pStateIdx0 may be set, e.g., to CtxParameterList[setId][2], and pStateIdx1 may be set, e.g., to CtxParameterList[setId][3], where i may be, e.g., an index of the context model, and setId may be equal, e.g., to ShiftParameterIdsSignFlag[i].

[0396] Otherwise (temporal_context_modeling_flag==0), for example, for each of the first, e.g., three, context models of the syntax element sign_flag, the associated context parameter shift0 may be set, e.g., to CtxParameterListsetId[0], shift1 may be set, e.g., to CtxParameterList[setId][1], pStateIdx0 may be set, e.g., to CtxParameterList[setId][2], and pStateIdx1 may be set, e.g., to CtxParameterList[setId][3], where i may be, e.g., the index of the context model, and setId may be equal, e.g., to ShiftParameterIdsSignFlag[i].

[0397] If temporal_context_modeling_flag is equal to 1, then, for example, for each of the 4*(cabac_unary_length_minus1+1) context models of the syntax element abs_level_greater_x, the associated context parameter shift0 may be set, for example, to CtxParameterList[setId][0], shift1 may be set, for example, to CtxParameterList[setId][1], pStateIdx0 may be set, for example, to CtxParameterList[setId][2], and pStateIdx1 may be set, for example, to CtxParameterList[setId][3], where i may be, for example, an index of the context model and setId may, for example, be equal to ShiftParameterIdsAbsGrX[i].

[0398] Otherwise (temporal_context_modeling_flag==0), for example, for each of the first, e.g., 2*(cabac_unary_length_minus1+1) context models of the syntax element abs_level_greater_x, the associated context parameter shift0 may be set, e.g., to CtxParameterList[setId][0], shift1 may be set, e.g., to CtxParameterList[setId][1], pStateIdx0 may be set, e.g., to CtxParameterList[setId][2], and pStateIdx1 may be set, e.g., to CtxParameterList[setId][3], where i may be, e.g., the index of the context model and setId may, e.g., be equal to ShiftParameterIdsAbsGrX[i].

[0399] Further embodiments in accordance with the present invention may include, for example, a decoding process flow as described below.

[0400] In general, the input to this process may be, for example, all the bin strings of the binarization of the requested syntax element.

[0401] The output of this process may be, for example, the value of a syntax element.

[0402] This process may, for example, specify for each syntax element, how each bin of the bin string is to be parsed, e.g., after parsing each bin, the resulting bin string may, for example, be compared with all bin strings of a binarization of the syntax element, and the following may apply:

[0403] If the bin string is equal to one of the bin strings, the corresponding value of the syntax element may be, for example, output. If not (the bin string is not equal to one of the bin strings), the next bit may be parsed, for example.

[0404] While analyzing each bin, the variable binIdx may be incremented by 1, for example, starting with binIdx being set equal to 0 for the first bin.

[0405] The analysis of each bin can be specified, for example, by the following two ordered steps:

[0406] 1. For example, the derivation process for ctxIdx and bypassFlag can be invoked, for example, with binIdx as input and ctxIdx and bypassFlag as outputs. 2. The arithmetic decoding process can be called, for example, with ctxIdx and bypassFlag as input and the bin value as output.

[0407] Further embodiments according to the present invention may include a derivation process for ctxInc of the syntax element sig_flag.

[0408] The input to this process can be, for example, the sig_flag decoded before the current sig_flag, the state value stateId, the associated sign_flag, if present, and the co-located parameter level (coLocParam) from the incremental update decoded before the current incremental update, if present. If a sig_flag was not decoded before the current sig_flag, it can be inferred to be, for example, 0. If a sign_flag associated with a previously decoded sig_flag was not decoded, it can be inferred to be, for example, 0. If a co-located parameter level from an incremental update decoded before the current incremental update is not available, it is inferred to be 0. The co-located parameter level refers to the parameter level in the same tensor at the same position in the previously decoded incremental update.

[0409] The output of this process is the variable ctxInc.

[0410] The variable ctxInc is derived as follows:

[0411] If coLocParam is equal to 0, the following applies: If sig_flag is equal to 0, ctxInc is set to stateId*3. Otherwise, if sign_flag is equal to 0, ctxInc is set to stateId*3+1. Otherwise, ctxInc is set to stateId*3+2. If coLocParam is not equal to 0, the following applies: If coLocParam is greater than 1 or less than -1, ctxInc is set to stateId*2+24. Otherwise, ctxInc is set to stateId*2+25.

[0412] Further embodiments according to the present invention may include a derivation process for ctxInc of the syntax element sign_flag.

[0413] The input to this process can be, for example, the sig_flag decoded before the current sig_flag, the associated sign_flag, if present, and the co-located parameter level (coLocParam) from the incremental update decoded before the current incremental update, if present. If a sig_flag was not decoded before the current sig_flag, it can be inferred to be, for example, 0. If a sign_flag associated with a previously decoded sig_flag was not decoded, it can be inferred to be, for example, 0. If a co-located parameter level from an incremental update decoded before the current incremental update is not available, it can be inferred to be, for example, 0. The co-located parameter level refers to the parameter level in the same tensor at the same position in the previously decoded incremental update.

[0414] The output of this process may be, for example, the variable ctxInc.

[0415] The variable ctxInc may be derived, for example, as follows:

[0416] If coLocParam is equal to 0, the following can be applied: If sig_flag is equal to 0, then ctxInc may be set to 0, for example. Otherwise, if sign_flag is equal to 0, ctxInc may be set to 1, for example. If not, ctxInc may be set to 2, for example. If coLocParam is not equal to 0, the following can be applied: If coLocParam is less than 0, ctxInc may be set to 3, for example. If not, ctxInc may be set to 4, for example.

[0417] Further embodiments may include a derivation process for ctxInc of the syntax element abs_level_greater_x[j].

[0418] The input to this process can be, for example, the sign_flag decoded before the current syntax element abs_level_greater_x[j] and, if present, the collocated parameter level (coLocParam) from the incremental update decoded before the current incremental update. If the collocated parameter level from the incremental update decoded before the current incremental update is not available, it can be inferred to be, for example, 0. The collocated parameter level means the parameter level in the same tensor at the same position in the previously decoded incremental update.

[0419] The output of this process may be, for example, the variable ctxInc.

[0420] The variable ctxInc may be derived, for example, as follows:

[0421] If coLocParam is equal to zero, the following can be applied:

[0422] If sign_flag is equal to 0, ctxInc may be set to, for example, 2*j. Otherwise, ctxInc may be set to, for example, 2*j+1. If coLocParam is not equal to zero, the following may apply: If coLocParam is greater than or equal to j or less than or equal to -j, ctxInc may be set to, for example, 2*j+2*maxNumNoRemMinus1. Otherwise, ctxInc may be set to, for example, 2*j+2*macNumNoRemMinus1+1.

[0423] Further notes: Below, different inventive embodiments and aspects are described in the "Introduction" section and the "Parameter Update Tree (PUT) (Example)" section, and their respective subsections.

[0424] Further embodiments are defined by the appended claims.

[0425] It should be noted that any embodiment defined by the claims may be supplemented by any of the details (features and functions) described in the above sections and / or subsections, respectively, and / or by any of the details (features and functions) described in the above disclosure.

[0426] Furthermore, the embodiments described in the above chapters and / or subchapters can be used individually or can be supplemented by any of the features of the other chapters or by any features included in the claims.

[0427] It should also be noted that the individual aspects described herein can be used individually or in combination, and thus details can be added to each individual aspect without adding details to another one of the aspects.

[0428] It should also be noted that this disclosure explicitly or implicitly describes features that can be used in neural network encoders (devices that provide encoded representations of neural network parameters) and neural network decoders (devices that provide decoded representations of neural network parameters based on the encoded representations). Thus, any of the features described herein can be used in the context of a neural network encoder as well as in the context of a neural network decoder.

[0429] Furthermore, features and functions disclosed herein with respect to a method may also be used in an apparatus (configured to perform such functions). Furthermore, any feature and function disclosed herein with respect to an apparatus may also be used in the corresponding method. In other words, the method disclosed herein may be complemented by any of the features and functions described with respect to the apparatus.

[0430] Additionally, any of the features and functions described herein may be implemented in hardware or software, or using a combination of hardware and software, as described in the "Alternative Implementations" section.

[0431] The following section (eg, including subsections or chapters 1 and 2) may be entitled Efficient Signaling of Neural Network Updates in Distributed Scenarios.

[0432] 1. Introduction The following provides background information for embodiments in accordance with the present invention. However, the features and / or functions and / or details and / or aspects described in this section may optionally be incorporated into multiple embodiments of the present invention, both individually and in combination. Embodiments in accordance with the present invention may include or be used in conjunction with such aspects and / or features.

[0433] As neural networks (NNs) have brought breakthroughs in many application fields, efficient transmission mechanisms have also been developed. This may enable, for example, a potentially complex NN training process to be performed, for example, on a central server device, and optionally transmit the trained NN to, for example, client devices. Here, compression and representation of neural networks have been standardized in recent years. A newer application area is federated learning (FL) and training scenarios, where a NN can be trained, for example, on many devices simultaneously. In FL scenarios, for example, frequent communication between client devices and a central server device may be beneficial or even necessary. Initially, a first version of a pre-trained NN can be sent to all clients, for example, for further training, using, for example, neural network compression. Then, all clients can further train the NN and transmit updated NN versions to one (or more) servers, for example, as shown in FIG. 14.

[0434] Figure 14 shows a schematic diagram of an example federated learning scenario according to an embodiment of the present invention. Figure 14 shows multiple clients 1410, 1420, 1430 that can be configured to train neural networks. A server 1430 can receive training results, e.g., updates, from each client and can provide aggregated updated neural network parameters to the clients based on the training results, e.g., neural network parameters.

[0435] An update may be, for example, a differential signal between the initial NN and a newer NN version at the client. Thus, one or more, or, for example, all, arrows between the server and the client (e.g., in FIG. 14 ) may represent sending an NN update. The server may then collect some, or, for example, all, of the local client versions and aggregate a new server version of the NN. The aggregation process may be, for example, a simple averaging of multiple, or, for example, all, available network versions, or a more advanced process, such as only averaging output labels. This latter method, known as federated distillation, may allow, for example, more flexibility at the local client. This may even include, for example, different networks and / or topologies at each client, as long as, for example, the output labels of all client versions can still be aggregated at the server. This process up to this point is also referred to as a communication round (CR). Afterwards, for example, the new server version may again be sent to multiple, or, for example, all, clients, e.g., as a differential signal relative to the previous NN version, for further training, and the process may be repeated. An FL scenario can continue for many CRs, for example, until a certain accuracy (of inference) is reached. In general, an FL scenario can allow for communication flexibility. In a synchronous communication scenario, multiple, or for example, all N, clients can send updates in each communication round, and the server can aggregate versions, or for example, N versions, into a new version and then send this version to multiple, or for example, all N clients, for example, for the next CR. In an asynchronous FL scenario, a client can send updates only after a certain number of CRs. Alternatively, it can send updates for consecutive numbers of CRs and then pause for a certain amount of time.This can mean that the server can, for example, have only a subset K < N of the client versions in a particular CR, for example, to aggregate new server versions to be sent to the client.

[0436] Many or for example all of these conditions in the federated scenario may, for example, require specific signaling to, for example, (e.g., in both directions) increase the coding efficiency of the NN updates and / or differential data sent between the server(s) and the client(s), if desired, for example, to handle different FL deformation forms. Therefore, in the following subsections, related aspects of the present invention will be described.

[0437] 2. Parameter Update Tree (PUT) (example) As an example, consider an embodiment with a plurality of devices (e.g., including server devices and client devices) and a so-called base model, which may be, for example, a neural network where updates are transmitted or can be transmitted between devices. Embodiments according to the present invention include and / or describe a scheme for representing such updates, for example, using a tree structure. A plurality of or for example individual parameters (or groups of parameters) of the base model can be associated with the root node of a tree, for example, a tree having the aforementioned tree structure. Updates to such parameters can correspond to child nodes added to the root node. The child nodes can include, for example, instructions on how to update the parameters associated with the parent node. Any node of the tree can be further updated, for example, by adding child nodes in the same way. An example is shown in FIG. 15, where R is the root node representing one parameter of the base model.

[0438] FIG. 15 shows an example of a parameter update tree, for example, a schematic diagram of an exemplary parameter update tree, according to an embodiment of the present invention.

[0439] Nodes U1 and U2 may describe updates to node R, and node U3 may describe updates to node U2. Each node in the tree may represent a version of the parameters of base model R, and it may be decided, for example, to run the model using a particular updated version U1, U2, or U3 instead of R. In other words, according to an embodiment, the decoder may be configured to decide, for example, to run the model using a particular updated version U1, U2, or U3 instead of R, e.g., corresponding to a particular version of the neural network parameters.

[0440] To represent such a tree, a unique node identifier may be associated with multiple nodes, or for example, each node. For example, this may be or may be an integer, a string, and / or a cryptographic hash (such as, for example, SHA-512) associated with the node. However, within such a tree, each node identifier may be, or even, for example, must be, unique.

[0441] In a distributed scenario, multiple devices, or for example, each device (e.g., a server or client), may maintain an update tree, e.g., of parameters of a model. For example, to transmit a particular update from one device to another, by way of example only, the corresponding update may or may need to be transmitted along with an associated node identifier (e.g., a pointer to the parent node of the update).

[0442] In the following, examples according to embodiments are disclosed to provide a better understanding of aspects of the present invention. For example, consider a client and a server, both of which have an available node R (e.g., as in FIG. 15). Suppose the client creates an update U2 and wants to make it available to the server. To do this, it can send U2 to the server along with the node identifier of R. The server can look up the node identifier of R in its version of the tree and add U2 as a child node of R. Now, both the server and the client can have available trees with R as the root node and U2 as a child node of R.

[0443] 2.1 Parameter update procedure (example) This section describes several different types of update instructions associated with nodes in a PUT, for example, according to embodiments of the present invention. A parameter of a base model that may be associated with a PUT may be, or shall be, denoted as a tree parameter. For example, a so-called node parameter may be, or shall be, associated with each node in the PUT. However, in other examples, a so-called node parameter may be, for example, associated with a set of nodes in the PUT, for example, a set of reachable nodes (e.g., reachable from the root node to a particular current node), or a set of consecutive nodes starting from the root node, or, for example, associated with each node in the PUT. This node parameter may be derived, for example, by traversing the PUT from the root node to the desired node and applying an update instruction for each visited node to the tree parameter. For example (e.g., as in FIG. 15 ), after applying an update instruction for, for example, U1, the node parameter of R may be equal to the tree parameter, and the node parameter of U1 may be equal to the tree parameter.

[0444] If a node parameter (and consequently also a tree parameter, e.g.) is a tensor (i.e., a multidimensional array of values, e.g.), it may be, or shall be denoted as, a node tensor.

[0445] In a preferred embodiment, the update instructions associated with, for example, a node may comprise a so-called product tensor, e.g., of the same shape as the node tensor. Updating a parameter may correspond, for example, to an element-wise product of the node tensor elements and the product tensor elements.

[0446] In a preferred embodiment, for example, an update instruction associated with a node may include at least one of a so-called sum tensor, e.g., of the same shape as the node tensor, a scalar node tensor weight value, and / or a scalar sum tensor weight value. Updating the parameters may, for example, correspond to an element-wise weighted sum of the node tensor elements and the sum tensor elements. As an example, or more precisely, for example, each element of the node tensor may be multiplied by a node tensor weight value, and each element of the sum tensor may be multiplied by a sum tensor weight value, and then an element-wise sum of both scaled tensors may be calculated. Note that both weights may also be set to 1, which may, for example, correspond to an unweighted sum as a special case.

[0447] In another preferred embodiment, the update instructions associated with, for example, a node may comprise a so-called permutation tensor, for example of the same shape as the node tensor. Updating a parameter may correspond to replacing the values ​​of the node tensor with, for example, the values ​​of the permutation tensor.

[0448] In another preferred embodiment, the update instructions using the update tensors (e.g., as permutation tensors, sum tensors, and / or product tensors) may include, for example, implicit tensor shape transformations, for example, as follows:

[0449] In the following, examples according to embodiments are disclosed to provide a better understanding of aspects of the present invention.

[0450] The update tensor shape is assumed to be identical to the node tensor shape except for one or more individual dimensions, which may be equal to 1. For example, if the dimension of the update tensor is 1, then the tensor elements along this axis may be, or shall be, assumed to be the same. For example, assume the node tensor is given as a 2D tensor [[a,b,c],[d,e,f]] (dimensions are [2,3]). An update tensor given as [[x],[y]] (dimensions are [2,1]) can be, or will be, implicitly expanded to [[x,x,x],[y,y,y]]. An update tensor given as [[z]] (dimensions are [1,1]) can be, or will be, implicitly expanded to [[z,z,z],[z,z,z]]. An update tensor given as [[r,s,t]] (with dimensions [1,3]) can be or will be implicitly expanded to [[r,s,t],[r,s,t]]. In other words, a decoder according to an embodiment can be configured to update the tensor shape according to, for example, the examples described above.

[0451] TBD: Quantized domain update (check the effect of different qps) (optional) Optionally, quantized domain updates may be used, for example (eg, alternatively or in combination with the above concepts).

[0452] 2.2 Server-Side Update Aggregation Example In a distributed scenario according to an embodiment of the present invention, where a server can maintain a base model and, for example, receive updates from different clients, a server-side PUT can, for example, collect multiple update nodes for the same model. The server can decide to combine the multiple update nodes, for example, create a new update node from this combination, and, for example, distribute it to clients as, for example, a collectively updated model. Then, multiple nodes, or for example, each node, can decide to continue federated learning, for example, based on this collectively updated model.

[0453] 2.3 Parameter Update Tree Signaling (Example) This section presents a method for representing elements of a PUT as, for example, a data unit (e.g., a byte sequence) optimized for transmission between devices (e.g., including a server and a client) according to an embodiment of the present invention. For example, this method can be combined with existing techniques for representing neural networks, such as the emerging standard "ISO / IEC 15938 Part 17: Compression of neural networks for multimedia content description and analysis," referred to throughout this specification as NNR. NNR can represent individual parameters of a neural network as so-called compressed data units (NDUs). For example, implementing the PUT concept by using efficient compression techniques available in NNR can be performed, for example, as follows: According to an embodiment of the present invention, consider the case where a node includes an update tensor (e.g., a permutation tensor, a sum tensor, and / or a product tensor, as described above). Such an update tensor can be efficiently represented as an NDU. For example, additional syntax elements may be added to extend an NDU with all the necessary information so that it can be used as, for example, a PUT node.

[0454] In a preferred embodiment, a new syntax element "parent_node_id_present_flag" is introduced, for example, in the nnr_compressed_data_unit_header of the NDU, which can indicate, for example, whether a parent node identifier is present in the NDU. Depending on the value of parent_node_id_present_flag, a further new syntax element "parent_node_id" can be transmitted, which uniquely identifies another NDU that can, for example, contain the parent node of the current PUT node.

[0455] In another preferred embodiment, the parent_node_id may be a cryptographic hash (such as SHA-512) of the parent NDU.

[0456] In another preferred embodiment, the parent_node_id may be a cryptographic hash (such as SHA-512) of the nnr_compressed_data_unit_payload of the parent NDU.

[0457] In another preferred embodiment, the parent_node_id may be a combined value representing, for example, a device identifier and / or a serial number, both of which may be associated with the parent NDU.

[0458] In another preferred embodiment, a new syntax element "node_id" may be encoded within a parent node to be used as the parent_node_id of its child nodes. The node_id may be, for example, a unique identifier.

[0459] In another preferred embodiment, the syntax element "node_id" (e.g., which may uniquely identify a node) may consist of a device identifier and / or parameter update tree depth information (i.e., information about the number of nodes visited when walking the tree from the current node to the root node, for example) and / or a parameter update tree identifier.

[0460] In another preferred embodiment, a flag can be signaled to the node indicating whether node_id is present or not. Depending on the value of this flag, the syntax element node_id may or may not be present.

[0461] In another preferred embodiment, a new syntax element "parent_node_id_type" can indicate what type the syntax element parent_node_id is. For example, the possible different types of parent_node_id may be as described in the previous preferred embodiment.

[0462] In another preferred embodiment, whether the syntax element parent_node_id_type is signaled or not may depend on the value of the syntax element parent_node_id_present_flag.

[0463] 2.3.1 Topology Change Signaling (Example) In distributed scenarios, it may be important to change the topology of the neural network: for example, the number of output neurons in a particular layer can be increased or decreased, or both.

[0464] In a preferred embodiment, for example, within an NDU, a syntax element "shape_update" can be signaled that indicates whether the shape of a tensor associated with a parent node is modified. If the parent tensor shape is updated, the new tensor dimensions can be transmitted (e.g., using the syntax element tensor_dimensions).

[0465] Alternative embodiments: While some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0466] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementations can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective methods are performed. Thus, the digital storage medium may be computer-readable.

[0467] Some embodiments according to the invention comprise a data carrier having an electronically readable control signal, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0468] Generally, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine-readable carrier.

[0469] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0470] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0471] A further embodiment of the inventive methods is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory.

[0472] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may for example be adapted to be transmitted via a data communication connection, for example via the Internet.

[0473] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0474] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0475] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0476] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0477] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0478] The devices described herein, or any components of the devices described herein, may be implemented at least in part in hardware and / or software.

[0479] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0480] The methods described herein, or any components of the apparatus described herein, may be at least partially implemented by hardware and / or software.

[0481] The described embodiments merely exemplify the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details presented by the description and illustration of the embodiments herein.

Claims

1. A decoder (100, 100b) for decoding parameters of a neural network, the decoder is configured to obtain, based on the encoded bitstream, a plurality of neural network parameters (104, 104b, 204, 304, 512, 522, 532, 542, 552, 562, 572, 582, 592, 602, 612, 622, 652, 720, 740, 812) of the neural network; the decoder is configured to obtain node information (112, 112b) describing nodes of a parameter update tree; The node information includes a parent node identifier (114, 524, 534, 544, 554, 564, 574, 584, 594, 604, 614, 624), the node information includes parameter update information (116, 526, 536, 546, 556, 566, 576, 586, 596, 606, 616, 626); the decoder is configured to derive one or more neural network parameters (104, 104b, 204, 304, 512, 522, 532, 542, 552, 562, 572, 582, 592, 602, 612, 622, 652, 720, 740, 812) using parameter information of a parent node identified by the parent node identifier and using the parameter update information; Decoder.

2. an encoder (200, 300, 300b, 400) for encoding parameters of a neural network to obtain an encoded bitstream, the encoder is configured to provide node information (212, 312, 312b, 412) describing nodes of a parameter update tree (500); The node information includes a parent node identifier (214, 314, 524, 534, 544, 554, 564, 574, 584, 594, 604, 614, 624), the node information includes parameter update information (216, 316, 416, 526, 536, 546, 556, 566, 576, 586, 596, 606, 616, 626); the parameter update information describes a difference between a neural network parameter associated with a parent node defined by the parent node identifier and a current neural network parameter; encoder.

3. A neural network controller (800), the neural network controller is configured to train a neural network to obtain updated neural network parameters (812) based on initial neural network parameters; the neural network controller is configured to determine parameter update information (832) based on reference neural network parameters (822) and the updated neural network parameters; the parameter update information includes one or more update instructions that describe how to derive the updated neural network parameters, at least approximately, from the initial neural network parameters; the neural network controller is configured to provide node information (112, 112b, 212, 312, 312b, 412, 802) including a parent node identifier and the parameter update information; the parent node identifier defines a parent node, and the parameter information of the parent node serves as a starting point for application of the parameter update information. Neural network controller.

4. A method (1000) for decoding parameters of a neural network, comprising: obtaining (1010) a plurality of neural network parameters of the neural network based on the encoded bitstream; obtaining 1020 node information describing nodes of a parameter update tree, The node information includes a parent node identifier; the node information includes parameter update information; and deriving (1030) one or more neural network parameters using parameter information of a parent node identified by the parent node identifier and using the parameter update information. method.

5. 1. A method (1100) for encoding parameters of a neural network to obtain an encoded bitstream, comprising: providing 1110 node information describing nodes of the parameter update tree; The node information includes a parent node identifier; the node information includes parameter update information; the parameter update information describes a difference between a neural network parameter associated with a parent node defined by the parent node identifier and a current neural network parameter; method.

6. A method (1200) for controlling a neural network, comprising: training 1210 the neural network to obtain updated neural network parameters based on the initial neural network parameters; determining 1220 parameter update information based on reference neural network parameters and the updated neural network parameters; the parameter update information includes one or more update instructions that describe how to derive the updated neural network parameters, at least approximately, from the initial neural network parameters; providing (1230) node information including a parent node identifier and the parameter update information; the parent node identifier defines a parent node, and parameter information of the parent node serves as a starting point for application of the parameter update information. method.

7. A computer program for carrying out the method according to any one of claims 4 to 6 when said computer program is run on a computer.

Images