Apparatus and method for topology design in distributed learning

Abstract

There is described a device (1000) in a network for performing reasoning on a hierarchically structured task, the network comprising a plurality of nodes (206, 207, 208, 209, 210, 211, 1050), each node for processing respective data relating to a task of the hierarchically structured task, the device for: sending (801) to each node in the network a respective current collaboration mode (203, 204), each respective current cooperation mode is derived from a current connectivity model (202, 205, 601) of the network, the current connectivity model (202, 205, 601) indicating one or more other nodes in the network to communicate with a respective node; receiving (802), from each node in the network, a respective loss vector corresponding to the task of the hierarchy; from the received respective loss vectors, an updated connectivity model of the network is formed (803). In this way, an optimal network topology for tasks of a given hierarchy can be determined.

Description

Technical Field

[0001] The present disclosure relates to data processing, for example, to distributed learning and reasoning in a network of data processing devices. Background Art

[0002] The main category of approaches to solving many distributed learning problems is represented by federated learning. In this case, the master node (usually called the parameter server) has much larger memory and processing power than the edge devices in the network (usually called the clients) in order to perform calculations and store the results. In addition, the parameter server coordinates the information exchange communication in the network throughout the training and inference phases.

[0003] The connectivity pattern of the clients is called the network topology. The topology can have a variety of patterns. However, the most common is the star topology, as schematically shown in Figure 1(a). In this network topology, for the network 100, communication is only carried out between the parameter server 101 and each node 102, 103, 104, 105, 106, 107, and these connections remain fixed throughout the training and inference phases.

[0004] The communication pattern in a network can be defined by a connectivity matrix (CM), as shown in Figure 1(b). Therefore, the cooperation pattern of devices in a network can be stored as a matrix whose rows and columns represent the indexes of the devices and the matrix contains the numbers 1 and 0 as its elements. For example, an element of row i and column j equal to 1 indicates that devices i and j exchange information. A value of 0 indicates that the corresponding device does not communicate in the current round of message exchange. The connectivity matrix shown in Figure 1(b) is a full connectivity matrix, in which every node communicates with every other node in the network.

[0005] For cases with independent and identically distributed (iid) data at the client (which means the data has the same statistical characteristics throughout the network), the star network architecture performs very well while minimizing communication overhead. However, in real-world applications, where data does not have homogeneous properties, this becomes very undesirable.

[0006] The main drawback of existing techniques in this field is that they usually assume that the data is iid, so most federated learning algorithms are tailored for this case. When they are applied to non-iid situations, the learning performance may drop to a very low level.

[0007] To ensure the same performance as in the iid case, greater flexibility is needed in the design of device cooperation patterns in such networks that process non-iid data. In addition, in order to achieve optimality in performance and resource consumption, the network topology needs to be customized for the specific task or hierarchical composition of the tasks that the network is to solve. Summary of the invention

[0008] According to one aspect, a device for performing reasoning on hierarchical tasks in a network is provided, wherein the network includes multiple nodes, each node is used to process corresponding data related to tasks in the hierarchical tasks, and the device is used to: send a corresponding current collaboration mode to each node in the network, wherein each corresponding current collaboration mode is derived from a current connectivity model of the network, and the current connectivity model indicates one or more other nodes in the network to communicate with the corresponding node; receive a corresponding loss vector corresponding to the hierarchical task from each node in the network; and form an updated connectivity model of the network based on the received corresponding loss vector.

[0009] This method can provide an efficient way to determine the optimal network topology for a given hierarchical task. This method can be used in image classification applications to provide a customized network topology that ensures high accuracy of image classification on each device in the network. This method can also address the non-iid and privacy nature of the data and multi-task reasoning issues.

[0010] Each respective loss vector is determined based on one or more gradients of a respective neural network implemented by a respective node in the network and each node in the network that communicates with the respective node based on the current connectivity model of the network. This can improve models and performance, and protect data privacy.

[0011] The updated connectivity model may define multiple node clusters, wherein each node in a cluster is used to communicate with other nodes in the cluster. Each node in a cluster may be used to communicate only with other nodes in the cluster. For example, the connectivity model may be a connectivity matrix. This may allow each node cluster to be obtained in a manner that utilizes data with similar statistical properties and associated with a specific task in the hierarchical task structure. This may improve reasoning performance and communication efficiency.

[0012] The updated connectivity model may also define an inter-cluster collaboration mode for each of the plurality of node clusters, which may enable efficient communication between clusters.

[0013] The nodes of the cluster can be respectively used to output data related to the same task in the tasks of the hierarchy. This can improve reasoning performance and communication efficiency. The latter can be achieved because the cluster can make only nodes with relevant data exchange information between themselves.

[0014] The corresponding data processed by each node in the network may be non-IID data, with the data having different statistical properties depending on which node in the network processes the data. This may reflect many real-world applications where data does not have homogeneous properties.

[0015] The device may also be configured to: combine the corresponding loss vectors received from each node in the network to determine a combined loss value; and form the updated connectivity model based on the combined loss value. This may enable the loss received from each node in the network to be used to update the connectivity model.

[0016] The apparatus may be used to form the updated connectivity model to minimize the global average training loss of the tasks of the hierarchy. This may enable the connectivity model to be optimized as the network converges.

[0017] According to a second aspect, there is provided a node in a network for performing reasoning on a hierarchical task, the network comprising a plurality of nodes, each node being used to implement a corresponding neural network to process corresponding data related to a task in the hierarchical task, the node being used to: receive a current collaboration mode from a device in the network, wherein the current collaboration mode is derived from a current connectivity model of the network, the current connectivity model indicating one or more other nodes in the network to communicate with the node; determine one or more gradients of the corresponding neural network implemented by the node; send the one or more gradients to one or more other nodes in the network indicated by the current collaboration mode; determine a loss vector corresponding to the hierarchical task; and send the loss vector to the device.

[0018] This method can provide an efficient way to determine the optimal cooperation mode of nodes in the network for a given hierarchical task through the information provided by the nodes. This method can also solve the non-iid and privacy nature of data and multi-task reasoning problems, because nodes can only exchange gradients with other nodes, but not the original input data.

[0019] The node may also be configured to: receive one or more gradients of the one or more corresponding neural networks implemented by one or more other nodes in the network defined by the current connectivity model of the network; and determine the loss vector corresponding to the task of the hierarchy based on the received one or more gradients. This may improve models and performance, and protect data privacy.

[0020] The node may be configured to update parameters of the node's neural network based on the one or more gradients received from the one or more nodes in the network defined by the current connectivity model of the network. This may allow each node to optimize its own neural network and may be compatible with existing methods (e.g., stochastic gradient descent).

[0021] The node may also be configured to receive an updated collaboration pattern from the device, the updated collaboration pattern indicating other nodes in the cluster to communicate with the node. This allows nodes in the network to communicate so that data having similar statistical properties and related to a specific task in a hierarchy can be utilized.

[0022] The node can be used to send the output of the neural network of the node to the other nodes in the cluster indicated by the updated cooperation mode. Since only node devices with relevant data will exchange information between themselves, this can improve reasoning performance and communication efficiency.

[0023] The nodes can be used to process data related to tasks in the hierarchy of tasks. This allows nodes in the network to communicate so that data with similar statistical properties and related to specific tasks in the hierarchy can be utilized. This allows hierarchical tasks to be solved through distributed learning and reasoning.

[0024] The data processed by the node may be non-IID data, which has different statistical properties from data processed by one or more other nodes in the network. This may reflect many real-world applications where data does not have homogeneous properties.

[0025] According to another aspect, a computer-readable storage medium storing computer-readable instructions is provided, which, when executed at a computer system located at a device or node, causes the computer system to perform the above steps. The computer system may include one or more processors. The computer-readable storage medium may be a non-transitory computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present disclosure is described below by way of examples in conjunction with the accompanying drawings. In the accompanying drawings:

[0027] FIG1( a ) schematically shows a traditional star network topology.

[0028] Fig. 1(b) shows an example of a connectivity matrix.

[0029] FIG2( a ) schematically shows that multiple cooperation modes are derived from the connectivity matrix of the network through a parameter server.

[0030] FIG. 2( b ) schematically illustrates the information exchange in terms of the cooperation mode of each edge device and the loss vector between the parameter server and the edge devices 1 to N (nodes in the network).

[0031] Figure 3 An example of an objective function to be optimized so that a network of devices can complete a given task is shown.

[0032] Figure 4 The interaction between the optimization of the cooperation mode of the node devices and the optimization of their local models is schematically shown.

[0033] Figure 5 Schematically shows a network topology that can be used for non-iid image classification.

[0034] FIG6 schematically shows that the connectivity matrix reveals the inter-cluster and intra-cluster communication patterns of nodes at convergence.

[0035] Figure 7 An example of the results of a cluster-based topology design method for non-iid data is schematically shown.

[0036] Figure 8 A flowchart of steps illustrating an example of a method implemented at a device in a network for performing reasoning on hierarchical tasks.

[0037] Fig. 9 A flowchart of steps illustrating an example of a method implemented at a node in a network for performing reasoning on hierarchical tasks.

[0038] FIG. 10( a ) shows an example of a device that may act as a parameter server as described herein and some of its associated components.

[0039] FIG. 10( b ) shows an example of a node and some of its associated components as described herein.

[0040] Fig.11 Shown is the Top-1 classification accuracy for one of the five clients used in an experiment.

[0041] Fig.12 Shown is the Top-1 classification accuracy for one of the two clients used in another experiment. DETAILED DESCRIPTION

[0042] In the federated learning method, the main node device in the network (usually called the parameter server) usually has much larger memory and processing power than the node device acting as an edge device in the network (usually called the client device) in order to perform calculations and store results. In addition, the parameter server can coordinate the communication mode of information exchange in the network throughout the training and inference phases.

[0043] The communication pattern of nodes is called the network topology. In some implementations, the communication pattern in the network can be defined by a connectivity matrix. Therefore, the cooperation pattern of devices in the network can be stored as a matrix whose rows and columns represent the indexes of the devices and the matrix contains the numbers 1 and 0 as its elements. For example, an element of row i and column j equal to 1 indicates that devices i and j exchange information. A value of 0 indicates that the corresponding device does not communicate in the current round of message exchange.

[0044] In the methods described herein, communication patterns can be determined for devices in a network, and in some implementations, the devices can be advantageously clustered to perform resource and communication efficient distributed learning to solve multi-task reasoning problems. The result of the clustering is an optimal topology of the network tailored to the specific hierarchy of tasks to be solved by the system during the reasoning phase.

[0045] The network typically includes multiple nodes. For example, a node can be a client device and / or an edge device. Each node is a connection point in the network and can serve as an endpoint for data transmission or redistribution. The parameter server can be one of the nodes (for example, one of the client devices or edge devices) or a separate device in the network. Each node can include at least one processor and at least one memory. The memory stores code in a non-transient manner, which can be executed by one or more processors to implement the node in the manner described herein. The node can also include a transceiver for sending and receiving data.

[0046] The parameter server device may also include at least one processor and at least one memory. The memory stores code in a non-transient manner, which code can be executed by one or more processors to implement the node in the manner described herein. The parameter server device may also include a transceiver for sending and receiving data.

[0047] The network is preferably a wireless network. In an alternative implementation, the network may be a wired network.

[0048] The cooperation pattern of devices in the network is stored at the parameter server as a connectivity model. In the example described in this article, the connectivity model is a connectivity matrix (CM), the rows and columns of which represent the index of the devices, and the CM contains the numbers 1 and 0 as its elements. As described above, an element of row i and column j equal to 1 indicates that devices i and j exchange information. A value of 0 indicates that the corresponding device does not communicate in the current round of message exchange. Alternatively, the connectivity model can be another model or data structure that indicates one or more other nodes in the network to communicate with the corresponding node.

[0049] During training, the connectivity matrix can change its contents based on the changing communication patterns between nodes until convergence is reached. This occurs when any new changes in the nodes' collaboration patterns fail to significantly improve the objective function used to measure the network's performance in achieving a given task.

[0050] The network can be trained so that during the inference phase, the node devices in the network can perform multi-task inference on multiple tasks in a hierarchy of tasks. For example, in an image classification problem, the hierarchy of tasks may include:

[0051] - First level: The task is to classify the image into the correct category, i.e. one of C categories.

[0052] - Second level: After identifying the correct category, the task is to classify the objects in the image into their correct sub-categories.

[0053] -Level 3: After each object in the image is correctly identified, the task is to identify their specific properties.

[0054] Such networks can be trained to perform other types of tasks in areas such as natural language processing (NLP), medical imaging, robotics, autonomous driving, and neuroscience.

[0055] An implementation is described below with reference to FIG. 2( a ) and FIG. 2( b ).

[0056] A parameter server (PS) 201 in the network 200 is selected. A parameter server can be a device in the network that has a larger amount of memory and processing power than other devices in the network in order to perform calculations and store the results. A parameter server can be a node in the network that also performs data processing in the same manner as other nodes, or it can be a separate device that coordinates the communication mode of information exchange between nodes in the network throughout the training and inference phases without processing its own data for a specific task.

[0057] PS201 receives the current connectivity matrix 202 of network 200 and sends the corresponding cooperation mode of the index to each node in the network so that the node knows one or more other nodes in the network to communicate with them and / or nodes not to communicate with them. Two exemplary cooperation modes derived from matrix 202 are shown at 203 and 204.

[0058] Each collaboration mode may include multiple indexes, each index corresponding to a node i in the network. A value of 1 in a mode corresponding to node i indicates that a specific node with the collaboration mode communicates with node i. A value of 0 in a mode corresponding to node i indicates that a specific node with the collaboration mode communicates with node i.

[0059] As shown in Figure 2(b), at the beginning of the first iteration of the process, the connectivity matrix is ​​the initial connectivity matrix 205. The initial connectivity matrix can be a fully connected matrix, defining that every node must communicate with every other node in the network, and the value is "1" everywhere except the diagonal line with a value of '0' (which means that the node does not communicate with itself). Alternatively, the initial connectivity matrix can have random values.

[0060] PS 201 determines the cooperation mode of each node from the connectivity matrix and sends the corresponding cooperation mode to each node in a downlink message. Node devices 1, 2, 3, j, N-1 and N are shown at 206, 207, 208, 209, 210 and 211 respectively.

[0061] Each node device is used to implement a trained artificial intelligence model, such as a neural network. The weights of the model (also known as parameters) can also be updated during each iteration of the process.

[0062] During the training phase, each node device is used to process at least a portion of the training data set. Each node can be used to process a subset of the training data set. The training data set may include multiple pairs of training data, each pair of training data includes input data and corresponding expected outputs. In the example of image classification, each input data may include an image, and each expected output may include a classification of the corresponding image, which may be compared with the classification result output by the network.

[0063] During training, the global loss function of the tasks of the hierarchy can be minimized. The gradient is the rate of change of the loss function relative to the change of the neural network weights. During each iteration, each node device calculates its corresponding gradient and sends the gradient to the device indicated by the cooperation mode defined by the connectivity matrix 205.

[0064] Each node uses the gradients received from other nodes that it is instructed to collaborate with, as defined by the connectivity model and the derived collaboration pattern, to average its own gradients and the received gradients, and uses the average to update its corresponding model. Each node calculates a loss vector corresponding to a task in the hierarchy. Each node sends the loss vector to PS201 in an uplink message, as shown in Figure 2(b).

[0065] The parameter server receives the loss vector from all node devices and calculates the average loss of the network. The parameter server updates the connectivity matrix based on the average loss. The parameter server then sends the corresponding new cooperation mode to each node for the next iteration of the process. The above process can be repeated until the network loss function is minimized.

[0066] In summary, the parameter server uses the information from the connectivity model and sends the corresponding cooperation pattern to each node. Each node calculates its gradient for its corresponding local model and sends the gradient to the appropriate nodes defined in the received cooperation pattern. Each node aggregates the received gradients, updates the parameters of its local model and calculates the loss. Each node sends their corresponding loss back to the parameter server. This can be sent as a loss vector. The parameter server calculates the average global loss and decides whether to change the connectivity model to further reduce the average global loss. If necessary, the server can update the connectivity model. After the step of connectivity model optimization, each node performs a local model update using its local dataset.

[0067] Therefore, the parameter server device sends a corresponding current collaboration mode to each node in the network. Each corresponding current collaboration mode is derived from a current connectivity model of the network (e.g., a current connectivity matrix), and the current connectivity model indicates one or more other nodes in the network to communicate with the corresponding node. The parameter server receives a corresponding loss vector corresponding to a task of the hierarchy from each node in the network. The parameter server then updates the connectivity model of the network based on the received corresponding loss vector. Each node device receives the current collaboration mode from a parameter server device in the network. Each node device determines one or more gradients of a corresponding neural network implemented by the node. Each node sends one or more gradients to one or more other nodes in the network indicated by the current collaboration mode. Each node determines a loss vector corresponding to a task of the hierarchy and sends the loss vector to the parameter server device.

[0068] As described above, each node is used to implement a model, such as a neural network. In a preferred embodiment, the nodes do not exchange raw input data to protect privacy, but the nodes exchange features (e.g., gradients) of their neural networks in order to improve the model and performance. Therefore, in some implementations, for privacy or other reasons, the nodes may only send the output of the corresponding model they implement, rather than the raw data itself. The raw data is input into the model. The model can be used to encrypt the input data. Since the nodes do not share their raw data (i.e., data before being processed using the corresponding model implemented by the corresponding node), the nodes cannot decide who to communicate with before the training phase to improve their task performance.

[0069] During the inference phase, each node can only access data similar to that during the training phase. Therefore, there may be no benefit in training using data from other nodes with different distributions.

[0070] Due to the non-iid nature of typical real-world data and the heterogeneity of the tasks performed by different nodes in the network, aggregating information from all nodes may degrade task performance in some cases. The benefit of finding the optimal topology is that it reduces the communication cost between nodes during inference while improving task performance, because some subsets of the data are suitable for some tasks, but not all tasks.

[0071] Nonlinear optimization methods can be applied to the process of updating the connectivity matrix of the network to minimize the global average training loss. The connectivity matrix of the nodes in the network can be changed until the optimal pattern of node exchange gradients is obtained, that is, until convergence.

[0072] In a preferred implementation, finding the optimal network topology involves two alternating minimization steps, both of which are performed through cooperation between the parameter server and the node devices.

[0073] Minimizing the global average objective function (global loss) in the connectivity matrix of the network can be performed using a sequential least squares program with inequality constraints. Figure 3 An exemplary loss function and formula for a nonlinear optimization process for determining a connectivity matrix (CM) are shown, where is the global loss, n is the number of nodes (clients), It is a local loss.

[0074] The global average objective function (global loss) is minimized over the training data. This can be performed using methods such as stochastic gradient descent (SGD) with backpropagation. The global loss can be optimized so that the network of node devices can complete a given task.

[0075] Figure 4 The interaction between device cooperation mode optimization and its local neural network cooperation mode optimization is shown to find the optimal network topology through these two alternating optimization (minimization) steps.

[0076] These two optimization steps are performed simultaneously: updating the parameters of the mode implemented by each node device, and finding the best cooperative mode for communication between node devices. These optimization steps are performed jointly because each of them affects the performance of the other. The right balance between the execution frequency of these two steps can be determined as a salient feature of the optimization algorithm.

[0077] At 401, the connectivity matrix is ​​initialized using the connectivity matrix determined in the last iteration, or if this is the first optimization iteration, an initial matrix or a full connectivity matrix with random values ​​is used to initialize the connectivity matrix. At 402, the network receives a batch of training data and performs iterations of the above process. At 403, the current connectivity matrix is ​​updated to minimize the global average classification loss. This is performed after each iteration until convergence. As shown in 404, the model implemented by each node device is trained locally, for example, by SGD. The parameters of the model can be updated after each iteration until convergence. As shown in 405, if convergence has not yet occurred, the updated connectivity matrix can be used in the next iteration of the process.

[0078] Figure 5 The schematic diagram shows the communication in the network 500 during the inference phase of the non-iid image classification task, wherein the final connectivity matrix (CM) 501 has been determined by the above process. Each node device with index i processes a process represented by X (i) , where the local model (in this case a neural network) is specified as (W, b) by weights W and biases b. Nodes 1 to 5 of the network are indicated at 502, 503, 504, 505, 506 and 507 respectively. The arrows indicate the communication pattern between the nodes in the network, which is also given in numerical form by the connectivity matrix 501. Therefore, the connectivity matrix 501 gives the intra-node communication of the tasks performing the hierarchy.

[0079] As shown in Figure 6(a) and Figure 6(b), at convergence, the connectivity matrix can define clusters of node devices based on the shared statistical properties of the data processed at each node and the related tasks in a given hierarchy. The connectivity matrix can provide the cooperation pattern within each cluster, as well as which nodes communicate with which other nodes in other discovered clusters in the network.

[0080] The connectivity matrix shown at 601 in Fig. 6 (a) is the initial full connectivity matrix, wherein each node communicates with each other node in the network. The sparse structure shown at 602 is the final connectivity matrix when converged. Optionally, the final matrix can be further permuted. For example, the final matrix can be further permuted, and the purpose is to rearrange the elements of the matrix in some way to obtain a matrix that is block diagonal or close to the block diagonal. The block diagonal matrix is ​​a matrix whose diagonal elements are square matrices, as shown in the squares marked as clusters 1, 2 and 3 in the matrix 603 of Fig. 6 (b).

[0081] FIG6( b ) shows another example of a connectivity matrix 603 , where, at convergence, optionally after applying permutations to the final matrix obtained during training, the connectivity matrix reveals the inter-cluster (between clusters) and intra-cluster (between nodes in the same cluster) communication patterns of nodes.

[0082] The connectivity matrix 603 defines three node clusters: 604, 605, and 606. Each cluster includes a plurality of nodes. Cluster 604 includes four nodes, and clusters 605 and 606 include three nodes respectively. Each node in a cluster communicates with each other node in the cluster (intra-cluster communication). Nodes of the same cluster process data related to the same task in the hierarchical task structure. There may also be communication between node clusters (inter-cluster communication), as shown by the circled number "1" in Figure 6(b). In the training phase, clusters can exchange gradients of neural networks implemented by nodes in the cluster, as described above. In the inference phase, clusters can exchange results of specific tasks in the hierarchical task structure to be solved by the network. Clusters can only communicate with other clusters through one node in each cluster. Therefore, nodes in each cluster can be used to receive data from nodes in other clusters. Nodes in each cluster can be used to send data to corresponding nodes in one or more other clusters.

[0083] Figure 7 An example of the network topology obtained by the above process is schematically shown, wherein multiple nodes in the distributed network 700 have been clustered. Figure 7Not shown in the figure, the nodes are respectively used to receive their respective cooperation modes from the parameter server and send their respective losses to the parameter server during each iteration of the training process, as described above. Each node can receive its final optimized cooperation mode from the parameter server to be used in the inference phase to perform tasks in the hierarchy of tasks received by the node in the network. The cooperation mode of each node to be used in the inference phase is derived from the final connectivity model obtained in the training phase. The cooperation mode indicates other nodes in the network to communicate with a specific node. The network includes multiple node clusters 701, 702, 703, 704, 705, 706. Each cluster includes multiple nodes. The nodes in the cluster communicate with other nodes in the cluster (intra-cluster communication). The node cluster can also communicate with at least one other node cluster (inter-cluster communication). As described above, in the training phase, the clusters can exchange gradients of the neural networks implemented by the nodes in the cluster. In the inference phase, the clusters can exchange results of specific tasks in the hierarchy of tasks to be solved by the network. In Figure 7 In the example shown, cluster 701 communicates with clusters 702 and 703. Cluster 702 communicates with clusters 704 and 705 as well as cluster 701. Cluster 703 communicates with cluster 706 as well as cluster 701. Nodes in cluster 701 process data related to tasks on the first level of the hierarchy of tasks. Nodes in clusters 702 and 703 receive data from nodes in cluster 701. A single node in cluster 701 can send data to a single node in each of clusters 702 and 703. Clusters 702 and 703 process data related to tasks on the second level of the hierarchy of tasks, respectively. Clusters 704, 705, and 706 process data related to tasks on the third level of the hierarchy of tasks, respectively. Nodes in clusters 704 and 705 receive data from nodes in cluster 702. A single node in cluster 702 can send data to a single node in each of clusters 704 and 705. Nodes in cluster 706 receive data from nodes in cluster 703. A single node in cluster 703 may send data to a single node in cluster 706. Thus, at least one cluster of nodes in the network may receive data from one or more clusters of nodes that are above the level of the corresponding cluster in the hierarchy.

[0084] In some embodiments, during the training and / or inference phase, one or more of the following may be true. The hierarchy of multiple tasks may be known. The loss function to be optimized for each task may be given and known. Each node device may process a subset of data received by the network that is only relevant to one of the tasks in the hierarchy. The data may be non-iid, meaning that the data has different statistical properties depending on the node where the data is located. Each node device may not know which task its input data is useful for.

[0085] Therefore, the goal of the process is to find the best cooperation mode for the nodes so that each node finds the most suitable neighboring node in the network to exchange information with in order to maximize its own learning ability. The purpose is to exchange only information related to a specific task to reduce the communication cost between nodes. To this end, nodes can be grouped into clusters so that nodes of the same cluster process data related to / useful for the same task. Although in some embodiments, clusters of nodes can communicate with each other (inter-cluster communication) as described above, nodes can only exchange data with nodes of the same cluster (intra-cluster communication). The method can determine the appropriate ordering of inter-cluster communication based on the connectivity matrix.

[0086] Figure 8 An example of a general method 800 for performing reasoning on hierarchical tasks in a network is shown. The network includes a plurality of nodes, each node being used to process corresponding data related to a task in the hierarchical task. The method includes: at step 801, sending a corresponding current collaboration mode to each node in the network, wherein each corresponding current collaboration mode is derived from a current connectivity model of the network, and the current connectivity model indicates one or more other nodes in the network to communicate with the corresponding node. At step 802, the method includes: receiving a corresponding loss vector corresponding to the hierarchical task from each node in the network. At step 803, the method includes: forming an updated connectivity model of the network based on the received corresponding loss vectors.

[0087] Fig. 9 An example of a general method 900 performed at a node in a network for performing reasoning on a hierarchical task is shown. The network includes a plurality of nodes, each node for implementing a corresponding neural network to process corresponding data related to a task in the hierarchical task. The method includes: at step 901, receiving a current collaboration mode from a device in the network, wherein the current collaboration mode is derived from a current connectivity model of the network, and the current connectivity model indicates one or more other nodes in the network to communicate with the node. At step 902, the method includes: determining one or more gradients of the corresponding neural network implemented by the node. At step 903, the method includes: sending one or more gradients to one or more other nodes in the network indicated by the current collaboration mode. At step 904, the method includes: determining a loss vector corresponding to the hierarchical task. At step 905, the method includes: sending the loss vector to the device.

[0088] Figures 10(a) and 10(b) show examples of devices for implementing the above methods. Figure 10(a) shows an example of a device for acting as the above parameter server. Figure 10(b) shows an example of a device for acting as the above node. Devices 1000, 1050 may include at least one processor (e.g., processors 1001, 1051) and at least one memory (e.g., memory 1002, 1052). The memory stores code in a non-transient manner, which can be executed by one or more processors to implement the device in the manner described herein. The device may also include transceivers 1003, 1053.

[0089] The following describes the results of a specific embodiment of non-iid image classification using a cluster-based topology design method. This embodiment is represented by heterogeneous image classification at a node. For example, N number of node devices need to collaborate at each node to train a corresponding local neural network model to perform image classification.

[0090] Each node device can access a local dataset of non-iid images through the network. Some nodes have images with the same statistical properties, but the nodes may not know this information in advance. In other words, the nodes in the network do not know which other nodes can access the same type of images as themselves, or which nodes can access different types of images. Therefore, at each node, the statistical properties of the datasets being processed by other nodes in the network may be unknown.

[0091] In this example, a parameter server device is specified which coordinates the distributed learning process and the communication between the parameter server device and the nodes and between the nodes themselves. The network is assigned to classify images into their corresponding categories.

[0092] A large image training dataset is split into N smaller parts and distributed to each of the N nodes through the network. Images belong to C categories. Each image belongs to one of the C categories. The non-iid nature of the data stems from the fact that the categories are unevenly distributed in the network such that some nodes may only have access to one or a few categories in their corresponding training data and they do not own the entire set of images belonging to a certain category. Therefore, each node has access to only a small subset of images in a category and can therefore collaborate with other nodes with similar images to improve their classification performance.

[0093] The image classification problem is formulated as a hierarchy of tasks as follows. In this example, when the network receives an image, the hierarchy of tasks includes the following:

[0094] - First level: The task is to classify the image into the correct category, i.e. one of C categories.

[0095] - Second level: After identifying the correct category, the task is to classify the objects in the image into their correct sub-categories.

[0096] -Level 3: After every object in the image has been correctly identified, the task is to identify specific attributes of the object.

[0097] In preliminary experiments on tasks on the first level in the hierarchy, the Canadian Institute for Advanced Research (CIFAR-10) benchmark dataset (see https: / / en.wikipedia.org / wiki / CIFAR-10) was used to test the performance of classical federated learning for iid data image classification.

[0098] Exemplary results are in Fig.11 and Fig.12 The Top-1 classification accuracy is reported in , which refers to the fact that the category predicted by the neural network classifier is the same as the original category. The number of epochs refers to the number of times the entire dataset is passed to the neural network model for training. Training is performed by SGD with momentum and weight decay, with a learning rate equal to 0.1 and a batch size equal to 24.

[0099] Fig.11 The top-1 classification accuracy of one of the five node devices used in this experiment is shown. The validation accuracy is shown at 1101, and the training and test accuracies are shown at 1102 and 1103, respectively. Each node is trained using the entire CIFAR-10 dataset. The parameter server aggregates and averages the gradients, and the nodes and parameter server exchange gradients. The Resnet-18 architecture is used for the classification of CIFAR-10 images.

[0100] Fig.12 The top-1 classification accuracy of one of the two node devices used in this experiment is shown. The validation accuracy is shown at 1201, and the training and test accuracies are shown at 1202 and 1203, respectively. Each node has half of the CIFAR-10 dataset for training. The parameter server aggregates and averages the gradients, and the nodes and parameter server exchange gradients. The Resnet-18 architecture is used for the classification of CIFAR-10 images.

[0101] like Fig.11 and Fig.12 As shown, even in simple cases where less data is available at the nodes, the image classification accuracy drops significantly. However, using the approach described in this paper, the classification accuracy can be improved relative to previous techniques using distributed learning.

[0102] The star topology (Figure 1), which is the state-of-the-art approach in the literature, can be very suboptimal in the case of non-iid data. Improved performance can be achieved if the traditional star topology is replaced with a customized topology for each specific use case where distributed learning is used.

[0103] Even with a small number of nodes in the network, the number of possible topologies is exponential. The method described in this paper provides an efficient search method for finding the optimal topology for a task with a given hierarchy. The method can be used in image classification applications to provide customized network topologies that ensure the highest possible accuracy for image classification on each device. The method can also address the non-iid and privacy nature of the data and multi-task reasoning issues.

[0104] Some other advantages of the methods described herein include the following.

[0105] At inference time, only a few devices in a given cluster communicate with each other, which results in a significant overall reduction in the communication cost in the network, leading to improvements such as energy saving and battery life of edge devices. This feature is a result of the sparse nature of the connectivity matrix. Clusters can communicate with each other through only one node, which further reduces the communication cost. Since each discovered cluster is tailored for a specific task in the hierarchy of tasks given at the beginning of the optimization process, the network achieves higher performance in terms of the task.

[0106] Furthermore, each cluster of nodes is obtained in such a way as to utilize data with similar statistical properties and relevant to a specific task in the hierarchy. This improves both inference performance and communication efficiency. The latter is possible because only devices with relevant data can exchange information between themselves, rather than with other nodes, for which the data is not useful for their specific learning tasks.

[0107] Furthermore, the hierarchical nature of the obtained topology makes the information flow in the network efficient and energy-efficient. Advantageously, the data remains private at each client, as the clients can be trained indirectly through clustering and optimization methods, rather than by directly exchanging raw data. The method is scalable to a large number of nodes.

[0108] The network topology design method described in this paper is given by an optimization process that alternates between optimizing the cooperation mode of the devices and the cooperation mode of the neural network model, where classic stochastic gradient descent training is used on the local dataset. This optimization process can cluster the devices according to the shared statistical properties of the data from the network and fit the related tasks in the hierarchy to be solved by the network.

[0109] The optimization process can involve the interplay between selecting the best cooperation mode of the devices and updating the model parameters of the local neural network at the device. This allows for indirect training of the local neural network on the edge device without exchanging raw data, thereby preserving privacy. This can be an important feature in complex real-world applications where end-user privacy is highly desired.

[0110] The method optimizes task performance and communication cost by performing collaborative distributed learning for a given number of client devices in a network coordinated by a parameter server. When the tasks of the hierarchy are known and the data at each node is non-iid and private, the method can be used to find the optimal network topology to perform multi-task reasoning. Therefore, the method can help clients find the best way to collect and send information in the network, thereby solving the multi-task problem faced by the entire network.

[0111] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, without considering whether such features or combinations of features can solve any problems disclosed herein, and without limiting the scope of the claims. Applicants point out that various aspects of the present invention may include any such individual features or combinations of features. In view of the above description, it will be clear to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A device (201, 1000) for performing reasoning on hierarchical tasks in a network (200), the network comprising a plurality of nodes (206, 207, 208, 209, 210, 211, 1050), each node being used to process corresponding data related to a task in the hierarchical tasks, the device being used to: Sending (801) a corresponding current collaboration mode (203, 204) to each node in the network, wherein: Each respective current cooperation mode is derived from a current connectivity model (202, 205, 601) of the network, the current connectivity model (202, 205, 601) indicating one or more other nodes in the network with which the respective node is to communicate; receiving (802) from each node in the network a respective loss vector corresponding to a task of the hierarchy; Based on the received corresponding loss vectors, an updated connectivity model (602) of the network is formed (803).

2. The device according to claim 1, wherein: Each respective loss vector is determined based on one or more gradients of a respective neural network implemented by a respective node in the network and each node in the network that is used to communicate with the respective node based on the current connectivity model of the network.

3. The device according to claim 1 or 2, wherein: The updated connectivity model defines a plurality of node clusters (604, 605, 606, 701, 702, 703, 704, 705, 706), wherein each node in a cluster is operable to communicate with other nodes in the cluster.

4. The device according to claim 3, wherein: The updated connectivity model also defines an inter-cluster cooperation mode for each of the plurality of node clusters.

5. The device according to claim 3 or 4, wherein: The nodes of the cluster are respectively used to output data related to the same task in the tasks of the hierarchical structure.

6. A device according to any one of the preceding claims, wherein: The corresponding data processed by each node in the network is non-IID data, and the data has different statistical properties depending on which node in the network processes the data.

7. A device according to any one of the preceding claims, wherein: The device is also used for: combining the corresponding loss vectors received from each node in the network to determine a combined loss value; The updated connectivity model is formed according to the combined loss value.

8. A device according to any one of the preceding claims, wherein: The device is used to form the updated connectivity model to minimize the global average training loss of the tasks of the hierarchy.

9. A node (1050) in a network (200) for performing reasoning on a hierarchical task, the network comprising a plurality of nodes (206, 207, 208, 209, 210, 211, 1050), each node being configured to implement a corresponding neural network to process corresponding data associated with a task in the hierarchical task, the node being configured to: A current collaboration mode (203, 204) is received (901) from a device in the network, wherein: The current cooperation mode is derived from a current connectivity model (202, 205, 601) of the network, the current connectivity model (202, 205, 601) indicating one or more other nodes in the network with which the node is to communicate; determining (902) one or more gradients of the corresponding neural network implemented by the node; sending (903) the one or more gradients to one or more other nodes in the network indicated by the current collaboration mode; determining (904) a loss vector corresponding to a task of the hierarchy; The loss vector is sent (905) to the device.

10. The node according to claim 9, wherein: The node is also used to: receiving one or more gradients of the one or more corresponding neural networks implemented by one or more other nodes in the network defined by the current connectivity model of the network; The loss vector corresponding to the task of the hierarchy is determined based on the received one or more gradients.

11. The node according to claim 10, wherein: The node is configured to update parameters of a neural network of the node based on the one or more gradients received from the one or more nodes in the network defined by the current connectivity model of the network.

12. A node according to any one of claims 9 to 11, wherein: The node is further configured to receive an updated cooperation mode from the device, wherein the updated cooperation mode indicates other nodes in the cluster that are to communicate with the node.

13. The node according to claim 12, wherein: The node is used to send the output of the neural network of the node to other nodes in the cluster indicated by the updated cooperation mode.

14. A node according to any one of claims 9 to 13, wherein: The nodes are used to process data related to tasks in the hierarchical structure.

15. A node according to any one of claims 9 to 14, wherein: The data processed by the node is non-IID data, which has different statistical properties from data processed by one or more other nodes in the network.

Images