Methods, systems, and apparatus, including computer programs, for federated learning

Abstract

Broadly speaking, the present technology relates generally to methods, systems and apparatuses for training ML (machine learning) models using federated learning. In particular, there is provided a method of training a ML (machine learning) model using a plurality of client devices performing federated learning, the method comprising determining a computing power of each client device, associating each client device to a value based on the determined computing power, the value defining how many of each neural network layer in the ML model is to be included in a sub-model to be trained by each client device, and generating sub-models of the ML model by using the value associated with each client device to perform an ordered pruning of at least one neural network layer of the ML model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to methods, systems, and apparatuses for training machine learning (ML) models using federated learning. In particular, the present disclosure relates to a computer-implemented method for performing ordered dropout to dynamically prune an ML model being trained based on the capabilities of the device performing the training without requiring subsequent retraining of the model. BACKGROUND

[0002] In the past few years, the progress in deep learning has revolutionized the way people interact with everyday devices. These successes have largely relied on the availability of large-scale training infrastructure and the collection of large amounts of training data. However, users and vendors are increasingly aware of the privacy concerns that come with this increasing collection of data, leading to various privacy-protecting initiatives by service providers and government regulators.

[0003] Federated learning is a relatively new subfield of machine learning, ML, that allows for training of ML models without requiring training data to leave a user device. Instead, federated learning allows individual client devices to collaboratively train an ML model by moving the training computation to the client devices while keeping all training data private. In each round of training, participating client devices download the latest version of a global model and use their local data (i.e., data local to the client or stored on the client device) to compute an updated model. These locally trained models are then sent back from the participating client devices to a central server, which aggregates all of the locally trained models received to generate an updated version of the global model. This is in contrast to centralized training, in which a training mechanism (e.g., a central server) has access to all of the training data. The client devices can be mobile devices like smartphones, home appliances, wearable devices, or even servers or systems of entities such as hospitals and institutions. Thus, there is a need to keep data private on the client devices and avoid sharing data with the central server to train the global model from the perspective of data privacy and security.

[0004] Another key challenge in deploying federated learning in natural environments is the vast heterogeneity of client devices, from low-end Internet of Things (IOT) devices to flagship high-end smartphones. The different computing capabilities of these devices make it difficult to achieve federated learning of a global model within a reasonable time frame. Typically, the solution is to prevent bottom-end devices from participating in federated learning (which can introduce bias in the end model) or to reduce the size of the model so that it can be trained using a large number of devices (which can result in a less accurate model).

[0005] Accordingly, the present applicant has recognized the need for an improved technique to perform federated learning on client devices. SUMMARY

[0006] TECHNICAL SOLUTION

[0007] In a first method herein, there is provided a method (implemented by a server or master node or coordinator) for training an ML (machine learning) model using federated learning performed by a plurality of client devices, the method comprising: determining a capability of each client device; associating each client device with a value, the value defining how many of each neural network layer in the ML model is to be included in a sub-model to be trained by each client device based on the determined capability, and generating sub-models of the ML model to perform an ordered pruning of at least one neural network layer of the ML model by using the value associated with said each client device. BRIEF DESCRIPTION OF DRAWINGS

[0008] Implementations of the present technology will now be described, by way of example only, with reference to the attached drawings in which:

[0009] Figure 1 A schematic illustration of random dropping is shown;

[0010] Figure 2A A schematic illustration of ordered dropping according to exemplary embodiments of the present disclosure is shown;

[0011] Figure 2B A schematic illustration of ordered dropping according to exemplary embodiments of the present disclosure is shown;

[0012] Figure 2C A schematic illustration of ordered dropping according to exemplary embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic illustration of federated learning using ordered dropping to generate nested sub-models for client device training according to exemplary embodiments of the present disclosure is shown;

[0014] Figure 4A is a flowchart of example steps performed by a server to train an ML model using federated learning according to exemplary embodiments of the present disclosure;

[0015] Figure 4B is a flowchart of example steps performed by a server to train an ML model using federated learning according to exemplary embodiments of the present disclosure;

[0016] Figure 5 Experimental results are shown for implementing federated learning using ordered dropping according to exemplary embodiments of the present disclosure;

[0017] Figure 6AExperimental results showing performance versus dropout rate for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0018] Figure 6B Experimental results showing performance versus dropout rate for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0019] Figure 6C Experimental results showing performance versus dropout rate for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0020] Figure 7A Results of an ablation experiment for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0021] Figure 7B Results of an ablation experiment for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0022] Figure 7C Results of an ablation experiment for federated learning using ordered dropout according to demonstrative embodiments of the present disclosure are shown;

[0023] Figure 8A Experimental results testing the scalability of the present technology according to demonstrative embodiments of the present disclosure are shown;

[0024] Figure 8B Experimental results testing the scalability of the present technology according to demonstrative embodiments of the present disclosure are shown;

[0025] Figure 9A Experimental results testing the scalability of the present technology according to demonstrative embodiments of the present disclosure are shown;

[0026] Figure 9B Experimental results testing the scalability of the present technology according to demonstrative embodiments of the present disclosure are shown;

[0027] Figure 10 An example use of the present technology for training an ML model according to demonstrative embodiments of the present disclosure is shown;

[0028] Figure 11 Another example use of the present technology for training an ML model according to demonstrative embodiments of the present disclosure is shown;

[0029] Figure 12 is a system for training an ML model using federated learning according to demonstrative embodiments of the present disclosure; and

[0030] Figure 13 is a flowchart of example steps performed by an apparatus using federated learning to train an ML model according to demonstrative embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] In a first method of the present technology, there is provided a method (performed by a server or master node or coordinator) of training an ML (machine learning) model using federated learning performed by a plurality of client devices, the method comprising: determining capabilities of each client device; associating each client device with a value based on the determined capabilities, the value defining how many of each neuron network layer in the ML model is to be included in a sub-model to be trained by each client device, and generating sub-models of the ML model by using the value associated with each client device in order to perform an ordered pruning of at least one neuron network layer of the ML model.

[0032] In a related method of the present technology, the method of training an ML (machine learning) model using federated learning performed by a plurality of client devices further comprises: during each federated learning round, assigning to each client device a sub-model of the ML model based on the value associated with each client device.

[0033] In a related method of the present technology, there is provided a central computing device (which can be a server, master node or coordinator device) for training an ML (machine learning) using federated learning performed by a plurality of client devices. The central computing device comprises at least one processor coupled with a memory, configured to: determine capabilities of each client device; associate each client device with a value based on the determined capabilities, the value defining how many of each neuron network layer in the ML model is to be included in a sub-model to be trained by each client device; generate sub-models of the ML model by using the value associated with each client device in order to perform an ordered pruning of at least one neuron network layer of the ML model.

[0034] The central computing device is further configured to: during each federated learning round, assign to each client device a sub-model of the ML model based on the value associated with each client device.

[0035] A machine learning (ML) model can be viewed as a super model from which multiple nested sub-models can be extracted and trained. The term "nested sub-models" as used herein means that the size of the sub-models is hierarchical and one sub-model is inside another sub-model. Thus, a sub-model sent to a client device for training can itself contain nested sub-models. For example, a super model can contain multiple nested sub-models A, B, C, D, and E, where A is the largest sub-model (more suitable for higher tier client devices) and E is the smallest sub-model (more suitable for lower tier client devices). Sub-model A contains sub-models B through E in a nested manner, sub-model B contains sub-models C through E in a nested manner, and so on. Thus, when sub-model A is sent to a client device, the client device can choose to train 5 models: A, B, C, D, or E. Similarly, when sub-model E is sent to a client, the client device can only train sub-model E. Thus, it can be appreciated that due to the nested nature of the sub-models in the super model, a sub-model can itself contain sub-models.

[0036] Beneficially, when a sub-model is sent to a client device for training via federated learning, each client device can train the received sub-model without needing to train the larger / entire super model. For example, if sub-model A (containing models B through E) is sent to a client device, the client device can train sub-model A. Similarly, when a sub-model is sent to a client device, each client device can extract one nested sub-model from the received sub-model based on the current capabilities of the client device and train the extracted sub-model rather than attempting to train the entire sub-model. For example, if sub-model A is sent to a client device but the sub-model is too large for the client device to train at the moment, the client device can extract one of the nested sub-models (B through E) and train that extracted nested sub-model. Thus, client devices of different specifications are able to participate in the training process. Moreover, a client device can extract and train a sub-model from a received sub-model based on its capabilities or resources at the time of training. That is, a client device that is being used to perform other tasks at the time of training (e.g., capturing images, participating in a call, etc.) can still participate in the training by extracting a sub-model based on the resources available to train at a particular time.

[0037] The sub-models sent to the client devices are based on the maximum or ideal computing capabilities of each client device. This ensures that the client device receives a sub-model that it will be able to train. However, as noted above, a sub-model can itself contain nested sub-models, which allows each client device to extract a smaller sub-model if it does not have the resources to perform training on the received sub-model.

[0038] Associating each client device with a value can include associating each client device with a value from a distribution of discrete values. The distribution of discrete values can be a uniform distribution. Alternatively, the distribution of discrete values can be a non-uniform distribution that reflects a distribution of capabilities of the plurality of client devices.

[0039] The number of discrete values in the distribution of discrete values can depend on any one or more of: the number of device capability levels, the size or width of the neuron network layer, and the number of device capability levels included in the ML model.

[0040] Generating the plurality of nested submodels of the ML model can include performing an ordered pruning of each prunable neuron network layer or plurality of prunable neuron network layers of the ML model using the value associated with each client device. That is, two or more prunable (i.e., “prunable”) neuron network layers can be pruned using the same value associated with each client device. For example, the prunable layers can be convolutional layers and linear layers. For example, layers such as “batchNorm” require special handling by the submodel, RNN layers can be implemented as linear layers, and thus benefit from the same advantages.

[0041] Alternatively, generating the plurality of nested submodels of the ML model can perform an ordered pruning of one neuron network layer of the ML model using the value associated with each client device and an ordered pruning of at least one other neuron network layer of the ML model using at least one other value. That is, two prunable neuron network layers can be pruned with two different values.

[0042] During each federated learning training round, distributing the submodels of the ML model to each client device can include broadcasting a submodel generated using one value to each client device associated with the value for local training by the client device.

[0043] The method can further include during each federated learning training round, receiving from each client device a gradient of the maximally updated submodel, wherein each gradient corresponds to a change in a weight of the submodel.

[0044] The method can further include using the received gradients, aggregating the changes in the weights of the submodels received from each client device and updating the ML model.

[0045] The method can include repeating the steps of generating and distributing using the updated ML model.

[0046] In a second method of the present technology, there is provided a system for training an ML (machine learning) model using federated learning performed by a plurality of client devices, the system comprising: a plurality of client devices; at least one processor coupled to a memory configured to: determine a capability of each client device; associate each client device with a value based on the determined capability, the value defining how many of each neural network layer in the ML model is to be included in a sub-model to be performed by each client device; and generate a sub-model of the ML model by using the value associated with each client device to perform an ordered pruning of at least one neural network layer of the ML model.

[0047] The at least one processor in the system is configured to: during each federated learning training round, assign each client device a sub-model of the ML model based on the value associated with each client device.

[0048] The features described above in relation to the first method apply equally to the second method.

[0049] In a third method of the present technology, there is provided an apparatus for training machine learning (ML) using federated learning, the apparatus comprising: a storage device storing private training data collected by the apparatus; and at least one processor coupled to a memory configured to: receive a sub-model of an ML model from a server, the sub-model generated based on a capability of the apparatus; train the sub-model using the stored private training data and update weights of the sub-model; and send gradients of the maximum updated sub-model to the server, wherein each gradient corresponds to a change in a weight of the sub-model.

[0050] The processor can be configured to: determine a current computing capability of the apparatus; and extract a smaller nested sub-model from the received sub-model for training using the stored private training data.

[0051] The apparatus can further comprise: at least one image capture device for capturing images or video to be used as the private training data; and / or, at least one interface for collecting data forming the private training data.

[0052] In a related method of the present technology, there is provided a non-transitory data carrier carrying processor control code to implement the methods described herein.

[0053] As will be appreciated by one skilled in the art, the present technology can be embodied as a system, method, or computer program product. Accordingly, the present technology can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware.

[0054] Furthermore, the technology can take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. For example, the computer readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0055] Computer program code for carrying out operations of the technology can be written in any combination of one or more programming languages, including an object oriented programming language, a functional programming language, and conventional procedural programming languages. The code components can be embodied as procedures, methods, or the like, and can include sub-components, each of which can take the form of an instruction or a set of instructions arranged to perform a particular operation or set of operations. The code components can be stored on a computer readable medium, which can be any medium, tangible or intangible, for storing code components.

[0056] Embodiments of the technology also provide a carrier containing code, which when executed on a processor, can cause the processor to perform any of the methods described herein.

[0057] The technology further provides processor control code, for example, on a general purpose computer system or on a digital signal processor (DSP), to implement the above described methods. The technology also provides a carrier containing the processor control code, for example, on a non-transitory data carrier, to perform any of the methods described above when the code is executed. The code can be provided on a carrier, such as a disk, a microprocessor, a CD or DVD-ROM, in a program memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or in a data carrier such as an optical or electrical signal carrier. Code (and / or data) implementing embodiments of the technology described herein can comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for setting up or controlling hardware

[0058] It will also be apparent to those skilled in the art that all or part of the logic method according to embodiments of the present technology can be embodied in a logic apparatus that comprises logic elements that perform the steps of the methods described above, and that these logic elements can comprise components such as logic gates for example in a programmable logic array or application- specific integrated circuit. Such logic configuration can be further embodied in enablement elements for temporarily or permanently establishing the logic structures in such arrays or circuits using for example virtual hardware description languages, which can be stored and transmitted using fixed or transmittable carrier media.

[0059] In one embodiment, the present technology can be realized in the form of a data carrier having functional data embodied therein, the functional data comprising functional computer data structures that, when loaded into a computer system or network and operated upon thereby, enable the computer system to perform all the steps of the above-described methods.

[0060] The above-described method can be executed in whole or in part on a device, i.e., an electronic device, using a machine learning or artificial intelligence model. The model can be processed by an artificial intelligence dedicated processor, which is designed in a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model can be obtained by training. Here, "obtained by training" means that a basic artificial intelligence model is trained using a plurality of training data by a training algorithm to obtain a predefined operation rule or an artificial intelligence model configured to perform a desired feature (or purpose). The artificial intelligence model can include a plurality of neuron network layers. Each layer of the plurality of neuron network layers includes a plurality of weight values, and the calculation of the neuron network is performed by a calculation between the calculation result of the previous layer and the plurality of weight values.

[0061] As described above, the present technology can be implemented using an AI model. An AI-related function can be executed by a non-volatile memory, a volatile memory, and a processor. The processor can include one or more processors. At this time, the one or more processors can be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an artificial intelligence dedicated processor such as a neural processing unit (NPU). The one or more processors control the processing of input data according to a predefined operation rule or an artificial intelligence (AI) model stored in a non-volatile memory or a volatile memory. The predefined operation rule or the artificial intelligence model is provided by training or learning. Here, provided by learning means that an AI model of a predefined operation rule or a desired feature is formed by applying a learning algorithm to a plurality of learning data. Learning can be performed in the device itself (AI according to an embodiment is executed in the device), and / or can be performed through a separate server / system.

[0062] An AI model can be composed of multiple neuron network layers. Each layer has multiple weight values and performs a layer operation by computing the operation of the previous layer and the multiple weight values. Examples of neuron networks include, but are not limited to, a convolutional neuron network (CNN), a deep neuron network (DNN), a recurrent neuron network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neuron network (BRDNN), a generative adversarial network (GAN), and a deep Q network.

[0063] A learning algorithm is a method for training a predetermined target device (e.g., a robot) that uses multiple learning data to cause, allow, or control the target device to make a judgment or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0064] Broadly speaking, the present technology is generally directed to methods, systems, and apparatuses for training a machine learning (ML) model using federated learning. In particular, the present disclosure is directed to a computer-implemented method that performs ordered dropping based on capabilities of client devices to prune an ML model and generate nested submodels for client device training. Beneficially, this means that nested submodels of the ML model can be trained with private training data that is only stored on the client devices, thereby enabling the ML model to be trained on useful training data without compromising the privacy of users of the client devices. Further beneficially, this means that training of the ML model can be distributed across multiple devices in a manner that takes into account the computational capabilities of the devices.

[0065] As explained in more detail below with respect to the accompanying drawings, the present technology provides a new adaptive training framework that enables heterogeneous devices to participate in federated learning by dynamically adjusting the model size, and thus the computation, memory, and data exchange size, to the available client device resources. To this end, ordered dropping (OD) is introduced. OD is a run-time ordered (importance-based) pruning mechanism that enables submodels to be extracted and trained in a nested fashion. Thus, OD can enable all client devices to independently participate in the federated learning process by training submodels of the original global model, while still providing knowledge to the global model. In addition to OD, the present technology provides a self-distillation method based on the largest supported submodel on the client devices to strengthen the feature extraction of smaller nested submodels. Finally, based on the hardware and load constraints of the devices, the framework has the additional advantage of generating models that can be dynamically scaled during inference.

[0066] As explained below, evaluation of the federated learning process using ordered dropout shows that the present technology provides significant accuracy benefits over baselines on different datasets and networks, while allowing extraction of submodels of different FLOPs (floating point operations per second) and sizes without retraining.

[0067] By design, federated learning assumes that data is not independent or uniformly distributed across client devices (i.e., non-IID), and that client devices can be unreliable (e.g., offline). However, it should be noted that the present technology can also be applicable to IID settings.

[0068] Federated learning of a typical machine learning (ML) model requires multiple rounds where i) a number of participating client devices are selected, ii) the latest ML model is sent from a central server or coordinator of the federated learning to the client devices, iii) the client devices use local data to update the ML model (local training), iv) the client devices share the updated model with the server, v) the server aggregates the generated models (one per client) into a single ML model before starting another round. There are also some other mechanisms that can ensure privacy, security, and robustness.

[0069] However, federated learning is typically bandwidth-intensive, as the entire global model needs to be sent to the client devices participating in each round (downstream), and these devices in turn need to send the updated gradients (upstream). In addition, federated learning is typically resource-intensive, as training neural networks significantly spends more computational resources and memory due to the backpropagation steps of the training process and its data dependencies (e.g., activations, optimization data, etc.).

[0070] Further, client devices in the natural environment can be very heterogeneous, with a large number of different computing capabilities. For example, smartphones can range from low-tier versions at a low price to flagship high-tier devices, each with potentially unique data. Even simple Internet of Things (IoT) devices, another low-tier device, can store data relevant to training ML models, as can wearable devices, virtual assistive devices, and other limited resource consumer devices. Since some client devices are resource-constrained, they can prolong the training process, as training can need to wait for the client devices to perform training and return updated gradients, i.e., training can need to wait for stragglers. Likewise, some client devices do not participate in the training process at all, so they are actively removed or excluded from model training due to their inability to perform training or perform training within the required time frame. This has an impact on the fairness, bias, and accuracy of the global model, as “invisible” data is missing from the training process. Since it is a low-tier device (most of which are resource-constrained and thus tend not to participate), the ML model trained by federated learning is at risk of being trained almost entirely on high-tier devices, which can cause bias due to different social demographics of owning low-tier devices versus owning high-tier devices and generally just missing critical data.

[0071] Current solutions typically either obtain bandwidth gains by compressing the communication channel (e.g., via pruning) or use dropping techniques to reduce the load of clients in the training process. Most current solutions ignore the computational and / or memory heterogeneity of the clients, thus not allowing for resource-constrained training.

[0072] Accordingly, there is a need to train large and global accurate models across heterogeneous devices in a federated setting. The present technology meets this need with ordered dropping, which is facilitated by carefully constructed network pruning. More specifically, the present technology orders knowledge representations in nested submodels of an original global model. In other words, a large global model can be used to generate smaller submodels, which are formed by pruning portions of the neural network under the global model. The models are nested, in the sense that a larger submodel contains all smaller submodels. The smaller submodels can be more amenable to training using resource-constrained devices. The submodels can be generated to accommodate the computing capabilities of the client devices participating in the federated learning process. This increases the likelihood of resource-constrained devices participating in the training process, thereby reducing the risk of the above-mentioned problems.

[0073] The present technology allows for an adaptive training process that allows heterogeneous devices to participate in training by adapting to the computational and memory requirements of a given resource. The present technology also entails a load balancing computation that limits stragglers. The trained model exhibits an ordered structure that enables efficient deployment to a wide range of devices. The generated model can also be dynamically adjusted during inference to adapt to a wide range of devices and runtime conditions (e.g., device load) at deployment.

[0074] Figure 1 A schematic of random dropout applied to neurons and filters in, for example, a convolutional layer is shown. Random dropout is a mechanism in which a number of units (i.e., neurons or channels) are randomly dropped from computation within a given layer of a neural network of a machine learning model. Random dropout is typically used for regularization purposes during the training process. In each batch, a different, random set of units is dropped. The number of units to be kept is typically controlled by a probability p. For example, in Figure 1 , at each layer at least one neuron is kept and the associated connections between the selected neurons are kept, while the dark neurons are randomly selected to participate, and the remaining greyed-out neurons will not participate in the computation.

[0075] Figure 2A , Figure 2B and Figure 2C A schematic of ordered dropout is shown in accordance with exemplary embodiments of the present disclosure. Ordered dropout (OD) is a mechanism of ordered importance-based pruning. The variable p e (0, 1] still defines how many to be included (dark shade) of each layer of the network, but the dropped neurons (greyed-out) are not randomly selected. Rather, they are dropped in a predefined order. For example in Figure 2A , Figure 2B and Figure 2C The leftmost units of p are kept, while the right (1 - p) units are dropped: Compare in particular Figure 2A and Figure 1 , both of which show an example of p = 0.2. In this way, the pruning submodels of p are nested, with the submodels of higher p values containing the same neurons as the submodels of lower p values. As will be discussed later, Figure 2A , Figure 2B and Figure 2CThe pruning sub-models of p in the p = 0.2, 0.5, 0.75 each can correspond to a class of devices with a specific computational and memory footprint, i.e. for higher level devices, there is a higher p value. In one embodiment, the class of devices means that the hierarchical level of the convergence of multiple heterogeneous devices depends on their hardware capabilities. The p value can be determined according to the class of the device.

[0076] By performing OD during training, the network is encouraged to emphasize learning towards the highest ranked units, as these units are always less likely to be dropped. This also means that the computation can be adapted by dynamically dropping the least important neurons during inference.

[0077] Figure 3 A schematic diagram of federated learning using ordered dropout to generate nested sub-models for training of client devices is shown, in accordance with the present exemplary embodiments. A remote or cloud server stores a master global ML model that will be trained using federated learning. Ordered dropout (OD) is used for ordered, importance-based pruning of the global ML model. By pruning the ML model, sub-models of the global model are generated, where each sub-model has a desired computational and memory footprint that makes it suitable for training with one or more client devices. The generation of the sub-models is controlled by a variable p e (0, 1] that defines how many of each layer of the network are to be included in the target sub-model. Based on the hardware capabilities of the devices, each device d is divided into a device level c and associated with a maximum p value (p (c) max ) that defines the maximum number of neurons that can be pruned from each layer of the global model. Thus, as shown in Figure 3 a sub-model with more units dropped can be generated for a low level device such as a wearable device, and a sub-model with no units dropped can be generated for a higher level device such as a high-end smartphone.

[0078] At each FL training round, sub-models are assigned to each participating device based on their capabilities. Since p (c) max < = 1, this means that the present technology allows i) a downstream bandwidth gain (as smaller models are assigned), ii) a computational gain (by not extending the computation to pruned neurons) and iii) an upstream gain (by sending only the gradients of the non-pruned neurons).

[0079] The present technology also provides the following benefits:

[0080] The ability to train a more global model, as the global model size and target device capabilities are decoupled.

[0081] More devices can participate in the training process, and their non-IID (independent and identically assigned) data will be included in the training process, which brings improved fairness and accuracy.

[0082] Because the model of a particular device is overweight, there are fewer laggards, which leads to faster convergence.

[0083] Figure 4A This is a flowchart of example steps performed by a server to train an ML model using joint learning and ordered dropout according to an exemplary embodiment of this disclosure. Generally speaking, the ordered dropout method is parameterized for: i) the dropout rate value p∈(0,1] for each layer, ii) a set of candidate values ​​P such that p∈P.

[0084] The key hyperparameter of OD is the dropout rate P, which defines how much of each layer should be included, with the remaining units discarded in a structured and ordered manner. The value of P is derived from the dropout distribution D. p Selected by sampling, D p It consists of a set of discrete values ​​P = [s1, s2, ..., s... |P|}(satisfying 0 < s1 < ... < s |P| ≤1) and probability (satisfy It is represented by ) . For example, a uniform distribution on P is represented by ) to indicate (i.e.) In the experiment described below, the set A uniform distribution on the surface is called (or uniform-k). The discrete property of the distribution is derived from the inherent number of discrete elements of the selected neurons or filters.

[0085] The method includes determining the capabilities of each of a plurality of client devices to be used for training the ML model (step S410). For example, this may include determining the computational or processing power and memory capacity of each client device. The hardware capabilities of each client device can be obtained from the client device (e.g., by requesting information), in which case the temporary computational load of the device can also be used to determine the computational or processing power and memory capacity of each client device.

[0086] In one embodiment, the hardware capabilities of each client device can be measured by a test operation passing through the entire ML model. When a new device participates in federated learning, the entire ML model is sent to the new device and executed. Through the execution of the ML model, the hardware capabilities of the new client device can be measured and obtained by the central computing device. The central computing device can determine the p-value of the new client device according to the obtained hardware capabilities of the new client device. In one embodiment, the central computing device can determine the class of the new client device according to the obtained hardware capabilities of the new client device. The p-value of the new client device can be determined according to the class of the new client device.

[0087] The method can include associating each client device with a value P (dropout rate) based on the determined capabilities, the value P defining how many of each neural network layer of the ML model will be included in the sub-model implemented by each client device (step S420), wherein the remaining neurons / channels are dropped in an ordered manner. Client devices can be grouped into groups or sets based on having similar capabilities. For example, client devices with limited processing and / or memory capabilities can be grouped together (e.g., low-tier smartphones and smartwatches), while client devices with good processing and / or memory capabilities can be grouped together (e.g., high-tier smartphones and tablets). This enables the sub-models to be sent to a group of client devices, i.e., having the same p (c) max of devices.

[0088] The method can include generating sub-models of the ML model by using the values associated with each client device to perform an ordered pruning of at least one neural network layer of the ML model (step S430).

[0089] The ordered pruning (also referred to herein as ordered dropout) operates as follows. The dropout rate p is sampled from a given distribution of discrete candidates p (e.g., a uniform distribution of discrete values p = [s1, s2,...], 0 < s1 < s2 <... < s |P| < = 1, denoted as p ~ U P (i.e., D = U). The discrete nature of this distribution stems from the inherent discrete number of neurons / filters to be selected.

[0090] Associating each client device with a value p can include associating each client device with a value from a distribution of discrete values. The distribution of discrete values can be a uniform distribution. Alternatively, the distribution of discrete values can be a non-uniform distribution that reflects the capabilities of the plurality of client devices. Thus, the distribution followed by p can be different to uniform to reflect data distribution specific to each client (e.g. a flagship device can have significantly more data and thus can require more network capacity and finer-grained sub-models). For ease of illustration, the following assumes that P is a uniform discrete distribution.

[0091] The number of discrete values in the discrete value distribution |P| can depend on: the number of different device tiers in the natural environment, i.e. the number of device capability levels; the size (width) of the network; and the number of device capability levels included in the ML model.

[0092] This p value can be the same for all layers of a given network, or can be configured per layer L, which leads to p L ~ D L P We will continue to use a single p over all layers for ease of understanding without loss of generality of the approach.

[0093] If a given layer L has K L width (i.e. neurons for linear layers, filters for convolutional layers), the sub-model for a given layer p is defined as a mask form where all neurons / filters 0:ceil((p)*K L ) are included and ceil((p)*K L ):K L are pruned. Unnecessary connections between pruned neurons are also removed.

[0094] This leads to a gain in terms of total FLOP (forward and backward propagation) and parameters. The latter also translates to optimizer (e.g. momentum vector) and gradient memory gain.

[0095] For each linear and convolutional layer, the number of weight parameters and FLOP is reduced by approximately p^2, with a slight difference in the accuracy formula due to the ceil operation, i.e. ceil(p*K1)*ceil(p*K2) / (K1*K2), where K1 and K2 correspond to the number of input and output neurons / channels, respectively. For the bias term, the reduction factor is p, more precisely (p*K2) / K2.

[0096] For normalization layers, the reduction in FLOP and number of parameters is the same as for the bias in linear / convolutional layers. This also applies to all activation and pooling layers in terms of FLOP.

[0097] Use weight w p The subnetwork F represents the p-pruned subnetwork. p Here, F and w represent the original (global model) network and weights, respectively. Advantageously, the network F can be directly obtained after training without requiring fine-tuning. p This eliminates the need to access any tagged data.

[0098] Therefore, this technique provides a pruning-aware training mechanism to train a DNN with this ordered pruning capability. To achieve this, this technique:

[0099] Choose a global model architecture and initialize it randomly or via a pre-trained network.

[0100] Create a uniform distribution U with |P| discrete values. P See the previous slide. Each p-value is transformed into a sub-model of the global model with fewer FLOPS and parameters.

[0101] The devices participating in the training are clustered into N levels, and the performance of the model on each device within that cluster is correlated with the p-value of that cluster. max Value. This p max The value represents the maximum network capacity that devices in that cluster can handle without sacrificing anything (defined by the Service Level Objective (SLO)).

[0102] This correlation can be achieved through analysis (via network and hardware FLOP throughput) or through an overview (i.e., time-based execution on the device). No data access is required for this step.

[0103] Step S420, which generates a sub-model of the ML model, may include performing ordered pruning of each neuron network layer of the ML model using a value p associated with each client device.

[0104] Alternatively, step S420 of generating a sub-model of the ML model may include performing ordered pruning of each neuron layer of the ML model using a value p associated with each client device and performing ordered pruning of at least one more neuron layer of the ML model using at least one more value.

[0105] The steps for S410, S420, and S430 will be discussed later. Figure 4B The details are described in the text.

[0106] Figure 4BThis is a flowchart illustrating example steps performed by a server according to an exemplary embodiment of this disclosure for training an ML model using joint learning and ordered dropout. Generally speaking, the ordered dropout method is parameterized for the following values: i) the dropout rate value p ∈ (0, 1] at each layer, ii) a set of candidate values ​​P such that p ∈ P, and iii) a sampling method for P on the candidate value set such that P ~ D p D p It is a distribution on P.

[0107] A key hyperparameter of the dropout distribution (OD) is the dropout rate P, which defines how much of each layer should be included, with the remaining units dropped in a structured and ordered manner. This value P is derived from the dropout distribution D. p Selected for sampling, D p It consists of a set of discrete values ​​P = {s1, s2, ..., s} |P|}(satisfying 0 < s1 < ... < s |P| ≤1) and probability (satisfy ) represents a uniform distribution on P. For example, a uniform distribution on P is represented by ) (Right now In the experiment explained below, the set Uniform distribution on (called) (or uniform-k). The discreteness of the distribution stems from the inherent discreteness of the number of neurons or filters that need to be selected.

[0108] The method includes determining the capabilities of each of a plurality of client devices that will be used to train the ML model (step S100). For example, this may include determining the computational or processing power and memory capacity of each client device. The hardware capabilities of each client device can be obtained from the client device (e.g., by requesting information), in which case the transient computational load of the device can also be used to determine the computational or processing power and memory capacity of each client device.

[0109] In one embodiment, the hardware capabilities of each client device can be measured through test operations across the entire ML model. When a new device participates in federated learning, the entire ML model is sent to that new device and executed. Through the execution of the ML model, the hardware capabilities of the new client device can be measured and obtained by a central computing device. The central computing device can determine the p-value of the new client device based on its obtained hardware capabilities. In one embodiment, the central computing device can determine the category of the new client device based on its obtained hardware capabilities. The p-value of the new client device can then be determined based on its category.

[0110] The method may include associating each client device with a value (dropout rate) P based on a determined capability, where P defines how many of each neuron network layer of the ML model is included in the target sub-model executed on the client device (step S102), with the remaining neurons / channels dropped in an ordered manner. Client devices may be grouped into groups or sets based on having similar capabilities. For example, client devices with limited processing and / or memory capabilities may be grouped together (e.g., low-level smartphones and smartwatches), while client devices with good processing and / or memory capabilities may be grouped together (e.g., high-level smartphones and tablets). This allows sub-models to be sent to the client device group based on the hardware capabilities of the devices within that group, i.e., those with the same P. (c) max The equipment group.

[0111] The method includes generating multiple sub-models of an ML model by using values ​​associated with each client device to perform ordered pruning of at least one neuron network layer of the ML model (step S104).

[0112] Ordered pruning, also known as ordered discarding, operates as follows: From a given distribution of discrete candidate values ​​p (e.g., discrete values ​​P = [s1, s2, ..., s...]), ... |P| ], 0 <s1<s2<...<s |P| In a uniform distribution (<=1), the sampling discard rate p is expressed as p ~ U P (i.e., D = U). The discreteness of this distribution comes from the inherently discrete number of neurons / filters that need to be selected.

[0113] Associating each client device with a value p can include associating each client device with a value from a discrete value distribution. The discrete value distribution can be uniform. Alternatively, the discrete value distribution can be a non-uniform distribution reflecting the capabilities of multiple client devices. Therefore, the distribution following p can be dissimilar to uniform to reflect the specific data distribution of each client (e.g., a flagship device may have significantly more data and therefore may require more network capacity and finer-grained sub-models). For ease of interpretation, the following assumption is that P is a uniform discrete distribution.

[0114] The number of discrete values ​​|P| in the discrete value distribution may depend on: the number of different device layers in the natural environment, i.e. the number of device capability levels; the size (width) of the network; and the number of device capability levels included in the ML model.

[0115] For all layers of a given network, this p value can be the same or configured for each layer L, thus resulting in p L ~D L PWe will continue to use a single p on all layers for ease of understanding without loss of generality of the approach.

[0116] If a given layer L has K L neurons (i.e. linear layers) or filters (i.e. convolutional layers), the sub-model of a given layer p is defined as a masked version, containing all neurons / filters 0:ceil((p)*K L ) are included and ceil(p)*K L :K L are pruned. Unnecessary connections between pruned neurons are also removed.

[0117] This results in a gain in both total FLOP (forward and backward propagation) and parameters. The latter also translates into a gain in optimizer (e.g. momentum vector) and gradient memory.

[0118] For each linear and convolutional layer, the number of weight parameters and FLOP is reduced by roughly p^2, with a slight difference in the accuracy formula due to the ceil operation, i.e. ceil(p*K1)*ceil(p*K2) / (K1*K2), where K1 and K2 correspond to the number of input and output neurons / channels, respectively. For the bias term, the reduction is p, more precisely (p*K2) / K2.

[0119] For normalization layers, the reduction in FLOP and number of parameters is the same as for the bias of linear / convolutional layers. The same applies to all activation and pooling layers in terms of FLOP and number of parameters.

[0120] Let w p denote the weights of the pruned subnetwork F p , where F and w are the original (global model) network and weights, respectively. Beneficially, without the need for fine-tuning, the network F p can be obtained directly after training, thus eliminating the need to access any labeled data.

[0121] The method comprises, during each federated learning training round, assigning a sub-model of the ML model to each client device based on a value p associated with each client device (step S106).

[0122] The present technology thus provides a pruning-aware training mechanism in order to train a DNN with this ordered pruning capability. To achieve this, the present technology:

[0123] A global model architecture is chosen and initialized randomly or via a pre-trained network.

[0124] A uniform distribution U P, see previous slide. Each p-value is translated into a sub-model of the global model with less FLOPS and parameters.

[0125] The devices participating in the training are clustered into N levels and p max values are associated to each cluster based on the performance of the model on the devices of that cluster. max This p max value represents the maximum capacity of the network that the devices in that cluster can handle without sacrificing (defined by the Service Level Objective (SLO)).

[0126] This association can be achieved through profiling (through network and hardware FLOP throughput) or via outlining (i.e. time execution on the device). No access to data is required for this step.

[0127] When ready to start the federated learning training mechanism, the global model architecture F is initialized with weights w0. According to the assignment D P The dropout rate space P is selected, each P corresponding to a sub-model of the global model with different FLOPS and parameters. The participating devices are clustered into C levels and p C max values are associated to each cluster; p (C) max represents the maximum capacity network that the devices in cluster C can handle without violating latency or memory constraints.

[0128] At the beginning of each communication round t, the number of participating devices S t and the number of local iterations are selected. The pool of available devices S t is sampled until the required number of devices to participate is obtained; the required number can be all available clients A t or a subset of A t , depending on the capacity of the server. When only a subset of A t is required, it is possible to over-subscribe the client devices with respect to the number of devices required to train the model; i.e. more client devices are selected than required to train the model to allow for failures to occur.

[0129] The master node (coordinator) (which can be the server) broadcasts the p C max defined sub-models through each participating device i's respective cluster or group, each client device i receives w p i max This will result in saving bandwidth as p max <=1.

[0130] At each local round t, device i samples p(I, k) ~ U P , st. p e (0, pC max ) and update the corresponding weights w of the local sub-models p (i, k); the weights can be updated with a suitable rule such as FedAvg. Each device i runs E local iterations (rounds) k and at the end of the local round each device sends back the gradients of the maximum updated sub-model (due to max(p i，k )<=p C max <=1, upstream saving in bandwidth.

[0131] The server aggregates these changes across devices and aggregates and updates the global model that will be distributed in the next global federated learning round, in which a different set of devices will be used. Appropriate aggregation techniques can be employed. Since changes with different p values are sampled, the ordered importance of features is maintained throughout the training process.

[0132] The variety of devices results in a variety of model updates. This is best computed in the global aggregation step using an aggregation rule

[0133]

[0134] where is the weight belonging to but not to w t+1 is the global weight w (i ，t，E) after E local iterations, on client i in communication round t, is the set of clients with the ability to update w

[0135] A typical procedure in FL is to perform element-wise averaging to aggregate model updates from clients. However, due to the permutation invariance of the hidden layers, coordinate-wise averaging of the updates can have a detrimental effect on the accuracy of the global model. Recent techniques address this issue by matching the "neurons" of the clients before averaging. Unfortunately, this is computationally expensive and impairs scalability. In contrast, the present approach alleviates this problem as it exhibits the natural importance of neurons / channels of each hidden layer by design. In this way, ordered dropout can replace the neuron matching algorithm without the computational overhead.

[0136] The step S104 of generating a plurality of nested sub-models of the ML model can comprise performing an ordered pruning of each neural network layer of the ML model using the value p associated with each client device.

[0137] Optionally, the step S104 of generating a plurality of nested sub-models of the ML model can comprise performing an ordered pruning of one neural network layer of the ML model using a value p associated with each client device and performing an ordered pruning of at least one further neural network layer of the ML model using at least one further value.

[0138] The step S106 of assigning to each client device a sub-model of the ML model during each federated learning training round t can comprise broadcasting the sub-model generated using one value to each client device associated with that value for local training by the client device.

[0139] Figure 4A And Figure 4B The method illustrated can further comprise receiving from each client device during each federated learning training round a gradient of the maximum updated sub-model, wherein each gradient corresponds to a change in a weight of the sub-model. The method can comprise receiving the weights directly.

[0140] Figure 4A And Figure 4B The method illustrated can further comprise aggregating the changes in the weights of the sub-model received from each client device using the received gradients and updating the ML model.

[0141] Figure 4A And Figure 4B The method illustrated can comprise repeating the steps of generating and assigning using the updated ML model.

[0142] No high-end device training. Imagine a model that is too heavy for devices in the natural environment or you do not have enough high-end devices or even you reserve this model for the next generation of high-end devices in the future. This means that under the current deployment, the entire (global) model (p = 1) will be rarely or never updated. Therefore, the global model is actually larger than the most trained model.

[0143] One way to solve this problem is to use federated dropout at the client side so that the p max is higher for this cluster of devices. This leads to more devices updating a larger part of the network. Optionally, before assigning the sub-model to the client devices, federated dropout can be used on the server side so that the p = 1 model is correspondingly smaller due to randomly dropped neurons / filters, but still preserves a higher level of neurons / filters that would always be dropped orderedly. Therefore, more devices can help to update a larger part of the network.

[0144] With federated dropout, we randomly and uniformly sample neurons (FC layers) and filters (CONV layers) to exploit random dropout to counteract a higher structured budget (p). Formally, previously: p ~ UP , s.t. p <= p' max Now: d ~ U P , s.t. p < p' max where p' is the maximum submodel size the device can handle, p is the sampled value, satisfying p <= p' max . The loss function is defined as follows: max , 1) and defines a trade-off between joint dropping and ordered dropping. When p e (p max , p' max ), the joint dropping rate is defined as d = p / p' max , otherwise, d = 1.

[0145] For clarity, d and p are different in the way they translate to dropping in the network. Joint dropping randomly keeps ceil(d*K) neurons / filters within a layer. Ordered dropping drops the neurons / filters with indices ceil((p)*K) to K, where K is the width of the current layer.

[0146] Another solution to this problem is to use quantization on the client side, so that the p max of the device cluster is higher. This results in more devices updating a larger portion of the network.

[0147] Optionally, the quantization variables (weights + activations + gradients) of the submodels are used in order to increase p max to higher values. For this, the present technology supports multiple quantization techniques (e.g. linear quantization with random rounding).

[0148] While the computational gain of quantization is hardware dependent, the additional advantage of quantized models is that they can be run on different processors (such as GPU (float16), NPU (int8) and DSP (int8)).

[0149] Increase the accuracy of small submodels. For small p values, you might aggregate updates from all clients, but the final accuracy can still be too low to be practically useful.

[0150] To solve this problem, we use the structure of ordered dropping of the present technology, which explicitly enables knowledge distillation between submodels (p) and supermodels (p max ), where p max is the maximum submodel size the device can handle, p is the sampled value, satisfying p <= p' max . The loss function is defined as follows:

[0151] L client (softmax p , softmaxp max , y label) = (1 - a) * CE(max(softmax p ), y label ) + a * KL(softmax p , softmax p , T)

[0152] Here, softmax p is the softmax output of the sampled sub-model, y label is the groundtruth label, CE is the cross-entropy function, KL is the KL-divergence function, a is the relative weight of the two losses, and T is the distillation temperature.

[0153] Note that this technique uses some additional computation to improve accuracy (the overhead of one additional forward pass of the sub-model p max ). Since we are still sampling to p max , the devices are still able to participate. This computation is expected to have more impact during the first few rounds of the global training process, where the smallest sub-model is not yet a strong feature extractor.

[0154] Number of values in the distribution. The number of discrete values k that we can sample from depends on:

[0155] The number of different device levels |D tiers | in the natural environment.

[0156] For example, we might want to have three model variants that cater to high-end, mid-tier, and low-end devices. This should be defined by the service operator based on their needs and the devices that their service is deployed on.

[0157] Since we want to consider extensibility (next generation devices), we recommend k >= 2 * |D tiers |.

[0158] The number of different device load levels in our model.

[0159] Similarly, but at runtime, we can also consider the online load that the device has, which defines the p max that the device can temporarily handle. Thus, we can have, for example, the quartiles (25%, 50%, 75%, 100%) of the load associated with different p values. The load levels can represent different resource types or a combination of these resources (e.g., computation, memory, energy, etc.).

[0160] The size (width) of the network. The number of layers 1, in the DNN. l Channels / neurons beyond this cannot be distinguished. Thus, k < max {|P|1，max}.

[0161] Performance evaluation. In Tables 1(a) and 1(b) below, the joint learning with ordered dropout (also referred to as FjORD) of the present technology is evaluated with respect to two vision and one text prediction tasks.

[0162] Table 1(a):

[0163]

[0164] (a) Dataset description

[0165] Table 1(b)

[0166]

[0167] (b) MAC and parameters of the network for each reduction p

[0168] For CIFAR 10 (Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009), the“CIFAR” version of ResNet 18 (Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.) was used. The dataset was federated by randomly splitting the dataset into equal sized parts (each part assigned to a particular client) and thus effectively maintaining IID. For FEMNIST, a CNN with two convolutional layers followed by a softmax layer was used. For Shakespeare, an RNN with an embedding layer followed by two LSTMs (Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997) and a softmax layer (without dropout) was employed. The performance of the model on the last epoch provided on the test set was constructed by this each client’s test data. The top ranked accuracy for the visual task and negative perplexity for the text prediction.

[0169] FjORD at the top of Flower (v0.14dev) (Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi, Titouan Parcollet, Nicholas D Lane). Flower: A Friendly Federated Learning Research Framework. arXiv preprint arXiv:2007.14390, 2020) framework and PyTorch (v1.4.0) (Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward, Zachary DeVito, Martin Raison, Paszke Tejani, Sasank Chilamkurthy, Benoit, Benoit Steiner, Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8026-8037, 2019).

[0170] All experiments were conducted on a private cloud farm consisting of Nvidia V100 GPUs. To scale to hundreds of clients on a single machine, Flower was optimized so that clients only allocate GPU resources when actively participating in federated client rounds. The average performance and standard deviation of three runs of all experiments are reported. To simulate client availability, up to 100 Flower clients ran in parallel, each sampling 10% of the global round, with the ability to switch identities at the start of each round to over-provision a larger federated dataset. Additionally, client heterogeneity was simulated by assigning each client to a device cluster. The following settings were provided:

[0171] Uniform-{5, 10}: This refers to the distribution, i.e., p ~ k , k = 5 or 10.

[0172] Drop Scale ∈ {0.5, 1.0}: This parameter affects the possible deviation in the number of devices in each cluster. It refers to the number of drops per client when a device in each cluster enters a higher P. Formally, for uniform-n and drop scale ds, the high-end cluster n contains The devices in the cluster each contain a DSN. Therefore, for uniform-5 with ds=1.0, all devices can run a subnet with p=0.2, 80% of the devices can run with p=0.4, and so on, resulting in a device allocation of (0.2, ..., 0.2). For ds=0.5, this percentage is halved, resulting in a larger high-end cluster, such as (0.1, 0.1, ..., 0.6).

[0173] Baselines. To evaluate performance against existing techniques, the FjORD algorithm is compared to the following baselines: i) Extended Joint Dropout (eFD), ii) Comparison with eFD (w / eFD). eFD builds upon the Joint Dropout (FD) technique[?], employing random dropout (RD) at the neuron / filter level to minimize the model's footprint. However, FD does not support adaptability to uncertain heterogeneous client capabilities because it inherits a single dropout rate across devices. To address this, an extension to FD is proposed that allows the dropout rate to be applied to the device capabilities defined by the respective cluster members. Clearly, eFD outperforms FD in performance and provides a more stringent baseline, as the latter requires applying the same dropout rate to accommodate the atheand model across all devices, resulting in a larger dropout rate (i.e., an average dropout rate of 80% across the entire model to support low-end devices).

[0174] To evaluate the performance of Fjord, it was compared with two baselines, eFD and OD+eFD. A uniform-5 setting was considered with a dropout scale of 1.0 (i.e., uniform clustering). For each baseline, for each p, an independent model F was trained end-to-end. p For eFD, this means that model F cannot be run. p The device cluster is compensated for by randomly dropping neurons / filters. It should be noted that p=0.2 is omitted from the eFD results because it essentially doesn't use any dropping anyway. For the case of FjORD+eFD, RD is controlled by capping it at d=0.25. This allows larger sub-models to be updated more frequently, since devices belonging to cluster c during training can now have more data. c+1 is the next, more robust cluster, which also prevents the destructive effects of excessively high drop values ​​shown in the eFD baseline.

[0175] Figure 5Experimental results are shown for implementing federated learning using ordered dropout according to demonstrative embodiments of the present disclosure. In Figure 5 In the middle row, top-1 validation accuracy is shown for CIFAR-10 / Resnet18, and in the bottom row, top-1 validation accuracy is shown for CIFAR-100 / Resnet28_8. The plots show the relationship between top-1 validation accuracy and network footprint (neurons, parameters, MAC) in a classification task. Initial data is based on models initialized from random and trained with ordered dropout; pre-trained data is based on models initialized from pre-trained and trained with ordered dropout; pre-trained data with knowledge distillation (KD) is based on smaller sub-models trained with ordered dropout using knowledge distillation; and full model data shows ordered dropout sub-models trained end-to-end without ordered dropout. Preliminary experiments show the required performance of a classification task using ResNet-based networks in a non-federated learning setting, here considering only one node. The present techniques result in: ordered representations of learned models; comparable or even better performance compared to models trained from scratch; and even better results than models trained from scratch.

[0176] Figure 6A , Figure 6B and Figure 6C are results of experiments using ResNet18-CIFAR10, CNN-FEMNIST, and RNN-shakespeare datasets according to demonstrative embodiments of the present disclosure. A uniform distribution p ~ U k with 5 discrete values (i.e., k = 5) was applied. Ordered dropout models with knowledge distillation (labeled FjORD w / KD in the figures) were compared to extended federated dropout “eFD” and FjORD with eFD. Extended federated dropout is essentially a process of providing p-pruned sub-models by random dropout instead of ordered dropout, i.e., while federated dropout “FD” does not support the heterogeneity of clients due to its single dropout rate, eFD allows the dropout rate to adapt to device capabilities (based on p). FjORD + eFD corresponds to the case listed above, i.e., federated dropout is applied to the server-side model in order to update a larger portion of the model from low-tier devices.

[0177] It can be seen that FjORD w / KD outperforms eFD on all datasets with improvements ranging from 1.53-34.87 percentage points (pp) on CIFAR10 (average 19.22 pp), 1.57-6.27 pp on FEMNIST (average 3.41 pp) and 0.01-0.82 pp (average 0.46 pp) on Shakespeare. Compared to FjORD+eFD, the performance gain for FjORD on FENMIST is 0.71-2.66 pp (average 1.79), up to 2.56 pp (average 1.35 pp) and 0.12-0.22 pp (average 0.18 pp) on Shakespeare. In all tasks, FjORD is able to improve its performance with increased p due to the nested structure of the ordered dropping method of FjORD. We also conclude that eFD on top of FjORD does not seem to bring better results. Interestingly, considering the heterogeneous pool of devices, to obtain the highest performing model of eFD, multiple models have to be trained (i.e. one model per device cluster). For example, for CIFAR10, FEMNIST and Shakespeare, the highest performing model of eFD is F0.4, F0.6 and F0.4 respectively, such results are only obtained a posteriori, i.e. after all model variables have been trained. However, despite the device heterogeneity, FjORD requires a single training procedure leading to a global model that significantly outperforms the best eFD model (2.98 and 2.73 pp for CIFAR10 and FEMNIST respectively, 0.13 pp for Shakespeare) while allowing direct and seamless extraction of sub-models due to the nested structure of OD.

[0178] Figure 7A 、 Figure 7B and Figure 7C The effect of FjORD using knowledge distillation (KD) in a federated learning setting is shown according to an exemplary embodiment of the present disclosure. When p > 0.4, KD of FjORD consistently improves performance on all three datasets with an average gain of 0.18; 0.68 and 0.87 pp for sub-models of size 0.6; 0.8 and 1 for CIFAR-10; 2.39 and 2.65 for FEMNIST and 0.10 pp for Shakespeare. For the case where p 0.4, the effect of KD is gradually diminishing, which can be due to the average precision of the sub-models being optimized. However, it can be easily seen that KD significantly improves the performance of the global model, yielding a gain of 0.71 and 2.63 pp for CIFAR10 and FEMNIST respectively and 0.10 pp for Shakespeare.

[0179] One important feature of the technology is that the training technique scales to larger clusters of devices, or performs better with higher granularity of p-values. To illustrate this, the performance of OD is tested on both uniform-5 and uniform-10 settings.

[0180] Figure 8A and Figure 8B It is shown that ordered dropout scales to larger clusters of devices; in other words, it improves performance with higher granularity of p-values. Figure 8A and Figure 8B The uniform-5 and uniform-10 settings, i.e., p ~ U5 and p ~ U 10 It can be seen that ordered dropout (FjORD) still maintains performance even with higher granularity of p-values. Therefore, ordered dropout still has great value for applications that require finer-grained modeling of clients, without any significant drop in the accuracy of each sub-model. This further supports the use cases that require explicit modeling of device loads in a cluster of devices (e.g., modeling device capabilities and loads in decile values).

[0181] Figure 9A and Figure 9B It is shown that ordered dropout is resilient to the distribution of devices in each cluster according to an exemplary embodiment of the present disclosure. Here, the uniform-5 setting is used in conjunction with the dropout scale ds e {0.5, 1.0}, which is a parameter that affects the deviation of the number of devices that can appear in each cluster. In this way, the high-end cluster n contains devices, so that for ds = 1.0, all devices can run a network of p = 0.2, 80% of the devices can run a network of p = 0.4, and so on, i.e., resulting in a distribution of devices (0.2, …, 0.2). For ds = 0.5, the dropout percentage is halved, resulting in a larger high-end cluster; e.g., (0.1, …, 0.6). In other words, in both cases, the clients can support models of p i max e {0.2, …, 0.8}, but in the case of ds = 0.5, the highest cluster occupies 60% of the total number of devices. This reflects the case where most of the participating devices are able to run the entire original model.

[0182] Figure 9A and Figure 9BThe results show that larger sub-models are expected to be more accurate and updated more frequently. However, the same plots also show that FjORD does not significantly reduce the accuracy of smaller sub-models (i.e. ds=0.5) in the presence of higher level devices. This is a direct result of sampling p values during the local rounds, rather than simply tying each level to the largest sub-model it can handle. It should also be noted that the uniform sampling does not change in the presence of more high-end devices, accurately illustrating the adaptability of the ordered dropping to potential user device distribution changes that the server can not be aware of.

[0183] Example embodiments. Figure 10 An example use of the present technology for training ML models is shown in accordance with exemplary embodiments of the disclosure. The present technology can enable activity recognition and health care applications such as sleep tracking without the vendor of the ML model having access to specific sensor data that can be collected from a wearable device (e.g. a smartwatch or fitness tracker) or a smartphone. The present technology advantageously means that user-specific data does not need to be sent to the vendor of the ML model or the owner of any app using the ML model. The local training is done on the device, as explained above, with the gradients aggregated between clients to update the global model.

[0184] Figure 11 Another example use of the present technology for training ML models is shown in accordance with exemplary embodiments of the disclosure. The present technology can enable tasks involving environment recognition such as augmented reality (AR) and robotic assistants. AR devices can track and interact with multiple rooms regardless of where the user is using or wearing them. Robotic assistants and robotic vacuum cleaners can move around the room and interact with their environment. Figure 11 The trajectory light trails of a robotic vacuum cleaner are shown, illustrating the extent of the interaction and movement of the device within the room. However, due to their semi-mobile nature, the computational capabilities of these devices are typically limited and cannot be increased. For example, in the case of AR devices, additional computational resources would increase the weight of the device, making it difficult to wear and use. Similarly, battery life is important for robotic devices, so increasing computational resources to implement or train ML models would decrease battery life. Furthermore, as these devices interact with the user’s home and environment, it is important that any data obtained by these devices remains private.

[0185] Another example use of the present technology is to improve handwriting recognition. Many Samsung products come with an S-Pen. The present technology can be used to improve the ability to recognize handwriting (e.g., what they wrote with the S-Pen) without directly accessing user data and with less training overhead. This in turn can be used to implement a better handwriting recognition global model, which can be implemented across multiple devices. The global model can be further personalized using meta-learning techniques that enable the global model to perform better on the handwriting of a particular user.

[0186] Figure 12 is an example system 100 for training an ML model using federated learning in accordance with demonstrative embodiments of the present disclosure. The system 100 includes a server 102 and a plurality of client devices or apparatuses 110. For simplicity, a single apparatus 110 is shown here.

[0187] The server 102 includes at least one processor 104 coupled with a memory 106. The at least one processor 104 can include one or more of a microprocessor, a microcontroller, and an integrated circuit. The memory 106 can include a volatile memory, such as a Random Access Memory (RAM) for temporary storage of data, and / or a non-volatile memory, such as a flash memory, a read-only memory (ROM), or an Electrically Erasable Programmable ROM (EEPROM) for storing data, programs, or instructions.

[0188] The server 102 stores a global ML model 108 that is to be trained using federated learning and the plurality of apparatuses 110.

[0189] The at least one processor 104 coupled to the memory 106 can be arranged to determine a capability of each client device 110, associate each client device with a value based on the determined capability, the value defining how many of each neural network layer in the ML model 108 is to be included in a target model implemented by the client device 110, generate a plurality of nested sub-models 109 of the ML model 108 to perform an ordered pruning of at least one neural network layer in the ML model by using the value associated with each client device, and assign to each client device a sub-model 109 of the ML model during each federated learning training round based on the value associated with each client device.

[0190] The apparatus 110 can be any one of a smartphone, a tablet, a notebook, a computer or computing device, a virtual assistant device, a vehicle, a drone, an autonomous vehicle, a robot or robotic device, a robotic assistant, an image capturing system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, a smart consumer device, a smart watch, a fitness tracker, and a wearable device. It can be appreciated that this is a non-exhaustive, non-limiting list of example apparatuses.

[0191] Each client device or apparatus 110 can comprise at least one processor 112 coupled to a memory 114. The at least one processor 112 can comprise one or more of a microprocessor, a microcontroller and an integrated circuit. The memory 114 can comprise volatile memory, such as random access memory (RAM) for temporary storage of data, and / or non-volatile memory, such as flash memory, read only memory (ROM) or electrically erasable programmable ROM (EEPROM) for storing, for example, data, programs or instructions.

[0192] Each apparatus 110 receives from the server 102 a (nested) sub-model 109 for local training based on the hardware capabilities of the apparatus, i.e. the storage capacity of the memory 114 and the processing power of the processor 112. The apparatus comprises a storage for private training data 122 collected or acquired by the apparatus 110. The private training data 122 can be health data (e.g. heart rate, step count, etc.), images or videos, handwritten documents, etc. At the time of training, the processor 112 can be configured to determine the current computing power of the apparatus 110 and to extract from the received sub-model a smaller nested sub-model for training using the stored private training data. That is, if the computing power of the apparatus 110 is currently used to perform one or more other tasks (e.g. capturing images, participating in a call, etc.), less resources are available for training the sub-model. In this case, since the received sub-model 109 is sent to the apparatus 110 based on its maximum computing power, the apparatus 110 can only train the received sub-model at this particular time.

[0193] The apparatus 110 can further comprise at least one image capturing device 116 for capturing images or videos to be used as private training data. Additionally or alternatively, the apparatus can further comprise at least one interface 118 for collecting data forming the private training data. For example, the interface 118 can be a communication interface receiving (via a wired or wireless connection) data collected by another device. For example, the apparatus 110 can be a smartphone which receives heart rate data from a wearable fitness tracker via the interface 118.

[0194] Figure 13is a flowchart of example steps performed by a device to train an ML model using federated learning according to the demonstrative embodiments of the present disclosure. The storage device 120 stores private training data collected by the device 110 for locally training an ML model. The at least one processor 112 coupled to the memory 114 can be configured to receive, from the server 102, sub-models 109 of the ML model 108 generated based on the capabilities of the device (step S200). Optionally, as described above, the processor 112 can be configured to determine the current computing capability of the device 110; and extract, from the received sub-models, a smaller nested sub-model for training using the stored private training data (step 202). The processor 112 can train the sub-model (i.e. the full received sub-model or one of the nested sub-models therein) using the stored private training data 122 and update the weights of the sub-model (step 204); and send, to the server 102, the gradients of the maximum updated sub-model, where each gradient corresponds to a change in the weights of the sub-model 109 (step S206).

[0195] In summary, the present technology provides a federated learning method for heterogeneous device training. To this end, FjORD builds on the ordered dropout technique as a mechanism to extract smaller footprint sub-models from the master model in a way that the training partial models also participate in training the whole model. It is shown that ordered dropout outperforms the performance of competing techniques in both local and federated settings while maintaining flexibility across different environmental settings. Advantageously, the present method can enable more types of client devices to participate in federated learning and compensate for the potential drawbacks of training with custom sub-models by enabling more client devices and more diverse training data to participate in the training process. In a typical federated learning process, client devices that do not provide a locally trained model to the server within a baseline time period (e.g. 30 seconds) are typically automatically excluded from the training process. In contrast, in the present technology, the use of custom sub-models means that all devices will be able to send the results of local training back to the server within the baseline time period.

[0196] Those skilled in the art will appreciate that, while the foregoing description sets forth what are believed to be the preferred embodiments of the present technology, other and various embodiments of the present technology can be devised without departing from the spirit or the principles of the present technology. The present technology is intended to cover adaptations or variations of the preferred embodiments discussed herein. Therefore, it is manifestly intended that the present technology be limited only by the claims hereinafter set forth.

Claims

1. A method of training a machine learning (ML) model by utilizing federated learning performed by a plurality of client devices, the method comprising: determining a computing power of each client device; associating each client device with a value based on the determined computing power, the value defining how many of each neural network layer of the ML model is to be included in a target submodel to be trained by the each client device; generating a plurality of nested submodels of the ML model by using the value associated with the each client device, wherein the ML model comprises all submodels and a larger submodel comprises all smaller submodels to perform pruning of at least one neuron of at least one neural network layer of the ML model according to a predefined order such that pruned neurons are not used by a submodel; and during each federated learning training round, assigning a submodel of the ML model to the each client device based on the value associated with the each client device.

2. The method of claim 1, wherein, associating each client device with a value comprises associating the each client device with a value according to a distribution of discrete values.

3. The method of claim 2, wherein, The distribution of discrete values is a uniform distribution.

4. The method of claim 2, wherein, The distribution of discrete values is a non-uniform distribution that reflects a distribution of capabilities of the plurality of client devices.

5. The method of claim 2, wherein, A number of discrete values in the distribution of discrete values depends on any one or more of: a number of device computing power levels, a size or width of a neural network layer, and a number of device computing power levels contained in the ML model.

6. The method of claim 1, wherein, Generating a plurality of nested submodels of the ML model comprises performing an ordered pruning of each neural network layer of the ML model using the value associated with the each client device.

7. The method of claim 1, wherein, Generating a plurality of nested submodels of the ML model comprises performing an ordered pruning of one neural network layer of the ML model using the value associated with the each client device and using at least one more value to perform an ordered pruning of at least one more neural network layer of the ML model.

8. The method of claim 1, wherein, During each federated learning training round, assigning the submodel of the ML model to the each client device comprises broadcasting a submodel generated using the value to each client device associated with the value for local training by the each client device.

9. The method of claim 1, further comprising: during each federated learning training round, receiving a gradient of a maximum supported updated submodel from the each client device, wherein each gradient corresponds to a change in a weight of the submodel.

10. The method of claim 9, further comprising: using the received gradients, aggregating the changes in weights of the submodels received from the each client device; and updating the ML model.

11. The method of claim 10, further comprising: repeating the generating and assigning steps using the updated ML model.

12. A non-transitory data carrier carrying code which, when executed on a processor, causes the processor to perform the method of claim 1.

13. A system of training a machine learning (ML) model utilizing federated learning performed by a plurality of client devices, the system comprising: a plurality of client devices; and A central computing device comprising at least one processor coupled to a memory and configured to: determine a computing power of each client device; associate each client device with a value defining how many of each network of neurons in the ML model is to be included in a target sub-model to be implemented by the each client device based on the determined computing power; generate a plurality of nested sub-models of the ML model by using the values associated with the each client device, wherein the ML model comprises all sub-models and larger sub-models comprise all smaller sub-models to perform pruning of at least one neuron of at least one network of neurons of the ML model according to a predefined order such that pruned neurons are not used by a sub-model; and assign a sub-model of the ML model to the each client device based on the value associated with the each client device during each federated learning training round.

Images