FEDERATE LEARNING WITH BASIC MODEL DISTILLATION

Abstract

Methods and systems for training neural networks with federated learning. Machine learning models are sent from a server to clients, resulting in local machine learning models. On each client, the models are trained using locally stored data, including determining a cross-entropy loss for each of the multiple local machine learning models. The weights for each local model are updated and transmitted to the server without transmitting the locally stored data. The transmitted weights are aggregated on the server to obtain an aggregated, server-managed machine learning model. A distillation loss based on a base model is generated on the server.The aggregated, server-managed machine learning is updated to obtain the respective aggregated weights, which are then passed to the clients for updating in their local models.

Description

STATE OF THE ART

[0001] The present disclosure relates to methods and systems of federated learning with base models in machine learning models. BACKGROUND

[0002] Federated learning (a type of collaborative learning) is a machine learning technique in which a machine learning algorithm is trained across multiple independent nodes, each using its own dataset. Federated learning aims to train a global machine learning algorithm, such as deep neural networks, on multiple local datasets contained within local nodes (also called clients), without explicitly sharing data samples. The learning task is solved by a federation of participating devices coordinated by a central server. Each participating device (client) has a local training dataset that is not uploaded to the server or shared with other clients. Instead, each client computes an update to the current global model maintained by the server.The clients transmit this update, but not the training dataset, to the server. The server aggregates the received updates to update the global model. The resulting joint model can be trained by learning from the clients' training, allowing users to benefit from joint models trained from the clients' data without having to transfer the clients' data or store it centrally. This is particularly useful in situations where the exchange of sensitive or personal data is prohibited (e.g., medical information, Internet of Things devices, personal devices such as smartphones). SHORT DESCRIPTION OF THE CHARACTERS Fig. 1 shows a system for training a neural network according to one embodiment. Fig. 2 shows a computer-implemented method for training and using a neural network, according to one embodiment. Fig. 3A illustrates an example representation or layout of a federated learning system including a base model according to one embodiment; Fig. 3B illustrates a flowchart of a first example method for training neural networks via client-side federated learning according to a first embodiment. Fig. 3C illustrates a flowchart of a second example method for training neural networks through server-side federated learning according to a second embodiment. Fig. 4 shows a schematic representation of a deep neural network with nodes in an input layer, a plurality of hidden layers, and an output layer, according to one embodiment. Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine and a control system, according to one embodiment. Fig. Figure 6 shows a schematic diagram of the control system of Fig. 5, which is configured to control a vehicle, which according to one embodiment may be a semi-autonomous vehicle, a fully autonomous vehicle, a semi-autonomous robot or a fully autonomous robot. Fig. Figure 7 shows a schematic diagram of the control system of Fig. 5, which is designed to control a manufacturing machine, such as a punching machine, a cutting machine or a drill, of a manufacturing system, such as a part of a production line. Fig. Figure 8 shows a schematic diagram of the control system of Fig. 5, which is designed to control a power tool, such as a drill or a drill driver, which has an at least partially autonomous mode. Fig. Figure 9 shows a schematic diagram of the control system of Fig. 5, which is configured to control an automated personal assistant. Fig. Figure 10 shows a schematic diagram of the control system of Fig. 5, which is designed to control a monitoring system, such as an access control system or a surveillance system. Fig. Figure 11 shows a schematic diagram of the control system of Fig. 5, which is designed to control an imaging system, e.g. an MRI device, an X-ray imaging device or an ultrasound device. DETAILED DESCRIPTION

[0003] Embodiments of the present disclosure are described herein. It should be understood, however, that the disclosed embodiments are merely examples, and further embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or reduced to show details of particular components. Specific details of structure and function disclosed herein are therefore not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously utilize the embodiments.Those of ordinary skill in the art will understand that various features illustrated and described with respect to any of the figures may be combined with features illustrated in one or more additional figures to create embodiments not explicitly illustrated or described. The illustrated feature combinations provide representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0004] The terms "A," "a," and "the" refer to both the singular and the plural, unless the context clearly indicates otherwise. For example, the term "a processor" programmed to perform different functions refers to one processor programmed to perform each function individually, or to multiple processors programmed together to perform each of the different functions.

[0005] The present disclosure relates to federated learning. References to a "server" of the federated learning system refer to the central server(s) that communicate with the clients to exchange data. A server may, for example, consist of one or more servers in the cloud or remote from the end users. References to a "client" of the federated learning system refer to local nodes or devices corresponding to a particular end user. For example, a smartphone, an Internet of Things device, or the like would constitute a client. Both the server and the client may use a corresponding machine learning computer system as described herein.

[0006] Federated learning (also known as collaborative learning) is a machine learning technique in which a machine learning algorithm is trained in multiple independent sessions, each using its own dataset. Federated learning aims to train a global machine learning model (e.g., deep neural networks) based on multiple local datasets contained in local nodes (also called clients), without explicitly sharing data samples. The learning task is solved by a federation of participating devices coordinated by a central server. Each participating device (client) has a local training dataset that is not uploaded to the server. Instead, each client computes an update to the current global model maintained by the server. The clients submit this update, but not the training dataset, to the server.The server aggregates the received updates to update the global model. The resulting joint model can be trained by learning from the clients' training, allowing users to benefit from joint models trained from the clients' data without having to transfer or centrally store the clients' data.

[0007] Federated learning is particularly useful in situations where the sharing of sensitive or personal data is precluded (e.g., medical information, Internet of Things devices, personal devices such as smartphones). A simple, everyday example is smartphones. When a person types an email or text message, they may have an autocorrect or autocomplete feature that automatically corrects spelling mistakes or suggests corrections or additions to the text. This type of system typically relies on a machine learning model that is trained based on the user's previous word usage, which can vary from user to user. Due to the sensitive nature of the information used, the raw data of the words entered by the user is not sent to a central server for model training.Rather, the user's smartphone can receive a base model from the server and train that model locally to perform autocorrection in a way tailored to the user's word usage. Information about how the model was trained locally can be sent to the server without needing to send the raw data (e.g., the user's words and / or corrections) to the server.

[0008] Federated learning is a technique that allows data to be stored locally on the clients' devices, while the server acts as an orchestrator for the learning process, aggregating the learned information and synchronizing it with the clients (e.g., users, AI tasks, etc.). However, the system can lose efficiency and robustness due to issues such as data heterogeneity, distribution shifts, unreliable communication, and even hardware failures. Furthermore, various federated learning architectures have been proposed, such as single global models and peer-to-peer with individual models, some of which include personalized versions.

[0009] FedAvg is a federated learning technique in which a distributed machine learning algorithm encourages clients to collaboratively train a global server model while keeping their data stored locally and private. The original model is sent from the server to the clients, the client updates are aggregated into a global model by averaging the client updates, and this process is repeated until convergence is achieved. FedAvg enables the global model to be updated by merging the knowledge and insights of multiple clients. A typical federated learning system suffers from issues such as reduced performance and robustness, especially when faced with problems such as data heterogeneity and distribution changes.Recently, base models pre-trained on a large dataset have become popular for initializing model parameters. In the scenario where clients have access to their own base models but have strong privacy concerns, the maximum achievable performance would be limited.

[0010] Considering the above challenges and the plethora of federated learning architectures and frameworks, the present disclosure aims to address these problems by proposing a novel federated learning framework that uses base models locally within clients to improve their performance, while using a smaller proxy model to coordinate and synchronize information across different clients. While others propose using base models within clients to enhance their customization capabilities, they suggest directly customizing the base model by adding customization layers or prompts. They do not use distilled knowledge of base models to guide the training of the smaller proxy model.

[0011] To address the problems of insufficient representation and computational efficiency, this disclosure proposes embodiments that incorporate the following techniques. Clients have the freedom to choose their own base models instead of using the same base model used in the literature. By consistently distilling the performance of base models into a smaller student model on the client side and updating the student model with aggregated weights from the server, our framework leverages the prior knowledge of pre-trained base models while preserving the information presented by the client population. Clients distill knowledge from their own selection of pre-trained base models, which are fine-tuned using local data, and have access to aggregated knowledge in the federated learning framework.In this way, our framework strikes a balance between efficient communication setup between server and clients, the privacy protection that comes from not directly revealing the base model, and a high level of performance and robustness that can be achieved with base models.

[0012] The federated learning system can integrate the machine learning training and the Fig. use the processes shown. Fig. Figure 1 illustrates a system 100 for training a neural network, e.g., a deep neural network. The neural network to be trained may be located on the server or the client. In other words, both the server and the client can learn from Fig. 1. The system 100 may include an input interface for accessing training data 102 for the neural network. As in Fig. 1, the input interface can be formed, for example, by a data storage interface 104 that can access the training data 102 from a data storage 106. The data storage interface 104 can be, for example, a memory interface or an interface for persistent storage, e.g., a hard disk or SSD interface, but also a personal, local, or wide area network interface such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage 106 can be an internal data storage of the system 100, such as a hard disk or SSD, but also an external data storage, e.g., a data storage accessible via a network.

[0013] In some embodiments, the data store 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 may access from the data store 106. However, it will be appreciated that the training data 102 and the data representation 108 of the untrained neural network may each be accessed from a different data store, e.g., via a different subsystem of the data store interface 104. Each subsystem may be of a type as described above for the data store interface 104. In other embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on design parameters for the neural network and therefore does not need to be explicitly stored on the data store 106.The system 100 may further comprise a processor subsystem 110 that may be configured to provide an iterative function as a replacement for a stack of layers of the neural network to be trained during operation of the system 100. The respective layers of the stack of layers being replaced may have common weights and receive as input an output of a previous layer or, for a first layer of the stack of layers, an initial activation and a portion of the input of the stack of layers. The processor subsystem 110 may further be configured to iteratively train the neural network using the training data 102. An iteration of the training by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion.The processor subsystem 110 may be configured to perform the forward propagation portion by determining, among other operations defining the forward propagation portion, an equilibrium point of the iterative function at which the iterative function converges to a fixed point, wherein determining the equilibrium point comprises using a numerical root-finding algorithm to find a root solution to the iterative function minus its input, and providing the equilibrium point as a substitute for an output of the stack of layers in the neural network. The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network, which data may also be referred to as trained model data 112. For example, as also shown in FIG. Fig. 1, are formed by the data storage interface 104, which in these embodiments is an input / output interface (“EA”) via which the trained model data 112 can be stored in the data storage 106. For example, the data representation 108 defining the “untrained” neural network can be at least partially replaced by the data representation 112 of the trained neural network during or after training by adjusting the parameters of the neural network, such as weights, hyperparameters, and other types of neural network parameters, to reflect the training on the training data 102. This is also illustrated in Fig. 1 by reference numerals 108, 112, which refer to the same data set on the data storage 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 that defines the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of a type described above for the data storage interface 104.

[0014] The structure of system 100 is an example of a system that can be used to train the models employed by the federated learning system described here. Additional structure for operating and training these machine learning models is described in Fig. 2 shown.

[0015] Fig. 2 illustrates a federated learning system 200 configured to execute and train the machine learning models described herein, e.g., the neural networks or deep neural networks. The system 200 may be implemented to perform the federated learning processes described herein. The system 200 may include at least one computing system 202. The computing system 202 may be part of or executed by a client device, such as a smartphone, an Internet of Things device, a medical device, or other device, such as those described herein with reference to the following. Fig. 6 to Fig. 11. Computing system 202 may be, for example, an embedded computing system, a system-on-chip (SoC), a single-board computing system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a laptop computer, a personal device such as a smartphone or tablet, a network of personal devices, or a combination thereof. Computing system 202 may include at least one processor 204 coupled to a memory device 208. Processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206. CPU 206 may be a commercially available processing unit implementing an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, the CPU 206 may execute stored program instructions retrieved from the memory 208.The stored program instructions may include software that controls the operation of the CPU 206 to perform the process described herein. In some examples, the processor 204 may be a system on a chip (SoC) that integrates the functionality of the CPU 206, the memory unit 208, a network interface, and input / output interfaces into a single integrated device. The computing system 202 may implement an operating system to manage various aspects of operation. While in . Fig. 2 a processor 204, a CPU 206 and a memory 208 are shown, of course more than one can be used in an overall system.

[0016] The storage unit 208 may include volatile and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses electrical power. The volatile memory may include static and dynamic random access memory (RAM) that store program instructions and data. For example, the storage unit 208 may store a machine learning model 210 or algorithm, a training dataset 212 for the machine learning model 210, and a source raw dataset 216.

[0017] Computing system 202 may include a network interface device 222 configured to enable communication with external systems and devices. Network interface device 222 may, for example, include a wired and / or wireless Ethernet interface according to the IEEE 802.11 (Institute of Electrical and Electronics Engineers) family of standards. Network interface device 222 may include a cellular interface for communicating with a cellular network (e.g., 3G, 4G, 5G). Network interface device 222 may further be configured to provide a communication interface to an external network 224 or a cloud so that the device executing computing system 202 (e.g., the client device) can communicate with server 230.

[0018] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communications protocol between computing devices. The external network 224 may enable the easy exchange of information and data between computing devices and networks.

[0019] One or more servers 230 may be connected to the external network 224. Each server may include a computing system, such as the computing system 202, such that the server 230 is configured to perform machine learning and train neural networks. Of course, for the purposes of this disclosure, certain personal or sensitive source raw data 216 originating from a particular client device may not be transmitted to the server 230, such that the source raw data is not present on the server or is completely independent of the source raw data on the computing system 202 of a client device. During operation of the federated learning system, the computing system 202 connected to a computing device may, as described further below, exchange portions of the training data 212, but not the source raw data 216 or personally identifiable information, in order to maintain the confidentiality of sensitive personally identifiable information residing on the client device.The server 230 can then access this information via a connection to the network 224 and update its stored models on the server side.

[0020] Computing system 202 may include an input / output (I / O) interface 220 that may be configured to provide digital and / or analog inputs and outputs. I / O interface 220 is used to transfer information between internal memory and external input and / or output devices (e.g., HMI devices). I / O interface 220 may include associated circuitry or bus networks to transfer information to or between the processor(s) and memory. For example, I / O interface 220 may include digital I / O logic lines that can be read or set by the processor(s), handshake lines for monitoring data transmission over the I / O lines, timing and counting devices, and other known structures for providing such functions. Examples of input devices include a keyboard, a mouse, sensors, etc.Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include additional serial interfaces for communication with external devices (e.g., a Universal Serial Bus (USB) interface). The I / O interface 220 may be referred to as an input interface (for transferring data from an external input, such as a sensor) or an output interface (for transferring data to an external output, such as a display).

[0021] Computing system 202 may include a human-machine interface (HMI) 218, which may include any device with which system 200 can receive control input. Examples of input devices may include human interface inputs such as a keyboard, mouse, touchscreen, voice input devices (e.g., microphone), and other similar devices. Computing system 202 may include a display device 232. Computing system 202 may include hardware and software for outputting graphic and text information to display device 232. Display device 232 may include an electronic screen, a projector, a printer, or other suitable device for displaying information to a user or operator. Computing system 202 may be further configured to enable interaction with remote HMI and remote display devices via network interface device 222.

[0022] System 200 may be implemented using one or more computing systems. While the example shows a single computing system 202 implementing all of the described features, various features and functions may also be separated and implemented by multiple communicating computers. In particular, a client device may implement computing system 202, and server 230 may also include its own computing system 202. The particular system architecture selected may depend on a variety of factors.

[0023] The system 200 may implement a machine learning algorithm 210 configured to analyze the source raw data set 216. The source raw data set 216 may contain raw or unprocessed sensor data that may be representative of an input data set for a machine learning system. The source raw data set 216 may contain video, video segments, images, text-based information, audio or human speech, time series data (e.g., a pressure sensor signal over time), raw or partially processed sensor data (e.g., a radar map of objects). The source raw data set 216 may contain sensitive or personal data that requires increased security, which is why the source raw data set 216 may not be transferred from the client device to the server 230. Fig. Various example inputs are illustrated and described in Figures 5-11. In some examples, the machine learning algorithm 210 may be a neural network algorithm (e.g., a deep neural network) configured to perform a predetermined function. For example, in automotive applications, the neural network algorithm may be configured to identify road signs or pedestrians in images. The neural network algorithm may be configured to automatically correct text or speech based on the context of a person's words.

[0024] Computing system 202 may store a training data set 212 for computing system 210. Training data set 212 may represent a set of previously created data for training machine learning algorithm 210. Training data set 212 may be used by machine learning algorithm 210 to learn weighting factors associated with a neural network algorithm. Training data set 212 may include a set of source data having corresponding results that machine learning algorithm 210 attempts to duplicate through the learning process.

[0025] The machine learning algorithm 210 may operate in a learning mode using the training data set 212 as input. The machine learning algorithm 210 may be executed over a number of iterations using the data from the training data set 212. At each iteration, the machine learning algorithm 210 may update internal weighting factors based on the results obtained. For example, the machine learning algorithm 210 may compare the output results (e.g., a reconstructed or augmented image when image data is used as input) with the results from the training data set 212. Because the training data set 212 contains the expected results, the machine learning algorithm 210 may determine when the performance is acceptable. After the machine learning algorithm 210 has achieved a predetermined level of performance (e.g.,Once the machine learning algorithm 210 has achieved a certain (e.g., 100% agreement with the results associated with the training dataset 212) or convergence, the machine learning algorithm 210 may be executed using data not included in the training dataset 212. For the purposes of this disclosure, "convergence" may mean that a certain (e.g., predetermined) number of iterations has occurred, or that the residual is sufficiently small (e.g., that the change in the approximation probability across iterations is less than a threshold), or that other convergence conditions are met. The trained machine learning algorithm 210 may be applied to new datasets to generate annotated data.

[0026] The machine learning algorithm 210 may be configured to identify a specific feature in the source raw data 216. The source raw data 216 may contain a plurality of instances or input data sets for which supplementary results are desired. For example, the machine learning algorithm 210 may be configured to identify the presence of a pedestrian in video images and annotate the occurrences. The machine learning algorithm 210 may be programmed to process the source raw data 216 to detect the presence of specific features. The machine learning algorithm 210 may be configured to identify a feature in the source raw data 216 as a predetermined feature (e.g., a specific word if text or spoken words are the input). The source raw data 216 may originate from a variety of sources.For example, the source raw data 216 may be actual input data acquired by a machine learning system. The source raw data 216 may be machine-generated for testing the system. The source raw data 216 may include, for example, images or videos from a camera, spoken words from a microphone, typed or written words from a keyboard or touchscreen, or the like.

[0027] Federated learning (FL) is a distributed training paradigm that enables clients scattered across the real world to cooperatively learn a global model while preserving privacy. These clients typically face strict constraints on communication bandwidth and inference speed. However, heterogeneous data distributions across clients and the increased computational effort required to maintain high performance under such conditions without increasing the risk of compromising privacy are critical bottlenecks that hinder the practical application of FL. Typically, heterogeneous data distributions among clients lead to client drift, where the global model fails to reach its optimal point due to divergent behavior between local models.When coupled with small models to compensate for inference speed and communication bandwidth limitations, there is an overall reduction in the performance and robustness achievable with both global and local models. These issues are further compounded by privacy concerns when information about local data can be leaked from standard model updates. To address these issues, we present a framework that leverages the power of baseline models to help small proxy models correct their local biases and improve the overall performance of the global model under non-IID (Independent and Identically Distributed) settings, while maintaining varying privacy guarantees and fixed communication and inference budgets.A thorough evaluation of this framework under various conditions is presented, highlighting its effectiveness in improving the performance of small-scale models under strong non-IID conditions. Furthermore, the impact of fine-tuning baseline models on local data and the degradation of obtaining an optimal global model under highly heterogeneous data distributions are presented.

[0028] Federated learning (FL) is a popular training paradigm in large-scale machine learning that can address cooperative learning across distributed clients, similar to the real-world, while maintaining the privacy of the data stored there. A typical FL framework may involve a central server that coordinates the training of a shared model by periodically averaging the local models of multiple clients, each of which must be trained only on the data stored locally on the client. In practice, due to their deployment in real-world environments, the clients are subject to strict constraints on inference speed and communication bandwidth, which in turn limits them to using small models.Combined with heterogeneous data distributions and data confidentiality constraints, these issues pose a significant challenge to the achievable overall performance and robustness.

[0029] Baseline models represent a potential solution to poor performance on heterogeneous data distributions, with the well-known advantages of portable representations for a wide range of downstream tasks and high robustness to distributional shifts. Several recent methods explore the fine-tuning of transformer-based architectures and baseline models in federated environments. However, in some cases, under extreme non-IID conditions, federated training of baseline models results in worse performance than purely local training. Furthermore, the existing method does not address the relatively large increase in inference time that arises when switching to baseline models instead of smaller alternatives such as EfficientNet, MobileNet, etc.Furthermore, most federated training algorithms require the exchange of a complete set or subset of weights with the server, which is privacy incompatible, as it has previously been shown that by capturing gradient updates or weights, a malicious user can infer several properties of the original data distribution and, in turn, exploit this information in a detrimental way.

[0030] Considering heterogeneous data distribution, inference speed, and privacy, a novel federated training framework is presented that improves the overall performance of a small-scale model by distilling information from locally stored base models in clients. This framework allows clients flexibility in the number and type of base models stored in each client, allowing the small-scale model to take advantage of improved performance and robustness against non-IID distributions and privacy-preserving optimization schemes, which are known to degrade overall performance. Furthermore, this framework eliminates the computational overhead associated with running a large base model during inference time by relying on the small-scale model to measure the overall client performance.By treating the base model as a local model and the small-scale model as a proxy that can be shared, and by training it with privacy-preserving optimization techniques, privacy can be ensured.

[0031] The unique aspects of this framework include: First, this framework is the first to deploy baseline models in federated learning environments with a limited computational budget. The federated training framework raises the upper bound of performance and robustness achievable in small client models under strict non-IID data settings.

[0032] By distilling knowledge from baseline models, this framework can overcome the degradation in overall performance observed when using optimization techniques with privacy guarantees.

[0033] Finally, this framework allows flexibility in choosing locally stored base models, allowing clients to benefit from their choice of base models and scaling of computational resources without sacrificing the privacy of their models or inference speed. Federated learning

[0034] Federated learning is a distributed machine learning approach that allows multiple clients to collaboratively train a common model while keeping their data local, thus preserving data privacy. Consider using FedAvg to provide an effective solution for jointly training a global model with periodic model averaging. Federated learning is being adopted in a number of application areas, including computer vision, natural language processing, speech recognition, healthcare, the Internet of Things, and many others. Recently, there have been concerted efforts to address certain key aspects of federated learning, such as performance under heterogeneous data conditions, reduced communication efficiency, and improved data protection. Heterogeneous data distributions:

[0035] Typically, client data may have non-identical distributions, leading to large accuracy losses during federated training, so naive aggregation such as FedAvg cannot guarantee model convergence to a local minimum. One way to address such challenges is personalized FL, which adds a proximal term to the local training objectives to allow for variations in the client model without overfitting. Other ways to address this problem include regularization, model mixing, and clustered clients, which can be used to stabilize FL with personalized models. Multi-task learning, in which different tasks are executed simultaneously, and meta-learning, in which a global model is quickly adapted to the local data distribution of each client, can also mitigate the problem.In this work, the problem of heterogeneous data distribution is addressed by using baseline models, as they can provide a more general solution while maintaining the same amount of data as traditional FL methods. Basic models

[0036] Baseline models can be advantageous in specific areas such as the integration of speech, vision, and multimodal pretraining approaches. By leveraging the power of pretraining on large datasets, these models can be seamlessly applied to a wide variety of tasks. This framework can apply baseline models in FL to improve client robustness to distribution shifts and heterogeneous data distributions, or to boost overall system performance. While baseline models greatly improve performance in heterogeneous data environments, directly training and evaluating baseline models increases the computational overhead for the client and latency during inference, making their use in real-world conditions difficult.Instead, this framework takes advantage of locally stored base models to improve the performance and robustness of small models while keeping communication and inference overhead low. Inference efficiency

[0037] We turn to FL strategies that address the problem of inference efficiency. Consider, for example, whether to use dropouts to ensure efficient inference, develop a method for adaptively pruning parameters during federated training, or provide a theoretical analysis of the design strategy for a local model's pruning mask. Consider whether federated learning with personalized and structured sparse masks is feasible, or whether to create different local models as subnetworks of the global network by selecting important continuous parameters layer by layer through structured pruning using static batch normalization.An important difference between existing work addressing inference efficiency and the approach presented here is the use of pruning as a platform for personalizing and improving communication and inference times per client. Instead, the available budget is set and the performance cap is improved for smaller models. These two approaches are complementary and can be combined to enhance their impact. Data protection

[0038] To mitigate privacy concerns in FL, an FL framework should be considered that allows clients to train a meme / proxy model and a private model through mutual learning. The meme / proxy model can be shared only with the server, thus reducing the risk of data exposure. However, there may be vulnerabilities in capturing model weights / updates during the information exchange between clients and server. To provide stronger privacy protection, one can use techniques such as differential privacy, secure multi-party computation, secret sharing, homomorphic encryption, and hybrid techniques. To address the privacy issue in FL, ProxyFL can use a formulation where clients maintain a private model and a proxy model, which are trained through mutual learning.The client communicates with others exclusively by exchanging its proxy model, thus ensuring data and model confidentiality. However, in ProxyFL, each client can use DP-SGD during training, which provides measurable guarantees for differential privacy. Similarly, an FL framework based on differential privacy (DP) can obscure and protect information about the clients' data distributions by adding artificial noise to the parameters during local training in the clients. Inspired by methods proposed to actively address privacy concerns, this framework can adopt the method proposed in ProxyFL and leverage the impact of baseline model distillation on the performance of DP-based optimizers. distillation

[0039] Knowledge distillation is a teaching technique that allows valuable insights and generalization capabilities to be transferred from a teacher model to a student model. In the FL domain, this framework explores customizable aggregation methods using ensemble distillation to fuse models on the server and then considers an auxiliary dataset to weight and stitch local models from each client. Consider using FedDistill, a method that extracts statistics related to the logit vector of client models, which are then shared with other clients to facilitate knowledge distillation. Furthermore, a data-free approach to knowledge distillation (FEDGEN) is considered to address data heterogeneity in FL. In this approach, a generative model is trained on the server by combining client information, and the generator data is used to improve the performance of the client models.Or a co-distillation-based personalized federated learning approach that enables cross-architecture training. This framework considers the impact of the base formulation of knowledge distillation on the performance of the small model while avoiding the use of excess data, augmentations, or model sharing to provide guidance on how to effectively leverage base models in federated learning. USE OF BASE MODELS IN FEDERATED LEARNING

[0040] Consider the conventional FL framework FedAvg shown below, supplemented by the key components of the framework of this disclosure. STANDARD FL FRAMEWORK

[0041] FedAvg can be briefly described by the following equation, minθ∈ℝdf(θ):=1N∑i=1Nf(θi) where θ ∈ R dthe parameters of the global model and N is the number of clients. f(θi) is the loss function of the i-th client, where each client contains its local data distribution denoted by D i is called. APPROACH

[0042] This paper presents a new framework in which each of the N clients contains two different types of local models: (a) pre-trained base models (private) θFMi1,θFMi2,…,θFMiMi, and (b) small-scale model (proxy), denoted by θiS. In this framework, the objective of FL remains the same as in Eq. 1, with the models to be optimized being the small-scale proxy models. Framework Core

[0043] A summary of the core of the framework is presented in Algorithm 1. During training, the server receives locally updated small-scale models from each client. These models are aggregated using formulations similar to Equation 1 or more complex weighting schemes such as minθ∈ℝdθ:=1|Di|∑i=1NθiS which takes into account the distribution of data among the clients.

[0044] After model aggregation is complete, the server sends the aggregated model to each client, which then updates the weights of its local small-scale model using the aggregated model. For each client, the small-scale model is trained using the local data distribution with the following loss: L(θiS)=λLCE(θiS)+(1−λ)LDistill(θiS,θiFM1,…,θiFMMi), LCE(θi)=E(xy)∼DiICE(h(x;θi),y) LDistilli(θiS,θiFM1,…,θiFMMi)=∑m=1MiE(x,y)∼DilKL[h(x;θi)‖h(x;θFMmi)] where CE is the cross-entropy loss and LDistill is the Kullback-Leibler (KL) divergence loss used to distill knowledge from the baseline model to the small-scale proxy model. The parameter λ controls the proportion of knowledge distilled from the baseline model compared to the ground-truth labels. Once local training is complete, each client pushes its local small-scale model to the server, and the entire process of aggregation, transfer, and client update is repeated until convergence.

[0045] Fig. Figure 3A: Overview of the proposed lightweight federated learning system. For each client, there are two models: the student model and the frozen baseline model. The frozen baseline model generates the logit to guide the training of the student models. The server-side aggregate model aggregates information from various local models without observing their data. In the inference phase, the baseline model is discarded.

[0046] Let's consider a first exemplary embodiment and recall that federated learning is the process of cooperative learning between different clients while preserving the privacy of the data stored in them. The server acts as a supervisor for the learning process by receiving and aggregating the information provided by each client before disseminating the aggregated model to all clients. A typical federated learning system suffers from issues such as reduced performance and robustness, especially when faced with problems such as data heterogeneity and distribution changes. Baseline models, pre-trained on a large amount of data, can be widely customized as initialization for the model parameters.In the scenario where clients have access to their own base models but have strong privacy concerns, the maximum achievable performance would be limited.

[0047] Considering the above challenges, this embodiment of the framework aims to address these problems by proposing the use of base models at the local level within clients to improve their performance, while using a smaller student model (proxy) to coordinate and synchronize information between different clients. Then, leverage the knowledge gained from the base models to guide the training of the smaller model. This embodiment presents a framework that can take advantage of different feature embeddings of base models and distill knowledge to improve learning performance across a variety of clients. This allows clients to choose their own base models instead of using the same base model.By consistently distilling the performance of base models into a smaller student model on the client side and updating the student model with aggregated weights from the server, this framework leverages the prior knowledge of pre-trained base models while preserving the information presented by the client population. Clients distill knowledge from their own selection of pre-trained base models, which are fine-tuned using local data, and have access to aggregated knowledge in the federated learning framework. Thus, this framework strikes a balance between efficient communication setup between server and clients, the privacy protection achieved by not directly revealing the base model, and the high level of performance and robustness achievable with base models.

[0048] The idea here is to distill knowledge from the base models and train a smaller model. The distilled, smaller model is sent to the server, which provides an updated aggregated overall model. Prototypes generated from pre-trained models are then combined with a contrastive loss to update the representations learned on each client. This implementation provides an alternative solution that approximates the performance of base models while training a smaller model. By distilling knowledge from a proprietary selection of pre-trained base models, combined with updates from clients and adaptations to local client data, this framework provides a high-performance, stable, and small-scale baseline for the client.Since the selection of base models is private to the client, this embodiment preserves the client's privacy more than using the same base model for all clients.

[0049] This embodiment can be used for training models with many different sensor input systems using federated / decentralized / collaborative learning in industrial IoT, sensor networks, and manufacturing automation scenarios.

[0050] Fig. Figure 3B is a flowchart for privacy-preserving federated learning with client-side distillation of base models. The server sends model weights collected and aggregated across all clients to each client. At each client, the smaller model is initialized using the weights sent by the server. The model is then updated using two losses, one distilling information from the pre-trained base models stored in the client, and the second being a cross-entropy loss. Once the smaller client model is updated, its weights are submitted to the server, and the entire process repeats.

[0051] Suppose N each client stores M i(i ∈ [N]) each client stores pre-trained base models alongside a smaller student model, and a central server hosts a smaller model. The selection of base models can cover a wide range of variations, including, but not limited to, variations in the architecture, the loss function, the pre-training dataset, etc. The server contains a smaller model (θ S ) that aligns with the backbone of the client model and serves as the primary synchronization point for the clients. Overall, the server acts as an orchestrator for federated training, sending and receiving information from the clients.

[0052] For each client i, the parameters of the M i Basic models and the distilled smaller model each with θFM1i,θFM2i,…,θFMMii and θDSi Each client-i has access to locally stored data, which is denoted by (D i , y i ). In addition, there are some key parameters in the federated training framework, such as the number of epochs in which a client is trained between each round of communication (to the server), the batch size, the learning rate, the type of optimizer, and so on. Input:

[0053] The central server (orchestrator) hosts a smaller student model that mirrors the client model.

[0054] N clients in the federated learning system, with their locally stored data (D i , y i ) and initialized local models θDS1,θds2,…,θDSN, and stores a number of M pre-trained base models. Steps:

[0055] At the beginning of each round (r) of federated training, the server receives model weights from each client in [N]. The model weights are aggregated and used to initialize θ DS used in the following way, θSr=∑k=1N1|Dk|θkDS

[0056] Here we assume that the data is evenly distributed across all clients, so that we can assign a reweighted importance to each client model.

[0057] After the model aggregation described in the previous step is completed, the student model is sent to each client.

[0058] After each client receives the information from the server, it resets its model by accepting and reinitializing its weights using the aggregated model and starts training on its local data (D i , y i), using the loss determined by combining the distilled knowledge from the base models and fitting it to its local data. For the part of the knowledge distillation from the pre-trained base models θFM1i,θFM2i,…,θFMMii on the local data set (D i , y i ) the corresponding loss item is given by, Ldistill=1|Mi|∑m=1Mi∑i=1CqFMmilog(qFMmiqDSi), referred to as KL divergence loss [5] where qi=exp(zi / T)∑j=1Cexp(zj / T),Zi the logits of class i th and T denotes the distillation temperature. Combining the training fit with local data, the following loss function results: minθi1|Di|∑d∈Dil(qi,yi)+Ldistill, where l() is the cross-entropy loss function.

[0059] These operations are then repeated until the termination criterion is reached (e.g., the total number of rounds, the loss / power tolerance, etc.).

[0060] Consider a second exemplary embodiment and recall that federated learning is the process of cooperative learning across multiple clients while maintaining the privacy of the data stored within them. The server acts as a supervisor for the learning process, receiving and aggregating the information provided by each client before disseminating the aggregated model to all clients. Clients are often deployed to solve problems in real-world contexts, constraining their infrastructure with limited computational resources, runtime, memory, etc. Given this, the clients may not be suitable for storing and executing modern models, which are often extremely large. Therefore, the clients may be limited to using smaller models, limiting the maximum achievable performance.Such adjustments can reduce the overall system's performance and robustness, especially when faced with problems such as data heterogeneity and distribution shifts, which are common in federated learning.

[0061] Considering the above challenges, this second embodiment aims to address these problems by proposing the use of base models within the server to coordinate and synchronize information between different clients. While the use of base models within clients improves their personalization capabilities, limitations in client computing resources may not allow the storage and execution of base models within all clients, regardless of whether all or a subset of the base model contents are shared between server and client. Combining the feature embeddings generated by pre-trained networks in a federated manner is considered, assuming the existence of pre-trained models on each client.This second embodiment presents a framework for leveraging the benefits of different feature embeddings of base models hosted on a central server to coordinate and improve learning performance across multiple clients, without the additional overhead of directly sharing the base model. By consistently distilling the performance of frozen base models into a smaller student model on the server, reflecting a model backbone more responsive to a client device, coupled with updating the student model with aggregate weights from the clients, our system leverages the prior knowledge of pretrained base models while preserving the information presented by the clients.The clients, in turn, receive a better initial model, the distilled student model on the server, and perform further fine-tuning using local data to improve their performance. In this way, this framework strikes a balance between establishing efficient communication between the server and clients and the high level of performance and robustness achieved by using base models.

[0062] In addition to using base models on individual clients of a federated system, this second embodiment focuses on adapting base models hosted on a central server by distilling them into a smaller model. The distilled, smaller base model receives updates from all clients simultaneously and then synchronously sends an updated global model to each client. It combines prototypes generated from pre-trained models with a contrastive loss to update the representations learned on the individual clients. An alternative solution approximates the powerful base models while using a smaller model that is better suited to the available resources.By placing the base model on a central server and using a distilled version of the model in conjunction with updates by the clients, the global distilled model provides a powerful, stable, and small-scale baseline for the client that can be further optimized using local data.

[0063] This embodiment can be used for training models with many different sensor input systems using federated / decentralized / collaborative learning in industrial IoT, sensor networks, and manufacturing automation scenarios.

[0064] Fig. is a flowchart for federated learning with server-side model distillation, where clients send locally adjusted model weights to the server. On the server, all client models are aggregated. Once aggregation is complete, the model is updated by distilling from the base model using a public dataset, after which the distilled model is sent to all clients. At each client, the smaller model is initialized with the weights sent by the server, and the entire process is repeated.

[0065] Suppose a central server stores M pre-trained base models, each of which is θFM1,θFM2,…,θFMM are parameterized. The selection of base models can cover a wide range of variations, including, but not limited to, variations in architecture, loss function, pre-training dataset, etc. In addition, the server includes a smaller Student model (θ DS ), which aligns with the backbone of the client model and serves as the primary synchronization point for the clients. Overall, the server acts as an orchestrator for federated training, sending and receiving information from the clients.

[0066] The federated learning system consists of N clients. Typically, the clients are constrained to use as few hardware resources as possible. Therefore, it makes sense to use a common small model for all clients to mimic this behavior.

[0067] In general, there is a common public dataset (D p , y p), which is accessible to both the server and all clients. For each client-i, its locally stored data and model are stored with (D i , y i ) and θ i In addition, there are some key parameters in federated training, such as the number of epochs in which a client is trained between each round of communication (to the server), the batch size, the learning rate, the type of optimizer, and so on. Input:

[0068] The central server (Orchestrator) stores a number of M pre-trained base models (deep neural networks) and a smaller student model θ DS that reflects the customer's model.

[0069] N clients in the federated learning system, where each client i has locally stored data (D i , y i ) and a local model with initialized parameters θ i has.

[0070] Public data (D P , y P ), which are accessible to both the server and the clients. Steps:

[0071] 1. At the beginning of each round (r) of federated training, the server receives model parameters θ i from each client in [N]. The model parameters are aggregated and used to initialize θrDS used in the following way, θrDS=∑k=1N|Dk|∑i=1N|Di|θrk

[0072] The importance of each client model is re-weighted to take into account the amount of local data in each client relative to the total amount of data in the system.

[0073] 2. After the model aggregation is completed, the public dataset is used to incorporate knowledge from the pre-trained base models into the student model with aggregated parameters θrDS to distill. The loss used for distillation is given by, 1|M|∑m=1M∑i=1CqiFMm log(qiFMmqiDS), referred to as KL divergence loss [5] where qi=exp(zi / T)∑j=1Cexp(zj / T), the logits of the i th Class and T denotes the distillation temperature.

[0074] 3. When the distillation is complete, the student model is transferred to each client, where the local model weights (θr1,θr2,…,θrN) , be reinitialized with the student model transmitted from the server (θrDS). ). Then the clients start training on their local data, (D i , y i ), using the minθri1|Di|∑d∈Dil(qi,yi), given loss, where 1() is the loss function.

[0075] Steps 1-3 are then repeated until the stopping criterion is reached (e.g. total number of laps, loss / power tolerance, etc.).

[0076] The machine learning models described here can be used in many different areas. As described above, the source raw data stored locally can be image data, audio data, or similar, and therefore Fig. 6 to Fig. 11 different applications in which this data is retrieved or used are shown as examples. The structure used for training and using the machine learning models for these (and other) applications is shown in Fig. 5 is shown as an example. Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine 500 and a control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to detect a state of the computer-controlled machine 500. The sensor 506 may be configured to encode the detected state into sensor signals 508 and transmit the sensor signals 508 to the control system 502. Non-limiting examples of sensors 506 include video, radar, LiDAR, microphone, ultrasonic, and motion sensors. In one embodiment, the sensor 506 is an optical sensor configured to detect optical images of an environment proximate the computer-controlled machine 500.

[0077] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As explained below, the control system 502 may further be configured to calculate actuator control commands 510 depending on the sensor signals and to transmit actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0078] As in Fig. 5, the control system 502 includes a receiving unit 512. The receiving unit 512 may be configured to receive sensor signals 508 from the sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x without a receiving unit 512. Each input signal x may be a part of each sensor signal 508. The receiving unit 512 may be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to an image captured by the sensor 506.

[0079] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify input signals x into one or more labels using a machine learning algorithm, such as a neural network described above. The classifier 514 is configured to be parameterized by parameters such as those described above (e.g., parameter θ). The parameters θ can be stored in and provided by the non-volatile main memory 516. The classifier 514 is configured to determine the output signals y from the input signals x. Each output signal y contains information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signals y to the conversion unit 518. The conversion unit 518 is configured to convert output signals y into the actuator control commands 510.The control system 502 is configured to transmit actuator control commands 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control commands 510. In another embodiment, the actuator 504 is configured to directly actuate the computer-controlled machine 500 based on the output signals y.

[0080] Upon receipt of actuator control commands 510 by actuator 504, actuator 504 is configured to perform an action corresponding to the corresponding actuator control command 510. Actuator 504 may include control logic configured to convert actuator control commands 510 into a second actuator control command used to control actuator 504. In one or more embodiments, actuator control commands 510 may be used to control a display instead of, or in addition to, an actuator.

[0081] In another embodiment, the control system 502 includes a sensor 506 instead of or in addition to the computer-controlled machine 500 with sensor 506. The control system 502 may also include an actuator 504 instead of or in addition to the computer-controlled machine 500 with the actuator 504.

[0082] As in Fig. 5, control system 502 also includes a processor 520 and a memory 522. Processor 520 may include one or more processors. Memory 522 may include one or more storage units. Classifier 514 (e.g., machine learning algorithms such as those described above) of one or more embodiments may be implemented by control system 502, which includes non-volatile main memory 516, processor 520, and memory 522.

[0083] Non-volatile main memory 516 may include one or more persistent data storage devices, such as a hard disk, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or another device capable of permanently storing information. Processor 520 may be one of the processors or processor subsystems described above with reference to Fig. 1-2, and may comprise one or more devices selected from high performance computing (HPC) systems, including high performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, tensor processing units, graphics processing units, ASICs, or other devices that process (analog or digital) signals based on computer-executable instructions in memory 522.The memory 302 may include a single memory device or a number of memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0084] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions located in non-transitory main memory 516 that embody one or more machine learning algorithms and / or methods of one or more embodiments. Non-transitory main memory 516 may include one or more operating systems and applications. Non-transitory main memory 516 may store compiled and / or interpreted computer programs created using a variety of programming languages ​​and / or technologies, including, without limitation, and either alone or in combination, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL.

[0085] When executed by processor 520, the computer-executable instructions of non-volatile memory 516 may cause control system 502 to implement one or more of the algorithms and / or methods disclosed herein. Non-volatile main memory 516 may also contain machine learning data (including data parameters) that support the functions, features, and processes of the embodiments described herein.

[0086] The program code embodying the algorithms and / or methods described herein may be distributed individually or collectively as a program product in a variety of different forms. The program code may be distributed using a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to perform aspects of one or more embodiments. Computer-readable storage media, which are inherently non-transitory, may include volatile and non-volatile, as well as removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technology, portable compact disc read-only memory (CD-ROM) or other optical storage device, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer.Computer-readable program instructions may be downloaded to a computer, other type of programmable data processing apparatus, or other device from a computer-readable storage medium or over a network to an external computer or storage device.

[0087] Computer-readable program instructions stored on a computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to function in a particular manner such that the instructions stored on the computer-readable medium produce an article of manufacture, including instructions that implement the functions, acts, and / or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, acts, and / or operations specified in the flowcharts and diagrams can be rearranged, processed serially, and / or processed concurrently in accordance with one or more embodiments.Moreover, each of the flowcharts and / or illustrations may include more or fewer nodes or blocks than those illustrated in accordance with one or more embodiments.

[0088] Alternatively, the processes, methods, or algorithms may be embodied in whole or in part using suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, control units, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0089] The Fig. 6 to Fig. 11 illustrate examples of environments in which the federated learning systems described herein may be implemented. Each of these embodiments illustrates an embodiment of a client device. In these embodiments, the dataset originating from sensor 506 may be the source raw data used to train a machine learning model on the device (client) but is not transmitted to the server to protect the data. Fig. 6 shows a schematic diagram of control system 502 configured to control vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. Vehicle 600 includes actuator 504 and sensor 506. Sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). One or more of the specific sensors may be integrated into vehicle 600. Sensor 506 may include a software module configured to determine a state of actuator 504 when executed. One non-limiting example of a software module includes a weather information software module configured to determine a current or future state of the weather in the vicinity of vehicle 600 or another location.

[0090] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect objects in the vicinity of the vehicle 600 depending on input signals x. In such an embodiment, the output signal y may contain information characterizing the proximity of the objects to the vehicle 600. The actuator control command 510 may be determined according to this information. The actuator control command 510 may be used to avoid collisions with the detected objects. The source raw data for federated learning may include the raw images of the vehicle's surroundings, but the vehicle's processing of the objects in the surroundings may change the weights in the machine learning model used onboard the vehicle; these adjusted weights may then be sent back to the server's models for updating.

[0091] In embodiments where the vehicle 600 is an at least partially autonomous vehicle, the actuator 504 may be embodied in a brake, a drive system, an engine, a powertrain, or a steering system of the vehicle 600. Actuator control commands 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids collisions with detected objects. Detected objects may also be classified according to what the classifier 514 considers most likely, e.g., pedestrians or trees. The actuator control commands 510 may be determined depending on the classification. In a scenario where a hostile attack may occur, the system described above may be further trained to better detect objects or to identify a change in lighting conditions or angle for a sensor or camera on the vehicle 600.

[0092] In other embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control command 510 may be determined such that a drive unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot can avoid collisions with identified objects.

[0093] In another embodiment, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as sensor 506 to determine a condition of plants in an environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray chemicals. Depending on an identified type and / or condition of the plants, the actuator control command 510 may be determined to cause the actuator 504 to spray the plants with an appropriate amount of appropriate chemicals.

[0094] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect a condition of an object to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor 506 may detect a condition of the laundry in the washing machine. The actuator control command 510 may be determined based on the detected condition of the laundry.

[0095] Fig. Figure 7 shows a schematic diagram of control system 502 configured to control system 700 (e.g., a manufacturing machine), such as a punching machine, a cutting machine, or a drill, of manufacturing system 702, e.g., as part of a production line. Control system 502 may be configured to actuate actuator 504 configured to control system 700 (e.g., the manufacturing machine).

[0096] The sensor 506 of the system 700 (e.g., the manufacturing machine) may be an optical sensor configured to detect one or more properties of the manufactured product 704. The classifier 514 may be configured to determine a state of the manufactured product 704 from one or more of the detected properties. The actuator 504 may be configured to control the system 700 (e.g., the manufacturing machine) for a subsequent manufacturing step of the manufactured product 704 depending on the determined state of the manufactured product 704. The actuator 504 may be configured to control functions of the system 700 (e.g., the manufacturing machine) on the subsequently manufactured product 706 of the system 700 (e.g., the manufacturing machine) depending on the determined state of the manufactured product 704.

[0097] Fig. Figure 8 shows a schematic diagram of the control system 502 configured to control the power tool 800, such as a drill or a drill driver, that operates at least partially autonomously. The control system 502 may be configured to control the actuator 504 configured to control the power tool 800.

[0098] The sensor 506 of the power tool 800 may be an optical sensor configured to detect one or more characteristics of the work surface 802 and / or the fastener 804 being driven into the work surface 802. The classifier 514 may be configured to determine a condition of the work surface 802 and / or the fastener 804 relative to the work surface 802 from one or more of the detected characteristics. The condition may be that the fastener 804 is aligned with the work surface 802. Alternatively, the condition may be the hardness of the work surface 802. The actuator 504 may be configured to control the power tool 800 such that the drive function of the power tool 800 is adjusted depending on the determined condition of the fastener 804 relative to the work surface 802 or one or more detected characteristics of the work surface 802.For example, the actuator 504 may interrupt the drive function when the state of the fastener 804 is aligned relative to the working surface 802. As another non-limiting example, the actuator 504 may apply additional or lesser torque depending on the hardness of the working surface 802.

[0099] Fig. Figure 9 shows a schematic diagram of the control system 502 configured to control the automated personal assistant 900. The control system 502 may be configured to actuate the actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, a stove, an oven, a microwave, or a dishwasher.

[0100] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of gestures 904 from user 902. The audio sensor may be configured to receive a voice command from user 902.

[0101] The control system 502 of the automated personal assistant 900 may be configured to determine actuator control commands 510 configured to control the system 502. The control system 502 may be configured to determine actuator control commands 510 depending on sensor signals 508 from the sensor 506. The automated personal assistant 900 is configured to transmit sensor signals 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify the gesture 904 of the user 902, determine the actuator control commands 510, and transmit the actuator control commands 510 to the actuator 504.The classifier 514 may be configured to retrieve information from a non-volatile memory in response to the gesture 904 and output the retrieved information in a form suitable for receipt by the user 902.

[0102] Fig. 10 shows a schematic diagram of control system 502 configured to control surveillance system 1000. Control system 1000 may be configured to physically control access through door 1002. Sensor 506 may be configured to detect a scene relevant to deciding whether to grant access. Sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. Such data may be used by control system 502 to detect a person's face.

[0103] The classifier 514 of the control system 502 of the surveillance system 1000 may be configured to interpret the image and / or video data by comparing the identities of known individuals stored in the non-volatile main memory 516, thereby determining the identity of an individual. The classifier 514 may be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 may be configured to lock or unlock the door 1002 in response to the actuator control command 510. In other embodiments, non-physical, logical access control is also possible.

[0104] The surveillance system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene to be monitored, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of a scene, e.g., whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adapt the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that the classifier 514 classifies as suspicious.Using an embodiment of the disclosed system, the monitoring system can predict the appearance of objects at specific times in the future.

[0105] Fig.11 shows a schematic diagram of the control system 502 configured to control the imaging system 1100, e.g., an MRI machine, an X-ray imaging machine, or an ultrasound machine. The sensor 506 may, for example, be an image sensor. The classifier 514 may be configured to determine a classification of all or part of the acquired image. The classifier 514 may be configured to determine or select an actuator control command 510 depending on the classification obtained by the trained neural network. For example, the classifier 514 may interpret a region of an acquired image as potentially anomalous. In this case, the actuator control command 510 may be determined or selected to cause the display 1102 to display the image and highlight the potentially anomalous region.The captured image can be used in the hospital environment to train machine learning systems within the hospital (client). However, this data is not sent to the server to train the server's models. Instead, the hospital's adjusted weights, based on the captured images, can be sent to the server to adjust the weights on the server side.

[0106] While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As previously described, the features of various embodiments may be combined to form other embodiments of the invention that need not be expressly described or illustrated. While various embodiments may be described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or moreone or more features or characteristics may be compromised to achieve desired overall system characteristics that depend on the specific application and implementation. These characteristics may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Therefore, to the extent that any embodiments are described as less desirable than other prior art embodiments or implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and may be desirable for certain applications.

Claims

[1] A method for training neural networks with federated learning, the method comprising: Sending at least a portion of a server-managed machine learning model from a server to a plurality of clients, resulting in a plurality of local machine learning models; at each of the plurality of clients, training the plurality of local machine learning models with locally stored data stored locally at that respective client, wherein the training at each client includes determining a respective cross-entropy loss for each of the plurality of local machine learning models; Updating the respective weights for each of the plurality of local machine learning models; Transferring the respective updated weights from each client to the server without transferring the clients' locally stored data; aggregating the respective weights of each of the plurality of local machine learning models on the server to obtain an aggregated server-managed machine learning model; Generating a distillation loss on the server based on a base model on the server; Updating the aggregated server-managed machine learning model on the server to obtain updated aggregated respective weights; transmitting the updated aggregated respective weights to each of the plurality of clients; and Update each of the plurality of local machine learning models with the updated aggregated respective weights. [2] The method according to claim 1, wherein the aggregation is carried out according to θsr=∑k=1N1|Dk|θDSk. [3] System for training neural networks with federated learning, the system comprising: Memory that stores instructions; and a multitude of processors that work together to execute the instructions stored in memory: Sending at least a portion of a server-managed machine learning model from a server to a plurality of clients, resulting in a plurality of local machine learning models; at each of the plurality of clients, training the plurality of local machine learning models with locally stored data stored locally at that respective client, wherein the training at each client includes determining a respective cross-entropy loss for each of the plurality of local machine learning models; Updating the respective weights for each of the plurality of local machine learning models; Transferring the respective updated weights from each client to the server without transferring the clients' locally stored data; aggregating the respective weights of each of the plurality of local machine learning models on the server to obtain an aggregated server-managed machine learning model; Generating a distillation loss on the server based on a base model on the server; Updating the aggregated server-managed machine learning model on the server to obtain updated aggregated respective weights; transmitting the updated aggregated respective weights to each of the plurality of clients; and Update each of the plurality of local machine learning models with the updated aggregated respective weights. [4] Method according to claim 3, wherein the aggregation is carried out according to θsr=∑k=1N1|Dk|θDSk. [5] A neural network-driven robot system comprising: Memory that stores instructions; and at least one processor that collectively trains the neural networks with federated learning when executing the instructions stored in memory, by: Sending at least a portion of a server-managed machine learning model from a server to a plurality of clients, resulting in a plurality of local machine learning models; at each of the plurality of clients, training the plurality of local machine learning models with locally stored data stored locally at that respective client, wherein the training at each client includes determining a respective cross-entropy loss for each of the plurality of local machine learning models; Updating the respective weights for each of the plurality of local machine learning models; Transferring the respective updated weights from each client to the server without transferring the clients' locally stored data; aggregating the respective weights of each of the plurality of local machine learning models on the server to obtain an aggregated server-managed machine learning model; Generating a distillation loss on the server based on a base model on the server; Updating the aggregated server-managed machine learning model on the server to obtain updated aggregated respective weights; transmitting the updated aggregated respective weights to each of the plurality of clients; and Update each of the plurality of local machine learning models with the updated aggregated respective weights. [6] Method according to claim 5, wherein the aggregation is carried out according to θsr=∑k=1N1|Dk|θDSk. [7] Robot system according to claim 5, wherein the robot system is an autonomously driving vehicle. [8] The robot system of claim 5, wherein the robot system is a medical system.

Images