Computationally efficient machine learning

Abstract

A computer-implemented method for computationally efficient machine learning includes: a) generating, by a server computing device, a set of initial model parameters; b) generating, by the server computing device, a set of partition parameters, the set of partition parameters comprising at least the following parameters: a number of partitions and a minimum number of data points in the partitions; c) transmitting, by the server computing device, the set of initial model parameters and the set of partition parameters to each client computing device; d) obtaining, by each client computing device, a global value from the server computing device corresponding to the set of initial parameters; e) selecting, by the server computing device, a client computing device based on the global sequence of all client computing devices; f) transmitting, by the server computing device, the machine learning-based model parameters to the selected client computing device; g) obtaining, by the selected client computing device, global values corresponding to the machine learning-based model parameters; h) incrementally training, by the selected client computing device, the machine learning model based at least in part on a partition of the local data set selected from the local data set according to a predefined scheme to obtain updated machine learning-based model parameters; i) transmitting, by the selected client computing device, the updated machine learning-based model parameters to the server computing device; and j) repeating the steps d) to i) until all predefined conditions are met.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning. More specifically, the present disclosure relates to computationally efficient federated machine learning. Background Art

[0002] Federated learning relies on transferring model parameters from client computing devices to server computing devices, and from server computing devices to client computing devices. Federated learning (also known as collaborative learning) is a machine learning technique in which multiple participants collaborate to train machine learning models without sharing original data. However, current research shows that traditional federated learning schemes based on gradient model updates cannot save communication costs and show lower accuracy performance on notidentically distributed and independent (non-IID) data structures. Communication costs and non-IID are two obstacles in traditional federated learning schemes. How to build computationally efficient machine learning with non-IID data is a key challenge.

[0003] Figure 1 The concept of local learning is shown in . In this learning scheme, there are multiple local machine learning models 10, which are trained separately by multiple local computing devices 12 using samples from local data sets. Multiple local data sets are usually distributed in different geographical areas. However, this learning scheme is not universal, and the local models 10 reflect the characteristics and biases of local data, and when they are used in other geographical areas, they usually provide biased or inaccurate predictions. The performance, model accuracy and complexity of this learning scheme are heavily dependent on the properties of each local computing device 12.

[0004] Figure 2The concept of central learning is shown in . Compared with local learning, central learning centralizes data and parameters from different sources to solve the problem of insufficient local data for local learning. In this learning scheme, there is a single (global) machine learning model 20, which is then trained by using aggregated samples from multiple local data sets. The samples are sent by the local computing device 22 to the server (not shown) responsible for training the global model 20 through a communication channel. The access of the local computing device 22 to the global model 20 is managed by the server that also acts as an arbitration device. However, the arbitration device itself does not make any changes to the global model 20. Typical applications of central learning are, for example, expansions in the form of globally deployed applications, such as Google's advertising recommendation system or Netflix's video recommendation system. However, with the increase in the data traffic required to move data in a centralized scheme, this type of learning may raise concerns about data ownership, confidentiality, privacy and security among data owners, as well as the possibility of creating data monopolies that benefit data aggregators.

[0005] In order to improve the security and solve the data confidentiality problem in centralized learning schemes, federated learning schemes have been proposed. Figure 3 The concept of federated learning is shown. In the federated learning scheme, there are multiple local machine learning models 30, which are independently trained by several local computing devices 32 using samples from local data sets. The local models 30 are periodically forwarded to a central server computing device 34 that maintains a global machine learning model 36. The server computing device 34 manages the access of the local computing device 32 to the global model 30, and once any one or more of the multiple local models 30 are updated, the multiple local models 30 are securely aggregated into the global model 36.

[0006] In centralized federated learning protocols, the server coordinates communication, but learning between peer models is not incremental, while private algorithms based on incremental learning are decentralized (Sheller et al. "Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation." International MICCAI Brainlesion Workshop, pp. 92-104. Springer, Cham, 2018). Although decentralized systems have some advantages over centralized frameworks, they are generally more difficult to implement and more prone to implementation errors. In addition, in the Cyclic Institutional Incremental Learning (CIIL) scheme (Chang et al., "Distributed deep learning networks among institutions for medical imaging", Journal of the American Medical Informatics Association, Volume 25, Issue 2, 8, August 2018, pp. 945–954 (Chang et al., “Distributed Deep Learning Networks Across Medical Imaging Institutions,” Journal of the American Medical Informatics Association, Vol. 25, No. 8, August 2018, pp. 945–954), does not describe a mechanism to provide robustness (e.g., to erroneous connections to clients).

[0007] In the CIIL scheme, the order of each client computing device in the update queue is fixed. If the datasets of the client computing devices are non-identically distributed independent (non-IID), catastrophic forgetting (after updating the model on a new dataset, the updated model will not be very accurate on the previously learned data, i.e., "it forgets what it previously learned") may be even higher, because the datasets of the client computing devices with all-data distribution may have subsets in which two datasets are not too far from each other.

[0008] Continuous model updating is an inherent feature of incremental learning, so catastrophic learning is a major challenge in this field. Recently, many solutions have been provided to solve this problem. One of these solutions is the so-called elastic weight consolidation (EWC) (Kirkpatrick et al, "Overcoming catastrophic forgetting in neural networks", Proceedings of the National Academy of Sciences 114, No.13 (2017): 3521-3526. (Kirkpatrick et al., "Overcoming catastrophic forgetting in neural networks", Proceedings of the National Academy of Sciences Vol. 114, No. 13 (2017): 3521-3526), ​​EWC penalizes the loss function based on the distribution of individual weights in the previous model. The method according to the present invention is designed to use this tool from the beginning.

[0009] An innovative chained secure multi-party computing technology based on federated learning, named Preserving Federated Learning (PPEL), is proposed and applied to edge computing (Li et al, "Privacy-Preserving Federated Learning Framework Based on Chained Secure Multi-party Computing"). In this framework, the server sends global model parameters to all clients, which then train the global model in a chained manner, and the last client sends the global model back to the server. This Chained-PPEL focuses on privacy-preserving federated learning that prevents sensitive information leakage.

[0010] Therefore, an object of the present invention is to at least partially eliminate the shortcomings of currently used collaborative learning schemes and provide an improved method of federated machine learning, which provides computational efficiency with significantly reduced computational cost, improves the accuracy of federated learning under non-IID data structures (including extreme non-IID scenarios), and combines the robustness of server-based federated learning architectures with the low computational and communication costs of recurrent schemes.

[0011] The above object is achieved by providing a computer-implemented method for efficient machine learning, the method being performed by a server computing device and a plurality of client computing devices connected to the server computing device, wherein each of the plurality of client computing devices has a local data set stored locally, the method comprising:

[0012] a) generating a set of initial model parameters by a server computing device;

[0013] b) generating, by the server computing device, a set of partitioning parameters, the set of partitioning parameters comprising at least parameters: the number of partitions and the minimum number of data points in a partition;

[0014] c) transmitting, by the server computing device, to each client computing device, the set of initial model parameters and the set of partition parameters;

[0015] d) obtaining, by each client computing device, a global value corresponding to the set of initial parameters from the server computing device;

[0016] e) selecting, by the server computing device, the client computing device based on a global sequence of all client computing devices;

[0017] f) transmitting, by the server computing device, model parameters based on machine learning to the selected client computing device;

[0018] g) obtaining, by the selected client computing device, global values ​​corresponding to the machine learning-based model parameters;

[0019] h) incrementally training the machine learning model based at least in part on a partition of the local dataset by the selected client computing device to obtain updated machine learning-based model parameters, the partition being selected from the local dataset according to a predefined scheme;

[0020] i) transmitting updated machine learning-based model parameters from the selected client computing device to the server computing device;

[0021] j) Repeat steps d) to i) until all predefined conditions are met.

[0022] Various preferred embodiments of the method are defined in the dependent claims.

[0023] The above object is also achieved by providing a server computing device and a client computing device configured to perform each step of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will now be described in more detail with reference to the accompanying drawings. In the drawings:

[0025] Figure 1 is a schematic diagram illustrating the concept of local learning according to the prior art.

[0026] Figure 2 is a schematic diagram illustrating the concept of centralized learning according to the prior art.

[0027] Figure 3 is a schematic diagram illustrating the concept of federated learning according to the prior art.

[0028] Figure 4 is a schematic diagram illustrating the concept of improved federated learning according to the present invention.

[0029] Figure 5 is a flowchart illustrating the main steps of an example system for improved federated learning according to the present invention.

[0030] Figure 6 is a flow chart showing the main steps of a prior art collaborative iterative learning (CIIL) method.

[0031] Figure 7 is a flow chart showing the main steps of a method performed by a client computing device according to the present invention.

[0032] Figure 8 The accuracy of various trained models over training rounds is shown. DETAILED DESCRIPTION

[0033] Example System

[0034] exist Figure 4 The concept of the improved federated learning according to the present invention is schematically illustrated in FIG. For the sake of clarity, the improved federated learning of the present invention will be referred to as “serial federation” hereinafter.

[0035] In the serial federated learning method according to the present invention, there is a single global machine learning model 40, which is subsequently trained by multiple local client computing devices 42 using the following samples: the samples are from a data set 44 stored locally in the client computing device 42; or the samples are located in a separate database (not shown) that is locally accessible or remotely accessible to the client computing device 42.

[0036] The global learning model 40 is forwarded to each client computing device 42 in a round-robin manner by the central server computing device 46 according to a predefined global sequence of all client computing devices. The server computing device 46 only acts as an arbitration entity, which is responsible for receiving the locally updated global model 40 from the client computing device 42 and forwarding the model 40 to the next client computing device 42 based on the actual arbitration rules. There is no aggregation stage in this serial federation because each training result is an intermediate state of the final global model 40.

[0037] During the initialization phase, the local data set 44 is divided into n partitions 48 , wherein the value of n is initially defined by the server computing device 46 as a global parameter common to each client computing device 42 .

[0038] The client computing devices 42 use the local data partitions 48 to dynamically train the global learning model 40 in the order specified by the arbitration server computing device 46. In this case, since no local parameters are transmitted from the client computing devices 42, the local data cannot be inferred and tracked.

[0039] In the example system, secure behavior settings and communication channel settings can be defined. Typically, the server computing device 46 is honest but curious. In this case, a semi-honest server will follow the instructions of the protocol but try to use the received data to infer private information. Most federated learning research adopts semi-honest schemes because it is in the interest of all parties to honestly abide by the protocol. Second, in cryptography, the standard approach is to first build a secure protocol against semi-honest adversaries, which can then be strengthened through zero-knowledge proofs to defend against malicious adversaries.

[0040] The communication channel between the server computing device 46 and the client computing device 42 should be secure. For each client-server connection, a secure internal virtual private network (VPN) can be established, which is also used to verify the identity of the client. At the same time, the entire communication process is carried out through a secure Hypertext Transfer Protocol Security (HTTPS).

[0041] In the serial federation method according to the present invention, the central server computing device enhances robustness. Since the server computing device is a single point of failure (SPF) in the federated system, it is recommended to run multiple instances of the server computing device in parallel. This also ensures high availability and reduces the possibility of downtime, so that in the event of a failure of one instance of the server computing device, other instances of the server computing device can still process requests. Another benefit of having multiple instances of the server computing device is that more requests can be processed simultaneously to provide a suitable response time.

[0042] Example Method

[0043] The following will refer to Figure 5 Describe in detail the serial federation method according to the present invention, Figure 5 The main steps of the method are shown in the flow chart.

[0044] In a serial federation, the communication is coordinated by the server computing device. A federated learning system is robust if it can maintain its performance, reliability, and security in the face of various challenges (e.g., noise, communication problems, adversarial attacks, and other unexpected events). The central server computing device can play an important role in making the system more robust by:

[0045] 1. Monitoring and quality assurance: A central server can monitor the performance of the global model, identify potential issues, and take corrective action as needed. This can include detecting and handling low-quality updates, malicious actors, or even triggering model retraining when necessary.

[0046] 2. Data validation and preprocessing: The central server can perform data validation and preprocessing tasks to ensure that the data used for model updates is accurate, clean, and consistent, which helps maintain the quality of the global model.

[0047] 3. Adaptive learning: The central server can adjust the learning process based on the observed global model and the performance of participating devices. This may involve adjusting the learning rate, update frequency, or even changing the model architecture to better suit the problem at hand.

[0048] 4. Security and privacy enforcement: The central server can implement various security and privacy measures (e.g., secure aggregation and differential privacy) to protect the sensitive information of participating devices and maintain the overall trustworthiness of the system.

[0049] 5. Resource allocation and scheduling: The central server can efficiently allocate and schedule resources among participating devices, ensuring that the system remains balanced and can handle fluctuations in device availability or communication limitations.

[0050] By managing these aspects, a central server can significantly improve the robustness of a federated learning system or any distributed computing system.

[0051] In a preferred embodiment of the method of the present invention, the server computing device can change the state information of the model. For example, the server can act as a regularizer in the learning process and can skip suspicious model updates. In this embodiment, the server computing device is not only responsible for routing the updated model from one client computing device to another client computing device, but also responsible for regulating the learning process.

[0052] In a first step 500, the server computing device defines the following parameters: the number of partitions (n); the minimum size of a partition (s min ); and the client's random sequence.

[0053] The parameter number of partitions defines the number of partitions n into which the local data set of the client computing device should be divided.

[0054] The minimum partition size parameter is the minimum number of data points in a partition. This parameter is used to make each training process more balanced in data size. min The definition of s means that the size of all datasets should be equal to or greater than the minimum size s min Multiply by the number of data partitions n. The use of a minimum size for data partitions supports the efficiency and accuracy of model training. The impact of data partitioning on performance is significant because it amplifies the effect of randomizing the order of client updates. In the non-IID case, setting a large number of partitions n ensures that during each training round, a different dataset representing each mode is used to update the model. This allows the model to converge faster, thereby reducing computational cost.

[0055] The random sequence of clients is a vector of indices that determine the order of client updates. Preferably, the server computing device defines a global sequence of clients based on a random or pseudo-random selection of clients, which guarantees that the client computing device never has information about previous client computing devices within the training sequence. The effect of randomization of the update order of client computing devices is similar to the randomization of data points before model training in a central learning scheme. Therefore, the randomization of client devices is such that:

[0056] 1. Avoiding order effects: Randomizing the order of clients helps avoid order effects that can occur when processing client data in a specific order. If the order of clients is not randomized, the model may learn certain patterns or trends in the partitions that are related to the order in which they are presented. (This problem is more prevalent in the non-IID case.)

[0057] 2. Avoid overfitting: Randomization can also help avoid overfitting, which occurs when the model becomes too adapted to the training data and does not generalize well to new data. By randomizing the order of clients, the model is forced to learn a more general representation of the data.

[0058] 3. Enhanced performance: Randomization can also improve the performance of the model by ensuring that the model is exposed to more diverse data during training. This can help the model learn to recognize and respond to patterns that the model may not encounter if the client is not randomized.

[0059] 4. Improve convergence speed: Randomization can help the model converge faster by ensuring that the model encounters a different set of samples in each epoch. This exposure helps the model learn the underlying patterns in the data more effectively, thereby preventing the model from falling into the dilemma of local optimality or overfitting to a specific sample sequence.

[0060] In step 501 , the server computing device transmits initial partition parameters to each client computing device through a secure communication channel and instructs the client computing device to create a partition.

[0061] Preferably, at the initialization of the training process, the minimum partition size s is sent from the server computing device to all client computing devices via broadcast. min and the number n.

[0062] In step 502, each client computing device obtains global values ​​corresponding to the set of initial partitioning parameters.

[0063] In step 503, each client computing device divides its local data set into a specific number n of data partitions defined by the partition parameters, each partition containing at least s min data points.

[0064] Both step 503 of partitioning the data and step 500 of randomizing the client update order have a significant impact on the convergence speed of federated learning. The main factor of computational cost in federated learning is the convergence speed which is proportional to the amount of computational resources used.

[0065] In step 504, the server computing device generates initial untrained machine learning-based model parameters. These parameters are used as an initial model for training on the selected partition of the client computing device.

[0066] In step 505 , the server computing device selects a client computing device based on the random sequence of parameter clients determined in step 500 and predefined quality control criteria.

[0067] Quality control criteria can be defined by a server computing device, which can also be the data owner, and it can use its own data to track the training process and intervene based on predefined thresholds when there is a significant drop in accuracy compared to the previous model. In this case, the server computing device can send an earlier, better performing model to the next client computing device for further training.

[0068] In step 506, the server computing device transmits the machine learning based model parameters to the selected client computing device. The first selected client computing device always receives the untrained initial model parameters.

[0069] In step 507 , the selected client computing device obtains a global set of machine learning-based model parameters.

[0070] In step 508, the selected client computing device incrementally trains the machine learning model based at least in part on a partition of the local data set to obtain updated machine learning-based model parameters.

[0071] The incremental training step 508 solves the problem of continuous private updates to the federated learning model in the presence of continuously increasing data. By adopting data partitioning in the serial federated structure, only another round of model training is required when new data is added, which reduces the complexity of the method.

[0072] In step 508, a partition is selected from the local data set according to a predefined scheme, preferably the next partition is randomly selected from the set of local partitions without replacement, and the process is restarted when there are no more partitions.

[0073] To train the model, the selected client computing device uses a local machine learning model. The type of local learning model used strictly depends on the application under consideration. For example, in a model with the right model update scheme, the model can be a convolutional neural network (CNN), a feedforward neural network (FNN), a long short-term memory network (LSTM), K-means or a naive Bayes classifier. Due to the incremental training step, in general, any online learning algorithm can be used to train the model.

[0074] In the example method, the time set for each client training can also be fixed and supervised by the server computing device. Depending on the size of the data partition, when the data set is small, a shorter waiting time (e.g., 1 minute) can be set, and when the data set is large, a longer waiting time (e.g., 5 minutes) can be set. In this case, even if one or more of the multiple client computing devices exit the system, it will not cause training failure.

[0075] Example Algorithm 1 shows the specific steps of the training algorithm executed by the server computing device.

[0076]

[0077] Example Algorithm 2 shows the specific steps of the training algorithm executed by the client computing device.

[0078]

[0079] In step 509 , the client computing device transmits updated machine learning-based model parameters to the server computing device via a secure communication channel.

[0080] In step 510 , the server computing device receives the updated machine learning-based model from the selected client computing device and repeats the iteration until a predefined condition is met.

[0081] For example, the predefined condition may be satisfied when the model reaches a state where further training does not significantly improve its accuracy (e.g., the change between the model trained in the previous round and the model trained in the current round is less than a predefined percentage or difference). In another example, the predefined condition may be satisfied, for example, when the number of training rounds has reached or exceeded a predefined threshold number (e.g., defined by an administrator of the server computing device).

[0082] The following will refer to Figure 6 and Figure 7 The main differences between the exemplary method of cyclic learning (i.e., the CIIL scheme) and the exemplary embodiment of the serial federation method according to the present invention are described in detail according to the flowchart shown in FIG. Figure 6 and Figure 7 The main steps of the method executed by the client computing device are respectively shown.

[0083] Both methods involve the steps of obtaining a set of parameters for a machine learning based model (i.e., steps 600 and 720, respectively), and incrementally training the model with local data (i.e., steps 610 and 740, respectively). The CIIL method focuses on training the model with the entire local dataset in step 610 and then passing the updated parameters to the next client in step 620, while the serial federated method divides the local dataset into multiple partitions in step 710 and uses one partition at a time to train the learning model in step 740.

[0084] In the CIIL method, in step 620, the updated model parameters are transmitted to the next client computing device, while in the serial federation method, in step 750, the updated model parameters are sent to the server computing device.

[0085] The serial federation method involves an additional setup phase to determine the number of partitions n and the minimum number of data points per partition s in step 700. min , unlike the CIIL method which does not require this step.

[0086] Compared with the machine learning process in the prior art, the serial federation method according to the present invention has the following advantages:

[0087] - It works accurately in extreme non-IID scenarios, thus generating new capabilities for federated scenarios.

[0088] - Due to data partitioning and random sequence of clients, it significantly reduces the computation cost on the client side.

[0089] -It can adopt any kind of online machine learning method.

[0090] -It addresses the challenge of continuously updating machine learning models with new data.

[0091] -The server provides robustness to the federated training process.

[0092] Example Experiment

[0093] To evaluate the capabilities of the method, multiple training settings were tested on the CIFAR10 dataset with varying degrees of non-IID on a standard DenseNet neural network that can achieve up to 90% accuracy when trained centrally (Huang et al., Densely Connected Convolutional Networks. https: / / arxiv.org / abs / 1608.06993). Convergence was measured to quantify the computational cost in the experiments. The main goal of these experiments was to show that serial federation performs as well as other protocols in a general setting, and has significantly better performance when the dataset is non-IID, providing increased accuracy and reduced convergence time for training on the client, resulting in a significant reduction in computational cost on the client side.

[0094] Another objective is to show that in the case of non-IID edges, when each client computing device has samples with only one type of label, the serial federation method according to the present invention still converges and provides an accurate model, while other algorithms will never converge or will converge very slowly. These experiments also aim to provide quantitative proof of the impact of client sequence randomization and data partitioning on convergence, i.e., computational efficiency.

[0095] Experimental methods

[0096] In these experiments, the following methods were compared and analyzed: a cyclic institutional learning (CIIL) scheme without client-side sequence randomization (Chang et al., "Distributed deep learning networks among institutions for medical imaging", Journal of the American Medical Informatics Association, Vol. 25, No. 8, August 2018, pp. 945-954); a cyclic institutional learning scheme (CIIL) with client-side sequence randomization; an adaptive FedAvg protocol called FedYogi; and a serial federation method according to the present invention. For each protocol, the same model with the same optimizer and loss criterion was used. The neural network included a DenseNet (Huang et al., Densely Connected Convolutional Networks. https: / / arxiv.org / abs / 1608.06993) with a stochastic gradient descent optimizer with a learning rate of 0.01, a momentum of 0.9, and a weight decay of 5e-4. For each test, a total of 100 epochs were performed, which is defined as the product of the number of local epochs and the number of training rounds. The tests were performed using 6 client computing devices in order to simulate the real-world scenario of cross-silo federated learning.

[0097] To represent the capabilities of the most widely used federated learning method (FedAvg), an advanced approach was chosen that aims to address known issues with its unfavorable convergence behavior. Based on the work of Reddi et al., “Adaptive Federated Optimization” ( https: / / arxiv.org / abs / 2003.00295v5 ), among the three proposed adaptive federated optimizations (FedYogi, FedAdam, FedAdagrad), the FedYogi scheme converges the fastest and has the highest accuracy on cifar10, so it is used as the benchmark. For these experiments, the Flower1.4.0 federated learning framework adaptive federated optimization implementation was used. For all tests, the following FedYogi-specific parameters were used: server-side learning rate (eta): 1e-1; Beta2 momentum: 0.9; adaptability (tau): 1e-9.

[0098] An example implementation of the serial federation functionality of the server computing device and the client computing device is based on example Algorithms 1 and 2. For each global training round, a random sequence of client computing devices is uniformly sampled from the permutation {1,2,3,4,5,6} (a global sequence can be generated immediately by concatenating the permutations together). The algorithm persists after 100 global training rounds (predefined conditions). For all tests, the following serial federation specific parameters are further used: the number of partitions is selected as n=2; the minimum number of data points in a partition is s min =100; uniformly sample the client update order from the arrangement {1,2,3,4,5,6}.

[0099] For all three methods, the following common parameters are used: Client-side learning rate (eta1) 1e-2. For all tests, the local ML batch size is fixed to 128 and the local epoch is set to 1 to maximize accuracy at the expense of increasing total network usage. All client computing devices participate in each training round.

[0100] Convergence is achieved when the global model reaches a state where further training cannot significantly improve its accuracy. The speed of convergence is inversely proportional to the minimum number of training rounds required to achieve convergence. Regardless of infrastructure, federated learning methods can have a huge impact on the speed of convergence and, therefore, the computational cost.

[0101] When the training reaches convergence, the number of training rounds that varies with accuracy can be approximated by a simple ramp function defined as follows:

[0102]

[0103] In this case, the fitting parameter T is proportional to the minimum number of training rounds required to achieve convergence. The parameter A is proportional to the accuracy of the model after convergence, while A0 represents the initial accuracy and is not fitted. The fitting process can be performed using analytical solutions through least squares, Monte Carlo maximum likelihood (ML), or Bayesian methods.

[0104] The total computational cost is a function of the number of rounds (T) and the number of computers (n) involved in each learning step. c ). The total computation factor (TF) is a simple integer representing the number of model updates required to converge.

[0105] TF=n c T

[0106] Experimental data

[0107] The CIFAR10 dataset is randomly split between the client computing devices, and depending on the test, a label shift is performed during the split. In the case of non-IID testing, the dataset is created in two ways. In the first case, two groups of client computing devices are defined so that half of the client computing devices in the plurality of client computing devices receive more data with labels from 0 to 4, while the other group receives data with labels from 5 to 9. The second case is the most extreme case possible, in which 10 client computing devices are used, each of which has only one label. The numbers are numerical representations of the image labels. The following non-IID datasets were tested for less extreme cases: ratios between labels of 30% to 70%, 10% to 90%, and 1% to 99%.

[0108] Experimental Results

[0109] Table 1 below summarizes the accuracy values ​​and total computational coefficients of various schemes under different label ratios obtained from the tests.

[0110]

[0111]

[0112] After each training round, the accuracy of the trained model was evaluated and both the accuracy and the number of computation rounds were determined. Figure 8 As shown, this is numerically represented by fitting a ramp function of accuracy versus training rounds.

[0113] Experimental results show that in the case of IID data, the serial federated method according to the present invention has the same performance as the cyclic mechanism learning with or without randomization. However, as the degree of non-IID increases, the serial federated method begins to significantly outperform other protocols in both accuracy and convergence speed.

[0114] Each additional feature added to the recurrent mechanism learning scheme (i.e., client randomization, partitioning) has been shown to produce significant technical improvements over the original approach, including faster convergence (i.e., reduced computational cost) and improved accuracy. Introducing update order randomization alone leads to faster convergence and higher accuracy in both IID and moderately non-IID scenarios. Even in the non-IID case, partitioning the client data can further improve performance and reduce computational cost. Notably, the serial federation approach according to the present invention performs well even in the extreme non-IID case.

[0115] Experimental results show that the serial federation method according to the present invention has better performance on non-IID data sets compared to the adaptive FedAvg algorithm of the prior art. Compared with other cyclic federation schemes, the present scheme is more robust. Experiments further show that the serial federation according to the present invention can achieve higher accuracy with a smaller total computation coefficient. The serial federation of the present invention provides similar bandwidth usage as CIIL, but uses less computing resources due to faster convergence.

[0116] Additional disclosure

[0117] The technology discussed herein relates to server and client computing devices, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The information exchanged within computer-based systems exhibits their inherent flexibility, which allows for a variety of configurations, task divisions, and functional combinations between components. Either a single device may be used to perform each process, or multiple devices may be used to work in concert to perform each process. Databases and applications may be implemented on one system, or may be distributed across multiple systems.

[0118] The present subject matter has been discussed in depth with respect to various specific example embodiments for purposes of explanation and not limitation. Those skilled in the art can easily generate changes, modifications, and equivalents to these embodiments after understanding the subject matter. Therefore, the present disclosure does not exclude any modifications, modifications, or additions that are obvious to those of ordinary skill in the art. For example, a feature described or explained as part of one embodiment can be used with another embodiment to produce a new embodiment. It is the intent of the present disclosure to cover such changes, modifications, and equivalents.

Claims

1. A client computing device, comprising: Processing unit; as well as A non-volatile computer-readable storage medium containing a local data set and instructions, which, when executed by the processing unit, instruct the client computing device to perform a series of functions, including: a) obtaining a global value corresponding to a set of parameters associated with a machine learning based model; b) incrementally training the machine learning model based at least in part on the partitions of the local data set to obtain updated machine learning-based model parameters; and c) transmitting the updated machine learning-based model parameters to a server computing device.

2. The client computing device of claim 1, wherein: Generating the partition of the local data set includes: a) obtaining a global value corresponding to a set of parameters associated with: the number of partitions to generate; and the minimum number of data points in a partition; and b) partitioning the local data set into a plurality of (n) partitions defined by the parameter said number of partitions, wherein a partition has at least s min data points, where s min The value of is defined by the parameter Minimum number of data points in the partition.

3. A server computing device, comprising: Processing unit; as well as A non-volatile computer-readable storage medium containing instructions that, when executed by the processing unit, instruct the server computing device to perform a series of functions, the functions comprising: a) selecting a client computing device based on a global sequence of all client computing devices and predefined quality control criteria; b) transmitting machine learning based model parameters to the selected client computing device; and c) receiving an updated machine learning-based model from the selected client computing device.

4. The server computing device according to claim 3, wherein: The global sequence of the client computing device is generated by a calculation method, the calculation method comprising: a random mathematical algorithm; a random statistical algorithm; a pseudo-random mathematical algorithm; or a pseudo-random statistical algorithm.

5. The server computing device of claim 3, wherein the server computing device is further configured to: a) generating a set of initial parameters associated with a machine learning based model; b) Generation parameters: the number of partitions; c) Generation parameters: minimum number of data points in a partition; and d) Generation parameters: random sequence of the client computing device.

6. A computer-implemented method for computationally efficient machine learning, comprising: a) generating a set of initial model parameters by a server computing device; b) generating, by the server computing device, a set of partitioning parameters, the set of partitioning parameters comprising at least parameters: the number of partitions and the minimum number of data points in a partition; c) transmitting, by the server computing device, the set of initial model parameters and the set of partition parameters to each of a plurality of client computing devices; d) obtaining, by each client computing device, global values ​​corresponding to the set of initial parameters from the server computing device; e) selecting, by the server computing device, a client computing device based on a global sequence of all client computing devices; f) transmitting, by the server computing device, model parameters based on machine learning to the selected client computing device; g) obtaining, by the selected client computing device, a global value corresponding to the machine learning-based model parameter; h) incrementally training the machine learning model based at least in part on a partition of the local dataset by the selected client computing device to obtain updated machine learning-based model parameters, wherein the partition is selected from the local dataset according to a predefined scheme; i) transmitting, by the selected client computing device, the updated machine learning-based model parameters to the server computing device; j) Repeat steps d) to i) until all predefined conditions are met.

Images