An efficient and robust federated learning algorithm for heterogeneous data

Abstract

The present invention provides an efficient and robust federated learning algorithm for heterogeneous data, wherein the method is used for a federated learning architecture of a parameter server, including: S1) defining a local loss function f of a local model; i (w) and global loss function f(w); S2), using small batch stochastic gradient descent method to update local model; S3), according to the client's non-independent and identically distributed data, construct corresponding personalized correction factor #imgabs0# for each client i; S4), using personalized correction factor #imgabs1# to update the local model of each client i. Compared with the prior art, the present invention can improve the prediction accuracy; it can also better resist general malicious nodes and find hidden lazy nodes; the present invention can solve the technical problems of poor model convergence stability, low time efficiency, and poor robustness when facing non-independent and identically distributed data.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to an efficient and robust federated learning algorithm for heterogeneous data. Background Art

[0002] Federated learning is a distributed machine learning based on parallel computing. Its structure consists of a server and several clients. Each client uses its own local data to train a model and only sends the trained model back to the server. The server aggregates the received models to obtain a global model. In this process, the client's local data will not be shared.

[0003] In federated learning, how to utilize non-independent and identically distributed training data (Non-IIDData) between different clients has always been one of the most important challenges in this field. In order to solve the problems caused by non-independent and identically distributed training data, researchers have proposed many targeted improved algorithms, such as FedProx, FoolsGold, Scaffold, and STEM. Based on the traditional federated averaging algorithm FedAvg, these algorithms use global model parameters or global update gradients at different stages of the training process to correct the local model, pointing the local model to the global optimum rather than the local optimum, to alleviate the degradation of model performance caused by Non-IID data. However, they still have the following disadvantages and shortcomings:

[0004] 1. Poor model convergence stability: Compared with the classic federated averaging algorithm FedAvg, the improved federated learning algorithm for Non-IID data shows stronger instability during the training process. In some specific situations, its performance is not as good as the classic FedAvg algorithm, and even the model convergence fails.

[0005] 2. Low time efficiency: The above-mentioned improved federated learning algorithms for Non-IID data often use "training round-accuracy" as the key evaluation indicator, which mainly compares the model accuracy of the algorithm after the same number of training rounds. However, "training round-accuracy" cannot accurately reflect the time efficiency of the algorithm because different federated learning algorithms often have different local updates or global aggregation processes, which leads to different computational complexity and completion time of different algorithms in a single round. For example, the STEM algorithm is significantly better than other algorithms in the "training round-accuracy" indicator, but because its computational complexity in the local update step is twice that of the general algorithm, although the algorithm can complete model convergence with fewer training rounds, its training time for each round will also be significantly higher than other similar algorithms. Therefore, the actual time efficiency of the STEM algorithm is greatly reduced.

[0006] 3. Poor robustness: Most central parameter servers used in federated learning algorithms for non-IID data often require the clients participating in the training to upload some additional auxiliary variables, such as the number of updates to the local model and the size of the local data set, in addition to the necessary local models or local model gradients. This parameter server's reliance on client information not only increases the computational and communication overhead of the central parameter server and the client, but also adds additional risks. The client can seriously affect the convergence of the global model simply by reporting false information such as the amount of local data or the size of local gradients. Therefore, these federated learning algorithms lack basic robustness in the face of some simple and easy-to-implement malicious attacks. Although algorithms like FoolsGold can resist these malicious attacks to a certain extent, they cannot identify another type of malicious users, namely lazy users who do not contribute to federated learning and only contribute stale gradients. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an efficient and robust federated learning algorithm for heterogeneous data, which can solve the technical problems of poor model convergence stability, low time efficiency and poor robustness when facing non-independent and identically distributed data.

[0008] The technical solution of the present invention is: an efficient and robust federated learning algorithm for heterogeneous data, wherein the method is used for a federated learning architecture of a parameter server, including N clients, defined as a set and a central parameter server for global aggregation;

[0009] And each client Have heterogeneous computing and communication resources, and have corresponding private local data sets The local dataset Contains D i Data samples

[0010] The federated learning method comprises the following steps:

[0011] S1), define the local loss function f of the local model i (w) and the global loss function f(w) are:

[0012]

[0013]

[0014] Where D i Indicates the number of samples in the local dataset; x i,j represents the jth data sample of the i-th client; w is the local model parameter; l(w,x i,j) is the sample data x i,j The loss function of

[0015] S2) Update the local model using a small batch stochastic gradient descent method, and upload the accumulated gradient of the local model update to the central parameter server for aggregation. The central parameter server updates the global model based on the obtained global gradient;

[0016] S3) According to the client's non-independent and identically distributed data, a corresponding personalized correction factor is constructed for each client i.

[0017] S4) Using personalized correction factors Update the local model of each client i.

[0018] As a preferred embodiment, in step S1), the local data set of each client i is not independent and identically distributed.

[0019] Preferably, in step S2), the small batch stochastic gradient descent method is used to update the local model as follows:

[0020] Each client i will get the local dataset Randomly extract a small batch of data from Used to perform the kth local model update in the tth training round;

[0021] Each training round includes multiple local model updates. The loss function of client i when performing local model updates is Defined as:

[0022]

[0023] Where s is the size of the training batch.

[0024] Preferably, in step S2), small batch data is used The rules for updating the local model are:

[0025]

[0026] In the formula, represents the model parameters after k+1 local updates of client t in the tth federated learning training round, represents the model parameters after k local updates of client t, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of From local dataset is a small batch of data randomly selected from ; K represents the number of times the local model is updated in each training.

[0027] As a preferred embodiment, in step S2), after completing k rounds of local model updates, each client i will update the cumulative gradient of this round of local model updates The central parameter server aggregates the accumulated gradients of each client according to the weight of the client. Perform weighted summation to obtain the global gradient Δ of the t+1th federated learning training round t+1 ,Right now:

[0028]

[0029]

[0030] Where η l is the learning rate of the local model, represents the initial model parameters of client t at the beginning of the tth federated learning training round, represents the local model parameters of client i after K updates in the tth federated learning training round, v i Represents the weight of client i among all clients.

[0031] Preferably, in step S2), the central parameter server is used to obtain the global gradient Δ t+1 After that, through the global gradient Δ t+1 The global model will be updated and the latest global model will be synchronized to all clients to start a new round of federated model training. The global model update formula is as follows:

[0032] w t+1 =w t ―η g Δ t+1

[0033] Among them, η g is the global learning rate, w t+1 represents the local model parameters of the t+1th federated learning training round, w t Represents the local model parameters of the tth federated learning training round.

[0034] Preferably, in step S3), the personalized correction factor of the client i for:

[0035]

[0036] In the formula, is the vector transpose symbol;

[0037] By calculating the cumulative gradient uploaded by client i The amplitude And the cumulative gradient and the global model gradient Δ t+1 The cosine similarity of is used to determine the value of the correction factor for each client i.

[0038] As a preferred embodiment, in step S3), the central parameter server uploads the accumulated gradient of the local model Use personalized correction factors when performing weighted summation Get the global gradient Δ t+1 ;Right now:

[0039]

[0040] Where K represents the number of times the local model is updated, η l is the learning rate of the local model.

[0041] As a preference, in step S4), each client i uses a personalized correction factor The rules for updating the local model are as follows:

[0042]

[0043] In the formula, represents the model parameters after k+1 local updates of client t in the tth federated learning training round, represents the model parameters after k local updates of client i in the tth federated learning training round, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of ,γ∈(0,1) is used to limit the maximum value of the correction factor, Δ t Indicates that in the tth federated learning training round, the central parameter server uses the personalized correction factor The calculated global gradient.

[0044] Preferably, in step S4), the personalized correction factor The following conditions are met to find and remove nodes that do not contribute during training, namely:

[0045]

[0046] In the formula, κ is the preset threshold.

[0047] The beneficial effects of the present invention are:

[0048] 1. Compared with the prior art, the present invention can improve the prediction accuracy; it can also better resist general malicious nodes and find hidden lazy nodes;

[0049] 2. The present invention can solve the technical problems of poor model convergence stability, low time efficiency, and poor robustness when facing non-independent and identically distributed data;

[0050] 3. The present invention designs a corresponding personalized correction factor for each client to achieve personalized correction of each client's local update, thereby avoiding the problem of over-correction or under-correction and improving the robustness of the algorithm;

[0051] 4. The present invention presets a threshold value so as to find and eliminate nodes that do not contribute during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0053] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0054] like Figure 1 As shown, this embodiment provides an efficient and robust federated learning algorithm for heterogeneous data, wherein the method is used for a federated learning architecture of a parameter server, including N clients, defined as a set and a central parameter server for global aggregation;

[0055] And each client Have heterogeneous computing and communication resources, and have corresponding private local data sets The local dataset Contains D i Data samples And the local dataset of each client i is not independent and identically distributed.

[0056] As a preferred embodiment of this invention, the federated learning method includes the following steps:

[0057] S1), define the local loss function f of the local model i (w) and the global loss function f(w) are:

[0058]

[0059]

[0060] Where D i Indicates the number of samples in the local dataset; x i,jrepresents the jth data sample of the i-th client; w is the local model parameter; l(w,x i,j ) is the sample data x i,j The loss function of

[0061] S2) Update the local model using a small batch stochastic gradient descent method, and upload the accumulated gradient of the local model update to the central parameter server for aggregation. The central parameter server updates the global model based on the obtained global gradient;

[0062] S3) According to the client's non-independent and identically distributed data, a corresponding personalized correction factor is constructed for each client i.

[0063] S4) Using personalized correction factors Update the local model of each client i.

[0064] As a preferred embodiment of this invention, in step S2), the small batch stochastic gradient descent method is used to update the local model as follows:

[0065] Each client i will get the local dataset Randomly extract a small batch of data from Used to perform the kth local model update in the tth training round;

[0066] Each training round includes multiple local model updates. The loss function of client i when performing local model updates is Defined as:

[0067]

[0068] Where s is the size of the training batch.

[0069] As a preferred embodiment of this invention, in step S2), small batch data is used The rules for updating the local model are:

[0070]

[0071] In the formula, represents the model parameters after k+1 local updates of client i in the tth federated learning training round, represents the model parameters after k local updates of client i, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of From local dataset is a small batch of data randomly selected from ; K represents the number of times the local model is updated in each training.

[0072] As a preferred embodiment of this invention, in step S2), after completing k rounds of local model updates, each client i will update the cumulative gradient of the local model update in this round The central parameter server aggregates the accumulated gradients of each client according to the weight of the client. Perform weighted summation to obtain the global gradient Δ of the t+1th federated learning training round t+1 ,Right now:

[0073]

[0074]

[0075] Where η l is the learning rate of the local model, represents the initial model parameters of client i at the beginning of the tth federated learning training round, represents the local model parameters of client i after K updates in the tth federated learning training round, v i Represents the weight of client i among all clients.

[0076] As a preferred embodiment of this invention, in step S2), the central parameter server is used to obtain the global gradient Δ t+1 After that, through the global gradient Δ t+1 The global model will be updated and the latest global model will be synchronized to all clients to start a new round of federated model training. The global model update formula is as follows:

[0077] w t+1 =w t ―η g Δ t+1

[0078] Among them, η g is the global learning rate, w t+1 represents the local model parameters of the t+1th federated learning training round, w t Represents the local model parameters of the tth federated learning training round.

[0079] As a preferred embodiment of this invention, in step S3), the personalized correction factor of the client i for:

[0080]

[0081] In the formula, is the vector transpose symbol;

[0082] By calculating the cumulative gradient uploaded by client i The amplitude And the cumulative gradient and the global model gradient Δ t+1 The cosine similarity of is used to determine the value of the correction factor for each client i.

[0083] As a preferred embodiment of this invention, in step S3), the central parameter server processes the accumulated gradients uploaded by the local model. Use personalized correction factors when performing weighted summation Get the global gradient Δ t+1 ;Right now:

[0084]

[0085] Where K represents the number of times the local model is updated, η l is the learning rate of the local model.

[0086] As a preferred embodiment of this invention, in step S4), each client i uses a personalized correction factor The rules for updating the local model are as follows:

[0087]

[0088] In the formula, represents the model parameters after k+1 local updates of client i in the tth federated learning training round, represents the model parameters after k local updates of client i in the tth federated learning training round, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of ,γ∈(0,1) is used to limit the maximum value of the correction factor, Δ t Indicates that in the tth federated learning training round, the central parameter server uses the personalized correction factor The calculated global gradient.

[0089] As a preferred embodiment of this invention, in step S4), the personalized correction factor The following conditions are met to find and remove nodes that do not contribute during training, namely:

[0090]

[0091] In the formula, κ is the preset threshold.

[0092] Among them, Table 1 shows the results of learning and testing the method of this embodiment with the existing learning algorithms FedAvg, fedprox, foolsgold, scaffold, and stem on the data sets MNIST, FMNIST, SVHN, CIFAR-10, and adult. The method described in this embodiment can improve the final test accuracy by 0.27% to 8.19% on the MNIST and FMNIST data sets compared with the benchmark algorithm, and improve the final test accuracy by 0.67% to 57.1% on the SVHN, CIFAR-10 data sets and adult data sets. Compared with other classic federated learning algorithms, this embodiment can better resist general malicious nodes and find hidden lazy nodes.

[0093] Table 1 Final model accuracy results of the method in this embodiment and other benchmark algorithms in multiple data sets and scenarios

[0094]

[0095] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. An efficient and robust federated learning algorithm for heterogeneous data, characterized by: The federated learning algorithm described is used in the federated learning architecture of the parameter server, including N clients, defined as the set and a central parameter server for global aggregation; And each client Have heterogeneous computing and communication resources, and have corresponding private local data sets The local dataset Contains D i Data samples The federated learning method comprises the following steps: S1), define the local loss function f of the local model i (w) and the global loss function f(w) are: Where D i Indicates the number of samples in the local dataset; x i,j represents the jth data sample of the i-th client; w is the local model parameter; l(w,x i,j ) is the sample data x i,j The loss function of S2) Use the small batch stochastic gradient descent method to update the local model, and upload the accumulated gradient of the local model update to the central parameter server for aggregation. The central parameter server updates the global model according to the obtained global gradient; S3) According to the client's non-independent and identically distributed data, a corresponding personalized correction factor is constructed for each client i. Among them, the personalized correction factor of client i for: Where T is the vector transpose symbol; By calculating the cumulative gradient uploaded by client i The amplitude And the cumulative gradient and the global model gradient Δ t+1 The cosine similarity of is used to determine the value of the correction factor of each client i; The central parameter server uses the personalized correction factor Accumulated gradients uploaded to the local model Perform weighted summation to obtain the global gradient Δ t+1 ;Right now: Where K represents the number of times the local model is updated, η l is the learning rate of the local model; S4) Using personalized correction factors Update the local model of each client i; Each client i uses a personalized correction factor The rules for updating the local model are as follows: In the formula, represents the model parameters after k+1 local updates of client i in the tth federated learning training round, represents the model parameters after k local updates of client i in the tth federated learning training round, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of ,γ∈(0,1) is used to limit the maximum value of the correction factor, Δ t Indicates that in the tth federated learning training round, the central parameter server uses the personalized correction factor The calculated global gradient; The personalized correction factor The following conditions are met to find and remove nodes that do not contribute during training, namely: In the formula, κ is the preset threshold.

2. The efficient and robust federated learning algorithm for heterogeneous data according to claim 1, characterized in that: In step S1), the local data set of each client i is not independent and identically distributed.

3. The efficient and robust federated learning algorithm for heterogeneous data according to claim 1, characterized in that: In step S2), the small batch stochastic gradient descent method is used to update the local model as follows: Each client i will get the local dataset Randomly extract a small batch of data from Used to perform the kth local model update in the tth training round; Each training round includes multiple local model updates. The loss function of client i when performing local model updates is Defined as: Where s is the size of the training batch.

4. The efficient and robust federated learning algorithm for heterogeneous data according to claim 3, characterized in that: In step S2), use small batch data The rules for updating the local model are: In the formula, represents the model parameters after k+1 local updates of client i in the tth federated learning training round, represents the model parameters of client i after k local updates, η l is the learning rate of the local model, Represents the loss function when the local model is updated The gradient of From local dataset is a small batch of data randomly selected from ; K represents the number of times the local model is updated in each training.

5. The efficient and robust federated learning algorithm for heterogeneous data according to claim 4, characterized in that: In step S2), after completing k rounds of local model updates, each client i will update the cumulative gradient of this round of local model updates The central parameter server aggregates the accumulated gradients of each client according to the weight of the client. Perform weighted summation to obtain the global gradient Δ of the t+1th federated learning training round t+1 ,Right now: Where η l is the learning rate of the local model, represents the initial model parameters of client i at the beginning of the tth federated learning training round, represents the local model parameters of client i after K updates in the tth federated learning training round, v i Represents the weight of client i among all clients.

6. The efficient and robust federated learning algorithm for heterogeneous data according to claim 5, characterized in that: In step S2), the central parameter server is used to obtain the global gradient Δ t+1 After that, through the global gradient Δ t+1 The global model will be updated and the latest global model will be synchronized to all clients to start a new round of federated model training. The global model update formula is as follows: w t+1 =w t -or g D t+1 Among them, η g is the global learning rate, w t+1 represents the local model parameters of the t+1th federated learning training round, w t Represents the local model parameters of the tth federated learning training round.

Images