A personalized federated learning method for efficient communication on non-identically distributed data

Abstract

The application realizes a personalized federated learning method for efficient communication on non-identically distributed data. The major challenge of federated learning is the non-identically distributed nature of client data, i.e., statistical heterogeneity, and the limited communication resources between clients and cloud servers. Statistical heterogeneity and communication resource constraints pose obstacles to the implementation of federated learning. To address these challenges, the application provides a personalized federated learning method for efficient communication on non-identically distributed data, enabling each client to train a personalized model while significantly reducing communication costs. Specifically, each client compresses its gradient before uploading it and mitigates the adverse effects of gradient compression by saving the gradient error before and after compression and then adding the error to the gradient to be compressed in the next round. By uploading only the compressed sparse gradient, communication costs can be significantly reduced. At the cloud server end, the received sparse gradient is restored to the model, and the similarity of these models is aggregated to facilitate collaboration between clients. After receiving the global model, the client performs local updates to obtain a personalized model. The application not only obtains a model with high accuracy, but also significantly reduces the communication cost of the client.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication and training prediction models in federated learning. BACKGROUND

[0002] Deep neural networks have been widely applied in various fields, such as disease diagnosis, image reconstruction, earthquake detection, self-driving cars, etc. Generally speaking, training a good deep neural network requires a large amount of data. However, the available data on a single client is usually very limited, which will result in poor performance of the deep neural network trained independently by the client locally. Centralized training has been used as a method to improve the accuracy of the trained model, which requires all participating clients to upload their local data to a central server to train a shared model. However, the centralized training method is prone to data privacy leakage.

[0003] Federated learning is a distributed machine learning paradigm. Federated learning solves the problem of data privacy by allowing multiple clients participating in federated learning to collaboratively train a globally shared model without transmitting the local data of the clients to a central server. The basic process of traditional federated learning is that the client downloads the global model from the cloud server, and after local training, the client uploads the gradient or model weight to the server for model aggregation. Repeat the process until the model reaches the required accuracy.

[0004] The goal of federated learning is to achieve a high-performance global model. Traditional federated learning assumes that the data on the client's local is independently and identically distributed. However, in many real-world scenarios, the data on the client's local is not independently and identically distributed, which will result in statistical heterogeneity. Statistical heterogeneity refers to the significant difference in the distribution of local data between clients, which greatly limits the performance of federated learning, making it challenging to train a high-performance global model.

[0005] In addition, in federated learning, the client needs to transmit a large amount of model parameters to the cloud server, and the communication resources of the client participating in federated learning are very limited. Therefore, how to achieve efficient communication is also a key challenge in the practical application of federated learning. SUMMARY

[0006] To solve the problem of limited communication resources of the client in federated learning and the poor performance of the model trained when the data on the client's local is not independently and identically distributed, the present application proposes a personalized federated learning method with efficient communication, which not only effectively improves the performance of the model by personalizing the client's model, but also greatly reduces the communication cost of the training process.

[0007] The technical scheme adopted by the present application is that a group of recovery models and a group of global models are maintained on a server, and the number of recovery models and global models is the same as the number of clients participating in federated learning. A local model, a threshold, an error and a gradient are maintained on a client. The method of the present application iteratively trains T rounds, wherein the training of each round includes the following seven steps:

[0008] First step: the client adds the gradient of the last round and the error of the last round to obtain the accumulated gradient to be compressed this round;

[0009] Second step: the client compares the L2 norm of the accumulated gradient of this round with the threshold maintained by the client. If the L2 norm of the accumulated gradient is greater than the threshold, it means that the client needs to communicate with the cloud server in this round. If the L2 norm of the accumulated gradient is less than the threshold, it means that the client does not need to communicate with the cloud server in this round. If the client is judged to need to communicate with the cloud server, the client compresses the accumulated gradient using the "Topk" method to obtain the sparse gradient of this round. If the client is judged to not need to communicate with the cloud server, the client sets the sparse gradient of this round to be equal to 0;

[0010] Third step: the client subtracts the sparse gradient of this round from the accumulated gradient of this round to obtain the error of this round;

[0011] Fourth step: if the sparse gradient obtained by the client in the second step is not equal to 0, the client uploads the sparse gradient to the cloud server. If the sparse gradient obtained by the client in the second step is equal to 0, the client does not upload in this round;

[0012] Fifth step: after receiving the sparse gradient uploaded by the client, the cloud server restores the sparse gradient to a model to obtain the recovery model of this round, and then performs similarity aggregation on the recovery model to obtain the global model of this round;

[0013] Sixth step: the cloud server distributes the global model of this round to the corresponding clients;

[0014] Seventh step: after receiving the global model of this round distributed by the cloud server, the client performs local training to obtain the local model of this round and the gradient of this round. The local model of this round is the personalized model of this round.

[0015] The beneficial effects of this invention are as follows: First, the client decides whether to upload its gradient based on the L2 norm of the gradient and uses the "Topk" compression method to compress the gradient to be uploaded, which greatly reduces the communication cost of the client during federated learning training. Second, the client saves the gradient error before and after compression in each round and adds the error to the gradient to be compressed in the next round, which can alleviate the problem of model performance degradation caused by compression. Finally, the cloud server aggregates a global model for each client through similarity aggregation. Then, after receiving the model from the cloud server, the client updates it locally to obtain a personalized model, which realizes the personalization of the client model and improves the model performance. Attached Figure Description

[0016] The invention will now be further described with reference to the accompanying drawings, in which: Figure 1 This describes the training process of the present invention. Step ① represents the first step of the training process, where the client adds the gradient and error from the previous round to obtain the cumulative gradient to be compressed in the current round. Step ② represents the second step, where the client compares the L2 norm of the cumulative gradient with a threshold maintained by the client and uses the "Topk" method to compress the cumulative gradient to obtain the sparse gradient. Step ③ represents the third step, where the client subtracts the cumulative gradient from the sparse gradient to obtain the error in the current round. Step ④ represents the fourth step, where clients with non-zero sparse gradients upload the sparse gradients to the cloud server. Step ⑤ represents the fifth step, where the cloud server first reconstructs the recovered model for the current round from the received sparse gradients, and then performs similarity aggregation on the recovered model to obtain the global model for the current round. Step ⑥ represents the sixth step, where the cloud server distributes the global model to the corresponding clients. Step ⑦ represents the seventh step, where the client performs local training to obtain the local model and gradient for the current round; the local model for this round is the personalized model for this round. The above process went through T rounds, and the client received the final personalized model. Detailed Implementation

[0017] Consider a federated learning system that has a cloud server that maintains a set of recovery models. and a set of global models u1,…,u m Connecting m clients to a cloud server, using c i (1≤i≤m) represents one of the clients. Let w i Indicates client c i The local model, using α i c i The threshold, denoted by e i c i The error is expressed in g. i c iThe gradient of . Let T represent the total number of rounds of federated learning.

[0018] Before the first round of training begins, the cloud server initializes and restores the model. And send them to the corresponding clients as the clients' local models. Right now Client c i Set your own threshold α i Client c i Initialization error and gradient The initial values ​​are all 0. The task of the t-th (1≤t≤T) training round is to update To obtain a new model The process in round t is as follows:

[0019] Step 1: Client c i First of all and Add them together to get the cumulative gradient. Right now:

[0020]

[0021] in This represents the error in round t-1. It is the gradient of the (t-1)th round. and By adding them together, we can ensure that the gradient information lost due to compression in round t-1 is uploaded to the cloud server for aggregation in round t, thereby improving the accuracy of the model.

[0022] Step 2: c i get After that, c i Will With α i Compare. If Greater than α i Then it represents c i This round requires communication with the cloud server. Otherwise, it represents c. i This round requires communication with the cloud server. If c i If the client is determined to need to communicate with the cloud server, it uses the "Topk" method to compress the accumulated gradient. Obtain sparse gradient If the client is determined not to need to communicate with the cloud server, then the client settings... It equals 0. Specifically, given a vector p∈R d Topk(p)∈R d The l-th element of Topk(p) is defined as:

[0023]

[0024] Where |.| is a function that returns the absolute value, p l Let represent the l-th element of p, and thr be the k-th largest absolute value among the elements of p. In practice, k can be two to three orders of magnitude smaller than d, and this will only sacrifice model accuracy to some extent. In this case, the communication overhead consumed in gradient propagation can be significantly reduced. In summary, The compression process can be summarized as follows:

[0025]

[0026] Step 3: Compression inevitably introduces errors, affecting the accuracy of personalized models. To improve model accuracy, c... i The error is retained for the next round. That is, c i use and The difference between them

[0027]

[0028] Step 4: c i get After that, c i Decide whether to Upload to the cloud server. If... Then c i Will Upload to the cloud server; otherwise, c i No communication with cloud servers is required.

[0029] Step 5: The cloud server receives... Then, firstly according to and The recovery model for this round was obtained. Right now:

[0030]

[0031] Note that if the cloud server does not receive Then make Also note that when t=1, cloud servers get Afterwards, Aggregation yields a global model Next, we will introduce in detail The calculation.

[0032] First, we compute any model Weights:

[0033]

[0034] where σ is a scaling hyperparameter, is a hyperparameter that specifies c i 's contribution to the model of c i 's aggregate model. Then, the cloud server uses the computed weights to aggregate the global model u i of this round for c i :

[0035]

[0036] Tan() is the Tanimoto coefficient, which is used to measure the similarity between models. The calculation method of the Tanimoto coefficient is:

[0037]

[0038] Step 6: The cloud server sends to the corresponding client c i .

[0039] Step 7: The client trains the local model of the t-th round, i.e., updates To minimize communication overhead, c i only communicates with the cloud server after R local model training, rather than transmitting updates to the cloud server after each local update. We use to represent the result of the r-th (1≤r≤R) training. After the r-th training, updates as follows:

[0040]

[0041] where F i () is the loss function. When r=1, After R iterations, c i gets and let So far, the local training process of the t-th round is complete, and c i takes as the personalized model of the t-th round. Finally, the client c i gets to perform the next round of training:

[0042]

[0043] The above is the specific implementation process of the application, through which the individualization of each client model in federated learning is realized, and the statistical heterogeneity problem between clients is solved. In addition, through threshold judgment, some clients do not communicate with the cloud server in a specific round, and the clients communicating with the cloud server use the "Topk" method to compress the data volume communicated with the server, greatly reducing the communication cost of the clients in federated learning and improving the communication efficiency.

Claims

1. A personalized federated learning method for achieving efficient communication on non-independent and identically distributed data, characterized in that, Includes the following steps: Step 1: The client adds the gradient from the previous round to the error from the previous round to obtain the cumulative gradient to be compressed in this round; Step 2: The client compares the L2 norm of the cumulative gradient in this round with the threshold maintained by the client. If the L2 norm of the cumulative gradient is greater than the threshold, it means that the client needs to communicate with the cloud server in this round. If the L2 norm of the cumulative gradient is less than the threshold, it means that the client does not need to communicate with the cloud server in this round. If the client is determined to need to communicate with the cloud server, the client uses the "Topk" method to compress the accumulated gradient to obtain the sparse gradient for this round. If the client is determined not to need to communicate with the cloud server, the client sets the sparse gradient for this round to 0. Step 3: The client calculates the error for this round by subtracting the cumulative gradient from the sparse gradient of this round. Step 4: If the sparse gradient obtained by the client in step 2 is not equal to 0, the client will upload the sparse gradient to the cloud server. If the sparse gradient obtained by the client in step 2 is equal to 0, the client will not upload it in this round. Step 5: After receiving the sparse gradient uploaded by the client, the cloud server first restores the sparse gradient into a model to obtain the restored model for this round, and then performs similarity aggregation on the restored model to obtain the global model for this round. Step 6: The cloud server distributes the global model for this round to the corresponding clients; Step 7: After receiving the global model from the cloud server for this round, the client performs local training to obtain the local model and gradient for this round. The local model for this round is the personalized model for this round. Cloud server maintains a set of recovery models and a set of global models u1,...,u m Connect m clients to the cloud server using c. i (1≤i≤m) represents one of the clients, denoted by w. i Indicates client c i The local model, using α i c i The threshold, denoted by e i c i The error is expressed in g. i c i The gradient of is denoted by T, which represents the total number of rounds of federated learning. Before the first round of training begins, the cloud server initializes and restores the model. And send them to the corresponding clients as the clients' local models. Right now Client c i Set your own threshold α i Client c i Initialization error and gradient The initial values ​​are all 0. The task of training in the t-th round (1≤t≤T) is to update To obtain a new model Gradient of the t-th round in It's the local model from the previous round. This is the new model obtained after this round of training; Will and Add them together to get the cumulative gradient. in This represents the error in round t-1. It is the gradient in the (t-1)th round; The Topk method: Given a vector p∈R d Topk(p)∈R d The i-th element of Topk(p) is defined as Where |·| is a function that returns the absolute value, p l Let represent the l-th element of p, and thr be the k-th largest absolute value among the elements of p; The compression process is summarized as follows: Error in round t in This is the cumulative gradient for this round. It is the compressed sparse gradient; The cloud server received Afterwards, according to and Obtain the recovery model for this round in It's the recovery model from the previous round; If the cloud server did not receive Then make When t=1 cloud servers get Will Aggregation yields a global model Any model Weights: Where σ is the scaling hyperparameter. Is it specified c i The model in c i The hyperparameters of contribution in the aggregation model are used by the cloud server to aggregate c using calculated weights. i The global model u in this round i :

Images