Communication efficient federal learning method capable of resisting privacy data reconstruction
By adopting unbiased sampling and local differential privacy methods in federated learning, the problems of large communication overhead and privacy data leakage of federated learning on resource-constrained devices are solved, and efficient communication and strong privacy protection are achieved.
Patent Information
- Application Number
- CN202311198516.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-15
- Publication Date
- 2025-03-18
AI Technical Summary
Existing federated learning methods have the risk of large communication overhead and privacy data leakage on resource-constrained devices, especially gradient discretization and low-rank factor decomposition will reduce model accuracy, while secure multi-party computing requires huge computational overhead.
The conditional random sampling method based on unbiased sampling technology is adopted to reduce communication overhead, and protect private data through local differential privacy Laplace random perturbation to avoid data reconstruction attacks.
While ensuring the accuracy of the model, the communication overhead during model transmission is greatly reduced, with a reduction of up to 93%, and the private data sets of participants are effectively protected and resisting data reconstruction attacks.
Smart Images
Figure CN119646862A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer information security, and specifically is a privacy-enhanced, communication-efficient federated learning method for resource-constrained devices. Background Art
[0002] With the increasing emphasis on data security and user privacy, as a distributed machine learning method, federated learning aims to train a model that benefits multiple parties without transmitting the private data of each participant. However, for participants with resource-constrained devices, they cannot afford the communication overhead required to transmit the model for multiple iterations. In addition, studies have shown that federated learning still has the possibility of leaking the privacy of participants. For example, an adversary can extract some key features from the eavesdropped global updates and reconstruct the private data set of the participants, which seriously damages the privacy rights of the participants in federated learning.
[0003] In addition to the gradient sparsification method, the mainstream efficient communication methods based on gradient compression also include gradient discretization, knowledge distillation, and low-rank factor decomposition methods. However, gradient discretization and low-rank factor decomposition will greatly reduce the accuracy of the model, and the knowledge distillation method will have additional local computing overhead. Among the methods based on gradient sparsification, there are sketch-based, cropping-based, and unbiased sampling-based methods. However, sketch-based and cropping-based methods cannot accurately control the size of gradient transmission. The above shortcomings cannot be applied to federated learning scenarios of resource-constrained devices. Therefore, this patent uses a conditional random sampling method based on an unbiased sampling method to improve the communication efficiency of the federated learning system.
[0004] The mainstream federated learning privacy protection methods are based on differential privacy and secure multi-party computing. However, although secure multi-party computing can maximize security and model accuracy, it requires huge computing overhead, which is unbearable for resource-constrained devices. Therefore, this patent provides a federated learning method based on local differential privacy to improve the privacy security of the federated learning system.
[0005] In summary, in order to improve the privacy protection capability of federated learning and reduce the communication overhead of resource-constrained devices, the present invention discloses a communication-efficient federated learning method that is resistant to privacy data reconstruction. Based on the communication efficiency of unbiased sampling technology and the privacy security based on local differential privacy, a communication-efficient federated learning method that is resistant to privacy data reconstruction is implemented. Summary of the invention
[0006] In order to solve the problems existing in the prior art, the present invention provides a communication efficient federated learning method that is resistant to privacy data reconstruction, which reduces the communication overhead during model transmission and protects the privacy data of participants during model training while ensuring the accuracy of the federated learning model. The technical solution of the present invention is as follows:
[0007] A communication efficient federated learning method resistant to privacy data reconstruction, characterized in that the method comprises the following steps:
[0008] Step 1: Record local gradients. After receiving the global model from the federated server, use local private data to train the model and record the local gradients generated by the training.
[0009] Step 2: Conditional random sampling of local gradients. Conditional random sampling is performed on the local gradients generated by training, and the local gradients of the specified dimension size are unbiasedly selected through random probability under certain conditions;
[0010] Step 3: Randomly change the gradient sign. Due to the leakage of gradient direction by conditional random sampling, the local gradient sign after sampling is randomly changed while ensuring that the number of different gradient signs is certain.
[0011] Step 4: Add Laplace random perturbation. Further add Laplace noise that satisfies local differential privacy to achieve privacy amplification, and send the local gradient to the federated server;
[0012] Step 5: Compensate for local gradient update errors. Based on the local gradient in step 1, the gradient error caused by the privacy enhancement operation in this round is compensated with a certain coefficient in the next round of training to avoid error accumulation affecting model accuracy.
[0013] Optionally, the specific process of recording the local gradient in step 1 is:
[0014] The local client receives the global model parameters ω broadcast by the federated server g .
[0015] If it is the first training, calculate the local model If it is the rth training, based on the received global model gradient update And the r-1th local gradient Compute local model
[0016] Based on local privacy dataset Computational model Loss And calculate the local model gradient And local model gradient update Also recorded as Stored locally.
[0017] Optionally, the specific process of conditional random sampling of local gradients in step 2 is:
[0018] Based on the r-th training local model gradient update obtained in step 1 Perform conditional random sampling. It is a d-dimensional tensor, and a random probability α∈[0,p) with restricted conditions is used to calculate the priority value of each dimension of the original local model gradient.
[0019] In detail, first, if the gradient of the local model is updated to Calculate priority value in p∈(0,1-e -ε ].
[0020] Secondly, sort the priority values T into Top-k order, and sort the kth priority value Denoted as τ.
[0021] Finally, the gradient update dimension corresponding to the first k priority values is Perform Poisson sampling operation, that is, for The probability of the gradient update of this dimension being sampled is for The probability of the gradient update of this dimension being sampled is
[0022] So, yes Conditional random sampling of local gradients is done.
[0023] Optionally, the specific process of randomly changing the gradient sign in step 3 is:
[0024] Based on the sampled local model gradient update obtained in step 2 Randomly transform the sign of its gradient.
[0025] First, when training the model using the local dataset, record the The number of gradient directions in each dimension is denoted by β for positive gradients and γ for negative gradients.
[0026] Secondly, the gradient direction is randomly changed under the premise of ensuring that the number of positive and negative gradients remains unchanged. Let λ sign =1, and in the case of Pr[γ / (β+γ)], let λ sign =-1.
[0027] That is, for any dimension of gradient update, we have Then we can get the gradient update after the gradient sign is changed
[0028] Optionally, the specific process of adding Laplace random perturbation in step 4 is:
[0029] Based on the local model gradient update obtained in step 3 Perform a local differential privacy operation on , that is, add a Laplace random perturbation to each dimension, and share the privacy budget ε with that in step 2.
[0030] In detail, Then there is The gradient update after adding Laplace random perturbation is obtained And upload this gradient update to the federated server as the gradient update of the local client for this r-th training.
[0031] Optionally, the specific process of compensating the local gradient update error in step 5 is:
[0032] Original gradient without gradient sampling, direction transformation, and perturbation recorded in step 1 When the global gradient of the r+1th training When updates are broadcast to local clients, they are not used when computing the local model. And use Right now
[0033] Afterwards, using the local model Perform the r+1th training and calculate the local model gradient update, repeating steps 1, 2, 3, and 4.
[0034] Compared with the prior art, the present invention has the following beneficial technical effects:
[0035] The communication-efficient federated learning method resistant to privacy data reconstruction provided by the present invention greatly reduces the communication overhead during model transmission by up to 93% while ensuring that the model accuracy of the federated learning system is basically not negatively affected. In addition, the privacy protection method provided by the present invention can effectively protect the private data set data of the participants and resist data reconstruction attacks. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is the basic flowchart of federated learning;
[0037] Figure 2 A flow chart of the method provided by the present invention; DETAILED DESCRIPTION
[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings in specific embodiments.
[0039] like Figure 1 As shown, the present invention provides a method for dealing with the problems of privacy protection threats and huge communication overhead brought about by the upload phase of the federated learning model.
[0040] Specifically, when multiple device owners decide to maximize the use of each party’s private data to jointly train a global model that benefits all parties, a federated server for model aggregation needs to be established. The specific steps are as follows:
[0041] Initialization. The federated server initializes model ω and sends it to participants 1, 2, and 3.
[0042] Step 1: Local model training. If it is the first training, participants 1, 2, and 3 train based on the initialization model; if it is the rth training, they update based on the local gradient of the r-1th The global gradient update sent by the federated server is used to calculate the new model. After that, the model is trained using local private data and the gradient of the model loss is calculated. and local gradient update
[0043] Step 2: Upload the gradient update. Upload to the federation server.
[0044] Step 3: Model aggregation. The federated server receives the gradient updates submitted by each participant and aggregates them in a consensus manner. For example, the federated average aggregation is to add and average the gradients of each party. Then a new round of gradients is sent to each participant for the next training.
[0045] For this scenario, in step 2, the remote resource-constrained device may not be able to afford the additional computing overhead in addition to the local model training and the communication overhead of the frequent model transmission. In addition, if participant 2 is a malicious user, the global model sent by the federated server may reconstruct the private data of participants 1 and 3, causing serious privacy leakage accidents. Therefore, the present invention provides a communication-efficient federated learning method that is resistant to privacy data reconstruction, which is used to provide communication efficiency and privacy security while ensuring that the accuracy of the federated model is not damaged.
[0046] like Figure 2 As shown, the present invention is mainly implemented in Figure 1 In step 2, the model update upload part records local gradients, conditionally randomly samples local gradients, randomly changes gradient signs, adds Laplace random perturbations, and compensates for local gradient update errors. The specific steps are as follows:
[0047] Step 1: Record local gradients. After receiving the global model from the federated server, use local private data to train the model and record the local gradients generated by the training.
[0048] The local client receives the global model parameters ω broadcast by the federated server g .
[0049] If it is the first training, calculate the local model If it is the rth training, based on the received global model gradient update And the r-1th local gradient Compute local model
[0050] Based on local privacy dataset Computational model Loss And calculate the local model gradient And local model gradient update Also recorded as Stored locally.
[0051] Step 2: Conditional random sampling of local gradients. Conditional random sampling is performed on the local gradients generated by training, and local gradients of specified dimensions are unbiasedly selected through random probability under certain conditions.
[0052] Based on the r-th training local model gradient update obtained in step 1 Perform conditional random sampling. It is a d-dimensional tensor, and a random probability α∈[0,p) with restricted conditions is used to calculate the priority value of each dimension of the original local model gradient.
[0053] In detail, first, if the gradient of the local model is updated to Calculate priority value in p∈(0,1-e -ε ].
[0054] Secondly, sort the priority values T into Top-k order, and sort the kth priority value Denoted as τ.
[0055] Finally, the gradient update dimension corresponding to the first k priority values is Perform Poisson sampling operation, that is, for The probability of the gradient update of this dimension being sampled is for The probability of the gradient update of this dimension being sampled is
[0056] So, yes Conditional random sampling of local gradients is done.
[0057] Step 3: Randomly change the gradient sign. Due to the leakage of gradient direction by conditional random sampling, the local gradient sign after sampling is randomly changed while ensuring the number of different gradient signs is constant.
[0058] Based on the sampled local model gradient update obtained in step 2 Randomly transform the sign of its gradient.
[0059] First, when training the model using the local dataset, record the The number of gradient directions in each dimension is denoted by β for positive gradients and γ for negative gradients.
[0060] Secondly, the gradient direction is randomly changed under the premise of ensuring that the number of positive and negative gradients remains unchanged. Let λ sign =1, and in the case of Pr[γ / (β+γ)], let λ sign =-1.
[0061] That is, for any dimension of gradient update, we have Then we can get the gradient update after the gradient sign is changed
[0062] Step 4: Add Laplace random perturbation. Further add Laplace noise that satisfies local differential privacy to achieve privacy amplification, and send the local gradient to the federated server.
[0063] Based on the local model gradient update obtained in step 3 Perform a local differential privacy operation on , that is, add a Laplace random perturbation to each dimension, and share the privacy budget ε with that in step 2.
[0064] In detail, Then there is The gradient update after adding Laplace random perturbation is obtained And upload this gradient update to the federated server as the gradient update of the local client for this r-th training.
[0065] Step 5: Compensate for local gradient update errors. Based on the local gradient in step 1, the gradient error caused by the privacy enhancement operation in this round is compensated with a certain coefficient in the next round of training to avoid error accumulation affecting model accuracy.
[0066] Original gradient without gradient sampling, direction transformation, and perturbation recorded in step 1 When the global gradient of the r+1th training When updates are broadcast to local clients, they are not used when computing the local model. And use Right now
[0067] Afterwards, using the local model Perform the r+1th training and calculate the local model gradient update, repeating steps 1, 2, 3, and 4.
[0068] It should be noted that the present invention is not limited to the above embodiments. The protection scope of the present invention is determined by the appended claims. Technical solutions and improvements based on the spirit and scope of the present invention should all be covered by the claims of the present invention.
Claims
1. A communication-efficient federated learning method that is resistant to privacy data reconstruction, characterized in that: The method comprises: Step 1: Record local gradients. After receiving the global model from the federated server, use local private data to train the model and record the local gradients generated by the training. Step 2: Conditionally randomly sample local gradients. Conditionally randomly sample the local gradients generated by training. Select local gradients of specified dimensions unbiasedly through random probability under certain conditions. Step 3: Randomly change the gradient sign. Due to the leakage of gradient direction by conditional random sampling, the local gradient sign after sampling is randomly changed while ensuring the number of different gradient signs is certain. Step 4: Add Laplace random perturbation, further add Laplace noise that satisfies local differential privacy to achieve privacy amplification, and send the local gradient to the federated server; Step 5: Compensate for the local gradient update error. Based on the local gradient in step 1, the gradient error caused by the privacy enhancement operation in this round is compensated with a certain coefficient in the next round of training to avoid error accumulation affecting the model accuracy.
2. The method according to claim 1, characterized in that The specific process of recording local gradients in step 1 is: The local client receives the global model parameters ω broadcast by the federated server g , if it is the first training, calculate the local model If it is the rth training, based on the received global model gradient update And the r-1th local gradient Compute local model Based on local privacy dataset Computational model Loss And calculate the local model gradient And local model gradient update Also recorded as Stored locally.
3. The method according to claim 1, characterized in that The specific process of conditional random sampling of local gradients in step 2 is: Based on the r-th training local model gradient update obtained in step 1 Perform conditional random sampling; if is a d-dimensional tensor, and a random probability α∈[0,p) with constraints is used to calculate the priority value of each dimension of the original local model gradient. First, if the gradient of the local model is updated to Calculate priority value in p∈(0,1-e -ε ]; Secondly, sort the priority values T into Top-k order, and sort the kth priority value Recorded as τ; Finally, the gradient update dimension corresponding to the first k priority values is Perform Poisson sampling operation, that is, for The probability of the gradient update of this dimension being sampled is for The probability of the gradient update of this dimension being sampled is So, yes Conditional random sampling of local gradients is done.
4. The method according to claim 1, characterized in that The specific process of randomly changing the gradient sign in step 3 is: Based on the sampled local model gradient update obtained in step 2 Randomly transform its gradient sign. First, when training the model with the local data set, record it every 10 trainings. The number of gradient directions in each dimension, the number of positive gradients is recorded as β, and the number of negative gradients is recorded as γ; secondly, under the premise of ensuring that the number of positive and negative gradients remains unchanged, the gradient direction is randomly changed, and λ is set with probability Pr[β / (β+γ)]. sign =1, and in the case of Pr[γ / (β+γ)], let λ sign =-1, that is, for any dimension of gradient update, Then we can get the gradient update after the gradient sign is changed 5. The method according to claim 1, characterized in that The specific process of adding Laplace random perturbation in step 4 is: Based on the local model gradient update obtained in step 3 Perform local differential privacy operation on , that is, add Laplace random perturbation to each dimension, and share the privacy budget ε in step 2. In detail, for Then there is The gradient update after adding Laplace random perturbation is obtained And upload this gradient update to the federated server as the gradient update of the local client for this r-th training.
6. The method according to claim 1, characterized in that The specific process of compensating the local gradient update error in step 5 is: Original gradient without gradient sampling, direction transformation, and perturbation recorded in step 1 When the global gradient of the r+1th training When updates are broadcast to local clients, they are not used when computing the local model. And use Right now Afterwards, using the local model Perform the r+1th training and calculate the local model gradient update, repeating steps 1, 2, 3, and 4.