Privacy protection method for federated learning based on random participation differential privacy shuffling model

Abstract

The present invention relates to the field of data security technology, and more particularly to a method for protecting the privacy of federated learning based on a randomized differential privacy shuffling model. The method comprises the following steps: a client generates local weight data for a round of joint training using a local randomizer and uploads the data to the shuffler; the shuffler performs a random shuffling operation on the received local weight data and transmits the shuffled local weight data to an analyzer within a fixed time constraint; the analyzer aggregates the received local weight data, uses the aggregated weights to update the global model parameters for the next round of joint training, and broadcasts the updated global model parameters to all clients. The present invention can address the issues of existing federated learning models, such as network congestion leading to a long joint training process and thus affecting the feasibility of federated learning in practical application scenarios. The method can effectively sample clients during federated training while ensuring high model accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a federated learning privacy protection method based on a random participation differential privacy shuffling model. Background Art

[0002] Federated learning, a distributed collaborative learning solution, effectively addresses data silos and privacy leaks. During model training, local users participating in federated learning independently train models using private datasets and then upload the trained models to a server. The server collects and aggregates the models uploaded by all users and returns the new models to the users. Leveraging these advantages, federated learning achieves efficient data utilization and privacy protection, and has demonstrated strong performance in practical application scenarios such as smart healthcare, finance, and autonomous driving. However, relevant research has shown that model weights shared by local users may leak privacy. Therefore, differential privacy (DP), a widely accepted standard privacy concept favored by companies such as Google, Apple, and Microsoft, has been introduced into federated learning to protect shared model weights. Existing research on federated learning based on differential privacy is mainly categorized into centralized federated learning (DP-FL) and local federated learning (LDP-FL), each with its own advantages and disadvantages. DP-FL offers high accuracy but relies heavily on server trustworthiness. LDP-FL protects the privacy of model weights by local users before sharing them, eliminating the need to consider server trustworthiness. However, this results in lower model utility. Therefore, the issue of effectively balancing model accuracy and privacy has attracted the attention of a large number of researchers.

[0003] The emergence of new shuffling models enables an effective trade-off between model accuracy and privacy. One factor in this trade-off is the shuffler, a semi-honest server placed between the local randomizer and the server. Its primary task is to randomly permute the data submitted by the client, masking the mapping between inputs and outputs. This allows the server to only receive out-of-order, anonymous data records, ensuring data accuracy. This privacy protection achieved through shuffling is also known as shuffling privacy amplification. Another privacy amplification technique for shuffling models is subsampling. In subsampling, during each round of model training, only the server uniformly and randomly samples a random subset of the client or original dataset to serve as input to the privacy-preserving mechanism. Once all sampled models have completed training and shared their weights, the analyzer aggregates the received model weights and broadcasts the new round of model weights. When combined with shuffling, this privacy-preserving technique further confuses adversaries and achieves amplified privacy guarantees without sacrificing utility. Furthermore, this combination requires only a small amount of noise to achieve a satisfactory level of privacy.

[0004] Existing work on privacy amplification with shuffled models uses private mean estimates in several p-spaces to achieve efficient gradient aggregation during iterative model training. Alternatively, shuffled privacy amplification can be performed only on the n data records input to a ∈-differentially private local randomizer, achieving better privacy guarantees. However, these schemes typically use shufflers to achieve privacy amplification. However, when millions of mobile devices are simultaneously participating in model learning, some clients may drop out of training due to network and battery issues. This results in these schemes taking longer to complete a single round of model training and incurring significant communication resource consumption. Therefore, simulating the client sampling process during real-world training is crucial. Existing client sampling research proposes a model where each client participates in training independently, assuming that each client only stores a single sample. However, this scheme relies on a trusted data analyzer. While each client is allowed to have multiple data samples, the number of samples sampled is fixed. It remains unclear how to apply client random sampling mechanisms to shuffled models. In addition, the number of devices participating in a model iteration training is in the millions. When the joint training server aggregates all shared perturbation weights, the model aggregation dimension will increase, causing a surge in the data privacy budget and affecting the actual application effect of federated learning. Summary of the Invention

[0005] To this end, the present invention provides a federated learning privacy protection method based on a random participation differential privacy shuffling model, which solves the problem that the existing federated learning model has a long joint training process due to network congestion and other reasons, thereby affecting the feasibility of federated learning in actual application scenarios. During the alliance training, the client can be effectively sampled while ensuring the high accuracy of the model.

[0006] According to the design scheme provided by the present invention, a federated learning privacy protection method based on a random participation differential privacy shuffling model is provided, comprising:

[0007] The client uses the local randomizer to generate local weight data for this round of joint training and uploads it to the shuffler, where the number of clients is set to n, and n is greater than 1;

[0008] The shuffler performs a random shuffling operation on the received local weight data and transmits the randomly shuffled local weight data to the analyzer within a fixed constraint time;

[0009] The analyzer aggregates the received local weight data, uses the aggregated weights to update the global model parameters for the next round of joint training, and broadcasts the updated global model parameters to all clients.

[0010] As the federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, the client further uses a local randomizer to generate local weight data for this round of joint training, including:

[0011] First, the probability of each client independently exiting the training in this round of joint training is set, and the random participation rate of each client in this round of joint training is dynamically adjusted using the probability of each client independently exiting the training.

[0012] Next, the clients participating in this round of joint training are determined based on the random participation rate. The local training of this round is completed on the local private dataset using the clients participating in this round of joint training, and the model weight parameters obtained after training are trimmed.

[0013] Then, a local randomizer is used to add perturbation noise to the pruned local model weight parameters and uploaded to the shuffler via encrypted transmission.

[0014] As the federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, the process of dynamically adjusting the random participation rate of each client in this round of joint training is expressed as: β = p(1-p′), where p is the original random participation rate of the client independently determined to participate in this round of training, p′ is the set probability of the client independently exiting training, and β is the new random participation rate of the client after dynamic adjustment.

[0015] As a federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, further, a local randomizer is used to add perturbation noise to the pruned local model weight parameters, and the weight parameters to which the perturbation noise is added are limited to the k most important dimensions in the local model.

[0016] As the federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, further, a local randomizer is used to add perturbation noise to the pruned local model weight parameters, including:

[0017] First, the local randomizer uses the Top function to obtain the query results of the received local model weight parameters, and obtains the most important k index sets and non-important index sets according to the size of the query results;

[0018] Next, add disturbance noise to the weight elements in the most important k index sets by traversing the weight index;

[0019] Then, the local model weight parameters are updated, all elements in the most important k index sets and the unimportant index sets are added to the local model weight, and the updated local model weight parameters and client ID are encrypted and uploaded.

[0020] As the federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, further, the disturbance noise is Laplace noise or Gaussian noise.

[0021] As a federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, further, when the shuffler performs a random shuffling operation on the received local weight data, a fixed constraint time for the shuffler to perform the random shuffling operation on the received data is set according to the time of this round of joint training and the time required for each client to complete this round of joint training and share this round of local model weight data.

[0022] As a federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, the shuffler further performs a random shuffling operation on the received data within a preset shuffling time interval, including:

[0023] First, the initial time for the shuffler to transmit weight data is obtained based on the time it takes for each client to complete the current round of joint training and share the current round of weights, as well as the number of clients participating in the current round of joint training;

[0024] Then, when the shuffler receives the weight data uploaded by a valid client, it adds the time required for the client to complete this round of joint training and share this round of weights to the initial time or this training time to extend the weight data transmission time, and transmits the local model weight data of this round of joint training to the analyzer within the extended weight data transmission time.

[0025] As a federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, the shuffler further performs a random shuffling operation on the received data within a preset scrambling time interval, and also includes: if the extended weight data transmission time exceeds the final submission time and the number of client local model data received by the shuffler does not meet the minimum number of clients participating in the joint training in this round, the shuffler uses the global model parameters broadcast by this round of training to virtually fill the local model data, so that the number of filled client local model data reaches the minimum number of clients participating in the joint training in this round, and uses the virtual client ID of the non-participating training to encrypt the virtually filled local model data during encrypted transmission, wherein the final submission time is set based on the number of clients and the time required for each client to complete the joint training in this round and share the weights in this round.

[0026] As a federated learning privacy protection method based on the random participation differential privacy shuffling model of the present invention, further, when the analyzer performs aggregate evaluation on the received local weight data, it first decrypts the encrypted data transmitted, excludes invalid client ID weight data, obtains valid client ID local weight data, and then aggregates the valid client ID local weight data.

[0027] Beneficial effects of the present invention:

[0028] In the present invention, each client can independently and randomly decide whether to participate in each training session. Once participating in the training, the client will use its own private dataset for model training and upload its own perturbed model weights. The shuffler will also complete the shuffling and submission of the model weights within a fixed constraint time, thereby protecting the privacy of the model and avoiding excessive training time due to network congestion and other reasons. This also makes the federated training in this case feasible in model training in real-world scenarios. By setting the perturbation to only the most important K weights in the local model, the privacy budget after model aggregation is reduced. For the phenomenon of the shuffler receiving fewer weights due to severe network congestion, this risk is avoided by dynamically adjusting the shuffling time. When the shuffler reaches the maximum shuffling time but the number of weights is still small, virtual weights can be used to supplement the number of weights to protect the small number of model weights. Further experimental data verification shows that the federated training framework in this case can have excellent model accuracy and strong privacy protection capabilities, facilitating its application in distributed collaborative data protection scenarios such as smart healthcare, finance, and autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This figure illustrates the privacy protection process of federated learning based on the random participation differential privacy shuffling model in the embodiment;

[0030] Figure 2 This is a schematic diagram of the federated learning framework in the shuffle model in the embodiment;

[0031] Figure 3 This is an illustration of the model accuracy under the two data distributions of Test IId and Test Non-IID in the embodiment;

[0032] Figure 4 This is an illustration of the model training time and accuracy under two data distributions of three benchmark data sets in the embodiment. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below with reference to the accompanying drawings and technical solutions.

[0034] The following introduces the basic terms and properties related to differential privacy and subsampling.

[0035] In central differential privacy, for ∈, δ≥0, any random mechanism Any subset of the output on adjacent datasets D, D′ The following conditions are met, and the mechanism M satisfies (∈, δ)-differential privacy ((∈, δ)-DP):

[0036] Pr[M(D)∈S]≤e ∈ Pr[M(D′)∈S]+δ

[0037] Where adjacent datasets are two datasets that differ by at most one record; ∈ is the privacy budget (privacy parameter), representing the degree of privacy protection; a smaller ∈ is associated with higher privacy protection; and δ (δ∈[0, 1]) is the probability of differential privacy leakage. When δ = 0, mechanism M provides the strictest ∈-differential privacy protection, also known as "pure differential privacy." While the (∈, δ)-DP mechanism can provide sufficient data protection, this traditional differential privacy requires a trusted data processor, which is generally unrealistic.

[0038] To eliminate this limitation, local differential privacy is proposed, a random mechanism exist The same output result is obtained for any pair of inputs x, x′ in If the following inequality is satisfied, then the mechanism R satisfies ∈-local differential privacy (∈-LDP):

[0039]

[0040] In summary, differential privacy is an excellent privacy protection tool. Its combination properties have not only attracted widespread attention but also achieved the commonality of DP and LDP. The relevant combination properties are as follows:

[0041] Lemma 1 A mechanism that satisfies ∈-DP will satisfy k∈-DP under k-fold adaptive combination.

[0042] Lemma 2 For all ∈, δ, δ′>0, under k-fold adaptive combination, the group (∈, δ)-DP mechanism will satisfy the (∈′, kδ+δ′)-DP mechanism, where

[0043] The shuffle model is a distributed computing model that consists of three parts: They are analyzer, shuffler, and local randomizer. Assume that there are n users participating in the training, and their data can be represented as a dataset During training, each user uses a local randomizer Perturb your data to satisfy ∈ l - LDP m messages. Here we can focus more on the single message, that is, the case of m=1. Then, each user sends his report to the shuffler Here all reports are randomized to anonymize them. Finally, the server receives the reports from the shuffler and analyzes them.

[0044] Taking into account is an untrusted analyzer that, according to the backproperty of differential privacy, only needs to ensure Conform to (∈ c , δ c )-DP, then and The same level of privacy will be achieved. c <∈ l hour, This results in stronger privacy guarantees, a phenomenon known as "privacy amplification." Compared to the local differential privacy model, the shuffled model only needs to add a small amount of noise to achieve the same level of privacy guarantees. This privacy amplification is mainly achieved through the subsampling mechanism. The specific theory is as follows:

[0045] Theory 1 Privacy amplification through subsampling: sampling m records from a set of n records to form a new set with non-replacement sampling relationship, and the mechanism If (∈, δ)-DP is satisfied, then the mechanism satisfy

[0046] The shuffle model has received widespread attention as a trust model between the central federation model and the local federation model. It can achieve privacy amplification through subsampling data or shuffling, which means that the model will provide stronger privacy guarantees. With the increasing number of training devices, there is also a correlation between the number of users and the model. Users may fail to share the local model due to network congestion and other reasons, affecting the global training time and model accuracy. Therefore, it is particularly important to ensure that users can participate in training normally. Figure 1 As shown in the figure, a federated learning privacy protection method based on a random participation differential privacy shuffling model is provided, which includes:

[0047] S101. The client generates local weight data for joint training using a local randomizer and uploads it to the shuffler.

[0048] S102, the shuffler performs a random shuffling operation on the received local weight data and transmits the randomly shuffled local weight data to the analyzer within a fixed constraint time;

[0049] S103: The analyzer aggregates the received local weight data, uses the aggregated weights to update the global model parameters for the next round of joint training, and broadcasts the updated global model parameters to all clients.

[0050] Among them, the client uses a local randomizer to generate local weight data for this round of joint training. Specifically, it can be designed to include the following contents: first, set the probability of each client independently exiting training in this round of joint training, and use the probability of each client independently exiting training to dynamically adjust the random participation rate of each client in this round of joint training; then, determine the clients participating in this round of joint training based on the random participation rate, use the clients participating in this round of joint training to complete this round of local training on the local private dataset, and trim the model weight parameters obtained after training; then, use the local randomizer to add perturbation noise to the trimmed local model weight parameters, and upload them to the shuffler through encrypted transmission.

[0051] like Figure 1 In the shuffled model federated learning framework shown, it is assumed that there are n users participating in the training, and each user has a d-dimensional local model weight w i During the iterative training of the model, each user will independently and randomly make the decision to participate in this training and use the local randomizer Generate a result y i , and then the result y i Upload to shuffler; shuffler Perform random shuffling on the received information and send the shuffled data to the analyzer; The information is aggregated and evaluated, and the processed value w is added to the global model in round t, and the update method is θ t ←θ t-1 +w. The pseudo code of the framework is shown in Algorithm 1.

[0052]

[0053] In Algorithm 1, the local randomizer Shuffler Analyzer It is the part that implements model perturbation, shuffling and aggregation. In the tth round of training, the analyzer broadcasts the model θ to all clients t-1, asking them to participate in this learning. Each client will independently toss a biased coin with probability p and decide to participate in this round of training only when the head returns. The probability p of a client successfully participating in training can be called the random participation rate, and this probability follows the Bernoulli distribution Bern(p). Considering that even if the client decides to participate in training, it may exit due to various reasons such as power supply and network. In order to be more consistent with the participation situation in the real world, it is assumed that the client will have an independent probability p' of exiting training in each round of training, and this probability will be dynamically adjusted during each round of training, so the new random participation rate β = p(1-p').

[0054] When client i decides to participate in training, it downloads the model θ t-1 In the private dataset d i Complete this round of training, and the d-dimensional model weight w obtained by training i Perform weight parameter clipping and add Laplace noise or Gaussian noise using a local randomizer and send it (encrypted) to the shuffler, where C is the clipping threshold of the weight data, ∈ l The local privacy budget of each local weight is represented by . The shuffler will perform random shuffling operations on the received information within a specified time and send the shuffled data to the untrusted analyzer. This method avoids the problem of the client not being able to upload weights in time due to network communication problems, and also ensures that the global model completes a model training within a normal time. When the analyzer receives the data, it will evaluate, aggregate, and correct the uploaded data, and apply the new weight data w to the next round of model update tasks θ t ←θ t-1 +w, the final broadcast model θ t And seek clients for the next round of training.

[0055] Under the above settings, a simple protocol SFD-Ras can be constructed. in Denotes the probability of outputting an element from the privacy blanket, b represents the discrete domain [b], that is, the input value x will be encoded into the discrete domain [b] by the local randomizer and the output will be randomized. In simple terms, the extension of Lemma 3 and Corollary 1 is achieved by performing local random perturbations on each dimension of the weight and the final weight model aggregation. Since only the total privacy budget ∈ of the model weight is given l It is impossible to complete local randomness in each dimension. Combined with the combination property of Lemma 1, it can be concluded that will satisfy ∈ dl -LDP, where ∈ dl =∈ l / d. Therefore, the local client i executes Randomize(wi ,∈ l ) can be seen as After the shuffler receives the perturbation message and completes the random perturbation, it can dl Substitute into Lemma 3, and then derive the amplified central differential privacy level (∈ cd , δ cd )-DP. Since this is the privacy level in dimension, combining Lemma 1 and Lemma 2, we can easily derive Theorem 2, a privacy guarantee at the vector level.

[0056] Lemma 3. For if Satisfy∈ l -LDP, for But we will get (∈ c , δ c ),in

[0057] Corollary 1. In the shuffle model, if is∈ l -LDP, where will satisfy (∈ c , δ c )-DP and

[0058] Theory 2. For any neighboring datasets D, D′, which are different in the d-dimensional local vector of a user, then On the SFD-Rass protocol, it will satisfy (∈ c , δ c ):

[0059]

[0060] δ c =δ cd (d+1)

[0061] Since the client decides whether to participate in training based on the probability β, assuming It is a mechanism whereby clients independently participate in training with a probability of β. Therefore, under ideal expectations, each round of training will have βn clients choose to participate in the training. They will then send their perturbation weight data to the trusted shuffler for random permutation, and the untrusted analyzer will complete the evaluation and analysis of the data. Combining the amplification constraint of Corollary 1 and the subsampling mechanism of Theory 1, Theorem 3 can be easily derived. In addition, according to the probability of information leakage in differential privacy, it should be guaranteed that δ cd <2β, which is a very reasonable setting because according to the standard calculation Therefore, the final SFD-Ras The privacy guarantee in Theorem 3 will be achieved. Corollary 2 will also achieve the privacy budget ∈ l Towards Privacy Budget c of the amplification.

[0062] Theory 3. and δ cd <2β, for any neighboring datasets D, D′, they are different in the d-dimensional local vector of a user, the mechanism will satisfy (∈ cd , δ cd )-DP:

[0063]

[0064]

[0065] δ c =βδ cd (d+1)

[0066] Corollary 2. For SFD-Ras, in ∈ l ≤d·log(βn / log((d+1) / δ c ) / 2, the amplified central differential privacy ∈ c will look like this:

[0067]

[0068] By observing the privacy guarantee of Corollary 2, we can find that SFD-Ras provides very good privacy amplification, but the level of central differential privacy it provides depends on the number of sampled users and the dimension d of the weights. From a privacy perspective, the amplification effect will increase as the number of clients decreases, but no clients participate in training or the number of clients is small, which is contrary to the real world situation. From a utility perspective, when the privacy budget ∈ dl In the case that this can be ignored, performing local randomization on each dimension of the weight will generate a lot of noise, resulting in a decrease in model accuracy. In addition, sampling of all dimensions can easily lead to the loss of "important" dimensions of the model and an increase in the privacy budget after aggregation.

[0069] Therefore, further, in the embodiment of this case, a local randomizer is used to add disturbance noise to the pruned local model weight parameters, which can be designed to include the following content:

[0070] First, the local randomizer uses the Top function to obtain the query results of the received local model weight parameters, and obtains the most important k index sets and non-important index sets according to the size of the query results;

[0071] Next, add disturbance noise to the weight elements in the most important k index sets by traversing the weight index;

[0072] Then, the local model weight parameters are updated, all elements in the most important k index sets and the unimportant index sets are added to the local model weight, and the updated local model weight parameters and client ID are encrypted and uploaded.

[0073] For the problems of SFD-Ras, we use the enhanced privacy amplification protocol SFD-Rss to deal with them. dl The perturbation weights are different for each dimension. SFD-Rss only samples and perturbs the most important k dimensions in the model. This avoids the phenomenon of random subsampling treating all dimensions equally and causing the loss of "important" dimensions. It can also reduce the privacy budget after model aggregation. Therefore, it is particularly important to subsample and perturb only the most important k dimension weights. In addition, this method can also make each dimension get a larger privacy budget. benefit from it.

[0074]

[0075] As shown in Algorithm 2, the local randomizer will generate the model weight data w to be uploaded by the customer. i Perform local perturbation. When the local randomizer receives the weight data from the client, it executes the Top() function and obtains the most important k index sets S according to the size of α. top and a set S of unimportant indices non And by traversing the index of d-dimensional weight, the top The weight elements in the middle add noise that conforms to the Laplace or Gaussian distribution, which is non The weight elements in do nothing. The local randomizer then performs an update on the model weights, adding all elements to the model weights. and use Enc pka (·) Encrypt a unique client id to prove that the weights shared by the client in this round of training are valid.

[0076] The fixed constraint time for the shuffler to perform random shuffling operations on the received data can be set based on the joint training time of this round and the time required for each client to complete the joint training of this round and share the local model weight data of this round.

[0077] Specifically, the shuffler performs a random shuffle operation on the received data within a fixed constraint time, including:

[0078] First, the initial time for the shuffler to transmit weight data is obtained based on the time it takes for each client to complete the current round of joint training and share the current round of weights, as well as the number of clients participating in the current round of joint training;

[0079] Then, when the shuffler receives the weight data uploaded by a valid client, it adds the time required for the client to complete this round of joint training and share this round of weights to the initial time or this training time to extend the weight data transmission time, and transmits the local model weight data of this round of joint training to the analyzer within the extended weight data transmission time.

[0080] Considering the problem that the weights shared by the client cannot be uploaded to the shuffler in time due to external reasons such as network congestion, resulting in too few weights and reduced security of the shuffle model, we can use dynamic time constraints to address this challenge. Assuming that in a slightly congested environment, in the tth round of training, the time required for each client to complete training and share the weights of this round is Ti t , so the initial time for the shuffler to submit shuffle weights can be set to nβTi t When the shuffler receives a valid client weight data, it will increase the time of this training by a Ti t Extend the time for submitting weights. Once the final submission time nTi is exceeded t If the number of client weights received by the shuffler does not meet the minimum number of clients in this round, the shuffler will use the broadcast model θ trained in this round. t-1 Virtually fill in the remaining number of weights to meet the minimum weight requirement. This method is different from the method of randomly virtual filling data. This method will not affect the accuracy of the model to a certain extent.

[0081]

[0082] As shown in the pseudocode of Algorithm 3, it is easy to see that when the shuffler virtually supplements the number of model weights, it also generates an encrypted client ID that does not participate in training, indicating that the virtual filled weights in this round of training are invalid. Finally, the shuffler will send the randomly arranged weight data to the analyzer within the adaptively adjusted time. After receiving the data, the analyzer will decrypt the client ID according to the method in Algorithm 1 to exclude the weight data of the client that did not participate in training, and then update the next round of model θ t ←θ t-1 +w, and seek clients to participate in the next round of training.

[0083] For the SFD-rss protocol, it can be summarized as in is the s virtual weight information supplemented by the shuffler, It is a local randomizer that takes the most important k parameters in d dimensions with probability α. Compared with SFD-Rsa, the latter is obtained by Implement local sampling perturbation and use As the new privacy budget. Except that the shuffler dynamically adjusts the constraint submission time and adds the number of virtual weights, the other steps in the shuffling and analysis parts are the same. Therefore, Theorem 4 shows that A combination of complete privacy amplification bounds and vector-level differential privacy.

[0084] Theory 4. and δ cd <2βα, for any neighboring datasets D, D′, they are different in the d-dimensional local vector of a user, then the mechanism will satisfy (∈ cd , δ ca )-DP:

[0085]

[0086]

[0087] δ c =δ cd (αd+1)

[0088] Since only the most important k dimensions of data are sampled and disturbed, and virtual number padding is implemented in the shuffler part, combined with the sampling rates α, β, the local privacy budget ∈ l To the center privacy budget ∈ c amplification effect.

[0089] Corollary 3. For SFD-rss, in ∈ l ≤αdlog(βn / log((2αd+α) / δ c ) / 2, the amplified central differential privacy ∈ c will look like this:

[0090]

[0091] To verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0092] The security and accuracy of the model were examined using the most commonly used image datasets: MNIST, FMNIST, and Cifar-10. Considering the complexity and disorder of real-world data distributions, the datasets were divided into two types: independent and identically distributed (IID) and non-IID data distributions to specifically explore the model's performance in real-world scenarios. For the MNIST and FMNIST datasets, a convolutional neural network with a weight dimension of d = 50618 was used for model training. For the Cifar-10 dataset, a convolutional neural network with a weight dimension of d = 231562 was used to verify the model's performance. Experiments were conducted using 1000 clients in a local Linux environment, leveraging the powerful performance provided by a GeForce RTX3090Ti. To better evaluate the model, three commonly used models are used to compare with the SFD-Ras and SFD-Rss proposed in this solution. The three models are the non-private federated average model (FedAvg), the DP-FedAvg model and the LDP-FedAvg model, which are inspired by DP-FL's uniform addition of Gaussian noise to the model and LDP-FL's local random perturbation.

[0093] For SFD-Ras and SFD-Rss, a Laplace mechanism is used as the local random generator for each dimension. Provide perturbation value. For mnist and fmnist datasets, give ∈ l =506.18, α = 0.02, so the privacy budget of each dimension of SFD-Ras and SFD-Rss is ∈ dl =0.01,∈ kl = 0.5. After the model completes a global training, the analyzer The privacy guarantee will be amplified from (2.68, 5e-6)-DP of SFD-Ras to (0.37, 5e-6)-DP of SFD-Rss.

[0094] Figure 3 The performance of the model under two data distributions is shown in Figure 2, mnist and fmnist, ε = 506.18, α = 0.02, cifar-10, ε = 2315.62, α = 0.01. Figure 3 Observing plots a and b in Figure 1 clearly shows that the SFD-Rss model achieves significantly higher accuracy than the SFD-Ras model for both IID and Non-IID data distributions. Verification on the Cifar-10 dataset demonstrates that the privacy guarantee of the model is amplified from (6.64, 5e-6)-DP (SFD-Ras) to (1.39, 5e-6)-DP (SFD-Rss), while its accuracy for IID and Non-IID data distributions increases by 20.41% and 12.65%, respectively.

[0095] Clearly, under both data distributions and the same (0.37, 5e-6)-DP guarantee across all three datasets, SFD-Rss outperforms DP-FedAvgl and LDP-FedAvg. This performance is primarily due to the top-k subsampling and shuffler virtual padding mechanisms implemented in this solution. In summary, if only the client-side sampling and shuffling processes of SFD-Ras are used, SFD-Ras will not outperform DP-FL. Furthermore, when the privacy guarantee of DP-FedAvg2 is set to (34.78, 5e-6)-DP, it is observed that the line approaches NP-FedAvg and achieves higher model accuracy than SFD-Rss. However, the privacy guarantee of DP-FedAvg2 is significantly inferior to that of SFD-Rss. Therefore, it can be concluded that the top-k sampling and increased number of virtual weights in SFD-Rss are essential.

[0096] In order to analyze the effectiveness of adaptive time constraints, Figure 4 The model training time and accuracy of three benchmark datasets under two data distributions are given in Figure 4 As shown, SFD-Rss has good performance in terms of time and time consumption, especially Figure 4 On c in the figure, SFD-Rss takes 663 minutes less time than LDP-FedAvg and 9 minutes more time than DP-FedAvg2 on both data distributions. The model accuracy is 0.36 and 0.21 higher than LDP-FedAvg, and 0.04 and 0.06 lower than DP-FedAvg2. On less complex datasets (such as FMNIST and MNIST), it can be Figure 4 It is observed that this advantage gradually decreases when comparing a and b in the example. Specifically, on the MNIST dataset, there is no significant difference in model accuracy between the adaptive time and fixed time submission of model weights. This is because the model on CIFAR-10 is deeper and more complex, so it takes more time to perform local random perturbations on all model weights than to perform Top-k local random perturbations. In this case, the shuffler has to dynamically extend the time to accept weights uploaded by the client, which will increase the entire training time of the model. Top-k sampling can ensure that the shuffler receives the weight information from the client in succession within the effective time, thereby avoiding excessive extension of the submission time. This also explains why adaptive time constraints and Top-k sampling are required on complex datasets and neural networks.

[0097] In order to analyze the impact of privacy budget on model performance and privacy, two ∈ lThe value of is used, and experiments are conducted on two data distributions of three datasets: MNIST, FMNIT, and CIFAR-10.

[0098] Table 1 The effect of ∈ l on model

[0099]

[0100] As shown in Table 1, l =506.18 After introducing the two protocol models, we can see that the privacy guarantee after SFD-Ras amplification reaches ∈ c =2.68, while SFD-Rss can achieve stronger privacy guarantee∈ c =0.37. Under the IID and Non-IID distributions of MNIST, the SFD-Rss model achieved 0.23 and 0.26 higher accuracy than SFD-Ras, respectively. When the model was validated on Cifar-10, the privacy amplification effect became even more pronounced, but the model's accuracy decreased. This may be due to the high complexity of the model and the large number of weight parameters, which caused the fixed Top-k weight sampling perturbation to deviate from the true weight data, leading to a loss in accuracy. It can be assumed that for more complex neural networks and datasets, the fixed Top-k sampling perturbation will affect model accuracy to a certain extent.

[0101] The above experimental data further verifies that the random participation of the client and the Top-k sampling perturbation scheme of the local randomizer in this case can ensure that the model adapts to the real training environment and reduce the privacy budget after high-dimensional model aggregation; in addition, the dynamic time constraint and virtual data padding of the shuffler can effectively alleviate the long model training time caused by external reasons such as network congestion, and improve the privacy leakage problem caused by insufficient collection of weights, which facilitates the application in distributed data protection scenarios.

[0102] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0104] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0105] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0106] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A privacy protection method for federated learning based on a random participation differential privacy shuffling model, characterized by: Include: The client uses the local randomizer to generate local weight data for this round of joint training and uploads it to the shuffler, where the number of clients is set to n, and n is greater than 1; The shuffler performs a random shuffle operation on the received local weight data and transmits the shuffled local weight data to the analyzer within a fixed constraint time. When the shuffler performs a random shuffle operation on the received local weight data, the fixed constraint time for the shuffler to perform the random shuffle operation on the received data is set based on the current round of joint training time and the time required for each client to complete the current round of joint training and share the current round of local model weight data. The analyzer aggregates the received local weight data, uses the aggregated weights to update the global model parameters for the next round of joint training, and broadcasts the updated global model parameters to all clients.

2. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 1 is characterized in that: The client uses the local randomizer to generate local weight data for this round of joint training, including: First, the probability of each client independently exiting the training in this round of joint training is set, and the random participation rate of each client in this round of joint training is dynamically adjusted using the probability of each client independently exiting the training. Next, the clients participating in this round of joint training are determined based on the random participation rate. The local training of this round is completed on the local private dataset using the clients participating in this round of joint training, and the model weight parameters obtained after training are trimmed. Then, a local randomizer is used to add perturbation noise to the pruned local model weight parameters and uploaded to the shuffler via encrypted transmission.

3. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 2 is characterized in that: The process of dynamically adjusting the random participation rate of each client in this round of joint training is expressed as: β = p(1-p′), where p is the original random participation rate of the client independently determined to participate in this round of training, p′ is the set probability of the client independently exiting training, and β is the new random participation rate of the client after dynamic adjustment.

4. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 2 is characterized in that: When using a local randomizer to add perturbation noise to the weight parameters of the pruned local model, the weight parameters to which the perturbation noise is added are limited to the most important k dimensions in the local model.

5. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 4 is characterized in that: Use the local randomizer to add perturbation noise to the pruned local model weight parameters, including: First, the local randomizer uses the Top function to obtain the query results of the received local model weight parameters, and obtains the most important k index sets and non-important index sets according to the size of the query results; Next, add disturbance noise to the weight elements in the most important k index sets by traversing the weight index; Then, the local model weight parameters are updated, all elements in the most important k index sets and the unimportant index sets are added to the local model weight, and the updated local model weight parameters and client ID are encrypted and uploaded.

6. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 2 or 5, characterized in that: The disturbance noise is Laplace noise or Gaussian noise.

7. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 1 is characterized in that: The shuffler performs random shuffling on the received data within a preset shuffle interval, including: First, the initial time for the shuffler to transmit weight data is obtained based on the time it takes for each client to complete the current round of joint training and share the current round of weights, as well as the number of clients participating in the current round of joint training; Then, when the shuffler receives the weight data uploaded by a valid client, it adds the time required for the client to complete this round of joint training and share this round of weights to the initial time or this training time to extend the weight data transmission time, and transmits the local model weight data of this round of joint training to the analyzer within the extended weight data transmission time.

8. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 1 is characterized in that: The shuffler performs a random shuffling operation on the received data within a preset shuffling time interval, and also includes: if the extended weight data transmission time exceeds the final submission time and the number of client local model data received by the shuffler does not meet the minimum number of clients participating in the joint training in this round, the shuffler uses the global model parameters broadcast by this round of training to virtually fill the local model data, so that the number of filled client local model data meets the minimum number of clients in this round of joint training, and uses the virtual client ID of the non-participating training to encrypt the virtually filled local model data during encrypted transmission. The final submission time is set based on the number of clients and the time required for each client to complete the joint training in this round and share the weights in this round.

9. The privacy protection method for federated learning based on the random participation differential privacy shuffling model according to claim 1 is characterized in that: When the analyzer performs aggregate evaluation on the received local weight data, it first decrypts the encrypted data, excludes invalid client ID weight data, obtains valid client ID local weight data, and then aggregates the valid client ID local weight data.

Images