Adaptive top-k perturbation shuffle federated learning method and system based on differential privacy

Abstract

The present application belongs to the technical field of data privacy protection, and particularly relates to a self-adaptive Top-k perturbation shuffle federated learning method and system based on differential privacy. An analyzer initializes a business model weight and Top-k, and sends data to each data party. Each data party updates a local local model using the model weight parameter, trains the local local model using local data, randomly disturbs the Top-k dimension model parameter, and sends the parameter to a shuffler. The shuffler performs double disturbance processing on the model weight parameter, sends the weight parameter after shuffling to the analyzer, and the analyzer aggregates the received model weight parameter to generate a global model weight and a new Top-k, and broadcasts the global model weight and the new Top-k to the data party participating in the next round of training. The present application changes the disturbance of the Top-k model parameter of the client and the shuffler by dynamically adjusting the size of Top-k, improves the model efficiency, reduces the privacy budget value, and facilitates privacy data protection in federated learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data privacy protection, and particularly relates to a self-adaptive Top-k perturbation shuffle federated learning method and system based on differential privacy. BACKGROUND

[0002] Nowadays, mobile intelligent devices will generate a large amount of private personal data every day, and how to safely and effectively use these private data in machine learning without leaking the privacy information of users has become a problem to be solved. Unlike machine learning, federated learning does not need to share its own private data with other devices, but only needs to upload its own model parameters to the server for model aggregation and updating, thereby realizing effective use of personal data in the case of non-concentrated data, and showing better model performance than machine learning in practical application scenarios such as word prediction, epidemic prediction and automatic driving. However, some studies have shown that the model parameters uploaded by the client can also leak the privacy information of users. Existing methods introduce differential privacy to protect the model parameters uploaded by the client. Through exploration and research in recent years, the existing federated learning based on differential privacy is mainly divided into two categories: centralized federated learning model (DP-FL) and localized federated learning model (LDP-FL). In the centralized federated learning model, the data collector of the server is trusted by default, and the data collector of the server uniformly performs random perturbation on the model parameters, which makes the model have high accuracy. However, how to ensure that the data collector is trusted is still a problem to be solved. In the localized federated learning model, the local client completes the perturbation of the model parameters by itself to ensure the safety of the model, and does not need to worry about whether the analyzer of the server is trusted. Although this model is safe, too much noise is added to the model, which reduces the accuracy of the model and affects the communication efficiency of the model.

[0003] Currently, a semi-honest shuffler is added between the client and the server by using the shuffling model, and the client can only obtain the disordered message by shuffling and arranging the message uploaded by the client, which not only weakens the trust degree of the client to the server, but also guarantees the security of the client message. With the in-depth study of the shuffling model, the shuffling model has achieved more results in model efficiency and model security, but the existing shuffling model research still has the following problems. First, the devices participating in each model training are thousands, which leads to a large number of model parameters generated by each training, and a part of the existing research is to disturb all the model parameters, and once the server completes the aggregation operation of all the disturbed model parameters, it will lead to the explosion of the privacy budget value ∈ after aggregation, and then affect the security of the model. Secondly, there is also a part of the research that disturbs the model parameters of the client in a fixed range, but during the global training of the model, the model parameters of the client will change dynamically, which not only leads to the loss of the importance of the parameters due to excessive disturbance, but also affects the efficiency of the model due to the inadaptation to the dynamic changes of the model parameters in the global training. SUMMARY

[0004] Therefore, the application provides a self-adaptive Top-k perturbation shuffling federated learning method and system based on differential privacy, which changes the perturbation of Top-k model parameters of the client and the shuffler by dynamically adjusting the size of Top-k, improves the efficiency of the model, reduces the privacy budget value, and facilitates the protection of private data in federated learning.

[0005] According to the design scheme provided by the application, a self-adaptive Top-k perturbation shuffling federated learning method based on differential privacy is provided, which is used for joint training of a business model by multiple participants. The multiple participants at least include an analyzer and a shuffler arranged on a server, and a plurality of data parties arranged on a client, and a plurality of data parties in each training round in the joint training are randomly selected by the analyzer on the client, and the joint training process includes the following contents:

[0006] The analyzer initializes the business model weight and Top-k, and sends the weight parameters initialized by the business model and the preset Top-k ratio to each data party selected by the analyzer in the current training round;

[0007] Each data provider updates its local model using the received model weight parameters and trains the updated local model using its local private dataset. During training, the model parameters of the Top-k dimensions of the local model are randomly perturbed, and the randomized local model weight parameters are sent to the shuffler. The randomized local model weight parameters include: weight data for each dimension of the model, Top-k dimension index dataset, and non-Top-k dimension index dataset.

[0008] For the received weight parameters, the shuffler performs privacy perturbation on the model weight parameters of the Top-k dimensions, performs Gaussian distribution perturbation on the model weight parameters of the non-Top-k dimensions, and shuffles the perturbated weight parameters before sending them to the analyzer.

[0009] The analyzer aggregates the received model weight parameters and generates global model weights and a new Top-k, and then broadcasts the global model weights and the new Top-k to the next round of data participants.

[0010] As a differential privacy-based adaptive Top-k perturbation shuffling federated learning method in this invention, the data provider further perturbs the Top-k dimension model parameters of the local model during training by using a local randomization perturbation mechanism to perturb the Top-k dimension parameters in the model. The local randomization perturbation mechanism dynamically adjusts the Top-k size in each iteration.

[0011] As a differential privacy-based adaptive Top-k perturbation shuffling federated learning method in this invention, a local randomization perturbation mechanism is further implemented. Represented as:

[0012] tw d =(tk r [1]·d[1]+...+tk r [n l ]·d[n l ]),in, tw d tk represents the local vector dimension. r [n l ]、d[n l ]、∈ l ,∈ lt They represent the nth l Sampling rate of the layer model dimension, nth l Number of layer model dimensions, total privacy budget for the model, and privacy budget after being evenly distributed among the number of models.

[0013] As the adaptive Top-k perturbation shuffle federated learning method based on differential privacy in the application, further, the shuffler firstly shuffles the Top-k dimension index data set and the non Top-k dimension index data set for the received weight parameter, and then respectively performs perturbation processing on the model weight parameter of the corresponding dimension of the shuffled Top-k dimension index data set and the non Top-k dimension index data set.

[0014] As the adaptive Top-k perturbation shuffle federated learning method based on differential privacy in the application, further, in the privacy perturbation processing of the model weight parameter of the Top-k dimension in the shuffler, the perturbation value generated in the same privacy budget is reduced according to the model weight proportion size, and the Top-k perturbation limit in the model weight parameter of the Top-k dimension is amplified.

[0015] As the adaptive Top-k perturbation shuffle federated learning method based on differential privacy in the application, further, in the privacy perturbation and Gaussian distribution perturbation processing of the shuffler, the total number of parameters of the privacy perturbation and the Gaussian distribution perturbation is amplified, so that the number of parameters after perturbation is always equal to the number of model parameters fed back by each data party, and the model weight parameter is converted into a fixed security state.

[0016] As the adaptive Top-k perturbation shuffle federated learning method based on differential privacy in the application, further, in the shuffling process of the shuffler on the weight parameter after perturbation, firstly, the model is cut according to the local model hierarchical relationship of the data party, so that the local model of all data parties is arranged in a hierarchical relationship; then, random numbers corresponding to all model levels are randomly generated, and the local model is shuffled and reorganized in a hierarchical relationship through the random numbers, so as to obtain the model weight parameter after shuffling and reorganization.

[0017] Further, the application also provides a kind of adaptive Top-k perturbation shuffle federated learning system based on differential privacy, comprising server and client for business model joint training, analysis ware and cleaner are arranged on the server, a plurality of data parties are arranged on the client, wherein,

[0018] The analyzer is used to initialize the weight of the business model and Top-k, and the weight parameter and the preset Top-k ratio of the initialized business model are sent to each data party selected by the analyzer in the current training round;And the global model weight and new Top-k are generated by aggregating the received model weight parameters, and the global model weight and new Top-k are broadcast to the data party participating in the next round of training;

[0019] Each data party is configured to update a local local model using the received model weight parameters, train the updated local local model using a local private data set, and perform randomization disturbance on model parameters of a Top-k dimension of the local local model during the training process, and send the local local model weight parameters after the randomization disturbance to a shuffler, wherein the local local model weight parameters after the randomization disturbance comprise weight data of each layer dimension of the model, a Top-k dimension index data set, and a Non-Top-k dimension index data set.

[0020] The shuffler is configured to perform privacy disturbance on model weight parameters of the Top-k dimension, perform Gaussian distribution disturbance on model weight parameters of the Non-Top-k dimension, and send the disturbed weight parameters to an analyzer after shuffling.

[0021] The analyzer is configured to randomly select each data party participating in the joint training from a client to obtain the disturbed weight parameters.

[0022] The present application has the following beneficial effects:

[0023] The present application adds noise disturbance conforming to a Laplace distribution to the most important k elements in the model parameters, reduces the privacy budget value ∈, dynamically changes the Top-k in an iterative training manner, and makes the adaptive Top-k disturbance more suitable for the dynamic changes of the model parameters. The present application further expands the privacy amplification mechanism, reduces the added Laplace noise according to the model weight proportion for the model parameters in the Top-k, adds noise conforming to a Gaussian distribution to the model parameters in the Non-Top-k using a Bernoulli mechanism, and performs privacy combination and amplification on the model parameters in the Top-k and the Non-Top-k, thereby reducing the privacy budget value ∈ and improving the model security. The accuracy of the shuffling model of the present application is 43% higher than that of the fixed federated shuffling model, only 2.32% lower than that of the federated learning model without noise, and has a good application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The figure is a schematic diagram of the adaptive Top-k disturbance shuffling federated learning joint training process in the embodiment.

[0025] Figure 2 The figure is a schematic diagram of the comparison of three structures of the federated learning shuffling model in the embodiment.

[0026] Figure 3 The figure is a schematic diagram of the adaptive Top-k federated shuffling model architecture in the embodiment.

[0027] Figure 4 The figure is a schematic diagram of the model test performance under the same sampling client proportion in the embodiment.

[0028] Figure 5 The model training loss and accuracy under different data distributions of different data sets in the embodiment are shown, wherein (a) is the training loss, and (b) is the accuracy;

[0029] Figure 6 The influence of changing the Top-K ratio on the model accuracy under the data distribution in the embodiment is shown.

[0030] Figure 7 The influence of changing the Top-K ratio on the model accuracy under the data distribution in the embodiment is shown. l The influence of changing the Top-K ratio on the model accuracy under the data distribution in the embodiment is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application more clear, specific and apparent, the present application will be further described in detail below with reference to the drawings and technical scheme.

[0032] The shuffle model is a very promising federated learning method, which not only has high accuracy of central federated learning, but also has security of local federated learning. Referring to Figure 2 As shown in the figure, (a) is a DP-FL model, which realizes unified perturbation and aggregation operation of the model parameters of the client on the server side, and the client does not need to perform any perturbation operation; (b) is an LDP-FL model, which completes the local perturbation operation of the model parameters by the client itself, and the server side completes the model aggregation operation; (c) is a shuffle model, which completes the local perturbation operation of the model parameters by the client itself, and completes the shuffle arrangement operation of the model by the shuffler, and finally completes the aggregation operation of the model by the server. Although the shuffle model successfully balances the privacy security and model efficiency in federated learning, the existing shuffle model research still has the following problems: one is to perturb all model parameters, and only part of the parameters in all model parameters is important, which leads to the problem of rapid increase of privacy budget value. The other is to perturb the model parameters within a fixed range, and this part of the parameters loses the importance of the parameters themselves due to excessive addition of perturbation, resulting in a decrease in model efficiency. Therefore, in the embodiment of the present application, an adaptive Top-k perturbation shuffle federated learning method based on differential privacy is provided, which is used for joint training of a business model by a plurality of participants, wherein the plurality of participants at least include an analyzer and a shuffler arranged on a server, and a plurality of data parties arranged on a client, and the plurality of data parties in each training round in the joint training are randomly selected by the analyzer on the client, as shown in Figure 1 The joint training process includes the following contents:

[0033] S101, the analyzer initializes the business model weight and Top-k, and sends the weight parameters of the initialized business model and the preset Top-k ratio to each data party selected by the analyzer in the current training round.

[0034] S102, each data party updates the local local model using the received model weight parameters, and trains the updated local local model using the local private data set, and randomly disturbs the model parameters of the Top-k dimensions of the local local model in the training process, and sends the randomly disturbed local local model weight parameters to the shuffler, wherein the randomly disturbed local local model weight parameters include: weight data of each layer dimension of the model, Top-k dimension index data set and non-Top-k dimension index data set;

[0035] S103, for the received weight parameters, the shuffler performs privacy disturbance on the model weight parameters of the Top-k dimensions, performs Gaussian distribution disturbance on the model weight parameters of the non-Top-k dimensions, and sends the disturbed weight parameters to the analyzer after shuffling;

[0036] S104, the analyzer aggregates the received model weight parameters and generates global model weight and new Top-k, and broadcasts the global model weight and new Top-k to the data parties participating in the next round of training.

[0037] The existing work of applying differential privacy to federated learning mainly includes two directions: DP-FL and LDP-FL. The DP-FL model defaults that the server is trusted, and it completes the random disturbance task of the model parameters. This way realizes the improvement of model efficiency, but this way also produces a security problem, who guarantees the trustworthiness of the server. The security of the LDP-FL model is completed by the client itself. This mode not only guarantees the security of the model, but also weakens the degree of trust in the server, but the model efficiency is not high due to the addition of too much disturbance.

[0038] Let D = {d1,...,d n} denote a data set containing n points from X. Two data sets D = {d1,...,d n} and D ′ = {d ′ 1,...,d ′ n} are adjacent if and only if they differ in only one data point, i.e., there exists i ∈ [n] such that d ′ ≠ d i i ′ and for each j ∈ [n], j ≠ i, we have d i = d j ′ .

[0039] ​In centrally differentiated privacy, for ∈, δ≥0, a stochastic mechanism M:X→Y is (∈,δ)-differentially private (abbreviated as (∈,δ)-DP), if for all And for all datasets And ||xx ′ When ||1≤1, the resulting differential privacy representation is: Pr[M(x)∈S]≤exp(∈)Pr[M(x) ′ )∈S]+δ.

[0040] In Localized Differential Privacy (LDP), a stochastic mechanism R:X→Y satisfies ∈-locally differentiallyprivate (abbreviated as ∈-LDP). This holds true for any two inputs d, d ′ ∈X and for any output This will result in localized differential privacy, represented as: Pr[R(d)∈S]≤exp(∈)Pr[R(d′)∈S].

[0041] In the combinatorial properties of differential privacy, assuming A1(·) is an algorithm satisfying the (∈, δ)-DP mechanism, for any algorithm A2(·) and algorithm A1(·) combined into algorithm A = A2(A2(·)), algorithm A will also satisfy the (∈, δ)-DP mechanism. Assume... If (∈1, δ1)-DP and (∈2, δ2)-DP are satisfied respectively, then Combined differential privacy mechanisms This will satisfy (∈1+∈2, δ1+δ2)-DP. The ∈-DP-mechanism will satisfy k∈-DP under k-fold adaptive combination. For all ∈, δ, δ′>0, under k-fold adaptive combination, the group (∈, δ)-DP mechanism will satisfy the (∈′, kδ+δ′)-DP mechanism, where

[0042] The ESA (Electronic Shuffling Analysis) model mainly consists of three parts: the local encoder (R), the shuffler (S), and the analyzer (A). For a clearer understanding of the ESA model, let n represent the number of clients, where n... i This represents the i-th user, using x. i User n i The dataset we have is X = (x1, ..., xn) consisting of n users. n ).use The parameters represent the model after shuffling. The ESA operating mechanism can be described as follows:

[0043] Local encoder R: User n is randomized through a randomization mechanism. iData x i The data x is disturbed as an input data to output an array containing m pieces of information If m = 1, we will get a standard single-information shuffle model, which is also the focus of this paper.

[0044] Shuffler S: After accepting the data disturbed by the user, the shuffler removes the relationship between the data and the user and adds a fixed ID to the collected data. When the collected data reaches a certain threshold, the shuffler will randomly and uniformly arrange the data to make the ordered message become a disordered state, thereby improving the accuracy of subsequent data analysis.

[0045] Analyzer A: The analyzer is mainly deployed on the server side and does not need to add noise to the model. It only needs to accept the messages sent by the shuffler S and decrypt, unbiased estimate, aggregate analysis, and publish new aggregate models to the client for training in the next round.

[0046] In the embodiment of the case, the data side of the client randomly disturbs the Top-k dimensional model parameters of the local model during the training process, and uses a local randomization disturbance mechanism to disturb the Top-k dimensional parameters in the model. The local randomization disturbance mechanism dynamically adjusts the Top-k size in each iteration process. For the received weight parameters, the shuffler can first shuffle the Top-k dimensional index data set and the non-Top-k dimensional index data set, and then disturb the model weight parameters of the corresponding dimensions of the shuffled Top-k dimensional index data set and the non-Top-k dimensional index data set, respectively. Referring to Figure 3 As shown in the figure, in the scheme, an iterative adaptive Top-k federated shuffle model based on differential privacy can be used to realize joint training. The model architecture mainly consists of three parts: client (local training), shuffler, and server (analyzer). 1) The client obtains the initial model parameters transmitted by the server, then calls its own private data set for local model training, and uses a local randomization mechanism to adaptively disturb the trained model parameters. The operation can be simplified as: 2) Shuffler Accepts the model parameters uploaded from the client, and respectively performs a second privacy disturbance on the Top-k model parameters and a disturbance operation conforming to the Bernoulli distribution on the Non-Top-k. Finally, the expanded second privacy amplification technology is used to complete the mapping amplification processing of all parameters, which can be written as: 3) Analyzer Without adding any noise, the server analyzer will also satisfy the differential privacy. Therefore, Its main function is to receive The model weights are passed in, and these weight data are analyzed, aggregated, and updated with new Top-k values. Finally, The aggregated model weights and the new Top-k values ​​will be broadcast to the next round of clients participating in training. Therefore, the protocol can be used. This section summarizes the main mechanism and algorithm of the AdaSTopK model. The algorithm content is shown in Algorithm 1.

[0047]

[0048] In Algorithm 1, firstly, the analyzer... The model weights W will be initialized. 0 In each global iteration of training, The number of clients participating in the t-th round of training will be randomly selected from all clients n. Then, parameter W 0 ,tk r Broadcast to selected clients. During client-side parallel training, the ith-th client participating in the t-th round of global training... Model weights W will be used. t-1 The model is trained locally using SGD optimization on its own private dataset. This is to ensure client-side performance. The security of the original model weight data, The model's Top-k dimension parameters undergo local randomization perturbation in line 12 of the algorithm (RandomTopkData()). After the model randomization perturbation is completed, the client will upload the average model parameters of the perturbation. to the washing machine Perform a shuffling operation. In the ShuffleParameter(·) section of line 15 of the algorithm, we perform two privacy perturbations and shuffling operations on the client's weight data. Finally, Send the shuffled model data to the designated location. Perform private aggregation of the model and update tk for the next training iteration. r And send these parameters to the clients participating in the next round of global training.

[0049] In federated federated learning process, the data side of the client trains its own model by experience risk minimization, and uploads the trained model to the server to obtain the aggregated model, so that the model is more suitable for application in complex scenarios. During training, thousands of devices use the same model to generate, so the size and shape of the model parameters are uniform. However, due to the complexity of neural networks and the inconsistency of client datasets, the model parameters generated by device applications are also different. Once the same fixed method is used to perform perturbation on the model parameters of the device, it will cause the accuracy of the aggregated model to decrease, and also disturb the specificity of each device's own model. In addition, the number of model parameters generated by thousands of devices in one training is huge, and if local perturbation is performed on all parameters, it will cause the privacy budget value to increase after combination. Therefore, in the embodiment of the case, the method of changing the size of Top-k iteratively is used to cope with the above situation, which can not only adapt to the change of the model parameters of each client in the iterative training, but also ensure the accuracy of the model. Moreover, this method only performs dynamic perturbation on the Top-k dimensional parameters in the model parameters, so as to reduce the privacy budget value ∈ of the aggregated model. Local randomization perturbation The pseudo code of the local randomization perturbation mechanism is shown in Algorithm 2.

[0050]

[0051] When the local randomization perturbation mechanism The model weight of the ith client in the tth round is obtained After that, we will get the model d-dimensional weight data of each layer, as well as the data set of Top-k, Non-Topk dimensional index through iteration However, only The model parameters corresponding to the index are perturbed locally according to the Laplace mechanism Finally, the perturbed model parameters And the chaotic Top-k, Non-Topk set are sent to the shuffler, which avoids the shuffler directly operating the model data. And for any user's tw d The data in the vector will satisfy ||D-D' ||1≤1 if and only if there is only one data different in the tw After the local randomization mechanism lt -LDP, where: The instantiation form of the local randomization perturbation mechanism can be expressed as: tw d = (tk r [1]·d[1]+...+tkr [n l ]·d[n l ])。

[0052] Further, in the privacy disturbance processing of the model weight parameters of the Top-k dimension of the shuffler of the embodiment, the disturbance value generated in the same privacy budget is reduced according to the size of the model weight proportion, and the Top-k disturbance limit in the model weight parameters of the Top-k dimension is amplified. Further, the total number of parameters of the privacy disturbance and the Gaussian distribution disturbance is amplified, so that the number of disturbed parameters is always equal to the number of model parameters fed back by each data party, and the model weight parameters are converted into a fixed security state.

[0053] Although the iterative adaptive Top-k disturbance mechanism in Algorithm 2 reduces the privacy budget value, the proportion of the client randomizing and disturbing the Top-k dimension parameters depends on the size of tk r . Moreover, the size of tk r will be dynamically adjusted with the iterative training, and this adjustment will cause the number of randomly disturbed Top-k to change, thereby affecting the security of the model. For example, in the extreme case of tk r being extremely small or tk r = 0, the client will add a small amount of random disturbance or no disturbance to the Top-k model parameters, which is equivalent to directly uploading the original model data of the user to the server, ultimately leading to the leakage of the user's privacy data. To deal with this risk, the shuffler part of the embodiment can use two privacy technologies. One is to add corresponding disturbance to the model parameters in Top-k according to the proportion of the parameter weight, which ensures that the added disturbance does not affect the weight of the parameter. In addition, for the If satisfies ∈ l -LDP, for we will get (∈ c , δ c ), where Based on the above, the disturbance amplification limit of the Top-k parameters can be achieved. The other is to use the Bernoulli distribution idea to only execute the disturbance conforming to the Gaussian distribution on the model parameters with Pr[b n == 1]. Finally, we combine Top-k and Non-Topk to complete the double privacy disturbance of the entire model. Since the model is a dynamic iterative process, this way will cause the number of Top-k and Non-Topk in each disturbance to be inconsistent. If satisfies (∈, δ)-DP, the replacement relationship on the data set with a size of m is will be satisfied where m < n, the total number of Top-k, Non-Top-k perturbations can be amplified, and the number of double privacy perturbations can always be equal to the number of each client model parameters. In this way, it ensures that the client can add less noise at the same differential privacy level, and guarantees the accuracy of the model after aggregation. For the shuffler The double privacy perturbation is performed, and the specific algorithm can be shown in Algorithm 3.

[0054]

[0055] The model weight is obtained by the iteration technique and the chaotic Top-k weight index set S top and the chaotic Non-Top-k index set S non . Then, the corresponding perturbation mechanism is performed on the model parameters represented by S top and S non . For the model parameters in S top , because the second privacy perturbation is performed on all the model parameters in S top , the model parameters in S top may lose their own important value. In order to reduce this risk, in the embodiment, the ratio is used to reduce the perturbation value generated by the same privacy budget. And the privacy guarantee can be realized for the model parameters in S top . The privacy protection content can be embodied as: when δ cd <2tk r , the shuffling mechanism will satisfy (∈ ct , δ ct )-DP, wherein:

[0056]

[0057] Considering the δ constraint in differential privacy, it can be set to a reasonable constraint δ ct <2tk r , because when , it can be ignored.

[0058] In S non , with the increase of the proportion of S non , the attacker will obtain part of the privacy information of the user by analyzing the index set S non . In the face of this situation, the corresponding model parameters in S non can be added with differential privacy noise conforming to the Gaussian distribution to ensure the security of this part of the model parameters. When S top , the local client uploads S non , which means that the model parameters uploaded by the client do not perform local randomization perturbation to ensure the security of the model. Therefore, all model parameters uploaded by the client can be subjected to noise perturbation to ensure the security of the model. When , the idea of Bernoulli distribution is followed, and only the model parameters of S are subjected to Gaussian perturbation, which reduces the size of the privacy budget value in the index set S non . The privacy guarantee process of S non can be described as follows: for any neighboring data sets ||D-D ’ ||1≤1, if users all satisfy (∈ sn ,δ sn )-DP, two kinds of privacy mechanisms in the user group will respectively satisfy (∈ n1 ,δ n1 )-DP, (∈ n2 ,δ n2 )-DP, where:

[0059]

[0060]

[0061]

[0062] After performing noise perturbation on the parameters in S top , S non , it is found that due to the iterative change of tk r , the total number of perturbed model parameters in S top , S non is also dynamically changing, which can infer the degree of change of tk r and the size of the differential privacy budget value affected by each time to a certain extent. In order to avoid the insecurity caused by this dynamic change and the factors affecting the change of the privacy budget value, the total number of perturbed parameters in S top , S non is subjected to privacy amplification operation, so that the total number of perturbed parameters after amplification is always equal to the number of model parameters. If privacy amplification is performed on the data subjected to privacy perturbation, the amplified privacy is obtained: δ cd = β(δ ct + δ n1 ). Since all model parameters are subjected to noise perturbation, only the data at can be subjected to privacy amplification processing.

[0063] After the model parameters undergo double privacy perturbation, the model is guaranteed to be in a dynamically safe environment. Combined with privacy amplification operations, this can be further transformed into a fixed safe state. The complete privacy guarantee of double privacy perturbation can be described as follows: For any neighboring dataset ||D-D'||1≤1, the user in their own... Dimensional model fusion mechanism The general satisfies (∈ c ,δ c )-DP, where:

[0064]

[0065]

[0066] δ c =δ cd (k+1)

[0067] In a preferred embodiment, the shuffler further shuffles the perturbed weight parameters by first trimming the model according to the hierarchical relationship of the local models of the data parties, so that the local models of all data parties are arranged in a hierarchical relationship; then, it randomly generates random numbers corresponding to all model levels, and uses the random numbers to shuffle and recombine the local models in a hierarchical relationship to obtain the shuffled and recombined model weight parameters.

[0068] Even after the dual privacy perturbation ends, the local model parameters uploaded by the client still retain some records that can identify the client, such as IP address and VPN information. This information can, to some extent, verify which client uploaded the model, creating opportunities for attackers. In this situation, the model can be pruned according to the client's model hierarchy, ensuring all models exist in a hierarchical relationship. Then, by randomly generating... The model uses random numbers to correspond to all model layers, and then shuffles and reassembles the models according to their hierarchical relationships using a random array to form new models. This method ensures that the adversary obtains mixed model parameters, and they also have to spend a lot of time and effort to ensure correct model matching. For example, suppose the client has n data parties, and each data party has w model layers. n Therefore, the range of the opponent's correct model probability is within [number]. This approach once again ensures the safety of the model.

[0069] Further, based on the above method, the embodiment of the application also provides a differential privacy-based adaptive Top-k perturbation shuffle federated learning system, which comprises a server and a client for business model joint training, the server is provided with an analyzer and a cleaner, and the client is provided with a plurality of data parties, wherein,

[0070] The analyzer is used for initializing a business model weight and a Top-k, sending the initialized weight parameter of the business model and a preset Top-k ratio to each data party selected by the analyzer in a current training round, and generating a global model weight and a new Top-k by aggregating the received model weight parameters, and broadcasting the global model weight and the new Top-k to data parties participating in the next round of training;

[0071] Each data party is used for updating a local local model by using the received model weight parameters, training the updated local local model by using a local private data set, and randomly perturbing model parameters of a Top-k dimension of the local local model in the training process, and sending the randomly perturbed local local model weight parameters to the shuffler, wherein the randomly perturbed local local model weight parameters comprise weight data of each layer dimension of the model, a Top-k dimension index data set and a non-Top-k dimension index data set.

[0072] The shuffler is used for performing privacy perturbation on the model weight parameters of the Top-k dimension, performing Gaussian distribution perturbation on the model weight parameters of the non-Top-k dimension, and sending the perturbed weight parameters to the analyzer after shuffling.

[0073] The data parties participating in the joint training are randomly selected by the analyzer on the client.

[0074] To verify the effectiveness of the scheme, the following experimental data are used for further explanation and description:

[0075] The most commonly used image data sets MNIST, Fashion-MNIST (FMNIST) and Cifar-10 are used. Considering the complexity and disorder of data distribution in the real world, the data set is divided into three kinds of data distribution, namely, independent and identically distributed (iid), non-independent and identically distributed (Non-iid1) and non-independent and differently distributed (Non-iid2). Under the three kinds of data distribution, relying on the high efficiency provided by the GPU NVIDIA RTX3080 Ti, a large number of experiments are carried out in the Linux environment of the local server to explore the specific performance of the AdaSTopk model in the actual situation, and the FedAvg model is used as the baseline model.

[0076] 1. Communication efficiency analysis

[0077] In order to explore the communication performance of the AdaSTopk model in the scheme in the MNIST, Fashion-MNIST (FMNIST), Cifar-10 three data sets and the iid, Non-iid1, Non-iid2 three data distribution, the loss rate and the accuracy of the model are taken as the evaluation indexes to analyze the performance of the AdaSTopk model.

[0078] Some parameters of the experiment are set, such as setting E=20 as the period of a global training, and the learning rate lr=0.005. For the proportion of clients participating in each global training, starting from f r =0.1, the proportion of clients participating in each global training is gradually increased by a step size of 0.1 until f r =1. According to the results of Figure 4 , it can be seen that in the data sets FMNIST, MNIST, iid, Non-iid2 data distribution, the size of the proportion of data in the client participating in the training has little effect on the FedAvg and AdaSTopk models, but in the Non-iid1 data distribution, the two models will produce obvious fluctuations. In the data sets CIFAR-10, iid, Non-iid1, Non-iid2 data distribution, the size of the proportion of clients participating f r has no obvious change on the overall performance of the two models. However, when f r =1, the performance of the AdaSTopk model is better than that of the FedAvg.

[0079] In order to further study the performance of the model in the whole communication process, the whole communication process of the model is detected, as shown in Figure 5 , and the communication process of the model is shown in Figure 5The test loss (a) and accuracy (b) in the table show that the AdaSTopk model with security can achieve similar performance efficiency as the FedAvg model. Moreover, under the iid data distribution of the CIFAR-10 dataset, the AdaSTopk model not only maintains almost the same loss as the FedAvg model but also exceeds the FedAvg model in accuracy for the first time. In addition, under the iid, Non-iid2 distribution of the MNIST, FMNIST datasets, the performance gap between the AdaSTopk model and the FedAvg model is basically maintained within [0.05, 0.1]. Under the Non-iid data distribution, the performance of the AdaSTopk model lags behind the FedAvg model, especially under the Non-iid1 distribution of the MNIST dataset. It is not difficult to see that under the Non-iid1 distribution, the performance gap between the AdaSTopk model and the FedAvg model will gradually narrow down as the communication rounds continue. However, due to the complexity of the Non-iid1 data distribution and the task of the client, the model needs more communication rounds to achieve better performance.

[0080] 2. Top-k proportion analysis

[0081] To test the impact of the initial tk r value on the model performance, we compared the impact of fixed Top-k and adaptive Top-k on the model accuracy under three data distributions, as shown in Table 2. Figure 6 When tk r > = 0.5, the adaptive Top-k setting outperforms the fixed Top-k setting in terms of performance improvement on the iid, Non-iid2 distribution of the CIFAR-10, FMNIST, and MNIST datasets. For example, when the initial value tk r = 1.0, the adaptive Top-k model improves the accuracy of the fixed Top-k model by 0.07 and 0.06 on the CIFAR-10 dataset. When the initial value tk r = 0.8, the adaptive Top-k model improves the accuracy of the fixed Top-k model by 0.06 and 0.18 on the FMNIST dataset. When the initial value tk r = 1.0, the adaptive Top-k model improves the accuracy of the fixed Top-k model by 0.28 and 0.44 on the MNIST dataset. Under the Non-iid1 data distribution, the performance improvement of the model is not as fast as that under the other two data distributions, but overall, the accuracy of the adaptive Top-k model is still better than that of the fixed Top-k model, especially in the MNIST dataset, where this overall effect is more prominent.

[0082] With tkr The smaller the adaptive Top-k model is, the more the advantages of the model on the iid, Non-iid1, Non-iid2 data distribution disappear. Although there are individual cases where the accuracy of the adaptive Top-k model is greatly improved, the overall model performance is poor. The reason for this is mainly because the neural network model causes the dimension data of the model Top-k weight to change greatly when processing complex tasks. In this case, since the fixed Top-k always disturbs the model weight according to the size of tk r , but in the present scheme the value of tk r is always adjusted continuously with the communication between the client and the server, which causes the disturbance data added by the adaptive Top-k model to be always higher than that added by the fixed Top-k. Moreover, in actual application, the Top-k dimension data cannot be too small in one communication process, otherwise it will cause the model weight parameter to leak too much private information. Therefore, the selection of the AdaSTopk model in the present scheme on the Top-k dimension ratio is dynamically changed according to the size of tk r . This way well guarantees that no matter how large the initial value of tk r is, the AdaSTopk model will be in a safe state. Although the model accuracy is slightly worse when tk r <0.5, the model is safe, especially when tk r >0.5, the model in the present scheme achieves a better balance between security and accuracy.

[0083] 3. Privacy budget value analysis

[0084] Taking E=20 as a cycle of global training to test the model accuracy performance of the model under the condition of ∈ l ∈[10,100] on three kinds of data distribution, as shown in Figure 7 , it can be seen that with the change of privacy budget value ∈ l , the model performance of AdaSTopk model on iid, Non-iid1, Non-iid2 data distribution is also more and more close to FedAvg model. When ∈ l >30, the accuracy of the model gradually improves, especially on the data sets CIFAR-10 and MNIST under the iid, Non-iid2 data distribution, the accuracy of the model is rapidly improved, and when ∈ l= 90, the model first exceeds FedAvg on Non-iid2 of FMNIST. Although the accuracy of the model is not as good as the other two data distributions, it is still on the rise, especially on the MNIST dataset Non-iid2, the model improves the fastest. For l <30, the model performance is mainly because the disturbance value introduced by the model makes the model parameters lose their original characteristics. Moreover, when facing more complex tasks, the neural network will become more complex, causing the range of model weights to change more during the iteration process, which leads to fluctuations in the selection of model weights. Overall, under different privacy budget values, the AdaSTopk model can achieve good performance on different data distributions.

[0085] 4. The influence of iterative training on privacy budget value

[0086] Adopting c ,∈ n1 ,∈ n2 ,∈ ls as a detection index to detect the change process of the model's security in global iteration. Since the case of tk r = 0 does not appear in iterative training, only the specific changes of c ,∈ n1 ,∈ ls are given in Table 1. Before the experiment begins, some initial settings are made for the verification parameters. In CIFAR-10, the parameters are set to l = 20, f r = 0.2, tk r = 1. In FMNIST, the parameters are set to l = 80, f r = 0.2, tk r = 0.8. In MNIST, the parameters are set to l = 60, f r = 0.3, tk r = 0.8.

[0087] Table 1 Influence of communication times on privacy budget values c ,∈ n1 ,∈ 1s

[0088]

[0089] From Table 1, it can be clearly seen that in the fixed Top-k model, once the initial tk r ​The privacy budget value of the proportional model in global communication is always in a constant state. However, the adaptive Top-k in the scheme of the case is contrary to it, which dynamically adjusts tk with the progress of global communication r The proportional size further affects ∈ c , ∈ n1 , ∈ ls The actual value in each round, so as to ensure the security of the model Top-k data. In the table, it can also be obtained that ∈ c , ∈ n1 , ∈ ls The privacy budget value of the adaptive Top-k framework in the scheme of the case is less than that of the fixed Top-k. The results in the table [topk-rate] also verify that the adaptive Top-k framework in the scheme of the case can ensure that the user always satisfies a smaller ( ∈, δ )-DP through the privacy amplification mechanism.

[0090] Based on the experimental data above, it can be shown that the scheme in the case reduces the privacy budget value by performing privacy perturbation and privacy amplification techniques on the model parameters of Top-k and Non-Top-k, which can ensure that the model always satisfies a smaller ( ∈, δ )-DP in communication. Not only can it weaken the trust of the client to the server, but also it is more in line with the changes of the model weight in the communication process, at the same time, it can improve the performance of the business model, and within a certain Top-k range, it can achieve the trade-off between privacy security and model efficiency on different data distributions, which is convenient for practical application in business model joint training.

[0091] Unless specifically stated otherwise, the relative steps, numerical expressions and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0092] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0093] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0094] Those skilled in the art can understand that all or part of the steps in the foregoing method can be instructed by programs to the relevant hardware to complete, and the programs can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk. Alternatively, all or part of the steps of the foregoing embodiments can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the foregoing embodiments can be implemented in the form of hardware or in the form of a software functional module. The present application is not limited to any specific form of combination of hardware and software.

[0095] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, rather than limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. Such modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An adaptive Top-k perturbation shuffling federated learning method based on differential privacy, used for joint training of a business model by multiple participants, characterized in that... The multiple participants include at least: an analyzer and a shuffler set up on the server, and several data parties set up on the client. In the joint training, several data parties in each training round are randomly selected by the analyzer on the client. The joint training process includes the following: The analyzer initializes the business model weights and Top-k, and sends the initialized weight parameters and preset Top-k ratio to each data point selected by the analyzer in the current training round. Each data provider updates its local model using the received model weight parameters and trains the updated local model using its local private dataset. During training, the model parameters of the Top-k dimensions of the local model are randomly perturbed, and the Top-k dimension parameters in the model are perturbed using a local randomization perturbation mechanism. The randomized perturbed local model weight parameters are then sent to the shuffler. The local randomization perturbation mechanism dynamically adjusts the Top-k size in each iteration. The randomized perturbed local model weight parameters include: the weight data of each dimension of the model, the Top-k dimension index dataset, and the non-Top-k dimension index dataset. For the received weight parameters, the shuffler first shuffles the Top-k dimension index dataset and the non-Top-k dimension index dataset. Then, it perturbs the model weight parameters of the corresponding dimensions of the shuffled Top-k dimension index dataset and the non-Top-k dimension index dataset. After shuffling the perturbed weight parameters, it sends them to the analyzer. Among them, the Gaussian distribution perturbation is performed on the model weight parameters of the non-Top-k dimensions. The analyzer aggregates the received model weight parameters and generates global model weights and a new Top-k, and then broadcasts the global model weights and the new Top-k to the next round of data participants.

2. The adaptive Top-k perturbation shuffling federated learning method based on differential privacy according to claim 1, characterized in that, Local randomization perturbation mechanism Represented as: tw d =(tk r [1]·d[1]+...+tk r [n l ]·d[n l ]),in, tw d tk represents the local vector dimension. r [n l ]、d[n l ]、∈ l ,∈ lt They represent the nth l Sampling rate of the layer model dimension, nth l Number of layer model dimensions, total privacy budget for the model, and privacy budget after being evenly distributed among the number of models.

3. The adaptive Top-k perturbation shuffling federated learning method based on differential privacy according to claim 1, characterized in that, In the privacy perturbation processing of the model weight parameters in the Top-k dimension in the shuffler, the perturbation value generated in the same privacy budget is reduced according to the model weight ratio, and the Top-k perturbation boundary in the model weight parameters of the Top-k dimension is amplified.

4. The adaptive Top-k perturbation shuffling federated learning method based on differential privacy according to claim 1 or 3, characterized in that, The shuffler processes privacy perturbations and Gaussian distribution perturbations, and amplifies the total number of parameters for both perturbations and Gaussian distribution perturbations to ensure that the number of parameters after perturbation is always equal to the number of model parameters fed back by each data party, thus transforming the model weight parameters into a fixed and safe state.

5. The adaptive Top-k perturbation shuffling federated learning method based on differential privacy according to claim 1, characterized in that, During the shuffling process of the shuffling machine on the perturbed weight parameters, it first trims the model according to the hierarchical relationship of the local models of the data parties, so that the local models of all data parties are deployed in a hierarchical relationship. Then, random numbers corresponding to all model levels are randomly generated. The local models are then shuffled and recombined according to their hierarchical relationship using these random numbers to obtain the model weight parameters after shuffling and recombining.

6. An adaptive Top-k perturbation shuffling federated learning system based on differential privacy, comprising a server and a client for joint training of business models, characterized in that, Based on the method described in claim 1, the server is equipped with an analyzer and a cleaner, and the client is equipped with multiple data providers, wherein... The analyzer initializes the business model weights and Top-k, sends the initialized weight parameters and preset Top-k ratio to each data party selected by the analyzer in the current training round, and generates global model weights and a new Top-k by aggregating the received model weight parameters, and broadcasts the global model weights and the new Top-k to the data parties participating in the next round of training. Each data provider is used to update its local model using the received model weight parameters, and to train the updated local model using its local private dataset. During training, the model parameters of the Top-k dimensions of the local model are randomly perturbed, and the randomized local model weight parameters are sent to the shuffler. The randomized local model weight parameters include: the weight data of each dimension of the model, the Top-k dimension index dataset, and the non-Top-k dimension index dataset. The shuffler is used to perform privacy perturbation on the model weight parameters of the Top-k dimensions, perform Gaussian distribution perturbation on the model weight parameters of the non-Top-k dimensions, and shuffle the perturbated weight parameters before sending them to the analyzer. In this joint training, the data sources are randomly selected by the analyzer from the client.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is configured to execute a program stored in memory and, when the program is executed, implement the steps of the method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 5.

Images