Federal learning privacy protection method based on generative adversarial network and knowledge distillation

Abstract

The invention discloses a federated learning privacy protection method based on a generative adversarial network and knowledge distillation, and relates to the technical field of privacy protection. Comprising the following steps: S1, local model training; s2, generative adversarial network training; s3, communication in an initialization stage; s4, carrying out distillation training; s5, communication at a distillation stage; and S6, updating the client. According to the method, the generative adversarial network and the knowledge distillation technology are creatively combined, the generative adversarial network model is locally trained at the client, the synthetic data conforming to private data distribution is generated, dependence on a public data set is avoided, knowledge distillation is performed by using the generated synthetic data, and the knowledge distillation efficiency is improved. Knowledge migration without selecting a public data set is realized, and direct exposure of private data distribution is effectively avoided, so that the risk of privacy disclosure is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection technology, and in particular to a federated learning privacy protection method based on generative adversarial networks and knowledge distillation. Background Art

[0002] In recent years, with the widespread application of large-scale data sets, machine learning technology has achieved rapid development. However, data is usually stored in different geographical locations, which poses a major challenge to centralized data management. In addition, the data may contain sensitive information. Centralized storage not only increases the risk of data leakage, but may also infringe on personal privacy. Therefore, when processing training data containing sensitive information, traditional centralized machine learning methods may cause serious privacy leakage problems.

[0003] As a practical machine learning framework, federated learning can effectively solve the data island problem and reduce the risk of privacy leakage. The framework allows data to remain locally, and usually only the model weights or gradient parameters trained on local data need to be uploaded. By aggregating model parameters from different clients, federated learning achieves the goal of distributed training. However, since the model is trained based on local data, the uploaded parameters are closely related to local privacy information. Existing studies have shown that these parameters are extremely vulnerable to attacks during the upload process, such as model inversion attacks and inference attacks. Attackers can infer data privacy from the transmitted model parameters or gradients, leading to serious privacy leakage risks.

[0004] To further enhance privacy protection, existing methods combine federated learning with knowledge distillation technology, aggregating the prediction results of each participant by making predictions on large-scale public datasets. This method aims to achieve knowledge transfer and model updates by utilizing non-sensitive public datasets rather than directly sharing model parameters. However, the quality of public datasets is crucial to the effectiveness of knowledge distillation. Selecting high-quality public datasets usually requires understanding the distribution of private data or related information, and this selection process itself may bring the risk of privacy leakage. To solve this problem, data-free federated distillation provides a new idea, which generates synthetic data through generative adversarial networks (GANs) to avoid dependence on public datasets. However, existing data-free knowledge distillation methods still need to upload model parameters when updating generative adversarial networks, which is vulnerable to attacks. Therefore, the privacy protection problem of federated learning based on generative adversarial networks and knowledge distillation has not been fundamentally solved.

[0005] Generative adversarial networks (GANs) and knowledge distillation techniques are widely used in federated learning to achieve privacy protection and efficient model training. By using GAN to generate synthetic data and combining it with knowledge distillation techniques, the federated learning framework is able to perform model updates and knowledge transfer without directly accessing user private data. However, existing privacy protection methods based on GAN and knowledge distillation have the following shortcomings:

[0006] First, existing methods do not pay enough attention to privacy protection during the distillation process. Although these methods generate synthetic data through GAN, avoiding the risk of privacy leakage caused by relying on public datasets, they usually still require uploading the prediction parameters of the local model to update GAN, a process that may also lead to privacy leakage; second, existing privacy-preserving knowledge distillation methods mostly focus on selecting samples with the most information, while ignoring the accuracy and quality of soft labels. These methods fail to effectively solve the problem of balancing privacy protection and model utility.

[0007] Therefore, in order to solve the privacy problem in federated learning, it is necessary to explore and develop a new privacy protection framework. This framework should use GAN to generate synthetic data for knowledge distillation, and deploy GAN locally to avoid the risk of privacy leakage during parameter uploading. In addition, it is necessary to study a sampling strategy that takes into account both sample information and label quality to control the growth of the privacy budget and maximize the utility in the knowledge distillation process.

[0008] In summary, proposing a privacy protection method for federated learning based on generative adversarial networks and knowledge distillation to solve the difficulties existing in the existing technology is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0009] In view of this, the present invention provides a privacy protection method for federated learning based on generative adversarial networks and knowledge distillation to solve the problem of privacy leakage in the federated learning process and achieve a balance between privacy protection and model utility.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A privacy protection method for federated learning based on generative adversarial networks and knowledge distillation, comprising the following steps:

[0012] S1. Local model training: Initialize local model parameters, perform local model training based on local data sets of multiple clients, and obtain a trained local model;

[0013] S2. Generative adversarial network training: The trained local model is used as the discriminator to perform adversarial training with the generator to generate new synthetic sample data;

[0014] S3. Initialization phase communication: Each client randomly selects the same number of synthetic sample data for noise processing, uploads the noisy synthetic sample data to the server for sample merging, and obtains a public data set. Multiple clients use the public data set to generate prediction labels and send the prediction results to the server again;

[0015] S4. Distillation training: Multiple clients perform knowledge distillation training based on a public dataset to extract knowledge features;

[0016] S5. Communication in the distillation phase: Multiple clients perform balanced sampling of the extracted knowledge features, select prediction labels and synthetic data for noise processing, and upload the noise-processed prediction labels and synthetic data to the server. The server averages the prediction label results and merges the synthetic data of the clients to obtain the average prediction label result, and sends the average prediction label result to multiple clients.

[0017] S6. Client update: Multiple clients execute S2, S4, and S5 in a loop based on the average predicted label results to update the local model.

[0018] The above method is optional. Each client in S1 uses its own local data set to perform local model training by minimizing the local loss function:

[0019]

[0020] Among them, w i is the local parameter initialized by each client, L(D i ,w i ) is the client i in the dataset D i The loss function on is the updated parameter;

[0021] Using the stochastic gradient descent optimization algorithm on each client, update the local parameter w initialized by each client i , for each iteration t, w i The update formula is:

[0022]

[0023] Where η is the learning rate, is the local loss function L(D i ,w i ) for parameter w i The gradient of

[0024] On each client i, the local model will be trained for multiple rounds of iterations, each round using the dataset D iThe weights are updated using a batch of samples in until the preset convergence condition is reached.

[0025] In the above method, optionally, the training of the generative adversarial network in S2 is to set the generator locally on the client and directly perform adversarial training with the trained local model as the discriminator, without uploading the updated parameters to the server;

[0026] The discriminator is a trained local model network, and the generator is the same initialized model. Let p data (x) = D i represents the real data set, z represents the noise distribution from the prior z The noise vector sampled in (z) is generated by the generator by mapping z to the data space G(z);

[0027] During the training process, the goal of the discriminator is to maximize its ability to correctly classify real data and generated data, that is, to maximize the following objective function:

[0028]

[0029] At the same time, the goal of the generator G is to deceive the discriminator so that it judges the generated data as real data, that is, to minimize or maximize the following objective function:

[0030]

[0031] The above method optionally uses the Gaussian noise method to add noise to the model prediction label e uploaded by the client. k and synthetic data d k Adding Gaussian noise, the formula is as follows:

[0032]

[0033] where s is the sensitivity of f, and It means that the center is 0 and the variance is σ 2 Gaussian normal distribution.

[0034] In the above method, optionally, the steps of distillation training in S4 are as follows: first, the synthetic data after aggregation, balanced sampling and noise addition are uploaded to the server side as a public data set for knowledge distillation; second, the predicted labels after balanced sampling and noise addition are uploaded to the server side for averaging.

[0035] In the above method, optionally, the specific content of balanced sampling of the extracted knowledge features by multiple clients in S5 is:

[0036] According to Renyi entropy, the amount and quality of knowledge are evaluated. First, the Renyi entropy of the predicted label is calculated, and the synthetic data corresponding to the predicted label with large Renyi entropy is selected, and the predicted label with large Renyi entropy is removed. Let the predicted label be e i =(p i1 ,p i2 ,..,p ij ) and is evaluated based on the following equation:

[0037]

[0038] Among them, α>0 and α≠1.

[0039] The above method optionally classifies the predicted labels into two categories based on the calculated Renyi entropy of the predicted labels. The classification result is represented by C, one of which includes accurate and clear label knowledge E l , and the other category covers fuzzy label knowledge E h , the specific classification formula is as follows:

[0040]

[0041] Among them, τ represents the threshold;

[0042] After selecting the label knowledge, we further screen out the datasets containing rich information, including E h The corresponding synthetic data s h And the newly generated synthetic data s k , thus forming a dynamically updated synthetic proxy dataset D p , as follows:

[0043] D p =β(s h )+γ(s k )

[0044] Among them, β and γ are the control s h 、s k Two weight parameters for the two-term balance.

[0045] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a privacy protection method for federated learning based on generative adversarial networks and knowledge distillation, which has the following beneficial effects:

[0046] (1) The present invention innovatively combines the generative adversarial network (GAN) with knowledge distillation technology. By training the generative adversarial network model locally on the client and generating synthetic data that conforms to the distribution of private data, the dependence on public datasets is avoided. The generated synthetic data is used for knowledge distillation, and knowledge transfer is achieved without selecting a public dataset. This effectively avoids direct exposure of the distribution of private data, thereby significantly reducing the risk of privacy leakage.

[0047] (2) The present invention introduces a balanced sampling method based on Renyi entropy in the knowledge distillation process. By performing balanced sampling and adding noise to the predicted labels and synthetic data, the knowledge quality can be effectively measured and optimized. This not only improves the privacy protection effect, but also achieves a balance between model utility while ensuring privacy, enhances the robustness to noise, and improves the reliability and stability of the model in the federated learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A flow chart of a method for privacy protection of federated learning based on generative adversarial networks and knowledge distillation provided by the present invention;

[0050] Figure 2 This is a flow chart of the generative adversarial network training provided by the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Reference Figure 1 As shown, the present invention discloses a method for protecting privacy in federated learning based on a generative adversarial network and knowledge distillation, comprising the following steps:

[0053] S1. Local model training: Initialize local model parameters, perform local model training based on local data sets of multiple clients, and obtain a trained local model;

[0054] S2. Generative adversarial network training: The trained local model is used as the discriminator to perform adversarial training with the generator to generate new synthetic sample data;

[0055] S3. Initialization phase communication: Each client randomly selects the same number of synthetic sample data for noise processing, uploads the noisy synthetic sample data to the server for sample merging, and obtains a public data set. Multiple clients use the public data set to generate prediction labels and send the prediction results to the server again;

[0056] S4. Distillation training: Multiple clients perform knowledge distillation training based on a public dataset to extract knowledge features;

[0057] S5. Communication in the distillation phase: Multiple clients perform balanced sampling of the extracted knowledge features, select prediction labels and synthetic data for noise processing, and upload the noise-processed prediction labels and synthetic data to the server. The server averages the prediction label results and merges the synthetic data of the clients to obtain the average prediction label result, and sends the average prediction label result to multiple clients.

[0058] S6. Client update: Multiple clients execute S2, S4, and S5 in a loop based on the average predicted label results to update the local model.

[0059] Furthermore, each client in S1 uses its own local dataset to perform local model training by minimizing the local loss function:

[0060]

[0061] Among them, w i is the local parameter initialized by each client, L(D i ,w i ) is the client i in the dataset D i The loss function on is the updated parameter;

[0062] Using the stochastic gradient descent optimization algorithm on each client, update the local parameter w initialized by each client i , for each iteration t, w i The update formula is:

[0063]

[0064] Where η is the learning rate, is the local loss function L(D i ,w i ) for parameter w i The gradient of

[0065] On each client i, the local model will be trained for multiple rounds of iterations, each round using the dataset D i The weights are updated using a batch of samples in until the preset convergence condition is reached.

[0066] Furthermore, the training of the generative adversarial network in S2 is to set the generator locally on the client and directly perform adversarial training with the trained local model as the discriminator, without uploading the updated parameters to the server;

[0067] The discriminator is a trained local model network, and the generator is the same initialized model. Let p data (x) = D i represents the real data set, z represents the noise distribution from the prior z The noise vector sampled in (z) is generated by the generator by mapping z to the data space G(z);

[0068] During the training process, the goal of the discriminator is to maximize its ability to correctly classify real data and generated data, that is, to maximize the following objective function:

[0069]

[0070] At the same time, the goal of the generator G is to deceive the discriminator so that it judges the generated data as real data, that is, to minimize or maximize the following objective function:

[0071]

[0072] Specifically, the Generative Adversarial Network (GAN) is defined as follows: Let G represent the generator and D represent the discriminator. The task of the generator is to generate data that is as realistic as possible, and the task of the discriminator is to distinguish whether the input data is real or generated by the generator.

[0073] This paper proposes a new privacy-preserving federated learning framework based on generative adversarial networks and simple distillation. The core idea is to set the generator locally on the client, so that adversarial training can be directly performed with the locally trained model (i.e., the discriminator), thereby eliminating the need to upload updated parameters to the server.

[0074] The entire training process can be viewed as a game process, in which the generator and the discriminator continuously compete with each other to achieve their respective goals. After the training of the generative adversarial network is completed, synthetic data that conforms to the private data distribution is generated, such as Figure 2 Shown is the training process of GAN.

[0075] Furthermore, the noise addition method is used to add Gaussian noise to the model prediction label e uploaded by the client. k and synthetic data dk Adding Gaussian noise, the formula is as follows:

[0076]

[0077] where s is the sensitivity of f, and It means that the center is 0 and the variance is σ 2 Gaussian normal distribution.

[0078] Specifically, in federated learning, Gaussian noise is a common privacy protection method. Gaussian noise is a differential privacy noise method that adds precisely calculated noise to the data, ensuring that any small change in a single entry in the data set will not significantly change the output result.

[0079] Furthermore, the steps of distillation training in S4 are as follows: first, aggregate the synthetic data after balanced sampling and noise addition and upload it to the server side as a public data set for knowledge distillation; second, upload the predicted labels after balanced sampling and noise addition to the server side for averaging.

[0080] Specifically, in federated learning, when each client has a heterogeneous model structure, knowledge distillation technology can effectively solve this problem. Through knowledge distillation, the knowledge of each client can be migrated to a unified global model, thereby achieving model collaboration and fusion. The overall distillation steps in the present invention are as follows: first, aggregate the synthetic data uploaded to the server after balanced sampling and noise, and use it as a public data set for knowledge distillation; second, upload the predicted labels after balanced sampling and noise to the server, and average them; finally, send them to the client for knowledge distillation update. The pseudo code is as follows:

[0081]

[0082] Furthermore, the specific contents of balanced sampling of the extracted knowledge features by multiple clients in S5 are as follows:

[0083] According to Renyi entropy, the amount and quality of knowledge are evaluated. First, the Renyi entropy of the predicted label is calculated, and the synthetic data corresponding to the predicted label with large Renyi entropy is selected, and the predicted label with large Renyi entropy is removed. Let the predicted label be e i =(p i1 ,p i2 ,..,p ij ) and is evaluated based on the following equation:

[0084]

[0085] Among them, α>0 and α≠1.

[0086] Furthermore, based on the calculated Renyi entropy of the predicted labels, the predicted labels are classified into two categories. This classification result is represented by C, one of which includes accurate and clear label knowledge E l , and the other category covers fuzzy label knowledge E h , the specific classification formula is as follows:

[0087]

[0088] Among them, τ represents the threshold;

[0089] After selecting the label knowledge, we further screen out the datasets containing rich information, including E h The corresponding synthetic data s h And the newly generated synthetic data s k , thus forming a dynamically updated synthetic proxy dataset D p , as follows:

[0090] D p =β(s h )+γ(s k )

[0091] Among them, β and γ are the control s h 、s k Two weight parameters for the two-term balance.

[0092] Specifically, the balanced sampling strategy achieves a balance between the quantity and quality of knowledge. The balanced sampling strategy selects synthetic data with rich information for knowledge extraction and selects high-quality predictive knowledge to participate in knowledge distillation training, thus achieving efficient knowledge distillation.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A privacy protection method for federated learning based on generative adversarial networks and knowledge distillation, characterized in that: The following steps are involved: S1. Local model training: Initialize local model parameters, perform local model training based on local data sets of multiple clients, and obtain a trained local model; S2. Generative adversarial network training: The trained local model is used as the discriminator to perform adversarial training with the generator to generate new synthetic sample data; S3. Initialization phase communication: Each client randomly selects the same number of synthetic sample data for noise processing, uploads the noisy synthetic sample data to the server for sample merging, and obtains a public data set. Multiple clients use the public data set to generate prediction labels and send the prediction results to the server again; S4. Distillation training: Multiple clients perform knowledge distillation training based on a public dataset to extract knowledge features; S5. Communication in the distillation phase: Multiple clients perform balanced sampling of the extracted knowledge features, select prediction labels and synthetic data for noise processing, and upload the noise-processed prediction labels and synthetic data to the server. The server averages the prediction label results and merges the synthetic data of the clients to obtain the average prediction label result, and sends the average prediction label result to multiple clients. S6. Client update: Multiple clients execute S2, S4, and S5 in a loop based on the average predicted label results to update the local model.

2. According to claim 1, a method for privacy protection of federated learning based on generative adversarial network and knowledge distillation is characterized in that: Each client in S1 uses its own local dataset to train a local model by minimizing the local loss function: Among them, w i is the local parameter initialized by each client, L(D i ,w i ) is the client i in the dataset D i The loss function on is the updated parameter; Using the stochastic gradient descent optimization algorithm on each client, update the local parameter w initialized by each client i , for each iteration t, w i The update formula is: Where η is the learning rate, is the local loss function L(D i ,w i ) for parameter w i The gradient of On each client i, the local model will be trained for multiple rounds of iterations, each round using the dataset D i The weights are updated using a batch of samples in until the preset convergence condition is reached.

3. According to claim 1, a method for privacy protection of federated learning based on generative adversarial network and knowledge distillation is characterized in that: In S2, the training of the generative adversarial network is to set the generator locally on the client and directly perform adversarial training with the trained local model as the discriminator, without uploading the updated parameters to the server; The discriminator is a trained local model network, and the generator is the same initialized model. Let p data (x) = D i represents the real data set, z represents the noise distribution from the prior z The noise vector sampled in (z) is generated by the generator by mapping z to the data space G(z); During the training process, the goal of the discriminator is to maximize its ability to correctly classify real data and generated data, that is, to maximize the following objective function: At the same time, the goal of the generator G is to deceive the discriminator so that it judges the generated data as real data, that is, to minimize or maximize the following objective function:

4. According to claim 1, a method for privacy protection of federated learning based on generative adversarial network and knowledge distillation is characterized in that: The noise processing uses the Gaussian noise method to predict the label e of the model uploaded by the client. k and synthetic data d k Adding Gaussian noise, the formula is as follows: where s is the sensitivity of f, and It means that the center is 0 and the variance is σ 2 Gaussian normal distribution.

5. The method for privacy protection of federated learning based on generative adversarial networks and knowledge distillation according to claim 1, characterized in that: The steps of distillation training in S4 are as follows: first, aggregate the synthetic data after balanced sampling and noise addition and upload it to the server side as a public data set for knowledge distillation; second, upload the predicted labels after balanced sampling and noise addition to the server side for averaging.

6. The method for privacy protection of federated learning based on generative adversarial networks and knowledge distillation according to claim 1, characterized in that: The specific contents of balanced sampling of extracted knowledge features by multiple clients in S5 are as follows: According to Renyi entropy, the amount and quality of knowledge are evaluated. First, the Renyi entropy of the predicted label is calculated, and the synthetic data corresponding to the predicted label with large Renyi entropy is selected, and the predicted label with large Renyi entropy is removed. Let the predicted label be e i =(p i1 ,p i2 ,..,p ij ) and is evaluated based on the following equation: Among them, α>0 and α≠1.

7. The method for privacy protection of federated learning based on generative adversarial networks and knowledge distillation according to claim 6, characterized in that: Based on the calculated Renyi entropy of the predicted labels, the predicted labels are classified into two categories. This classification result is represented by C. One category includes accurate and clear label knowledge E l , and the other covers fuzzy label knowledge E h , the specific classification formula is as follows: Among them, τ represents the threshold; After selecting the label knowledge, we further screen out the datasets containing rich information, including E h The corresponding synthetic data s h And the newly generated synthetic data s k , thus forming a dynamically updated synthetic proxy dataset D p , as follows: D p =β(s h )+γ(s k ) Among them, β and γ are the control s h 、s k Two weight parameters for the two-term balance.

Images