A federated learning method for non-independent and identically distributed heterogeneous data
By using conditional generative networks to generate virtual data in federated learning and combining it with knowledge distillation techniques, the problem of insufficient model training under heterogeneous data with non-independent and identical distributions is solved, achieving efficient and accurate global model training while protecting client privacy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2022-12-21
- Publication Date
- 2026-06-05
AI Technical Summary
Existing federated learning methods struggle to train high-quality global models applicable to all clients when faced with heterogeneous data that are not independent and are identically distributed, and they also suffer from privacy protection and low training efficiency issues.
A conditional generative network is used to generate virtual data on the server side. This virtual data is then used for training. In conjunction with knowledge distillation techniques, the classification models on both the client and server sides are used as teacher models to train the generative network, generating fake data or latent representations to compensate for data heterogeneity issues without exposing client privacy.
It improves the model's generalization and training efficiency, enhances the accuracy and training effect of the global model, and protects the client's privacy data from being leaked.
Smart Images
Figure CN115879542B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of federated learning technology, and specifically relates to a federated learning method for non-independent, identically distributed, heterogeneous data. Background Technology
[0002] Federated learning collaboratively trains a global model while ensuring that a group of clients do not upload their local datasets. Each user can only access their own data, thus protecting the privacy of users participating in the training. Due to its advantages, federated learning has broad application prospects in industries such as medicine, finance, and artificial intelligence, and has become a research hotspot in recent years. However, federated learning focuses on obtaining a high-quality global model by learning from the local data of all participating clients. But because the data of each client is heterogeneous in real-world scenarios, it cannot train a global model applicable to all clients when faced with data heterogeneity issues.
[0003] To achieve high prediction accuracy and address the data and model heterogeneity issues encountered in federated learning, most existing federated learning methods employ transfer learning, meta-learning, semi-supervised learning, or direct algorithm modification. However, these methods have limitations in terms of security and performance, especially under highly heterogeneous data distributions. Applying generative networks to federated training, specifically using conditional generative networks (GANs) to address the heterogeneity problem in federated learning, represents a promising new direction. GANs are widely used in artificial intelligence; their core idea is a variation of the original GAN network where both the generator and discriminator receive additional information C as conditions. These conditions can be category information or other modalities. By feeding this additional information C into the discriminative and generative models as part of the input layer, the conditional generative network is implemented. In federated learning, data and model heterogeneity is one of the main research directions. The non-independent and identically distributed nature of client data affects the model's performance and training efficiency. Conditional generative networks can generate data of specific categories. For clients with different data heterogeneity, they can generate data that does not expose privacy and apply it to the client's model training. This improves the generalization of the client model, solves the problem of missing client data categories, and reduces the differences between client data (increasing the possible shared data) without exposing users' local private data, thus solving the heterogeneity problem.
[0004] Therefore, how to ensure better training results for each client model while meeting privacy protection constraints, and at the same time train the GAN network to assist model training, is an urgent problem to be solved. Summary of the Invention
[0005] This invention provides a federated learning method for non-independent, identically distributed heterogeneous data, addressing the technical challenges of model and data heterogeneity in existing distributed deep learning.
[0006] This application provides a federated learning method for heterogeneous, non-independent data, the method comprising:
[0007] Step 1, Model Initialization:
[0008] Based on the settings of the federated average algorithm, a server and n clients are established, where each client has private data x. i , where i is the client number;
[0009] Initialize a pre-trained conditional generative model G(z|y) and a classification model E(x) on the server side, and then use the client-side classification model E... i (x) and the conditional generation model G(z|y) are transmitted to the client; where the client classification model E i (x) is the same as the server-side classification model E(x);
[0010] The server-side uses several rounds of server-to-client communication to aggregate client communication information and update parameters to obtain a classification model E(x) applicable globally.
[0011] Step 2, Client-side training:
[0012] Using client-assigned private data x i The local classification model E is trained using a conditional generative model G(z|y) shared with the server. i (x);
[0013] Step 3, Train the generative model: On the server side, utilize each client model E i The conditional generative network G is trained using E(x) and the server-side model E(x);
[0014] Step 4, Aggregate Parameters: After receiving the data from the client, the server weights and aggregates the parameters of each client model according to the federated average algorithm to update W. i And adjust the server classification model E(x);
[0015] Propagate the current parameters of the new model E(x) after aggregation in the current round to all client models;
[0016] Step 5, Model Testing: In the current round of communication, update the E value of all client classification models with the current parameters. i (x), test the accuracy using the test set, and obtain the average test accuracy of the federated model by weighting it according to the amount of data for each client;
[0017] Step 6: The server determines whether to continue the next communication. If it does, it returns to step 2; otherwise, it ends the communication and saves the global network model parameters.
[0018] Optionally, use private data x allocated by the client. i The local classification model E is trained using a conditional generative model G(z|y) shared with the server. i (x), including:
[0019] Step 2-1, the client uses local private data x i And client classification model E i (x) The cross-entropy loss of the model obtained during training;
[0020] Step 2-2: Sample random target category h on the client side, and generate fake image G(h) based on target category h using conditional generation model G(z|y). Use G(h) and its category h as an additional distillation dataset to train the client model and obtain additional loss; wherein, the sampling method includes random sampling or weighted sampling.
[0021] Steps 2-3: Calculate the loss function on the client side and train the client-side model E. i (x), update the model W i The result of the loss function is then passed back to the server.
[0022] Optionally, various client models E can be utilized on the server side. i The conditional generator network G is trained on the server-side model E(x) and the server-side model E(x), including:
[0023] Step 3-1: Extract private data from each client x i Randomly sampled category noise h from available categories * The generated fake image G(h) is obtained through the conditional generation network G. * |y);
[0024] Step 3-2: Classify each client's classification model E i The server-side model E is used as the teacher model to train the conditional generator network G;
[0025] Fake image G(h) * |y) directly through each classification model E i Calculate and minimize the following three loss functions for both the server-side model E and the server-side model E:
[0026] Activation loss: G(h) * |y) is the sum of the negative absolute values of the feature layers output by each classification model;
[0027] Single-heat vector loss: G(h)* |y) The classification layer outputs of each classification model and the target category h * Cross-entropy loss and diversity loss between them: G(h) * |y) The kl distance between the classification layer outputs of each classification model and the Gaussian distributed noise;
[0028] Steps 3-4: Based on the different data distributions of each client, obtain the weight proportion of data samples of different categories according to different clients and sum them with the loss in a weighted manner;
[0029] Steps 3-5: Return the loss function and update the generated model parameters W. G .
[0030] Optional, private data x i The general dataset is divided into independent and identically distributed datasets according to the Dirichlet distribution and assigned to each client. Different clients have different amounts of private data with different category distributions.
[0031] Optionally, during the first round of execution, step 2-2 is not executed. Instead, the model parameter updates and loss function values are directly calculated and uploaded to the server.
[0032] Optionally, the conditional generation network G includes fully connected layers, activation layers, batch normalization layers, deconvolution layers, and embedding layers;
[0033] The conditional generative network G is trained using Adam as the optimizer. The output of the conditional generative network G is either the spoof image output or the latent representation output of the target classification model E(x).
[0034] This invention proposes a federated learning method for non-independent, identically distributed heterogeneous data, with several key innovations: 1)
[0035] This invention provides a method for implementing federated learning. The method constructs a conditional generative network on the server side, which generates virtual data. This virtual data, used for training on various clients, increases the generalization ability between models and compensates for the uneven data distribution caused by data heterogeneity among different clients. This mitigates the problem of insufficient global network model training due to insufficient data volume and uneven data distribution. 2) Using the server-side model and the client-side training model as teacher models, the client-side and server-side classifiers are used instead of discriminators on the server side to train the conditional generative model. Simultaneously, the generative model is used to generate fake data or latent representations on various clients. Because the training of the generative model does not use clients' private data, this method improves the efficiency and accuracy of federated learning without exposing privacy. 3) This method combines federated learning with knowledge distillation, using all client-side models and server-side models as teacher networks and the fake data generated by the generative network as the distillation dataset. Data distillation improves model performance and efficiency when training client models. Attached Figure Description
[0036] Figure 1 This is a flowchart of a federated learning method for heterogeneous, non-independent data provided in an embodiment of this application;
[0037] Figure 2 This is a schematic diagram of the model provided in the application embodiment. Detailed Implementation
[0038] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0039] As shown in the figure, this application provides a federated learning method for heterogeneous data that are not independent and are distributed in the same way. The method includes:
[0040] Step 1, Model Initialization:
[0041] Step 1-1: Based on the settings of the federated averaging algorithm, establish a server and n clients, where each client possesses private data x. i , where i is the client number;
[0042] Steps 1-2: Initialize a pre-trained conditional generation model G(z|y) and a classification model E(x) on the server side, and then initialize the client-side classification model E... i (x) and the conditional generation model G(z|y) are transmitted to the client; where the client classification model E iThe conditional generative network G is identical to the server-side classification model E(x). G contains fully connected layers, activation layers, batch normalization layers, deconvolutional layers, and embedding layers. G is trained using Adam as the optimizer. The output of G is either the spoofed image output or the latent representation output of the target classification model E(x), depending on whether spoofed images or latent representations are used as the distillation dataset in step 2-2.
[0043] In the experiment, LeNet, CNN, and Mobilenet_v3 were used as classification models E. Under the condition of non-independent and identically distributed, the training time was shorter and the global accuracy was higher than that of the federated averaging algorithm.
[0044] Steps 1-3 involve aggregating client communication information during several rounds of server-to-client communication on the server side and updating parameters to obtain a global classification model E(x).
[0045] Step 2, Client-side training:
[0046] Using client-assigned private data x i The local classification model E is trained using a conditional generative model G(z|y) shared with the server. i (x).
[0047] It should be noted that the private data x i The general dataset was divided into independent and identically distributed datasets according to the Dirichlet distribution and assigned to each client. Different clients have different amounts of private data with different category distributions. The CIFAR10 and MNIST datasets were used as the general datasets in the experiment.
[0048] Specifically, step 2 includes:
[0049] Step 2-1, the client uses local private data x i And client classification model E i (x) The cross-entropy loss of the model obtained during training;
[0050] Step 2-2: Sample random target category h on the client side, and generate fake image G(h) based on target category h using conditional generation model G(z|y). Use G(h) and its category h as an additional distillation dataset to train the client model and obtain additional loss; wherein, the sampling method includes random sampling or weighted sampling.
[0051] This step improves the efficiency and speed of model aggregation by generating a corresponding distillation dataset for each client using a generative model G(z|y) trained on the server side and performing the distillation operation on the client side.
[0052] It should be noted that in step 1, the model G generated in the first round of communication between the client and the server has not yet been trained. Therefore, in the first round of execution, step 2-2 is not executed. The model parameter update and loss function value are directly calculated and uploaded to the server.
[0053] Steps 2-3: Calculate the loss function on the client side and train the client-side model E. i (x), update the model W i The result of the loss function is passed back to the server.
[0054] Step 3, Train the generative model: On the server side, utilize each client model E i The server-side model E(x) and the conditional generator network G are trained.
[0055] Specifically, step 3 includes:
[0056] Step 3-1: Extract private data from each client x i Randomly sampled category noise h from available categories * The generated fake image G(h) is obtained through the conditional generation network G. * |y);
[0057] Step 3-2: Classify each client's classification model E i The server-side model E is used as the teacher model to train the conditional generator network G;
[0058] Unlike classic generative adversarial networks that train the generative network through a discriminator, this method trains the fake image G(h) in step 3-1. * |y) directly through each classification model E i Calculate and minimize the following three loss functions for both the server-side model E and the server-side model E:
[0059] Activation loss: G(h) * |y) is the sum of the negative absolute values of the feature layers output by each classification model;
[0060] Single-heat vector loss: G(h) * |y) The classification layer outputs of each classification model and the target category h * Cross-entropy loss and diversity loss between: G(h) * |y) The kl distance between the classification layer outputs of each classification model and the Gaussian distributed noise.
[0061] Classify each client model E iThe purpose of using the server-side model E as the teacher model to train the generator network G is to: use the existing trained classification network as the discriminator, eliminating the need to train an additional discriminator and saving training time; and at the same time, it does not require any private data from the client to participate in the training, thus protecting user privacy.
[0062] The purpose of the one-hot vector loss is to make the output of the generated image after passing through each classification model similar to the output of the data from dataset x after passing through the classification model. In this way, the generated image of the conditional generative network G will also have a roughly the same distribution as the training data of the teacher model. The purpose of the activation loss is to make the generated image of the conditional generative network G more like an image from a real dataset. The output of the feature layer corresponds to the output of the classification model before passing through the fully connected layer. If the input image is real and not some random vector, the feature map will often obtain a higher activation value. The diversity loss is to make the generated images of the conditional generative network G as unique as possible for a certain type of image, that is, to make the generated images as diverse as possible.
[0063] Steps 3-4: Based on the different data distributions of each client, obtain the weight proportion of data samples of different categories according to different clients and sum them with the loss in a weighted manner;
[0064] Steps 3-5: Return the loss function and update the generated model parameters W. G .
[0065] Step 4, Aggregate Parameters: After receiving the data from the client, the server weights and aggregates the parameters of each client model according to the federated average algorithm to update W. i And adjust the server classification model E(x);
[0066] The current parameters of the new model E(x) after aggregation in the current round are propagated to all client models to update the client models;
[0067] Step 5, Model Testing: In the current round of communication, update the E value of all client classification models with the current parameters. i (x), test the accuracy using the test set, and obtain the average test accuracy of the federated model by weighting it according to the amount of data for each client;
[0068] Step 6: The server determines whether to continue the next communication. If it does, it returns to step 2; otherwise, it ends the communication and saves the global network model parameters.
[0069] The method provided in this application constructs a conditional generative network on the server side, which generates virtual data. This virtual data, which is used for training on various clients, increases the generalization between models and can compensate for the uneven data distribution caused by data heterogeneity between different clients. This can mitigate the problem of insufficient training of the global network model caused by insufficient data volume and uneven data distribution.
[0070] This application uses a server-side model and a client-side trained model as teacher models. On the server side, classifiers from both the client and server sides are used instead of discriminators to train a conditional generative model. This generative model is then used to generate fake data or latent representations on various clients. Because the training of the generative model does not use clients' private data, this method improves the efficiency and accuracy of federated learning without exposing privacy.
[0071] The method presented in this application combines federated learning with knowledge distillation, using all client-side and server-side models as the teacher network and the spoof data generated by the generative network as the distillation dataset. Data distillation improves model performance and efficiency during client-side model training.
Claims
1. A federated learning method for heterogeneous, non-independent data, characterized in that, The method includes: Step 1, Model Initialization: Based on the settings of the federated averaging algorithm, a server and n clients are established, where each client has private data. , where i is the client number; Initialize a pre-trained conditional generative model on the server side. and classification models And the client classification model and conditional generative models Transmitted to the client; among which, the client-side classification model Server-side classification model same; The server aggregates client communication information through several rounds of server-to-client communication and updates parameters to obtain a classification model applicable globally. ; Step 2, Client-side training: Using client-assigned private data Conditional generation model shared with the server Training a local classification model ; Step 3, Train the generative model: On the server side, utilize the various client models and server-side model Training Conditional Generative Network ; Step 4, Aggregate Parameters: After receiving the data from the client, the server weights and aggregates the parameters of each client model according to the federated average algorithm to update the parameters. And adjust the server classification model ; The new model after aggregating the current rounds The current parameters are propagated to all client models; Step 5, Model Testing: In the current round of communication, update the classification models of all clients with the current parameters. The accuracy was tested using a test set, and the average test accuracy of the federated model was obtained by weighting the data volume of each client. Step 6: The server determines whether to continue the next communication. If it does, it returns to step 2; otherwise, it ends the communication and saves the global network model parameters. Utilize various client models on the server side and server-side model Training Conditional Generative Network ,include: Step 3-1: Extract private data from each client Randomly sampled category noise from available categories After conditional generation network Get the generated fake image ; Step 3-2: Classify the various client models and server-side model As a teacher model, the conditional generation network is trained. ; Fake images Directly through various classification models Calculate and minimize the following three loss functions respectively: Activation loss: The sum of the negative absolute values output by the feature layers of each classification model; Single-heat vector loss: After the classification layer outputs of each classification model and the target category Cross-entropy loss between Diversity loss: The kl distance between the classification layer outputs of each classification model and the Gaussian distributed noise; Steps 3-4: Based on the different data distributions of each client, obtain the weight proportion of data samples of different categories according to different clients and sum them with the loss in a weighted manner; Steps 3-5: Return the loss function and update the generated model parameters. .
2. The federated learning method for non-independent, identically distributed heterogeneous data according to claim 1, characterized in that, Using client-assigned private data Conditional generation model shared with the server Training a local classification model ,include: Step 2-1, the client uses local private data and client classification model The cross-entropy loss of the model is obtained through training; Step 2-2: Sample random target categories on the client side. Through conditional generation model According to target category Generated fake images ,by and its categories The client model is trained using an additional distillation dataset to obtain additional loss; the sampling method includes random sampling or weighted sampling. Steps 2-3: Calculate the loss function and train the client-side model. Update the model The result of the loss function is then passed back to the server.
3. The federated learning method for non-independent, identically distributed heterogeneous data according to claim 1, characterized in that, Private data The general dataset is divided into independent and identically distributed datasets according to the Dirichlet distribution and assigned to each client. Different clients have different amounts of private data with different category distributions.
4. The federated learning method for non-independent, identically distributed heterogeneous data according to claim 1, characterized in that, In the first round of execution, step 2-2 is not executed. The model parameter updates and loss function values are directly calculated and uploaded to the server.
5. A federated learning method for non-independent, identically distributed heterogeneous data according to claim 1, characterized in that, Conditional Generative Networks It includes fully connected layers, activation layers, batch normalization layers, deconvolution layers, and embedding layers; Conditional Generative Networks The conditional generative network is trained using Adam as the optimizer. The output is the target classification model. The output of fake images or potential representations.