Federal learning method based on data enhancement and data-knowledge-free distillation

Abstract

The invention provides a federated learning method based on data enhancement and data-knowledge-free distillation, and belongs to the technical field of federated learning. The method comprises the following steps: a client deploys a generative adversarial network and a local model, and a server deploys a global model and the generative adversarial network; participating clients participating in the training are randomly selected, and each participating client downloads the global model of the server as a local model; the participating client generates an added data set based on the local data set through a generative adversarial network; training a local model of a participating client by using the enhanced data set, and updating local model parameters and data distribution; the server aggregates the updated local model parameters uploaded by each participating client, and updates the global model through a data-knowledge-free distillation operation; and the server sends the updated global model to a next-round participating client for next-round training. According to the method, the generalization ability and performance of the federal learning model in the data non-independent identically distributed environment are improved.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning method based on data augmentation and data-free knowledge distillation. Background Technology

[0002] Federated Learning (FL) is a distributed machine learning framework designed to train a global model jointly by multiple clients (such as mobile devices or edge nodes) while protecting data privacy. Its standard algorithms typically assume that the data from each client satisfies the condition of Independent and Identically Distributed (IID). In this case, the updates of the local model to the global model can be considered unbiased estimates, thus achieving efficient convergence and good performance. However, in real-world applications, client data often exhibits Non-IID characteristics, meaning that the data distribution varies significantly across different clients. This heterogeneity in data distribution leads to the accumulation of biases between the local and global models, resulting in degraded model performance and slower convergence.

[0003] To address the Non-IID data problem in federated learning, existing technologies primarily improve upon it in two ways: first, by optimizing algorithms to reduce the bias between local and global models, such as through regularization constraints, client-side aggregated weight adjustments, or dynamic learning rate strategies; and second, by using data augmentation to alleviate data imbalance, such as using Generative Adversarial Networks (GANs) to generate synthetic data or sharing partial data among clients to balance the distribution. However, the former can only suppress bias propagation to a certain extent and cannot fundamentally eliminate the impact of data distribution differences, especially in extreme Non-IID scenarios where its effectiveness is limited. While the latter can improve data imbalance, most methods rely on shared generators or raw data, posing a privacy risk and violating the core principle of federated learning to protect data privacy.

[0004] Therefore, there is an urgent need for a federated learning method that can effectively solve the Non-IID problem of data between clients without compromising user privacy. Summary of the Invention

[0005] In view of this, the present invention provides a federated learning method based on data augmentation and data-free knowledge distillation. By deploying a generative adversarial network locally on the client, data augmentation is performed on the client's local dataset to alleviate the imbalance in data distribution among clients. Furthermore, data-free knowledge distillation is adopted on the server. A generator deployed on the server generates pseudo-data that conforms to the client's data distribution to replace the real data required for distillation. This transfers the knowledge of each local model to the global model, thereby providing the global model with generalization ability and accuracy in scenarios where data is not independent and identically distributed.

[0006] Therefore, the present invention provides the following technical solution:

[0007] A federated learning approach based on data augmentation and data-free knowledge distillation includes:

[0008] The client deploys the generative adversarial network and local model, while the server deploys the global model and generative adversarial network.

[0009] Randomly select participating clients to participate in this round of training, and each participating client downloads the server's global model as a local model;

[0010] Participating clients construct augmented datasets based on local datasets using deployed generative adversarial networks;

[0011] Train the local model participating in the client using the augmented dataset, and update the local model parameters and data distribution;

[0012] The server aggregates the updated local model parameters uploaded by each participating client and updates the global model through a knowledge-free distillation operation.

[0013] The server sends the updated global model to the next round of participating clients to begin the next round of training.

[0014] Furthermore, the participating client constructs an augmented dataset based on the local dataset using a deployed generative adversarial network, including:

[0015] Calculate the local data distribution based on the local dataset;

[0016] Select labels with a data volume that reaches a preset threshold to train the generative adversarial network;

[0017] Synthetic data is generated using a trained generative adversarial network.

[0018] Combine synthetic data and local datasets to build an enhanced dataset.

[0019] Furthermore, the server aggregates the updated local model parameters uploaded by each participating client and updates the global model through a knowledge-free distillation operation, including:

[0020] Each local model is used as the teacher model, and the global model is used as the student model;

[0021] Generative adversarial networks deployed on the server are used to generate pseudo-data that conforms to the data distribution of participating clients;

[0022] The difference between the global model and the local model is calculated based on the pseudo-data and used as the distillation loss.

[0023] By minimizing the distillation loss, knowledge from the local model is transferred to the global model.

[0024] Furthermore, the process of generating synthetic data using the trained generative adversarial network includes:

[0025] Using the trained generator, the target label vector is determined based on the local data distribution; and based on the target label vector, synthetic data matching the label is generated.

[0026] Furthermore, the generative adversarial network deployed on the client is an auxiliary classifier generative adversarial network.

[0027] Furthermore, the generative adversarial network deployed on the server is a conditional generative adversarial network.

[0028] Furthermore, the loss function for training the generative adversarial network is:

[0029]

[0030] Where z~N(0,1) is standard Gaussian noise, (D k G k ) represents a set of discriminators and generators for client k, where x is the real image and y is the category label. This represents the optimization objective of the generative adversarial network; The value function represents the adversarial game relationship between the generator and the discriminator.

[0031] Furthermore, the distillation loss:

[0032]

[0033] in, This represents pseudo data. The difference between the global and local models is represented by distillation loss; σ is the softmax function; D KL It is the Kullback-Leibler divergence; Representing the local model ω k for The predicted score; Represents the global model ω g for The predicted score.

[0034] Advantages and positive effects of the present invention:

[0035] This method deploys a generative adversarial network (GAN) locally on the client side, achieving data augmentation of the client's local dataset without any sharing measures, thus ensuring user data privacy. Simultaneously, it uses a data-free knowledge distillation method on the server to transfer knowledge from each local model to the global model without involving the client's original data, fine-tuning the global model. This addresses the limitations of GANs in environments where data is not independent and identically distributed, thereby improving the generalization ability and performance of federated learning models in such environments. It also enhances the robustness and convergence efficiency of federated learning models. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of the federated learning method based on data augmentation and data-free knowledge distillation in an embodiment of the present invention;

[0038] Figure 2 This is a diagram of a federated learning framework based on data augmentation and data-free knowledge distillation in an embodiment of the present invention;

[0039] Figure 3 This is a graph showing the experimental results of the MNIST dataset in an embodiment of the present invention;

[0040] Figure 4 This is a graph showing the experimental results of the Fashion MNIST dataset in an embodiment of the present invention;

[0041] Figure 5 The figure shows the experimental results of the CIFAR-10 dataset in this embodiment of the invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] This invention provides a federated learning method based on data augmentation and data-free knowledge distillation. It involves deploying a generative adversarial network (GAN) model locally on the client side, training it on the client's dataset, and then using the trained generator to generate synthetic samples to augment the client's local dataset, initially alleviating the quantitative imbalance in data distribution among clients. Then, a data-free knowledge distillation method is employed on the server. A conditional generator deployed on the server generates pseudo-data that conforms to the client's local data distribution, replacing the real data required for traditional knowledge distillation, thus transferring knowledge from each local model to the global model.

[0045] Combination Figure 1 As shown, the steps of this method include:

[0046] The client deploys a generative adversarial network and a local model, while the server deploys a global model and a generative adversarial network.

[0047] S1. At the start of this training round, participating clients are randomly selected, and each participating client downloads the server's global model as a local model.

[0048] S2. Participating clients enhance their local datasets by generating adversarial networks:

[0049] 1) Calculate the local data distribution, expressed by the formula:

[0050]

[0051] in, This represents the data characteristics of client k. D represents the data label for client k. k This represents the local dataset of client k. This represents the amount of data labeled y for client k. The value is 1 if the condition is true, and 0 otherwise; p k(y) represents the amount of data for each label y in the client's local dataset, i.e., the local data distribution;

[0052] 2) Select labels with a data volume reaching a preset threshold to train the client's generative adversarial network:

[0053] Adversarial training is performed using a generator and a discriminator. The generator produces synthetic data that matches the characteristics of the original data, making the synthetic data as realistic as possible. The discriminator distinguishes between real and synthetic data and continuously provides feedback to the generator to improve the generation quality. The formula is expressed as:

[0054]

[0055] Where z~N(0,1) is standard Gaussian noise, (D k G k () represents a set of discriminators and generators for client k; This represents the optimization objective of the generative adversarial network; The value function is used to represent the adversarial game relationship between the generator and the discriminator.

[0056] 3) Utilize the generative adversarial network trained on the client side to generate synthetic data and construct an augmented dataset:

[0057] Each participating client has a trained generator capable of producing realistic samples.

[0058] ① The generator is trained based on the local data distribution p k (y) yields the target label vector Y target =y1,y2...y n , Y target It only contains tags with a preset amount of data;

[0059] ②Y target The data is fed into the post-training generator to generate synthetic data that matches the label y.

[0060]

[0061] ③ Combine the data Add to the local dataset to obtain an initially balanced augmented dataset.

[0062]

[0063] S3. Use the augmented dataset to train the local model participating in the client, and update the local model parameters and data distribution.

[0064] S4. The server aggregates the updated local model parameters uploaded by each participating client and updates the global model through a knowledge-free distillation operation:

[0065] Each local model is considered a teacher model, and the global model is considered a student model.

[0066] 1) Use the condition generator deployed on the server to generate pseudo-data that conforms to the data distribution of participating clients. The formula is as follows:

[0067]

[0068] Among them, G s Represents a condition generator; y represents pseudo-data; y is the category label.

[0069] 2) Calculate the difference between the global model and the local model as the distillation loss:

[0070]

[0071] in, The difference between the global and local models is represented by distillation loss; σ is the softmax function; D KL It is the Kullback-Leibler divergence; Representing the local model ω k for The predicted score; Represents the global model ω g for The predicted score.

[0072] 3) By minimizing the distillation loss, knowledge from the local model is transferred to the global model.

[0073] S6. After completing the knowledge distillation without data, the server sends the updated global model to the client for the next round of training.

[0074] The effectiveness of this method is further illustrated through comparative experiments:

[0075] Datasets include: MNIST, Fashion MNIST, and CIFAR-10.

[0076] Other federated learning methods for comparison include: FedAvg (Federated Averaging), FedProx (Federated Proximal Optimization), FedBN (Federated Batch Normalization), MOON (Model Agnostic Meta-Learning), and SCAFFOLD (Stochastic Controlled Averaging).

[0077] Experimental Setup

[0078] 1) Input image size: For the MNIST and Fashion MNIST datasets, the input image size is 28*28, and for the CIFAR-10 dataset, the input image size is 32*32.

[0079] 2) Model settings: The neural network selected is CNN (Convolutional Neural Network). The generative adversarial network deployed on the client is ACGAN (Auxiliary Classifier Generative Adversarial Network), and the generative adversarial network deployed on the server is CGAN (Conditional Generative Adversarial Network).

[0080] 3) Hyperparameter settings: The local training rounds are set to 5, the communication rounds are T=200, and the non-IID degree of the data conforms to the Dirichlet distribution. By default, the Dirichlet distribution parameters are α=0.2, the learning rate is β=0.01, and the batch size is 128.

[0081] 4) Evaluation indicators:

[0082] Test accuracy: The accuracy of the model on the test set;

[0083] Number of communication rounds: The number of communication rounds required for the algorithm to reach the target accuracy.

[0084] Experimental results (B):

[0085] Figure 3 , Figure 4 as well as Figure 5 The learning curves of this method and other federated learning methods are shown on three datasets. It can be seen that this method has the highest test accuracy in all settings, which is better than other methods. This shows that this method can effectively improve the performance of the model under non-independent and identically distributed data.

[0086] Table 1 compares the number of communication rounds required for our method to reach the target accuracy with other federated learning methods, using the number of communication rounds required to achieve the target accuracy as the evaluation metric. As shown in Table 1, our method achieves the target accuracy with the fewest communication rounds across the three datasets, indicating that our model exhibits superior overall performance.

[0087] Table 1

[0088] Dataset MNIST Fashion MNIST CIFAR-10 Target Accuracy 90％ 70％ 40％ FedAvg 169±6.66 54±2.01 93±1.53 FedProx 95±3.79 69±4 103±1 FedBN 77±4.04 40±3.06 94±2.33 MOON 74±5.10 67±7.51 108±1.32 SCAFFOLD 66±3.51 74±2.02 141±1.73 This method 30±2.79 21±2.08 31±2.65

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning method based on data augmentation and data-free knowledge distillation, characterized in that, include: The client deploys the generative adversarial network and local model, while the server deploys the global model and generative adversarial network. Randomly select participating clients to participate in this round of training, and each participating client downloads the server's global model as a local model; Participating clients construct augmented datasets based on local datasets using deployed generative adversarial networks; Train the local model participating in the client using the augmented dataset, and update the local model parameters and data distribution; The server aggregates the updated local model parameters uploaded by each participating client and updates the global model through a knowledge-free distillation operation. The server sends the updated global model to the next round of participating clients to begin the next round of training.

2. The method according to claim 1, characterized in that, The participating clients construct augmented datasets based on local datasets using a deployed generative adversarial network, including: Calculate the local data distribution based on the local dataset; Select labels with a data volume that reaches a preset threshold to train the generative adversarial network; Synthetic data is generated using a trained generative adversarial network. Combine synthetic data and local datasets to build an enhanced dataset.

3. The method according to claim 1, characterized in that, The server aggregates the updated local model parameters uploaded by each participating client and updates the global model through a knowledge-free distillation operation, including: Each local model is used as the teacher model, and the global model is used as the student model; Generative adversarial networks deployed on the server are used to generate pseudo-data that conforms to the data distribution of participating clients; The difference between the global model and the local model is calculated based on the pseudo-data and used as the distillation loss. By minimizing the distillation loss, knowledge from the local model is transferred to the global model.

4. The method according to claim 2, characterized in that, The process of generating synthetic data using the trained generative adversarial network includes: Using the trained generator, the target label vector is determined based on the local data distribution; and based on the target label vector, synthetic data matching the label is generated.

5. The method according to claim 1, characterized in that, The generative adversarial network deployed on the client is an auxiliary classifier generative adversarial network.

6. The method according to claim 1, characterized in that, The generative adversarial network deployed on the server is a conditional generative adversarial network.

7. The method according to claim 1, characterized in that, The loss function for training the generative adversarial network is: Where z~N(0,1) is standard Gaussian noise, (D k G k ) represents a set of discriminators and generators for client k, where x is the real image and y is the category label. This represents the optimization objective of the generative adversarial network; The value function represents the adversarial game relationship between the generator and the discriminator.

8. The method according to claim 1, characterized in that, The distillation loss: in, This represents pseudo data. The difference between the global and local models is represented by distillation loss; σ is the softmax function; D KL It is the Kullback-Leibler divergence; Representing the local model ω k for The predicted score; Represents the global model ω g for The predicted score.

Images