Data-free federated bidirectional knowledge distillation method and system based on feature adversarial

High-quality pseudo-data is generated through feature adversarial training and bidirectional distillation between the global model and the local model, solving the problems of inconsistent data distribution and privacy protection in federated learning, significantly improving the performance and stability of the model.

CN118627644BActive Publication Date: 2024-11-29JIANGNAN UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411093669.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-29
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

In federated learning, client data distribution is inconsistent, especially in the presence of a few and missing classes, resulting in significant differences in the model, affecting the aggregation effect and final performance of the global model. The application of traditional generative models is limited under the privacy protection requirements. , it is difficult to effectively utilize the knowledge-assisted training process generated by generative models.

Method used

Using the data-free federal two-way knowledge distillation method based on feature adversariality, through feature adversarial training between the client's local feature discriminator and the server's global generator, generates generated samples similar to the real global sample features, and utilizes generation in the global stage. The sample and local models are fine-tuned to achieve bidirectional distillation of the global model.

Benefits of technology

It significantly improves the performance and stability of the model, especially in scenarios where client data distribution is large, improves the model's ability to identify minority and missing classes, which is better than existing methods, and improves under the premise of privacy protection. The classification ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118627644B_ABST
    Figure CN118627644B_ABST
Patent Text Reader

Abstract

The present invention provides a data-free federated bidirectional knowledge distillation method and system based on feature adversarial, which relates to the field of federated learning technology in machine learning. The method includes constructing a federated learning framework, combining multiple clients and servers, using a global generator to generate pseudo data for enhanced training, and combining adversarial training between local feature discriminators and global generators to promote bidirectional knowledge distillation between global models and local models. During the iteration process, the global model continuously integrates local knowledge, and the local model also obtains the classification ability from a global perspective, ultimately achieving high performance and stability of the global model. The present invention is suitable for scenarios with uneven data distribution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology in machine learning, and in particular to a data-free federated bidirectional knowledge distillation method and system based on feature adversarial methods. Background Art

[0002] Federated Learning (FL), as an innovative distributed machine learning paradigm, has been widely studied to cope with the explosive growth of data volume and the increasing demand for client data privacy protection. In order to learn and train models from large amounts of decentralized and protected client data, FL builds a system consisting of a central server and multiple clients, avoiding direct access to the client's original data through communication of model parameters. To date, many advanced FL solutions have been developed and successfully applied in practical fields.

[0003] During the FL training process, the server and client optimize the model through multiple global iterations and local iterations. In the global iteration phase, the central server collects model updates from each client (usually the gradient or update amount of the model parameters), aggregates them, and generates a new global model based on them. Subsequently, this global model is distributed to all clients participating in the training. In the local iteration phase, the client uses its own local data to train the received global model, generates a new model update, and uploads it back to the server for the next global iteration.

[0004] However, FL has faced a fundamental and major challenge since its proposal: the data distribution between clients is highly inconsistent. Especially when there are minority classes and missing classes in the dataset, this inconsistency will be exacerbated, resulting in significant differences in the models obtained by the clients during the local training phase, which in turn affects the aggregation effect and final performance of the global model.

[0005] To solve this problem, generative models have attracted much attention due to their ability to generate new samples to alleviate the lack of training data. However, under the privacy protection requirements of FL, the application of traditional generative models is limited because they may leak sensitive information of the client. Therefore, how to effectively use the knowledge generated by the generative model to assist the training process of FL under the premise of protecting privacy has become one of the current research hotspots. The focus of this invention is to balance the relationship between the knowledge of the generated samples and privacy protection, ensuring that the generated samples can not only improve the model training effect, but also will not leak the sensitive data of the client. Summary of the invention

[0006] To this end, an embodiment of the present invention provides a dataless federated bidirectional knowledge distillation method and system based on feature adversarial, which is used to solve the problem in the prior art that the effect of dataless federated distillation is not significant when there are large differences in client data distribution and there are many minority classes and missing classes.

[0007] In order to solve the above problems, an embodiment of the present invention provides a data-free federated bidirectional knowledge distillation method based on feature confrontation, which includes:

[0008] Step S1: construct a federated learning framework, including multiple clients and one server, and jointly train a global model by combining multiple clients and one server to participate in federated learning;

[0009] The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier;

[0010] Each client initializes a local feature discriminator;

[0011] Step S2: In the local stage, each client downloads the global model and global generator from the server;

[0012] Each client uses the global model as the initial local model, and the generated data generated by the global generator and the local real data as training data. The output of the global model on the generated data serves as a teacher supervision and acts as a regular term for local model training to train the local model;

[0013] Each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator;

[0014] After completing the training, each client sends the trained local model and the trained local feature identifier to the server;

[0015] Step S3: Global stage, the server receives the trained local models and trained local feature identifiers sent by each client;

[0016] The server aggregates the trained local models as the initial global model;

[0017] The server trains the global generator so that the generated samples can deceive multiple local feature discriminators;

[0018] The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model.

[0019] Step S4: Return to step S2 and execute until the global model converges to obtain the final global model.

[0020] Preferably, the method further comprises: inputting the test image into the final global model to obtain an image classification result.

[0021] Preferably, step S1 specifically includes:

[0022] Build a federated learning framework, including multiple clients and one server, and jointly train a global model by uniting multiple clients and one server to participate in federated learning, where each client has its own private dataset;

[0023] The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier, wherein the global feature extractor is used to map data from the instance space To feature space , the global classifier is used to map features to the category space ; The global generator is used to generate Sampling generates samples with global knowledge;

[0024] The client initializes a local feature discriminator, which outputs a scalar that can identify whether the feature comes from the local domain or the global domain;

[0025] The global feature extractor is composed of Parameterized, the global classifier is given by Parameterized, the global model ; The global generator is composed of Parameterization; the local feature discriminator of the kth client is given by Parameterization, where It means "define as" or "set to", and is an assignment operator.

[0026] Preferably, step S2 specifically includes:

[0027] In the local stage, each client downloads the global model and global generator from the server;

[0028] Each client uses the global model as the initial local model , the generated data generated by the global generator and the local real data are used as training data, and the output of the global model on the generated data serves as the teacher supervision and acts as the regular term for local model training. Then the loss function of training the local model of the client is:

[0029] (1) Among them, is the loss function of the local model of the kth client, E represents the mathematical expectation, is the distribution of the training data set for the kth customer, It is a local model The cross entropy loss on the local data x, is the distribution of samples generated by the global generator, It means obedience A generated sample from the distribution, namely , the specific formula for generating samples using the global generator is:

[0030] (2) Among them, is a Gaussian distribution of noise, is a uniform distribution of class labels, z means obey A noise vector distributed by Show obedience The label value of a generated sample of the distribution, G is the output function of the generator, is a hyperparameter that adjusts the regularized distillation loss, is the regularized distillation loss, which constrains the training direction of the local model through the output of the global model on the generated data, so that the local model still maintains the ability of the global model on the generated data. The specific formula is as follows:

[0031] (3) Among them, are the outputs of the local model and the mth layer of the global model, respectively, M is the total number of model layers, is the mean square error loss;

[0032] Subsequently, each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator, maximize the score of the positive sample, and minimize the score of the negative sample. The loss function of the kth local feature discriminator is:

[0033] (4) Among them, is the loss function of the kth local feature discriminator, is the output function of the kth local feature discriminator, is the output function of the kth local feature extractor, is the output function of the global feature extractor, is the local feature extractor of the kth local model;

[0034] After completing the training, each client sends the trained local model and the trained local feature identifier to the server.

[0035] Preferably, step S3 specifically includes:

[0036] In the global stage, the server receives the trained local models and trained local feature discriminators sent by each client;

[0037] The local model after server aggregation training is used as the initial global model. The specific formula is as follows:

[0038] (5) Among them, is the set of clients participating in the current communication round, and its size is expressed as , is the size of the kth client dataset, is the size of the i-th client dataset;

[0039] After obtaining the initial global model, the server trains the global generator so that the generated samples can deceive multiple local feature discriminators and generate samples with global features. The training of the global generator is based on three losses: feature adversarial loss, semantic loss, and intra-class diversity loss.

[0040] Feature Adversarial Loss for:

[0041] (6) Semantic Loss for: (7) Among them, is the cross entropy loss, The label value is the label value given in advance when generating the sample according to formula (2) ; Intra-class diversity loss for: (8) Among them, and are two sampling points in the noise space, and two samples of the same category are generated according to formula (2) and ; and Is to generate samples and The features extracted by the local feature extractor are given by the formula: , ; e is the natural logarithm, is the two-norm;

[0042] Combining the three losses of the global generator, the final global generator loss for:

[0043] (9) Among them, and These are all adjustable hyperparameters, which can be used to adjust the balance of generated samples in feature space, category space, and diversity;

[0044] The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model. The global fine-tuning loss for:

[0045] (10) Among them, are the outputs of the local model and the mth layer of the global model, respectively, M is the total number of model layers, It is the mean square error loss. The form of the global fine-tuning loss is consistent with the regularization loss of the local model, ensuring that the knowledge of bidirectional distillation is consistent.

[0046] The embodiment of the present invention further provides a data-free federated bidirectional knowledge distillation system based on feature confrontation, which is used to implement the above-mentioned data-free federated bidirectional knowledge distillation method based on feature confrontation, and specifically includes:

[0047] The initialization module is used to build a federated learning framework, including multiple clients and one server, and jointly train a global model by combining multiple clients and one server to participate in federated learning;

[0048] The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier;

[0049] Each client initializes a local feature discriminator;

[0050] The local training module is used in the local stage, where each client downloads the global model and global generator from the server;

[0051] Each client uses the global model as the initial local model, and the generated data generated by the global generator and the local real data as training data. The output of the global model on the generated data serves as a teacher supervision and acts as a regular term for local model training to train the local model;

[0052] Each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator;

[0053] After completing the training, each client sends the trained local model and the trained local feature identifier to the server;

[0054] The global aggregation and update module is used in the global stage. The server receives the trained local models and trained local feature discriminators sent by each client.

[0055] The server aggregates the trained local models as the initial global model;

[0056] The server trains the global generator so that the generated samples can deceive multiple local feature discriminators;

[0057] The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model.

[0058] The convergence judgment and iteration module is used to return to execute the local training module until the global model converges to obtain the final global model.

[0059] Preferably, the system further comprises an image classification module, which is used to input the test image into the final global model to obtain an image classification result.

[0060] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned feature adversarial based data-free federated bidirectional knowledge distillation method.

[0061] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned feature-adversarial-based data-free federated bidirectional knowledge distillation method.

[0062] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0063] The embodiment of the present invention proposes a data-free federated bidirectional knowledge distillation method and system based on feature adversarial training. The feature adversarial training between the local feature discriminator of the client and the global generator of the server enables the global generator to generate generated samples with features similar to those of the real global samples. The generated samples and the local model are used to fine-tune the global model in the global stage, so that the global model can integrate and distill good classification capabilities from the local model. At the same time, the classification ability of the global model for the generated samples is distilled to the local model in the next local stage, so that the local model is not only good at classifying local data but also has the classification ability from a global perspective, which greatly alleviates the lack of classification ability of the local model for minority classes and missing classes. In this way, the bidirectional positive feedback distillation process of the local model and the global model improves the performance and stability of the model. In addition, a large number of experiments show that the method of the present invention is superior to most advanced methods in scenarios where the client data distribution is inconsistent. In scenarios where the data distribution is relatively different, the average test accuracy of the method of the present invention on the four data sets of FMNIST, SVHN, CIFAR10, and EMNIST is 4% to 6% higher than that of all the second-best methods, and the stability of the model performance is also better. Compared with other methods, the highest test accuracy of the method of the present invention on the EMNIST data set is the least affected by data distribution. The average performance of other methods decreases by 12.84% to 38.11%, while the method of the present invention only decreases by 7.45%. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0065] Figure 1 A flowchart of a data-free federated bidirectional knowledge distillation method based on feature adversarial provided in an embodiment;

[0066] Figure 2 A block diagram of a data-free federated bidirectional knowledge distillation system based on feature adversarial provided in an embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Embodiment 1

[0068] Most existing FL solutions attempt to mitigate the impact of uneven data distribution on model performance by adding model correction based on historical model parameters or historical update gradient information during the training process. These strategies aim to correct the update direction of the model on local data, causing the model to be over-biased to a certain extent. However, when there are many minority classes or missing classes in the client's data, such methods are not very effective, because the client is still training the model on its own data and cannot benefit from the knowledge about missing classes in historical information. Some data-free federated learning schemes hope to train a generator to compensate for this problem, but due to the limitations of privacy protection, the knowledge contained in the samples generated by the generator is often limited, resulting in poor performance of distillation. In addition, during the iterative communication process between the client and the server, this distilled knowledge is easily forgotten.

[0069] In order to solve the problem that the data distribution of clients is quite different in the existing technology, and the effect of data-free federated distillation is not significant when there are many minority classes and missing classes. Figure 1 As shown, the embodiment of the present invention proposes a data-free federated bidirectional knowledge distillation method based on feature confrontation, and the method includes:

[0070] Step S1: construct a federated learning framework, including multiple clients and one server, and jointly train a global model by uniting multiple clients and one server to participate in federated learning; the server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier; each client initializes a local feature discriminator;

[0071] Step S2: In the local stage, each client downloads the global model and global generator from the server;

[0072] Each client uses the global model as the initial local model, the generated data generated by the global generator and the local real data as training data, and the output of the global model on the generated data as a teacher supervision to act as a regular term for local model training to train the local model; each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator; after completing the training, each client sends the trained local model and the trained local feature discriminator to the server;

[0073] Step S3: In the global stage, the server receives the trained local models and the trained local feature discriminators sent by each client; the server aggregates the trained local models as the initial global model; the server trains the global generator so that the generated samples can deceive multiple local feature discriminators; the server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model;

[0074] Step S4: Return to step S2 and execute until the global model converges to obtain the final global model.

[0075] It can be seen from the above technical solution that the present invention proposes a data-free federated bidirectional knowledge distillation method based on feature adversarial, which effectively solves the problems of data islands and inconsistent distribution in federated learning by introducing feature adversarial and bidirectional knowledge distillation mechanisms. Through adversarial training of local feature discriminators and global generators, high-quality pseudo data is generated, which not only enriches the training samples, but also promotes the global model's recognition ability for minority classes and missing classes. The bidirectional distillation process of the global model and the local model enables the two to promote each other in classification ability, significantly improving the generalization performance and stability of the model, especially suitable for scenarios with large differences in client data distribution, showing significant advantages over existing methods.

[0076] In step S1, a federated learning framework is constructed, including multiple clients and a server. A global model is trained by uniting multiple clients and a server to participate in federated learning, where each client has its own private data set. The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier, where the global feature extractor is used to map data from the instance space to the global classifier. To feature space , the global classifier is used to map features to the category space ; The global generator is used to generate Sampling generates samples with global knowledge. The client initializes a local feature discriminator, which outputs a scalar that can identify whether the feature comes from the local domain or the global domain. Specifically, the global feature extractor consists of Parameterized, the global classifier is given by Parameterized, the global model ; The global generator is composed of Parameterization; the local feature discriminator of the kth client is given by Parameterization, where It means "define as" or "set to", and is an assignment operator.

[0077] In step S2, in the local stage, each client downloads the global model and the global generator from the server. First, each client uses the global model as the initial local model , the generated data generated by the global generator and the local real data are used as training data, and the output of the global model on the generated data serves as the teacher supervision and acts as the regular term for local model training. Then the loss function of training the local model of the client is:

[0078] (1) Among them, is the loss function of the local model of the kth client, E represents the mathematical expectation, is the distribution of the training data set for the kth customer, It is a local model The cross entropy loss on the local data x, is the distribution of samples generated by the global generator, It means obedience A generated sample from the distribution, namely , the specific formula for generating samples using the global generator is:

[0079] (2) Among them, is a Gaussian distribution of noise, is a uniform distribution of class labels, z means obey A noise vector distributed by Show obedience The label value of a generated sample of the distribution, G is the output function of the generator, is a hyperparameter that adjusts the regularized distillation loss, is the regularized distillation loss, which constrains the training direction of the local model through the output of the global model on the generated data, so that the local model still maintains the ability of the global model on the generated data. The specific formula is as follows:

[0080] (3) Among them, are the outputs of the local model and the mth layer of the global model, respectively, M is the total number of model layers, is the mean square error loss.

[0081] Subsequently, each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator, maximize the score of the positive sample, and minimize the score of the negative sample. The loss function of the kth local feature discriminator is:

[0082] (4) Among them, is the loss function of the kth local feature discriminator, is the output function of the kth local feature discriminator, is the output function of the kth local feature extractor, is the output function of the global feature extractor, is the local feature extractor of the kth local model.

[0083] After completing the training, each client sends the trained local model and the trained local feature identifier to the server.

[0084] In step S3, in the global stage, the server receives the trained local models and trained local feature identifiers sent by each client.

[0085] First, the local model trained by the server aggregation is used as the initial global model. The specific formula is as follows:

[0086] (5) Among them, is the set of clients participating in the current communication round, and its size is expressed as , is the size of the kth client dataset, is the size of the i-th client dataset.

[0087] Secondly, after obtaining the initial global model, the server trains the global generator so that the generated samples can deceive multiple local feature discriminators and generate samples with global features; the training of the global generator is based on three losses: feature adversarial loss, semantic loss, and intra-class diversity loss.

[0088] Feature Adversarial Loss for:

[0089] (6) Semantic Loss for: (7) Among them, is the cross entropy loss, The label value is the label value given in advance when generating the sample according to formula (2) ; Intra-class diversity loss for: (8) Among them, and are two sampling points in the noise space, and two samples of the same category are generated according to formula (2) and ; and Is to generate samples and The features extracted by the local feature extractor are given by the formula: , ; e is the natural logarithm, is the two-norm.

[0090] In summary, combining the three losses of the global generator, the final global generator loss is for:

[0091] (9) Among them, and These are all adjustable hyperparameters, which can be used to adjust the balance of generated samples in feature space, category space, and diversity.

[0092] Finally, the server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model. The global fine-tuning loss for: (10) Among them, are the outputs of the local model and the mth layer of the global model, respectively, M is the total number of model layers, It is the mean square error loss. The form of the global fine-tuning loss is consistent with the regularization loss of the local model, ensuring that the knowledge of bidirectional distillation is consistent.

[0093] In step S4, the process returns to step S2 until the global model converges to obtain the final global model.

[0094] Specifically, in each iteration, the client uses the updated global model and global generator for local training, and then sends the updated local model and updated local feature discriminator to the server. The server continues to aggregate local models, train the global generator, and fine-tune the global model with new generated data. After multiple iterations, the global generator is able to generate generated samples with features close to those of real global samples, and finally obtains a global model with excellent performance.

[0095] In addition, the method further comprises step S5: inputting the test image into the final global model to obtain an image classification result.

[0096] The algorithm flow of the data-free federated bidirectional knowledge distillation method based on feature adversarial provided by the present invention is shown in Table 1 below.

[0097] Table 1 Algorithm flow

[0098]

[0099] In order to illustrate the advantages of the method of the present invention, the following is an explanation in combination with specific experimental results. The data-free federated bidirectional knowledge distillation method based on feature adversarial is referred to as FedKCG, and the above algorithm flow is referred to as Algorithm 1. FedAvg, FedProx, Scafflod, FedGen, FedFTG and DaFKD are compared using four datasets: FMNIST, SVHN, CIFAR10 and EMNIST. Table 2 is a brief overview of these methods.

[0100] Table 2 A brief overview of several federated learning methods

[0101]

[0102] Experiment 1:

[0103] In this experiment, a simple convolutional neural network (CNN) was used as the basic skeleton for the global model of the technical solution, with three convolutional layers and two fully connected layers. The layers before the fully connected layers were considered feature extractors, and all fully connected layers were used as classifiers. The local feature discriminator architecture used three fully connected layers. For the global generator, one linear embedding layer and three convolutional layers were used.

[0104] According to Algorithm 1, set the communication round T=100, the number of clients K=100, and the active ratio C=0.1. In local training, use the SGD optimizer, the initial learning rate is 0.01, the momentum is 0.9, the batch size is 50, and the learning rate decay rate is 0.998. For all datasets, the dimension of the feature is 64. The discriminator and generator use the RMSprop optimizer, the learning rate is 5e-5, the batch size is 50, and the local feature extractor training rounds are =5, global generator training rounds = 20. For fine-tuning the global model using generated samples, the Adam optimizer is used with a learning rate of 2e-4. In addition, =0.1, =1 and = 1.

[0105] As shown in Table 3 below, in order to reduce the impact of randomness, the experimental results are the average of the Top-1 test accuracy based on three identical random seeds. Smaller β indicates more severe heterogeneity, and R represents the number of local training rounds. The results show that the method of the present invention achieves the best performance in most cases. More importantly, the method of the present invention is the one with the smallest drop in model performance due to enhanced data heterogeneity. For the FMNIST dataset, when β is reduced from 0.5 to 0.05, the average accuracy of the method of the present invention only drops by 2.12%, while FedAvg drops by 7.12%. Consistent phenomena can also be observed on other datasets.

[0106] Table 3 Average generalization accuracy of the proposed method and other methods

[0107]

[0108] Experiment 2:

[0109] In this experiment, a simple convolutional neural network (CNN) was used as the basic skeleton for the global model of the technical solution, with three convolutional layers and two fully connected layers. The layers before the fully connected layers were considered feature extractors, and all fully connected layers were used as classifiers. The local feature discriminator architecture used three fully connected layers. For the global generator, one linear embedding layer and three convolutional layers were used.

[0110] According to Algorithm 1, set the communication round T=100, the number of clients K=100, and the active ratio C=0.1. In local training, use the SGD optimizer, the initial learning rate is 0.01, the momentum is 0.9, the batch size is 50, and the learning rate decay rate is 0.998. For all datasets, the dimension of the feature is 64. The discriminator and generator use the RMSprop optimizer, the learning rate is 5e-5, the batch size is 50, and the local feature extractor training rounds are =5, global generator training rounds = 20. For fine-tuning the global model using generated samples, the Adam optimizer is used with a learning rate of 2e-4. In addition, =0.1, =1 and = 1.

[0111] Table 4 below shows the generalization stability of the method of the present invention and other methods. Under the same random seed 42, the client selected for each communication is ensured to be consistent, and the average accuracy and standard deviation of the last 10 communications of each method in Table 3 are calculated. In this experiment, we pay more attention to the generalization stability of the algorithm in heterogeneous federated learning. An important reason is that the inconsistent distribution and partial participation of customer data lead to unstable generalization of the model. In actual situations, the server cannot guarantee to show the validation set to verify the performance of the model. Therefore, even if an algorithm can achieve good performance but lacks stability at a certain moment in the training process, we cannot consider it to be an algorithm suitable for federated learning (FL). Existing studies have shown that under certain conditions, most federated learning algorithms can theoretically converge by controlling the learning rate decay, but the reasonable learning rate decay strategy is strongly a priori unknowable. Therefore, in the experiment, we ensure that the learning rate scheme of each algorithm is consistent. In Table 4, the results further demonstrate the superiority of the method of the present invention in heterogeneous scenarios. In the case of strong heterogeneity, FedKCG has an average accuracy of 4% to 6% higher than all the second-best methods on the four datasets, and also shows a smaller standard deviation. In particular, on the EMNIST dataset, when there are more categories, the present invention achieves better generalization stability, which is consistent with our expectation of using a generator with a global perspective to help the model learn missing categories.

[0112] Table 4 Generalization stability of the method of the present invention and other methods

[0113] Embodiment 2

[0114] like Figure 2 As shown, the present invention provides a data-free federated bidirectional knowledge distillation system based on feature confrontation, which is used to implement the data-free federated bidirectional knowledge distillation method based on feature confrontation of the above embodiment 1, and specifically includes:

[0115] Initialization module 100 is used to build a federated learning framework, including multiple clients and a server, and jointly train a global model by combining multiple clients and a server to participate in federated learning; the server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier; each client initializes a local feature identifier;

[0116] The local training module 200 is used in the local stage. Each client downloads the global model and the global generator from the server. Each client uses the global model as the initial local model, and uses the generated data generated by the global generator and the local real data as the training data. The output of the global model on the generated data serves as the teacher supervision and acts as the regular term for the local model training to train the local model. Each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator. After the training is completed, each client sends the trained local model and the trained local feature discriminator to the server.

[0117] The global aggregation and update module 300 is used in the global stage. The server receives the trained local models and the trained local feature identifiers sent by each client. The server aggregates the trained local models as the initial global model. The server trains the global generator so that the generated samples can deceive multiple local feature identifiers. The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model.

[0118] The convergence judgment and iteration module 400 is used to return to execute the local training module until the global model converges to obtain the final global model.

[0119] In addition, the system also includes an image classification module 500, which is used to input the test image into the final global model to obtain an image classification result.

[0120] A data-free federated bidirectional knowledge distillation system based on feature adversarial is provided in this embodiment, and is used to implement the aforementioned data-free federated bidirectional knowledge distillation method based on feature adversarial. Therefore, the specific implementation method of the data-free federated bidirectional knowledge distillation system based on feature adversarial can be seen in the implementation method part of the data-free federated bidirectional knowledge distillation method based on feature adversarial. For example, the initialization module 100, the local training module 200, the global aggregation and update module 300, the convergence judgment and iteration module 400, and the image classification module 500 are respectively used to implement steps S1, S2, S3, S4, and S5 in the aforementioned data-free federated bidirectional knowledge distillation method based on feature adversarial. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here. Embodiment 3

[0121] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned feature-adversarial-based data-free federated bidirectional knowledge distillation method. Embodiment 4

[0122] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned data-free federated bidirectional knowledge distillation method based on feature adversarial.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A data-free federated bidirectional knowledge distillation method based on feature adversarial, characterized in that: include: Step S1: construct a federated learning framework, including multiple clients and one server, and jointly train a global model by combining multiple clients and one server to participate in federated learning; The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier; Each client initializes a local feature discriminator; Step S1 specifically includes: Build a federated learning framework, including multiple clients and one server, and jointly train a global model by uniting multiple clients and one server to participate in federated learning, where each client has its own private dataset; The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier, wherein the global feature extractor is used to map data from the instance space X to the feature space I, and the global classifier is used to map the feature to the category space Y; the global generator is used to generate samples with global knowledge from the noise space Z; The client initializes a local feature discriminator, which outputs a scalar that can identify whether the feature comes from the local domain or the global domain; The global feature extractor is composed of Parameterized, the global classifier is given by Parameterized, the global model The global generator is composed of θ g Parameterization; the local feature discriminator of the kth client is given by Parameterization, where := means "define as" or "set to", and is an assignment operator; Step S2: In the local stage, each client downloads the global model and global generator from the server; Each client uses the global model as the initial local model, and the generated data generated by the global generator and the local real data as training data. The output of the global model on the generated data serves as a teacher supervision and acts as a regular term for local model training to train the local model; Each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator; After completing the training, each client sends the trained local model and the trained local feature identifier to the server; Step S2 specifically includes: In the local stage, each client downloads the global model and global generator from the server; Each client uses the global model as the initial local model w k :=w g , the generated data generated by the global generator and the local real data are used as training data, and the output of the global model on the generated data serves as the teacher supervision and acts as the regular term for local model training. Then the loss function of training the local model of the kth client is: in, is the loss function of the local model of the kth client, E represents the mathematical expectation, P k is the distribution of the training data set for the kth customer, L ce (w k ; x) is the local model w k The cross entropy loss on the local data x, P g is the distribution of samples generated by the global generator, then Indicates obedience to P g A generated sample from the distribution, namely The specific formula for generating samples using the global generator is: Among them, P z is a Gaussian distribution of noise, P y is a uniform distribution of category labels, z means it obeys P z A noise vector distributed by Indicates obedience to P y The label value of a generated sample of the distribution, G is the output function of the generator, λ reg is a hyperparameter that adjusts the regularized distillation loss, L reg is the regularized distillation loss, which constrains the training direction of the local model through the output of the global model on the generated data, so that the local model still maintains the ability of the global model on the generated data. The specific formula is as follows: in, are the outputs of the local model and the mth layer of the global model, respectively. M is the total number of model layers, and L mse is the mean square error loss; Subsequently, each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator, maximize the score of the positive sample, and minimize the score of the negative sample. The loss function of the kth local feature discriminator is: in, is the loss function of the kth local feature discriminator, D k is the output function of the kth local feature discriminator, F k is the output function of the kth local feature extractor, F g is the output function of the global feature extractor, is the local feature extractor of the kth local model; After completing the training, each client sends the trained local model and the trained local feature identifier to the server; Step S3: In the global stage, the server receives the trained local models and trained local feature identifiers sent by each client; The server aggregates the trained local models as the initial global model; The server trains the global generator so that the generated samples can deceive multiple local feature discriminators; The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model. Step S3 specifically includes: In the global stage, the server receives the trained local models and trained local feature discriminators sent by each client; The local model after server aggregation training is used as the initial global model. The specific formula is as follows: Among them, S t is the set of clients participating in the current communication round, and its size is represented by |S t |, N k is the size of the kth client dataset, N i is the size of the i-th client dataset; After obtaining the initial global model, the server trains the global generator so that the generated samples can deceive multiple local feature discriminators and generate samples with global features. The training of the global generator is based on three losses: feature adversarial loss, semantic loss, and intra-class diversity loss. Feature adversarial loss L adv for: Semantic loss L sem for: Among them, L ce is the cross entropy loss, The label value is the label value given in advance when generating the sample according to formula (2) Intra-class diversity loss L div for: Among them, z1 and z2 are two sampling points in the noise space, and two generated samples of the same category are generated according to formula (2): and v1 and v2 are generated samples and The features extracted by the local feature extractor are given by the formula: e is the natural logarithm, ∥·∥2 is the two-norm; Combining the three losses of the global generator, the final global generator loss L gen for: L gen =L adv +λ sem L sem +λ div L div (9) Among them, λ sem and λ div These are all adjustable hyperparameters, which can be used to adjust the balance of generated samples in feature space, category space, and diversity; The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model. The global fine-tuning loss L global for: in, are the outputs of the local model and the mth layer of the global model, respectively. M is the total number of model layers, and L mse It is the mean square error loss. The form of the global fine-tuning loss is consistent with the regularization loss of the local model, ensuring that the knowledge of bidirectional distillation is consistent; Step S4: Return to step S2 and execute until the global model converges to obtain the final global model; The method further includes: inputting the test image into the final global model to obtain an image classification result.

2. A data-free federated bidirectional knowledge distillation system based on feature adversarial, characterized in that: The system is used to implement the data-free federated bidirectional knowledge distillation method based on feature confrontation as described in claim 1, and specifically includes: The initialization module is used to build a federated learning framework, including multiple clients and one server, and jointly train a global model by combining multiple clients and one server to participate in federated learning; The server initializes a global model and a global generator, and divides the global model into a global feature extractor and a global classifier; Each client initializes a local feature discriminator; The local training module is used in the local stage, where each client downloads the global model and global generator from the server; Each client uses the global model as the initial local model, and the generated data generated by the global generator and the local real data as training data. The output of the global model on the generated data serves as a teacher supervision and acts as a regular term for local model training to train the local model; Each client uses the features of the local feature extractor on the real data as positive samples and the features of the global feature extractor on the generated data as negative samples to train the local feature discriminator; After completing the training, each client sends the trained local model and the trained local feature identifier to the server; The global aggregation and update module is used in the global stage. The server receives the trained local models and trained local feature discriminators sent by each client. The server aggregates the trained local models as the initial global model; The server trains the global generator so that the generated samples can deceive multiple local feature discriminators; The server uses the trained global generator to generate new generated data, and uses the new generated data as training data. The outputs of multiple local models on the generated data are used as teacher supervision to fine-tune the global model. The convergence judgment and iteration module is used to return to execute the local training module until the global model converges to obtain the final global model.

3. The data-free federated bidirectional knowledge distillation system based on feature adversarial according to claim 2 is characterized in that: The system also includes an image classification module, which is used to input the test image into the final global model to obtain an image classification result.

4. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the feature adversarial-based data-free federated bidirectional knowledge distillation method described in claim 1.

5. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the feature adversarial-based data-free federated bidirectional knowledge distillation method described in claim 1.

Citation Information

Patent Citations

  • Federal learning modeling optimization method and device, readable storage medium and program product

    CN113869533A

  • Federal comparative confrontation knowledge distillation machine account detection method and device

    CN117252204A