Federal learning knowledge distillation method based on data-free driving

Abstract

The invention discloses a federal learning knowledge distillation method based on data-free driving, and belongs to the technical field of personalized updating of federal learning. Comprising the steps that each client constructs a classification network and a generative adversarial network; the server side constructs a global model and a teacher model; each client enhances the private data set, is used for training a classification network and a generative adversarial network, and feeds back trained generator and classifier model parameters to the server; and the server side updates the global generator and the global classifier, trains the global model and the teacher model by using the distillation data, and issues optimized model parameters of the global generator and the global classifier to each client. According to the invention, rich knowledge of a teacher network is utilized to guide mutual learning with a global model at a server side; and a MixDistrition data enhancement technology is introduced into a client to carry out personalized updating, so that bidirectional flow and knowledge sharing of data are realized, and problems caused by data isomerism are relieved.

Description

Technical Field

[0001] The present invention relates to the field of personalized update technology of federated learning, and in particular to a federated learning knowledge distillation method based on non-data driven. Background Art

[0002] The rapid development of Internet of Things (IoT) technology has brought about data privacy and security issues, which are difficult to deal with with traditional centralized data processing methods. Federated Learning (FL), as a distributed machine learning method, avoids direct data sharing and effectively protects data privacy by training local models on different IoT devices and uploading model parameters. However, in its development process, challenges such as uneven client data distribution (Non-IID) and insufficient data for some clients (few-shot samples) still exist, which leads to significant performance degradation and poor convergence. To address these problems, most approaches alleviate the challenge of data heterogeneity by constraining the update of local models, but ignore the generalization limitations caused by simple aggregation of global models.

[0003] In order to cope with the challenge of data heterogeneity, the knowledge distillation (KD) method used to consolidate the aggregation strategy has attracted widespread attention. Ensemble Distillation for Robust Model Fusion in Federated Learning (FEDDF) is a representative method that uses unlabeled public datasets for knowledge distillation and guides the global model by integrating the integrated knowledge of local models. However, the use of carefully designed datasets on the server may be limited. If there is a significant distribution difference between public data and private data of the client, the global model will be difficult to adapt to the data distribution of the client. Therefore, the data-free knowledge distillation (DFKD) method was proposed and introduced into federated learning. FEDGEN (Data-free knowledge distillation for heterogeneous federated learning) is a representative method that introduces a global generator and trains the student network under the guidance of the teacher network by generating data without relying on external datasets for knowledge distillation. However, it guides the global model by integrating local models, which will cause the generalization limitation of the global model due to the compact nature (simple architecture) of the local model.

[0004] In summary, although these studies have made some progress in solving data heterogeneity, they have ignored the criticality of combining local and global. Specifically, it is difficult for the client to accurately obtain and utilize the rich distribution knowledge of global data, which is the main factor that leads to performance degradation due to data distribution heterogeneity. Therefore, the current update technology for FL faces three main problems: the use of public distilled datasets on the server will be limited and easily affect the performance of the global model; the simple teacher model architecture will cause generalization limitations of the global model; it is difficult for the client to accurately obtain and utilize the rich distribution knowledge of global data. Summary of the invention

[0005] In view of the defects of the prior art, the present invention establishes a federated learning framework for data-free knowledge distillation, introduces a teacher network in the knowledge distillation on the server side, uses the rich knowledge of the teacher network to guide mutual learning with the global model, and introduces MixDistribution data enhancement technology on the client side for personalized updates, so as to realize two-way flow of data and knowledge sharing, and alleviate the problem of data heterogeneity.

[0006] In order to achieve the above object, the present invention provides a data-free federated learning knowledge distillation method, the system involved includes K clients and 1 server, and includes the following steps:

[0007] (1) Each of the clients constructs a classification network and a generative adversarial network, wherein the classification network includes a feature extractor and a classifier, and the generative adversarial network includes a generator and a discriminator; the server constructs a global model and a teacher model, wherein the global model includes a global generator and a global classifier, and the teacher model includes a teacher generator and a teacher classifier;

[0008] (2) Each of the clients enhances the private data set to obtain an enhanced data set;

[0009] (3) The client receives the global generator and global classifier model parameters sent by the server, updates its generator and classifier; uses the private dataset and the enhanced dataset to train the client classification network and the generative adversarial network in turn, and then feeds back the generator and classifier model parameters to the server;

[0010] (4) The server receives the classifier and generator model parameters of all clients, and performs weighted average on them as updated global generator and global classifier model parameters;

[0011] (5) The server generates noise data and inputs it into the global model and the teacher model for training, and obtains optimized model parameters of the global generator and the global classifier through knowledge distillation;

[0012] (6) The server sends the optimized global generator and global classifier model parameters to each client for updating the client generator and classifier.

[0013] Furthermore, the step (2) is specifically as follows:

[0014] (2.1) The private data set data {x i} |D| Randomly shuffle and get Rand{x i} |D| ;

[0015] (2.2) Calculate the disturbance statistic ρ mix and mix

[0016] ρ mix =ψσ({x i} |D| )+(1-ψ)σ(Rand{x i} |D| )

[0017] υ mix =ψm({x i} |D| )+(1-ψ)m(Rand{x i} |D| )

[0018] (2.3) According to the disturbance statistics ρ mix and mix Reconstruct the data set and obtain the enhanced data set data

[0019]

[0020] Where: σ and μ represent the standard deviation and mean respectively; ψ is the weighting coefficient, and ψ ≥ 0.

[0021] Furthermore, the training process of the client classification network in step (3) is specifically as follows:

[0022] (3.1.1) The feature extractor E i Feature extraction is performed on the private dataset and the enhanced dataset respectively, and two features are obtained and

[0023] (3.1.2) The classifier C i The two extracted features are classified and predicted respectively to obtain two predicted values, and the classification loss between the two predicted values ​​and the true label is calculated respectively to obtain the first classification loss L C ;

[0024]

[0025] Among them: y represents the true label, x represents the private dataset data, Represents positive enhancement dataset data; represents the classifier model parameters, represents the feature extractor model parameters; H is the cross entropy function; E represents the expected value;

[0026] (3.1.3) Freeze the global generator G S The client generates Gaussian noise z with the same batch size as the private dataset, and inputs it and the label y into the global generator to obtain the feature

[0027] (3.1.4) According to the global generator G S The generated features are compared with the feature extractor E i The two features extracted give the second loss function;

[0028] The global generator G S The generated features are respectively compared with the feature extractor E i The minimum mean square error is calculated between the two extracted features;

[0029]

[0030] The global generator G S The generated features are respectively compared with the feature extractor E i The two extracted features calculate the similarity matrix, and obtain two probability distributions through the softmax function, and calculate their divergence loss;

[0031]

[0032] According to the minimum mean square error and divergence loss, the second loss function L is obtained E ;

[0033]

[0034] Where: φ represents the softmax function, T is the matrix transpose; Represents the generator model parameters;

[0035] (3.1.5) According to the first loss function and the second loss function, the total classification network loss L is obtained EC , optimize the classification network according to minimizing the total classification network loss;

[0036] L EC =LC +γL E

[0037] Among them: γ is a non-negative hyperparameter.

[0038] Furthermore, the training process of the client-side generated adversarial network in step (3) is specifically as follows:

[0039] (3.2.1) Freeze the feature extractor E of the client classification network i The model parameters of the private data set are obtained by extracting the features of the private data set to obtain a first feature;

[0040] (3.2.2) The client generates Gaussian noise z having the same batch size as the private dataset, and inputs it and the label y into the generator of the generative adversarial network to obtain a second feature;

[0041] (3.2.3) Input the first feature and the second feature into the discriminator D of the generative adversarial network respectively i , calculate the identification loss L D and the generation loss L G ;

[0042]

[0043] Where: Ω is the binary cross entropy function; 1 and 0 represent the scalar values ​​of the true label and the pseudo label respectively; E represents the expected value;

[0044] (3.2.4) Optimize the discriminator D of the adversarial network by minimizing the identification loss and generation loss i and the generator G i .

[0045] Furthermore, the step (5) is specifically as follows:

[0046] (5.1) The server generates a set of Gaussian noise z that follows a Gaussian distribution g , corresponding to label y g Obeying uniform distribution, it is input into the global generator of the global model and the teacher generator of the teacher model to generate distilled data, and then the distilled data is input into the global classifier and the teacher classifier respectively; and the probability distribution P is calculated T and P S ;

[0047]

[0048]

[0049] Where: φ is the softmax function; G S A generator representing the global model, represents the global model generator parameters; C S The classifier representing the global model, represents the global model classifier parameters; G T represents the generator of the teacher model, represents the teacher model generator parameters; C T represents the classifier of the teacher model, Represents the teacher model classifier parameters;

[0050] (5.2) Calculate the loss function based on the KL of the two probability distributions, and minimize the loss function to optimize the global model and the teacher model; L KD =E z～Z E y～ω [KL(P T ,P S )]

[0051] Where: KL is the KL divergence function; E represents the expected value.

[0052] Beneficial effects of the present invention:

[0053] The present invention uses a generator on the server side to generate the data required for knowledge distillation, thereby avoiding the use of a public data set, that is, a knowledge distillation method without public data. At the same time, by introducing a teacher network, the rich knowledge of the teacher network is used to guide mutual learning with the global model. Data enhancement technology is introduced on the client side. On the one hand, private data and enhanced data are classified and predicted, and the client model is trained by minimizing the classification loss; on the other hand, private data and enhanced data are used to calculate the similarity matrix of global knowledge respectively, and then the client learns global knowledge by minimizing the KL divergence loss, realizing two-way data flow and knowledge sharing, thereby alleviating the problem of data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the overall framework of the knowledge distillation method based on non-data-driven federated learning according to an embodiment of the present invention.

[0055] Figure 2 The figure is a schematic diagram of the client classification network training process according to an embodiment of the present invention.

[0056] Figure 3 A schematic diagram of the adversarial network training process generated by the client in an embodiment of the present invention.

[0057] Figure 4 Schematic diagram of the server-side knowledge distillation training process according to an embodiment of the present invention.

[0058] Figure 5 The figure is a schematic diagram for comparing the performance of the method according to the embodiment of the present invention with other methods. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and embodiments. Except for the contents specifically mentioned below, the remaining contents are common knowledge in the art and the present invention has no particular limitations.

[0060] like Figure 1 As shown, an embodiment of the present invention provides a knowledge distillation method for federated learning based on non-data-driven, and the system involved includes K clients and 1 server.

[0061] The federated learning framework proposed in the present invention includes four stages: client data enhancement stage, client classification network training stage, client generative adversarial network training stage and server knowledge distillation stage. Here, the complete federated process is called communication. Each communication round first trains the client. After the client training is completed, the parameters are uploaded to the server. After receiving the parameters, the server performs KD. After completing KD, the global model parameters of the server are sent back to the client. The method includes the following steps:

[0062] S101. Each of the clients constructs a classification network and a generative adversarial network, the classification network includes a feature extractor and a classifier, and the generative adversarial network includes a generator and a discriminator; the server side constructs a global model and a teacher model, the global model includes a global generator and a global classifier, and the teacher model includes a teacher generator and a teacher classifier.

[0063] Here we take a feature extractor E i and a classifier C i As a classification network, a generator G i and a discriminator D i As a generative adversarial network (GAN), the classification network and GAN are deployed together on the client. The server deploys the global generator G S and the global classifier C S , teacher generator G T and the teacher classifier C T .

[0064] S102: Each of the clients enhances a private data set to obtain an enhanced data set.

[0065] Consider the client private dataset as The augmented dataset is

[0066] First, the private dataset data {x i} D Randomly shuffle and get Rand{x i} |D| .

[0067] Then calculate the disturbance statistic ρ mix and mix , which respectively control the scaling and offset of the data distribution. Specifically, we transform the original data {x i} |D| and shuffle the data Rand{x i} |D| The standard deviation σ and mean μ of are weighted by ψ, where ψ is a non-negative hyperparameter from the beta distribution beta(δ,δ), where δ is a hyperparameter. The main function of ψ is to serve as a weighting coefficient. The specific method is as follows:

[0068] ρ mix =ψσ({x i} |D| )+(1-ψ)σ(Rand{x i} |D| )

[0069] υ mix =ψm({x i} |D| )+(1-ψ)m(Rand{x i} |D| )

[0070] Finally, the original data is reconstructed to obtain the enhanced data set data

[0071]

[0072] Where: σ and μ represent the standard deviation and mean respectively; ψ is the weighting coefficient, and ψ ≥ 0.

[0073] S103, the client receives the global generator and global classifier model parameters sent by the server, updates its generator and classifier; uses the private data set and the enhanced data set to train the client classification network and the generative adversarial network in turn, and then feeds back the generator and classifier model parameters to the server.

[0074] The client training includes the classification network training phase and the generative adversarial network training phase, as follows:

[0075] (1) Figure 2 As shown in the figure, during the client classification network training process, the client feature extractor E is optimized by minimizing the classification loss. i and classifier C i .

[0076] (1.1) Feature Extractor E i From xi and of Extract features from the and

[0077] (1.2) Classifier C i The two extracted features are classified and predicted respectively to obtain two predicted values, and the classification loss between the two predicted values ​​and the true label is calculated respectively to obtain the first classification loss L C .

[0078]

[0079] L cla and Describes the classifier for x i and The difference between the classification result and the true label y is calculated, and then the loss is integrated to obtain:

[0080]

[0081] Among them: y represents the true label, x represents the private dataset data, Represents the augmented dataset data; represents the classifier model parameters, represents the feature extractor model parameters; H is the cross entropy function.

[0082] (1.3) Freeze the received global generator G S The client generates Gaussian noise z with the same batch size as the private dataset and inputs it and the label y into the global generator to obtain the feature

[0083] Here, the shared knowledge and public knowledge (aggregated in the previous round) fused in the global generator are introduced into the local extractor, so there is an additional global generator on the client side, whose model parameters remain unchanged during the training process. The Gaussian noise z follows the Gaussian distribution N(0,1).

[0084] (1.4) Based on the features generated by the global generator and the feature extractor E i The two features are extracted to obtain the second loss function.

[0085] First, the features generated by the global generator are combined with the feature extractor E i The minimum mean square error loss is calculated for the two extracted features.

[0086]

[0087] Secondly, consider using the knowledge of the global generator to help the local extractor learn the structural knowledge of data distribution. The features generated by the global generator are combined with the feature extractor E i The two extracted features calculate the similarity matrix, and obtain two probability distributions through the softmax function, and calculate their divergence loss.

[0088]

[0089] According to the minimum mean square error and divergence loss, the second loss function L is obtained E ;

[0090]

[0091] Where: φ represents the softmax function, T is the matrix transpose; Represents the generator model parameters.

[0092] (1.5) According to the first loss function and the second loss function, the total classification network loss L is obtained EC , the classification network is optimized according to minimizing the total classification network loss.

[0093] L EC =L C +γL E

[0094] Among them: γ is a non-negative hyperparameter used to balance the proportion between the two loss terms. In the study, it was found that as the global generator generates more accurate feature representations during training, γ gradually increases from 0 to 1.

[0095] (2) Figure 3 As shown in the figure, during the training process of the client-side generative adversarial network, the discriminator D is optimized by minimizing the discriminator loss and the generator loss. i and the generator G i .

[0096] (2.1) Frozen feature extractor E i The model parameters are extracted features, and obtain the first feature.

[0097] The client will x i Feed into feature extractor E i , and obtain the “groundtruth” feature representation h.

[0098] (2.2) Client generation and Gaussian noise z with the same batch size (same as step (1.3)) is fed into the generator G of the generative adversarial network with the label y i , and obtain the second feature.

[0099] Feed (z, y) into the generator G i , to generate an estimated feature representation

[0100] (2.3) Input the first feature and the second feature into the discriminator D of the generative adversarial network respectively i , calculate the identification loss L D and the generation loss L G .

[0101]

[0102] Where: Ω is the binary cross entropy function; 1 and 0 represent the scalar values ​​of the true label and pseudo label, respectively.

[0103] (2.4) Optimize the discriminator D of the adversarial network by minimizing the discrimination loss and the generation loss i and the generator G i .

[0104] Finally, the parameters of the generator model after training are and classifier model parameters Feedback to the server.

[0105] S104, the server receives the classifier and generator model parameters of all clients, and performs weighted average on them as updated global generator and global classifier model parameters.

[0106] After the client training is completed, the parameters are uploaded to the server. After receiving the parameters, the server performs weighted averaging and then uses the weighted averaged generator and classifier model parameters to update the global generator and global classifier.

[0107] S105, the server generates noise data and inputs it into the global model and the teacher model for training, and obtains optimized model parameters of the global generator and the global classifier through knowledge distillation.

[0108] like Figure 4 As shown, the embodiment of the present invention introduces a large pre-trained teacher network on the framework of KD, and uses the rich knowledge of the teacher network to guide mutual learning with the global model. The specific process is as follows:

[0109] (5.1) The server generates a set of Gaussian noise z that follows a Gaussian distribution g , corresponding to label y g Obeying the uniform distribution, it is input into the global generator of the global model and the teacher generator of the teacher model to generate distilled data, and then the distilled data is input into the global classifier and the teacher classifier respectively, and the probability distribution P is calculated. T and P S :

[0110]

[0111] Where: z g Obeying Gaussian distribution N(0,1), y g Obeys uniform distribution U(0,c); φ is softmax function; G S A generator representing the global model, represents the global model generator parameters; C S The classifier representing the global model, represents the global model classifier parameters; G T represents the generator of the teacher model, represents the teacher model generator parameters; C T represents the classifier of the teacher model, Represents the teacher model classifier parameters.

[0112] (5.3) The loss function is calculated based on the KL of the two probability distributions, and the global model and the teacher model are optimized by minimizing the loss function.

[0113] L KD =E z～Z E y～ω [KL(P T ,P S )]

[0114] Where: KL is the KL divergence function.

[0115] S106. The server sends the optimized global generator and global classifier model parameters to each client for updating the client generator and classifier.

[0116] Once the server-side distillation is completed, the server will train the global model parameters θ S C and θ S G Distribute to each client.

[0117] In order to further illustrate the performance of the present invention, the embodiments of the present invention conduct sufficient experiments on three public data sets, namely FMNIST, SVHN, and Office-Caltech10, to verify the algorithm proposed in the present invention. Among them, FMNIST and SVHN simulate Non-IID settings (non-independent and identically distributed). Office-Caltech10 contains 10 office supplies from 4 fields, namely Amazon, DSLR, Webcam and Caltech. It is naturally suitable for Non-IID situations. At the same time, its sample size is limited (about 1910 pictures), which constitutes a few-sample scenario in FL. The 4 fields of Office-Caltech10 are regarded as 4 clients, 50% of the original data of each field is used as the training set, and the rest is evenly distributed to the test set and the validation set. For FMNIST and SVHN, four client scenarios are also provided, and 8,000 and 20,000 images are randomly sampled for FMNIST and SVHN respectively as local training sets. At the same time, in order to build Non-IID, the samples are distributed to the four clients according to the ratio of [10%, 20%, 30%, 40%].

[0118] In order to compare the performance, six existing methods are selected as comparative examples, and in-depth experiments are carried out on three different datasets. The effectiveness is verified by comparing the average client accuracy. Among them, FEDPROX regulates local targets by adding regularization terms; FEDDF introduces unlabeled datasets on the server side for knowledge distillation; FDSPLIT splits the local model and only uploads the classifier to the server for aggregation to protect privacy; FEDGEN trains the global generator to guide the client model training; FEDCG uploads the generator and classifier to the server, and aggregates the global model through data-free knowledge mutual distillation. At the same time, LOCAL is introduced as a comparison scheme, which is an ablation of FL. It only uses local data for training on the client without communicating with the server. FEDKDG represents an embodiment of the present invention. The experimental results are shown in Table 1:

[0119] Table 1

[0120]

[0121] It can be seen that FEDKDG shows the best accuracy in the datasets of the four Non-IID scenarios. The accuracy on FMNIST and SVHN is 1.34% and 0.48% higher than the second-ranked algorithm respectively, and the accuracy on Office is 5.69% higher than the second-ranked FEDCG, which proves that the present invention can alleviate the problem of few-sample scenarios in FL and make the client samples more diverse. It also shows that FEDKDG performs better in Non-IID scenarios with unbalanced sample numbers and different sample feature distributions.

[0122] In addition, in order to verify the performance of each client on FEDKDG, the accuracy of different clients of FEDKDG and 6 baselines under 3 datasets is also compared. Figure 5 As shown in the figure, it can be seen that FEDKDG has higher accuracy than other methods on the four clients of the datasets FMNIST, Office, and SVHN in the IID scenario. Especially on the dataset Office, the accuracy of each client is almost 12% higher than that of the lowest-accurate client in other baselines. On the dataset SVHN, FEDKDG has higher accuracy than other methods on three clients.

[0123] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for knowledge distillation based on federated learning without data drive, the system involved includes K clients and 1 server, characterized in that: The steps include: (1) Each of the clients constructs a classification network and a generative adversarial network, wherein the classification network includes a feature extractor and a classifier, and the generative adversarial network includes a generator and a discriminator; The server side constructs a global model and a teacher model, wherein the global model includes a global generator and a global classifier, and the teacher model includes a teacher generator and a teacher classifier; (2) Each of the clients enhances the private data set to obtain an enhanced data set; (3) The client receives the global generator and global classifier model parameters sent by the server, updates its generator and classifier; uses the private dataset and the enhanced dataset to train the client classification network and the generative adversarial network in turn, and then feeds back the generator and classifier model parameters to the server; (4) The server receives the classifier and generator model parameters of all clients, and performs weighted average on them as updated global generator and global classifier model parameters; (5) The server generates noise data and inputs it into the global model and the teacher model for training, and obtains optimized model parameters of the global generator and the global classifier through knowledge distillation; (6) The server sends the optimized global generator and global classifier model parameters to each client for updating the client generator and classifier.

2. The method for knowledge distillation based on non-data-driven federated learning according to claim 1, characterized in that: The step (2) is specifically: (2.1) The private data set data {x i } D Randomly shuffle and get Rand{x i } D ; (2.2) Calculate the disturbance statistic ρ mix and mix r mix =ψσ({x i } |D| )+(1-ψ)σ(Rand{x i } |D| ) u mix =ψm({x i } D )+(1-ψ)m(Rand{x i } D ) (2.3) According to the disturbance statistics ρ mix and mix Reconstruct the data set and obtain the enhanced data set data Where: σ and μ represent the standard deviation and mean respectively; ψ is the weighting coefficient, and ψ ≥ 0.

3. The method for knowledge distillation based on non-data-driven federated learning according to claim 1, characterized in that: The training process of the client classification network in step (3) is specifically as follows: (3.1.1) The feature extractor E i Feature extraction is performed on the private dataset and the enhanced dataset respectively, and two features are obtained and (3.1.2) The classifier C i The two extracted features are classified and predicted respectively to obtain two predicted values, and the classification loss between the two predicted values ​​and the true label is calculated respectively to obtain the first classification loss L C ; Among them: y represents the true label, x represents the private dataset data, Represents positive enhancement dataset data; represents the classifier model parameters, represents the feature extractor model parameters; H is the cross entropy function; E represents the expected value; (3.1.3) Freeze the global generator G S The client generates Gaussian noise z with the same batch size as the private dataset, and inputs it and the label y into the global generator to obtain the feature (3.1.4) According to the global generator G S The generated features are compared with the feature extractor E i The two features extracted give the second loss function; The global generator G S The generated features are respectively compared with the feature extractor E i The minimum mean square error is calculated between the two extracted features; The global generator G S The generated features are respectively compared with the feature extractor E i The two extracted features calculate the similarity matrix, and obtain two probability distributions through the softmax function, and calculate their divergence loss; According to the minimum mean square error and divergence loss, the second loss function L is obtained E ; Where: φ represents the softmax function, T is the matrix transpose; Represents the generator model parameters; (3.1.5) According to the first loss function and the second loss function, the total classification network loss L is obtained EC , optimize the classification network according to minimizing the total classification network loss; THE EC =L C +γL E Among them: γ is a non-negative hyperparameter.

4. The method for knowledge distillation based on non-data-driven federated learning according to claim 1, characterized in that: The training process of the client-generated adversarial network in step (3) is specifically as follows: (3.2.1) Freeze the feature extractor E of the client classification network i The model parameters of the private data set are obtained by extracting the features of the private data set to obtain a first feature; (3.2.2) The client generates Gaussian noise z having the same batch size as the private dataset, and inputs it and the label y into the generator of the generative adversarial network to obtain a second feature; (3.2.3) Input the first feature and the second feature into the discriminator D of the generative adversarial network respectively i , calculate the identification loss L D and the generation loss L G ; Where: Ω is the binary cross entropy function; 1 and 0 represent the scalar values ​​of the true label and the pseudo label respectively; E represents the expected value; (3.2.4) Optimize the discriminator D of the adversarial network by minimizing the identification loss and generation loss i and the generator G i .

5. The method for knowledge distillation based on non-data-driven federated learning according to claim 1, characterized in that: The step (5) is specifically as follows: (5.1) The server generates a set of Gaussian noise z that obeys Gaussian distribution g , corresponding to label y g Obeying uniform distribution, it is input into the global generator of the global model and the teacher generator of the teacher model to generate distilled data, and then the distilled data is input into the global classifier and the teacher classifier respectively; and the probability distribution P is calculated T and P S ; Where: φ is the softmax function; G S A generator representing the global model, represents the global model generator parameters; C S The classifier representing the global model, represents the global model classifier parameters; G T represents the generator of the teacher model, represents the teacher model generator parameters; C T represents the classifier of the teacher model, Represents the teacher model classifier parameters; (5.2) Calculating the loss function according to the KL of the two probability distributions, and minimizing the loss function to optimize the global model and the teacher model; L KD =E z～Z HAVE BEEN y～ω [KL(P T ,P S )] Where: KL is the KL divergence function; E represents the expected value.

Images