A federated learning method and system based on knowledge distillation

Abstract

The application provides a federated learning method and system based on knowledge distillation, which comprises the following steps: in the last step of the t-1th round of training, the server integrates all the local models uploaded by the clients in the active state in the current round to obtain an ACA model; all the local models in the cache slot are integrated to obtain an OCA model; in the training round, the server broadcasts the latest ACA model and OCA model to the clients in the active state; in the clients, the knowledge of the teacher model is transmitted to the local model by using a data-free knowledge distillation technology; then all the clients in the active state upload the local model to the server; the server updates the local model in the cache slot, wherein the local model in the cache slot corresponding to the client not in the active state in the current round remains unchanged; after the update, the server re-integrates the local model to obtain a new ACA model and a new OCA model; and the next round of training is returned.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of big data processing, and specifically relates to a federated learning method and system based on knowledge distillation. BACKGROUND

[0002] In the era of big data, big data within the same industry and between different industries can be combined to generate great value. In the field of artificial intelligence, the traditional data processing mode is often centralized, and the data collected by each party is centralized for preprocessing, cleaning and modeling. However, with the development and application of big data related technologies and artificial intelligence, data security and privacy are increasingly valued by relevant departments and enterprises. The increasingly stringent requirements for data privacy have caused the problem of "data silos". To solve this problem, Google first proposed the concept of federated learning. Federated learning is a machine learning framework that enables collaborative training by data owners while ensuring data security and privacy. It allows different data owners to collaborate without sharing data, sharing data value without sharing data.

[0003] However, the data in federated learning is independently generated by the data owner, and the data of each participant usually has different distribution characteristics, which means that the data distribution in the federated learning system does not necessarily meet the premise assumption of traditional distributed machine learning optimization algorithm that the data is independent and identically distributed (IID). In most cases, the data distribution in federated learning is non-independent and identically distributed (Non-IID). The Non-IID data distribution feature makes the model training speed in federated learning slower, the model accuracy lower, and the model training requiring more communication overhead, increasing the difficulty of model training. Therefore, the research on efficient federated learning algorithm under non-independent and identically distributed data has important research significance and important value for the application of federated learning system.

[0004] McMahan et al. proposed the Federated Averaging (FedAvg) algorithm, which greatly reduces the communication cost in federated learning, and enables the federated learning model to obtain an accuracy close to centralized training under independent and identically distributed data, but the performance of the algorithm will decrease significantly under non-independent and identically distributed data.

[0005] To solve the influence of non-independent and identically distributed data on the federated learning model, Li et al. introduced the idea of regularization into federated learning and proposed the FedProx algorithm, which modifies the loss function of the client local model training, prevents the local model from over "drifting" by limiting the difference between the local model update and the initial local model (i.e. the latest global model). However, this algorithm simply constrains the training of the local model and does not fully utilize the global information, so the performance of the final global model is not significantly improved.

[0006] Jeong et al. proposed the FAug algorithm, which attempts to prevent the local model from "drifting" by generating adversarial networks for data augmentation on local data. However, this algorithm requires the client to upload a small amount of local data as seed data for training the generative adversarial network, which seriously threatens the data privacy and security of the client.

[0007] Zhu et al. proposed the FedGen algorithm, which trains a feature generator on the server that conforms to the global data distribution. Before the client performs local model training, the server broadcasts the latest global model and the feature generator to the client, and the client generates features using the feature generator to "correct" the update of the local model during local model training, preventing the local model from over "drifting". However, this algorithm requires the client to upload the labels of the local data, which threatens the data privacy and security of the client.

[0008] Yao et al. proposed the FedGKD algorithm, which saves the latest 5 rounds of global models on the server and averages them to obtain a teacher model. The client needs to download the latest global model and the teacher model at the same time, and during the training of the local model, the knowledge of the teacher model is transferred to the local model using knowledge distillation to prevent the local model from "drifting". Since this algorithm requires the client to download both the global model and the teacher model, it requires more communication cost than FedAvg. The data used by the client during knowledge distillation is the local data of the client, which does not conform to the global data distribution in the non-independent and identically distributed scenario, and cannot transfer the global knowledge of the teacher model to the local model. SUMMARY

[0009] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a federated learning method and system based on knowledge distillation, which allows the client local model training to utilize global knowledge, uses knowledge distillation to transfer global knowledge to the local model, and fuses global knowledge and local knowledge to prevent the local model from "drifting", thereby accelerating the convergence speed of the global model and improving the accuracy of the global model.

[0010] To achieve the above purpose, the technical solution adopted by the present application is:

[0011] The first aspect of the application provides a federated learning method based on knowledge distillation, comprising:

[0012] A service end and a client are configured, wherein the client is used to train a local model, the service end maintains K different cache slots for storing the latest local model; only the client in an active state needs to upload the local model to the service end in each round of training;

[0013] The service end and the client implement federated learning in the following method:

[0014] Step 1, in the last step of the t-1th round of training, the service end integrates the local models uploaded by all the clients in the active state in the current round, that is, to obtain an ACA model representing a global model; at the same time, all the local models in the cache slots are integrated, that is, to obtain an OCA model representing global knowledge;

[0015] Step 2, in the tth round of training, only the client in the active state participates in the training, and c1,…,c m represent the ID of the client in the active state in the current round, and the service end broadcasts the latest ACA model and OCA model to c1,…,c m ;

[0016] Step 3, after receiving the ACA model and the OCA model, each client in the active state respectively takes the ACA model and the OCA model as an initial local model and a teacher model, and respectively uses w k and to represent, wherein the initial local model acts as a student model in the knowledge distillation process;

[0017] Step 4, a data-free knowledge distillation technology is used to transfer the knowledge of the teacher model to the local model; in the data-free knowledge distillation, a generator for generating pseudo samples to realize knowledge distillation is trained; at the same time, the local model is trained by using the local data of each client to fuse and transfer the global knowledge Global from the teacher model and the local knowledge Local from the local data to the local model;

[0018] Step 5, after the Global-Local knowledge fusion, all the clients in the active state upload the local model to the service end;

[0019] Step 6, after receiving the local model uploaded by the client, the service end updates the local model in the cache slot according to the ID of the client, wherein the local model of the cache slot corresponding to the client not in the active state in the current round remains unchanged;

[0020] After updating the local model of all active client corresponding cache slots, the server re-integrates the local model to obtain a new ACA model and a new OCA model;

[0021] Return to step 1 for the next round of training.

[0022] Based on the above, in step 4, the loss function of the generator for generating pseudo samples to realize knowledge distillation is trained in the following way:

[0023] Define g(·) to represent the output of the generator, and respectively represent the feature vector output and the probability vector output of the teacher model, input noise vector z ~ N(0,1, the generator output pseudo sample Input pseudo sample Teacher model output probability vector

[0024] Design One-Hot loss function:

[0025] Pseudo sample is expected to be predicted by the teacher model with a high probability as a certain specific class, therefore, by to calculate the pseudo label, the One-Hot loss function is defined as:

[0026]

[0027] Where CE is cross entropy, if L OH is minimized, the sample generated by the generator is classified into a certain class by the teacher model with a very high probability, which is also true for the real sample;

[0028] Design information entropy loss function:

[0029] In order to make the data generated by the generator cover all classes, information entropy is used to measure the uniformity of the distribution of generated data; that is, given a probability vector p = (p1,…,p d ), then the information entropy is expressed as Therefore, the information entropy loss function is defined as:

[0030]

[0031] When L IE is close to the minimum value, the data generated by the generator tends to have the same number of each class; therefore, minimizing the information entropy loss function will get a relatively balanced data set;

[0032] Design Activation loss function:

[0033] The feature vector of the real sample tends to obtain a higher activation value, and therefore, the Activation loss function is defined as:

[0034]

[0035] where ||·||1 represents the l1 norm;

[0036] The total loss function of the trained generator is:

[0037] L G =L IE +λ1L OH +λ2L A

[0038] where λ1 and λ2 are hyperparameters that balance the three loss functions.

[0039] Based on the above, the client uses the Global-Local knowledge fusion technology to train the local model, and trains the loss function in the following method:

[0040] Define h(·) to represent the probability vector output of the local model;

[0041] Design the KL loss function:

[0042] In order to let each client use the data generated by the generator to pass the global knowledge from the teacher model to the local model, and at the same time, the local model learns the local knowledge from the local data, then the KL loss function is defined as:

[0043]

[0044] where KL represents Kullback-Leibler divergence divergence; minimizing L KL makes the local model approach the teacher model, that is, learns the global knowledge;

[0045] Design the CE loss function:

[0046] Let represent the local data of the kth client, then the CE loss function is defined as:

[0047]

[0048] where CE represents cross-entropy, minimizing L CE makes the local model learn the local knowledge from the local data;

[0049] The total loss function of Global-Local knowledge fusion is:

[0050] L = L CE + gamma * L KL

[0051] where gamma is a hyper-parameter to balance the two loss functions; minimizing L makes the Global-Local knowledge fusion into the local model.

[0052] The second aspect of the application provides a federated learning system based on knowledge distillation, characterized in that it comprises a server and a client, wherein the client is used to train a local model, and the server maintains K different cache slots for storing the latest local model; in each round of training, only the client in the active state needs to upload the local model to the server.

[0053] The server and the client communicate to implement the federated learning method based on knowledge distillation. The application has outstanding substantial features and significant progress compared with the prior art, specifically:

[0054] 1. The application integrates Active-Inactive model integration technology and Global-Local knowledge fusion technology to design a privacy-protected federated learning method, which realizes high model performance and high fairness in the data heterogeneous (i.e. the local data of each client is not independent and identically distributed) scenario.

[0055] 2. When the server model is integrated, the application fully utilizes the local models of all clients and uses Active-Inactive model integration technology to obtain an OCA model that accurately represents global knowledge. For most previous federated learning methods, only the local models of the clients in the active state are involved in the server model integration, and a global model, i.e. an ACA model, is finally obtained. In contrast, the application integrates the local models of all clients, including the clients in the active state and the clients not in the active state, to obtain an OCA model that represents global knowledge. Therefore, the Active-Inactive model integration technology supports more accurate global knowledge representation and is a simple and accurate method for generating a global model. It can be combined with many federated learning methods (such as FedAvg, FedProx, etc.) to improve their performance.

[0056] 3、The client local model training considers global knowledge and local knowledge at the same time, and the global-local knowledge fusion technology is used to fuse the global knowledge and the local knowledge into the local model. In order to transfer the knowledge from the global model (teacher model) to the local model (student model), a data set used for training the global model is usually required. However, the client only has local data, and the distribution is usually inconsistent with the global data. Therefore, the data-free knowledge distillation technology is adopted to transfer the knowledge, and the problem that the client does not have global data is solved. In the data-free knowledge distillation, a generator needs to be trained to generate data for knowledge distillation. It is worth noting that the data generated by the application does not need to be very similar to the distribution of the real training data, and the only requirement is that the generated data can be used to promote knowledge transfer. Therefore, the requirements for the generator in the data-free knowledge distillation and the requirements for the generator in the traditional generative adversarial network (GAN) are different, because the generator in the traditional generative adversarial network needs to generate false pictures that are visually similar to real pictures, and can achieve the effect of deceiving the real; However, the generator in the data-free knowledge distillation only needs to generate false pictures that can transfer knowledge, and does not require visual effect. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The design schematic diagram of the application. DETAILED DESCRIPTION

[0058] The technical solutions of the application will be further described in detail through specific embodiments.

[0059] As shown in the figure, the embodiment provides a federated learning method based on knowledge distillation and a federated learning system based on knowledge distillation. Figure 1

[0060] Among them, the federated learning system includes: a server (Server) and a client (Client), wherein the client is used to train a local model, and the server maintains K different cache slots for storing the latest local model; In each round of training, only the active client (Active Client) needs to upload the local model to the server;

[0061] The server communicates with the client to realize the federated learning method based on knowledge distillation.

[0062] The specific federated learning method based on knowledge distillation includes:

[0063] Step 1, in the last step of the t-1th round of training, the server integrates all the local models uploaded by the active clients in the current round, that is, To obtain the ACA model representing the global model; at the same time, integrate all the local models in the cache slot, that is​ Obtain the OCA model representing global knowledge.

[0064] Step 2, in the t-th round of training, only the clients in the active state participate in the training, and the client IDs in the active state are denoted as c1,…,ct. m The server broadcasts the latest ACA model and OCA model to the clients in the active state. m Broadcast the latest ACA model and OCA model.

[0065] Step 3, after receiving the ACA model and OCA model, each client in the active state respectively takes the ACA model and OCA model as the initial local model and teacher model, and respectively uses w1,…,wtto denote the weights of the initial local model and teacher model. k and denote, wherein the initial local model acts as a student model in the knowledge distillation process.

[0066] Step 4, adopt the data-free knowledge distillation technology to transfer the knowledge of the teacher model to the local model; in the data-free knowledge distillation, a generator for generating pseudo samples to realize knowledge distillation is trained; at the same time, the local data of each client is used to train the local model, so as to fuse and transfer the global knowledge Global from the teacher model and the local knowledge Local from the local data to the local model.

[0067] wherein the loss function of the generator for generating pseudo samples to realize knowledge distillation is trained in the following manner:

[0068] wherein g(·) denotes the output of the generator, and denote the feature vector output and probability vector output of the teacher model respectively, and the input noise vector The generator outputs a pseudo sample The input pseudo sample The teacher model outputs a probability vector

[0069] Design the One-Hot loss function:

[0070] The pseudo sample is expected to be predicted by the teacher model with a high probability as a certain specific class, therefore, the pseudo label is calculated by The One-Hot loss function is defined as:

[0071]

[0072] wherein CE is the cross entropy, if L OH is minimized, the sample generated by the generator is classified into a certain specific class by the teacher model with a very high probability, and the same is true for the real sample;

[0073] Design information entropy loss function:

[0074] In order to make the data generated by the generator cover all classes, information entropy is used to measure the uniformity of the distribution of the generated data; that is, given a probability vector p = (p1, …, pK), the information entropy is expressed as d Therefore, the information entropy loss function is defined as:

[0075]

[0076] When L IE is close to the minimum value, the data generated by the generator tends to have the same number of each class; therefore, minimizing the information entropy loss function will obtain a relatively balanced data set;

[0077] Design Activation loss function:

[0078] The feature vector of the real sample tends to obtain a higher activation value, therefore, the Activation loss function is defined as:

[0079]

[0080] Where ‖·‖1 represents the l1 norm;

[0081] Integrating the above three loss functions, the total loss function of the trained generator is:

[0082] L G = L IE + λ1L OH + λ2L A

[0083] Where λ1 and λ2 are hyperparameters that balance the three loss functions.

[0084] The client uses the Global-Local knowledge fusion technology to train the local model, and trains the loss function as follows:

[0085] Define h(·) to represent the probability vector output of the local model;

[0086] Design KL loss function:

[0087] In order to let each client use the data generated by the generator to pass the global knowledge from the teacher model to the local model, and at the same time, the local model learns the local knowledge from the local data, the KL loss function is defined as follows:

[0088]

[0089] ​where KL denotes Kullback-Leibler divergence divergence; minimizing L KL So that the local model is close to the teacher model, that is, learning global knowledge;

[0090] Design the CE loss function:

[0091] Use to represent the local data of the kth client, then the CE loss function is defined as:

[0092]

[0093] where CE denotes cross-entropy, and minimizing L CE So that the local model learns the local knowledge from the local data;

[0094] The total loss function of Global-Local knowledge fusion is:

[0095] L=L CE +γL KL

[0096] where γ is a hyperparameter, used to balance the two loss functions; minimizing L makes Global-Local knowledge fusion into the local model.

[0097] Step 5, after Global-Local knowledge fusion, all clients in the active state upload the local model to the server.

[0098] Step 6, after the server receives the local model uploaded by the client, the local model in the cache slot is updated according to the ID of the client, wherein the local model in the cache slot corresponding to the client not in the active state in the current round remains unchanged;

[0099] After updating the local model in the cache slot corresponding to all clients in the active state, the server re-integrates the local model to obtain a new ACA model and a new OCA model;

[0100] Return to step 1 for the next round of training.

[0101] The present application will obtain two global models after training, which are ACA model and OCA model respectively, only the OCA model is retained as the final global model.

[0102] Comparative experiment

[0103] To verify the effect of the method of the application, the experiment is compared with five methods of FedAvg, FedProx, FedGen, FedGKD and q-FFL. The experiment is tested on three data sets of EMNIST, CIFAR-10 and CIFAR-100, for each data set, Dirichlet distribution Dir K (α) The data set is divided into 20 parts as the local data of 20 clients (where α is the concentration rate, the smaller α is, the more heterogeneous the data is), and each client divides the local data into training set and test set according to the ratio of 4:1. The client trains the local model on the training set and obtains the global model by integrating the server, and the client tests the accuracy of the global model on the test set to obtain the accuracy of the global model on the local test set of each client. The average (Mean) and variance (Var) of these accuracies are calculated, the average represents the performance of the global model, the higher the average is, the better the performance of the global model is, and the variance represents the fairness of the global model, the lower the variance is, the better the fairness of the global model is. The performance of each method is measured by using the two indicators of Mean and Var.

[0104] Table 1 Performance of different methods on MNIST

[0105]

[0106]

[0107] Table 2 Performance of different methods on CIFAR-10

[0108]

[0109] Table 3 Performance of different methods on CIFAR-100

[0110]

[0111]

[0112] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit them; although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or some technical features can be replaced by equivalent; without departing from the spirit of the technical solutions of the present application, they should be covered in the technical solution range of the present application.

Claims

1. A federated learning method based on knowledge distillation, characterized in that, Comprise: The configuration server and client, wherein the client is used to train the local model, and the server maintains K different cache slots for storing the latest local model; in each round of training, only the active client needs to upload the local model to the server; The server and the client implement federated learning by the following method: Step 1, in the last step of the t-1th training round, the server integrates all the local models uploaded by the active clients in the current round, i.e. , to obtain an ACA model representing a global model; at the same time, all the local models in the cache slot are integrated, i.e. , to obtain an OCA model representing global knowledge; Step 2, in the In the round training, only the client in the active state participates in the training, and the client in the inactive state does not participate in the training. represents the client ID in the active state in the current round, and the server broadcasts the latest ACA model and OCA model to the client in the active state. broadcasts the latest ACA model and OCA model; Step 3, after receiving the ACA model and the OCA model, each active client respectively takes the ACA model and the OCA model as the initial local model and the teacher model, respectively, uses the initial local model as the student model in the knowledge distillation process, and uses the ACA model as the teacher model in the knowledge distillation process. w k and represent, wherein the initial local model acts as a student model in the knowledge distillation process; Step 4, adopt the data-free knowledge distillation technology to transfer the knowledge of the teacher model to the local model; in the data-free knowledge distillation, a generator for generating pseudo samples to realize knowledge distillation is trained; at the same time, the local data of each client is used to train the local model, so as to fuse and transfer the global knowledge Global from the teacher model and the local knowledge Local from the local data to the local model; The loss function of the generator for generating pseudo samples to realize knowledge distillation is trained by the following method: define g(·) the output of the generator, and respectively the feature vector output and the probability vector output of the teacher model, an input noise vector z~N (0,1), the generator outputs a fake sample = g(z), the input fake sample , the teacher model outputs a probability vector ; Design One-Hot loss function: Pseudo samples are expected to be predicted as a certain class by the teacher model with a high probability, thus, pseudo labels are calculated by One-Hot loss function is defined as: where CE is the cross-entropy, if L OH is minimized, the samples generated by the generator are classified into a certain class by the teacher model with a high probability, as is also the case for the real samples; Design information entropy loss function: To enable the generator to generate data covering all classes, information entropy is employed to measure the uniformity of the distribution of the generated data; that is, given a probability vector p ( p 1,…, p d ), the information entropy is expressed as , and thus an information entropy loss function is defined as: When L IE Near the minimum, the data generated by the generator tends to have the same number of each class; therefore, minimizing the information entropy loss function results in a relatively balanced dataset; Design Activation loss function: The feature vector of the real sample tends to obtain a higher activation value, therefore, the Activation loss function is defined as: wherein denotes l 1-norm; The total loss function of the trained generator is: L G = L IE +λ1 L OH +λ2 L A Wherein, λ1 and λ2 are hyperparameters for balancing the three loss functions; The client trains the local model by using the Global-Local knowledge fusion technology, and trains the loss function by the following method: Define h(·) to represent the probability vector output of the local model; Design KL loss function: In order to let each client use the data generated by the generator to transfer the global knowledge from the teacher model to the local model, and at the same time, the local model learns the local knowledge from the local data, the KL loss function is defined as follows: where KL denotes the Kullback-Leibler divergence divergence; minimizing L KL such that the local model is pulled towards the teacher model, i.e. learns global knowledge; Design CE loss function: With Let denote the local data of the kth client, then the CE loss function is defined as: where CE denotes cross-entropy, minimized L CE such that the local model learns local knowledge from the local data; The total loss function of Global-Local knowledge fusion is: L = L CE + γL KL wherein, γ is a hyper-parameter to balance the two loss functions; minimizing L such that the Global-Local knowledge is fused into the local model; Step 5, after Global-Local knowledge fusion, all active clients upload the local model to the server; Step 6, after receiving the local model uploaded by the client, the server updates the local model in the cache slot according to the ID of the client, wherein the local model of the cache slot corresponding to the client not in the active state in the current round remains unchanged; After updating the local model of all cache slots corresponding to the active clients, the server recombines the local model to obtain a new ACA model and a new OCA model; Return to step 1 for the next round of training.

2. A federated learning system based on knowledge distillation, characterized in that, Comprise: The configuration server and client, wherein the client is used to train the local model, and the server maintains K different cache slots for storing the latest local model; in each round of training, only the active client needs to upload the local model to the server; The server and the client communicate to realize the federated learning method based on knowledge distillation of claim 1.

Images