A federated learning system based on data-free knowledge distillation performing model aggregation

Abstract

The application discloses a kind of federated learning systems based on model aggregation of execution without data knowledge distillation, which does not increase other operations except local model update to client, makes full use of the rich computing resources of server, and carries out the aggregation of model through the technology of data-free knowledge distillation on server.Data-free knowledge distillation model aggregation method includes generator training and model distillation two processes, generator training process fixes global model and local model parameters, and updates generator;Model distillation process fixes the parameters of generator and local model, and carries out knowledge distillation from local model to global model based on the current synthetic data generated by generator and the historical synthetic data in data buffer zone.The aggregation from local model to global model is realized by alternately executing generator training and model distillation two processes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data privacy, knowledge distillation and federated learning, and particularly relates to a federated learning system based on data-free knowledge distillation for model aggregation. BACKGROUND

[0002] With the rapid growth of data, deep learning algorithms have made significant progress in both mature and emerging fields, and data has become the core driving force in the field of artificial intelligence. However, in many real-world application scenarios (such as mobile device scenarios, Internet of Things scenarios, and autonomous driving scenarios), data is often scattered across different clients, forming so-called data island phenomenon. With the growing emphasis on data privacy protection, privacy protection-related regulations have also been introduced, marking the protection of personal data has risen to the legal level. Therefore, how to efficiently utilize the data scattered across various clients to create value while ensuring data privacy has become a pressing problem to be solved.

[0003] The emergence of federated learning is to solve this serious privacy protection problem. Unlike traditional data centralized training, federated learning enables various clients to participate in model training without revealing data privacy, which provides an effective solution for privacy-sensitive application scenarios. The core idea of federated learning is that data is saved locally on each client, and only model parameters are transmitted between the server and the client, without the need to store data centrally on a central server. The server aggregates the parameters uploaded by each client to obtain a global model, and distributes the global model to each client, thereby replacing the local model on the client. In actual training, the above process is repeated until the final global model is obtained. This decentralized data processing approach greatly reduces the risk of data leakage and also avoids the risk of privacy invasion caused by centralized data storage.

[0004] Although federated learning has made significant progress in research and practical application, it still faces the problem of data heterogeneity. In real-world scenarios, the data distribution between different clients is often different, which means that their data features and data distribution may differ. Data heterogeneity can lead to the so-called "client drift" phenomenon, that is, the difference in model performance between different clients increases, thereby reducing the performance and accuracy of the global model. The classic federated averaging algorithm simply performs weighted averaging of the model parameters uploaded by each client to obtain a global model. The weight of the model uploaded by each client in the weighted averaging is determined by the size of the local data on the client. This approach may result in a small contribution of the client with small data size to the global model, thereby affecting the generalization performance of the model and failing to solve the problem caused by data heterogeneity.

[0005] To solve the problem of data heterogeneity, researchers have proposed various improved algorithms based on federated averaging algorithm. These algorithms try to reduce the impact of data heterogeneity by adjusting the aggregation method of the model, weighting the contributions of different clients or introducing specific data preprocessing techniques. These algorithms more or less have some problems, some algorithms have to sacrifice some privacy protection ability to ensure the performance of the final model, and some need to perform complex operations on the client side, which cannot adapt to the reality of limited client computing power. SUMMARY

[0006] In order to solve the problem of poor model generalization ability caused by the weighted average aggregation algorithm mentioned above, the problem of data heterogeneity in the case of limited client computing power and the problem of privacy protection, the purpose of the present application is to provide a federated learning system based on data-free knowledge distillation for model aggregation, aiming to improve the model aggregation method of the server, without additional client computing operations, so that the present application can solve the problem of data heterogeneity in the case of limited client computing power while ensuring the privacy protection ability.

[0007] The specific technical solutions to achieve the purpose of the present application are:

[0008] A federated learning system based on data-free knowledge distillation for model aggregation, comprising:

[0009] At least two clients participate in federated training as participants, communicate with the server in each communication round, including uploading local model parameters and downloading global model parameters; the local model on each client and the global model on the server have the same neural network model architecture, the client first replaces the local model parameters with the global model parameters obtained in the current communication round, then updates the local model based on the local private data by random gradient descent algorithm, and finally uploads the updated local model parameters and label statistical information of the local training data set to the server to participate in the model aggregation of the current communication round;

[0010] A server, for the central server and each client to communicate, in each round of communication, first performs the distribution of global model parameters, according to the proportion of the communication round under the participation of the selected client in the federal training, the global model parameters are distributed to the selected client, and the local model parameters uploaded by the client and the label statistical information of the local training data set are received, the server first performs weighted average on the received local model parameters to obtain an initial global model, and then aggregates the label statistical information of the local training data set uploaded by the client to obtain a label statistical distribution, and then iteratively updates the global model in a data-free knowledge distillation manner; the server configures a label embedding network, a generator model and a data buffer area for data-free knowledge distillation of the local model to the global model, the data-free knowledge distillation manner involves two processes of generator training and model distillation, in each process, Gaussian noise z and label y are first sampled from Gaussian distribution and label statistical distribution respectively, then the label y is input into the label embedding network to obtain a label embedding vector ey, the embedding vector is spliced with the Gaussian noise z to obtain a vector h, and the vector h is input into the generator to generate synthetic data with the label y; in the generator training process, the synthetic data generated by the generator is directly input into the local model and the global model to obtain the outputs of the local model and the global model, and the outputs are used together with the label y of the synthetic data to construct a fidelity loss function Migration loss function And a diversity loss function to update the parameters of the generator to ensure that the synthetic data generated by the generator has fidelity, migratability and diversity; the data buffer area is used to store the synthetic data generated by the generator in the model distillation process, and the synthetic data generated by the generator and the synthetic data randomly sampled from the data buffer area are input into the local model and the global model in the model distillation process to obtain the outputs of the local model and the global model, thereby constructing a KL divergence loss function for updating the parameters of the global model to ensure that the knowledge of the local model can be migrated to the global model;

[0011] The client and the server continuously communicate until all clients are selected at least once.

[0012] The label statistical information uploaded by the client to the server only includes the number of data owned by each label in the local data set, and does not include specific data.

[0013] The fidelity loss function is in the form of:

[0014]

[0015] Wherein, CE is a cross-entropy function, S t is the client selected by the server at the communication round t, fi (s, θ i ) is the output result of the local model θ i uploaded by the client i based on the synthetic data s generated by the generator, y is the label of the synthetic data s;

[0016] The migration loss function is in the form of:

[0017]

[0018] Wherein, KL is the KL divergence function, S t is the selected client at the communication round t, f i (s, θ u ) is the output result of the local model θ i uploaded by the client i based on the synthetic data s generated by the generator, f(s, θ) is the output result of the global model θ based on the synthetic data s generated by the generator, and ε is a control coefficient, if argmaxf(s, θ)≠y and ε=1, otherwise ε=0;

[0019] The diversity loss function is in the form of:

[0020]

[0021] Wherein, B is the batch size of the data input to the generator, h i and h j are the vectors obtained by splicing the Gaussian noise z and the label embedding vector e y , s i and s j are the synthetic data generated by the generator according to h i and h j respectively.

[0022] The KL divergence loss function is in the form of:

[0023]

[0024] Wherein, KL is the KL divergence function, f(s, θ) is the output result of the global model θ based on the synthetic data s generated by the generator, S t is the selected client at the communication round t, f i (s, θ i ) is the output result of the local model θ i uploaded by the client i based on the synthetic data s generated by the generator.

[0025] The two processes of generator training and model distillation are alternately performed, that is, the generator training is performed for I g After the second iteration update, the model distillation process is performed for I d The iteration update is continued for I g The second generator training is performed, and the alternately training is continued back and forth until a preset iteration number is reached.

[0026] Advantages of the present application

[0027] 1) The federated learning system for model aggregation based on data-free knowledge distillation proposed in the present application adopts a data-free knowledge distillation manner to perform model aggregation on the server side, without contacting the original training data of the client, and only needs to obtain the label statistical data of the client. However, the disclosure of the label statistical information of the client is acceptable in most federated learning research work, so the present application ensures the privacy of the original data.

[0028] 2) The federated learning system for model aggregation based on data-free knowledge distillation proposed in the present application only performs necessary model update algorithms on each client, without performing additional operations, so it does not add computational burden to the client, and to some extent can be applied to real scenarios where the client's resources are limited.

[0029] 3) The federated learning system for model aggregation based on data-free knowledge distillation proposed in the present application ingeniously designs the fidelity loss function, transfer loss function and diversity loss function required in the generator training stage, so that the synthetic data generated by the generator has fidelity, transferability and diversity, and the knowledge of the local model can be better transferred to the global model in the model distillation stage.

[0030] 4) The federated learning system for model aggregation based on data-free knowledge distillation proposed in the present application introduces a data buffer area to save the synthetic data generated by the generator in the model distillation stage, so that the model can not only be distilled based on the synthetic data generated in the current iteration, but also based on the synthetic data in the data buffer area, to some extent, the catastrophic forgetting problem in the data-free knowledge distillation manner is alleviated. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the architecture diagram of the present application. DETAILED DESCRIPTION

[0032] The present application will be further described in detail in conjunction with the following specific embodiments and drawings, including the conditions, processes, training methods, etc. for implementing the present application. Except for the following specifically mentioned contents, the remaining contents are the general knowledge in the art, and the present application does not particularly limit the contents.

[0033] The specific embodiments of the present application are expected to make full use of the computing resources of the server, and apply the data-free knowledge distillation technology to the model aggregation process on the server. The present application focuses on the knowledge distillation technology, and hopes to effectively migrate the local model knowledge of each client to the global model through the knowledge distillation technology. However, the process of knowledge distillation requires the support of the training data of the client, and the server cannot access the training data of the client, so the present application adopts a data-free knowledge distillation method, first uses a generator to generate synthetic data, and then distills the local model knowledge of each client to the global model based on the synthetic data. In addition to the simple stochastic gradient descent process for local model updating on the client, no additional operations are performed, ensuring the feasibility of the federated learning algorithm under the condition of limited computing resources on the client.

[0034] The data-free knowledge distillation method adopted by the present application is divided into two processes: generator training and model distillation, and the two processes are alternately trained on the server. It is worth mentioning that the general data-free knowledge distillation method will only update the global model based on the synthetic data generated in the current iteration in the model distillation stage, and the synthetic data generated in the current iteration will be discarded in the next iteration, and the global model will be updated based on the newly generated synthetic data in the next iteration, which will cause the catastrophic forgetting problem of the model distillation process, that is, the global model will forget the knowledge learned in the previous iteration. In order to solve this problem, the present application additionally uses a fixed-size data buffer to store the historical synthetic data generated in the previous iteration in the model distillation process, and in addition to using newly generated synthetic data, a part of data randomly selected from the data buffer will also be used.

[0035] Referring to Figure 1 The federated learning system of the present application is composed of a central server, i.e. a server, and N clients. In addition to the local model and the global model on the central server, a generator model is configured for generating synthetic data required for the model distillation process, a label embedding layer is configured for converting label data y into label vector e y , and a data buffer is configured for storing historical synthetic data generated by the generator in the model distillation process. Each client has local private labeled data where obeys the data distribution in the feature space , i.e. , i.e. , and represents the label of X i , and C is the number of classes of the label. In addition, only the horizontal federated scenario is considered in the present application , i.e. each client has the same feature space, different data space and label space. The specific operation process is as follows:

[0036] • Step S1: Initialize the global model θ on the server side 0 ;

[0037] • Step S2: In each communication round t, the server randomly selects S clients from all clients in proportion, forming a client set S t , and distributes the global model θ t to each client S t in S i ;

[0038] • Step S3: Each client S i participating in this communication round t updates the local model based on the local data, and uploads the updated local model and local label statistics to the server side;

[0039] In each iterative update of the local model, the client S i first samples a batch size B of data from the local data set inputs it into the local model, calculates the gradient using the cross-entropy loss function after obtaining the output of the local model, and then updates the local model parameters using the stochastic gradient descent algorithm.

[0040] After the client S i performs I iterative updates, the updated local model and local label statistics n i are uploaded to the server side.

[0041] • Step S4: The server aggregates the local label statistics uploaded by the clients to form the label distribution y ~ p(y) := P[y];

[0042] • Step S5: The server performs a weighted average of the local models uploaded by the clients to obtain an initial global model θ t+1 ;

[0043]

[0044] where S t is the client subset randomly sampled from all clients at communication round t, p i is the weight of the client S i , determined by the size of the data set owned by the client S i , is the local model parameter of the client S i at communication round t.

[0045] Step S6: Fix the global model θ t+1 and the local model uploaded to the server Parameters, update generator G(·).

[0046] This step is the generator training process, which uses a fidelity loss function. migration loss function and diverse loss functions Ensure that the generated synthetic data is faithful, portable, and diverse.

[0047] The generator G(·) receives two inputs: noise data z randomly sampled from a Gaussian distribution and label data y. The label data y is sampled from the client's label distribution y ~ p(y): = P[y], and its purpose is to enable the generator to produce synthetic data with label information. To better fuse the label data y with the noise data z, the label data y is first input into a label embedding network to obtain the embedding vector e. y The embedding vector e y The data is concatenated with the noisy data z and then input into the generator G(·). The generator G(·) generates synthetic data based on the input. The local model and the global model obtain the model results based on the synthetic data. The model output is input into the loss function to calculate the gradient, thereby updating the generator parameters.

[0048] Fidelity loss function To ensure that the distribution of synthetic data generated by the generator approximates the distribution of training data from the local model, i.e., to maintain fidelity:

[0049]

[0050] Essentially, it is a cross-entropy loss function, where f i (s, θ) i ) is the client S i Local model θ i The output s is the synthesized data generated by the current generator G(·).

[0051] migration loss function The goal is to ensure the transferability of the generated synthetic data. For the same synthetic data s with label y, transferability ensures that the local model will classify the synthetic data s into the correct category y, while the global model will classify it into the wrong category. This allows the global model to better learn from the local model's knowledge during model distillation, meaning the generated synthetic data can maximize the difference between the global and local models. Transfer loss function. The formula is:

[0052]

[0053] where f(s, θ) is the output of the global model θ on synthetic data s, where s is generated based on label y. In addition, if argmaxf(s, θ)≠y and then ε = 1, otherwise ε = 0.

[0054] Diverse loss function The purpose is to ensure that the generated synthetic data has diversity, that is, to ensure that synthetic data of different categories has distinguishability, and synthetic data of the same category has diversity:

[0055]

[0056] where B is the size of the current batch, s i and s j are generated by h i and h j , h i and h j are obtained by splicing Gaussian noise z and label embedding vector e y .

[0057] Combining the above three loss functions, the final loss function is as follows:

[0058]

[0059] where α trans and α div are adjustable hyperparameters.

[0060] According to the above process, update I g times of iterations, and get the updated generator, which is used in the model distillation process of step S7.

[0061] • Step S7: Fix the conditional generator G(·) and the local model parameters uploaded to the server, generate synthetic data s according to the conditional generator G(·), and data s in the data cache area c Perform the distillation process of the local model to the global model, and update the data cache area with the newly generated synthetic data.

[0062] This step is the model distillation process, mainly minimizing the difference between the global model and the local model. Similar to step S6, the generator G(·) receives two inputs: noise data z randomly sampled from a Gaussian distribution and label data y. The label data y is sampled from the client's label distribution y ~ p(y): = P[y], and its purpose is to enable the generator to produce synthetic data with label information. To better fuse the label data y with the noise data z, the label data y is first input into a label embedding network to obtain the embedding vector e. y The embedding vector e y The data s is concatenated with the noisy data z and then input into the generator G(·). The generator G(.) generates synthetic data with high fidelity, transferability, and diversity. This synthetic data s is then input into the global model and the local model, and the model's output is fed into the KL loss function. middle:

[0063]

[0064] The gradient is calculated based on the loss function, and the parameters of the global model are updated, enabling the global model (student model) to learn better from the local model (teacher model).

[0065] It is important to note that the model distillation process introduces an additional data cache to store historical synthesis data generated by the generator in previous communication rounds. The model distillation process simultaneously uses both the newly generated synthesis data *s* and the historical synthesis data *s*. c Distillation, to some extent, alleviates the catastrophic forgetting problem in knowledge-free distillation, enabling knowledge from the local model to be better transferred to the global model.

[0066] Update I according to the above process d After the iteration, the updated global model is obtained, which is then used in the generator training process in step S6 to continue updating the generator.

[0067] • Step S8: Repeat steps S5-S7 until the preset number of iterations is reached.

[0068] By repeatedly iterating through steps S5-S7, the knowledge from each local model uploaded to the server is migrated to the global model, achieving the effect of model aggregation.

[0069] • Step S9: Repeat steps S2-S8 until all local clients are selected by the server.

[0070] Since each communication round randomly selects clients according to a certain proportion, in order for the global model to obtain knowledge from all clients, the termination condition should not be reached until all local clients have been selected by the server.

Claims

1. A federated learning system based on a data-free knowledge distillation execution model aggregation, characterized in that, include: At least two clients participate in federated training as participants, communicating with the server in each round of communication, including uploading local model parameters and downloading global model parameters. The local model on each client and the global model on the server have the same neural network model architecture. The client first replaces the local model parameters with the global model parameters obtained in the current communication round, then updates the local model by performing a stochastic gradient descent algorithm based on the local private data, and finally uploads the updated local model parameters and the label statistics of the local training dataset to the server to participate in the model aggregation of the current communication round. A server facilitates communication between a central server and various clients. In each round of communication, it first distributes global model parameters, selecting clients to participate in federated training according to a set ratio. The server then receives local model parameters and label statistics from the local training dataset uploaded by the clients. The server first performs a weighted average of the received local model parameters to obtain an initial global model. Next, it aggregates the label statistics from the client-uploaded local training dataset to obtain a label statistical distribution. Finally, it iteratively updates the global model using a knowledge-free distillation method. The server is configured with a label embedding network, a generator model, and a data cache for knowledge-free distillation from the local model to the global model. This knowledge-free distillation involves two processes: generator training and model distillation. Each process first samples Gaussian noise z and labels y from the Gaussian distribution and label statistical distribution, respectively. Then, the labels y are input into the label embedding network to obtain the label embedding vector e. y The embedding vector is concatenated with the Gaussian noise z to obtain the vector h, which is then input into the generator to generate synthetic data labeled y. The generator training process directly inputs the synthetic data generated by the generator into the local model and the global model to obtain the outputs of the local model and the global model, and constructs a fidelity loss function together with the label y of the synthetic data. migration loss function and diverse loss functions The generator parameters are updated to ensure the fidelity, transferability, and diversity of the synthesized data generated by the generator. The data buffer is used to store the synthesized data generated by the generator during the model distillation process. The model distillation process inputs the synthesized data generated by the generator and the synthesized data randomly sampled from the data buffer into the local model and the global model to obtain the outputs of the local model and the global model, thereby constructing the KL divergence loss function. Used to update the parameters of the global model to ensure that knowledge from the local model can be transferred to the global model; The client and server communicate continuously until all clients have been selected at least once.

2. The system according to claim 1, characterized in that, The tag statistics uploaded by the client to the server only include the amount of data for each tag in the local dataset, and do not include the actual data.

3. The system according to claim 1, characterized in that, The fidelity loss function The form is: Where CE is the cross-entropy function, S t f is the client selected by the server in communication round t. i (s,θ i The local model θ uploaded by client i i The output is obtained based on the synthesized data s generated by the generator, where y is the label of the synthesized data s; The migration loss function The form is: Where KL is the KL divergence function, S t For the client selected by the server in communication round t, f(s,θ) i The local model θ uploaded by client i i The output obtained from the synthesized data s generated by the generator, f(s,θ) is the global model θ, and ε is the control coefficient. If argmaxf(s,θ)≠y and Then ε = 1, otherwise ε = 0; The various loss functions The form is: Where B is the batch size of the data input to the generator, h i and h j It is Gaussian noise z and label embedding vector e y The concatenated vector, s i and s j The generator is based on h respectively i and h j The generated synthetic data.

4. The system according to claim 1, characterized in that, The KL divergence loss function The form is: Where KL is the KL divergence function, f(s,θ) is the output of the global model θ based on the synthetic data s generated by the generator, and S t f is the client selected by the server in communication round t. i (s,θ i The local model θ uploaded by client i i The output is obtained based on the synthetic data s generated by the generator.

5. The system according to claim 1, characterized in that, The generator training and model distillation processes are performed alternately, i.e., generator training is performed in I... g After the next iteration update, I is then applied to the model distillation process. d Iterative updates, then continue with I g The generator is trained in a round-trip manner until the preset number of iterations is reached.

Images