Efficient heterogeneous federated learning method and system based on hybrid distillation, and device and medium

Abstract

The present invention belongs to the technical field of federated learning. Disclosed are an efficient heterogeneous federated learning method and system based on hybrid distillation, and a device and a medium. The method comprises: a server initializing a global model parameter, and setting a preset total number of training rounds and the number of clients participating in each round of training; in each client, loading a corresponding local dataset, executing random transformation on the local dataset, and generating client distillation data; sampling several sub-networks from an original network of each client, training each sub-network on the client distillation data to obtain an updated local model parameter, and uploading the local model parameter to the server; and the server receiving the local model parameter, and then, in combination with a preset auxiliary dataset, performing server distillation, so as to obtain an updated global model parameter, and sending same to the client. The technical solution of the present invention can overcome the heterogeneity issue in federated learning, and can also improve the communication efficiency and robustness against attacks while avoiding privacy leakage.

Description

A high-efficiency heterogeneous federated learning method based on hybrid distillation, system, device and medium TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and particularly relates to a high-efficiency heterogeneous federated learning method based on hybrid distillation, system, device and medium. BACKGROUND

[0002] With the improvement of computer computing power, machine learning as a massive data analysis and processing technology has been widely used in human society. However, there are data barriers between different industries and departments, resulting in data forming "islands" and being unable to be safely shared, and the performance of machine learning models trained by independent data of each department cannot achieve global optimization. The federated learning technology is proposed to solve this problem, which effectively guarantees the privacy and security of users by transferring the data storage and model training stage of machine learning to local users and only interacting with the center server for model update, and has been widely used in practice and achieved good results. However, it also brings new challenges, that is, the data distribution of different parties is usually non-iid. When the client holds heterogeneous data, the local models will often diverge during training, resulting in client drift. Therefore, directly aggregating model parameters and updates will lead to a significant decline in the performance of the global model.

[0003] Since the problem of data heterogeneity was proposed, many methods have been proposed to alleviate various problems caused by data heterogeneity. Most of the existing federated learning solutions to different client data heterogeneity can be divided into three categories. The first category is the data-level method, which smoothes the statistical heterogeneity of the client local data through private data processing such as data augmentation and external data. The second category is the model-level method, which operates at the model level and aims to learn a local model for each client that adapts to its private data distribution while learning global information, mainly including adding regularization, combining contrastive learning or meta-learning, improving consistency, sharing part of the structure, etc. The third category is the server-level method, which requires the participation of the server, such as participating in client selection or client clustering.

[0004] Combining federated learning with knowledge distillation, namely federated distillation, can improve the performance of federated learning in a heterogeneous environment by utilizing external data sources and knowledge transfer, which belongs to data-level and model-level methods. Currently, there are mainly two ways of combination, one is client-side distillation, and the other is server-side distillation. In client-side distillation, each client obtains the average soft prediction from all clients to constrain local updates and prevent falling into local optimum. However, it requires data exchange between clients, which may lead to privacy leakage and is vulnerable to poisoning attacks by malicious clients. In server-side distillation, the server aggregates the client models and uses the average soft prediction output by the auxiliary dataset to fine-tune the global model. However, the existing server distillation method only uses soft prediction to learn global knowledge, which relies on the set of local predictors to output for distillation, making it sensitive to misleading and ambiguous knowledge injected by poorly performing local models, and sharing soft prediction will exacerbate the problem. SUMMARY

[0005] The purpose of the present application is to provide an efficient heterogeneous federated learning method, system, device and medium based on hybrid distillation to solve the problems existing in the prior art.

[0006] To achieve the above purpose, the present application provides an efficient heterogeneous federated learning method based on hybrid distillation, comprising:

[0007] Step one: the server initializes the global model parameters, and sets the preset total training round number and the number of clients participating in training per round;

[0008] Step two: in each client, load the corresponding local dataset, perform random transformation on the local dataset to generate client distillation data; sample a plurality of sub-networks from the original network of each client, each sub-network is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server;

[0009] Step three: the server receives the local model parameters and performs server distillation combined with the preset auxiliary dataset to obtain updated global model parameters, and sends the updated global model back to the client;

[0010] Repeat steps two to three until the model converges.

[0011] Optionally, the random transformation on the local dataset specifically comprises:

[0012] Scaling and rotating the local dataset to obtain the client distillation data.

[0013] Optionally, each of the sub-networks has a different network score width.

[0014] Optionally, each sub-network is trained on client distillation data, specifically including:

[0015] The KL divergence of the softmax output of the sub-network and the original softmax output of the corresponding client local model is calculated as a distillation loss, and the weight is dynamically allocated according to the prediction confidence of the sub-network;

[0016] Based on the distillation loss and the traditional cross-entropy loss, the local model parameters are updated using an optimization algorithm.

[0017] Optionally, the step three specifically includes:

[0018] The server receives the updated local model parameters of all clients, aggregates the weights, and obtains a global model;

[0019] Based on the preset auxiliary data set, the global model is distilled based on joint soft prediction distillation and feature distillation, the updated global model parameters are obtained, and the updated global model is sent back to the client.

[0020] An efficient heterogeneous federated learning system based on hybrid distillation, comprising:

[0021] An initialization module for initializing global model parameters by a server, and setting a preset total training round number and a number of clients participating in training per round;

[0022] A client distillation module for loading a corresponding local data set in each client, performing random transformation on the local data set to generate client distillation data, sampling a plurality of sub-networks from the original network of each client, each sub-network being trained on client distillation data, and obtaining updated local model parameters after training, and uploading the local model parameters to the server;

[0023] A server distillation module for the server to receive local model parameters and combine a preset auxiliary data set for server distillation to obtain updated global model parameters, and send the updated global model back to the client; repeat the client distillation and server distillation until the model converges.

[0024] An electronic device comprising a memory for storing a computer program and a processor for running the computer program to make the electronic device execute the method of the efficient heterogeneous federated learning based on hybrid distillation.

[0025] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the efficient heterogeneous federated learning based on hybrid distillation.

[0026] The technical effects of the present application are:

[0027] The present application reduces local overfitting and improves model aggregation and generalization performance of federated learning on heterogeneous and long-tail client data by designing a two-stage learning scheme consisting of client self-distillation and server integrated distillation. The scheme does not require any information exchange between clients or joint optimization with global data distribution, thus preventing privacy leakage.

[0028] The server distillation method proposed by the present application can achieve better representation learning and a flatter loss landscape to fine-tune the aggregated model, both of which contribute to improving the accuracy of the global model under different degrees of data heterogeneity and different numbers of clients.

[0029] The method designed by the present application not only overcomes the heterogeneity problem of federated learning, but also improves communication efficiency and attack robustness while avoiding privacy leakage. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0031] The drawings forming part of the present application are used to provide further understanding of the present application, the illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitation on the present application. In the drawings:

[0032] Figure 1 is an implementation flowchart in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Now a variety of exemplary embodiments of the present application will be described in detail, which should not be considered as limiting the present application, but should be understood as a more detailed description of certain aspects, characteristics and embodiments of the present application.

[0034] It should be understood that the terms described in the present application are only for describing the specific embodiments, and are not used to limit the present application. In addition, for the numerical range in the present application, it should be understood that each intermediate value between the upper limit and the lower limit of the range is also specifically disclosed. Each smaller range between any stated value or intermediate value within the range and any other stated value or intermediate value within the range is also included in the present application. The upper limit and the lower limit of these smaller ranges can be independently included or excluded from the range.

[0035] All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains unless otherwise specifically defined herein. Although preferred methods are described herein, any method similar or equivalent to those described herein can be used in the practice or testing of the present application. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods associated therewith. In case of conflict between the content of the specification and that of any document incorporated herein by reference, the content of the specification prevails.

[0036] Many modifications and variations of the present application described in the detailed description of the application can be made by those skilled in the art without departing from the scope or spirit of the application. Other implementations of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application. The specification and examples given are exemplary only.

[0037] As used herein, the terms "comprises", "comprising", "includes", "including", "has", "having", and the like are open-ended terms that are intended to mean including, but not limited to.

[0038] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0039] Embodiment one

[0040] As shown in FIG. 1, the present embodiment provides a high-efficiency heterogeneous federated learning method based on hybrid distillation, which includes:

[0041] Step one: the server initializes the global model parameters, and sets the preset total training rounds and the number of clients participating in training per round;

[0042] Step two: in each client, load the corresponding local data set, perform random transformation on the local data set to generate client distillation data; sample a number of sub-networks from the original network of each client, each sub-network is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server;

[0043] Step three: the server receives the local model parameters, combines the preset auxiliary data set for server distillation, obtains the updated global model parameters, and sends the updated global model back to the client;

[0044] Repeat steps two to three until the model converges.

[0045] The embodiment is designed by designing a two-stage learning scheme composed of client self-distillation and server integrated distillation to reduce local overfitting and improve model aggregation and generalization performance of federated learning on heterogeneous and long-tail client data. The scheme does not require any information exchange between clients or joint optimization of global data distribution, thereby preventing privacy leakage.

[0046] The server distillation method proposed in the embodiment can achieve better representation learning and a more flat loss landscape to fine-tune the aggregated model, both of which help to improve the accuracy of the global model under different degrees of data heterogeneity and different numbers of clients.

[0047] The method designed in the embodiment not only overcomes the heterogeneity problem of federated learning, but also improves communication efficiency and attack robustness while avoiding privacy leakage.

[0048] The embodiment proposes a two-stage learning paradigm for heterogeneous federated learning, namely federated hybrid knowledge distillation (FedHyb). The two stages are mainly knowledge distillation on the client side and the server side. Among them, the client distillation limits local updates and reduces local overfitting by dynamically learning subnetworks, while avoiding information exchange between clients. A server integrated distillation scheme is proposed on the server side, which more comprehensively transmits aggregated client information to the global model at three different levels. The server distillation process is guided by a class-balanced unlabeled dataset, which can come from a third party or be generated by a generator independent of the client data distribution to supervise integrated knowledge transfer. In this way, a more general feature representation can be learned from model aggregation, and a classifier can be retrained using a balanced data set to achieve better final test accuracy.

[0049] The specific scheme is as follows:

[0050] 1. Traditional federated learning (FedAvg): Consider a federated learning system with K clients. For client k = 1, 2,..., K, all clients can only access their own local private data D k : = (x, y), x and y are the local private data and labels of the client, respectively. N k : = |D k | represents the amount of client data. Participating clients use to perform local training, and the cross-entropy (CE) loss is:

[0051] After training a specified number of rounds, the local client model parameters ω k = ω1, ω2,..., ω K are sent to the server, which uses to perform simple model aggregation to obtain global model parameters:

[0052] Where pk = |D k | / |D| represents the amount of data from client k relative to the set of data from all clients. The weight ratios are determined. The server sends the global model back to the client. This process is repeated T times until convergence.

[0053] 2. Client-side Distillation Process: Most existing methods involve local logarithmic distillation, which requires information exchange between clients. If a malicious client participates, the privacy of other clients may be leaked during the collaborative learning process. To address this issue, this embodiment designs a client-side self-distillation scheme that protects privacy, requiring neither information exchange nor a shared public dataset between clients. In the self-distillation process, firstly, the local data sample D... k Perform random transformations, including scaling and rotation, on the data to obtain the client distillation data R. m (x). Secondly, this embodiment also samples M subnetworks with different network score widths from the original network, denoted as S. m (ω), where ω is the local client model parameter, m = 1, 2, ..., M. This embodiment then uses the cumulative distillation loss of the M sub-networks to constrain local updates and mitigate local overfitting. Specifically, this embodiment introduces the following regularization term to the traditional cross-entropy (CE) loss function to regularize the local model update at client k:

[0054] Here, the softmax output of the m-th sub-network trained on the corresponding client distillation dataset is denoted as Q. k (S m (ω);R m (ω)), the original softmax output of the k-th client local model is Q. k (ω; x) are aligned by calculating the KL divergence between the two. The proposed client distillation involves sampling multiple subnetworks with different network score widths, which are fine-tuned using client distillation data generated by random transformations of local datasets. This enriches the learned local semantic information without leaking it to other clients.

[0055] 3、Dynamic subnetwork weights: Due to the different score widths of the subnetworks, the simple transformation of the original image is different, and the performance of each subnetwork may be biased. The best case is that the subnetwork with better performance can contribute more to the original network. On this basis, the embodiment designs a mechanism of dynamically weighting the loss of the subnetwork, which dynamically allocates the loss weight to the subnetwork according to the performance of the different subnetworks. Specifically, as the training proceeds, each subnetwork has different prediction confidence for the sample, so that the subnetwork dynamically contributes to the training of the original network, and using an adaptive way to allocate these subnetwork weights is beneficial to the original network, that is, based on the prediction output of different subnetworks, the contribution of the subnetworks with different confidence levels to the original network is different in the current training. The definition of the formula is:

[0056] wherein is the cross-entropy (CE) loss calculated on the original label of the mth subnetwork of the client k. In this way, the subnetwork with higher prediction confidence will contribute more in the client self-distillation process. In combination with the above design, the objective function of the local model at the client k is:

[0057] wherein the first part is the traditional cross-entropy (CE) loss as shown in formula (1), and the second part is the cumulative knowledge distillation (KD) loss based on soft label prediction, and the process is dynamic weighting with adaptive model regularization. The contributions of CE and KD loss are balanced by using a hyperparameter σ.

[0058] 4、Server distillation: Due to the existence of data heterogeneity, there may be great differences between the client models, and the performance of the server model obtained by directly aggregating the client models is not good, therefore, the aggregated server model is further fine-tuned at the server. Specifically, further distillation is performed at the server, and an unlabeled class-balanced proxy dataset D S :=x s is used, which can come from a third party or be generated by a generator irrelevant to the data distribution of the client, and based on the global proxy dataset, the integrated knowledge of the client model is transferred to the server model, and the drift of the server model caused by the aggregation of the client models due to data heterogeneity is alleviated. At the same time, in order to better transfer the knowledge, three levels of aggregation are performed, including weight aggregation, soft prediction aggregation and feature aggregation. First, weight aggregation is performed, that is, the traditional federated learning aggregation method is used to obtain a global model Ω, and then the proxy dataset D S is used to fine-tune Ω by jointly soft prediction distillation and feature distillation. The formula of soft prediction knowledge transfer is as follows:

[0059] where the first term in KL is the aggregated soft prediction output of the uploaded client models, i.e., soft prediction aggregation, and the second is the global model output with current network parameters Ω before fine-tuning. The knowledge of feature extraction of K clients is then transferred to the server, i.e., representation knowledge transfer, which is done by minimizing the mean square error (MSE) distance between the server model data representation output and the data representation output of the client models. The formula of representation knowledge transfer is as follows:

[0060] where the first term in MSE represents the second-to-last feature extracted by the global model Ω, and the second term is the aggregation of the second-to-last feature obtained from the uploaded client models. By combining these two types of knowledge transfer, the server ensemble distillation loss is thus:

[0061] where η and ν are used to adjust the proportion of soft prediction knowledge and representation knowledge.

[0062] Experiments show that the model trained using the embodiment scheme has higher accuracy than traditional schemes and the latest scheme in the federated learning heterogeneous and long-tail environment, and has a leading advantage in communication efficiency and attack robustness. The evaluation dataset used in this embodiment is: 10-class SVHN and CIFAR-10, 100-class CIFAR-100.

[0063] This embodiment tests the accuracy of the server global model and the accuracy of the client model when the number of clients is 10, 20, and 50. The method proposed in this embodiment can achieve the highest accuracy in most cases, especially for the global model on the server side, and is not affected by the number of clients. In the case of more clients close to reality, the best performance can also be obtained. Compared with the traditional FedAvg, this embodiment obtains an 11-24% gain in server accuracy. Compared with the latest proposed method, this embodiment obtains a 2-19% gain in server accuracy.

[0064] In terms of heterogeneity, this embodiment uses Dirichlet distribution to change the heterogeneity of client data. By setting the heterogeneity factor, different heterogeneities of clients are achieved. The smaller the heterogeneity factor, the more heterogeneous the data. Through experiments, the method of this embodiment is optimal compared with traditional schemes and the latest scheme. Under the condition of setting the same heterogeneity, a 3-20% gain is obtained.

[0065] In terms of communication efficiency, by comparing the number of communication rounds required by the embodiment and other methods to reach the specified test accuracy, the fewer the communication rounds, the higher the communication efficiency. Through experiments, the embodiment method requires the least number of communication rounds at all specified test accuracies, indicating that the embodiment has a leading advantage in communication efficiency.

[0066] For attack robustness, the embodiment uses two attack methods to evaluate the robustness of malicious clients trying to poison federated learning. One is random noise (RN) attack, which generates perturbations based on Gaussian distribution and introduces random noise during training to mislead the training process and reduce model performance. The other is label flipping (LF) attack, which attacks the global model by modifying the client dataset, and the attack involves changing the class of each instance in the dataset to the wrong classification. By comparing the embodiment method with traditional federated learning methods and the latest methods, as the number of malicious clients (attackers) increases, the global accuracy of all methods decreases to varying degrees, but the embodiment method has the smallest reduction, indicating that the method significantly improves the robustness of the model under the two types of data poisoning attacks.

[0067] A high-efficiency heterogeneous federated learning system based on hybrid distillation, comprising:

[0068] An initialization module for initializing the global model parameters by the server and setting the preset total training rounds and the number of clients participating in training per round;

[0069] A client distillation module for loading the corresponding local dataset in each client, performing random transformation on the local dataset to generate client distillation data, sampling a plurality of sub-networks from the original network of each client, training each sub-network on the client distillation data, obtaining updated local model parameters after training, and uploading the local model parameters to the server;

[0070] A server distillation module for the server to perform server distillation on the updated local model parameters combined with the preset auxiliary dataset to obtain updated global model parameters, and send the updated global model back to the client; repeat the client distillation and server distillation until the model converges.

[0071] An electronic device comprising a memory for storing a computer program and a processor for running the computer program to make the electronic device execute the high-efficiency heterogeneous federated learning method based on hybrid distillation.

[0072] A computer-readable storage medium storing a computer program, which is executed by a processor to implement the high-efficiency heterogeneous federated learning method based on hybrid distillation.

[0073] The above description is only preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A high-efficiency heterogeneous federated learning method based on hybrid distillation, characterized in that, Comprising: Step one: make the server initialize the global model parameters, and set the preset total training rounds and the number of clients participating in training per round; Step two: in each client, load the corresponding local data set, perform random transformation on the local data set to generate client distillation data; sample several subnetworks from the original network of each client, each subnetwork is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server; Step three: the server receives the local model parameters and combines the preset auxiliary data set to perform server distillation to obtain updated global model parameters, and sends the updated global model back to the client; Repeat steps two to three until the model converges.

2. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. Performing random transformation on the local data set, specifically including: Perform scaling and rotation on the local data set to obtain the client distillation data.

3. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. Each of the subnetworks has a different network score width.

4. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. Each subnetwork is trained on the client distillation data, specifically including: Calculate the KL divergence of the softmax output of the subnetwork and the original softmax output of the corresponding client local model as the distillation loss, and dynamically allocate weights according to the prediction confidence of the subnetwork; Based on the distillation loss and the traditional cross-entropy loss, update the local model parameters using an optimization algorithm.

5. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. The step three, specifically includes: The server receives all the updated local model parameters of the clients, aggregates the weights to obtain the global model; Based on the preset auxiliary data set, combine joint soft prediction distillation and feature distillation to distill the global model to obtain updated global model parameters, and send the updated global model back to the client.

6. A high efficient heterogeneous federated learning system based on hybrid distillation, characterized in that, Comprising: An initialization module for making the server initialize the global model parameters, and setting the preset total training rounds and the number of clients participating in training per round; A client distillation module for loading the corresponding local data set in each client, performing random transformation on the local data set to generate client distillation data; sampling several subnetworks from the original network of each client, each subnetwork is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server; A server distillation module for the server to receive the local model parameters and combine the preset auxiliary data set to perform server distillation to obtain updated global model parameters, and send the updated global model back to the client; repeat the client distillation and server distillation until the model converges.

7. An electronic device, comprising: Comprising a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to make the electronic device execute the efficient heterogeneous federated learning method based on mixed distillation according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which is executed by a processor to realize the efficient heterogeneous federated learning method based on mixed distillation according to any one of claims 1-5.

Images