A high-efficiency heterogeneous federated learning method, system, device and medium based on hybrid distillation

Abstract

The application belongs to the technical field of federated learning, and discloses an efficient heterogeneous federated learning method, system, device and medium based on mixed distillation, comprising: a server initializes global model parameters, sets a preset total training round number and a number of clients participating in training per round; in each client, a corresponding local data set is loaded, random transformation is performed on the local data set, and client distillation data is generated; a plurality of sub-networks are sampled from the original network of each client, each sub-network is trained on the client distillation data, updated local model parameters are obtained, and the local model parameters are uploaded to the server; after receiving the local model parameters, the server performs server distillation in combination with a preset auxiliary data set, obtains updated global model parameters, and sends them back to the client. The technical scheme of the application can overcome the heterogeneity problem of federated learning, and improve communication efficiency and attack robustness while avoiding privacy leakage.

Description

Technical Field

[0001] This invention belongs to the field of federated learning technology, and in particular relates to an efficient heterogeneous federated learning method, system, device and medium based on hybrid distillation. Background Technology

[0002] With the advancement of computing power, machine learning, as a technology for analyzing and processing massive amounts of data, has been widely applied to human society. However, data silos exist between different industries and departments, resulting in data "islands" that cannot be securely shared. Furthermore, machine learning models trained solely on data from each department cannot achieve global optimization. Federated learning technology was proposed to address this problem. By transferring the data storage and model training phases of machine learning to local users, and only interacting with a central server for model updates, it effectively protects user privacy and security. It has been widely used in practice with good results. However, it also brings new challenges: the data distribution of different parties is often non-independent and identically distributed (non-iID). When clients hold heterogeneous data, local models often diverge during training, causing client drift. Therefore, directly aggregating model parameters and updating them leads to a significant decrease in global model performance.

[0003] Since the problem of data heterogeneity was first proposed, many methods have been devised to mitigate the various issues it brings. Existing federated learning approaches to address data heterogeneity across different clients can largely be categorized into three types. The first type is data-level methods, which smooth out statistical heterogeneity in client-local data through private data processing such as data augmentation and external data. The second type is model-level methods, which operate at the model level to learn a local model for each client that adapts to its private data distribution while simultaneously learning global information. These methods primarily include adding regularization, combining contrastive or meta-learning to improve consistency, and sharing partial structures. The third type is server-level methods, which require server involvement, such as participating in client selection or client clustering.

[0004] Federated learning, combining it with knowledge distillation, is known as federated distillation. This method leverages external data sources and knowledge transfer to improve the performance of federated learning in heterogeneous environments, belonging to both data-level and model-level approaches. Currently, there are two main approaches: client-side distillation and server-side distillation. In client-side distillation, each client obtains the average soft prediction from all clients to constrain local updates and prevent getting trapped in local optima. However, it requires data exchange between clients, which can lead to privacy breaches and vulnerability to poisoning attacks from malicious clients. In server-side distillation, the server aggregates client models and uses the average soft prediction from auxiliary datasets to fine-tune the global model. However, existing server-side distillation methods only utilize soft predictions to learn global knowledge. It relies on the set of local predictor outputs for distillation, making it sensitive to misleading and ambiguous knowledge injected by poorly performing local models. Sharing soft predictions exacerbates this problem. Summary of the Invention

[0005] The purpose of this invention is to provide an efficient heterogeneous federated learning method, system, device, and medium based on mixed distillation to solve the problems existing in the prior art.

[0006] To achieve the above objectives, this invention provides an efficient heterogeneous federated learning method based on hybrid distillation, comprising:

[0007] Step 1: Initialize the global model parameters on the server and set the preset total number of training rounds and the number of clients participating in training in each round;

[0008] Step 2: Load the corresponding local dataset in each client, perform random transformation on the local dataset to generate client distillation data; sample several sub-networks from the original network of each client, train each sub-network on the client distillation data, obtain the updated local model parameters after training, and upload the local model parameters to the server.

[0009] Step 3: After receiving the local model parameters, the server performs server distillation by combining them with a preset auxiliary dataset to obtain the updated global model parameters, and then sends the updated global model back to the client.

[0010] Repeat steps two through three until the model converges.

[0011] Optionally, a random transformation is performed on the local dataset, specifically including:

[0012] The local dataset is scaled and rotated to obtain the client-side distillation data.

[0013] Optionally, each of the sub-networks has a different network score width.

[0014] Optionally, each sub-network is trained on client-side distillation data, specifically including:

[0015] The KL divergence between the softmax output of the subnetwork and the original softmax output of the corresponding client local model is calculated as the distillation loss, and weights are dynamically allocated according to the prediction confidence of the subnetwork.

[0016] Based on the distillation loss and the traditional cross-entropy loss, the local model parameters are updated using an optimization algorithm.

[0017] Optionally, step three specifically includes:

[0018] The server receives all updated local model parameters from clients, performs weight aggregation, and obtains the global model.

[0019] Based on the preset auxiliary dataset, the global model is distilled using joint soft prediction distillation and feature distillation to obtain updated global model parameters, and the updated global model is sent back to the client.

[0020] An efficient heterogeneous federated learning system based on hybrid distillation includes:

[0021] The initialization module is used to initialize the global model parameters of the server and set the preset total number of training rounds and the number of clients participating in training in each round;

[0022] The client-side distillation module is used to load the corresponding local dataset in each client, perform random transformation on the local dataset to generate client-side distillation data; sample several sub-networks from the original network of each client, train each sub-network on the client-side distillation data, obtain updated local model parameters after training, and upload the local model parameters to the server.

[0023] The server distillation module is used to perform server distillation after the server receives the local model parameters and combines them with a preset auxiliary dataset to obtain updated global model parameters, and then send the updated global model back to the client; the client distillation and server distillation are repeated until the model converges.

[0024] An electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to cause the electronic device to perform an efficient heterogeneous federated learning method based on hybrid distillation as described above.

[0025] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned efficient heterogeneous federated learning method based on hybrid distillation.

[0026] The technical effects of this invention are as follows:

[0027] This invention employs a two-stage learning scheme, consisting of client-side self-distillation and server-side ensemble distillation, to mitigate local overfitting and improve model aggregation and generalization performance in federated learning on heterogeneous and long-tail client data. This scheme eliminates the need for any information exchange between clients and joint optimization via global data distribution, thereby preventing privacy breaches.

[0028] The server distillation method proposed in this invention can achieve better representation learning and a flatter loss landscape for fine-tuning the aggregation model, both of which help improve the accuracy of the global model under varying degrees of data heterogeneity and different numbers of clients.

[0029] The method designed in this invention not only overcomes the heterogeneity problem of federated learning, but also improves communication efficiency and attack robustness while avoiding privacy leaks. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 This is a flowchart illustrating the implementation of an embodiment of the present invention. Detailed Implementation

[0033] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0034] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0035] Unless otherwise stated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. While only preferred methods have been described herein, any methods similar or equivalent to those described herein may be used in the implementation or testing of this invention. All references to this specification are incorporated by way of citation to disclose and describe the methods associated with those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

[0036] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0037] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment provides an efficient heterogeneous federated learning method based on hybrid distillation, including:

[0041] Step 1: Initialize the global model parameters on the server and set the preset total number of training rounds and the number of clients participating in training in each round;

[0042] Step 2: Load the corresponding local dataset in each client, perform random transformation on the local dataset to generate client distillation data; sample several sub-networks from the original network of each client, train each sub-network on the client distillation data, obtain the updated local model parameters after training, and upload the local model parameters to the server.

[0043] Step 3: After receiving the local model parameters, the server performs server distillation by combining them with a preset auxiliary dataset to obtain the updated global model parameters, and then sends the updated global model back to the client.

[0044] Repeat steps two through three until the model converges.

[0045] This embodiment employs a two-stage learning scheme, consisting of client-side self-distillation and server-side integrated distillation, to mitigate local overfitting and improve model aggregation and generalization performance in federated learning on heterogeneous and long-tail client data. This scheme eliminates the need for any information exchange between clients and joint optimization via global data distribution, thereby preventing privacy breaches.

[0046] The server distillation method proposed in this embodiment can achieve better representation learning and a flatter loss landscape for fine-tuning the aggregation model, both of which help improve the accuracy of the global model under varying degrees of data heterogeneity and different numbers of clients.

[0047] The method designed in this embodiment not only overcomes the heterogeneity problem of federated learning, but also improves communication efficiency and attack robustness while avoiding privacy leaks.

[0048] This embodiment proposes a two-stage learning paradigm for heterogeneous federated learning, namely Federated Hybrid Knowledge Distillation (FedHyb). The two stages primarily involve knowledge distillation performed on the client and server sides respectively. Client-side distillation uses dynamically learned sub-networks to constrain local updates and mitigate local overfitting, while avoiding information exchange between clients. On the server side, a server-integrated distillation scheme is proposed, which comprehensively transmits aggregated client information to the global model at three different levels. The server-side distillation process is guided by a class-balanced unlabeled dataset. This auxiliary dataset can come from a third party or be generated by a generator independent of the client data distribution, to supervise the integrated knowledge transfer. In this way, more general feature representations can be learned from model aggregation, and the classifier can be retrained using a balanced set of data, thereby achieving better final test accuracy.

[0049] The specific plan is as follows:

[0050] 1. Traditional Federated Learning (FedAvg): Consider a federated learning system with K clients. For clients k = 1, 2, ..., K, each client can only access its own labeled local private data D. k := (x, y), where x and y are the client's local private data and label, respectively. Using N k :=|D k | represents the amount of client data. Participating clients use local training, with cross-entropy (CE) loss as follows:

[0051]

[0052] After training for a specified number of epochs, send the local client model parameters ω to the server. k =ω1, ω2, ..., ω KThen, the server performs a simple model aggregation through weight aggregation to obtain the global model parameters:

[0053]

[0054] Where pk = |D k | / |D| represents the amount of data from client k relative to the set of data from all clients. The weight ratios are determined. The server sends the global model back to the client. This process is repeated T times until convergence.

[0055] 2. Client-side Distillation Process: Most existing methods involve local logarithmic distillation, which requires information exchange between clients. If a malicious client participates, the privacy of other clients may be leaked during the collaborative learning process. To address this issue, this embodiment designs a client-side self-distillation scheme that protects privacy, requiring neither information exchange nor a shared public dataset between clients. In the self-distillation process, firstly, the local data sample D... k Perform random transformations, including scaling and rotation, on the data to obtain the client distillation data R. m (x). Secondly, this embodiment also samples M subnetworks with different network score widths from the original network, denoted as S. m (ω), where ω is the local client model parameter, m = 1, 2, ..., M. This embodiment then uses the cumulative distillation loss of the M sub-networks to constrain local updates and mitigate local overfitting. Specifically, this embodiment introduces the following regularization term to the traditional cross-entropy (CE) loss function to regularize the local model update at client k:

[0056]

[0057] Here, the softmax output of the m-th sub-network trained on the corresponding client distillation dataset is denoted as Q. k (S m (ω);R m (ω)), the original softmax output of the k-th client local model is Q. k (ω; x) are aligned by calculating the KL divergence between the two. The proposed client distillation involves sampling multiple subnetworks with different network score widths, which are fine-tuned using client distillation data generated by random transformations of local datasets. This enriches the learned local semantic information without leaking it to other clients.

[0058] 3. Dynamic Sub-network Weights: Due to the different fractional widths of the sampling sub-networks and the different simple transformations of the original image, the performance of each sub-network may vary. Ideally, a better-performing sub-network should contribute more to the original network. Based on this, this embodiment designs a dynamic weighted sub-network loss mechanism. This mechanism dynamically allocates loss weights to the sub-networks according to their performance. Specifically, as training progresses, each sub-network has different prediction confidence levels for the samples, allowing the sub-networks to dynamically contribute to the training of the original network. Adaptively allocating these sub-network weights is beneficial to the original network; that is, based on the prediction outputs of different sub-networks, in the current training, sub-networks with different confidence levels contribute differently to the original network. The formula is defined as follows:

[0059]

[0060] in This represents the cross-entropy (CE) loss calculated on the original label and the m-th subnetwork at client k. Thus, the subnetwork with higher prediction confidence will contribute more to the client-side self-distillation process. Combining the above design, the objective function of the local model at client k is:

[0061]

[0062] The first part is the traditional cross-entropy (CE) loss, as shown in Equation (1), and the second part is the cumulative knowledge distillation (KD) loss based on soft label prediction, which is a dynamically weighted process with adaptive model regularization. The contributions of CE and KD losses are balanced using the hyperparameter σ.

[0063] 4. Server-side distillation: Due to data heterogeneity, client-side models may differ significantly, and directly aggregating client-side models to obtain a server-side model results in poor performance. Therefore, further fine-tuning of the aggregated server-side model is performed. Specifically, further distillation is carried out on the server using an unlabeled class-balanced proxy dataset D. S :=x s The proxy dataset can come from a third party or be generated by a generator independent of the client data distribution. Based on the global proxy dataset, the ensemble knowledge of the client model is transferred to the server model, mitigating the drift of the server model formed by aggregating client models due to data heterogeneity. Simultaneously, to improve knowledge transfer, three levels of aggregation are performed: weight aggregation, soft prediction aggregation, and feature aggregation. First, weight aggregation, i.e., the traditional federated learning aggregation method, is performed to obtain the global model Ω. Then, the proxy dataset D is used... S The soft prediction distillation and feature distillation are combined to fine-tune Ω. The formula for soft prediction knowledge transfer is as follows:

[0064]

[0065] The first term in KL is the aggregated soft prediction output of the uploaded client model, i.e., soft prediction aggregation; the second is the global model output with the current network parameters Ω before fine-tuning. Then, the knowledge extracted from the features of the K clients is transferred to the server, i.e., representation knowledge transfer, which is accomplished by minimizing the mean squared error (MSE) distance between the server model's data representation output and the client model's data representation output. The formula for representation knowledge transfer is as follows:

[0066]

[0067] In MSE, the first term represents the penultimate feature extracted by the global model Ω, while the second term is an aggregation of the penultimate feature obtained from the uploaded client model. By combining these two types of knowledge transfer, the server ensemble distillation loss is:

[0068]

[0069] η and ν are used to adjust the ratio of soft predictive knowledge to representational knowledge.

[0070] Experiments show that the model trained using the scheme in this embodiment improves accuracy compared to traditional and state-of-the-art schemes in heterogeneous and long-tail federated learning environments, while also exhibiting leading advantages in communication efficiency and attack robustness. The evaluation datasets used in this embodiment are: 10-class SVHN and CIFAR-10, and 100-class CIFAR-100.

[0071] This embodiment tested the accuracy of the server-side global model and the client-side model with 10, 20, and 50 clients. The method proposed in this embodiment achieves the highest accuracy in most cases, especially for the server-side global model, and is unaffected by the number of clients, maintaining optimal performance even with a larger number of clients in a near-realistic scenario. Compared to the traditional FedAvg, this embodiment achieves an 11-24% gain in server accuracy, and compared to the latest proposed methods, it achieves a 2-19% gain in server accuracy.

[0072] Regarding heterogeneity, this embodiment uses a Dirichlet distribution to alter the heterogeneity of client data. Different heterogeneity levels are achieved by setting a heterogeneity factor; the smaller the heterogeneity factor, the more heterogeneous the data. Experiments show that the method in this embodiment is optimal compared to traditional and state-of-the-art solutions, achieving a gain of 3-20% under relatively heterogeneous conditions.

[0073] Regarding communication efficiency, by comparing the number of communication rounds required to reach the specified test accuracy with other methods, the fewer the number of communication rounds, the higher the communication efficiency. Through experiments, the method of this embodiment requires the fewest communication rounds for all specified test accuracies, indicating that this embodiment has a leading advantage in communication efficiency.

[0074] For attack robustness, this embodiment uses two attack methods to evaluate the robustness against malicious clients attempting to poison federated learning. One is a random noise (RN) attack, which generates perturbations based on a Gaussian distribution and introduces random noise during training to mislead the training process and degrade model performance. The other is a label flipping (LF) attack, which targets the global model by modifying the client dataset, involving changing the class of each instance in the dataset to a misclassified one. By comparing the method in this embodiment with traditional federated learning methods and state-of-the-art methods, all methods show varying degrees of decrease in global accuracy as the number of malicious clients (attackers) increases, but the method in this embodiment shows the smallest decrease, indicating that it significantly improves the robustness of the model under both types of data poisoning attacks.

[0075] An efficient heterogeneous federated learning system based on hybrid distillation includes:

[0076] The initialization module is used to initialize the global model parameters of the server and set the preset total number of training rounds and the number of clients participating in training in each round;

[0077] The client-side distillation module is used to load the corresponding local dataset in each client, perform random transformation on the local dataset to generate client-side distillation data; sample several sub-networks from the original network of each client, train each sub-network on the client-side distillation data, obtain updated local model parameters after training, and upload the local model parameters to the server.

[0078] The server distillation module is used to perform server distillation after the server receives the local model parameters and combines them with a preset auxiliary dataset to obtain updated global model parameters, and then send the updated global model back to the client; the client distillation and server distillation are repeated until the model converges.

[0079] An electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to cause the electronic device to perform an efficient heterogeneous federated learning method based on hybrid distillation as described above.

[0080] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned efficient heterogeneous federated learning method based on hybrid distillation.

[0081] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A high-efficiency heterogeneous federated learning method based on hybrid distillation, characterized in that, Comprising: Step one: make the server initialize the global model parameters, and set the preset total training rounds and the number of clients participating in training per round; Step two: in each client, load the corresponding local data set, perform random transformation on the local data set to generate client distillation data; sample several subnetworks from the original network of each client, each subnetwork is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server; Each subnetwork is trained on the client distillation data, specifically including: Calculate the KL divergence of the softmax output of the subnetwork and the original softmax output of the corresponding client local model as the distillation loss, and dynamically allocate weights according to the prediction confidence of the subnetwork; Based on the distillation loss and the traditional cross-entropy loss, update the local model parameters using an optimization algorithm; Step three: the server receives the local model parameters and combines the preset auxiliary data set for server distillation to obtain updated global model parameters, and sends the updated global model back to the client; Repeat steps two to three until the model converges.

2. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. Perform random transformation on the local data set, specifically including: Perform scaling and rotation on the local data set to obtain the client distillation data.

3. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. Each of the subnetworks has a different network score width.

4. The method of claim 1, wherein the method is a hybrid distillation based efficient heterogeneous federated learning method. The step three, specifically includes: The server receives all the updated local model parameters of the clients, aggregates the weights to obtain the global model; Based on the preset auxiliary data set, combine joint soft prediction distillation and feature distillation to distill the global model to obtain updated global model parameters, and send the updated global model back to the client.

5. A high efficient heterogeneous federated learning system based on hybrid distillation, applying the method of any one of claims 1-4, characterized in that, Comprising: An initialization module for making the server initialize the global model parameters, and setting the preset total training rounds and the number of clients participating in training per round; A client distillation module for loading the corresponding local data set in each client, performing random transformation on the local data set to generate client distillation data; sampling several subnetworks from the original network of each client, each subnetwork is trained on the client distillation data, and the updated local model parameters are obtained after training, and the local model parameters are uploaded to the server; A server distillation module for the server to receive the local model parameters and combine the preset auxiliary data set for server distillation to obtain updated global model parameters, and send the updated global model back to the client; repeat the client distillation and server distillation until the model converges.

6. An electronic device, comprising: Comprising a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to make the electronic device execute the efficient heterogeneous federated learning method based on mixed distillation according to any one of claims 1-4.

7. A computer readable storage medium characterized in that, It stores a computer program, which is executed by a processor to realize the efficient heterogeneous federated learning method based on mixed distillation according to any one of claims 1-4.

Images