Knowledge distillation-based efficient heterogeneous client federal learning method

Abstract

The invention provides an efficient heterogeneous client federated learning method based on knowledge distillation. The efficient heterogeneous client federated learning method comprises the following steps: selecting a chairman model, namely an RC model, of a training round from client models participating in federated learning by using a heuristic chairman selection strategy; uploading the selected chairman model to a server; sending the chairman model to each local client; learning knowledge in the local model through knowledge distillation; the knowledge is obtained from a client model through local data training learning; uploading the trained chairman model to the server again; the server collects and aggregates the chairman models uploaded by the client to obtain the aggregated chairman model; redistributing the aggregated chairman model to each client; the chairman model guides the client model to update, and the client model is trained; and judging whether 200 rounds of training are reached or not, if not, returning, and if so, ending.

Description

Technical Field

[0001] This invention relates to the technical field of federated learning methods, and more particularly to an efficient heterogeneous client federated learning method based on knowledge distillation. Background Technology

[0002] With the rapid development of the Internet, the Internet of Things (IoT), and mobile communication technologies, the global volume of data is exploding. Various digital platforms, including social media, e-commerce, smart terminals, and industrial sensors, generate massive amounts of structured and unstructured data daily (such as user behavior logs, transaction records, biometrics, and equipment monitoring data). This data contains rich potential value and can provide strong data support for training artificial intelligence models, thereby promoting the widespread application of machine learning in fields such as computer vision, natural language processing, recommender systems, and medical diagnosis.

[0003] However, despite the sheer volume of data, its practical utilization faces significant challenges. On one hand, data privacy and security issues are increasingly prominent. Much data involves sensitive personal information (such as identity information, financial records, and health data) or contains core business secrets (such as user preferences and production processes). Under strict privacy regulations, cross-institutional data sharing and centralized storage face extremely high compliance risks. On the other hand, data silos are prevalent. Due to competition, technological barriers, or management policy restrictions, data is difficult to exchange between different industries, enterprises, and even departments, resulting in a large amount of high-value data being stored fragmented and unable to fully realize its training value.

[0004] To address the aforementioned issues, Federated Learning (FL) emerged. As an emerging distributed machine learning paradigm, Federated Learning adopts the core idea of ​​"data remains stationary, model moves," allowing multiple participants to collaboratively train a model without sharing the original data. Its basic process includes: (1) Local training: Each participant (client) trains a local model based on its own data; (2) Parameter aggregation: The client only uploads the model parameters (not the original data) to the central server; (3) Global update: The server aggregates the parameters and distributes the new model for iterative optimization.

[0005] This framework effectively addresses privacy risks while promoting cross-institutional data value mining, and has been successfully implemented in scenarios such as financial risk control, smart healthcare, and smart terminals. For example, multiple banks can jointly train anti-fraud models based on federated learning without sharing customer transaction data; hospitals can collaborate to build disease prediction models while protecting patient privacy. Furthermore, the rise of open-source communities (such as FATE and TensorFlow Federated) has further promoted the standardization and popularization of federated learning.

[0006] However, despite the excellent performance of federated learning in terms of privacy protection, existing methods still have significant technical limitations, especially in terms of data heterogeneity (Non-IID) and personalized needs: 1. Data heterogeneity leads to degraded model performance: Non-IID data distribution problem: In real-world scenarios, data from different clients often exhibits highly non-independent and identically distributed (Non-IID) characteristics. For example, user consumption habits vary significantly across different regions, or case distributions differ greatly between different medical institutions. This data heterogeneity causes the local model's optimization direction to deviate from the global optimum, triggering the "Client Drift" phenomenon, making global model convergence difficult and reducing generalization performance. Experiments show that the traditional federated averaging algorithm (FedAvg) may experience a 20%-40% drop in test accuracy under Non-IID data, and requires more communication rounds to reach an acceptable performance level.

[0007] 2. Insufficient support for heterogeneous models: Model homogeneity limitations: Traditional federated learning forces all clients to use the same global model architecture. However, in real-world scenarios, clients often require highly personalized models due to differences in hardware (e.g., mobile phones, servers), task requirements (e.g., classification, generation), or business goals (e.g., different user group preferences). Existing methods cannot flexibly support heterogeneous model structures, resulting in poor local deployment performance. Summary of the Invention

[0008] In view of the technical problems mentioned in the background, an efficient heterogeneous client federated learning method based on knowledge distillation is provided.

[0009] The technical means employed in this invention are as follows: An efficient heterogeneous client federated learning method based on knowledge distillation includes the following steps: Step 1: Use a heuristic chair selection strategy to select the chair model (RC model) for this training round from the client models participating in federated learning; Step 2: Upload the selected chairman model to the server; Step 3: Send the chairman model to each local client; Step 4: Learn the knowledge in the local model through knowledge distillation; the knowledge is acquired from the client model through training and learning on local data. Step 5: Upload the trained chairman model to the server again; Step 6: The server collects and aggregates the chairman models uploaded by the client to obtain the aggregated chairman model; Step 7: Redistribute the aggregated chairman model to each client; Step 8: The chairman model guides the client model update, and the client model is trained. Step 9: Determine if 200 rounds of training have been reached. If not, return to step 1; if so, end the process.

[0010] Furthermore, in step 8, cross-entropy loss and KL loss are used for interactive learning; the loss functions of the two models are set as follows: ; ; in, Represents cross-entropy loss; Indicates channel distillation loss; This represents the knowledge distillation loss due to decoupling. Denotes KL divergence, and These represent the predicted logit values ​​of the local model and the RC model, respectively. Labels representing data, and These represent the feature maps of the local model and the RC model, respectively.

[0011] Furthermore, in step 2, the global model and the local client's personalized model are initialized, and the structure of the global model is set to be the same as the model of the first client.

[0012] Furthermore, in step 4, the RC model forks the global model and performs an interactive learning round; the local model learns the RC model through channel distillation and decoupled knowledge distillation to complete the update; the RC model then learns the local model through KL loss to complete the update.

[0013] Furthermore, in step 6, the trained RC model parameters are weighted, averaged, and aggregated before being passed to the global model: ; in, Represents global model parameters. This indicates the local client.

[0014] Furthermore, the RC model is tested using test data, specifically by recording the highest predicted probability value for each model. The second highest probability prediction value ; Use them for each client model value minus Value, obtain the difference

[0015] ; Find all clients The model with the smallest value will be used as the coordinating node for the next round.

[0016] Compared with the prior art, the present invention has the following advantages: (1) This method achieves efficient knowledge propagation while meeting the needs of heterogeneous client models by using an interactive learning strategy based on channel distillation and decoupled knowledge distillation, thus solving the problem of insufficient support for model heterogeneity in existing federated learning.

[0017] (2) At the same time, by using the “rotating chairman system” and heuristic chairman selection strategy, knowledge can be better transferred between clients, greatly improving the training effect and solving the problem of poor training effect caused by data heterogeneity in federated learning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] like Figure 1 As shown, this invention provides an efficient heterogeneous client federated learning method based on knowledge distillation, comprising the following steps: Step 1: Use a heuristic chair selection strategy to select the chair model (RC model) for this training round from the client models participating in federated learning; Step 2: Upload the selected chairman model to the server; Step 3: Send the chairman model to each local client; Step 4: Learn the knowledge in the local model through knowledge distillation; the knowledge is acquired from the client model through training and learning on local data. Step 5: Upload the trained chairman model to the server again; Step 6: The server collects and aggregates the chairman models uploaded by the client to obtain the aggregated chairman model; Step 7: Redistribute the aggregated chairman model to each client; Step 8: The chairman model guides the client model update, and the client model is trained. Step 9: Determine if 200 rounds of training have been reached. If not, return to step 1; if so, end the process.

[0023] Because the local clients participating in the training have different model architectures, the global model parameters cannot be directly federated through weighted averaging. Knowledge distillation can transfer knowledge without relying on model structure; therefore, it is necessary to use knowledge distillation to solve this problem. To enable better interactive learning between the RC model and the local model, this application employs channel distillation and decoupled knowledge distillation loss to support mutual learning between heterogeneous models, while mitigating the negative impact of client drift caused by data heterogeneity. When the local model learns the RC model, this application uses channel distillation + decoupled knowledge distillation + cross-entropy loss; when the RC model learns the local model, it uses cross-entropy loss + KL loss for interactive learning. In step 8, cross-entropy loss and KL loss are used for interactive learning; the loss functions of the two models are set as follows: ; ; in, Represents cross-entropy loss; Indicates channel distillation loss; This represents the knowledge distillation loss due to decoupling. Denotes KL divergence, and These represent the predicted logit values ​​of the local model and the RC model, respectively. Labels representing data, and These represent the feature maps of the local model and the RC model, respectively.

[0024] In reality, models and data are typically heterogeneous. Given this model heterogeneity, each client will naturally prefer to use its own customized model rather than a generic one. This necessitates the use of federated learning to improve the performance of local models on each client.

[0025] This application designs a novel federated learning mechanism called the "rotating chairperson system." This transforms the RC model into a dynamic model, enabling better interactive learning between the local model and the RC model. In step 2, the global model and the local client's personalized model are initialized, setting the structure of the global model to be the same as the model of the first client.

[0026] Furthermore, in step 4, the RC model forks the global model and performs an interactive learning round; the local model learns the RC model through channel distillation and decoupled knowledge distillation to complete the update; the RC model then learns the local model through KL loss to complete the update.

[0027] Furthermore, in step 6, the trained RC model parameters are weighted, averaged, and aggregated before being passed to the global model: ; in, Represents global model parameters. This represents the local client. The local model then updates itself by learning from the updated RC model through channel distillation and decoupled knowledge distillation. If the RC model and the local model are identical, they are directly copied without distillation. After training, the RC model and the global model become identical to the next local client. The RC model then learns from the local model using KL loss to update itself. After updating, it is passed to the global model via Eq.3 aggregation. The global model updates itself and then forks its parameters back to the RC model, and so on.

[0028] Here, the RC model structure is set to be dynamic, changing with each round of communication to become the model of each client in turn. This allows each local personalized client to take turns acting as the server. The advantage of this is that it allows the client to better learn the knowledge of other clients, mitigates the performance degradation caused by data heterogeneity, and improves the effectiveness of the local model.

[0029] Furthermore, the RC model is tested using test data, specifically by recording the highest predicted probability value for each model. The second highest probability prediction value ; Use them for each client model value minus Value, obtain the difference

[0030] ; Find all clients The model with the smallest value will be used as the coordinating node for the next round.

[0031] Example 1 Table 1

[0032] Table 1 shows the training results of this method and other methods on the MNIST dataset, trained for 200 rounds under three imbalance levels.

[0033] In a specific implementation, this invention uses three image classification datasets—MNIST, CIFAR10, and CIFAR100—for validation. Three local models with different architectures—CNN, LeNet, and AlexNet—are used to simulate heterogeneous model scenarios. Data partitioning employs a Dirichlet distribution (Dir(α)) to generate a Non-IID data distribution. The α values ​​for CIFAR10 / CIFAR100 are set to 0.1, 0.5, and 1.0, while the α values ​​for MNIST are set to 0.05, 0.1, and 0.5 to simulate varying degrees of data heterogeneity. The experiment is set up with 10 clients by default, performing 120 rounds of communication training. Each round of local training consists of 5 epochs, with a batch size of 128 and a learning rate of 0.005. The evaluation metric is the average accuracy of the personalized models on each client, comprehensively measuring the algorithm's actual performance in a heterogeneous environment.

[0034] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient heterogeneous client federated learning method based on knowledge distillation, characterized in that, Includes the following steps: Step 1: Use a heuristic chair selection strategy to select the chair model (RC model) for this training round from the client models participating in federated learning; Step 2: Upload the selected chairman model to the server; Step 3: Send the chairman model to each local client; Step 4: Learn the knowledge in the local model through knowledge distillation; The knowledge is acquired from the client-side model through training and learning on local data. Step 5: Upload the trained chairman model to the server again; Step 6: The server collects and aggregates the chairman models uploaded by the client to obtain the aggregated chairman model; Step 7: Redistribute the aggregated chairman model to each client; Step 8: The chairman model guides the client model update, and the client model is trained. Step 9: Determine if 200 rounds of training have been reached. If not, return to step 1; if so, end the process.

2. The efficient heterogeneous client federated learning method based on knowledge distillation according to claim 1, characterized in that, In step 8, cross-entropy loss and KL loss are used for interactive learning; the loss functions of the two models are set as follows: ； ； in, Represents cross-entropy loss; Indicates channel distillation loss; This represents the knowledge distillation loss due to decoupling. Denotes KL divergence, and These represent the predicted logit values ​​of the local model and the RC model, respectively. Labels representing data, and These represent the feature maps of the local model and the RC model, respectively.

3. The efficient heterogeneous client federated learning method based on knowledge distillation according to claim 1, characterized in that, In step 2, the global model and the local client's personalized model are initialized, and the structure of the global model is set to be the same as the model of the first client.

4. The efficient heterogeneous client federated learning method based on knowledge distillation according to claim 1, characterized in that, In step 4, the RC model is forked from the global model and an interactive learning round is performed. The local model is updated by learning the RC model through channel distillation and decoupled knowledge distillation. The RC model is then updated by learning the local model through KL loss.

5. The efficient heterogeneous client federated learning method based on knowledge distillation according to claim 1, characterized in that, In step 6, the trained RC model parameters are weighted, averaged, and aggregated before being passed to the global model. ； in, Represents global model parameters. This indicates the local client.

6. The efficient heterogeneous client federated learning method based on knowledge distillation according to claim 5, characterized in that, The RC model is tested using test data, specifically by recording the highest predicted probability value for each model. The second highest probability prediction value ; Use them for each client model value minus Value, obtain the difference ； Find all clients The model with the smallest value will be used as the coordinating node for the next round.

Images