Heterogeneous edge calculation multi-teacher federated distillation method based on course learning

Abstract

The invention discloses a heterogeneous edge calculation multi-teacher federated distillation method based on course learning, and aims to reduce the influence of statistical heterogeneity on model performance on the premise of protecting the heterogeneity of an edge device model. The method specifically comprises the following steps: S1, a client performs local training by using a private data set and global knowledge, and then transmits output obtained by reasoning on a public data set to a server; s2, the server evaluates the quality of the output uploaded by each client by taking a sample as a unit, and then performs weighted aggregation to generate comprehensive knowledge; and S3, the server follows the thought of course learning, dynamically trains a global model by using comprehensive knowledge, and then sends global knowledge output by the global model to the client. And repeating the above process, and training until the global model converges. Compared with a previous federated learning method, the method has the advantages that the performance on four data sets is obviously improved, and the method is generally suitable for various training tasks.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a heterogeneous edge computing multi-teacher federated distillation method based on curriculum learning. Background Technology

[0002] With the rapid development of edge computing, numerous edge devices and sensors are generating massive amounts of data in real time across diverse application areas, such as autonomous driving, healthcare, and finance. However, the sensitivity and privacy of certain information make direct data collection for training impractical. Federated learning, by training local models on edge nodes and exchanging model updates with the server, can achieve collaborative intelligence at the edge while protecting user privacy and reducing communication overhead. However, due to the limited computing and memory capabilities of edge nodes, only low-complexity models can be trained, and federated learning requires the server to have a consistent model with the edge device, which limits the performance of the global model. Furthermore, differences in device type, deployment environment, and other factors can lead to statistical heterogeneity among clients, causing divergence in the direction of model updates and making it difficult for the global model to converge to the optimal solution. These issues undoubtedly limit the performance of the global model.

[0003] Recently, an increasing number of studies have turned to addressing the challenges posed by non-independent and identically distributed (i.i.d.) data. Common approaches can be categorized into two types: one is for traditional federated learning, which limits client-side model updates by adding proximal constraints. FedProx applies regularization to federated learning, making the local model as close as possible to the global model, effectively mitigating the impact of data heterogeneity. SCAFFOLD adds a correction term in each local iteration, continuously correcting the model update direction to prevent deviations. MOON uses contrastive learning to correct client-side local training by leveraging the similarity between model representations, requiring similarity between the extracted features of the local and global models. These methods mostly improve the local training process of the client model to prevent the client from deviating excessively from the server model. The other type is personalized federated learning, which allows clients to train a personalized model for their local data distribution. FedRep only aggregates the model's representation layer, allowing clients to personalize their head to better suit their local data. FedBABU advocates that clients use a fixed model head during training, updating only the model body parameters; after model training, the head is personalized. pFedHN, on the other hand, designs a hypernetwork for the server that can generate personalized local models for each client to better adapt to local data distribution. While these methods can mitigate the impact of data heterogeneity, they still require the transfer of models between the server and the client, resulting in significant communication costs; furthermore, they require the client and server models to have the same architecture, leading to a waste of server resources and underutilization of the server's powerful computing capabilities.

[0004] Federated distillation introduces knowledge distillation into federated learning. Unlike uploading model gradients or weights, clients send their outputs or intermediate embeddings to a central server, eliminating the need for strict consistency between client and server models and significantly reducing communication costs. Currently, federated distillation is widely used to address statistical heterogeneity in federated learning. FedGKD effectively mitigates client drift by integrating knowledge from historical global models to guide local training. FedBR applies block-level regularization and knowledge distillation to the standard federated learning framework to reduce the impact of statistical heterogeneity during local training. DS-FL introduces an entropy reduction aggregation strategy to effectively integrate client outputs, thereby improving global model alignment and performance. FedSND employs self-distillation on the client side and introduces a pseudo-sample generation module on the server side to jointly improve model accuracy and communication efficiency. CDKT-FL introduces a cross-device knowledge transfer scheme based on client outputs and representations, utilizing lightweight proxy data to achieve robust model aggregation. However, most existing methods still employ standard training procedures after acquiring knowledge from teacher models, often failing to fully utilize this knowledge to improve local training performance. Summary of the Invention

[0005] This invention provides a multi-teacher federated distillation method for heterogeneous edge computing based on course learning, which aims to maintain model heterogeneity while reducing the performance degradation caused by statistical heterogeneity.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A heterogeneous edge computing multi-teacher federated distillation method based on curriculum learning, which specifically includes the following steps:

[0008] S1. The client uses a private dataset and global knowledge to train locally, and then passes the output obtained by inference on a public dataset to the server.

[0009] S2. The server evaluates the quality of the output uploaded by each client on a sample basis, and then performs weighted aggregation to generate comprehensive knowledge.

[0010] S3. Following the principles of course learning, the server dynamically trains the global model using comprehensive knowledge, and then sends the global knowledge output by the global model to the client.

[0011] In each round, the S3 process is repeated. Next, the trained model is obtained in The output is broadcast to the client for the next round of training, and then the S1-S3 process is repeated; training continues until the global model converges.

[0012] Furthermore, step S1 specifically includes:

[0013] In the In the round, the client Gain global knowledge of server broadcasts Then, using this knowledge and its own local dataset, the local model is trained. Once training is complete, the client will use the public dataset. Latest output on Upload to the server. (Client) Local loss It can be written as:

[0014]

[0015] in, Indicates client Local dataset, This represents the cross-entropy loss value on the local dataset. Used to measure public datasets The difference between the output of the local model and the global model Used to balance the weights of the two loss terms;

[0016] Furthermore, step S2 specifically includes:

[0017] This invention treats each client as a teacher model, while the global model on the server acts as a student model, using multiple teacher models to train a single student model. Due to statistical heterogeneity among clients, the quality of knowledge provided by each client may vary depending on the sample. Therefore, to obtain high-quality knowledge, it is necessary to evaluate the quality of knowledge generated by each client on a sample-by-sample basis. First, calculations are performed on a public dataset... The loss between the model's predicted value and the true label This, to some extent, reflects the model's predictive accuracy on that sample. Then, for the... The sample, according to the first The loss value of each client is used to evaluate the output. Weighting is applied to ensure that higher-quality knowledge is given greater weight; therefore, the client... right The Middle Weights of each sample It can be represented as:

[0018]

[0019]

[0020] in, This indicates the total number of clients. Indicates client In the Loss on a single sample express function, This indicates that the sample in the client output belongs to the first... The probability of class; if the first Each sample belongs to category ,but ;otherwise, This ensures that clients whose predictions are closer to the true values ​​will receive higher weights, thus playing a dominant role in training.

[0021] Finally, the first [model] used to train the global model Comprehensive knowledge of individual samples Defined as:

[0022]

[0023] It integrates weights from multiple clients, providing more reliable and informative guidance for training a global model.

[0024] Furthermore, step S3 specifically includes:

[0025] Following the principles of course learning, the training difficulty is gradually adjusted based on model performance, enabling the model to adapt to updates of diverse data samples. This gradually improves its adaptability and generalization ability under different data distributions, effectively mitigating the negative impact of statistical heterogeneity on global model performance. Specifically, parameters are adjusted in reverse based on the gradient of each model update. The difficulty is increased when accuracy improves and decreased when accuracy declines. This allows for dynamic adjustment of model training without explicitly changing the sample sequence.

[0026] During global model training, the update operation is based on the global model loss. The process is represented as:

[0027]

[0028] in, This represents the output of the global model. This represents the comprehensive knowledge obtained through weighted summation in step S2. express function; The weights representing the KL divergence loss are used to balance the two loss values; It is a hyperparameter used to adjust the difficulty of training, but most current knowledge distillation methods will... Set to a fixed value. In fact, when At lower levels, the probability distribution becomes sharper, causing the distillation process to focus primarily on the maximum value within the probability distribution. However, as... As the number of samples gradually increases and the distribution becomes smoother, the boundaries between sample classes become blurred, increasing the complexity of training. This invention dynamically adjusts the model based on its performance. The value of is determined to ensure that as model performance improves, the difficulty increases, promoting deeper knowledge acquisition; conversely, when performance declines, the difficulty is appropriately reduced to maintain the stability and robustness of the training process and prevent further performance degradation.

[0029] Therefore, this invention introduces a new parameter adjustment module. global model The goal is to minimize ,and Conversely, they attempt to increase the difficulty of training; therefore, and The parameter update method is as follows:

[0030]

[0031]

[0032] However, a limited number of training epochs may prevent the model from reaching its optimal performance, resulting in relatively low-quality knowledge that provides incorrect guidance for model updates. It is more sensitive to the direction of updates, and even small changes can have a profound impact; therefore, a warm-up period is established, and additional parameters are introduced. To control right The extent of the impact should be considered to prevent excessive early impact; therefore, The update process is as follows:

[0033]

[0034] Initial reduction right The influence of inaccurate knowledge is gradually increased as model performance improves, which helps prevent misleading information and ensures that the impact of inaccurate information is minimized. Initially, the influence is gradually increased using a cosine or linear increase strategy. The value of ; when the model performance reaches a stable state. It will eventually become fixed;

[0035] get Output Afterwards, Scale to an appropriate range and ensure it is non-negative:

[0036]

[0037] in and They represent Minimum and maximum values Make Determine based on the current training situation and maintain it. Within the range.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1) Compared with the traditional federated distillation framework, this invention filters and integrates the knowledge of each client, improves the quality of the knowledge used to train the global model, and helps to improve the performance of the global model.

[0040] 2) This invention adopts the concept of course learning in the training process of the global model. By dynamically adjusting the difficulty of the model training process, it is beneficial for the model to explore deeper knowledge and gradually improve the model performance. Attached Figure Description

[0041] Figure 1 This is a flowchart of the present invention;

[0042] Figure 2 This is a schematic diagram of the framework of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. The processes, conditions, experimental methods, etc., for implementing the present invention, except as specifically mentioned below, are all common knowledge and general knowledge in the art. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0044] Example

[0045] See Figures 1-2 The present invention provides a multi-teacher federated distillation method for heterogeneous edge computing based on course learning, using the following steps:

[0046] Consider a typical federated learning framework with a total of 20 clients. Each client iterates twice per round on its local dataset using the SGD optimizer, and the global model is also trained twice per round on the public dataset.

[0047] S1: In the In the round, the server will use the trained global model knowledge Broadcast to selected clients. Each client trains its local model by dividing the local and public datasets into multiple batches, with each batch... Each sample, in each training round Next. After training is complete, the client... In public datasets Calculation output The data is uploaded to the server. (Client) Local loss It can be written as:

[0048]

[0049] in, Indicates client Local dataset, This represents the cross-entropy loss value on the local dataset. Used to measure public datasets The difference between the output of the local model and the global model Used to balance the weights of the two loss terms;

[0050] S2: The server evaluates the quality of knowledge uploaded by each client on a sample-by-sample basis. First, it evaluates the knowledge on a public dataset. For each sample, the cross-entropy loss between the client model's predicted value and the true label is calculated. .for The first in The sample, according to the first Loss value calculated by each client Determine the output Weighting ensures that higher-quality knowledge is given greater weight. Therefore, for The Middle One sample, client weight It can be represented as:

[0051]

[0052]

[0053] in, This indicates the total number of clients. Indicates client In the Loss on a single sample express function, This indicates that the sample in the client output belongs to the first... The probability of class. If the first Each sample belongs to category ,but ;otherwise, .

[0054] Finally, the first [model] used to train the global model Comprehensive knowledge of individual samples Defined as:

[0055]

[0056] It integrates weights from multiple clients, providing reliable and informative guidance for training a global model.

[0057] S3: Adjust the model parameters in reverse based on the gradient of each round of global model update. Increase the difficulty when the accuracy increases and appropriately reduce the difficulty when the accuracy decreases, thus dynamically adjusting the training of the model.

[0058] The global model is based on the global model loss. Update It can be represented as:

[0059]

[0060] in, This represents the output of the global model. This represents the comprehensive knowledge obtained by weighting S2. express function; The weights representing the KL divergence loss are used to balance the two loss values; It is a hyperparameter used to adjust the difficulty of training, but most current knowledge distillation methods will... Set to a fixed value. This invention dynamically adjusts the value based on the model's performance. The value of is determined to ensure that as model performance improves, the difficulty increases, promoting deeper knowledge acquisition; conversely, when performance declines, the difficulty is appropriately reduced to maintain the stability and robustness of the training process.

[0061] Global Model The goal is to minimize And parameter adjustment module This attempts to increase the difficulty of training. Therefore, and The parameters are updated as follows:

[0062]

[0063]

[0064] However, a limited number of training epochs may prevent the local model from achieving optimal performance, resulting in relatively low-quality knowledge that provides incorrect guidance for model updates. It is more sensitive to the direction of updates; even small changes can have a profound impact. Therefore, a warm-up period is established, and additional parameters are introduced. To control right The extent of the impact, to prevent excessive early impact. Therefore, The update process is as follows:

[0065]

[0066] Initial reduction right The degree of influence is gradually increased as model performance improves, which helps prevent misleading information from inaccurate knowledge. Initially, a cosine strategy is used, gradually increasing the influence. The value of when the model performance reaches a stable state. It will eventually be fixed.

[0067] get Output Afterwards, Scale to an appropriate range and ensure it is non-negative:

[0068]

[0069] in and They represent The minimum and maximum values. Make Determine based on the current training situation and maintain it. Within the range, set , .

[0070] In each round of training, the S3 process is repeated. Next, the trained model is obtained in The output is broadcast to the client for the next round of training, and then the S1-S3 process is repeated; training continues until the global model converges.

[0071] Experimental verification

[0072] To verify the universality of this invention, four datasets were selected for the experiment: FashionMNIST, CIFAR-10, CIFAR-100, and EMNIST. For FashionMNIST, CIFAR-10, and CIFAR-100, two-layer, three-layer, and four-layer CNNs were used for training, respectively; for EMNIST, a two-layer DNN was used for training. Each task was trained for 100 communication rounds.

[0073] The dataset uses the Dirichlet distribution ( The data is divided into segments to simulate statistical heterogeneity among clients. The degree of heterogeneity control is controlled; smaller values ​​indicate a higher level of heterogeneity and more significant differences. In the experiment, [the following values ​​were used to control for heterogeneity]. Set to 0.01 and 0.1.

[0074] The model was trained using the SGD optimizer, with a client-side model learning rate of 0.01 and a global model learning rate of 0.08. The loss function was set to... and To balance the relative weights of the various loss terms. and Set them to 1 and 10 respectively. In each round, select... Training is performed on one client. The value can be either 1 or 0.5.

[0075] Table 1 reports the test accuracy of the trained model in this embodiment (abbreviated as FedCMTD) under three different distribution scenarios. To demonstrate the beneficial effects of the present invention, it is also compared with other existing methods, including (1) FedAvg: directly averaging and aggregating the client model parameters. (2) FedProx: introducing proximal term regularization on the basis of FedAvg, effectively mitigating the impact of statistical heterogeneity by making the local model closer to the global model. (3) Scaffold: maintaining a control variable for each client, and the client adjusts the update direction of the model according to the control variable to prevent deviation in the update direction. (4) MOON: using contrastive learning, using the similarity between model representations to correct the local training of the model. The model of the previous round is regarded as a negative sample, and the global model features are regarded as positive samples. (5) FedDF: incorporating ensemble distillation on the basis of FedAvg to promote model fusion and allow flexible aggregation of heterogeneous client models. (6) FedKT: introducing a model split distillation method and a learnable aggregation network to achieve effective server-side knowledge integration and transfer. Meanwhile, a consistency constraint loss was designed on the client side to correct local updates. (7) CDKT-FL: A knowledge transfer mechanism was introduced on both sides, and the model was updated on a small proxy dataset using the average value of the model output.

[0076] Table 1 shows the experimental results, demonstrating that the proposed method significantly outperforms other methods in most experimental settings. When client-side statistical heterogeneity exists, the performance of knowledge distillation methods (including FedDF, FedKT, CDKT-FL, and FedCMTD) is significantly better than that of models obtained through parameter exchange. This is because knowledge distillation-based methods enable the global model to more effectively absorb diverse knowledge contributed by different clients. FedDF and FedKT incorporate knowledge distillation on top of parameter exchange, using the average value of knowledge to guide the update of the aggregated model. This results in the server being unable to selectively utilize client knowledge, leading to a slight decrease in overall performance. Compared to CDKT-FL, FedCMTD consistently demonstrates superior performance, with the highest accuracy improvement being approximately 2.4%.

[0077] Additionally, generally speaking, when the number of participating clients decreases (i.e. When heterogeneity is low (i.e., when the model's accuracy decreases accordingly), the model's accuracy also decreases. For methods using knowledge distillation, the model performance changes relatively little in all cases, exhibiting good robustness. However, for methods based on model parameter exchange, when heterogeneity is low (i.e., when the model's accuracy decreases accordingly), the model's accuracy also decreases accordingly. When heterogeneity is high (i.e., when the model performance is relatively small, the impact on model performance is small); when the heterogeneity is high (i.e., when the model performance is relatively small, the impact on model performance is small). When heterogeneity is high, model performance degrades significantly, a phenomenon most pronounced on the FashionMNIST dataset. This is because high heterogeneity means different clients are selected in each round, leading to different convergence directions and performance oscillations, making it difficult to find the optimal direction. Conversely, low heterogeneity results in updates closer to the optimal direction, reducing the impact of selecting only some clients for training. FedDF and FedKT incorporate knowledge distillation into standard federated learning, mitigating the impact of different convergence directions on model aggregation. However, FedKT's accuracy drops significantly with increasing training difficulty, indicating weaker generalization than the method presented in this invention. Overall, this invention achieves the best results in most settings. FedCMTD automatically adjusts weights to reduce the impact of inaccurate knowledge, ensuring that even some clients learn relatively high-quality knowledge and reducing model bias. Furthermore, by adjusting training difficulty, it allows for the gradual learning of deeper knowledge while mastering fundamental concepts. The experimental results show that regardless of the number of clients participating in the training, the model of the present invention can achieve high accuracy in most cases, which proves the effectiveness and generalization of the method of the present invention.

[0078] Table 1 Performance Evaluation of FedCMTD

[0079]

[0080] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A heterogeneous edge computing multi-teacher federated distillation method based on curriculum learning, characterized in that, The method specifically includes the following steps: S1. The client uses a private dataset and global knowledge to train locally, and then passes the output obtained by inference on a public dataset to the server. S2. The server evaluates the quality of the output uploaded by each client on a sample basis, and then performs weighted aggregation to generate comprehensive knowledge. S3. Following the principles of course learning, the server dynamically trains the global model using comprehensive knowledge, and then sends the global knowledge output by the global model to the client. In each round, the S3 process is repeated. Next, the trained model is obtained on a public dataset. The output is broadcast to the client for the next round of training, and then the S1-S3 process is repeated; Train until the global model converges.

2. The heterogeneous edge computing multi-teacher federated distillation method based on course learning according to claim 1, characterized in that, Step S1 specifically includes: In the In the round, the client Gain global knowledge of server broadcasts Then, using this knowledge and its own local dataset, the local model is trained. Once training is complete, the client will use the public dataset. Latest output on Upload to server; client Local loss Written as: ； in, Indicates the client Local dataset, This represents the cross-entropy loss value on the local dataset. Used to measure public datasets The difference between the output of the local model and the global model The weights used to balance the two loss terms.

3. The heterogeneous edge computing multi-teacher federated distillation method based on course learning according to claim 1, characterized in that, Step S2 specifically includes: Each client is treated as a teacher model, while the global model on the server acts as a student model. Multiple teacher models are used to train a single student model. The server evaluates the quality of knowledge generated by each client on a sample-by-sample basis. First, it computes data on a public dataset. The loss between the model's predicted value and the true label Then for the first The sample, according to the first The loss value of each client is paired with the output. Weighting is applied to ensure that higher-quality knowledge is given greater weight; therefore, the client... right The Middle Weights of each sample It can be represented as: in, This indicates the total number of clients. Indicates the client In the Loss on a single sample express function, This indicates that the sample in the client output belongs to the first... The probability of class; if the first Each sample belongs to category ,but ;otherwise, Clients whose predictions are closer to the true values ​​will receive higher weights, thus playing a dominant role in training. Finally, the first [model] used to train the global model Comprehensive knowledge of individual samples Defined as: It integrates the weights of multiple clients.

4. The heterogeneous edge computing multi-teacher federated distillation method based on course learning according to claim 1, characterized in that, Step S3 specifically includes: Following the principles of course learning, parameters are adjusted backward based on the gradient of each round of model updates. The difficulty is increased when the accuracy improves and decreased when the accuracy decreases, thus dynamically adjusting the training of the model. During global model training, the update operation is based on the global model loss. The process is represented as: in, This represents the output of the global model. This represents the comprehensive knowledge obtained through weighted summation in step S2. express function; The weights representing the KL divergence loss are used to balance the two loss values; It is a hyperparameter used to adjust the difficulty of training, and it is dynamically adjusted according to the model's performance. The value of is determined to ensure that as model performance improves, the difficulty increases, promoting deeper knowledge acquisition; conversely, when performance declines, the difficulty is appropriately reduced to maintain the stability and robustness of the training process and prevent further performance degradation. Introducing a new parameter adjustment module global model The goal is to minimize ,and Conversely, they attempt to increase the difficulty of training; therefore, and The parameter update method is as follows: It is more sensitive to the direction of updates; even small changes can have a profound impact. A warm-up period should be established, and additional parameters should be introduced. To control right The extent of the impact should be considered to prevent excessive early impact; therefore, The update process is as follows: Initial reduction right The influence of inaccurate knowledge is gradually increased as model performance improves, which helps prevent misleading information and ensures that the impact of inaccurate information is minimized. In the early stages of training, the influence is gradually increased using cosine or linear increment strategies. The value of ; when the model performance reaches a stable state. It will eventually become fixed; In obtaining Output Afterwards, Scale to an appropriate range and ensure it is non-negative: in and They represent Minimum and maximum values Make Determine based on the current training situation and maintain it. Within the range.

Images