A heterogeneous federated learning framework and method based on multi-knowledge distillation fusion

Abstract

This invention relates to the field of machine learning technology, specifically to a heterogeneous federated learning framework and method based on multi-knowledge distillation fusion. The framework includes: a client using local data to compute local knowledge; a server receiving class-level knowledge uploaded by the client and weighted aggregating all client-level class-level knowledge through a class-level knowledge aggregation module to generate global class-level knowledge; the server distributing the global knowledge to each client as teacher knowledge for subsequent client model training through a knowledge distribution module; and the client receiving the global knowledge and performing temperature-adaptive knowledge distillation through a knowledge distillation module to maximize the effectiveness of teacher-student knowledge transfer. This invention effectively alleviates many challenges faced by traditional federated learning frameworks when dealing with heterogeneous devices and data environments by proposing and integrating two different knowledge distillation methods: temperature-adaptive knowledge distillation and batch sample category-related knowledge distillation.

Description

Technical Field

[0001] This invention relates to the fields of federated learning and knowledge distillation technology, specifically to a heterogeneous federated learning framework and method based on multi-knowledge distillation fusion. Background Technology

[0002] Federated learning is a distributed machine learning approach designed to collaboratively train a model among multiple participants while protecting data privacy. Traditional centralized machine learning requires data to be aggregated onto a central server for training, which presents challenges in terms of data privacy, security, and data transmission costs. Federated learning effectively protects data privacy by training the model locally on each participant, sharing only model parameters or gradient information, thus avoiding the direct transmission of raw data. However, federated learning faces the challenge of heterogeneity in devices and data in practical applications. Different clients may have different computing capabilities, network bandwidth, and data distributions, making it difficult for a unified model architecture to adapt to the needs of all clients. Furthermore, the non-independent and identically distributed nature of the data can affect the model's convergence speed and accuracy. To address these issues, knowledge distillation has been introduced into federated learning. Knowledge distillation is a method for model compression and acceleration, transferring knowledge from a complex model (the teacher model) to a simpler model (the student model), allowing the student model to achieve performance similar to the teacher model while maintaining a smaller scale. In federated learning, knowledge distillation can be used to aggregate knowledge from different clients on the server side and deliver it to each client, thereby achieving model personalization and adaptability. However, existing heterogeneous federated learning methods, when employing knowledge distillation techniques, often lack sufficient consideration for client heterogeneity and data diversity, resulting in poor model performance in practical applications. Therefore, a heterogeneous federated learning method is urgently needed to improve the model's adaptability and performance in heterogeneous environments.

[0003] CN116227624A discloses a federated knowledge distillation method and system for heterogeneous models, aiming to address the problems of non-independent and identically distributed data and heterogeneous models. By introducing self-supervised learning of the domain classifier to extract domain-specific data from open datasets, and combining knowledge distillation from the global and local models, the model is iteratively updated round by round to optimize the performance of the global and local models in heterogeneous environments. This method weakens the dependence on open datasets and improves the robustness and generalization ability of the model by utilizing intermediate layer features and domain-specific data. However, this method has high complexity in the design and training of the domain classifier and the global model, and requires significant computational and communication resources.

[0004] CN118153666A discloses a personalized federated knowledge distillation model construction method (Fed-PKD). Addressing the heterogeneity of clients and data in the Internet of Things (IoT), it designs a federated learning framework combining personalized model construction, a second-normative weight aggregation algorithm, and a federated knowledge distillation strategy to improve model generalization ability, accelerate model convergence, and reduce communication overhead. However, this method requires a public dataset, and the quality of the selected public dataset directly affects the effectiveness of knowledge distillation. If client data is highly private or lacks high-quality public data samples, knowledge transfer may fail.

[0005] CN116629376A discloses a federated learning aggregation method and system based on data-free distillation (FedDTG). Its core lies in utilizing distributed generative adversarial networks (GANs) and knowledge distillation techniques to address the issues of traditional federated learning methods, such as lack of support for model heterogeneity, privacy leakage, and dependence on public datasets. However, this method relies on generator training, and the quality of the generator is crucial to the entire method. When data is scarce or highly unevenly distributed, generator training may fail to achieve ideal results, directly impacting distillation quality. Secondly, computational and communication overhead increases. The client needs to perform three-way adversarial training and upload generator and discriminator parameters multiple times, potentially leading to high computational and communication costs, especially on resource-constrained devices. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a heterogeneous federated learning framework and method based on multi-knowledge distillation fusion.

[0007] The technical solution adopted to solve the above technical problems is: a heterogeneous federated learning framework based on multi-knowledge distillation fusion, including: an edge server and at least one client device;

[0008] Each client k∈{1,2,3,…K} has its local dataset, which is expressed as follows:

[0009]

[0010] And there is |D k There are | samples, each with D data dimensions, belonging to one of C different classes. Due to different user behaviors, the local training and testing datasets of the clients are not independent and identically distributed. The personalized model parameters of client k are denoted as . Where d k This represents the number of parameters in the client-side model k. There is system heterogeneity among client devices, and the model size deployed on each client varies, for example, d. l ≠d m , Each client k has a local optimization objective that depends on its corresponding local data distribution. The overall objective is to minimize the expected objective of all clients, which is expressed as:

[0011]

[0012] The edge server is configured with: a granular knowledge aggregation module, used to receive granular knowledge from multiple clients and generate global granular knowledge; and a knowledge distribution module, used to distribute the global granular knowledge to each client. The client is configured with: a local knowledge calculation module, used to calculate local granular knowledge based on local data; and a knowledge distillation module, used to perform knowledge distillation based on the received global and local granular knowledge.

[0013] The technical solution adopted to solve the above-mentioned technical problems is: a heterogeneous federated learning method based on multi-knowledge distillation fusion, which is applicable to the aforementioned heterogeneous federated learning framework based on multi-knowledge distillation fusion, including:

[0014] Step 1: The client uses local data to calculate local knowledge. For example, the first client input is... This represents the sample data of the u-th batch size from client 1, obtained by predicting labels. and real label y u The error between them is used to calculate the cross-entropy loss L. ce Meanwhile, each client, through its local knowledge computation module, calculates local class-level granular knowledge based on local data according to the formula. Calculate and aggregate into a kind of granular knowledge;

[0015] Step 2: The server receives the class-level granularity knowledge uploaded by the client, and generates global class-level granularity knowledge by weighting and aggregating all client class-level granularity knowledge through the class-level granularity knowledge aggregation module.

[0016] Step 3: The server distributes global knowledge through the knowledge distribution module. This knowledge is distributed to each client as teacher knowledge for training the client-side model described later.

[0017] Step 4: The client receives global knowledge and performs temperature-adaptive knowledge distillation through the knowledge distillation module to maximize the effect of knowledge transfer between teachers and students;

[0018] Step 5: The client receives global knowledge and performs batch sample correlation knowledge distillation through the knowledge distillation module. By introducing batch-level sample correlation distillation loss, the model's excessive bias towards specific samples or categories is suppressed, and the model's understanding of the overall characteristics of the data is strengthened.

[0019] Step 6: The client updates the parameters of the local model and the temperature prediction model based on the knowledge distillation loss and cross-entropy loss.

[0020] Preferably, in step 2, a knowledge aggregation scheme that considers differences in knowledge clarity is adopted, specifically including the following steps:

[0021] Step 2.1: Calculate the clarity of knowledge for both the teacher and student, using a logarithmic summation function to quantify the smoothness of the output, expressed as clarity. Assume... Where C represents the probability of the sample belonging to a category, and the clarity is defined as follows:

[0022]

[0023] Step 2.2: Assign different aggregation weights based on the clarity of the knowledge differentiation. Specifically, the definition of knowledge weight aggregation for class j is as follows:

[0024]

[0025] Among them, Sharpness j This represents the sum of clarity for all knowledge belonging to category j. This represents global knowledge of class j. This represents the set of clients that own samples of type j.

[0026] Preferably, the temperature-adaptive knowledge distillation in step 4 specifically includes the following steps:

[0027] Step 4.1: For each client, an independent lightweight model is used to learn a dynamic temperature prediction module θ. k At the start of training, the parameters of the temperature prediction module model are first optimized in reverse to maximize the distillation loss between students and teachers.

[0028] Step 4.2: Input the student logits and teacher logits into the temperature prediction module to predict the distillation temperature T suitable for the current sample. The prediction process can be represented as:

[0029]

[0030] in, For the temperature prediction module, a relational mapping is used to ensure the predicted temperature value T. pred Within a reasonable range, the predicted values ​​are mapped to a preset temperature range [T] using an activation function. start ,T end The specific formula is as follows:

[0031] T pred =T start +T end δ(T pred )

[0032] Where T start and T end The set temperature range is δ(·), which is the activation function, and its purpose is to map the model predictions to the range between 0 and 1.

[0033] Step 4.3: Standardize the teacher's and student's knowledge to ensure that the student model can learn knowledge more effectively rather than mechanically imitate it, thereby reducing distillation loss in cases of accurate prediction and making the model's performance more balanced across all categories during the learning process, according to the formula. The knowledge obtained from teachers and students after standardization Let Z be the average value of the knowledge of students or teachers, and σ(Z) be the corresponding standard deviation.

[0034] Step 4.4: Based on the standardized clarity of knowledge between teachers and students, obtain the teacher-student differentiated distillation temperature. Specifically, the clarity sharp is calculated as follows:

[0035]

[0036] Where Z represents the student's or teacher's knowledge, knowledge with relatively high clarity should generally have a higher temperature compared to student knowledge, while knowledge with lower clarity should have a lower temperature. The distillation temperature between teachers and students is dynamically adjusted for each sample to better reflect the sample's characteristics. This adaptive adjustment mechanism compensates for the lack of sample-level characteristic consideration in standardization, making knowledge transfer more targeted. The distillation temperature of teachers and students is adjusted in the following ways:

[0037]

[0038]

[0039] Where T s and T t Differential temperature used for knowledge distillation and T s ≠T t This step further enables personalized knowledge transfer at the sample level by adaptively generating appropriate temperature values ​​for each sample and category, fine-tuning the temperature based on sample characteristics during the knowledge transfer process, and maximizing the absorption of teacher knowledge by the student model.

[0040] Step 4.5: Calculate the temperature at which each element c∈[C] in the knowledge vector based on the teacher-student differentiated temperature at temperature T. sOr T t The transformation mapping under the action, taking the knowledge output by the student model as an example, shows that each element c∈[C] in the knowledge vector at temperature T s The transformation mapping under the action is:

[0041]

[0042] Where C represents the possible categories to which the sample may belong;

[0043] Step 4.6: Based on the knowledge of the teachers and students, at temperature T under differentiated distillation temperatures. s and T t Transformation mapping under action and The temperature-adaptive knowledge distillation loss is calculated using the following formula:

[0044]

[0045] Preferably, in step 5, the knowledge distillation of intra-batch sample correlation specifically includes the following steps:

[0046] Step 5.1: Standardize the performance of each sample in a batch within a single class. This ensures that the model considers the influence of other samples while predicting individual samples, preventing excessive bias towards certain samples or classes. Formally, the normalized logits values ​​of the teacher and student models for sample i in class c∈[C] are expressed as:

[0047]

[0048] Where B is the batch size and T is the set distillation temperature. and These are the logits values ​​of teacher knowledge and student knowledge for sample i in category c, respectively.

[0049] Step 5.2: Using the normalized knowledge in the category direction as input, the batch sample correlation knowledge distillation calculation method is as follows:

[0050]

[0051] In-batch class normalization introduces an implicit regularization effect, preventing the model from overfitting to predictions of individual samples during training. In the loss function of in-batch class normalization, the student model's output... It is the logits normalization function for all samples in the batch on class c, therefore the loss function is related to the student model logits. The gradient is:

[0052]

[0053] Using the chain rule, the formula Taking the partial derivative, we can obtain the gradient formula:

[0054]

[0055] The gradient of sample i with class c depends not only on its own probability distribution but also on the probability distributions of other samples in the batch. Let represent Kronecker delta, which is 1 when i = b and 0 otherwise. This gradient formula shows that intra-batch class normalization introduces mutual constraints between samples within the batch, thus limiting the gradient update magnitude of a single sample.

[0056] Preferably, step 6 specifically includes the following steps:

[0057] Step 6.1: Based on the temperature-adaptive knowledge distillation method, obtain the temperature-adaptive knowledge distillation loss L. TAKD ;

[0058] Step 6.2: Based on the knowledge distillation method for the correlation of samples within the batch, obtain the temperature-adaptive knowledge distillation loss L. CRKD ;

[0059] Step 6.3: Combine the two loss functions with the cross-entropy loss and perform a weighted sum to form the final distillation loss function:

[0060] Loss=λ·L ce +α·L TAKD +β·L CRKD

[0061] Where λ, α, and β are the weight parameters of the three loss components, respectively. The overall fusion strategy improves the robustness and generalization ability of the model by balancing the contributions of each loss function.

[0062] Step 6.4: The client calculates the total loss (Loss) relative to the local model parameters W using the backpropagation algorithm. k and temperature prediction module parameter θ k gradient Then, the model parameters are updated using the stochastic gradient descent optimization algorithm, as shown in the following formula:

[0063]

[0064] Where η is the learning rate of the local model, η T The learning rate for the parameters of the temperature prediction module. and These are the gradients of the total loss function with respect to the local model parameters and the temperature prediction module parameters, respectively.

[0065] The beneficial effects of this invention are as follows: By proposing and integrating two different knowledge distillation methods, namely temperature-adaptive knowledge distillation and batch sample category association knowledge distillation, this invention achieves higher accuracy with minimal communication overhead, effectively alleviating many challenges faced by traditional federated learning frameworks when dealing with heterogeneous devices and heterogeneous data environments, especially in cases of large differences in device performance, high communication overhead, and uneven data distribution, where the accuracy and efficiency of client model training are significantly limited. Attached Figure Description

[0066] Figure 1 This is a diagram of the heterogeneous federated learning framework for multi-knowledge distillation and fusion in this invention.

[0067] Figure 2 This is a schematic diagram of a temperature-adaptive knowledge distillation method under the heterogeneous federated learning framework of multi-knowledge distillation fusion according to the present invention.

[0068] Figure 3 This is a schematic diagram of the batch sample category association knowledge distillation method under the heterogeneous federated learning framework of the present invention, which integrates multi-knowledge distillation and fusion.

[0069] Figure 4 This is a comparison chart of the effects of the heterogeneous federated learning method based on multi-knowledge distillation and fusion of the present invention with other federated learning methods. Detailed Implementation

[0070] Example 1, as Figure 1 As shown, the present invention proposes a heterogeneous federated learning framework based on multi-knowledge distillation fusion, comprising: an edge server and at least one client device.

[0071] Each client k∈{1,2,3,…K} has its local dataset, where the dataset is expressed as follows:

[0072]

[0073] And there is |D k There are | samples, each with D data dimensions, belonging to one of C different classes. Due to different user behaviors, the local training and testing datasets of the clients are not independent and identically distributed. The personalized model parameters of client k are denoted as . Where d k This represents the number of parameters in the client-side model k. There is system heterogeneity among client devices, and the model size deployed on each client varies, for example, d. l ≠d m , Each client k has a local optimization objective that depends on its corresponding local data distribution. The overall objective is to minimize the expected objective of all clients, which is expressed as:

[0074]

[0075] The edge server is configured with: a class-level granular knowledge aggregation module, used to receive class-level granular knowledge from multiple clients and generate global class-level granular knowledge; and a knowledge distribution module, used to distribute global class-level granular knowledge to each client. The client is configured with: a local knowledge calculation module, used to calculate local class-level granular knowledge based on local data; and a knowledge distillation module, used to perform knowledge distillation based on the received global class-level granular knowledge and local class-level granular knowledge.

[0076] Example 2, as Figures 2-3 As shown, this invention proposes a heterogeneous federated learning method based on multi-knowledge distillation fusion, which is applicable to a heterogeneous federated learning framework based on multi-knowledge distillation fusion, including:

[0077] Step 1: The client uses local data to calculate local knowledge. For example, the first client input is... This represents the sample data of the u-th batch size from client 1, obtained by predicting labels. and real label y u The error between them is used to calculate the cross-entropy loss L. ce Meanwhile, each client, through its local knowledge computation module, calculates local class-level granular knowledge based on local data according to the formula. Calculate and aggregate into a kind of granular knowledge;

[0078] Step 2: The server receives the class-level granularity knowledge uploaded by the client, and generates global class-level granularity knowledge by weighting and aggregating all client class-level granularity knowledge through the class-level granularity knowledge aggregation module.

[0079] Step 3: The server distributes global knowledge through the knowledge distribution module. It is distributed to each client as teacher knowledge for subsequent client model training;

[0080] Step 4: The client receives global knowledge and performs temperature-adaptive knowledge distillation through the knowledge distillation module to maximize the effect of knowledge transfer between teachers and students;

[0081] Step 5: The client receives global knowledge and performs batch sample correlation knowledge distillation through the knowledge distillation module. By introducing batch-level sample correlation distillation loss, the model's excessive bias towards specific samples or categories is suppressed, and the model's understanding of the overall characteristics of the data is strengthened.

[0082] Step 6: The client updates the parameters of the local model and the temperature prediction model based on the knowledge distillation loss and cross-entropy loss.

[0083] In this embodiment, step 2 adopts a knowledge aggregation scheme that considers differences in knowledge clarity, specifically including the following steps:

[0084] Step 2.1: Calculate the clarity of knowledge for both teachers and students, using a logarithmic summation function to quantify the smoothness of the output, expressed as clarity. Assume... Where C represents the probability of the sample belonging to a category, and the clarity is defined as follows:

[0085]

[0086] Step 2.2: Assign different aggregation weights based on the clarity of knowledge differentiation. Specifically, the definition of knowledge weight aggregation for class j is as follows:

[0087]

[0088] Among them, Sharpness j This represents the sum of clarity for all knowledge belonging to category j. This represents global knowledge of class j. This represents the set of clients that own samples of type j.

[0089] It should be noted that, in an optional embodiment, the temperature-adaptive knowledge distillation in step 4 specifically includes the following steps:

[0090] Step 4.1: For each client, use an independent lightweight model to learn a dynamic temperature prediction module θ. k At the start of training, the parameters of the temperature prediction module are first optimized in reverse to maximize the distillation loss between the student and the teacher.

[0091] Step 4.2: Input the student logits and teacher logits into the temperature prediction module to predict the distillation temperature T suitable for the current sample. The prediction process can be represented as:

[0092]

[0093] in, For the temperature prediction module, a relational mapping is used to ensure the predicted temperature value T. pred Within a reasonable range, the predicted values ​​are mapped to a preset temperature range [T] using an activation function. start ,T end The specific formula is as follows:

[0094] T pred =T start +T end δ(T pred )

[0095] Where T start and T end The set temperature range is δ(·), which is the activation function, and its purpose is to map the model predictions to the range between 0 and 1.

[0096] Step 4.3: Standardize both teacher and student knowledge to ensure the student model learns knowledge more effectively rather than mechanically imitating it, thereby reducing distillation loss in cases of accurate prediction and making the model perform more evenly across all categories during the learning process, according to the formula. The knowledge obtained from teachers and students after standardization Let Z be the average value of the knowledge of students or teachers, and σ(Z) be the corresponding standard deviation.

[0097] Step 4.4: Based on the standardized clarity of knowledge between teachers and students, obtain the teacher-student differentiated distillation temperature. Specifically, the clarity sharp is calculated as follows:

[0098]

[0099] Here, Z represents student or teacher knowledge. Compared to student knowledge, knowledge with relatively high clarity should generally have a higher temperature, while knowledge with lower clarity should have a lower temperature. The distillation temperature between teachers and students is dynamically adjusted for each sample to better reflect the characteristics of the sample. This adaptive adjustment mechanism compensates for the lack of sample-level characteristic consideration in standardization, making knowledge transfer more targeted. The distillation temperature of teachers and students is adjusted in the following ways:

[0100]

[0101] Where T s and T t Differential temperature used for knowledge distillation and T s ≠T t This step further enables personalized knowledge transfer at the sample level by adaptively generating appropriate temperature values ​​for each sample and category, fine-tuning the temperature based on sample characteristics during the knowledge transfer process, and maximizing the absorption of teacher knowledge by the student model.

[0102] Step 4.5: Calculate the temperature at which each element c∈[C] in the knowledge vector is at temperature T based on the teacher-student temperature differences. s Or T t The transformation mapping under the action, taking the knowledge output by the student model as an example, shows that each element c∈[C] in the knowledge vector at temperature Ts The transformation mapping under the action is:

[0103]

[0104] Where C represents the possible categories to which the sample may belong;

[0105] Step 4.6: Based on the knowledge of teachers and students, at temperature T under differentiated distillation temperatures. s and T t Transformation mapping under action and The temperature-adaptive knowledge distillation loss is calculated using the following formula:

[0106]

[0107] It should be noted that, in an optional embodiment, step 5, a knowledge distillation of intra-batch sample correlation specifically includes the following steps:

[0108] Step 5.1: Standardize the performance of each sample in a batch within a single class. This ensures that the model considers the influence of other samples while predicting individual samples, preventing excessive bias towards certain samples or classes. Formally, the normalized logits values ​​of the teacher and student models for sample i in class c∈[C] are expressed as:

[0109]

[0110] Where B is the batch size and T is the set distillation temperature. and These are the logits values ​​of teacher knowledge and student knowledge for sample i in category c, respectively.

[0111] Step 5.2: Using the knowledge normalized in the category direction as input, the knowledge distillation calculation method for the intra-batch sample correlation is as follows:

[0112]

[0113] In-batch class normalization introduces an implicit regularization effect, preventing the model from overfitting to predictions of individual samples during training. In the loss function of in-batch class normalization, the student model's output... It is the logits normalization function for all samples in the batch on class c, therefore the loss function is related to the student model logits. The gradient is:

[0114]

[0115] Using the chain rule, the formula Taking the partial derivative, we can obtain the gradient formula:

[0116]

[0117] The gradient of sample i with class c depends not only on its own probability distribution but also on the probability distributions of other samples in the batch. Let represent Kronecker delta, which is 1 when i = b and 0 otherwise. This gradient formula shows that intra-batch class normalization introduces mutual constraints between samples within the batch, thus limiting the gradient update magnitude of a single sample.

[0118] It should be noted that, in an optional embodiment, step 6 specifically includes the following steps:

[0119] Step 6.1: Obtain the temperature-adaptive knowledge distillation loss L based on the temperature-adaptive knowledge distillation method. TAKD ;

[0120] Step 6.2: Based on the knowledge distillation method of in-batch sample correlation, obtain the temperature-adaptive knowledge distillation loss L. CRKD ;

[0121] Step 6.3: Combine the two loss functions with the cross-entropy loss and perform a weighted sum to form the final distillation loss function:

[0122] Loss=λ·L ce +α·L TAKD +β·L CRKD

[0123] Where λ, α, and β are the weight parameters of the three loss components, the overall fusion strategy improves the robustness and generalization ability of the model by balancing the contributions of each loss function.

[0124] Step 6.4: The client calculates the total loss (Loss) relative to the local model parameters W using the backpropagation algorithm. k and temperature prediction module parameter θ k gradient Then, the model parameters are updated using the stochastic gradient descent optimization algorithm, as shown in the following formula:

[0125]

[0126] Where η is the learning rate of the local model, η T The learning rate for the parameters of the temperature prediction module. and These are the gradients of the total loss function with respect to the local model parameters and the temperature prediction module parameters, respectively.

[0127] To verify the effectiveness of the proposed method, this invention conducted comparative experiments with other federated learning methods:

[0128] 1) FD: A federated distillation method based on a granular knowledge interaction system;

[0129] 2) FedCache: A federated learning algorithm for knowledge caching based on sample granularity;

[0130] 3) FedMkd: This invention discloses a heterogeneous federated learning method based on multi-knowledge distillation fusion;

[0131] Figure 4 Experimental results comparing the aforementioned federated learning algorithms on the MNIST dataset are presented. Similar conditions were set for each method in the experiments, such as the types of heterogeneous models, the number of clients, and the degree of data heterogeneity, to ensure fairness. The results show that the convergence curve of the proposed method is much steeper than FedCache's, similar to FD, but FedMkd achieves a higher MAUA of 89.02%, while reducing communication overhead by an order of magnitude compared to FedCache.

[0132] This invention proposes and integrates two different knowledge distillation methods: temperature-adaptive knowledge distillation and batch sample category association knowledge distillation. It achieves higher accuracy with minimal communication overhead, effectively alleviating many challenges faced by traditional federated learning frameworks when dealing with heterogeneous devices and heterogeneous data environments. In particular, it addresses the problem that the accuracy and efficiency of client model training are significantly limited when there are large differences in device performance, high communication overhead, and uneven data distribution.

[0133] It should be noted that the embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A heterogeneous federated learning method based on multi-knowledge distillation fusion, characterized in that, Applied to an edge server and at least one client device, comprising: Step 1: Client computes local knowledge using local data, where the first client input is The sample data of the u-th Batchsize of Client 1 is represented by the error between the predicted label and the real label y u , the cross-entropy loss L ce is calculated, and each client calculates the local class granularity knowledge based on the local data according to the formula computing and aggregating into a kind of class granularity knowledge, wherein W k is the personalized model parameter of the client k; Step 2: The server receives the class granularity knowledge uploaded by the client, and aggregates all client class granularity knowledge by class granularity knowledge aggregation to generate global class granularity knowledge; Step 3: The server distributes the global knowledge through knowledge distribution to each client as teacher knowledge for the training of the subsequent client models. Step 4: The client receives the global knowledge, and performs temperature adaptive knowledge distillation through knowledge distillation to maximize the effect of teacher-student knowledge transmission; Step 5: The client receives the global knowledge, and performs batch sample relevance knowledge distillation through knowledge distillation to suppress the excessive bias of the model to specific samples or categories and strengthen the understanding of the model to the overall characteristics of the data by introducing batch-level sample relevance distillation loss; Step 6: The client updates the local model and temperature prediction model parameters according to the knowledge distillation loss and cross-entropy loss.

2. The heterogeneous federated learning method based on multi-knowledge distillation fusion according to claim 1, characterized in that, In the step 2, a knowledge aggregation scheme considering knowledge clarity difference is adopted, specifically including the following steps: Step 2.1: Calculate the distinctness of the knowledge of the teacher and student respectively, using the log-sum function to quantify the smoothness of the output, denoted as distinctness, where where c represents the likelihood of the sample belonging to a class, and distinctness is defined as follows: Step 2.2: According to the clarity of the knowledge difference, different aggregation weights are given, specifically, the definition of the knowledge weight aggregation for j classes is as follows: where Sharpness j represents the total sharpness of all knowledge belonging to class j, represents the global knowledge of class j, represents the set of clients that possess a sample of class j.

3. The heterogeneous federated learning method based on multi-knowledge distillation fusion according to claim 1, characterized in that, The temperature adaptive knowledge distillation in step 4 specifically includes the following steps: Step 4.1: Learn a dynamic temperature prediction module θ for each client with a separate lightweight model k The training starts by first optimizing the temperature prediction module model parameters in a backward direction, aiming to maximize the distillation loss between the student and the teacher. Step 4.2: The student logits and teacher logits are input into the temperature prediction module to predict the distillation temperature T that adapts to the current sample, and the prediction process is represented as: Wherein, The relationship mapping of the temperature prediction module is to ensure that the predicted temperature value T pred In a reasonable range, the predicted value is mapped to the preset temperature range [T start , T end ] by the activation function, and the specific formula is as follows: T pred = T start + T end δ(T pred ) where T start and T end are the temperature ranges set, and δ(·) is an activation function whose purpose is to map the model prediction values between 0 and 1. Step 4.3: Standardize the teacher and student knowledge to ensure that the student model can learn the knowledge more effectively rather than mechanically imitating, thereby reducing the distillation loss in the case of accurate prediction, making the model's performance more balanced for all classes in the learning process, according to the formula Get the standardized knowledge of teacher and student after standardization, is the average value of student or teacher knowledge, and σ(Z) is the corresponding standard deviation; Step 4.4: According to the clarity of the standardized teacher and student knowledge, the teacher-student difference distillation temperature is obtained, specifically, the calculation method of the clarity sharp is as follows: Wherein, Z is the student or teacher knowledge, the teacher knowledge and the student knowledge, the knowledge with relatively large clarity should have a higher temperature, and the knowledge with small clarity has a low temperature, and the distillation temperature between the teacher and the student is dynamically adjusted for each sample to reflect the characteristics of the sample, and the distillation temperature between the teacher and the student is adjusted through the following way to compensate for the defects of the standardization lacking sample-level characteristics consideration: where T s and T t Differentiated temperature for knowledge distillation and T s ≠ T t This step further forms the personalized migration of knowledge at the sample level, by adaptively generating appropriate temperature values for each sample and class, the temperature is fine-tuned for sample characteristics in the knowledge transfer process, maximizing the absorption of student models to teacher knowledge; Step 4.5: According to the teacher-student differentiable temperature calculation, each element c∈[C] in the knowledge vector is transformed and mapped under the action of temperature T s or T t , and the transformed and mapped result is: each element c∈[C] in the knowledge vector is transformed and mapped under the action of temperature T s , and the transformed and mapped result is: Wherein, c is the category that the sample may belong to; Step 4.6: The teacher-student knowledge at the difference distillation temperature under the temperature T s and T t under the action of the transformation mapping and calculates its temperature adaptive knowledge distillation loss, and the calculation formula is as follows:

4. The heterogeneous federated learning method based on multi-knowledge distillation fusion according to claim 1, characterized in that, In step 5, the knowledge distillation of the batch sample relevance specifically includes the following steps: Step 5.1: Quantify the performance of each sample in a class within a batch to prevent the model from generating excessive bias to certain specific samples or categories while considering the prediction of a single sample, and the normalized logits value of the teacher and student model on sample i in class c∈[C] is represented as: where B is batch size, T is the set distillation temperature, and are logits values for teacher and student knowledge, respectively, for sample i on class c. Step 5.2: The knowledge normalized in the class direction is input, and the batch sample relevance knowledge distillation calculation method is as follows: Batch-wise class normalization introduces an implicit regularization effect, such that the model does not overfit to the prediction of a single sample during training. In the loss function with batch-wise class normalization, the output of the student model is normalized by the logits of all samples in the batch at class c, so the gradient of the loss function with respect to the student logits is: By the chain rule, the partial derivative of the formula with respect to x is given by the gradient formula The gradient of sample i on class c depends not only on its own probability distribution, but also on the probability distributions of other samples in the batch, denoted by the Kronecker function ib which equals 1 when i = b and 0 otherwise. This gradient formula shows that the class normalization within the batch introduces mutual constraints between samples in the batch, thus limiting the magnitude of the gradient update for a single sample.

5. The heterogeneous federated learning method based on multi-knowledge distillation fusion according to claim 1, characterized in that, The step 6 specifically includes the following steps: Step 6.1: Obtain the temperature-adaptive knowledge distillation loss L according to the temperature-adaptive knowledge distillation method TAKD ; Step 6.2: Obtain the temperature-adaptive knowledge distillation loss L according to the knowledge distillation method of sample correlation within batch CRKD ; Step 6.3: Combine the two loss functions and the cross-entropy loss to form the final distillation loss function: Loss = λ · L ce + α · L TAKD + β · L CRKD Wherein, λ, α, β are weight parameters of the three loss components, and the overall fusion strategy balances the contribution of each loss function to improve the robustness and generalization ability of the model; Step 6.4: The client computes the gradient of the total loss Loss with respect to the local model parameters W by backpropagation algorithm k and the temperature prediction module parameters θ k The gradient of the total loss Loss with respect to the local model parameters W and the temperature prediction module parameters θ is computed by backpropagation algorithm. The model parameters are updated using a stochastic gradient descent optimization algorithm, as follows: where η is the learning rate of the local model, η T is the learning rate of the temperature prediction module parameters, and are the gradients of the total loss function with respect to the local model parameters and the temperature prediction module parameters, respectively.

Images