Diffusion distillation and self-distillation integrated optimization method and device based on federal learning
Through the integrated optimization method of diffusion distillation and self-distillation, the data heterogeneity and model coordination problems in federated learning are solved, and the personalization and generalization performance of the local model is balanced, which improves the robustness and accuracy of the model.
Patent Information
- Application Number
- CN202510461614.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-08-01
AI Technical Summary
Existing federated learning methods are difficult to coordinate effectively when facing the problem of data heterogeneity and the balance between global models and local models' generalization capabilities, especially in the absence of public data sets, the application of knowledge distillation is limited.
The integrated optimization method of diffusion distillation and self-distillation is adopted to distillate global feature knowledge into the local model through the diffusion model, and historical personalized knowledge is retained through the self-distillation mechanism, and the local model is optimized by combining diffusion distillation loss and self-distillation loss.
Effectively alleviate the impact of data heterogeneity, enhance the personalization and global adaptability of local models, support diversified client collaboration, and improve the robustness and accuracy of the model.
Smart Images
Figure CN120409618A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of federated learning, and particularly relates to an integrated optimization method and device for diffusion distillation and self-distillation based on federated learning. Background Art
[0002] With the rapid development of the digital age, artificial intelligence (AI) has gradually penetrated into fields such as healthcare, finance, smart homes, and autonomous driving, bringing a series of revolutionary changes. By analyzing and processing massive amounts of data, AI not only excels in improving decision-making efficiency but also demonstrates great potential in various application scenarios such as personalized services and predictive analytics. However, with the widespread application of AI technology, privacy issues have become a challenge that urgently needs to be addressed. Especially globally, the implementation of privacy protection laws such as GDPR and CCPA has further intensified the focus on how to effectively train models while protecting data privacy.
[0003] In the process of uploading data to a central server for processing using traditional centralized model training methods, there is a risk of sensitive data leakage, and this method cannot meet the cross-institutional and cross-regional data collaboration requirements, often restricting the application of machine learning algorithms. Therefore, how to achieve effective collaboration between multiple data sources without sacrificing data privacy has become the focus of current academic and industrial research. Federated Learning (FL), as an emerging distributed machine learning framework, provides an effective solution to this problem. By training models on local clients and only sharing model parameters instead of raw data, federated learning can achieve collaborative training of multi-party data while ensuring data privacy. This method breaks through the limitations of traditional centralized learning and is widely applied in scenarios with high data privacy requirements, such as the fields of finance, healthcare, and smart devices. However, there are still some key problems in federated learning, especially the heterogeneity problem of client data. Due to the differences in data distributions among different clients, the "non-independent and identically distributed" (Non-IID) phenomenon occurs during model training, resulting in impaired performance of the global model and increasing the complexity of coordinating the global model with local models of each client.
[0004] To address these challenges, Personalized Federated Learning (pFL) has gradually become a research hotspot. By combining the advantages of global and local data, pFL aims to customize a model for each client to improve its performance on specific tasks. Existing pFL methods can be roughly divided into two categories: one is the model-based method, which trains a global model and fine-tunes or adapts the model for each client; the other is the representation-based method, which uses compact representations such as class prototypes to optimize the local model, so as to better match the distribution of local data. These methods effectively improve the local performance of the model, but there are still certain challenges, especially in terms of how to balance the generalization ability of the personalized model and the global model.
[0005] In addition, with the introduction of technologies such as Knowledge Distillation (KD), researchers have tried to use the global model for knowledge transfer in federated learning to improve the performance of personalized models. For example, through knowledge distillation, the client can obtain the soft labels of the global model from the server, thus accelerating the personalized process and improving the performance of the local model. However, existing federated learning methods based on knowledge distillation usually rely on sharing a common dataset, which is not practical in many real-world applications. Therefore, how to effectively utilize knowledge distillation to improve model personalization and global performance without a common dataset has become an important research topic in this field. Summary of the Invention
[0006] Aiming at the problems that existing federated learning technologies cannot overcome data heterogeneity and balance the generalization ability of the global model and the local model, the present invention proposes an integrated optimization method and system for diffusion distillation and self-distillation based on federated learning. On the one hand, the global feature knowledge is distilled into the local model through a diffusion model, and on the other hand, the historical personalized knowledge is retained through a self-distillation mechanism, which can improve the performance of the local model in data heterogeneous scenarios and balance personalization and generalization performance.
[0007] In a first aspect, the present invention provides an integrated optimization method for diffusion distillation and self-distillation based on federated learning, including:
[0008] Step 1: The client receives the first global model sent by the server and initializes the local model based on the parameters of the first global model to obtain the first local model;
[0009] Step 2: The client receives the first global feature sent by the server and uses the first global feature to train a diffusion model;
[0010] Step 3: The client extracts a first local feature from the first local model, aligns the first local feature, and inputs it into the diffusion model to obtain a second local feature. The diffusion distillation loss is calculated based on the difference between the second local feature and the first global feature.
[0011] Step 4: The client trains the first local model using the diffusion distillation loss to obtain a second local model, and performs self-distillation using the second local model from the previous round to calculate the self-distillation loss;
[0012] Step 5: The client calculates the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and trains the first local model based on the total loss of the local model to obtain a third local model;
[0013] Step 6: Upload the third local model and corresponding local features of each client to the server, and the server performs global aggregation to update the first global model, obtain the second global model, and distribute the second global model and the corresponding second global features to each of the clients for the next round of federated learning training until the first local model converges.
[0014] Furthermore, in step 2, the neural network is optimized by minimizing the diffusion loss function to obtain a diffusion model; the neural network is used to predict the difference between the noise and the initial features, and the minimized diffusion loss function is shown in formula (1):
[0015]
[0016] Where t represents the time step, t∈{0,1,…,T}, represents a neural network, ∈ t represents the prediction noise, represents minimizing diffusion loss, x t Represents the noisy feature at time step t, which is composed of the original feature x0 and the predicted noise ∈ t Linear composition.
[0017] Furthermore, the step 3 specifically includes:
[0018] Step 301: The client extracts corresponding first local features from the first local model, and inputs the first local features into an adaptive noise alignment network for noise alignment;
[0019] Step 302: inputting the aligned first local features into the diffusion model to obtain second local features;
[0020] Step 303: Calculate the difference between the second local feature and the first global feature using the mean square error function to obtain the diffusion distillation loss for feature distillation.
[0021] Further, in step 301, the adaptive noise alignment network is used to align the first local feature, and the first local feature is aligned with the noise level of the initial noise feature at time step T, and the processing process is shown in formula (2):
[0022]
[0023] where, represents the aligned first local feature, represents the first local feature, ∈ T represents Gaussian noise, and G represents the noise weight.
[0024] Further, the diffusion distillation loss is calculated by formula (3):
[0025]
[0026] where, represents the diffusion distillation loss, Z (tea) represents the first global feature, represents the second local feature.
[0027] Further, self-distillation is performed using the second local model of the previous round, specifically including:
[0028] Using the second local model of the previous round as the teacher model and the second local model of the current round as the student model for knowledge distillation.
[0029] Further, the classifier of the current local model generates a probability distribution p s , while the distributor of the teacher model generates a probability distribution p t ,
[0030] Correspondingly, the calculation formula of the self-distillation loss is as follows:
[0031]
[0032] where, represents the self-distillation loss, T temp represents the temperature parameter for smoothing the probability distribution, and KL represents the KL divergence.
[0033] Further, the server performs global aggregation to update the first global model, including: the server aggregates each client in a weighted average manner to update the first global model, and the process is shown in formula (6):
[0034]
[0035] Among them, |D m | represents the size of the local dataset of client m, K represents the aggregation weight normalization factor, k represents the number of clients participating in this round of training, and W R+1 represents the second global model, represents the local model of client m after the R-th round of training.
[0036] Furthermore, the second global feature update rule includes: for the local features uploaded by each client, the server performs weighted averaging based on the weights of each type of feature to generate global knowledge features, as specifically shown in formula (7):
[0037]
[0038] Among them, represents the average feature representation of client m for category n in round R, and p m represents the feature weight, which is calculated by formula (8):
[0039]
[0040] Among them, and N n respectively represent the number of samples of category n in client m and all clients.
[0041] Second, the present invention provides a diffusion distillation and self-distillation integrated optimization device based on federated learning, including:
[0042] A local model generation unit, configured to receive the first global model sent by the server by the client, and initialize the local model based on the parameters of the first global model to obtain the first local model;
[0043] A diffusion model generation unit, configured to receive the first global feature sent by the server by the client, and train the diffusion model using the first global feature;
[0044] A diffusion distillation calculation unit, configured to extract the first local feature from the first local model by the client, align the first local feature and input it into the diffusion model to obtain the second local feature, and calculate the diffusion distillation loss based on the difference between the second local feature and the first global feature;
[0045] A self-distillation calculation unit, configured to train the first local model using the diffusion distillation loss by the client to obtain the second local model, and perform self-distillation using the second local model of the previous round to calculate the self-distillation loss;
[0046] A local model update unit, which is used for the client to calculate the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and train the first local model based on the total loss of the local model to obtain a third local model;
[0047] A global model update unit, which is used to upload the third local model and the corresponding local features of each client to the server. The server performs global aggregation to update the first global model to obtain a second global model, and distributes the second global model and the corresponding second global features to each client for the next round of federated learning training until the local model converges.
[0048] The beneficial effects of the present invention are as follows:
[0049] (1) The present invention can alleviate the impact of data heterogeneity: By introducing a denoising mechanism based on a diffusion model, the client can effectively align local features with global knowledge, reducing the challenges brought by data heterogeneity. The difference between the denoised local features and the global features is significantly reduced, improving the collaborative ability of the local model and the global model, and enabling the system to train more efficiently in a heterogeneous data environment.
[0050] (2) The present invention can enhance the personalization of the local model and its global adaptability: By introducing a self-distillation mechanism, the local model of the previous round is used as a teacher model to guide the training of the current round. This method not only retains the personalized features of the local model, but also gradually enhances its adaptability to the global data distribution, effectively balancing the personalization and generalization performance, and improving the robustness and accuracy of the model.
[0051] (3) The present invention can support the collaboration of diverse clients: The denoising and self-distillation knowledge transfer method based on a diffusion model proposed by the present invention supports different numbers and types of clients to participate in federated learning collaboration. Whether it is clients with similar data distributions or highly heterogeneous data clients, they can all perform efficient collaborative training through this system to meet the needs of different application scenarios. Description of the Drawings
[0052] Figure 1 It is a schematic diagram of the architecture of an integrated optimization method of diffusion distillation and self-distillation based on federated learning provided by an embodiment of the present invention;
[0053] Figure 2 It is a schematic diagram of the process of an integrated optimization method of diffusion distillation and self-distillation based on federated learning provided by an embodiment of the present invention;
[0054] Figure 3 It is a schematic diagram of the comparison of local accuracy and global accuracy of different algorithms when there are 10 clients provided by an embodiment of the present invention;
[0055] Figure 4 Schematic diagram of the comparison of local accuracy and global accuracy of different algorithms for 20 clients provided by the embodiments of the present invention;
[0056] Figure 5 Schematic diagram of the comparison of local accuracy and global accuracy of different algorithms for 50 clients provided by the embodiments of the present invention. Detailed implementation manners
[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0058] An integrated optimization method of diffusion distillation and self-distillation based on federated learning provided by the embodiments of the present invention is directed to the data heterogeneity scenario in federated learning. In this scenario, there is at least one server and multiple clients. The server is communicatively connected to the multiple clients. Different clients adopt a homogeneous model but have heterogeneous data distributions. Among them, the server maintains a global model and extracts global features to distribute to each client. The client executes a dual knowledge distillation strategy during model training: on the one hand, distill the global feature knowledge into the local model through a diffusion model, and on the other hand, retain the historical personalized knowledge through a self-distillation mechanism. In addition, the system allows different numbers of clients to participate in federated training. The specific architecture is as Figure 1 shown, and the working process is specifically as Figure 2 shown, including:
[0059] Step 1: The client receives the first global model sent by the server and initializes the local model based on the parameters of the first global model to obtain the first local model.
[0060] Each client receives the global model W R sent by the server and initializes the local model with the global model parameters where R represents the training round and m represents the client number.
[0061] Step 2: The client receives the first global feature sent by the server and trains the diffusion model using the first global feature.
[0062] The client receives the first global feature Z (tea) sent by the server and trains a diffusion model using the first global feature. Specifically, by minimizing the diffusion loss to optimize the neural network to obtain a diffusion model; the neural network is designed to predict the noise ∈ t (where ), and the difference between the initial feature x0. In the embodiments of the present invention, the neural network includes two 3×3 Bottleneck blocks. The minimization of the loss function is shown in formula (1):
[0063]
[0064] where t represents the time step, t ∈ {0, 1, …, T}, represents the neural network, ∈ t represents the predicted noise, represents the minimization of the diffusion loss, x t represents the noisy feature at time step t, which is linearly composed of the original feature x0 and the predicted noise ∈ t During the training process, it should be noted that the global feature Z of the diffusion model used for training (tea) remains independent and does not receive any backward gradients from the diffusion model, thus ensuring the integrity of the global feature and the isolation of the training process.
[0065] Step 3: The client extracts the first local feature from the first local model, aligns the first local feature and then inputs it into the diffusion model to obtain the second local feature, and calculates the diffusion distillation loss based on the difference between the second local feature and the first global feature.
[0066] The client can extract the corresponding first local feature from the first local model, horizontally align the first local feature with the noise of the initial noise feature at time step T, and input the aligned first local feature into the diffusion model obtained by the above training for denoising to obtain the second local feature. Knowledge distillation is performed using the first global feature and the second local feature, and the diffusion distillation loss is calculated based on the difference between the two.
[0067] In some embodiments, Step 3 includes the following sub-steps S301 to S30:
[0068] Step S301: The client extracts the corresponding first local feature from the first local model and inputs the first local feature into the Adaptive Noise Alignment Network (ANN) for noise alignment.
[0069] Here, the Adaptive Noise Alignment Network ANN is designed to learn the weight G that combines the output of the first local feature with Gaussian noise, and horizontally align the first local feature with the noise level of the initial noise feature at time step T. The processing process of the first local feature is shown in formula (2):
[0070]
[0071] Among them, represents the first local feature after alignment, represents the first local feature, ∈ T represents Gaussian noise, G represents the noise weight, that is, the noise scale factor predicted from the student features through the adaptive noise network, and its value dynamically controls the mixing ratio of the original feature and the Gaussian noise ∈ T of the mixture ratio.
[0072] Step S302: Input the aligned first local feature into the diffusion model to obtain the second local feature
[0073] Step S303: Use the mean square error function MES to calculate the difference between the second local feature and the first global feature to obtain the diffusion distillation loss for feature distillation. The diffusion distillation loss is calculated by formula (3):
[0074]
[0075] Among them, represents the diffusion distillation loss, Z (tea) represents the first global feature, represents the second local feature.
[0076] The client trains the diffusion model by receiving global knowledge from the server, thereby denoising the local features. This denoising process effectively reduces the difference between the denoised local features and the global features, enhancing the collaborative ability of the local model and the global model.
[0077] Step 4: The client uses the diffusion distillation loss to train the first local model to obtain the second local model, and uses the second local model of the previous round for self-distillation to calculate the self-distillation loss.
[0078] The client uses the diffusion distillation loss to train the first local model to obtain the second local model and stores it as the historical local model. This model is used to retain personalized historical knowledge and as a reference for the next round of training, while only retaining it locally on the client to ensure privacy. During the local training process of the next round, the client uses the local model stored in the previous round as the teacher model to guide the training of the second local model in the current round.
[0079] Specifically, use the second local model of the previous round as the teacher model and the second local model of the current round as the student model for knowledge distillation. The classifier of the current local model generates a probability distribution of p s, the classifier of the teacher model generates a probability distribution of p t , the difference between the two is optimized through the self-distillation loss function, as shown in formula (4) specifically:
[0080]
[0081] Among them, represents the self-distillation loss, T temp represents the temperature parameter for smoothing the probability distribution, and KL represents the KL (Kullback-Leibler) divergence, which is used to measure the difference between two probability distributions p s and p t .
[0082] During the training process of the client's model, the self-distillation mechanism is introduced, and the self-distillation loss function is used to transfer historical personalized knowledge to the local model of the current round, so as to ensure that the local model gradually improves its adaptability to the global data distribution while maintaining personalized performance. In this way, the client model achieves a better balance between personalized and generalization performance, thus enhancing the robustness and accuracy of the model.
[0083] Step 5: The client calculates the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and trains the first local model based on the total loss of the local model to obtain the third local model.
[0084] Specifically, the total loss of the local model is calculated by formula (5):
[0085]
[0086] Among them, represents the cross-entropy loss, represents the self-distillation loss, represents the diffusion loss for training the diffusion model, represents the feature distillation loss.
[0087] Step 6: Upload the third local model and the corresponding local features of each client to the server. The server performs global aggregation to update the first global model to obtain the second global model, and distributes the second global model and the corresponding second global features to each client for the next round of federated learning training until the first local model converges.
[0088] Specifically, the server aggregates each client by weighted average to update the first global model, and the process is as shown in formula (6):
[0089]
[0090] Among them, |D m| represents the size of the local dataset of client m; K represents the aggregation weight normalization factor, usually the sum of the sizes of all client datasets; k represents the number of clients participating in this round of training, and W R+1 represents the second global model, represents the local model of client m after the R-th round of training.
[0091] The second global feature update rule includes: the server performs weighted averaging on the local features uploaded by each client based on the weights of each type of feature to generate global knowledge features. Specifically, as shown in formula (7):
[0092]
[0093] where, represents the average feature representation of client m for category n in round R, and p m represents the feature weight, that is, the number of samples of category n of client m accounting for the total number N of samples of this category of all clients n The proportion is calculated by formula (8):
[0094]
[0095] where, and N n respectively represent the number of samples of category n of client m and all clients, p m objectively quantifies the contribution degree of each client to the global feature representation of category n.
[0096] The embodiment of the present invention proposes a joint optimization method for local and global models. During the training process of the client, the local model is jointly optimized with the global knowledge through the denoised features. And by synchronously optimizing the local personalized model and the global model, the synergy between the two in performance is ensured, thereby further improving the overall performance of the federated learning system in a heterogeneous data environment.
[0097] The embodiment of the present invention also provides a diffusion distillation and self-distillation integrated optimization device based on federated learning, including:
[0098] A local model generation unit, configured to receive, by the client, the first global model sent by the server, and initialize the local model based on the parameters of the first global model to obtain the first local model.
[0099] A diffusion model generation unit, configured to receive, by the client, the first global feature sent by the server, and train the diffusion model using the first global feature.
[0100] The diffusion distillation calculation unit is used for the client to extract the first local feature from the first local model, align the first local feature and then input it into the diffusion model to obtain the second local feature, and calculate the diffusion distillation loss based on the difference between the second local feature and the first global feature.
[0101] The self-distillation calculation unit is used for the client to train the first local model using the diffusion distillation loss to obtain the second local model, and perform self-distillation using the second local model of the previous round to calculate the self-distillation loss.
[0102] The local model update unit is used for the client to calculate the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and train the first local model based on the total loss of the local model to obtain the third local model.
[0103] The global model update unit is used to upload the third local model and the corresponding local feature of each client to the server. The server performs global aggregation to update the first global model to obtain the second global model, and distributes the second global model and the corresponding second global feature to each client for the next round of federated learning training until the first local model converges.
[0104] Embodiments of the present invention use the common CIFAR-100 image classification dataset to conduct experiments to test the performance of the local model and the global model trained by the method proposed by the present invention. The dataset uses the partitioning method of Dirichlet distribution with different concentrations. The lower the Dirichlet concentration, the more unbalanced the data distribution and the stronger the data heterogeneity. The experimental results are as Figures 3 to 5 shown. Under different numbers of clients (10, 20, and 50), the present invention achieves the highest model accuracy compared with 7 baseline methods.
[0105] Table 1 Comparison table of local accuracy and global accuracy of different algorithms under different Dirichlet concentrations (10 clients)
[0106]
[0107]
[0108] As shown in Table 1, using the common CIFAR-100 image classification dataset to conduct experiments, in the case of 10 clients, comparing the local accuracy and global accuracy of different algorithms under different Dirichlet concentrations, the method provided by the present invention achieves the highest model accuracy compared with 7 baseline methods.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. An integrated optimization method of diffusion distillation and self-distillation based on federated learning, characterized in that, Including: Step 1: The client receives the first global model sent by the server and initializes the local model based on the parameters of the first global model to obtain the first local model; Step 2: The client receives the first global feature sent by the server and trains the diffusion model using the first global feature; Step 3: The client extracts the first local feature from the first local model, aligns the first local feature and then inputs it into the diffusion model to obtain the second local feature, and calculates the diffusion distillation loss based on the difference between the second local feature and the first global feature; Step 4: The client trains the first local model using the diffusion distillation loss to obtain the second local model, and performs self-distillation using the second local model of the previous round to calculate the self-distillation loss; Step 5: The client calculates the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and trains the first local model based on the total loss of the local model to obtain the third local model; Step 6: Upload the third local model and the corresponding local feature of each client to the server. The server performs global aggregation to update the first global model to obtain the second global model, and distributes the second global model and the corresponding second global feature to each client for the next round of federated learning training until the first local model converges.
2. The integrated optimization method of diffusion distillation and self-distillation based on federated learning according to claim 1, wherein In Step 2, the neural network is optimized by minimizing the diffusion loss function to obtain the diffusion model; the neural network is used to predict the difference between the noise and the initial feature, and the minimized diffusion loss function is shown in Formula (1): where \(t\) represents the time step, \(t\in\{0,1,\ldots,T\}\), denotes the neural network, \(\in\) t represents the predicted noise, denotes minimizing the diffusion loss, \(x\) t represents the noisy feature at time step \(t\), which is linearly composed of the original feature \(x_0\) and the predicted noise \(\in\) t linearly.
3. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 1, characterized in that, The specific content of Step 3 includes: Step 301: The client extracts the corresponding first local feature from the first local model and inputs the first local feature into the adaptive noise alignment network for noise alignment; Step 302: Input the aligned first local feature into the diffusion model to obtain the second local feature; Step 303: Use the mean square error function to calculate the difference between the second local feature and the first global feature to obtain the diffusion distillation loss for feature distillation.
4. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 1, characterized in that, In step 301, the first local feature is aligned using an adaptive noise alignment network, and the first local feature is aligned with the noise level of the initial noise feature at time step T, and the processing process is as shown in formula (2): Among them, represents the first local feature after alignment, represents the first local feature, ∈ T represents Gaussian noise, and G represents the noise weight.
5. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 1, characterized in that The diffusion distillation loss is calculated by Formula (3): Among them, represents the diffusion distillation loss, Z (tea) represents the first global feature, represents the second local feature.
6. The integrated optimization method of diffusion distillation and self-distillation based on federated learning according to claim 1, wherein Performing self-distillation using the second local model of the previous round specifically includes: Using the second local model of the previous round as the teacher model and the second local model of the current round as the student model for knowledge distillation.
7. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 6, characterized in that, The classifier of the current local model generates a probability distribution p s , while the distributor of the teacher model generates a probability distribution p t , Correspondingly, the calculation formula of the self-distillation loss is as follows: Among them, represents the self-distillation loss, and T temp represents the temperature parameter for smoothing the probability distribution, and KL represents the KL divergence.
8. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 1, characterized in that When the server performs global aggregation to update the first global model, it includes: The server aggregates each client in a weighted average manner to update the first global model, and the process is shown in Formula (6): Among them, |D m | represents the size of the local dataset of client m, K represents the aggregation weight normalization factor, k represents the number of clients participating in this round of training, and W R+1 represents the second global model, represents the local model of client m after the R-th round of training.
9. A diffusion distillation and self-distillation integrated optimization method based on federated learning according to claim 1, characterized in that, The update rule of the second global feature includes: For the local features uploaded by each client, the server performs weighted average based on the weight of each type of feature to generate the global knowledge feature, specifically as shown in Formula (7): Among them, represents the average feature representation of client m for category n in round R, p m represents the feature weight, which is calculated by formula (8): Among them, and N n respectively represent the number of samples of category n in client m and all clients.
10. An integrated optimization device for diffusion distillation and self-distillation based on federated learning, characterized in that, Including: A local model generation unit, configured to enable the client to receive the first global model sent by the server and initialize the local model based on the parameters of the first global model to obtain the first local model; A diffusion model generation unit, which is used for the client to receive the first global feature sent by the server and train a diffusion model by using the first global feature; A diffusion distillation calculation unit, which is used for the client to extract a first local feature from the first local model, input the aligned first local feature into the diffusion model to obtain a second local feature, and calculate a diffusion distillation loss based on the difference between the second local feature and the first global feature; A self-distillation calculation unit, which is used for the client to train the first local model by using the diffusion distillation loss to obtain a second local model, and perform self-distillation by using the second local model of the previous round to calculate a self-distillation loss; A local model update unit, which is used for the client to calculate the total loss of the local model based on the self-distillation loss and the diffusion distillation loss, and train the first local model based on the total loss of the local model to obtain a third local model; A global model update unit, which is used to upload the third local model of each client and the corresponding local feature to the server. The server performs global aggregation to update the first global model to obtain a second global model, and distributes the second global model and the corresponding second global feature to each client for the next round of federated learning training until the local model converges.