Personalized federal learning method based on knowledge fusion distillation and storage medium

Abstract

The invention discloses a personalized federated learning method based on knowledge fusion distillation and a storage medium, and belongs to the technical field of personalized federated learning, and the method comprises the steps: a server carries out the pre-training and distributes a diffusion model to each client, and generates a local synthesis data set, initializing a global model as a student model of a client to perform knowledge fusion distillation training, and guiding the student model training by the teacher model optimized in the last round; and after training is completed, the client uploads student model parameters to the server for federal aggregation to update the global model, and meanwhile, the personalized feature extraction capability and reliability of the next round of teacher model are enhanced by using synthetic data, so that a closed-loop learning framework of global cooperation and local personalized collaborative optimization is formed. According to the method, the problems of client model drifting, performance attenuation and convergence rate slowing caused by data heterogeneity can be solved.

Description

Technical Field

[0001] This invention belongs to the field of personalized federated learning technology, specifically relating to a personalized federated learning method and storage medium based on knowledge fusion distillation. Background Technology

[0002] In recent years, the rapid advancement of computer technology has provided strong hardware support for the development of artificial intelligence. The exponential growth in computing power has made it possible to process massive amounts of data and complex models, while continuous breakthroughs in storage technology have laid the foundation for the storage and management of large-scale data. The widespread adoption of the internet and the upgrading of mobile communication technology have made data transmission and sharing more convenient and efficient, significantly increasing the speed of information flow globally. In centralized training models, all computing tasks are concentrated on a central server, placing enormous computational pressure on the server and making it prone to insufficient computing resources, resulting in slow model training. Moreover, the central server becomes a single point of failure in the entire system; if the central server fails, the entire intelligent application system may be paralyzed and unable to operate normally. Faced with the many problems of centralized training scenarios, there is an urgent need for a new machine learning method that can fully utilize data resources distributed across different entities while protecting data privacy and security, achieving efficient and reliable model training. Against this backdrop, federated learning has emerged.

[0003] Federated learning is a promising and efficient distributed learning paradigm that demonstrates unique advantages when data cannot be shared at the edge due to privacy and storage issues. This method trains a global model collaboratively through distributed clients without transmitting the original data. As a solution that balances privacy protection and communication efficiency, federated learning has shown significant application value in fields such as smart healthcare, industrial IoT, and natural language processing. However, real-world applications often exhibit non-independent and identically distributed (Non-IID) data distribution characteristics, posing key challenges to traditional federated learning methods, such as performance degradation and slowed convergence rates due to client model drift. To address the Non-IID challenge, some existing research obtains shared models with strong generalization capabilities by calibrating the model's update direction. However, a single global model often cannot meet the personalized needs of clients, resulting in poor performance and slow convergence.

[0004] Personalized federated learning has been proposed as a novel solution to address the aforementioned problems. It aims to customize a personalized model for each client to excel in a specific task. Compared to traditional federated learning, personalized federated learning enables local models to better adapt to local data distributions and balances shared knowledge with local-specific knowledge to improve model performance. Some methods enhance the personalized performance of local models by improving the collaboration between clients. Other methods utilize additional data to adapt the model to the client's data distribution.

[0005] While these methods can improve the performance of personalized local models, they also have two limitations: First, the practice of introducing external shared datasets conflicts with the core principle of federated learning to protect data privacy, and the validity of the generated data is also difficult to guarantee in Non-IID scenarios. The other limitation is that existing methods rely too much on global knowledge or external data, while ignoring the personalized knowledge accumulated by the local model in historical training. This knowledge imbalance directly makes it difficult for the local model to achieve the optimal balance between personalized adaptation and generalization ability, ultimately restricting the overall performance. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a personalized federated learning method and storage medium based on knowledge fusion distillation, which solves the problems of client model drift, performance degradation and slowed convergence rate caused by data heterogeneity.

[0007] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0008] In a first aspect, the present invention provides a personalized federated learning method based on knowledge fusion distillation, comprising:

[0009] S1: The server uses a pre-built federated learning framework to pre-train a diffusion model and distributes it to each client; each client generates a synthetic dataset based on its local original dataset and the received diffusion model.

[0010] S2: The server initializes the global model and distributes it to each client. Each client initializes the student model with the received global model and trains it locally on the original dataset through the knowledge fusion distillation mechanism. The teacher model, which was adjusted by the teacher model in the previous round, guides the training of the student model.

[0011] S3: After training is complete, each client uploads the student model parameters to the server for federated aggregation and updates the global model; at the same time, the synthetic dataset is used to make the next round of adjustments to the teacher model to improve the teacher model's personalized feature extraction capability and reliability.

[0012] In the aforementioned personalized federated learning method based on knowledge fusion distillation, step S1 involves a federated learning framework comprising a server and multiple clients. The server includes a global dataset, and each client includes a local dataset with a limited amount of non-IID data. The diffusion model is a noise estimation model that supports category-conditional generation, composed of UNet network models. The pre-trained diffusion model includes: the server training the diffusion model using the global dataset, enabling it to generate data of the corresponding category under the guidance of guiding conditions.

[0013] The aforementioned personalized federated learning method based on knowledge fusion distillation, wherein the server trains the diffusion model using a global dataset, includes:

[0014] The time step t and the category y are defined as guiding conditions, which are connected to the intermediate modules of UNet through the mapping of the cross attention layer and are used as part of the diffusion model, so that the noise estimation model composed of the UNet network model supports category-conditional generation.

[0015] The training objective function for the diffusion model is:

[0016]

[0017] In the formula, E represents the expected value; It is Gaussian noise; It is an image with noise at time step t. , Indicates the preset maximum time step; It is noise estimated by UNet; It is a squared L2 norm operation.

[0018] In the aforementioned personalized federated learning method based on knowledge fusion distillation, step S1 involves each client generating a synthetic dataset based on its local original dataset and the received diffusion model, including:

[0019] Two initial noise samples are taken from a standard Gaussian distribution. and ;

[0020] Linear interpolation technique is used to reduce initial noise and Generate random noise The noise interpolation operation is as follows:

[0021]

[0022] In the formula, This represents the preset interpolation scaling factor;

[0023] Each client will generate random noise. The input is fed into the received diffusion model, and samples are generated using denoising diffusion implicit sampling techniques.

[0024] Based on the distribution characteristics of the original local dataset, the generated samples are filtered, and high-quality samples with matching distributions are retained to generate a synthetic dataset.

[0025] In the aforementioned personalized federated learning method based on knowledge fusion distillation, in step S2, the objective function trained locally... for:

[0026]

[0027] In the formula, This represents the student model for this round; This represents the teacher model for this round; q() represents the prediction result; It is the contribution rate of knowledge distillation; Represents cross-entropy loss; This represents the Kullback-Leibler divergence loss;

[0028] The student model is updated using the gradient descent mechanism of the locally trained objective function. The update process is represented as follows:

[0029]

[0030] In the formula, This indicates the preset learning rate.

[0031] In the aforementioned personalized federated learning method based on knowledge fusion distillation, step S3, the calculation formula for updating the global model by the server is as follows:

[0032] ,

[0033] In the formula, Represents the original dataset of client i; This represents the student model uploaded by client i.

[0034] In the aforementioned personalized federated learning method based on knowledge fusion distillation, step S3, where each client uses a synthetic dataset to perform the next round of adjustments to the teacher model, includes:

[0035] Initialize the teacher model to the student model after training in step S2;

[0036] Based on the objective function of the teacher model Using a synthetic dataset, the initialized teacher model is augmented and trained. The objective function of the teacher model is... for:

[0037]

[0038] In the formula, This represents the cross-entropy loss.

[0039] The enhanced teacher model is trained through the objective function. The gradient descent mechanism is used for updating, and the update process is represented as follows:

[0040]

[0041] In the formula, This indicates the preset learning rate.

[0042] In the aforementioned personalized federated learning method based on knowledge fusion distillation, step S3, where each client uses the synthetic dataset to adjust the teacher model for the next round, also includes: the client verifying the reliability of the updated teacher model on the original dataset to determine the teacher model for the next round. This process includes:

[0043] Calculate the initialized teacher model separately and the updated teacher model The accuracy of predictions on the original dataset, and the initialized teacher model. Results accuracy The calculation formula is:

[0044]

[0045] In the formula, Represents the original dataset of client i; It is an indicator function, representing the prediction result of the initialized teacher model. Compared with the true categories on the original dataset If they match, the indicator function outputs 1; if they do not match, the indicator function outputs 0. It is the original data from the original dataset; It is the raw data. The corresponding real category;

[0046] Updated teacher model Results accuracy The calculation formula is:

[0047]

[0048] Based on the accuracy of the calculation results and Determine the teacher model for the next round. The calculation formula is:

[0049] .

[0050] The aforementioned personalized federated learning method based on knowledge fusion distillation also includes step S4: distributing the updated global model to each client, and each client performing the next round of local training, repeating steps S2 to S3 until the global model converges.

[0051] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0052] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0053] This invention is based on a personalized federated learning method using knowledge fusion distillation. It generates synthetic data through a diffusion model to alleviate data heterogeneity, and uses knowledge distillation to fuse global and personalized knowledge to avoid model forgetting. While updating the global model through federated aggregation, it enhances the personalized guidance capability of the teacher model, achieving a balance between global collaboration and local personalization. This improves the convergence speed and local performance of the model under non-independent and identically distributed (Non-IID) data, and solves the problems of client model drift, performance degradation, and slowed convergence rate caused by data heterogeneity.

[0054] This mainly includes the following two points:

[0055] 1) This invention designs a data synthesis method based on a diffusion model. It uses a sampling method based on a denoising diffusion implicit model to accelerate the data synthesis process and ensure the generation of high-quality data. By generating a customized synthetic dataset locally, data privacy is protected, and the adaptation problem of local models under Non-IID can be solved.

[0056] 2) A personalized learning strategy based on knowledge fusion distillation is proposed. This strategy can accurately correct the training direction of the local model to guide local training, thus solving the model bias problem caused by Non-IID. Simultaneously, synthetic data is used for augmented training, combined with model reliability assessment, to obtain richer feature information, thereby improving the personalized performance of the model.

[0057] The personalized federated learning method based on knowledge fusion distillation of this invention is specifically divided into three stages:

[0058] In the first stage, the server pre-trains a conditional diffusion model using a public dataset. This model uses the UNet architecture to predict noise and introduces a class conditional mechanism to generate directional data. Its training objective is to minimize the mean squared error loss function of noise prediction. After pre-training, the server distributes the model to each client. The client uses a denoising diffusion implicit sampling method to improve the efficiency of synthetic samples and combines it with normalized linear interpolation to enhance the diversity of synthetic samples, generating a synthetic dataset that matches the local distribution.

[0059] In the second stage, the server initializes the global model and distributes it to the client. The client performs local training through a knowledge fusion distillation mechanism and uses the teacher model to accurately correct the training direction of the local model to guide local training. The global model is used as the student model, and the teacher model after the previous round of enhancement guides the student model to update through KL divergence loss. At the same time, it combines cross-entropy loss to maintain supervised learning, forming a composite loss function to achieve personalized knowledge retention and global knowledge fusion.

[0060] In the third stage, the client uploads the trained student model parameters to the server for federated aggregation to update the global model. At the same time, it uses synthetic datasets to enhance the training of the teacher model and uses accuracy verification to filter reliable personalized knowledge to ensure the effectiveness of the next round of knowledge fusion distillation.

[0061] This invention's knowledge fusion distillation-based personalized federated learning method addresses the Non-IID distribution problem by generating customized synthetic datasets using a diffusion model. It leverages iterative knowledge distillation to balance global collaborative optimization with local personalization, ultimately significantly improving model performance while protecting data privacy. Experimental results demonstrate that, compared to existing technologies, this knowledge fusion distillation-based personalized federated learning method outperforms existing methods in image classification accuracy tests under various variable conditions. Specifically, the method achieves the highest test accuracy, particularly under highly heterogeneous and data-scarce conditions; client performance is more concentrated and biased towards higher values, reflecting better fairness; and it maintains stable performance across different client numbers, demonstrating stronger scalability. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of a network model for a personalized federated learning method based on knowledge fusion distillation according to Embodiment 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of the pre-training process based on the diffusion model in Embodiment 1 of the present invention;

[0064] Figure 3 This is a comparison of the average accuracy of different methods on CIFAR10 in Embodiment 1 of the present invention;

[0065] Figure 4 This is a comparison of the average accuracy of different methods on CIFAR100 according to Embodiment 1 of the present invention;

[0066] Figure 5 This is a comparison result of the client test accuracy distribution on CIFAR10 according to Embodiment 1 of the present invention. Detailed Implementation

[0067] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0068] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0069] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention and therefore only show the components relevant to the present invention.

[0070] Example 1:

[0071] This embodiment discloses a personalized federated learning method based on knowledge fusion distillation, including:

[0072] S1: The server uses a pre-built federated learning framework to pre-train a diffusion model and distributes it to each client; each client generates a synthetic dataset based on its local original dataset and the received diffusion model.

[0073] S2: The server initializes the global model and distributes it to each client. Each client initializes the student model with the received global model and trains it locally on the original dataset through the knowledge fusion distillation mechanism. The teacher model, which was adjusted by the teacher model in the previous round, guides the training of the student model.

[0074] S3: After training is complete, each client uploads the student model parameters to the server for federated aggregation and updates the global model; at the same time, the synthetic dataset is used to make the next round of adjustments to the teacher model to improve the teacher model's personalized feature extraction capability and reliability.

[0075] like Figure 1 As shown, in step S1, the federated learning framework consists of one server and N clients, where N is greater than 1. Each local client's dataset has a limited amount of data and is non-independent and identically distributed (N-IID). The server deploys a server dataset. .

[0076] The client has two main functions: 1) During the pre-training phase, client i first obtains the data distribution of the local data, and then downloads the diffusion model parameters from the server. Afterwards, it utilizes the generative capabilities of the diffusion model to simulate the local data distribution, generating a suitable amount of high-quality data as a synthetic dataset. 1) Stored locally. 2) During the formal training phase, client i will utilize the original dataset. and synthetic datasets Local training is performed using synthetic knowledge distillation. Client i stores two classification models locally: a student model and a teacher model. The student model is uploaded to the server after local training is complete, while the teacher model is stored locally.

[0077] Knowledge fusion distillation guides the local training of student models by fusing knowledge from raw and synthetic data. Specifically, the client first downloads the global model to initialize a local student model. This model is then trained on the local raw data under the guidance of the teacher model. After training, the student model parameters are uploaded to the server for federated aggregation. Simultaneously, the local model uses the trained student model to initialize the teacher model. The teacher model then undergoes augmented training using synthetic data with a distribution similar to the original data, resulting in a teacher model that has learned more accurate and personalized knowledge. This knowledge fusion distillation mechanism allows the student model to fully learn the features of the local data while also better adapting to data heterogeneity through the augmented teacher model, ultimately improving the model's personalized performance.

[0078] The server has the following main functions: 1) During the pre-training phase, the server uses the deployed server dataset. Train a diffusion model and then distribute it to each client. The advantage of this approach is that, while ensuring data integrity, each client can utilize the diffusion model's generative capabilities to generate data locally for subsequent training. 2) During the formal training phase, federated aggregation is performed on the student models uploaded by the clients, and the global model is updated; finally, the updated global model is distributed to each client.

[0079] This embodiment presents a framework for a personalized federated learning method based on knowledge fusion distillation, such as... Figure 1 As shown, it includes the following steps:

[0080] This paper first proposes a data synthesis method based on a diffusion model to generate customized synthetic datasets for clients. In federated learning scenarios, data privacy and security are paramount. Due to privacy requirements, clients cannot directly share or access raw data, nor can they directly share or access it with the server. Furthermore, much sensitive data (such as medical records and financial information) is typically stored in non-IID format across various clients, and the amount of data stored locally on each client is often limited. These limitations not only increase the complexity of data distribution in federated learning but may also lead to biases and convergence difficulties during model training, negatively impacting the overall model performance. To address these issues, this invention first proposes introducing customized synthetic datasets for each client, utilizing a diffusion model to generate high-quality data to improve model training effectiveness.

[0081] Specifically, it includes:

[0082] Step 1: In the pre-training phase before federated training begins, the server trains the diffusion model using a public dataset. After pre-training, the diffusion model parameters are distributed to each client. Subsequently, each client uses the diffusion model to generate a synthetic dataset to assist training, tailored to its local data distribution.

[0083] like Figure 2 As shown, the specific steps of the diffusion model pre-training process in step 1 are as follows:

[0084] Step 1.1: Train the diffusion model. For example... Figure 2 As shown, during the pre-training phase, the server uses a sufficient global dataset to train the Training diffusion model. This diffusion model is a noise estimation model composed of UNet network models that supports class-conditional generation. Furthermore, to enable the trained diffusion model to synthesize data as needed, a class-conditional mechanism is added to the original UNet network model, allowing it to generate data of the corresponding class under the guidance of conditions.

[0085] like Figure 2 As shown, firstly, the time step t and class y are defined. Then, these are used as guiding conditions and concatenated into the intermediate module of the UNet network model through a mapping of the cross-attention layer, serving as part of the diffusion model. Therefore, the training objective of the diffusion model can be expressed as a noise prediction loss function in the form of mean squared error (MSE):

[0086] (1)

[0087] In the formula, It's Gaussian noise. , From standard Gaussian distribution Sampling noise; It is an image with noise at time step t. , This indicates the preset maximum time step. In this embodiment, The value is 1000; The noise in the output is estimated using UNet; It is the squared L2 norm operation, which is the numerator of the mean squared error (MSE); E represents the expected value, which is a probability-weighted average of all possible values ​​of the random variable.

[0088] In the specific implementation, the time step t is sinusoidally position-encoded and concatenated with the embedding vector of the class label y. This concatenation is then projected through a linear transformation to become the key and value of the cross-attention layer. The intermediate features of the UNet network model serve as the query, enabling dynamic fusion of conditional information and backbone features. The modified UNet network model structure allows noise estimation to depend simultaneously on the input data, time step, and target class, thereby driving the diffusion model to generate high-quality samples that conform to class constraints.

[0089] like Figure 1 and Figure 2 As shown, after the diffusion model is trained on the server, the server distributes the parameters of the diffusion model to each client (Client 1 to Client N). The trained diffusion model has the ability to generate conditional data, and through the distribution of model parameters, each client obtains a conditional diffusion model that can generate high-quality data while protecting data privacy.

[0090] Step 1.2: Synthesize personalized datasets. Since the data from each client is Non-IID, blindly using a diffusion model to generate data to assist local training will cause the optimization target of the local model to shift during training, thus affecting model performance. Therefore, as... Figure 2 As shown, a synthetic dataset needs to be generated based on the local data distribution to accurately assist the local model in training.

[0091] Since the distribution of the original local dataset is known, the client can directly enter the data synthesis stage after obtaining the diffusion model. To accelerate the data synthesis process, ensure synthesis quality, and reduce synthesis costs, this invention adopts Denoising Diffusion Implicit Sampling (DDIM), which requires significantly fewer steps than traditional diffusion model sampling methods, thus improving sample generation speed. Furthermore, it combines regularized linear interpolation for interpolation, enhancing the diversity of synthesized samples.

[0092] Specifically, two initial noise samples are first taken from a standard Gaussian distribution. and ,

[0093] Next, linear interpolation techniques are used to reduce the initial noise. and Generate random noise The noise interpolation operation is as follows:

[0094] (2)

[0095] In the formula, This represents the preset interpolation scaling factor; in this embodiment, The value is 0.5;

[0096] Then, each client will As random noise input into the receiving diffusion model, samples are generated through denoising diffusion implicit sampling techniques;

[0097] Then, each client will determine the distribution of its local original dataset. The generated samples are filtered, and high-quality samples with matching distribution characteristics are retained to generate a synthetic dataset. It is used for participating in the formal training in the subsequent stages.

[0098] This concludes the pre-training phase, which only needs to be executed once.

[0099]

[0100] After completing the pre-training phase in step 1, the system will enter the formal training phase:

[0101] Step 2: The server initializes the global model and distributes it to the client. The client then trains the model on its local original dataset using the knowledge fusion distillation method and uses the teacher model to guide the local training, thereby calibrating the update direction of the local student model and preventing the student model from shifting during training.

[0102] In this embodiment, the initial global model is a simple convolutional neural network model. The global model is initialized by initializing its parameters. The global model is as follows: Figure 1 The classification model shown is used to perform image classification tasks.

[0103] First, each client downloads the global model from the server for local initialization. However, this direct initialization method has two key problems: First, since the global model is obtained by aggregating client models with heterogeneous data distributions, its performance on each local client is often poor; second, this initialization causes the local model to forget previously learned personalized knowledge, which not only affects the model's performance on the current client but also significantly reduces the training convergence speed. To address these problems, this invention innovatively proposes a personalized learning method based on knowledge fusion distillation. By introducing a knowledge distillation mechanism during the model training phase, it can effectively retain key knowledge learned in the past and fuse personalized knowledge from synthetic data, thereby significantly improving the performance and convergence efficiency of the local model.

[0104] Specifically, in order to preserve personalized historical knowledge, the client-side constructed a teacher model. Teacher Model This is used to extract personalized features and guide local training. In the initial stage of federated learning, the client has not performed any local training, so the initial teacher model directly uses the global model downloaded from the server. At this point, no personalized knowledge has been introduced, and the teacher model is identical to the student model. In the formal training phase, the client first initializes the global model in round t. The model is used as a student model, and then trained locally under the guidance of the teacher model using the local raw dataset. Local training includes both student model training and teacher model training.

[0105] The definition of the local training objective function needs to combine the supervision signal from the task itself (cross-entropy loss) and the knowledge transfer signal provided by the teacher model (KL divergence loss). Therefore, the loss function is defined by combining the supervision loss and distillation loss. The objective function for local training is calculated using the following formula:

[0106] (3)

[0107] In the formula, This represents the student model for this round; This represents the teacher model for this round; q() represents the prediction result; This is the preset contribution rate of knowledge distillation. In this embodiment, The value is 0.3; Represents cross-entropy loss; This represents the Kullback-Leibler divergence loss. The client updates its local weights based on this loss function.

[0108] The student model is updated using the gradient descent mechanism of the locally trained objective function, as follows:

[0109] (4)

[0110] in, This represents the preset learning rate. In this embodiment, The value is 0.01.

[0111] Each client trains locally for a preset number of training rounds before completing the training.

[0112] Then, the client needs to perform two tasks. Task 1: The client uploads the locally trained model parameters to the server and participates in federated aggregation. Task 2: The client initializes the teacher model using the locally trained model parameters, and then further enhances the performance of the personalized teacher model using the synthetic dataset.

[0113] Step 3: After local training is complete, the client uploads its student models to the server, where the server aggregates all models. Next, the client uses a synthetic dataset to enhance the teacher model's training, improving its ability to extract personalized features.

[0114] Step 3.1: Upload the student model for aggregation. The client will upload the parameters of the locally trained student model to the server and participate in federated aggregation. The updated global model is represented as follows:

[0115] (9)

[0116] In the formula, This represents the dataset of client i; Represents the local model of client i;

[0117] Step 3.2, Teacher Model Enhancement Training. The client initializes the teacher model using the parameters of the locally trained student model, and then further enhances the performance of the personalized teacher model using a synthetic dataset. The synthetic dataset obtained through pre-training has a similar data distribution to the original data, thus possessing richer feature information, which can further improve the performance of the personalized model and mitigate the damage to model performance caused by Non-IID data. Therefore, the enhanced teacher model can more accurately guide the student model during local training. The loss function of the personalized teacher model is defined as:

[0118] (5)

[0119] The enhanced teacher model is trained through the objective function. The gradient descent mechanism is used for updating, and the update process is represented as follows:

[0120] (6)

[0121] In this embodiment, The value is 0.01.

[0122] After augmentation training, the model can learn more personalized knowledge to guide the student model in local training. However, the personalized knowledge learned by the model may be more applicable to the local context, which is called positive personalized knowledge; conversely, if it is not applicable to the local data distribution, it is called negative personalized knowledge. Therefore, this method also defines a criterion to ensure that the updated teacher model learns positive personalized knowledge. When the accuracy of the updated teacher model is greater than this criterion, the teacher model is considered reliable and saved for the next round of local training; otherwise, the teacher model without augmentation training is directly stored.

[0123] The initialized teacher model Accuracy of predictions on the original dataset As a discriminant for the updated teacher model Whether it is a reliable standard, the accuracy of the results The calculation process is as follows:

[0124] (7)

[0125] In the formula, This represents the dataset of client i; It is an indicator function, representing the prediction result of the initialized teacher model. Compared with the true categories on the original dataset If they match, the indicator function outputs 1; if they do not match, the indicator function outputs 0. It is the original data from the original dataset; It is the raw data. The corresponding real category;

[0126] Updated teacher model Results accuracy The calculation formula is:

[0127]

[0128] Based on the accuracy of the calculation results and Determine the teacher model for the next round. The calculation formula is:

[0129] (8)

[0130] The determined next round of teacher models They will guide the student models in the next round of local training.

[0131] The personalized federated learning method based on knowledge fusion distillation in this embodiment further includes step S4: distributing the updated global model to each client, and each client performing the next round of local training, and repeating steps S2 to S3 until the global model converges.

[0132] In one specific embodiment, global model convergence determination includes: the global model on the server in the server dataset. The accuracy of the results fluctuated by less than 1% for three consecutive rounds, and the formula for calculating the accuracy of the results was Formula 7.

[0133] To facilitate understanding, Algorithm 2 demonstrates the training process of personalized learning for knowledge fusion distillation in steps S2 and S3.

[0134]

[0135] Based on the above, this invention designs a simulation experiment of a personalized federated learning method based on knowledge fusion distillation in a real-world scenario, using the CIFAR-10 and CIFAR-100 datasets for image classification tasks. Both CIFAR-10 and CIFAR-100 consist of 50,000 training images and 10,000 test images, but CIFAR-10 has only ten categories with 6,000 images per category, while CIFAR-100 has one hundred categories with only 600 images per category. Therefore, the training task for CIFAR-100 is more challenging than that for CIFAR-10. This invention uses the Dirichlet distribution. To configure the data using Non-IID settings, and thereby allocate the dataset, where A higher value indicates a lower degree of data heterogeneity. This invention assumes... The experiment was conducted with a threshold of 0.1, and all datasets were used with 80% as the training set and 20% as the test set.

[0136] The diffusion model used in the experiment is based on the UNet model structure for noise estimation and consists of three main components: encoder, decoder, and output. The encoder contains three downsampling modules, each consisting of two ResNet blocks, a linear attention module, and a downsampling convolution operation. The decoder symmetrically contains three upsampling modules, each consisting of two ResNet blocks, a linear attention module, and an upsampling deconvolution operation. The final output layer consists of a residual block and a convolutional layer. The diffusion model is trained for a fixed number of 1000 steps using the Adam optimizer for 200 iterations. The initial global model of this invention employs a simple convolutional neural network model, consisting of three convolutional layers with 3*3 kernels and two fully connected layers. During the training of the global model for image classification, this method uses an Adam optimizer with a learning rate of 0.01 and a momentum of 0.9.

[0137] The experiment compared our method Ours with five existing federated learning algorithms, including the basic FedAvg, non-IID federated learning algorithms namely FedProx, MOON and pFedSD, and finally FedGen, an algorithm that uses generated data for assisted training.

[0138] The experiment uses the default settings of 20 clients, 100 global communication rounds, 3 local training rounds, and a batch size of 128 for training the dataset. The contribution rate of knowledge distillation is discussed. Set to 0.3. All the baselines mentioned above were implemented in PyTorch, and all experiments were conducted on a device equipped with an NVIDIA GeForce RTX 4090 GPU and 24 GB of GDDR6X memory.

[0139] Table 1 Comparison of accuracy results of different methods

[0140]

[0141] The local test accuracy of all methods is shown in Table 1. As can be seen from the table, with the increase of heterogeneity ( (The accuracy of all federated learning algorithms decreased on both datasets), but the accuracy of this method was still the best, demonstrating its effectiveness.

[0142] As can be observed from the table, the higher the degree of heterogeneity ( The lower the value, the more obvious the advantages of this method become, especially when At that time, our method outperformed the second-best method, pFedSD, by approximately 5.3% on CIFAR10. This is because our method uses synthetic data for auxiliary training, which allows us to better extract features from local data, thereby enhancing the model's adaptability to local data distributions.

[0143] In addition, compared with the FedGen method, which also uses synthetic data, this method uses knowledge fusion distillation, which can effectively solve the training bias problem caused by Non-IID and ensure that the model can fully learn the knowledge of each category.

[0144] Furthermore, in Figure 3 and Figure 4The paper presents the change in Test Accuracy during federated training. Specifically, the proposed method does not perform well in the early stages of training, but its learning efficiency continuously improves as training progresses, achieving its best performance in the later stages and maintaining it until the end. This is because the knowledge that the teacher model can learn is relatively limited in the early stages of training, making it unable to effectively guide the student model in local training. However, as training progresses, the teacher model constructed by this method learns more data features through augmented training, enabling it to better adapt to highly heterogeneous data and continuously improve model performance.

[0145] Due to the complexity of the CIFAR100 dataset, all methods exhibited relatively low accuracy, with smoother accuracy fluctuations. This is because, in the Non-IID setting, each client is missing data for some categories, further increasing the training difficulty. However, our method still maintains the highest accuracy even in challenging scenarios, and its accuracy continues to increase as other methods converge, further demonstrating its superiority.

[0146] Due to data heterogeneity, the personalization performance of different clients may vary significantly. To investigate the fairness of personalization performance improvements across clients, this experiment compared the overall performance distribution of all clients under different degrees of data heterogeneity. Figure 5 The comparison of client test accuracy distributions on CIFAR10 is shown. The horizontal axis represents the range of testing accuracy (0-100), with higher accuracy towards the right. The vertical axis represents the number of clients, under different levels of data heterogeneity. and This method demonstrates superior personalization performance compared to other methods. As shown in the graph, compared to the other two methods, pFedSD and FedProx, the client performance distribution of this method is closer to the right (accuracy increases from left to right). The closer the client distribution is to the right, the better the client's personalization performance. In other words, this method achieves better fairness and enables more clients to achieve higher accuracy.

[0147] Table 2 Comparison of test accuracy results for different numbers of clients

[0148]

[0149] Furthermore, the accuracy of all methods was investigated on the CIFAR10 dataset with varying numbers of clients to validate the scalability of the proposed method. The accuracy of all methods is shown in Table 2. Overall, the accuracy of most methods decreases with the increase of clients. This is because, with the total number of samples remaining constant, the amount of data allocated to each client decreases as the number of clients increases, leading to a more significant bias in the model during local training and thus reducing model performance. However, our proposed method still maintains an accuracy of around 69% with different numbers of clients, indicating that the model using this method has good generalization ability and scalability. Notably, our method exhibits the most significant performance advantage when the number of clients is 40, with an accuracy 4.81% higher than the second-highest FedGen. This further demonstrates that the federated knowledge fusion distillation strategy can effectively mitigate the negative impact of Non-IID.

[0150] Table 3. Test accuracy of different knowledge distillation contribution rates

[0151]

[0152] Finally, the performance of the proposed method was investigated using different knowledge distillation contribution rates, and the results are shown in Table 3. It can be observed that without an appropriate knowledge distillation contribution rate, the model's performance is significantly worse. Therefore, it is speculated that a high knowledge distillation contribution rate allows the knowledge from the synthetic data to dominate, thus misleading the direction of local training and leading to performance degradation. When a low knowledge contribution rate is set, the local model cannot acquire enough knowledge from the synthetic data to aid training, ultimately reducing the model's performance. In summary, by selecting an appropriate knowledge distillation contribution rate, the personalized historical knowledge of the local model and the new knowledge from the synthetic data can be effectively balanced. Therefore, this method uses a specific value for the knowledge distillation contribution rate. The value is set to 0.3 to obtain better model performance.

[0153] Example 2:

[0154] Based on the same inventive concept as Embodiment 1, this embodiment introduces a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A personalized federated learning method based on knowledge fusion distillation, characterized in that, include: S1: The server uses a pre-built federated learning framework to pre-train a diffusion model and distributes it to each client; Each client generates a synthetic dataset based on its local original dataset and the received diffusion model. S2: The server initializes the global model and distributes it to each client. Each client initializes the student model with the received global model and trains it locally on the original dataset through the knowledge fusion distillation mechanism. The teacher model, which was adjusted by the teacher model in the previous round, guides the training of the student model. S3: After training is complete, each client uploads the student model parameters to the server for federated aggregation and updates the global model; at the same time, the synthetic dataset is used to make the next round of adjustments to the teacher model to improve the teacher model's personalized feature extraction capability and reliability.

2. The personalized federated learning method based on knowledge fusion distillation according to claim 1, characterized in that, In step S1, The federated learning framework includes a server and multiple clients. The server includes a global dataset, and each client includes a local dataset with a non-IID and a limited amount of data. The diffusion model is a noise estimation model that supports category-conditional generation, composed of a UNet network model. The pre-trained diffusion model includes: the server trains the diffusion model using a global dataset, enabling it to generate data of the corresponding category under the guidance of guiding conditions.

3. The personalized federated learning method based on knowledge fusion distillation according to claim 2, characterized in that, The server uses a global dataset to train the diffusion model, including: The time step t and the category y are defined as guiding conditions, which are connected to the intermediate modules of UNet through the mapping of the cross attention layer and are used as part of the diffusion model, so that the noise estimation model composed of the UNet network model supports category-conditional generation. The training objective function for the diffusion model is: In the formula, E represents the expected value; It is Gaussian noise; It is an image with noise at time step t. , Indicates the preset maximum time step; It is noise estimated by UNet; It is a squared L2 norm operation.

4. The personalized federated learning method based on knowledge fusion distillation according to claim 3, characterized in that, In step S1, each client generates a synthetic dataset based on its local original dataset and the received diffusion model, including: Two initial noise samples are taken from a standard Gaussian distribution. and ; Linear interpolation technique is used to reduce initial noise and Generate random noise The noise interpolation operation is as follows: In the formula, This represents the preset interpolation scaling factor; Each client will generate random noise. The input is fed into the received diffusion model, and samples are generated using denoising diffusion implicit sampling techniques. Based on the distribution characteristics of the original local dataset, the generated samples are filtered, and high-quality samples that match the distribution characteristics are retained to generate a synthetic dataset.

5. The personalized federated learning method based on knowledge fusion distillation according to claim 1, characterized in that, In step S2, the objective function of local training for: In the formula, This represents the student model for this round; This represents the teacher model for this round; q() represents the prediction result; It is the contribution rate of knowledge distillation; Represents cross-entropy loss; This represents the Kullback-Leibler divergence loss; The student model is updated using the gradient descent mechanism of the locally trained objective function. The update process is represented as follows: In the formula, This indicates the preset learning rate.

6. The personalized federated learning method based on knowledge fusion distillation according to claim 5, characterized in that, In step S3, the calculation formula for updating the global model by the server is as follows: ， In the formula, Represents the original dataset of client i; This represents the student model uploaded by client i.

7. The personalized federated learning method based on knowledge fusion distillation according to claim 3, characterized in that, In step S3, each client uses the synthetic dataset to perform the next round of adjustments to the teacher model, including: Initialize the teacher model to the student model after training in step S2; Based on the objective function of the teacher model Using a synthetic dataset, the initialized teacher model is augmented and trained. The objective function of the teacher model is... for: In the formula, This represents the cross-entropy loss. The enhanced teacher model is trained through the objective function. The gradient descent mechanism is used for updating, and the update process is represented as follows: In the formula, This indicates the preset learning rate.

8. The personalized federated learning method based on knowledge fusion distillation according to claim 7, characterized in that, In step S3, each client uses the synthetic dataset to adjust the teacher model for the next round. This also includes: the client verifying the reliability of the updated teacher model on the original dataset to determine the teacher model for the next round. The process includes: Calculate the initialized teacher model separately and the updated teacher model The accuracy of predictions on the original dataset, and the initialized teacher model. Results accuracy The calculation formula is: In the formula, Represents the original dataset of client i; It is an indicator function, representing the prediction result of the initialized teacher model. Compared with the true categories on the original dataset If they match, the indicator function outputs 1; if they do not match, the indicator function outputs 0. It is the original data from the original dataset; It is the raw data. The corresponding real category; Updated teacher model Results accuracy The calculation formula is: Based on the accuracy of the calculation results and Determine the teacher model for the next round. The calculation formula is: 。 9. The personalized federated learning method based on knowledge fusion distillation according to claim 1, characterized in that, It also includes step S4: the updated global model is distributed to each client, each client performs the next round of local training, and steps S2 to S3 are executed in a loop until the global model converges.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method of claim 1.

Images