Trusted personalized federated learning method and device based on clustering and knowledge distillation

Abstract

This invention belongs to the field of federated learning technology and discloses a trusted personalized federated learning method and apparatus based on clustering and knowledge distillation. First, clustering is performed in cloud collaboration to divide the user terminal into multiple clusters, and then each cluster is locally aggregated to obtain a local model. Second, a method based on cyclic knowledge distillation is designed to sequentially train the local model in a cyclical manner, extracting common knowledge through knowledge distillation. Then, a relearning operation is performed on the global model on the user terminal to effectively recover local knowledge. Furthermore, to address the potential privacy leakage problem during parameter upload, a distributed differential privacy mechanism based on a shuffling algorithm is designed. By adding an extra hybrid encoding step on both the client and server sides, it allows users to achieve a higher level of privacy protection by adding only a small amount of noise. This invention introduces knowledge distillation and shuffling algorithms, improving the framework's security while achieving personalized training.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a reliable personalized federated learning method and apparatus based on clustering and knowledge distillation. Background Technology

[0002] Traditional centralized artificial intelligence technologies require the analysis of massive amounts of data to make inferences and provide feedback. However, smart terminals often only store individual user data, and the data sources needed for machine learning training are scattered from large data centers to numerous terminal devices. Furthermore, these terminal devices contain a large amount of users' private data; uploading this information to data centers raises privacy concerns. Federated learning, a distributed machine learning framework, allows multiple data sources to collaboratively train a model without requiring participants to upload training data. As an emerging machine learning paradigm, federated learning breaks down data silos while ensuring that user data is stored locally, a feature that guarantees the privacy of training data.

[0003] In practical applications, training data across different user terminals is often non-independent and identically distributed. When the data distribution differs significantly between user terminals, directly acquiring knowledge learned from other user terminals would drastically reduce the performance of the user terminal model. Therefore, researchers have proposed a series of solutions to address the problems caused by data heterogeneity.

[0004] In 2022, Long et al. proposed a multi-center aggregation mechanism in their paper "Multi-Center Federated Learning: Clients Clustering for Better Personalization," which uses the parameters of client-side models to cluster data. It learns multiple global models from the data as cluster centers and derives the optimal match between users and centers. This method uses k-means as the clustering algorithm, thus suffering from its drawbacks, such as computational efficiency for high-dimensional data and robustness to outliers. In early 2023, Chen et al. proposed Federated Hyper-Knowledge Distillation in their paper "The Best of BothWorlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation." Clients rely on knowledge distillation to train local models. Each client extracts and sends its local data representation and corresponding soft predictions to the server. The server aggregates this information and broadcasts it to the clients to support local training. However, this method increases training time compared to other models.

[0005] In summary, although federated learning provides a new solution for personalized model training, it still has some shortcomings: (1) Data is often unevenly distributed among devices, and the storage, computing, and communication capabilities of different devices vary. (2) The global model extracts common knowledge from all participants, and direct aggregation in the case of heterogeneous data will reduce the performance of the global model. (3) Most studies on personalized federated learning have not considered privacy issues. Therefore, how to reduce accuracy loss, how to reduce communication and computing overhead, and how to make privacy protection methods effective under a personalized framework are problems that need to be solved. Summary of the Invention

[0006] The purpose of this invention is to propose a reliable personalized federated learning method and apparatus based on clustering and knowledge distillation to solve the above-mentioned technical problems.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A trusted personalized federated learning method based on clustering and knowledge distillation is proposed. A trusted personalized federated learning scenario model based on clustering and knowledge distillation is established. The federated learning scenario model includes a cloud layer and a user layer. The cloud layer is a cloud server, and the user layer mainly consists of M user terminals. Each user terminal stores private local data, and all user terminals jointly train the personalized model through the federated learning algorithm.

[0009] The training process runs in a time-slot manner; the training time is T, which is divided into W consecutive time slots of equal duration; the federated learning algorithm iterates once in each time slot. In each iteration, the user terminal downloads the initial model from the cloud server for local training. After uploading the similarity between user terminals to the cloud server, the cloud server clusters the local models trained by the user terminals, resulting in N clusters; the local models in each cluster are aggregated in the cloud to obtain N local models; the cloud server performs knowledge distillation on the local models to obtain the global model, and each user terminal downloads the global model to its local machine for relearning, training, and updating to obtain a personalized model.

[0010] The local model is clustered to obtain N clusters. The local model is clustered based on Canopy's k-means clustering algorithm, and the steps are as follows:

[0011] (1) The Canopy clustering algorithm is used for "coarse clustering";

[0012] Define the local model set as L = {m1, m2, ..., m} M Given two prior values ​​T1 and T2, where T1 > T2; obtain a node m from set L. r Calculate the distance D(m) from this node to all current cluster centers. r ,a s ), a s The current cluster center is m. If no cluster center exists, m is used as the current cluster center. r A new cluster is formed around the center point, and the minimum distance D is selected. min The distance is calculated using Euclidean distance:

[0013]

[0014] When the distance D min If the distance is less than T1, it means the node belongs to the cluster and is added to the cluster; when the distance is D... min If the distance is less than T2, add the node to the cluster and remove it from the set L; if the distance is less than D, add the node to the cluster and remove it from the set L. min If the value is greater than T1, a new cluster is formed with that node; the loop ends when set L no longer changes or the number of elements is 0.

[0015] (2) Perform "fine clustering" using the k-means clustering algorithm;

[0016] The K cluster centers obtained in step (1) are used as initial centers for "fine clustering";

[0017] For the cluster centers obtained in step (1), calculate the distance from all nodes to each cluster center, and gather each node into the cluster with the nearest cluster center to obtain a cluster; calculate the mean coordinate of all nodes in each cluster as the new cluster center point; repeat the above process until the clustering result no longer changes; divide M local models into N clusters.

[0018] The cloud server performs knowledge distillation on local models using a cyclic knowledge distillation method.

[0019] Local models within a cluster are uploaded to the cloud. The cloud server aggregates the local models in each cluster to obtain N local models, denoted as {F1, F2, ..., F...}. N};

[0020] The dataset for the local model is represented as {H1, H2, ..., H...} N}, each dataset Including training set Prediction set and test set in The entire knowledge distillation process is defined as solving the problem of minimizing the loss function:

[0021]

[0022] Among them, f i For the local model with shared knowledge obtained through training, l is the loss function, which serves as the global model.

[0023] The local models are trained sequentially in a cyclical manner, with the previous local model guiding the next. Without data exchange, the information from all local models is combined until convergence. Convergence is defined as the extraction of all common knowledge.

[0024]

[0025] Among them, g tea It is the feature extractor of the previous local model, g stu It is the feature extractor of the current local model, and x is a data sample from the current local model;

[0026] Through knowledge distillation, knowledge relevant to the current local model is retained, while irrelevant knowledge is discarded. After multiple rounds of training, knowledge relevant to all local models is retained, resulting in the global model. The total loss for training the local models is:

[0027]

[0028] Where λ is the weight of knowledge transfer and focusing on current data, and cls is the cross-entropy loss. Where c i For a fully connected layer in a local model, g i For feature extractors.

[0029] The weight λ is designed as follows: The effective precision on the current local model's valid data is used to determine whether the knowledge of the previous local model should be completely retained; when… When, it means that the training data on the current local model has enough knowledge to train the model; when When this occurs, it indicates that the current local model has insufficient training data and needs to be initialized using previous local models. Let l be the prediction accuracy for the (i+1)th round. t The total loss function for training the local model is λ = λ0, which is fixed to ensure that sufficient common knowledge is preserved.

[0030] The relearning process is used for local knowledge recovery of the global model;

[0031] Global model f i For z ij The predicted probability distribution is q(z) ij ), local model m i For z ij The predicted probability distribution is p(z) ij ), where z ij The global model is obtained through knowledge distillation; the KL divergence between the predicted probability distributions of the two models is used as the loss function for relearning the local knowledge in the global model.

[0032]

[0033] Among them, u ij For user terminal i local private dataset The j-th training sample, I i for The total number of samples;

[0034] The user terminal downloads the global model from the cloud, trains and updates it using local data, and the cross-entropy loss function is:

[0035]

[0036] Among them, v ij For user terminal i local private dataset The label of the j-th training sample For the predicted results;

[0037] The total loss function of the global model during the relearning process is:

[0038] l total ′=l KL +l cls ′

[0039] When the global model f i With local model m i When the performance is the same, that is

[0040]

[0041] in λ1 is the prediction accuracy on the local validation set. After the iteration, the global model is used to update the local model to obtain the final personalized model.

[0042] The user terminal and cloud server are equipped with a secure shuffling model to protect the privacy of parameters during data transmission between the user layer and the cloud layer.

[0043] The secure shuffling model consists of three parts: encoding, shuffling, and decoding.

[0044] The encoding part specifically involves: deploying an encoder to add Laplace-distributed noise to the parameters of the secure shuffling model to ensure local difference privacy; the probability density function of the Laplace distribution is:

[0045]

[0046] when When ε-differential privacy is satisfied, μ, λ, and e are constants, and λ > 0, ε is the privacy budget, and Δo is the sensitivity.

[0047] The encoder transmits data with added Laplace noise to the shuffling section;

[0048] The shuffling part specifically involves: deploying a shuffler to infrequently and in batches mix and shuffle the data with added Laplace noise; the shuffling process is an algorithm that generates a random permutation from a finite set, scrambling the parameters to prevent attackers from making inferences; and forwarding the shuffled data to the decoding part.

[0049] The decoding part specifically involves deploying a decoder that decrypts, stores, aggregates, and ultimately recycles the data received from the shuffler.

[0050] A trusted personalized federated learning device based on clustering and knowledge distillation includes: a memory for storing a federated learning scenario model and a federated learning algorithm; the federated learning scenario model includes a cloud layer and a user layer; the cloud layer is one cloud server, and the user layer mainly consists of M user terminals; each user terminal stores private local data, and all user terminals jointly train a personalized model through the federated learning algorithm.

[0051] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, the processor is configured to:

[0052] Training is conducted in a time-slot manner; the training time is T, which is divided into W consecutive time slots of equal duration; the federated learning algorithm iterates once in each time slot. In each iteration, the user terminal downloads the initial model from the cloud server for local training. After uploading the similarity between user terminals to the cloud server, the cloud server clusters the local models trained by the user terminals, resulting in N clusters; the local models in each cluster are aggregated in the cloud to obtain N local models; the cloud server performs knowledge distillation on the local models to obtain the global model, and each user terminal downloads the global model to its local machine for relearning, training, and updating to obtain a personalized model.

[0053] The beneficial effects of this invention are as follows: This invention proposes a trusted personalized federated learning method and apparatus based on clustering and knowledge distillation, establishing a two-layer framework: a user layer and a cloud layer. In cloud collaboration, user terminals are divided into multiple clusters using clustering methods. The k-means clustering algorithm based on Canopy reduces the number of iterations. Iterative knowledge distillation sequentially trains each cluster in a cyclical manner to extract common knowledge. Relearning is performed on the global model at the user terminal to recover local knowledge. A distributed differential privacy mechanism based on a shuffling algorithm is designed. By adding an extra hybrid encoding step between the user terminal and the server, a high level of privacy protection can be achieved with only a small amount of added noise. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a cloud-based collaborative federated learning model scenario according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the shuffling algorithm model described in an embodiment of the present invention;

[0056] Figure 3 This is a flowchart of the trusted personalized federated learning algorithm based on clustering and knowledge distillation as described in an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of a trusted personalized federated learning device based on clustering and knowledge distillation. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0059] A trustworthy personalized federated learning method and apparatus based on clustering and knowledge distillation is proposed for training personalized health monitoring models in the healthcare field.

[0060] The scene diagram of the present invention is as follows Figure 1 As shown, a two-layer federated learning framework is established, comprising a user layer and a cloud layer. The user layer includes user terminals, such as health monitoring devices like wristbands, while the cloud layer consists of cloud servers. These two layers form a cloud-based collaborative model that jointly participates in the federated learning process to train a personalized local model.

[0061] The structure diagram of the shuffling algorithm model of this invention is as follows: Figure 2 As shown, it consists of an encoder, a shuffler, and a decoder. The data first enters the encoder, where Laplace noise is added, then it is shuffled by the shuffler, and finally decrypted by the decoder.

[0062] The flowchart of this invention is as follows Figure 3 As shown, firstly, a two-layer cloud-based collaborative federated learning scenario model is established; then, the local model is clustered using the k-means clustering algorithm based on Canopy to obtain N clusters; the N clusters are then aggregated to obtain N local models; next, the local models are subjected to iterative knowledge distillation to obtain the common knowledge part of the local models; finally, a relearning process is designed for local recovery, and personalized models are obtained through local training.

[0063] A schematic diagram of the device of the present invention is shown below. Figure 4 As shown, the device consists of a memory and a processor. The memory stores the federated learning scenario model and the federated learning algorithm; the processor executes the computer program stored in the memory.

[0064] The specific steps are as follows:

[0065] A trustworthy personalized federated learning scenario model based on clustering and knowledge distillation is established. The federated learning scenario model includes a cloud layer and a user layer. The cloud layer is a cloud server, and the user layer mainly consists of M user terminals (health monitoring devices). Each user terminal stores private local data, and all user terminals jointly train the personalized model through the federated learning algorithm.

[0066] The training process runs in a time-slot manner; the training time is T, which is divided into W consecutive time slots of equal duration; the federated learning algorithm iterates once in each time slot. In each iteration, the user terminal downloads the initial model from the cloud server for local training. After uploading the similarity between user terminals to the cloud server, the cloud server clusters the local models trained by the user terminals, resulting in N clusters; the local models in each cluster are aggregated in the cloud to obtain N local models; the cloud server performs knowledge distillation on the local models to obtain the global model, and each user terminal downloads the global model to its local machine for relearning, training, and updating to obtain a personalized model.

[0067] The local model is clustered to obtain N clusters. The local model is clustered based on Canopy's k-means clustering algorithm, and the steps are as follows:

[0068] (1) The Canopy clustering algorithm is used for "coarse clustering";

[0069] Define the local model set as L = {m1, m2, ..., m} M Given two prior values ​​T1 and T2, where T1 > T2; obtain a node m from set L. r Calculate the distance D(m) from this node to all current cluster centers. r ,a s ), a s The current cluster center is m. If no cluster center exists, m is used as the current cluster center. r A new cluster is formed around the center point, and the minimum distance D is selected. min The distance is calculated using Euclidean distance:

[0070]

[0071] When the distance D min If the distance is less than T1, it means the node belongs to the cluster and is added to the cluster; when the distance is D... min If the distance is less than T2, it means that the node not only belongs to this cluster but is also very close to the current cluster center. Therefore, the node is added to this cluster and removed from set L; when the distance is D... min If the value is greater than T1, a new cluster is formed with that node; the loop ends when set L no longer changes or the number of elements is 0.

[0072] (2) The k-means clustering algorithm is used for "fine clustering";

[0073] The K cluster centers obtained in step (1) are used as initial centers for "fine clustering";

[0074] For the cluster centers obtained in step (1), calculate the distance from all nodes to each cluster center, and gather each node into the cluster with the nearest cluster center to obtain a cluster; calculate the mean coordinate of all nodes in each cluster as the new cluster center point; repeat the above process until the clustering result no longer changes; divide M local models into N clusters.

[0075] The cloud server performs knowledge distillation on local models using a cyclic knowledge distillation method.

[0076] Local models within a cluster are uploaded to the cloud. The cloud server aggregates the local models in each cluster to obtain N local models, denoted as {F1, F2, ..., F...}. N};

[0077] The dataset for the local model is represented as {H1, H2, ..., H...} N}, each dataset Including training set Prediction set and test set in The entire knowledge distillation process is defined as solving the problem of minimizing the loss function:

[0078]

[0079] Among them, f i For the local model with shared knowledge obtained through training, l is the loss function, which serves as the global model.

[0080] The local models are trained sequentially in a cyclical manner, with the previous local model guiding the next. Without data exchange, the information from all local models is combined until convergence. Convergence is defined as the extraction of all common knowledge.

[0081]

[0082] Among them, g tea It is the feature extractor of the previous local model, g stu It is the feature extractor of the current local model, and x is a data sample from the current local model;

[0083] Through knowledge distillation, knowledge relevant to the current local model is retained, while irrelevant knowledge is discarded. After multiple rounds of training, knowledge relevant to all local models is retained, resulting in the global model. The total loss for training the local models is:

[0084]

[0085] Where λ is the weight of knowledge transfer and focusing on current data, and cls is the cross-entropy loss. Where c i For classification layer, g i For feature extractors.

[0086] The weight λ is designed as follows: The effective precision on the current local model's valid data is used to determine whether the knowledge of the previous local model should be completely retained; when… When, it means that the training data on the current local model has enough knowledge to train the model; when When this occurs, it indicates that the current local model has insufficient training data and needs to be initialized using previous local models. Let l be the prediction accuracy for the (i+1)th round. t The total loss function for training the local model is λ = λ0. Since knowledge needs to be accumulated during the knowledge distillation process, λ = λ0 is fixed to ensure that enough common knowledge can be retained.

[0087] The relearning process is used for local knowledge recovery of the global model;

[0088] Global model f i For z ij The predicted probability distribution is q(z) ij ), local model m i For z ij The predicted probability distribution is p(z) ij ), where z ij The global model is obtained through knowledge distillation; the KL divergence between the predicted probability distributions of the two models is used as the loss function for relearning the local knowledge in the global model.

[0089]

[0090] Among them, u ij For user terminal i local private dataset The j-th training sample, I i for The total number of samples;

[0091] The user terminal downloads the global model from the cloud, trains and updates it using local data, and the cross-entropy loss function is:

[0092]

[0093] Among them, v ij For user terminal i local private dataset The label of the j-th training sample For the predicted results;

[0094] The total loss function of the global model during the relearning process is:

[0095] l total ′=l KL +l cls ′

[0096] When the global model f i With local model m i When the performance is the same, that is

[0097]

[0098] in λ1 is the prediction accuracy on the local validation set. After the iteration, the global model is used to update the local model to obtain the final personalized model.

[0099] The user terminal and cloud server are equipped with a secure shuffling model to protect the privacy of parameters during data transmission between the user layer and the cloud layer.

[0100] The secure shuffling model consists of three parts: encoding, shuffling, and decoding.

[0101] The encoding part specifically involves: deploying an encoder to add noise following a Laplace distribution to the model parameters to ensure local difference privacy; the probability density function of the Laplace distribution is:

[0102]

[0103] when When ε-differential privacy is satisfied, μ and λ are constants, and λ > 0, ε is the privacy budget, and Δo is the sensitivity.

[0104] The encoder transmits data with added Laplace noise to the shuffling section;

[0105] The shuffling part specifically involves: deploying a shuffler to infrequently and in batches mix and shuffle the data with added Laplace noise; the shuffling process is an algorithm that generates a random permutation from a finite set, scrambling the parameters to prevent attackers from making inferences; and forwarding the shuffled data to the decoding part.

[0106] The decoding part specifically involves deploying a decoder that decrypts, stores, aggregates, and ultimately recycles the data received from the shuffler.

[0107] In summary, this invention proposes a trustworthy personalized federated learning method and apparatus based on clustering and knowledge distillation. Specifically, an embodiment is presented for health monitoring in the healthcare field. Personalization is achieved through clustering, knowledge distillation, and relearning methods, resulting in a personalized model. Furthermore, to address privacy concerns, a distributed differential privacy mechanism based on a shuffling algorithm is employed to achieve efficient privacy protection.

[0108] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for trusted personalized federated learning based on clustering and knowledge distillation, characterized in that, A trusted personalized federated learning scenario model based on clustering and knowledge distillation is established, the federated learning scenario model comprises a cloud layer and a user layer; the cloud layer is 1 cloud server, and the user layer mainly comprises A plurality of user terminals; each user terminal stores private local data, and all user terminals jointly train a personalized model through a federated learning algorithm; The training process runs in a time slot manner; the time used by the training process is , divided into consecutive time slots with the same duration; the federated learning algorithm is iterated in each time slot, and in each round of iteration, the user terminal downloads the initial model from the cloud server for local training, uploads the similarity between the user terminals to the cloud server, and then the cloud server clusters and divides the local models trained by the user terminals to obtain clusters; the local models in each cluster are aggregated in the cloud layer to obtain local models; the cloud server obtains a global model after knowledge distillation of the local models, each user terminal downloads the global model to the local for relearning, and obtains a personalized model through training and updating; the local models are clustered and divided to obtain clusters, and the local models are clustered and divided based on the Canopy mean clustering algorithm, and the steps are as follows: (2.1) Canopy clustering algorithm is used for "coarse clustering"; Definition of local model set is , given two prior values and , and ; a node is obtained from the set , the distance between the node and all current cluster centers is calculated , is the current cluster center, when there is no cluster center, a new cluster is formed with as the center point, and the minimum distance is selected; the distance is calculated by using the Euclidean distance: ； When distance is less than , it indicates that the node belongs to the cluster, and is added to the cluster; when distance is less than , the node is added to the cluster, and is deleted from the set ; when distance is greater than , a new cluster is formed with the node; the loop operation is ended until the set has no change or the number of elements is 0. (2.2) by "fine clustering" by means of a mean-shift clustering algorithm; K cluster centers obtained in step (2.1) are used as initial center points for "fine clustering"; For the cluster center obtained in step (2.1), the distance from all nodes to each cluster center is calculated, each node is clustered into the cluster closest to the cluster center, and the cluster is obtained; The coordinate mean of all nodes in each cluster is calculated as a new cluster center point; The above process is repeatedly performed until the clustering result no longer changes; The local model is divided into clusters; the cloud server adopts a cyclic knowledge distillation manner for knowledge distillation of the local model. The local models in the cluster are uploaded to the cloud layer, and the cloud server aggregates the local models in each cluster to obtain a local model, denoted as ; The dataset of the local model is represented as Each dataset includes a training set , a prediction set and a test set , wherein , The whole process of knowledge distillation is defined as a problem of solving loss function minimization: ； wherein, is the locally trained model with common knowledge as the global model, is the loss function; The local model is sequentially trained in a loop, the previous local model guides the next local model, and the information of all local models is combined together without data exchange until convergence; the convergence is defined as all common knowledge is extracted: ； wherein, is a feature extractor of the previous local model, is a feature extractor of the current local model, is a data sample from the current local model; Through knowledge distillation, the knowledge related to the current local model is retained, and irrelevant knowledge is discarded, and after multiple rounds of loop training, the knowledge related to all local models is retained, and the global model is obtained; the total loss of training the local model is: ； wherein, is the weight of knowledge transfer and focusing current data, is the cross-entropy loss, ; wherein is a fully connected layer in the local model, is a feature extractor.

2. The cluster and knowledge distillation based trusted personalized federated learning method according to claim 1, characterized in that, The weights The design is as follows: using the effective accuracy on the current local model effective data to determine whether to completely retain the knowledge of the previous local model; when , it indicates that the training data on the current local model has enough knowledge to train the model; when , it indicates that the training data information of the current local model is less, and the previous local model needs to be used to initialize the current local model, wherein, is the prediction accuracy of the first round, is the total loss function of training the local model; fix to ensure that enough common knowledge can be retained.

3. The cluster and knowledge distillation based trusted personalized federated learning method according to claim 1 or 2, characterized in that, The relearning process is used for local knowledge recovery of the global model; Global model The prediction probability distribution of is Local model The prediction probability distribution of is wherein, is the global model obtained by knowledge distillation; the KL divergence between the two prediction probability distributions as the loss function of the global model relearning to recover local knowledge: ； in, For user terminals Local private dataset The training samples, for The total number of samples; The user terminal downloads the global model from the cloud layer, and trains and updates it using local data, and the cross entropy loss function is: ； wherein, a user terminal a local private dataset a first training sample label, a prediction result; The total loss function of the global model in the relearning process is: ； When the global model is the same as the local model performance, i.e. ； wherein is the prediction accuracy on the local validation set, is the hyper-parameter, the global model updates the local model after the iteration, and the final personalized model is obtained.

4. The method of claim 3, wherein, The user terminal and the cloud server are arranged with a secure shuffling model, and the parameters are protected in the process of data transmission in the user layer and the cloud layer; The secure shuffling model is divided into three parts of encoding-shuffling-decoding; The encoding part is specifically: deploying an encoder, adding noise obeying Laplace distribution to the secure shuffling model parameters to ensure local differential privacy; the probability density function of Laplace distribution is: ； When , satisfies - differential privacy; wherein, , , is a constant, and , is a privacy budget, is a sensitivity; The encoder transmits the data added with Laplace noise to the shuffling part; The shuffling part is specifically: deploying a shuffler, mixing and shuffling the data added with Laplace noise infrequently and in batches, the shuffling process is an algorithm for generating a random arrangement of a finite set, which scrambles the parameters to prevent attackers from inferring; the shuffled data is forwarded to the decoding part; The decoding part is specifically: deploying a decoder, the decoder decrypts, stores, aggregates and finally recycles the data received from the shuffler.

5. A trusted personalized federated learning device based on clustering and knowledge distillation, characterized in that, The method comprises a memory for storing a federated learning scenario model and a federated learning algorithm; the federated learning scenario model comprises a cloud layer and a user layer; the cloud layer is one cloud server, and the user layer mainly comprises a plurality of user terminals; each user terminal stores private local data, and all the user terminals jointly train a personalized model through the federated learning algorithm. The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, the processor is configured to: The training is operated in a time slot manner; the time used by the training process is , is divided into consecutive time slots with the same duration; the federated learning algorithm is iterated in each time slot, and in each round of iteration, the user terminal downloads an initial model from the cloud server for local training, uploads the similarity between the user terminals to the cloud server, and then the cloud server clusters and divides the local models trained by the user terminals to obtain clusters; the local models in each cluster are aggregated in the cloud layer to obtain local models; the cloud server obtains a global model after knowledge distillation of the local models, and each user terminal downloads the global model to the local for relearning to obtain a personalized model.

Images