A personalized federated learning method and system based on dual-teacher knowledge distillation

Abstract

The present invention proposes a personalized federated learning method and system based on dual-teacher knowledge distillation, which belongs to the technical field of federated learning. The method is executed by a client, including: receiving a global model sent by a server in the previous round; obtaining a sparse vector based on local data; obtaining a personalized model based on local data training; uploading the sparse vector and personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model; receiving the updated global model in this round; performing dual-teacher knowledge distillation training based on the previous round of personalized models and the updated global model, and obtaining the final personalized model after multiple rounds of knowledge distillation training iterations. The method makes the personalized model of the previous round serve as the first teacher and the global model serve as the second teacher, thereby achieving a balance between the generalization ability and personalization ability of the personalized model.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a personalized federated learning method and system based on dual-teacher knowledge distillation. Background Art

[0002] Personalized federated learning can be divided into two categories: global model personalization and learning personalized models. The main means of global model personalization is to fine-tune the global model on the local data of each participant to generate a model suitable for the data characteristics of the participant. This also means that if the data differences between the participants are large, the global model may not perform well on some participants, and even fine-tuning cannot fully compensate. Learning personalized models focuses on learning an independent personalized model for each participant from the beginning, rather than just fine-tuning the global model. Existing methods are implemented through clustering, meta-learning, etc. Among them, meta-learning can enable participants to adapt to local data faster by training meta-models, but the training process is complicated and the computational cost is high, especially when there are many participants or the data is highly heterogeneous. Clustering groups participants according to data characteristics so that participants in each cluster get a personalized model, but this method only considers the collaboration between users within the cluster, but ignores the communication between clusters, and does not fully utilize the local data of all users for federated learning. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a personalized federated learning method and system based on dual-teacher knowledge distillation, which uses dual-teacher knowledge distillation to guide the training of personalized models. The method can be executed by the client, so that the personalized model of the previous round serves as the first teacher to guide the personalization of the personalized model, and prevents the regression of personalized performance and knowledge forgetting caused by the aggregation of model parameters; at the same time, the global model serves as the second teacher to guide the generalization ability of the personalized model, and prevents the personalized model from overfitting the local knowledge, thereby achieving a balance between the generalization ability and the personalization ability of the personalized model.

[0004] In order to achieve the above object, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a personalized federated learning method based on dual-teacher knowledge distillation, which is executed by a client and includes:

[0006] The global model sent by the receiving server in the last round;

[0007] Get sparse vectors based on local data;

[0008] Get a personalized model based on local data training;

[0009] Upload the sparse vector and the personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model;

[0010] This round receives and updates the global model;

[0011] Based on the previous round of personalized model and the updated global model, dual-teacher knowledge distillation training is performed. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0012] In a second aspect, the present invention provides a personalized federated learning method based on dual-teacher knowledge distillation, which is executed by a server and includes:

[0013] In the previous round, the global model is sent to each client. For any client, the client obtains a sparse vector and personalized model parameters based on local data and uploads them to the server.

[0014] Construct coefficient matrix based on sparse vectors;

[0015] The coefficient matrix is ​​used to aggregate the personalized model parameters sent by each client to obtain an updated global model;

[0016] In this round, the updated global model will be sent to each client separately, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0017] In a third aspect, the present invention provides a personalized federated learning system based on dual-teacher knowledge distillation, which is executed by a client and includes:

[0018] A global model receiving module is used to receive the global model sent by the server in the previous round;

[0019] A sparse vector extraction module, used to obtain sparse vectors based on local data;

[0020] The local model training module is used to obtain a personalized model based on local data training;

[0021] A data upload module is used to upload the sparse vector and the personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model;

[0022] The global model receiving module is updated, and is used to receive and update the global model in this round;

[0023] The dual-teacher knowledge distillation module is used to perform dual-teacher knowledge distillation training based on the previous round of personalized model and updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0024] In a fourth aspect, the present invention provides a personalized federated learning system based on dual-teacher knowledge distillation, which is executed by a server and includes:

[0025] A data sending module is used to send the global model to each client in the previous round. For any client, the client obtains a sparse vector and personalized model parameters based on local data and uploads them to the server;

[0026] Correlation and sparse matrix calculation module, used to construct coefficient matrices based on sparse vectors;

[0027] A personalized and global model aggregation module, used to aggregate the personalized model parameters sent by each client using a coefficient matrix to obtain an updated global model;

[0028] The updated global model sending module is used to send the updated global model to each client in this round, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0029] In a fifth aspect, the present invention provides an electronic device, the electronic device comprising:

[0030] at least one processor;

[0031] and a memory communicatively coupled to the at least one processor;

[0032] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the personalized federated learning method based on dual-teacher knowledge distillation described in the first aspect or the second aspect of the present invention.

[0033] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement a personalized federated learning method based on dual-teacher knowledge distillation as described in the first aspect or the second aspect of the present invention when executed.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This invention breaks the barrier of clustering in traditional clustering by constructing a coefficient matrix based on the client sparse vector. Instead of setting up clusters separately and aggregating models from the perspective of clusters, the invention aggregates models from the perspective of each participant (client). Compared with clustering limited to collaboration within a cluster, this invention makes full use of all participants to collaborate, and determines the closeness of collaboration based on the depth of the correlation between the participants, thereby maximizing the benefits of user collaboration.

[0036] The dual-teacher knowledge distillation method provided by the present invention jointly guides the training of the current round of personalized models (students) by combining the previous round of personalized models (first teacher) with the global model updated by the server (second teacher). This dual guidance mechanism not only utilizes the specificity of local data for accurate modeling, but also incorporates the generalization ability of the global model, effectively alleviating the problem of data islands. By calculating the dual optimization of supervised learning loss and knowledge distillation loss, it is ensured that the final personalized model not only conforms to the local data characteristics, but also has high generalization performance, thereby protecting data privacy while improving the overall effect and adaptability of the model.

[0037] Compared with the traditional federated learning method, the method of the present invention can not only retain the advantages of privacy protection and distribution, but also perform personalized optimization for the data characteristics of each participant, thereby improving the practical application effect of the model. Compared with the traditional personalized federated learning method, the present invention can not only train a personalized model for each participant, but also use the correlation of the participants to improve the generalization ability of the personalized model under the same feature distribution. Compared with the personalization of traditional model parameter aggregation, the present invention can not only retain personalized local knowledge while learning the relevant knowledge of other participants, but also review the learned personalized knowledge and global knowledge according to the knowledge distillation technology, and determine the amount of knowledge required to be learned according to the closeness of the correlation. The present invention can solve the problem of performance degradation of the traditional federated learning global model under the condition of data heterogeneity, and has achieved good accuracy in related domain drift data sets (digit, office-home).

[0038] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0040] Figure 1 A detailed flow chart of a personalized federated learning method based on dual-teacher knowledge distillation provided in Example 1 of the present invention;

[0041] Figure 2 A main flow chart of a personalized federated learning method based on dual-teacher knowledge distillation provided in Example 1 of the present invention;

[0042] Figure 3 A main flow chart of a personalized federated learning method based on dual-teacher knowledge distillation provided in Example 2 of the present invention;

[0043] Figure 4 The main flow chart of a personalized federated learning system based on dual-teacher knowledge distillation provided in Embodiments 3 and 4 of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] Embodiment 1

[0046] Figure 1 This is a detailed flow chart of a personalized federated learning method based on dual-teacher knowledge distillation provided in the first embodiment of the present invention. This embodiment can be applied to the personalized federated learning of data with non-independent and identically distributed data features. The method can be executed by a model training device based on dual-teacher knowledge distillation and federated learning, and the model training device based on dual-teacher knowledge distillation and federated learning can be configured in the client. Figure 2 As shown, the method includes:

[0047] S1: the global model sent by the receiving server in the last round;

[0048] Specifically, assume that there are a total of M federated learning clients as participants and one server participating in the training. The number of the participants is i=1,2,……,M , each participant i There are local data sets , indicating the participants i The amount of private data is Represents. The dataset contains data samples x and sample labels y.

[0049] The purpose of the method in this embodiment is to train a personalized model for each participant that is unique to the participant and adapts to the participant's local data. The model parameters are This computational objective can be expressed by maximizing the local performance of the personalized model on the participant (i.e., minimizing the loss function of the personalized model on the participant), namely:

[0050]

[0051] in, represents the sum of local losses of personalized models of all participants, which needs to be minimized. Indicates that the participants i The total local loss, in this embodiment, consists of three parts. Indicates the participants i The personalized model parameters, Indicates the participants i local dataset.

[0052] In this embodiment, before the federated learning training begins, the auxiliary participants are first constructed. The data sets of the auxiliary participants are independent and identically distributed, that is, there is no data heterogeneity problem. The model trained using the data set is used as the initial global model. .

[0053] Since the main purpose of personalized federated learning is to train a personalized model for each participant in the training, the server will send the initial global model Sent to the local participants, each participant receives the initial global model Perform two parts of work: channel sparsity collection and local training. For channel sparsity collection:

[0054] S2: Obtain sparse vectors based on local data. Sparse vectors are composed of sparse values ​​extracted from channels by the participant by inputting their local data into the global model. The sparse vectors of the participant are obtained by combining the sparse values ​​of multiple channels. Specifically:

[0055] S201: inputting local data into a personalized model, extracting feature vectors in a mapping channel to calculate the proportion of zero elements, and averaging the proportion of zero elements to obtain a sparse value;

[0056] S202: Combining sparse values ​​collected from multiple channels into a sparse vector.

[0057] It should be understood that the current theoretical knowledge is sufficient to prove that channel sparsity can represent the characteristic distribution of data. By inputting the data of the entire data set into the model, extracting the feature map in the mapping channel to calculate the proportion of zero elements, and taking the mean to represent the sparse value of the participant, the sparse values ​​collected by multiple channels can form a sparse vector.

[0058] In S3, a personalized model is obtained based on local data training. For local training, each client trains personalized model parameters unique to the participant based on local data. It should be understood that the training of personalized models by the client is known to those skilled in the art, and Figure 1 The model parameters involved are exemplary and those skilled in the art may select them according to actual conditions.

[0059] S4: Upload the sparse vector and the personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model.

[0060] Since personalized model training needs to focus not only on local data distribution but also on the correlation between participants, it is necessary to upload personalized model parameters for processing in order to allow participants to learn from each other and avoid overfitting.

[0061] Among them, the coefficient matrix is ​​constructed based on the sparse vector, specifically: based on the sparse vectors of the two participants, the Euclidean distance between the two participants is obtained. The stronger the correlation between the participants, the smaller the value calculated by the Euclidean distance. By collecting the Euclidean distances between two different participants, the correlation matrix can be obtained .

[0062] The coefficient matrix can be further calculated through the correlation matrix, which is used for the aggregation of personalized model parameters. The calculation formula is as follows:

[0063]

[0064] in, and From the incidence matrix Extract from Indicates the participants i With the participants j A Euclidean distance of a sparse vector, Represents the Euclidean distance between the sparse vector of participant i and any other participant except i.

[0065] is a coefficient matrix, which indicates how much of the personalized model parameters of participant i come from the personalized model of participant j during the aggregation process. This weighted aggregation method is more convincing than average aggregation, because the stronger the correlation between the participants, the larger the coefficient matrix, and the greater the proportion of the associated model parameters in the aggregation, that is, the participant will learn more knowledge from the participants with whom it is strongly correlated.

[0066] Furthermore, the personalized model parameters are uploaded for processing, the purpose is to allow the participants to learn from each other, avoid overfitting, and allow the personalized model to learn the knowledge of other participants through model parameter aggregation; the coefficient matrix is ​​used to aggregate the personalized model parameters sent by each client to obtain an updated global model. First, the personalized model parameters of each participant in this round are calculated, and the calculation formula is as follows:

[0067]

[0068] in, Indicates the participants i The personalized model parameters of this round are obtained after aggregation. Indicates the participants i The personalized model parameters uploaded to the server in the previous round. k is a hyperparameter used to indicate how much proportion of the learning process is retained and how much proportion is learned from others. is the coefficient matrix, which represents the i What percentage of the knowledge learned comes from the participants? j This also indirectly shows that the closer the relationship between the parties involved, the more knowledge they need to learn. Indicates that except for the participants i The personalized model parameters of other participants are calculated. This aggregation method of personalized models makes full use of the relationship between the data features of all participants. The more similar the data features are, the more knowledge they can learn from each other.

[0069] Then, based on the newly calculated personalized model parameters of this round, the global model parameters can be aggregated with reference to the data volume. The calculation formula is as follows:

[0070]

[0071] in, Represents the global model parameters obtained by aggregating personalized model parameters. represents the amount of data of participant i, N Indicates the total data volume of all participants. It represents the personalized model parameters of each participant in this round calculated in the previous text.

[0072] S5: This round receives and updates the global model.

[0073] S6: Based on the previous round of personalized model and the updated global model, dual-teacher knowledge distillation training is performed. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained. Specifically, it includes:

[0074] S501: Perform supervised learning on the current round of personalized model based on local data to obtain supervised learning loss;

[0075] S502: Using the locally retained personalized model of the previous round as the first teacher, the global model sent by the server in this round as the second teacher, and the personalized model of this round as the student, knowledge distillation learning is performed to obtain the knowledge distillation loss;

[0076] S503: Calculate the supervised learning loss and the knowledge distillation loss, and obtain a final personalized model when the loss is minimum.

[0077] Specifically, the personalized model will experience a regression in personalized performance after a series of operations such as uploading, aggregated learning, and then issuing. Because in the process of aggregation, the personalized model learns the knowledge of other participants and forgets its own knowledge. Therefore, this embodiment considers introducing the first teacher model to guide the personalized model in personalized knowledge when it reaches this round after the previous round of updates.

[0078] In the specific implementation process, this embodiment retains the personalized model of the previous round. When local training is performed after the personalized model of this round is completed, this embodiment uses the technical means of knowledge distillation to review the knowledge of the personalized model of this round.

[0079] In addition, when reviewing personalized knowledge, the problem of forgetting global knowledge will occur, which is also a test of the generalization ability of the personalized model. Because when the model learns new knowledge, it will always forget the old knowledge, so the model must constantly review the knowledge.

[0080] Therefore, this embodiment considers introducing a second teacher model to guide the personalized model to generalize knowledge. In a specific embodiment, a global model is obtained while obtaining the personalized model of this round. Because the global knowledge learned by the global model must be richer than the global knowledge learned by the personalized model, the global model acts as a teacher model to review the global knowledge of the personalized model of this round by using the technical means of knowledge distillation. The calculation formula for the local training loss of the personalized model is as follows:

[0081]

[0082] in, It represents the total loss of the local training of the participants, which consists of three parts: the cross entropy loss generated by the local training ( Figure 1 Medium representation) and the distillation loss resulting from two teacher-guided learning. represents the difference between the soft labels output by the personalized model retained in the previous round and the personalized model in this round ( Figure 1 Medium ), and guides the current round of personalized model to review personalized knowledge by narrowing the difference. Represents the difference between the soft labels output by the global model and the personalized model of this round ( Figure 1 Medium ), which guides the current round of personalized model to review global knowledge by narrowing the difference. They respectively represent the soft labels generated by the corresponding models when facing the same samples.

[0083] The two teacher models simultaneously guide the personalized model with personalized and global knowledge. After multiple rounds of knowledge distillation training iterations, the performance of the personalized model converges and the iteration ends, and the final personalized model is obtained.

[0084] It should be understood that knowledge distillation technology is a common method of model compression and transfer learning, and the specific process of knowledge distillation of the present invention will not be repeated here.

[0085] The present invention solves the problem of performance degradation of the global model in federated learning participants caused by different data distributions of federated learning participants through the functions of these modules, and realizes the generation of personalized federated learning models unique to all participants. It also makes full use of the degree of association between participants, improves the generalization ability of personalized models in participants with similar data distribution, and improves the performance of personalized models in the participants themselves, that is, accuracy.

[0086] Embodiment 2

[0087] like Figure 3 As shown, this embodiment provides a personalized federated learning method based on dual-teacher knowledge distillation, which is executed by a server and includes:

[0088] In the previous round, the global model is sent to each client. For any client, the client obtains a sparse vector and personalized model parameters based on local data and uploads them to the server.

[0089] Construct coefficient matrix based on sparse vectors;

[0090] The coefficient matrix is ​​used to aggregate the personalized model parameters sent by each client to obtain an updated global model;

[0091] In this round, the updated global model will be sent to each client separately, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0092] Embodiment 3

[0093] like Figure 4 As shown, this embodiment provides a personalized federated learning system based on dual-teacher knowledge distillation (not all modules are shown in the figure), which is executed by the client and includes:

[0094] A global model receiving module is used to receive the global model sent by the server in the previous round;

[0095] A sparse vector extraction module, used to obtain sparse vectors based on local data;

[0096] The local model training module is used to obtain a personalized model based on local data training;

[0097] A data upload module is used to upload the sparse vector and the personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model;

[0098] The global model receiving module is updated, and is used to receive and update the global model in this round;

[0099] The dual-teacher knowledge distillation module is used to perform dual-teacher knowledge distillation training based on the previous round of personalized model and updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0100] It should be understood that Figure 4 The model parameters involved are exemplary and those skilled in the art may select them according to actual conditions.

[0101] Embodiment 4

[0102] like Figure 4 As shown, this embodiment provides a personalized federated learning system based on dual-teacher knowledge distillation (not all modules are shown in the figure), which is executed by a server and includes:

[0103] A data sending module is used to send the global model to each client in the previous round. For any client, the client obtains a sparse vector and personalized model parameters based on local data and uploads them to the server;

[0104] Correlation and sparse matrix calculation module, used to construct coefficient matrices based on sparse vectors;

[0105] A personalized and global model aggregation module, used to aggregate the personalized model parameters sent by each client using a coefficient matrix to obtain an updated global model;

[0106] The updated global model sending module is used to send the updated global model to each client in this round, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

[0107] Embodiment 5

[0108] This embodiment provides an electronic device, the electronic device comprising:

[0109] at least one processor;

[0110] and a memory communicatively coupled to the at least one processor;

[0111] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a personalized federated learning method based on dual-teacher knowledge distillation described in any embodiment of the present invention.

[0112] Embodiment 6

[0113] This embodiment provides a computer-readable storage medium, which stores computer instructions. The computer instructions are used to enable a processor to implement a personalized federated learning method based on dual-teacher knowledge distillation as described in any embodiment of the present invention when executed.

[0114] The steps or modules involved in the above embodiments 2 to 6 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A personalized federated learning method based on dual-teacher knowledge distillation, characterized in that: Executed by the client, including: The global model sent by the receiving server in the last round; Obtaining a sparse vector based on local data; specifically comprising: inputting local data into a personalized model, extracting feature vectors in a mapping channel, calculating the proportion of zero elements, and averaging the proportion of zero elements to obtain a sparse value; and combining the sparse values ​​collected from multiple channels into a sparse vector; Get a personalized model based on local data training; The sparse vector and the personalized model parameters are uploaded to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model; wherein, the coefficient matrix is ​​constructed based on the sparse vector, specifically including: calculating the Euclidean distance based on the sparse vectors of the two clients, aggregating the Euclidean distances between two different participants to obtain an association matrix, and constructing a coefficient matrix based on the association matrix: in, is the coefficient matrix; and Extract from the incidence matrix, Indicates the participants i With the participants j A Euclidean distance of a sparse vector, represents the Euclidean distance between the sparse vector of participant i and any other participant except i; M is the number of clients; This round receives and updates the global model; Based on the previous round of personalized model and the updated global model, dual-teacher knowledge distillation training is performed. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

2. A personalized federated learning method based on dual-teacher knowledge distillation as claimed in claim 1, characterized in that: The coefficient matrix is ​​used to aggregate the personalized model parameters sent by each client to obtain an updated global model, specifically: in, represents updated global model parameters obtained by aggregating personalized model parameters, and an updated global model is obtained based on the updated global model parameters; represents the amount of data from client i, N Indicates the total data volume of all clients; Represents the client i The personalized model parameters of this round are obtained after the personalized model parameters are aggregated; Represents the client i The personalized model parameters of the previous round uploaded to the server; k is a hyperparameter used to indicate the proportion of retaining oneself and the proportion of learning from others during the learning process; is the coefficient matrix, indicating that for the client i What percentage of the knowledge learned comes from the client? j ; Indicates that except client i The personalized model parameters of other clients.

3. A personalized federated learning method based on dual-teacher knowledge distillation as claimed in claim 1, characterized in that: The dual-teacher knowledge distillation training is performed based on the previous round of personalized model and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained, which specifically includes: Perform supervised learning on this round of personalized model based on local data to obtain supervised learning loss; The previous round of personalized model is used as the first teacher, the updated global model is used as the second teacher, and the current round of personalized model is used as the student to perform knowledge distillation learning and obtain the knowledge distillation loss; The supervised learning loss and the knowledge distillation loss are calculated, and a final personalized model is obtained when the loss is minimized.

4. A personalized federated learning method based on dual-teacher knowledge distillation, characterized in that: Executed by the server, including: In the previous round, the global model is sent to each client respectively. For any client, the client obtains a sparse vector and personalized model parameters based on local data and uploads them to the server; the sparse vector is obtained based on local data, specifically including: inputting local data into the personalized model, extracting feature vectors in the mapping channel to calculate the proportion of zero elements, averaging the proportion of zero elements to obtain a sparse value; and combining the sparse values ​​collected from multiple channels into a sparse vector; Constructing a coefficient matrix based on sparse vectors; specifically, calculating the Euclidean distance based on the sparse vectors of the two clients, collecting the Euclidean distances between two different participants to obtain an association matrix, and constructing a coefficient matrix based on the association matrix: in, is the coefficient matrix; and Extract from the incidence matrix, Indicates the participants i With the participants j A Euclidean distance of a sparse vector, represents the Euclidean distance between the sparse vector of participant i and any other participant except i; M is the number of clients; The coefficient matrix is ​​used to aggregate the personalized model parameters sent by each client to obtain an updated global model; In this round, the updated global model will be sent to each client separately, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

5. A personalized federated learning system based on dual-teacher knowledge distillation, characterized in that: Executed by the client, including: A global model receiving module is used to receive the global model sent by the server in the previous round; The sparse vector extraction module is used to obtain a sparse vector based on local data; specifically, the module includes: inputting local data into a personalized model, extracting feature vectors in a mapping channel to calculate the proportion of zero elements, averaging the proportion of zero elements to obtain a sparse value; and combining the sparse values ​​collected from multiple channels into a sparse vector; The local model training module is used to obtain a personalized model based on local data training; The data upload module is used to upload the sparse vector and the personalized model parameters to the server, so that the server constructs a coefficient matrix based on the sparse vector, and uses the coefficient matrix to aggregate the personalized model parameters sent by each client to obtain an updated global model; wherein, the coefficient matrix is ​​constructed based on the sparse vector, specifically including: calculating the Euclidean distance based on the sparse vectors of the two clients, aggregating the Euclidean distances between two different participants to obtain the association matrix, and constructing the coefficient matrix based on the association matrix: in, is the coefficient matrix; and Extract from the incidence matrix, Indicates the participants i With the participants j A Euclidean distance of a sparse vector, represents the Euclidean distance between the sparse vector of participant i and any other participant except i; M is the number of clients; An updated global model receiving module is used to receive and update the global model in this round; The dual-teacher knowledge distillation module is used to perform dual-teacher knowledge distillation training based on the previous round of personalized model and updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

6. A personalized federated learning system based on dual-teacher knowledge distillation, characterized in that: Executed by the server, including: The data sending module is used to send the global model to each client in the previous round. For any client, the client obtains the sparse vector and personalized model parameters based on the local data and uploads them to the server; the sparse vector is obtained based on the local data, specifically including: inputting the local data into the personalized model, extracting the feature vector in the mapping channel to calculate the proportion of zero elements, averaging the proportion of zero elements to obtain the sparse value; and combining the sparse values ​​collected from multiple channels into a sparse vector; The association and sparse matrix calculation module is used to construct a coefficient matrix based on sparse vectors; specifically, it includes: calculating the Euclidean distance based on the sparse vectors of the two clients, collecting the Euclidean distances between two different participants to obtain the association matrix, and constructing the coefficient matrix based on the association matrix: in, is the coefficient matrix; and Extract from the incidence matrix, Indicates the participants i With the participants j A Euclidean distance of a sparse vector, represents the Euclidean distance between the sparse vector of participant i and any other participant except i; M is the number of clients; A personalized and global model aggregation module, used to aggregate the personalized model parameters sent by each client using a coefficient matrix to obtain an updated global model; The updated global model sending module is used to send the updated global model to each client in this round, so that each client can perform dual-teacher knowledge distillation training based on the personalized model of the previous round and the updated global model. After multiple rounds of knowledge distillation training iterations, the final personalized model is obtained.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a personalized federated learning method based on dual-teacher knowledge distillation as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a personalized federated learning method based on dual-teacher knowledge distillation according to any one of claims 1 to 4 when executed.

Images