Personalized federal learning method and system for realizing feature alignment based on knowledge distillation

By using knowledge distillation technology in federated learning, the client uses the difference between the global feature extractor copy and the local model to calculate the loss during local training, optimizes the feature extractor, solving the problems of data heterogeneity and communication overhead, and improving the problem of improvement The generalization ability and efficiency of the model are achieved.

CN119005299AActive Publication Date: 2024-11-22HOHAI UNIV

Patent Information

Application Number
CN202411066474.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

The client drift problem caused by data heterogeneity in federated learning slows down the convergence speed and performance degradation of the model. The existing feature alignment methods require additional global feature information transmission, increasing communication overhead and privacy leakage risks.

Method used

Using knowledge distillation technology, the client can use the difference between the global feature extractor copy and the local model to calculate the distillation loss and classification loss during local training, optimize the local feature extractor, avoid transmitting additional feature information, and realize fine-grained features. Align.

Benefits of technology

It improves the generalization capability of local feature extractors, reduces model differences, promotes global aggregation, reduces communication overhead and privacy leakage risks, and improves the efficiency and scalability of federated learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005299A_ABST
    Figure CN119005299A_ABST
Patent Text Reader

Abstract

The invention discloses a personalized federal learning method and system for realizing feature alignment based on knowledge distillation, and the method comprises the steps: carrying out the distillation training of a client through the local data, inputting a sample into a global feature extractor and a local feature extractor at the same time, and obtaining the global features and local features of the sample; measuring the difference between the global features and the local features by using mean square error loss; by minimizing classification loss and distillation loss at the same time, the local feature extractor learns global feature information and local feature information at the same time, and the classifier learns local information; after local training is completed, each client uploads a local feature extractor to a server, and a classifier head is kept locally; the server aggregates the local feature extractors uploaded by the clients to obtain a new round of global feature extractor; the process is repeated until the model converges or reaches the specified model precision. According to the invention, the precision of the personalized model of the client is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of distributed computing and deep learning technology, and in particular to a personalized federated learning method and system for realizing feature alignment based on knowledge distillation. Background Art

[0002] In traditional centralized machine learning, data is usually concentrated in one place for training, which may involve uploading the data set to the cloud or a central server. However, this approach poses risks of privacy leakage and data security, especially when the data contains personally identifiable information or sensitive business data. In addition, when the data set is too large, transmitting and processing data will become very expensive and inefficient, and transmitting data will also bring very large communication overhead.

[0003] In recent years, with the popularity of mobile devices and the rise of edge computing, federated learning has received increasing attention. Federated learning achieves the goal of learning global models on multiple devices by training models on local devices and only sharing the gradients or parameters of model updates. Federated learning allows data to always be saved on the user's local device without having to transfer data to the cloud or server. This way of data storage and processing can effectively protect data privacy and reduce the risk of data leakage. Currently, federated learning has been applied to many real-world scenarios, such as recommendation systems, healthcare, finance, etc.

[0004] Although federated learning has made remarkable achievements in solving data privacy and security problems, it still faces some challenges, one of the most important of which is the data heterogeneity of each participant. Due to the unique training mode of federated learning, data is often generated by the client side. The data of these participants is often affected by factors such as their users, scenarios, preferences, etc., resulting in different data distributions for each participant. Therefore, non-independent and identically distributed data is a natural problem in federated learning. Data heterogeneity can lead to the phenomenon of "client drift", that is, the local update direction of the client deviates from the global update direction, which is caused by the inconsistency between the local optimization goal of the client and the global optimization goal, which will cause the model to converge more slowly and performance to degrade. Due to the different distribution of global and local data, the single global model obtained by using federated learning algorithms such as the classic Federated Average Algorithm (FedAvg) is not suitable for each client.

[0005] Therefore, in order to solve the difficulty of training a single global model, personalized federated learning was proposed. This approach is committed to building a personalized model for each client that conforms to its data distribution. Common methods include model regularization, data augmentation, and local fine-tuning. In addition, model decoupling is also an important research direction. The success of multi-task learning and representation learning shows that it is an effective strategy to decouple the model into a feature extractor responsible for extracting low-dimensional features and a classifier that processes tasks that are strongly related. In personalized federated learning, the feature extractor is trained jointly by all clients to learn a common representation, while the classifier is trained privately to complete local classification tasks. Figure 1 The training process of the model decoupling method is shown. First, the server sends the parameters of the global feature extractor to the client. After receiving the parameters, the client applies them to the local model. Then the client uses its local data to train the local model. After the training is completed, the client only uploads the parameters of the local feature extractor. After all clients have completed uploading, the server aggregates the feature extractor parameters of all clients to obtain a new round of global feature extractor parameters. This process is repeated until the model converges. However, local training of feature extractors can usually only learn local personalized feature information, while ignoring global feature information, which may cause the local feature extractor to deviate from the global feature standard and lose universality, thus affecting the aggregation effect of the global model. In addition, sharing feature extractors only at the parameter level is not enough to obtain universal features from heterogeneous data. Some recent studies have proposed to align the local features of local samples with the global features on the basis of sharing feature extractors, and learn additional universal features at the feature level. For example, FedPAC (Personalized Federated Learning with Feature Alignment and Classifier Collaboration) proposes to align local features with the centroid of global features. GPFL (GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning) proposes to align local features with global category embeddings to introduce global feature information into local training. However, both methods require communicating additional global feature information (global feature centroid or global category embedding) with the server, which will bring additional privacy and communication overhead issues, and cannot provide fine-grained feature guidance for the client's feature extractor. Summary of the invention

[0006] Purpose of the invention: The present invention proposes a personalized federated learning method for feature alignment based on knowledge distillation. The method realizes the effective transmission of feature information through knowledge distillation technology, so that the client can learn personalized and global feature information at the same time, which improves the generalization ability of local feature extractors to a certain extent. In addition, there is no need to transmit additional feature information between the client and the server, only the feature extractor parameters need to be transmitted, which can avoid additional communication overhead and privacy leakage problems. In addition, the method can also achieve fine-grained feature alignment, effectively limit the diversity of local feature extractors and promote global aggregation, which allows the client to run more local updates and learn universal representations in a communication-efficient manner.

[0007] The present invention also provides a personalized federated learning system for realizing feature alignment based on knowledge distillation.

[0008] Technical solution: In order to achieve the above invention objectives, the technical solution of the present invention is as follows:

[0009] A personalized federated learning method for feature alignment based on knowledge distillation includes the following steps:

[0010] The server sends the global feature extractor parameters to the client;

[0011] After receiving the global feature extractor parameters, the client saves a copy of the global feature extractor parameters and uses the parameters to overwrite the local feature extractor parameters;

[0012] The client uses local data for distillation training. For each training sample, the client uses the saved copy of the global feature extractor to obtain the global features, uses the local model to obtain the local features and predicted outputs, and obtains the distillation loss based on the difference between the global features and the local features. The client also obtains the classification loss based on the difference between the predicted output and the actual value. The client combines the distillation loss and the classification loss to obtain the total loss, and optimizes the local model by minimizing the total loss.

[0013] After local training is completed, each client uploads its own local feature extractor parameters to the server;

[0014] After receiving the local feature extractor parameters from the client, the server determines the aggregation weight according to the amount of data owned by each client, aggregates the local feature extractors of each client based on the aggregation weight, and obtains a new round of global feature extractors;

[0015] The above process is iterated until the model converges or reaches the specified model accuracy.

[0016] Preferably, for each training sample, obtaining a global feature using a saved copy of the global feature extractor includes:

[0017] For a single training sample x m , the client uses the saved copy of the global feature extractor to process it and obtain the global feature representation Among them, φ g represents the global feature extractor, It is represented by φ g Parameterized functions, is a vector processed by the function, containing the sample x m Global feature information of

[0018] Use local models to obtain local features and prediction outputs, including:

[0019] For a single training sample x m , the client uses the local model θ i Process the sample to obtain the local feature f φi (x m ) and predicted output Among them, the local model θ i By local feature extractor φ i and local classifier X i Composition, i represents the i-th client, is i Parameterized functions, Represents a vector after function processing, containing sample x m The local feature information of Represents the local classifier X i For sample x m prediction results.

[0020] Preferably, the distillation loss calculation method is as follows:

[0021]

[0022] in, represents the distillation loss of the i-th client, n i is the number of training samples for the i-th client.

[0023] Preferably, the classification loss is calculated as follows:

[0024]

[0025] in, represents the classification loss of the i-th client, y ij is the sample x m The probability of the true label belonging to category j, is the sample x predicted by the local model m The probability of a label belonging to class j.

[0026] Preferably, the total loss is obtained by combining the distillation loss and the classification loss, and the local model is optimized by minimizing the total loss, which is expressed as follows:

[0027]

[0028] Among them, β is the balanced classification loss and distillation losses is a hyperparameter that controls the degree of knowledge transferred from the global feature extractor to the local feature extractor.

[0029] Preferably, the method for the server to aggregate the local feature extractors of each client is as follows:

[0030]

[0031] in, is the global feature extractor of the t+1th global round, is the local feature extractor of the i-th client in the t-th global round, α i is the aggregation weight, k is the total number of clients that upload local feature extractors, n i is the amount of data owned by the i-th client.

[0032] The present invention also provides a personalized federated learning method system for implementing feature alignment based on knowledge distillation, comprising a server and a plurality of clients, wherein the server and the client are configured to iteratively perform the following process until the model converges or reaches a specified model accuracy:

[0033] The server sends the global feature extractor parameters to the client;

[0034] After receiving the global feature extractor parameters, the client saves a copy of the global feature extractor parameters and uses the parameters to overwrite the local feature extractor parameters;

[0035] The client uses local data for distillation training. For each training sample, the client uses the saved copy of the global feature extractor to obtain the global features, uses the local model to obtain the local features and predicted outputs, and obtains the distillation loss based on the difference between the global features and the local features. The client also obtains the classification loss based on the difference between the predicted output and the actual value. The client combines the distillation loss and the classification loss to obtain the total loss, and optimizes the local model by minimizing the total loss.

[0036] After local training is completed, each client uploads its own local feature extractor parameters to the server;

[0037] After receiving the local feature extractor parameters from the client, the server determines the aggregation weight according to the amount of data owned by each client, aggregates the local feature extractors of each client based on the aggregation weight, and obtains a new round of global feature extractors.

[0038] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the programs are executed by the processors, the following steps are implemented:

[0039] Send the global feature extractor parameters of this round to the client; and

[0040] After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors;

[0041] Alternatively, when the program is executed by a processor, the following steps are implemented:

[0042] Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters;

[0043] Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss.

[0044] After local training is completed, upload your own local feature extractor parameters to the server.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0046] Send the global feature extractor parameters of this round to the client; and

[0047] After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors;

[0048] Alternatively, when the computer program is executed by a processor, the following steps are implemented:

[0049] Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters;

[0050] Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss.

[0051] After local training is completed, upload your own local feature extractor parameters to the server.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] (1) The present invention proposes a novel personalized federated learning method, which introduces knowledge distillation technology into personalized federated learning to achieve effective transmission of feature information. Through knowledge distillation, the client can obtain rich feature knowledge from the global feature extractor and then apply this knowledge to the training of the local model. In this way, the client can not only train a personalized model based on local data, but also improve the generalization ability of the feature extractor with the help of global feature information. The client is allowed to learn personalized and global feature information at the same time during the training process. In this way, not only can the problems caused by data heterogeneity be solved, but also the generalization ability of the local feature extractor can be effectively improved, thereby further improving the performance of federated learning. (2) The present invention realizes fine-grained feature alignment, which can make the local feature extractor more consistent during the training process and reduce the differences between models. In this way, when the client performs local updates, the learned features are more consistent and more representative, and can better reflect the characteristics of the global data. At the same time, by promoting global aggregation, the feature information learned by each client can be effectively integrated to form a more complete and accurate global representation. The invention allows the client to perform more local updates in a communication-efficient manner, making better use of resources in a distributed environment and accelerating the convergence of the model. This not only improves the efficiency of federated learning, but also reduces communication overhead, making the invention more feasible and scalable in practical applications. (3) The invention opens up new opportunities for solving non-independent and identically distributed data in federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a training flowchart of the model decoupling method in general personalized federated learning;

[0055] Figure 2It is a general schematic diagram of knowledge distillation;

[0056] Figure 3 is a schematic diagram of client knowledge distillation according to the present invention;

[0057] Figure 4 This is a training flowchart of a personalized federated learning method for implementing feature alignment based on knowledge distillation according to the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0059] Knowledge distillation is a model compression technique that aims to transfer the "knowledge" of one model (usually a larger and more complex model) to another model (usually a smaller and simplified model) so that the latter can learn the "knowledge" of the former while maintaining or improving its performance. This method was originally proposed by Hinton et al. and has been widely used in the field of deep learning. In knowledge distillation, there are usually two models involved: the teacher model and the student model. The goal of knowledge distillation is to guide the learning of the student model by having the student model imitate the output or intermediate layer features of the teacher model. During the training process of knowledge distillation, the student model must minimize both the true label loss and the distillation loss in the optimization process in order to learn the "knowledge" of the teacher model. Figure 2 Schematic diagram of knowledge distillation.

[0060] The present invention divides the local model into a feature extractor and a classifier on the federated learning client. According to the federated learning mechanism, the client obtains the local feature extractor parameters of the client in each round of training, sends them to the server, and the server aggregates them to form a global feature extractor, which is sent to the client in the next round of iteration. In the context of the present invention, the global feature extractor is the shared feature extractor parameter distributed to all clients after aggregation by the server, and the local feature extractor is the feature extractor parameter of each client after personalized training on local data based on the global feature extractor parameter. The present invention combines the distillation technology, regards the global feature extractor as the teacher model, and the local feature extractor as the student model. Figure 3 The schematic diagram of local distillation is shown in Figure 2. By transferring knowledge from the global feature extractor to the local feature extractor, the client can learn both personalized and global feature information during the training process. This can not only solve the problem caused by data heterogeneity, but also effectively improve the generalization ability of the local feature extractor, thereby further improving the performance of federated learning.

[0061] Reference Figure 4 The personalized federated learning method for feature alignment based on knowledge distillation proposed in the present invention specifically includes the following steps:

[0062] Step 1: The server sends the global feature extractor parameters to the selected client.

[0063] At the beginning of each round of training, the server sends the parameters of the global feature extractor to the selected client. The parameters of the global feature extractor are the aggregation of the local feature extractors obtained by the previous round of client training. If it is the first round of global iteration, all model parameters initialized by the server are sent, including feature extractor parameters and classifier parameters. After this round, only feature extractor parameters are sent to the client.

[0064] Step 2: The client applies the global feature extractor parameters locally.

[0065] For clients that receive global feature extractor parameters, use the global feature extractor parameters to overwrite local feature extractor parameters to ensure that all clients start personalized training from the same feature extractor model. Ensure that the data is not damaged during transmission. The architecture of the received global feature extractor can be compared with the architecture of the local feature extractor model. If they are the same, the data is considered not damaged. Otherwise, it is considered damaged. If there is damage, the client needs to request the server to obtain the global feature extractor parameters. If there is no damage, the client saves a copy of the global feature extractor parameters locally, then parses the global feature extractor, including the architecture information and specific parameter data of the feature extractor, and then reads the local feature extractor parameters, and assigns the parameters of the global feature extractor to the corresponding parameters of the local feature extractor one by one.

[0066] Step 3: The client performs local training using local data.

[0067] In the embodiment of the present invention, the image recognition model is trained by sending the image training set to the network and adjusting the parameters according to the difference between the actual output of the network and the expected output. The steps of training the model are as follows:

[0068] a. The client divides the local model (θ) into a feature extractor (φ) and a classifier (χ), where χ is the last fully connected layer. The feature extractor is responsible for extracting high-dimensional features from the input data and consists of multiple convolutional layers or fully connected layers. The classifier uses the features extracted by the feature extractor for classification and is generally composed of several fully connected layers. The last layer outputs the category probability. Here we call the last fully connected layer the classifier, and all layers outside the classifier are called feature extractors.

[0069] The local model (θ) is a general concept, which refers to the part of the model that extracts features from the input data. It consists of a feature extractor (φ) and a classifier (χ). During model training, the feature extractor is mainly divided into a global feature extractor (φ g) and local feature extractor (φ i ), as mentioned above, the global feature extractor is the shared feature extractor parameters that are aggregated by the server and distributed to all clients. The client will save a copy of the global feature extractor parameters locally; the local feature extractor is the feature extractor parameters of each client after personalized training on local data based on the global feature extractor parameters. The feature extractor of client i is represented by φ i , the global feature extractor is denoted as φ g . Local model (θ i ) is extracted by the local feature extractor (φ i ) and local classifier (χ i )composition.

[0070] Remember f φ is a function parameterized by φ, which maps data points from d dimensions to k-dimensional feature space, φ: is a function parameterized by χ that maps K-dimensional features to the label space χ: Therefore, the client's local loss function can be expressed as:

[0071] b. Select an image sample (x m ), x n is the image data, is the label, that is, the category to which the image belongs;

[0072] c. The image sample x n Input to the locally saved global feature extractor φ g The actual output of the feature extractor is calculated in the copy of That is, the global characteristics of the sample;

[0073] d. Image sample x m Input to the local model θ i In the local feature extractor φ i Get the local features of the sample After the local classifier χ i Get the sample x m Prediction probability

[0074] e. Calculate sample x m The difference between the global features and local features of is the distillation loss. The distillation loss here uses the mean square error loss. The distillation loss can be expressed as:

[0075]

[0076] f. Calculate the error between the predicted value and the actual value. The cross entropy loss function is used to measure the error between the model prediction output and the true label. The classification loss can be expressed as:

[0077]

[0078] where y ij is the sample x on the i-th client m The probability of the true label is is the probability of the label predicted by the local model on the i-th client;

[0079] g. Calculate the overall loss. The overall loss is the weighted sum of the distillation loss and the classification loss:

[0080]

[0081] Where β is the balanced local cross entropy loss and distillation losses is a hyperparameter that controls the degree of knowledge transferred from the global feature extractor to the local feature extractor; by minimizing the local loss The client can use local data to learn personalized headers, and can also explicitly align local features with global features. The local feature extractor can learn local and global feature information at the same time.

[0082] h. Use back propagation according to the loss Compute gradient information about model parameters

[0083] i. Optimize model parameters. Use the stochastic gradient descent optimization algorithm to update the model parameters:

[0084]

[0085] Where η is a hyperparameter that controls the update amplitude of the model parameters. represents the local model parameters after the jth local update of the kth client in the tth global round of training.

[0086] j. Repeat the above AI process for each image sample until the entire image sample set is traversed and one training iteration of the local data set is completed;

[0087] k. Repeat the above aj process, and the client completes multiple rounds of training iterations of the local data set.

[0088] Step 4: The client uploads its own feature extractor parameters.

[0089] After the local training is completed, each client uploads the trained feature extractor parameters to the central server. At this time, the classifier part of each client remains locally and is not uploaded.

[0090] Step 5: The server aggregates the client’s feature extractor parameters.

[0091] After receiving the feature extractor parameters uploaded by all clients, the server determines the aggregation weight according to the amount of data of each client. The larger the amount of data of the client, the greater the weight of its model parameters in the aggregation process. The server uses these weights to perform a weighted average of the feature extractor parameters of each client to obtain a new round of global feature extractors. The aggregation method is as follows:

[0092]

[0093] where n i is the amount of data owned by the i-th client, k is the total number of clients that uploaded the feature extractor, is the global feature extractor of the t+1th global round.

[0094] Step 6: Repeat steps 1 to 5 above until the model converges or reaches the specified model accuracy.

[0095] The present invention also provides a personalized federated learning method system for implementing feature alignment based on knowledge distillation, comprising a server and a plurality of clients, wherein the server and the client are configured to iteratively perform the following process until the model converges or reaches a specified model accuracy:

[0096] The server sends the global feature extractor parameters to the client;

[0097] After receiving the global feature extractor parameters, the client saves a copy of the global feature extractor parameters and uses the parameters to overwrite the local feature extractor parameters;

[0098] The client uses local data for distillation training. For each training sample, the client uses the saved copy of the global feature extractor to obtain the global features, uses the local model to obtain the local features and predicted outputs, and obtains the distillation loss based on the difference between the global features and the local features. The client also obtains the classification loss based on the difference between the predicted output and the actual value. The client combines the distillation loss and the classification loss to obtain the total loss, and optimizes the local model by minimizing the total loss.

[0099] After local training is completed, each client uploads its own local feature extractor parameters to the server;

[0100] After receiving the local feature extractor parameters from the client, the server determines the aggregation weight according to the amount of data owned by each client, aggregates the local feature extractors of each client based on the aggregation weight, and obtains a new round of global feature extractors.

[0101] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the programs are executed by the processors, the following steps are implemented:

[0102] Send the global feature extractor parameters of this round to the client; and

[0103] After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors;

[0104] Alternatively, when the program is executed by a processor, the following steps are implemented:

[0105] Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters;

[0106] Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss.

[0107] After local training is completed, upload your own local feature extractor parameters to the server.

[0108] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0109] Send the global feature extractor parameters of this round to the client; and

[0110] After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors;

[0111] Alternatively, when the computer program is executed by a processor, the following steps are implemented:

[0112] Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters;

[0113] Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss.

[0114] After local training is completed, upload your own local feature extractor parameters to the server.

[0115] Embodiments of the present invention may be provided as methods, apparatuses, computer devices or computer program products, and may be implemented in the form of complete hardware, complete software or a combination of hardware and software. The present invention may be included in a computer program product on a computer-usable storage medium (e.g., disk storage, CD-ROM, optical storage). These program instructions may direct a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable device to generate a device for performing the specified functions in the flowchart. The computer program instructions may be stored in a computer-readable memory or loaded onto a computer or other programmable device to implement the operating steps of the specified functions.

Claims

1. A personalized federated learning method for feature alignment based on knowledge distillation, characterized in that: The following steps are involved: The server sends the global feature extractor parameters to the client; After receiving the global feature extractor parameters, the client saves a copy of the global feature extractor parameters and uses the parameters to overwrite the local feature extractor parameters; The client uses local data for distillation training. For each training sample, the client uses the saved copy of the global feature extractor to obtain the global features, uses the local model to obtain the local features and predicted outputs, and obtains the distillation loss based on the difference between the global features and the local features. The client also obtains the classification loss based on the difference between the predicted output and the actual value. The client combines the distillation loss and the classification loss to obtain the total loss, and optimizes the local model by minimizing the total loss. After local training is completed, each client uploads its own local feature extractor parameters to the server; After receiving the local feature extractor parameters from the client, the server determines the aggregation weight according to the amount of data owned by each client, aggregates the local feature extractors of each client based on the aggregation weight, and obtains a new round of global feature extractors; The above process is iterated until the model converges or reaches the specified model accuracy.

2. The method according to claim 1, characterized in that For each training sample, use the saved copy of the global feature extractor to obtain the global features, including: For a single training sample x m , the client uses the saved copy of the global feature extractor to process it and obtain the global feature representation Among them, φ g represents the global feature extractor, It is represented by φ g Parameterized functions, is a vector processed by the function, containing the sample x m Global feature information of Use local models to obtain local features and prediction outputs, including: For a single training sample x m , the client uses the local model θ i Process the sample to obtain local features and the predicted output Among them, the local model θ i By local feature extractor φ i and the local classifier x i Composition, i represents the i-th client, is i Parameterized functions, Represents a vector after function processing, containing sample x m The local feature information of represents the local classifier χ i For sample x m prediction results.

3. The method according to claim 2, characterized in that The distillation loss is calculated as follows: in, represents the distillation loss of the i-th client, n i is the number of training samples for the i-th client.

4. The method according to claim 2, characterized in that: The classification loss is calculated as follows: in, represents the classification loss of the i-th client, y ij is the sample x m The probability of the true label belonging to category j, is the sample x predicted by the local model m The probability of a label belonging to class j.

5. The method according to claim 1, characterized in that Combining the distillation loss and the classification loss to get the total loss, the local model is optimized by minimizing the total loss, which is expressed as follows: Among them, β is the balanced classification loss and distillation losses is a hyperparameter that controls the degree of knowledge transferred from the global feature extractor to the local feature extractor.

6. The method according to claim 1, characterized in that The method by which the server aggregates the local feature extractors of each client is as follows: in, is the global feature extractor of the t+1th global round, is the local feature extractor of the i-th client in the t-th global round, α i is the aggregation weight, k is the total number of clients that upload local feature extractors, n i is the amount of data owned by the i-th client.

7. A personalized federated learning method system for feature alignment based on knowledge distillation, comprising a server and several clients, characterized in that: The server and the client are configured to iteratively perform the following process until the model converges or reaches a specified model accuracy: The server sends the global feature extractor parameters to the client; After receiving the global feature extractor parameters, the client saves a copy of the global feature extractor parameters and uses the parameters to overwrite the local feature extractor parameters; The client uses local data for distillation training. For each training sample, the client uses the saved copy of the global feature extractor to obtain the global features, uses the local model to obtain the local features and predicted outputs, and obtains the distillation loss based on the difference between the global features and the local features. The client also obtains the classification loss based on the difference between the predicted output and the actual value. The client combines the distillation loss and the classification loss to obtain the total loss, and optimizes the local model by minimizing the total loss. After local training is completed, each client uploads its own local feature extractor parameters to the server; After receiving the local feature extractor parameters from the client, the server determines the aggregation weight according to the amount of data owned by each client, aggregates the local feature extractors of each client based on the aggregation weight, and obtains a new round of global feature extractors.

8. A computer device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the programs are executed by the processors, the following steps are implemented: Send the global feature extractor parameters of this round to the client; as well as After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors; Alternatively, when the program is executed by a processor, the following steps are implemented: Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters; Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss. After local training is completed, upload your own local feature extractor parameters to the server.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the following steps are implemented: Send the global feature extractor parameters of this round to the client; as well as After receiving the local feature extractor parameters of the client, the aggregation weight is determined according to the amount of data owned by each client, and the local feature extractors of each client are aggregated based on the aggregation weight to obtain a new round of global feature extractors; Alternatively, when the computer program is executed by a processor, the following steps are implemented: Receive the global feature extractor parameters sent by the server, save a copy of the global feature extractor parameters, and use the parameters to overwrite the local local feature extractor parameters; Use local data for distillation training. For each training sample, use the saved copy of the global feature extractor to obtain the global features, use the local model to obtain the local features and predicted outputs, and get the distillation loss based on the difference between the global features and the local features. Get the classification loss based on the difference between the predicted output and the actual value. Combine the distillation loss and the classification loss to get the total loss, and optimize the local model by minimizing the total loss. After local training is completed, upload your own local feature extractor parameters to the server.

Citation Information

Patent Citations

  • Heterogeneous software defect prediction method based on federal prototype learning

    CN114896169A

  • Personalized federal learning method based on model and data distillation

    CN117035057A

  • Federal element learning fault diagnosis method used under non-independent identical distribution condition

    CN118296329A

  • De-centralised learning for re-indentification

    US20230087863A1

Cited By

  • Federal consistency representation learning system and method based on prototype guidance

    CN119229223A

  • Two-stage federated learning method and system based on width learning

    CN120430383A

  • Personalized global prototype federal learning method and system based on adaptive feature alignment

    CN120509507A

  • Unified clustering method for unpaired multi-view data

    CN120832536A

  • A unified clustering method for unpaired multi-view data

    CN120832536B