A Personalized Federated Learning Method and System Based on Collaboration of Similar Features

Abstract

This invention discloses a personalized federated learning method and system based on similar feature collaboration, applied to a client and a server. The server manages at least two clients, each with a local model and a local dataset. The method includes: (1) the server receives local class-level feature vector sets sent by each client; (2) the server generates personalized class-level feature vector sets for each client based on the aggregate weights of the local class-level feature vector sets; (3) each client downloads its corresponding personalized class-level feature vector set from the server, trains and updates its local model on its local dataset, and obtains a local class-level feature vector set; (4) steps (2)-(3) are repeated until the average training loss of all clients no longer decreases. This invention can improve the performance degradation of local models caused by data heterogeneity, while also addressing issues such as model heterogeneity, communication costs, and interpretable participant contribution evaluation.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a personalized federated learning method and system based on collaboration of similar features. Background Technology

[0002] Federated Learning (FL) is essentially a distributed machine learning framework that enables data sharing and collaborative modeling while ensuring data privacy, security, and legal compliance. Its core idea is that when multiple data sources participate in model training, the original data can be transferred without requiring further processing; instead, the model can be jointly trained by exchanging intermediate parameters, and the original data can remain locally available.

[0003] Due to the heterogeneity of data and models, traditional FL still faces some practical challenges, and effective algorithms to overcome these challenges have not been fully developed or systematically studied. Existing solutions to the data heterogeneity problem include: (1) allocating and maintaining multiple global models for different local clients, such as clustering FL; (2) generating personalized models for each client using global and local information, such as personalized FL. Most of these methods rely on gradient-based aggregation, resulting in high communication costs and heavy reliance on local models with the same structure. Model heterogeneity is common because different clients have different hardware and computing capabilities. Knowledge distillation-based FL addresses this by transferring knowledge from the teacher model to student models with different model structures. However, these methods require an additional public dataset to align the outputs of student and teacher models, thus increasing computational costs. Furthermore, the performance of this method decreases significantly with the increasing distributional differences between the public dataset and the client dataset.

[0004] Furthermore, clients participating in federated learning may have additional needs beyond training useful FL models. For example, a pharmaceutical company might want to build a model that uses FL to leverage data from multiple hospitals to advance drug research. To compensate participating hospitals, the pharmaceutical company might need to provide incentive payments.

[0005] Therefore, in existing technologies, how to address the performance degradation of local models in federated learning under heterogeneous data scenarios, while taking into account model heterogeneity, communication costs, and interpretable evaluation of participant contributions, is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a personalized federated learning method and system based on similar feature collaboration, which improves the performance degradation of local models caused by data heterogeneity among clients in federated learning, while also taking into account problems such as model heterogeneity, communication costs, and interpretable evaluation of participant contributions.

[0007] The technical solution of the present invention is as follows:

[0008] A personalized federated learning method and system based on similar feature collaboration is applied to a client and a server. The server manages at least two clients, each running a local model and a local dataset for training the local model. The method includes:

[0009] (1) The server initializes and receives the local class-level feature vector sets sent by each client;

[0010] (2) The server generates a personalized class-level feature vector set for each client based on the aggregate weight of each client, according to the local class-level feature vector set sent by each client.

[0011] (3) Each client downloads the corresponding personalized class-level feature vector set from the server, and updates its local model on its local dataset by minimizing the sum of the local classification error and the distance between the local class-level feature vector set and the personalized class-level feature vector set; the updated local model obtains its updated local class-level feature vector set and sends it to the server;

[0012] (4) Repeat steps (2)-(3) until the average training loss of all clients no longer decreases.

[0013] On a certain client, its local dataset is input into its local model. The output of a fully connected layer (such as the penultimate fully connected layer) responsible for decision-making in the backend of the classification network is used as the feature vector of a certain class of the client. The feature vectors of all data samples belonging to this class are averaged to obtain the local class-level feature vector of this class.

[0014] A typical classification model consists of two parts: feature extraction and classification prediction. The feature extraction part usually comprises convolutional layers and downsampling, while the classification prediction part typically consists of fully connected layers, with the output dimension of the last fully connected layer having the same number of classes as the number of classes. This invention uses the output of a specific fully connected layer responsible for decision-making in the classification network's backend (e.g., the penultimate fully connected layer) as the feature vector for a particular class of the client. If multiple samples from the same client belong to the same class, the average of these class feature vectors is used as the final feature vector for that class, defined as a local class-level feature vector. The number of local class-level feature vectors corresponds to the number of classes a client has.

[0015] The fully connected layers used by each client to extract local class-level feature vectors have the same output size. Furthermore, given that the fully connected layers used by each client to extract local class-level feature vectors have the same output size, the local classification model structure does not need to be completely identical.

[0016] On the server, the local class-level feature vectors of all clients with samples of the same class are aggregated based on the aggregation weight of each client to obtain the personalized class-level feature vector of each client.

[0017] If multiple different clients have samples of the same class, assuming clients A, B, and C all have class 1, then taking client A as an example, the output obtained by weighted aggregation of the local class-1 feature vectors of clients A, B, and C through rules is called the personalized class-1 feature vector of client A. Similarly, the personalized class-1 feature vectors of clients B and C can be obtained.

[0018] Step (2) includes:

[0019] (i) For client i, calculate the local class-level feature vector of its class j. Local class-level feature vectors on other clients containing class j Distance between k∈[1,M (j) ], to obtain the distance vector Where: j∈[1,|C i |],|C i | represents the number of classes on client i; M (j) Indicates the number of clients containing class j; Set as a constant, it can be fixed (e.g., 1) or variable (e.g., the initial value is 1, and it gradually decreases). It affects the proportion of features of client i itself. The smaller the value, the greater the proportion.

[0020] (ii) For distance vectors Take distance The weight vector is obtained by inversely proportionalizing and normalizing the weight. in: Let represent the aggregate weight of client k with respect to class j of client i, and have .

[0021] (iii) Calculate the personalized class-level feature vector of class j on client i. The calculation formula is:

[0022]

[0023] In step (i), the distance For Euclidean distance, Manhattan distance, and cosine distance.

[0024] The formula for calculating Euclidean distance is: Where k∈[1,M] (j) ];

[0025] The formula for calculating Manhattan distance is: Where k∈[1,M] (j) ];

[0026] The formula for calculating the cosine distance of the included angle is: Where k∈[1,M] (j) ].

[0027] In step (3), the local training loss L of client i i for:

[0028]

[0029] Where: w i For the parameters of the local model of client i; x i For input data; y i For x i Corresponding label; F i It is x i The classification prediction obtained after inputting the local model; |D i | is the amount of local data for client i; |D i,j | represents the number of data points containing class j in the local dataset; N is the total number of samples across all clients; N j It is the total number of samples belonging to class j across all clients; and These are the local class-level feature vector and the personalized class-level feature vector of class j on client i, respectively; |C i | represents the number of classes on client i; λ is the regularization constant.

[0030] The optimization goal for federated learning across all clients is:

[0031]

[0032] Where N is the total number of samples across all clients; N j M is the total number of samples belonging to class j across all clients; M is the total number of clients.

[0033] The present invention also provides a personalized federated learning system based on similar feature collaboration, applied to a client and a server. The server is used to manage at least two clients, each client running a local model and a local dataset for training the local model. The system includes a training module, an interaction module, an aggregation module, and a visualization module.

[0034] The training module is applied to the client and updates its local model on the local dataset of each client by minimizing the sum of the distances between the local classification error and the local class-level feature vector set and the personalized class-level feature vector set; the updated local class-level feature vector set is obtained from the updated local model.

[0035] The interaction module is used for transmitting local class-level feature vector sets from each client to the server, and for transmitting corresponding personalized local class-level feature vector sets from the server to each client.

[0036] The aggregation module is applied to the server. It calculates the distance between the local class-level feature vectors of each client to obtain the aggregation weight of each client, and generates a personalized class-level feature vector set for each client based on the aggregation weight of each client.

[0037] The visualization module visualizes the aggregated weights during the training process, and is used to dynamically display and quantitatively evaluate the degree of contribution of each client and the changes in the degree of contribution.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. By calculating the distance between local class-level feature vectors of the same category from different clients, and further calculating the aggregation weight based on the distance, a personalized local class-level feature vector for each client is finally obtained based on the aggregation weight. During client-side local training, by narrowing the distance between the local class-level feature vectors and the personalized class-level feature vectors, collaboration between clients with similar class-level feature distributions is promoted, improving the classification performance of local models for each client in heterogeneous data scenarios and enhancing the applicability of the local models.

[0040] 2. Each client interacts with the server through a fixed-length feature vector. Compared with the traditional FL, which uses all parameters of the local model for interaction, this reduces the requirement for local models with the same structure, saves communication costs, and improves communication efficiency.

[0041] 3. By setting up a visualization module, the aggregate weights of each client during the training process can be visualized, which can dynamically display and quantitatively evaluate the contribution of each client's data and the changes in the contribution. This provides a class-level, interpretable quantitative evaluation reference for the contribution of FL participants, and offers a new approach for incentive payments in federated learning. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall framework of a personalized federated learning method and system based on similar feature collaboration according to the present invention.

[0043] Figure 2This is a schematic diagram of the local model.

[0044] Figure 3 A visualization of aggregated weights. Detailed Implementation

[0045] This invention proposes a personalized federated learning method and system based on similar feature collaboration. The key is how to generate personalized class-level aggregation weights. An example scheme is as follows:

[0046] Concept Explanation:

[0047] A typical classification model includes feature extraction and classification prediction. The feature extraction part usually consists of convolutional layers and downsampling, while the classification prediction part usually consists of fully connected layers, such as... Figure 2 As shown, the last fully connected layer ( Figure 2 The output dimension and number of categories of FC2 are the same, so the second to last fully connected layer can be used. Figure 2 The output of FC1 in the algorithm is used as the feature vector of a certain class of the client. The selection of feature vectors can be extended, but we will use this as an example for now. If multiple samples from the same client belong to the same class, the average of the feature vectors is used as the feature vector for that class, and this feature vector is defined as a local class-level feature vector. The number of local class-level feature vectors corresponds to the number of classes a client has.

[0048] If multiple different clients have samples of the same class, assuming clients A, B, and C all have class 1, then taking client A as an example, the output obtained by weighting and aggregating the local class-1 feature vectors of clients A, B, and C according to a certain rule is called the personalized class-1 feature vector of client A. Similarly, the personalized class-1 feature vectors of clients B and C can be obtained.

[0049] The overall process is as follows Figure 1 As shown, it mainly includes the following two key iterative steps:

[0050] (1) Client-side local update: Each client downloads its corresponding personalized class-level feature vector set from the server and updates its local dataset by minimizing the local training loss L. i (including classification error L) S The distance L between local class-level feature vectors and personalized class-level feature vectors R To update their local models w i Then, each client sets its local class-level feature vectors C i Send to the central server;

[0051] Local training loss L of client i i The definition is as follows:

[0052]

[0053] Among them, w i Let x be the local model parameter of client i. i For input data, y i For the corresponding label, F i It is x i Enter w i The subsequent classification prediction, |D i |D represents the amount of local data for client i. i,j |for|D i The number of data points containing class j, where N is the total number of samples across all clients. j It is the total number of samples belonging to class j on all clients. and These are the local feature vector and personalized feature vector of class j on client i, respectively, |C i | represents the number of classes on client i, and λ is the regularization constant.

[0054] The optimization objective for federated learning across all clients is defined as follows:

[0055]

[0056] Where N is the total number of samples across all clients, N j This represents the total number of samples belonging to class j across all clients, where M is the number of clients. The other parameters are explained as above.

[0057] (2) Server-level feature aggregation: The central server generates personalized class-level feature vectors for each client based on the aggregation weights and returns them to the corresponding clients to promote collaboration between similar clients.

[0058] Taking class j on client i as an example, a method for calculating aggregate weight is given. First, calculate... Local class-level feature vectors on other clients containing class j The distance between them (calculated using vector distance methods, such as Euclidean distance, Manhattan distance, cosine distance, etc.), i.e.:

[0059] European distance: Where k∈[1,M] (j) Equation (6)

[0060] Manhattan distance: Where k∈[1,M] (j) ]

[0061] Cosine distance of the included angle: Where k∈[1,M] (j) ]

[0062] Where i,k represent client numbers, and M (j) Let j represent the number of clients containing class j. Then, we have the distance vector:

[0063] Where j∈[1,|C i Equation (7)

[0064] in, Set to a constant. Considering that the smaller the distance, the closer the features are, and therefore the larger the aggregation weight, the weight vector is obtained by taking the inverse ratio of the distance and normalizing it, i.e.:

[0065] Where j∈[1,|C i Equation (8)

[0066] in, Let represent the aggregate weight of client k with respect to class j of client i, and have . Then the personalized feature vector of class j on client i is:

[0067]

[0068] Aggregate weights can serve as a reference for evaluating the contributions of clients during training, such as... Figure 3 As shown, the horizontal axis represents the client number, and the vertical axis represents the aggregation weight. Figure 3 In the diagram, (a) and (b) represent the aggregate weight distribution of the personalized feature vector of category 1 (label_0) on client 1 after the first and 101st rounds of communication (round_0 and round_100), respectively. Figure 3 It can be seen that after the first round, Client 1 itself has a weight of 0.40, and Client 19 (client_18) has a weight of 0.06; after the 101st round, Client 1 itself has a weight of 0.16, and Client 19 has a weight of 0.11. As the number of communication rounds increases, the weight of the client itself gradually decreases, while the weights of other clients gradually increase, and the weight differences of other clients are not significant. This indicates that the feature vectors of this category on these clients are relatively similar, and they have achieved a win-win situation through cooperation. Furthermore, by monitoring the weight increment of other clients on the current client, it is possible to promptly understand whether further communication and cooperation are necessary.

[0069] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A personalized federated learning method based on similar feature collaboration, characterized in that, The method is applied to a client and a server, the server is used for managing at least two clients, each client runs a local model and a local data set used for training the local model, and the method comprises the following steps: (1) the server initializes and receives a local class-level feature vector set sent by each client; on a certain client, a local data set of the client is input into a local model of the client, an output of a certain fully connected layer responsible for decision-making in a back end of a classification network is taken as a feature vector of a certain class of the client, feature vectors of all data samples belonging to the class are averaged, and a local class-level feature vector of the class is obtained; (2) the server generates a personalized class-level feature vector set of each client based on an aggregation weight of each client according to the local class-level feature vector set sent by each client; on the server, the local class-level feature vectors of all clients with the same class samples are aggregated based on the aggregation weight of each client, and a personalized class-level feature vector of each client is obtained; (3) each client downloads the corresponding personalized class-level feature vector set from the server, and updates the local model on the local data set by minimizing the sum of the local classification error and the distance between the local class-level feature vector set and the personalized class-level feature vector set; the updated local model is used to obtain an updated local class-level feature vector set of the client and send the updated local class-level feature vector set to the server; (4) steps (2)-(3) are repeated until the average training loss of all clients no longer decreases.

2. The personalized federated learning method based on similar feature cooperation according to claim 1, wherein, The output dimensions of the fully connected layer where the local class-level feature vectors are located are the same.

3. The personalized federated learning method based on similar feature collaboration according to claim 1, wherein, Step (2) comprises: (i) For client i, compute its local class-level feature vector for class j distance between the local class-level feature vector on client i and the local class-level feature vector on other clients containing class j get distance vector where: j e [1, |C i |], |C i | is the number of classes on client i; M (j) is the number of clients containing class j; is set to a constant;​ (ii) for distance vectors Taking the inverse of the distance and normalizing, we obtain the weight vector where: denotes the aggregate weight of client k for class j of client i, and has (iii) computing a personalized class-level feature vector for class j on client i The formula is:

4. The personalized federated learning method based on similar feature cooperation according to claim 3, characterized in that, In step (i), the distance is the Euclidean distance, the Manhattan distance, or the cosine of the angle distance.

5. The personalized federated learning method based on similar feature cooperation according to claim 4, wherein: The calculation formula of the Euclidean distance is: Wherein k ∈ [1, M (j) ]; The Manhattan distance is calculated by the following formula: where k ∈ [1, M (j) ]. The formula for calculating the cosine angle distance is: where k ∈ [1, M (j) ].

6. The personalized federated learning method based on similar feature cooperation of claim 1, wherein, In step (3), the local training loss L of the client i is: i is: Where: w i For the parameters of the local model of client i; x i For input data; y i For x i Corresponding label; F i It is x i The classification prediction obtained after inputting the local model; |D i | is the amount of local data for client i; |D i,j | represents the number of data points containing class j in the local dataset; N is the total number of samples across all clients; N j It is the total number of samples belonging to class j across all clients; and These are the local class-level feature vector and the personalized class-level feature vector of class j on client i, respectively; |C i | represents the number of classes on client i; λ is the regularization constant.

7. The personalized federated learning method based on similar feature collaboration of claim 1, wherein, the optimization objective of all the clients for federated learning is: where N is the total number of samples across all clients; N j is the total number of samples belonging to class j across all clients; and M is the total number of clients.

8. A personalized federated learning system based on similar feature collaboration, characterized in that, The system is applied to a client and a server, the server is used for managing at least two clients, each client runs a local model and a local data set used for training the local model, and the system comprises a training module, an interaction module, an aggregation module and a visualization module; the training module is applied to the client, and the local model of each client is updated on the local data set of each client by minimizing the sum of the local classification error and the distance between the local class-level feature vector set and the personalized class-level feature vector set; the updated local model is used to obtain an updated local class-level feature vector set of the client; on a certain client, a local data set of the client is input into a local model of the client, an output of a certain fully connected layer responsible for decision-making in a back end of a classification network is taken as a feature vector of a certain class of the client, feature vectors of all data samples belonging to the class are averaged, and a local class-level feature vector of the class is obtained; the interaction module is applied to transmitting the local class-level feature vector set from each client to the server and transmitting the corresponding personalized local class-level feature vector set from the server to each client; The polymeric module is applied to a server, and the distance between the local class-level feature vectors of each client is calculated to obtain the aggregation weight of each client, and the personalized class-level feature vector set of each client is generated based on the aggregation weight of each client; on the server, the local class-level feature vectors of all clients with the same class sample are aggregated based on the aggregation weight of each client to obtain the personalized class-level feature vector of each client; The visualization module visualizes the aggregation weight in the training process, and is used for dynamically displaying and quantitatively evaluating the contribution degree and the change of the contribution degree between the clients.

Images