Personalized federal learning method based on conditional separation network and weight adaptation

Abstract

The invention relates to the technical field of federated learning personalization, in particular to a personalized federated learning method based on a conditional separation network and weight self-adaption, which comprises the following steps: each client calculates a prototype of each label based on a local data training model in a prototype learning mode so as to constrain the distance between local and global features; decoupling the features output by the feature extractor into global features and local features by using a conditional separation network, and respectively inputting the global features and the local features into a global classifier and a local classifier; the server calculates an aggregation weight according to the Fisher information matrix of each client and the cosine similarity between the parameter updating direction and the global direction of the Fisher information matrix, and then performs weighted aggregation on the client parameters by using the weight; and the server issues the aggregated parameters to the client. According to the method, the problems of model performance degradation and client drifting caused by data isomerism are effectively solved, the personalized model performance is improved, and meanwhile, the problem of client disastrous forgetting is relieved through feature decoupling.

Description

Technical Field

[0001] This invention relates to the field of personalized federated learning technology, applicable to scenarios where data is not independent and identically distributed, and particularly to a personalized federated learning method based on conditional separation networks and weight adaptation. Background Technology

[0002] Federated Learning (FL), as an emerging distributed machine learning paradigm, allows multiple clients to collaboratively train models without using their local data, effectively solving the problems of data silos and privacy protection. Its classic algorithm, FedAvg, generates a global model by aggregating model updates from various clients, laying the foundation for the federated learning framework. However, in practical applications, the data distribution of each client often exhibits significant heterogeneity, i.e., non-independent and identically distributed (NID) data. This statistical heterogeneity leads to slow model convergence and decreased accuracy, making it difficult for a single global model to achieve optimal performance on all clients, severely limiting the effectiveness of federated learning in real-world scenarios. In edge device scenarios, client data exhibits significant statistical heterogeneity: on the one hand, the data distribution varies greatly (e.g., different hospital case types in medical terminals, different user motion data distributions in smartwatches), i.e., Non-IID characteristics; on the other hand, the data scale is uneven (e.g., some industrial sensors collect only a small number of samples, while core nodes have hundreds of thousands of samples). This heterogeneity leads to a 15%-30% drop in accuracy when traditional federated learning global models are deployed on local clients. For example, in the Non-IID scenario of the CIFAR-10 dataset, FedAvg's test accuracy is 22.6% lower than that of centralized training, which cannot meet the needs of practical applications.

[0003] To address the challenges posed by heterogeneous data, Personalized Federated Learning (PFL) has emerged, with its core objective being to customize a high-performance model for each client through federated learning. Existing PFL methods can be categorized into two types: global model training algorithms and local model learning algorithms. Global model training algorithms aim to improve model quality from the source, specifically including data-based methods (such as data augmentation and client selection) and model-based methods (such as cooperative game theory and transfer learning). Local model learning algorithms, on the other hand, focus on the personalized architecture and strategies on the client side, mainly including architecture-based methods (such as parameter decoupling and knowledge distillation) and similarity-based methods (such as multi-task learning, model interpolation, and group clustering). Among these, parameter decoupling, which divides the model into shared and personalized parts, is a widely adopted approach. For example, FedPer shares the basic feature extractor while personalizing the classifier head; FedRoD maintains both a global classifier and a personalized classifier for each client; and FURL further subdivides model parameters into federated parameters and private parameters. However, a thorough analysis of existing research reveals that, despite significant progress, these methods still have obvious limitations when dealing with highly complex and dynamic heterogeneous environments:

[0004] (1) Existing personalized federated learning (PFL) methods (such as FedPer and FURL) achieve partial personalization through an architecture of "shared feature extractor paired with personalized classifier," but they employ a static and rigid module partitioning strategy: treating the features output by the feature extractor as a coupling of global and local information, they cannot dynamically adjust the information flow based on input samples, resulting in extremely poor flexibility in adapting to actual scenarios. For example, in actual medical image diagnosis scenarios, when FedPer processes skin cancer images, it includes both global general features such as "skin contour" and local personalized features such as "lesion size and shape" (different hospitals have different imaging equipment and annotation standards). However, the static architecture forces all features to be input into the shared extractor, causing the local classifier to be interfered with by irrelevant global information, increasing the risk of overfitting, and potentially leading to misdiagnosis or missed diagnosis in practical applications. In intelligent recommendation scenarios, static decoupling cannot distinguish between "user general preferences" and "local personalized preferences," resulting in a decrease in recommendation accuracy.

[0005] (2) The parameter aggregation strategy fails to effectively distinguish the quality of updated parameters and cannot curb client drift. In highly heterogeneous data environments, there is an inherent deviation between the local optimization objectives of each client and the global objective, resulting in a discrepancy between the local update direction and the ideal global update direction, which leads to client drift. However, most existing methods still use FedAvg's weighted average strategy based on data volume when aggregating personalized parameters, or although they introduce contribution metrics such as Shapley values, they still do not fully consider the consistency between the client update direction and the global direction. This traditional aggregation method introduces local update parameters that conflict with the global optimization direction into the global model. This not only fails to effectively integrate beneficial personalized knowledge, but also acts as noise to interfere with the stability and performance of the global model, resulting in slow convergence of the federated learning system, and even performance degradation after multiple iterations. This problem is particularly prominent in scenarios with low client participation, significant differences in local data volume, or extremely heterogeneous data distribution, seriously affecting the robustness and practicality of the method.

[0006] Existing personalized federated learning methods have shortcomings in two key aspects: dynamic feature decoupling and parameter aggregation. Therefore, a more refined feature decoupling method is needed to build upon existing approaches. Simultaneously, a superior parameter aggregation mechanism is required, capable of comprehensively evaluating client information contributions and consistency in update direction. Only such methods can generate better and more robust personalized models for each client in highly heterogeneous data environments. Summary of the Invention

[0007] This invention addresses the problems of data heterogeneity and catastrophic forgetting on the client side in federated learning. For a local image classification task, the client extracts image features from the image using a local feature extractor, and then classifies the image based on the local classifier. The process of updating parameters for each client and the server specifically includes the following steps:

[0008] S1. The server distributes global model parameters to each client. The global model parameters cover the feature extractor, conditional separation network, and classifier.

[0009] S2. Each client trains a model based on local data and uses prototype learning to calculate the prototype of each label, thereby constraining the distance between local and global features.

[0010] S3. Use a conditional separation network to decouple the features output by the feature extractor into global features and local features, which are then input into the global classifier and the local classifier, respectively.

[0011] S4. The server calculates the aggregation weight based on the Fisher information matrix of each client and the cosine similarity between its parameter update direction and the global direction, and then uses this weight to perform weighted aggregation of the client parameters.

[0012] S5. The server sends the aggregated parameters to the client.

[0013] Compared with existing technologies, the personalized federated learning method based on conditional separation networks and adaptive weights provided by this invention uses prototype learning to calculate the prototype of each label, thereby constraining the distance between local and global features. By separating the features extracted by the feature extractor into global and local features, adaptive and refined separation of personalized and global information is achieved, enabling the model to better capture client-specific and general patterns, thereby improving personalized performance. Attached Figure Description

[0014] Figure 1 This is a model diagram of client training based on a personalized federated learning method with conditional separation network and weight adaptation provided in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram illustrating the parameter update of a personalized federated learning method based on conditional separation networks and weight adaptation, provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention proposes a personalized federated learning method based on conditional separation networks and weight adaptation. For a local image classification task, the client extracts image features from the image using a local feature extractor, and then classifies the image based on the local classifier. The parameter update process for each client and server specifically includes the following steps:

[0018] S1. The server distributes global model parameters to each client. The global model parameters cover the feature extractor, conditional separation network, and classifier.

[0019] S2. Each client trains a model based on local data and uses prototype learning to calculate the prototype of each label, thereby constraining the distance between local and global features.

[0020] S3. Use a conditional separation network to decouple the features output by the feature extractor into global features and local features, which are then input into the global classifier and the local classifier, respectively.

[0021] S4. The server calculates the aggregation weight based on the Fisher information matrix of each client and the cosine similarity between its parameter update direction and the global direction, and then uses this weight to perform weighted aggregation of the client parameters.

[0022] S5. The server sends the aggregated parameters to the client.

[0023] In this invention, for the classification task of local images, the client extracts features from the image based on the local feature extractor, and the classifier classifies the image according to the extracted features.

[0024] In this embodiment, the model sent by the server to the client includes feature extractor parameters. Conditional separation network parameters and global classifier parameters The local parameters of client i are initialized to: global feature extractor Local feature extractor Global classifier Local classifier and conditional separation networks Global feature extractor for client i It is frozen at the beginning of training and does not participate in the update of local parameters; it is used to extract global information from local data; local feature extractor These are trainable parameters used to extract local personalized information from local data; These are trainable parameters that decouple the features extracted from local data into local and global features during the training process. It is a frozen classifier that does not update its parameters during training. It is used for global feature classification and provides global context information. These are trainable parameters used during training to classify local features and provide local contextual information.

[0025] Each selected client i trains a model based on local data and uses prototype learning to calculate the prototype for each label. The model structure is as follows: Figure 1 As shown. The specific steps are as follows:

[0026] (1) In the training initialization phase, in order to calculate the global consistency of features, the local data of client i is used. Input to the global feature extractor with frozen parameters Extracting global features And calculate the global prototype for each category j. This serves as the global feature center for that category. Its calculation formula is as follows:

[0027]

[0028] in, It represents the number of samples with label j owned by client i. This represents an indicator function; if the condition in parentheses is true, the function value is 1, otherwise the function value is 0. i,k This represents the label of the sample owned by client i whose label is j; x i,k This represents the sample owned by client i whose k-th label is label j; This represents the global feature extractor through client i. From local sample x i,k Feature vectors extracted .

[0029] To reduce computational overhead and avoid repeatedly extracting all local features in each training round, this invention does not compute local prototypes during the initialization phase. Instead, it uses a linear weighting method to dynamically update local prototypes during training. Specifically, in each local training iteration, the client uses its local feature extractor. Extract features from the current batch of data and calculate the local prototype of category j in this batch. To overcome the prototype fluctuations caused by the randomness of batch samples, this invention introduces the exponential moving average method, which compares the current batch prototype with the previous batch prototype. Perform weighted fusion to smoothly update local prototypes:

[0030]

[0031]

[0032] Where m∈(0,1) is the smoothing coefficient, used to control the weight ratio of historical information to current information. This method significantly reduces computational redundancy while ensuring prototype stability; This indicates the local feature extractor through client i. From sample x i,k Feature vectors extracted .

[0033] (2) The client uses the minimum mean square error (MSE) function to calculate the global prototype. and local prototype The distance between them is used to obtain the alignment loss, i.e.:

[0034]

[0035] Where C represents the number of tag categories owned by client i.

[0036] A conditional separation network is used to decouple the features output by the feature extractor into global features and local features, which are then input into the global classifier and the local classifier, respectively. The implementation is as follows:

[0037] Since the classifier's parameters are a C×K matrix, where C represents the number of label categories and K represents the number of features obtained by the feature extractor... The dimension is such that each column of the classifier corresponds to a feature. The importance of each dimension is calculated, and the mean of the C-row classifier matrix is ​​obtained. This mean is then used as context information input to the conditional separation network; then... Separate into features with personalized information and features with global information Using personalized classifiers and global classifiers and The parameters generate context vectors with global and local information respectively. and The specific formula is as follows:

[0038]

[0039] in, For classifier The vector in the c-th row. Similarly, the global context vector can be obtained. In order to more fully classify global and local information, and By concatenating them together, we obtain the concatenated context vector. Its dimension becomes 2K. The obtained... Input is fed into a scaling factor mapping network consisting of two sets of linear layers. And bias coefficient mapping network Scaling factor mapping network And bias coefficient mapping network Use one first The linear layer maps a vector to a low-dimensional vector, then passes it through a ReLU activation function, and then passes the low-dimensional vector through a... The linear layer is mapped back to the original dimension, and the specific formula is as follows:

[0040]

[0041]

[0042] in, It is a scaling factor, including scaling factors that map local features. and scaling factor for mapping global features ; These are bias coefficients, including bias coefficients that map local features. and the bias coefficients of the global features The scaling factor and bias coefficient both have a dimension of K, therefore they can be correlated with the features. Perform affine transformation operations, that is:

[0043]

[0044]

[0045] in, For local features containing personalized information, It is a feature that provides global information.

[0046] Because the global feature extractor sent locally contains information from other clients, it can become out of sync with the local classifier. This invention addresses this by converting the feature extractor's output... Decoupling and They are then fed into a local classifier and a global classifier, respectively, with both classifiers sharing the same true label. The results of the two classifiers are then added together to obtain the output result. If directly Feeding data into a local classifier causes it to converge on global information, thus losing its individualized advantages and leading to catastrophic forgetting. Finally, the cross-entropy loss function is used to calculate the loss for this training session.

[0047]

[0048] in, It is client i about data (x) i ,y i The distribution of ) Let cross-entropy be the loss function. Let be the overall loss function. To minimize the local loss function, the client uses backpropagation to update its trainable parameters, including the local feature extractor. Conditional Separation Network and local classifier These trainable parameters continuously learn local personalized information from client i, while and During local training on the client, the parameters are frozen and do not participate in the parameter update process during training. Therefore, these two parameters retain globally common information.

[0049] Furthermore, in step S4, the server performs parameter aggregation, the specific process of which is as follows:

[0050] (1) The server receives the parameters uploaded by each client. Calculate its parameters The trace of the Fisher information matrix This is used to quantify the amount of information contained in the parameter. The specific formula is as follows:

[0051]

[0052]

[0053] in, Representative calculation parameters The negative log-likelihood function, Given the Fisher information matrix, calculate the trace of this matrix. This reflects the overall sensitivity of the model to parameters. Higher sensitivity indicates a higher level of importance for the model uploaded by the client. It can be used as one of the indicators for judging aggregation weight.

[0054] (2) Calculate the angle between the update direction of each client parameter and the update direction of the global parameter. The smaller the angle, the closer the current client is to the global update direction, and therefore it should receive a larger aggregation weight. This invention uses a non-linear mapping function to map this angle into an importance score. The specific formula is as follows:

[0055]

[0056] in This is a scaling constant. Based on the obtained... and Calculate the final aggregate weight for each client. The specific formula is as follows:

[0057]

[0058] (3) Using weights The uploaded parameters are weighted and averaged to update the global model:

[0059]

[0060] in, For the first The model is distributed to each client by the server. For the client The uploaded parameters include , and Furthermore, this invention uploads model parameters during the upload process, avoiding the risk of data leakage caused by uploading raw client data. Simultaneously, the parameter conditional separation strategy achieves isolated storage of global and local knowledge, preventing indirect leakage of local personalized data (such as medical records). This complies with GDPR, CCPA, and my country's Personal Information Protection Law and Data Security Law, and can be directly applied to privacy-sensitive fields such as healthcare, overcoming the application bottleneck of insufficient privacy compliance in existing solutions.

[0061] Furthermore, in step S5, the server sends the aggregated parameters to the client, as follows:

[0062] In the During round-robin communication, the server randomly selects... A set of clients and the parameters after the current aggregation Distribute to selected clients, such as Figure 2 As shown;

[0063] Use distribution Update the client's local global and local feature extractors, using the distributed ones. Only update the global classifier locally on the client, and use the distributed method. Update the local conditional separation network on the client.

[0064] Therefore, the personalized federated learning method based on conditional separation networks and adaptive weights provided in this embodiment can fully decouple personalized information from global information and optimize aggregate weights to generate a high-performance personalized model for each client. The image datasets used in this embodiment are MNIST, Cifar10, and Cifar100 datasets. The accuracy of this invention can reach 99.79%, 92.89%, and 60.95% respectively, even with heterogeneous data. In practical applications, the accuracy is improved compared to similar classification models for medical image data classification tasks, effectively assisting doctors in diagnosis.

[0065] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A personalized federated learning method based on conditional separation network and weight adaptation, wherein the client extracts image features from the image through a local feature extractor for a local image classification task, and then classifies the image based on the local classifier, characterized in that, The process of updating parameters for each client and server includes the following steps: S1. The server distributes global model parameters to each client. The global model parameters cover the feature extractor, conditional separation network, and classifier. S2. Each client trains a model based on local data and uses prototype learning to calculate the prototype of each label, thereby constraining the distance between local and global features. S3. Use a conditional separation network to decouple the features output by the feature extractor into global features and local features, which are then input into the global classifier and the local classifier, respectively. S4. The client updates its parameters based on the loss function calculated from the classification results of the local and global label prototypes, the global classifier, and the local classifier. The client then uploads the updated parameters to the server. S5. The server calculates the Fisher information matrix and the cosine similarity between its parameter update direction and the global direction based on the model parameters uploaded by each client, calculates the aggregation weight, and then uses this weight to perform weighted aggregation on the client parameters to obtain the updated global model parameters. S6. The server sends the updated global model parameters to the client.

2. The personalized federated learning method based on conditional separation network and weight adaptation according to claim 1, characterized in that, Step S1 specifically includes: The server first randomly samples a batch of clients from all clients as a client subset, and distributes the global feature extractor, conditional separation network and classifier parameters to each client in the subset. Each client maintains the following components: global feature extractor, local feature extractor, global classifier, local classifier and conditional separation network.

3. The personalized federated learning method based on conditional separation network and weight adaptation according to claim 2, characterized in that... The process of a client training a model based on local data includes: The average value of the prototype for each label is extracted using a global feature extractor, and the average value of the prototype for each label is calculated during training. By minimizing the distance between the local prototype and the global prototype, the local feature representation is constrained not to deviate from the global feature distribution.

4. The personalized federated learning method based on conditional separation network and weight adaptation according to claim 2, characterized in that, Conditional separation networks achieve feature decoupling in the following ways: Context vectors are generated based on the parameters of the global classifier and the local classifier. Then, the global context vector and the local context vector are concatenated and a feature scaling factor and bias coefficient are generated through a bottleneck structure network. By applying an affine transformation to the original features using a scaling factor and a bias coefficient, global and local features are generated, i.e.: ； ； in, Features extracted from local data by the local feature extractor; For local features containing personalized information, Features that provide global information; This is a scaling factor for mapping local features. This is the scaling factor for mapping global features; These are the bias coefficients that map local features. The bias coefficient is used to map global features.

5. The personalized federated learning method based on conditional separation network and weight adaptation according to claim 1, characterized in that, The calculation of the aggregation weight in step S4 includes: Calculate the trace of the Fisher information matrix for each client parameter as a measure of its contribution; Calculate the cosine similarity between the client parameter update direction and the global average update direction to measure the consistency between the local parameter update direction and the global average update direction; Aggregation weights are generated based on trace and cosine similarity using a nonlinear mapping function.

6. A personalized federated learning method based on conditional separation network and weight adaptation according to claim 5, characterized in that, The aggregate weight is represented as: ； in, Let be the aggregate weight of the i-th client; Let be the trace of the Fisher information matrix for the i-th client; The cosine similarity between the update direction of the i-th client parameter and the global average update direction; This refers to the set of clients selected for training in this round. It is a non-linear mapping function.

7. A personalized federated learning method based on conditional separation network and weight adaptation according to claim 2, characterized in that, After receiving the data from the server, the client updates the global feature extractor, local feature extractor, global classifier, and conditional separation network using the global model parameters.

8. The personalized federated learning method based on conditional separation network and weight adaptation according to claim 1, characterized in that, The client calculates the loss function based on the classification results of the local label prototype and the global label prototype, and the global classifier and the local classifier, including: ； in, This represents the total loss function for the client. It is a client Regarding data The distribution of xi is the client The data sample yi is the label of the data sample xi; The cross-entropy loss function; The classification results are from the global classifier and the local classifier; L PA global prototype and local prototype Alignment loss between them.

9. A personalized federated learning method based on conditional separation network and weight adaptation according to claim 8, characterized in that, Global Prototype and local prototype The alignment loss is the minimum mean square error between the global prototype and the local prototype in each category.

10. A personalized federated learning method based on conditional separation network and weight adaptation according to claim 9, characterized in that, The process of updating the prototype of the local client label j during a training session includes: The client uses its local feature extractor Extract features from the current batch of data and calculate the categories in that batch. Local prototype ,Right now: ； in, It is a client Have tags The number of samples; This represents an indicator function; if the condition in parentheses is true, the function value is 1, otherwise the function value is 0. i,k Indicates the client The kth tag is a tag The label of the sample; x i,k Indicates the client The kth tag is a tag The sample; Indicates through the client Global feature extractor From sample x i,k Feature vectors extracted from; Compare the current batch of prototypes with the previous batch of tags. prototype Perform weighted fusion to smoothly update the local prototype, that is: ； in, It is the smoothing coefficient.

Images