Personalized federated learning method of classifier asynchronous optimization and prototype perception reasoning

Abstract

This invention belongs to the field of federated learning technology, and particularly relates to a personalized federated learning method based on asynchronous optimization of classifiers and prototype-aware inference. This invention designs an asynchronously updated dual-classifier mechanism during the training phase and utilizes a prototype-aware technique during the testing phase to adaptively calculate the output weights of the two classifiers. This allows the model to adaptively determine the confidence levels of personalized and generalized knowledge, thereby achieving dynamic complementarity and balance between the two. To ensure that the features generated by each client feature extractor have globally consistent semantic representations, this invention introduces a bilateral prototype clustering strategy to generate unified category prototypes and adaptively guide each client feature extractor to update in a consistent direction, effectively mitigating the performance degradation problem caused by domain offset. This invention effectively addresses the challenges of multi-client collaboration under label and domain offset environments while protecting data privacy.

Description

Technical Field

[0001] This invention belongs to the field of federated learning technology, and in particular relates to a personalized federated learning method for asynchronous optimization of classifiers and prototype-aware reasoning. Background Technology

[0002] Federated learning, as a distributed machine learning paradigm, was initially designed to address the "data silo" problem, promoting multi-party data collaboration without the need for direct exchange of raw data. However, with continuous technological evolution and the rapid growth in the number of users, a single, generalized global model has become insufficient to meet the personalized service needs of various clients. Therefore, personalized federated learning has emerged. While sharing global knowledge, personalized federated learning optimizes the performance of local models for the specific data distribution and task requirements of each client. Compared to traditional federated learning, personalized federated learning not only effectively alleviates the challenges posed by data heterogeneity but also significantly improves the performance of local models on specific clients with only a minimal sacrifice in global generalization capabilities.

[0003] As an efficient and practical technical approach, model decoupling is widely used in personalized federated learning. A typical method involves dividing the model into a feature extractor and a classifier. The feature extractor is uploaded to a server for aggregation, while the classifier remains local to learn personalized features. This approach not only enhances the model's personalization capabilities but also preserves rich shared knowledge, thus becoming a research hotspot in personalized federated learning in recent years. However, due to the heterogeneity of client data distribution, simple feature extractor averaging aggregation strategies often fail to achieve effective knowledge transfer and may even lead to negative transfer, resulting in performance degradation. Therefore, maintaining semantic consistency of features across clients is crucial for achieving effective knowledge transfer. Currently, the mainstream approach is to guide feature extractor updates based on feature prototypes to encourage all clients to learn globally consistent feature representations.

[0004] While existing personalized federated learning methods based on model decoupling perform well in highly heterogeneous environments, they generally suffer from insufficient adaptability to weakly heterogeneous environments, particularly when dealing with label shifts. For example, FedPer and FedRep, by uploading feature extractors and retaining local classifiers, can effectively improve performance in highly heterogeneous environments, but their performance deteriorates significantly under weakly heterogeneous conditions. Regarding feature alignment, methods such as FedProto propose uploading only feature prototypes instead of model parameters, constructing a global prototype through an averaging strategy. However, when client class distributions are completely different or there is severe class missingness, the expressive power of the global prototype is limited, failing to accurately cover all local feature distributions, leading to decreased generalization performance and difficulty in effectively mitigating the impact of domain shifts.

[0005] Chinese patent document CN118734995A discloses a single-client, multi-domain heterogeneous federated learning system and method based on manifold learning. Each client determines multiple manifold point sets corresponding to the categories of the data in the training set based on its local model and local dataset, and sends the manifold point sets and local models to the server. The server determines the global manifold based on the manifold points of each client under any category, performs data domain partitioning on each global manifold through clustering to obtain sub-manifolds, updates each sub-manifold based on an attention mechanism, and then reconstructs the manifold points. The global model is updated based on each local model, and the reconstructed manifold points and the current global model are distributed to the corresponding clients. Each client determines the total loss function based on the received global model, updates its current local model based on the total loss function, and sends the updated local model to the server, enabling iterative training between the clients and the server until the global model converges. This achieves good federated performance in heterogeneous domain scenarios. However, this method uploads the entire model and manifold point sets, resulting in high communication costs and privacy risks. Furthermore, this method only addresses the domain generalization problem but neglects the personalized needs of clients, failing to effectively balance the needs of client generalization and personalization.

[0006] Chinese patent document CN119658639A discloses a Non-IID federated learning method and apparatus based on trajectory matching dataset distillation. The method includes: Step 1: There are N participants, each with its own local data distribution and labels. The data distributions among the clients do not satisfy independent and identically distributed (Io-distributed). On the client side, each client initializes its local model and replaces the original classifier of the given model architecture with the same ETF classifier. The server runs a standard federated averaging algorithm. The client trains its model on local data, updates the model parameters, and uploads the parameters to the server. The server aggregates the parameters and updates the global model. This process is repeated continuously. The server initializes the latent vector set of the synthetic data and optimizes it based on the early trajectories of the global model. The server uses the optimized synthetic data to correct the aggregated global model until the number of communication rounds reaches a certain number. It then initializes an expanded latent vector set and continues to optimize based on the later global model trajectories. The global model is further corrected using the synthetic data generated from the first two optimizations. With continuous model updates, aggregation, and correction, the global model gradually converges. However, it is based solely on latent vector synthesis and does not fully utilize global semantic consistency, which leads to missing semantic features. The model has difficulty learning shared knowledge across distributions during aggregation, resulting in a decrease in the predictive ability for data with unknown distributions. Furthermore, it may require more local data to participate in training to compensate for semantic missing features, thereby increasing the amount of data transmitted and the risk of privacy leakage. It mainly focuses on the alignment of feature space and does not adequately handle label offset.

[0007] To address the aforementioned issues, several improvement methods have been proposed in recent years. For example, FedPAC optimizes its approach by combining feature prototype alignment with classifier collaboration: on the one hand, it uses feature prototypes to guide feature extractor updates, mitigating domain bias; on the other hand, it reorganizes the classifier on the server side based on client distribution similarity. This strategy demonstrates good personalization and generalization performance in various heterogeneous environments. However, FedPAC requires clients to upload local data distribution information, thus introducing potential privacy risks.

[0008] Therefore, there is an urgent need to explore a new personalized federated learning method that does not expose privacy information, takes into account both label offset and domain offset, and can achieve efficient personalization and generalization performance. Summary of the Invention

[0009] This invention aims to overcome the shortcomings of existing technologies by proposing a personalized federated learning method based on asynchronous classifier optimization and prototype-aware inference. It employs an asynchronously updated dual-classifier mechanism during training and a prototype-aware technique during inference to adaptively calculate the output weights of the two classifiers. This allows the model to adaptively determine the confidence levels of personalized and generalized knowledge, achieving dynamic complementarity and balance between the two. Furthermore, to ensure that the features generated by each client feature extractor have globally consistent semantic representations, this invention introduces a bilateral prototype clustering strategy using both local and global prototypes to generate unified category prototypes and adaptively guide each client feature extractor to update in a consistent direction, effectively mitigating the performance degradation caused by domain shift. By combining asynchronous classifier updates and bilateral prototype clustering, this invention achieves synergistic optimization of personalization capabilities and generalization performance while protecting data privacy, effectively addressing the challenges of multi-client collaboration under label and domain shift environments.

[0010] To address the aforementioned technical problems, this invention provides a personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning, the method comprising:

[0011] Training phase:

[0012] S0, Server-side initialization of shared classifier and global clustering prototype The initial shared classifier and global clustering prototype The broadcast is sent to each client participating in the training;

[0013] S1. Asynchronous Update: The client receives the shared classifier and global clustering prototype from the server. The client first freezes the personalized classifier, uses the global clustering prototype to adaptively align features, and guides the update of the personalized feature extractor and the shared classifier. After the update is complete, the personalized classifier is unfrozen. The client then freezes the updated personalized feature extractor and the shared classifier, updates the personalized classifier, and after the update is complete, the personalized feature extractor and the shared classifier are unfrozen.

[0014] S2. Use the updated personalized feature extractor to extract features from the local training set, and then perform local prototype clustering on the extracted features to obtain a local clustering prototype set. Then, a weighted average of the prototypes in this set is used to calculate the locally unbiased prototype. ;

[0015] S3. The client updates the shared classifier from step S1 and the locally unbiased prototype obtained from step S2. Send to the server;

[0016] S4. The server uses the average aggregation method to aggregate the received shared classifiers, resulting in the aggregated shared classifier. ;

[0017] S5. The server will receive the locally unbiased prototype. Perform global prototype clustering to obtain the global clustering prototype. ;

[0018] S6. The server will use the aggregated shared classifier obtained in step S4. and the global clustering prototype obtained in step S5 Rebroadcast to each participating client, repeat steps S1-S6 until the preset number of rounds is reached or the model converges;

[0019] Prototype-aware adaptive reasoning stage:

[0020] S7. After training is completed, the corresponding features extracted from the local test set by the personalized feature extractor are input into the shared classifier and the personalized classifier to obtain personalized prediction output and shared prediction output. Based on prototype perception technology, the client adaptively calculates the output weights of the two classifiers and performs weighted fusion of the prediction outputs of the two classifiers to obtain the final prediction result, and thus obtain the final test accuracy.

[0021] Preferably, step S1, which utilizes a global clustering prototype to adaptively align features and guide the updates of the personalized feature extractor and the shared classifier, specifically involves:

[0022] S11. The client calculates the entropy value based on the label distribution of the local training set to obtain the adaptive alignment weights for feature alignment. :

[0023] The tag distribution of each client is statistically analyzed, and the entropy of the tag distribution is calculated using formula (1):

[0024] (1)

[0025] In equation (1), The entropy representing the label distribution of client k. This represents the random variable Y taking the t-th category in the label space. The probability, Let C represent the t-th category in the label space, and let C represent the total number of categories in the label space. represent The logarithm of the label distribution is used to assess the heterogeneity of the data distribution; the larger the value, the more uneven the label distribution.

[0026] The adaptive alignment weights of client k are calculated using the obtained entropy values. :

[0027] (2)

[0028] In equation (2), It is the adaptive alignment weight of client k. It is the entropy of the label distribution of client k. It is the scaling factor;

[0029] S12, Based on adaptive alignment weights Update the personalized feature extractor and the shared classifier:

[0030] For updating the personalized feature extractor, the previously obtained global clustering prototype from the server is used in conjunction with contrastive learning to guide it to generate semantically consistent and class-separable features, i.e., features of the same class are close to each other, and features of different classes are far apart. The loss for feature alignment is calculated as follows:

[0031] (3)

[0032] In equation (3), It is the loss function of the personalized feature extractor. The feature representation of the i-th sample before update, extracted by the personalized feature extractor of the k-th client. represent Cosine similarity between the class prototype c and the class prototype c Represents belonging to category The set of global clustering prototypes is the set of positive samples. This indicates that it does not belong to the category. The set of global clustering prototypes is the set of negative samples. By calculating this loss, the personalized feature extractors of each client can produce semantically consistent feature representations while maintaining the separability between categories.

[0033] For updating the shared classifier, the traditional cross-entropy loss is used:

[0034] (4)

[0035] In equation (4), It is the cross-entropy loss of the shared classifier, | | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. The loss represents the predicted value of the shared classifier for the i-th sample, and the shared classifier is updated by minimizing this loss.

[0036] The total loss for updating the personalized feature extractor and the shared classifier is:

[0037] (5)

[0038] In equation (5), It is the total loss of the update. It is the cross-entropy loss of the shared classifier. Adaptive weights representing feature alignment It is the loss function of the personalized feature extractor, which is minimized by This improves the performance of the personalized feature extractor and the shared classifier, resulting in an updated personalized feature extractor and shared classifier.

[0039] Step S1, updating the personalized classifier, specifically involves:

[0040] It is updated using cross-entropy loss, and its loss function is:

[0041] (6)

[0042] (7)

[0043] in, This refers to the total loss of the personalized classifier update. The cross-entropy loss represents the personalized classifier. | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. This represents the predicted output of the personalized classifier for the i-th sample; by minimizing... This improves the classification performance of the personalized classifier, resulting in an updated personalized classifier.

[0044] Preferably, in step S2, the extracted features are subjected to local prototype clustering to obtain a local clustering prototype set. Specifically:

[0045] Client k performs FINCH clustering on the features extracted from the personalized feature extractor. The clustering process is as follows:

[0046] (8)

[0047] In equation (8), This represents the local clustering prototype set of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m. Indicates the input sample Features obtained after the updated personalized feature extractor This represents the i-th input sample and its label. This represents the local training set where the k-th client belongs to category m;

[0048] The locally unbiased prototype is obtained by weighted averaging the prototypes in the set. Specifically:

[0049] After obtaining the feature prototypes for each category, the next step is to average the feature prototypes of each category to obtain the locally unbiased prototype. The calculation process is as follows:

[0050] (9)

[0051] In equation (9), The local unbiased prototype of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m.

[0052] Preferably, step S4 specifically comprises:

[0053] The server aggregates the local shared classifiers using the following formula:

[0054] (10)

[0055] In equation (10), Represents the aggregated shared classifier. This represents the number of clients participating in the aggregation. It is the local shared classifier for the kth client.

[0056] Preferably, in step S5, the server receives the locally unbiased prototype. Perform global prototype clustering to obtain the global clustering prototype. Specifically:

[0057] Using FINCH to process the uploaded partially unbiased prototype Clustering is performed, and the clustering process is as follows:

[0058] (11)

[0059] In equation (11), The set representing the global clustering prototypes of category m. The t-th global cluster prototype represents category m. The number of global cluster prototypes representing category m. The local unbiased prototype of class m representing client k. The set of locally unbiased prototypes representing each client class m.

[0060] Preferably, step S7 specifically includes:

[0061] S71. After training is completed, firstly, the local test set is sent to the personalized feature extractor to extract the corresponding features, and then the features are sent to the personalized classifier and the shared classifier respectively to obtain personalized prediction output and shared prediction output.

[0062] S72, Compare the features extracted by the personalized feature extractor with the global prototype. and locally unbiased prototypes The cosine similarity, where the global prototype For global clustering prototype The average was calculated.

[0063] S73. The two cosine similarities calculated in step S72 are normalized using the softmax function to obtain the weights of the prediction outputs of the shared classifier and the individual classifier. The two prediction outputs obtained in step S71 are then fused using these weights to obtain the final inference prediction output.

[0064] S74. Compare the final inference prediction output obtained in step S73 with the actual output to obtain the final test accuracy.

[0065] More preferably, in step S72, the features extracted by the personalized feature extractor are compared with the global prototype. and locally unbiased prototypes The cosine similarity is:

[0066] (12)

[0067] (13)

[0068] (14)

[0069] In equations (12), (13), and (14), For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories global prototype Cosine similarity between them yes Standardized metrics yes Standardized metrics yes Standardized metrics The number of global cluster prototypes representing category m. This represents the t-th global cluster prototype for category m.

[0070] More preferably, step S73 specifically includes:

[0071] The weights between the predicted outputs of the shared classifier and the individualized classifier are calculated using formula (15):

[0072] (15)

[0073] In equation (15), and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. + =1, For the client Features generated by personalized feature extractor and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by personalized feature extractor and corresponding categories global prototype Cosine similarity between them;

[0074] Based on the weights of the prediction outputs of the individualized classifier and the shared classifier, the final inference prediction output is:

[0075] (16)

[0076] in, For inference and prediction output, and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. The predicted output of the personalized classifier for client k. This is the predicted output of the shared classifier.

[0077] This invention also provides the application of a personalized federated learning method of asynchronous classifier optimization and prototype-aware reasoning in medical image analysis, for personalized model optimization tasks in medical image analysis.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] (1) The method of the present invention designs an asynchronous update dual classifier mechanism and introduces a bilateral prototype clustering strategy of local prototype and global prototype in the training phase. In the inference phase, a prototype awareness technology is used to adaptively calculate the output weights of the two classifier heads. This achieves the coordinated optimization of personalization capability and generalization performance while protecting privacy, and alleviates the impact of label offset and domain offset.

[0080] (2) Label Shift: Due to the existence of label shift, i.e., the data label distribution among clients exhibits non-independent and identically distributed (non-IID) characteristics, the update directions of each local model are inconsistent, which significantly reduces the performance of the global model. To alleviate this problem, personalized federated learning methods train personalized models for each client, which to some extent mitigates the impact of label shift. However, existing personalized federated learning methods still have performance limitations in weakly heterogeneous scenarios, so the label shift problem has not been completely solved. To this end, this invention introduces a shared classifier, which effectively integrates personalized knowledge and generalized knowledge by adaptively fusing the prediction outputs of two classifiers during the inference stage. This results in excellent performance in scenarios with different degrees of heterogeneity, further alleviating and solving the label shift problem. This invention uses a loss-guided dual classifier mechanism, which allows the model to address the label shift problem in strongly heterogeneous scenarios through a personalized head and alleviate the label shift problem in weakly heterogeneous scenarios through a shared head, thus solving the label shift problem in various heterogeneous environments.

[0081] (3) Domain Shift: When client data comes from different domains, existing personalized federated learning methods struggle to cope effectively, leading to a significant decrease in the model's domain generalization ability. By introducing a two-sided prototype clustering strategy, a global category prototype rich in more diverse knowledge and suitable for each client is obtained. This prototype guides the update of the personalized feature extractor, prompting each client to generate semantically consistent feature representations, thereby effectively mitigating the domain shift phenomenon and improving the model's generalization performance.

[0082] (4) Communication cost: Since the method of the present invention only uploads the shared classifier and the local unbiased prototype, the communication efficiency is greatly improved compared with the method of uploading the entire model.

[0083] (5) Protecting data privacy: Since the method of the present invention does not require uploading any information related to local data distribution to the server, the risk of privacy leakage is avoided.

[0084] (6) This invention is particularly applicable to personalized model optimization tasks in medical image analysis, such as scenarios where there are differences in data distribution between different hospitals, different imaging devices, or different patient groups. Traditional federated learning methods lack effective adaptation to local data characteristics and label offsets, making it difficult to simultaneously meet individualized diagnostic needs in medical imaging tasks. By applying the personalized federated learning method of this invention, feature-level consistency alignment and adaptive optimization of local models can be achieved without directly exchanging original medical data, thereby effectively improving the classification performance of each client in specific medical imaging tasks. Attached Figure Description

[0085] Figure 1This is a flowchart illustrating the personalized federated learning method used in this invention;

[0086] Figure 2 This is a schematic diagram of the framework principle of the training phase of the personalized federated learning method used in this invention;

[0087] Figure 3 This is a schematic diagram of the framework principle of the inference stage of the personalized federated learning method used in this invention;

[0088] Figure 4 This is Example 2, a comparison chart of the test accuracy of the personalized federated learning algorithm of the present invention with other federated learning algorithms under CIFAR100 data;

[0089] Figure 5 This is Example 2, a comparison chart of the test accuracy of the personalized federated learning algorithm of the present invention with other federated learning algorithms under Digit5 data. Detailed Implementation

[0090] Symbol definition: This invention selects Each client participates in the aggregation, and each client... Its local training set is , among which | | indicates the number of datasets it possesses; both the local training set and the local test set are image datasets. Global dataset This represents the collection of all client datasets.

[0091] Example 1

[0092] This invention provides a personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning, such as... Figure 1 , 2 As shown in Figure 3, the method includes:

[0093] Training phase:

[0094] S0, Server-side initialization of shared classifier and global clustering prototype The initial shared classifier and global clustering prototype The broadcast is sent to each client participating in the training;

[0095] S1. Asynchronous Update: The client receives the shared classifier and global clustering prototype from the server. The client first freezes the personalized classifier, then adaptively aligns features using the global clustering prototype to guide the update of the personalized feature extractor and the shared classifier. After the update, the personalized classifier is unfrozen. Subsequently, the client freezes the updated personalized feature extractor and the shared classifier, updates the personalized classifier, and then unfrozes both.

[0096] Step S1, which utilizes a global clustering prototype to adaptively align features and guides the updates of the personalized feature extractor and the shared classifier, specifically involves:

[0097] S11. The client calculates the entropy value based on the label distribution of the local training set to obtain the adaptive alignment weights for feature alignment. :

[0098] Specifically, in traditional feature alignment methods, all clients use the same alignment weight. That is, each client applies the same level of force during feature alignment, without considering the differences in data distribution among clients. When the data differences between clients are significant, forcibly using the same alignment force can lead to the following problems: ① Increased gradient of the alignment term: The system will "try hard" to align features with large distribution differences, resulting in a particularly large gradient during backpropagation; ② Interference with the main task learning: Over-emphasis on alignment operations can hinder the learning process of the main task (such as classification). Therefore, this invention designs an adaptive feature alignment method to reduce the alignment weight for clients with uneven data distribution. Here, we use an entropy-based method to calculate this alignment weight. First, we statistically analyze the label distribution of each client, and then calculate the entropy of the label distribution using the following formula:

[0099] (1)

[0100] In equation (1), This represents the random variable Y taking the t-th category in the label space. The probability, where C represents the total number of categories in the label space. Represents the t-th category in the label space. represent The logarithm of The entropy represents the label distribution of client k. The larger the value, the more uneven the label distribution, which is used to assess the degree of heterogeneity in the data distribution.

[0101] Next, the obtained entropy value is used to calculate the adaptive alignment weight of client k:

[0102] (2)

[0103] In equation (2), It is the adaptive alignment weight of client k. It is the entropy of the label distribution of client k. It's a scaling factor. Its function is to normalize the alignment term to prevent the entropy value from being too large or too small, which would affect the stability of training. It can also adjust the influence ratio of the alignment term relative to the main task, thereby setting a reasonable balance between the main task and the alignment task.

[0104] S12, Based on adaptive alignment weights The personalized feature extractor and shared classifier are updated as follows:

[0105] For updating the two classification heads, an asynchronous update strategy was used. Specifically, the personalized feature extractor and shared classifier were updated first, followed by the personalized classifier, to prevent synchronous updates from interfering with their training. First, the personalized classifier was frozen, and then the personalized feature extractor and shared classifier were updated. Specifically, for updating the personalized feature extractor, a global clustering prototype obtained from the server was used in conjunction with contrastive learning to guide it to generate semantically consistent and class-separable features. That is, features of the same class are close together, and features of different classes are far apart. The loss for feature alignment is calculated as follows:

[0106] (3)

[0107] In equation (3), It is the loss function of the personalized feature extractor. The feature representation of the i-th sample before update, extracted by the personalized feature extractor of the k-th client. represent Cosine similarity between the class prototype c and the class prototype c Represents belonging to category The set of global clustering prototypes is the set of positive samples. This indicates that it does not belong to the category. The set of global clustering prototypes is the negative sample set. By calculating this loss, the personalized feature extractors of each client produce semantically consistent feature representations while maintaining the separability between categories. Contrastive learning is used to guide the feature alignment of the client's personalized feature extractors, thereby solving the domain offset problem.

[0108] Next, for updating the shared classifier, the traditional cross-entropy loss was used:

[0109] (4)

[0110] In equation (4), It is the cross-entropy loss of the shared classifier, | | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. The loss represents the predicted value of the shared classifier for the i-th sample, and the shared classifier is updated by minimizing this loss.

[0111] The total loss for updating the personalized feature extractor and the shared classifier is:

[0112] (5)

[0113] In equation (5), It is the total loss of the update. It is the cross-entropy loss of the shared classifier. Adaptive weights representing feature alignment It is the loss function of the personalized feature extractor. By minimizing the total loss of the update, the performance of the personalized feature extractor and the shared classifier can be improved, and the updated personalized feature extractor and shared classifier can be obtained.

[0114] Step S1, updating the personalized classifier, is as follows:

[0115] It is updated using cross-entropy loss, and its loss function is:

[0116] (6)

[0117] (7)

[0118] in, This refers to the total loss of the personalized classifier update. The cross-entropy loss represents the personalized classifier. | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. This represents the predicted output of the personalized classifier for the i-th sample; by minimizing... This improves the classification performance of the personalized classifier, resulting in an updated personalized classifier.

[0119] S2. Local Prototype Clustering: Features are extracted from the local training set using the updated personalized feature extractor, and the extracted features are then subjected to local prototype clustering to obtain a local cluster prototype set. Then, a weighted average of the prototypes in this set is used to calculate the locally unbiased prototype. ;

[0120] Specifically as follows:

[0121] Local Prototype Clustering: Current research on local prototypes utilizes locally averaged prototypes, which tend to favor a dominant feature while neglecting other important characteristics. Therefore, we propose using FINCH clustering to obtain local cluster prototypes. Then, we average the local cluster prototypes belonging to the same category to obtain locally unbiased prototypes. These unbiased prototypes do not favor any dominant feature and better reflect the average characteristics of a domain, thus avoiding information loss.

[0122] First, client k performs FINCH clustering on the features extracted from the personalized feature extractor. The clustering process is as follows:

[0123] (8)

[0124] In equation (8), This represents the local clustering prototype set of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m. Indicates the input sample Features obtained after the updated personalized feature extractor This represents the i-th input sample and its label. This represents the local training set where the k-th client belongs to category m. After obtaining the feature prototypes for each category, the next step is to average the feature prototypes of all categories to obtain the locally unbiased prototype. The calculation process is as follows:

[0125] (9)

[0126] In equation (9), The local unbiased prototype of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m.

[0127] S3. The client updates the shared classifier from step S1 and the locally unbiased prototype obtained from step S2. Send to the server;

[0128] S4. Shared Classifier Aggregation: The server uses an average aggregation method to aggregate the received shared classifiers, resulting in an aggregated shared classifier. ;

[0129] Specifically, the server aggregates the locally shared classifiers using the following formula:

[0130] (10)

[0131] In equation (10), This represents the aggregated, globally shared classifier. This represents the number of clients participating in the aggregation. It is the local shared classifier for the kth client.

[0132] S5. Global Prototype Clustering: The server will cluster the received local unbiased prototypes. Perform global prototype clustering to obtain the global clustering prototype. ;

[0133] To better address domain offset, the global prototype needs to contain more knowledge of domain diversity. Since a standard global average prototype cannot describe diverse domain information and tends to favor the dominant lower-level domain, this invention uses a global clustering prototype to supplement this with richer knowledge of domain diversity. To obtain the global clustering prototype, the method of this invention uses FINCH to cluster the uploaded local unbiased prototype. The clustering process is as follows:

[0134] (11)

[0135] In equation (11), The set representing the global clustering prototypes of category m. The t-th global cluster prototype represents category m. The number of global cluster prototypes representing category m. The set representing the local unbiased prototypes of each client class m. The local unbiased prototype of class m representing client k.

[0136] S6. The server will use the aggregated shared classifier obtained in step S4. and the global clustering prototype obtained in step S5 Rebroadcast to each participating client, repeat steps S1-S6 until the preset number of rounds is reached or the model converges;

[0137] Prototype-aware adaptive reasoning stage:

[0138] S7. After training is complete, the features extracted from the local test set by the personalized feature extractor are input into the shared classifier and the personalized classifier to obtain personalized prediction outputs and shared prediction outputs. Based on prototype awareness technology, the client adaptively calculates the output weights of the two classifiers and performs weighted fusion of the prediction outputs of the two classifiers to obtain the final prediction result, and thus the final test accuracy. Specifically:

[0139] To adaptively integrate the prediction outputs of personalized and shared heads, thereby improving the model's adaptability to data distribution, this invention uses the cosine similarity between features and prototypes to approximate the confidence of the two classifiers. If the current feature is more similar to the global prototype, the shared classifier is trusted more; conversely, if the feature is more similar to the local unbiased prototype, the personalized classifier is trusted more. The specific process is as follows:

[0140] S71. After training the client, firstly, the local test set is sent to the personalized feature extractor to extract the corresponding features, and then the features are sent to the personalized classifier and the shared classifier to obtain personalized prediction output and shared prediction output respectively.

[0141] S72, Compare the features extracted by the personalized feature extractor with the global prototype. and locally unbiased prototypes Cosine similarity;

[0142] Calculating the similarity between features and prototypes: To measure the confidence level between the two, we first need to calculate the similarity between the features generated by the personalized feature extractor and the global prototype, as well as the cosine similarity between the features generated by the personalized feature extractor and the local prototype. Here, the global prototype refers to the average of the global clustered prototypes for the corresponding category, and the local prototype refers to the local unbiased prototype. The process of calculating the cosine similarity is as follows:

[0143] (12)

[0144] (13)

[0145] (14)

[0146] In equations (12), (13), and (14), For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories global prototype Cosine similarity between them yes Standardized metrics yes Standardized metrics yes Standardized metrics The number of global cluster prototypes representing category m. This represents the t-th global cluster prototype for category m.

[0147] S73. The two cosine similarities calculated in step S72 are normalized using the softmax function to obtain the weights of the prediction outputs of the shared classifier and the individual classifier. The two prediction outputs obtained in step S71 are then fused using these weights to obtain the final inference prediction output.

[0148] Specifically, the prototype-aware weights are calculated as follows: Based on the calculated similarity, the weights between the predicted outputs of the shared classifier and the individualized classifier are calculated using the following formula:

[0149] (15)

[0150] In equation (15), and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. + =1, For the client Features generated by personalized feature extractor and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by personalized feature extractor and corresponding categories global prototype Cosine similarity between them.

[0151] Finally, based on the weights of the prediction outputs of the individualized and shared classification heads, the final inference prediction output is:

[0152] (16)

[0153] in, For inference and prediction output, and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. The predicted output of the personalized classifier for client k. This is the predicted output of the shared classifier.

[0154] S74. Compare the final inference prediction output obtained in step S73 with the actual output to obtain the final test accuracy.

[0155] Figure 2 Step 1-Step 6 in this invention refers to S1-S6.

[0156] This invention also provides the application of a personalized federated learning method of asynchronous classifier optimization and prototype-aware reasoning in medical image analysis, for personalized model optimization tasks in medical image analysis.

[0157] Example 2

[0158] Next, we will conduct experimental verification. First, let's give a basic introduction to the experiment:

[0159] (1) Introduction to the dataset:

[0160] CIFAR100: This dataset consists of 60,000 color images, each 32×32 pixels in size, and is organized into 100 different categories, each containing 6,000 images.

[0161] Digit5: It contains handwritten digit image datasets from 5 different domains, namely MNIST, MNIST-M, SVHN, SYN and USPS, with each domain's data having different styles, backgrounds and other features;

[0162] (2) Task: Image classification task;

[0163] (3) Model: CNN, consisting of three convolutional-pooling layers and two fully connected layers;

[0164] (4) Label offset settings: The CIFAR100 dataset was used for training and testing. The data was split using practical non-IID splitting (α=0.1), and the distribution of the training set and the test set was the same.

[0165] (5) Domain offset settings: The Digit5 dataset was used for training and testing. During training, there were a total of 5 clients, each with data from one domain, and the data from each client belonged to different domains. During testing, the test set for each client came from the data of other clients' domains.

[0166] (6) Benchmark algorithms: FedAvg, FedProx, FedPer, FedRep, FedPAC, FedProto, FedAMP, FedAPEN, FedKD, FML;

[0167] (7) Hyperparameter settings: The learning rate of FedAvg and FedProx was set to 0.1, and the learning rate of FedPer, FedRep, FedPAC, FedProto, FedAMP, FedAPEN, FedKD, FML, and our algorithm was set to 0.05. All algorithms had a local iteration cycle of 3 and a batch size of 32.

[0168] To ensure fairness, all benchmark algorithms and this algorithm use the same network architecture, equipment, and hyperparameter settings. Next, we will discuss the appendix... Figure 3 and 4 The experimental results were analyzed in detail.

[0169] Regarding the appendix Figure 4 This section compares the performance of the proposed personalized federated learning algorithm with other benchmark methods on the CIFAR-100 dataset, with an experimental condition of heterogeneity coefficient α=0.1. As can be observed from the figures, traditional federated learning methods, such as FedAvg and FedProx, significantly outperform personalized federated learning methods, including FedPer, FedRep, FedProto, FedAPEN, FedPAC, and the method of this invention, in heterogeneous data scenarios. This difference is due to the fact that traditional federated learning methods primarily optimize the global generalization performance of the model, lacking attention to the personalized needs of the client; while personalized federated learning methods enhance performance under heterogeneous data conditions through local adaptation. It is particularly noteworthy that FedPer and FedRep exhibit a significant performance drop compared to other personalized methods, mainly because these two methods employ a shared feature extractor structure, forcing the model to generate more domain-invariant features. While this feature representation is beneficial for improving cross-domain generalization ability, it can mislead the classifier in personalized classification tasks, leading to performance degradation. In comparison, the method proposed in this invention outperforms all the comparative methods, verifying the effectiveness and superiority of the designed asynchronous classifier update strategy and prototype-aware adaptive inference strategy in a label heterogeneous environment.

[0170] Regarding the appendix Figure 5 The adaptability of the proposed method in scenarios with domain offset was evaluated using the Digit5 dataset. Experimental results show that, compared with other personalized methods, the proposed method exhibits superior performance when migrating between different source and target domains, further demonstrating the effectiveness of the proposed bilateral prototype clustering strategy in feature alignment and domain generalization.

[0171] In summary, the experimental results fully demonstrate that the method of the present invention has significant advantages in handling personalized federated learning scenarios involving label offset and domain offset, and that each key module plays a crucial role in improving the overall performance.

Claims

1. A personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning, characterized in that, The method includes: Training phase: S0, Server-side initialization of the shared classifier and global clustering prototype The initial shared classifier and global clustering prototype The broadcast is sent to each client participating in the training; S1. Asynchronous Update: The client receives the shared classifier and global clustering prototype from the server. The client first freezes the personalized classifier, uses the global clustering prototype to adaptively align features, and guides the update of the personalized feature extractor and the shared classifier. After the update is complete, the personalized classifier is unfrozen. The client then freezes the updated personalized feature extractor and the shared classifier, updates the personalized classifier, and after the update is complete, the personalized feature extractor and the shared classifier are unfrozen. S2. Use the updated personalized feature extractor to extract features from the local training set, and then perform local prototype clustering on the extracted features to obtain a local clustering prototype set. Then, a weighted average of the prototypes in this set is used to calculate the locally unbiased prototype. ; S3. The client updates the shared classifier from step S1 and the locally unbiased prototype obtained from step S2. Send to the server; S4. The server uses the average aggregation method to aggregate the received shared classifiers, resulting in the aggregated shared classifier. ; S5. The server will receive the locally unbiased prototype. Perform global prototype clustering to obtain the global clustering prototype. ; S6. The server will aggregate the shared classifier obtained in step S4. and the global clustering prototype obtained in step S5 Rebroadcast to each participating client, repeat steps S1-S6 until the preset number of rounds is reached or the model converges; Prototype-aware adaptive reasoning stage: S7. After training is completed, the corresponding features extracted from the local test set by the personalized feature extractor are input into the shared classifier and the personalized classifier to obtain personalized prediction output and shared prediction output. Based on prototype perception technology, the client adaptively calculates the output weights of the two classifiers and performs weighted fusion of the prediction outputs of the two classifiers to obtain the final prediction result, and thus obtain the final test accuracy.

2. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 1, characterized in that, Step S1, which utilizes a global clustering prototype to adaptively align features and guides the updates of the personalized feature extractor and the shared classifier, specifically involves: S11. The client calculates the entropy value based on the label distribution of the local training set to obtain the adaptive alignment weights for feature alignment. ; The tag distribution of each client is statistically analyzed, and the entropy of the tag distribution is calculated using formula (1): （1） In equation (1), The entropy representing the label distribution of client k. This represents the random variable Y taking the t-th category in the label space. The probability, Let C represent the t-th category in the label space, and let C represent the total number of categories in the label space. represent The logarithm of the label distribution is used to assess the heterogeneity of the data distribution; the larger the value, the more uneven the label distribution. The adaptive alignment weights of client k are calculated using the obtained entropy values. : （2） In equation (2), It is the adaptive alignment weight of client k. It is the entropy of the label distribution of client k. It is the scaling factor; S12, Based on adaptive alignment weights Update the personalized feature extractor and the shared classifier: For updating the personalized feature extractor, the previously obtained global clustering prototype from the server is used in conjunction with contrastive learning to guide it to generate semantically consistent and class-separable features, i.e., features of the same class are close to each other, and features of different classes are far apart. The loss for feature alignment is calculated as follows: （3） In equation (3), It is the loss function of the personalized feature extractor. The feature representation of the i-th sample before update, extracted by the personalized feature extractor of the k-th client. represent Cosine similarity between the class prototype c and the class prototype c Represents belonging to category The set of global clustering prototypes is the set of positive samples. This indicates that it does not belong to the category. The set of global clustering prototypes is the set of negative samples. By calculating this loss, the personalized feature extractors of each client can produce semantically consistent feature representations while maintaining the separability between categories. For updating the shared classifier, the traditional cross-entropy loss is used: （4） In equation (4), It is the cross-entropy loss of the shared classifier, | | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. The loss represents the predicted value of the shared classifier for the i-th sample, and the shared classifier is updated by minimizing this loss. The total loss for updating the personalized feature extractor and the shared classifier is: （5） In equation (5), It is the total loss of the update. It is the cross-entropy loss of the shared classifier. Adaptive weights representing feature alignment It is the loss function of the personalized feature extractor, which is minimized by This improves the performance of the personalized feature extractor and the shared classifier, resulting in an updated personalized feature extractor and shared classifier. Step S1, updating the personalized classifier, specifically involves: It is updated using cross-entropy loss, and its loss function is: （6） （7） in, This refers to the total loss of the personalized classifier update. The cross-entropy loss represents the personalized classifier. | represents the local training set of client K. The number of samples, Let be the label value of the i-th sample. This represents the predicted output of the personalized classifier for the i-th sample; by minimizing... This improves the classification performance of the personalized classifier, resulting in an updated personalized classifier.

3. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 1, characterized in that, Step S2 involves performing local prototype clustering on the extracted features to obtain a local clustering prototype set. Specifically: Client k performs FINCH clustering on the features extracted from the personalized feature extractor. The clustering process is as follows: （8） In equation (8), This represents the local clustering prototype set of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m. Indicates the input sample Features obtained after the updated personalized feature extractor This represents the i-th input sample and its label. This represents the local training set where the k-th client belongs to category m; Step S2 describes calculating the locally unbiased prototype by performing a weighted average of the prototypes in the set. Specifically: After obtaining the feature prototypes for each category, the next step is to average the feature prototypes of each category to obtain the locally unbiased prototype. The calculation process is as follows: （9） In equation (9), The local unbiased prototype of client k belonging to category m. This represents the j-th feature prototype of client k belonging to category m. This represents the number of cluster prototypes for which client k belongs to category m.

4. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 1, characterized in that, Step S4 specifically involves: The server aggregates the local shared classifiers using the following formula: （10） In equation (10), Represents the aggregated shared classifier. This represents the number of clients participating in the aggregation. It is the local shared classifier for the kth client.

5. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 1, characterized in that, In step S5, the server receives the locally unbiased prototype. Perform global prototype clustering to obtain the global clustering prototype. Specifically: Using FINCH to process the uploaded partially unbiased prototype Clustering is performed, and the clustering process is as follows: （11） In equation (11), The set representing the global clustering prototypes of category m. The t-th global cluster prototype represents category m. The number of global cluster prototypes representing category m. The local unbiased prototype of class m representing client k. The set of locally unbiased prototypes representing each client class m.

6. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 1, characterized in that, Step S7 specifically includes: S71. After training is completed, firstly, the local test set is sent to the personalized feature extractor to extract the corresponding features, and then the features are sent to the personalized classifier and the shared classifier respectively to obtain personalized prediction output and shared prediction output. S72, Compare the features extracted by the personalized feature extractor with the global prototype. and locally unbiased prototypes The cosine similarity, where the global prototype For global clustering prototype The average was calculated. S73. The two cosine similarities calculated in step S72 are normalized using the softmax function to obtain the weights of the prediction outputs of the shared classifier and the individual classifier. The two prediction outputs obtained in step S71 are then fused using these weights to obtain the final inference prediction output. S74. Compare the final inference prediction output obtained in step S73 with the actual output to obtain the final test accuracy.

7. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 6, characterized in that, Step S72 involves calculating the features extracted by the personalized feature extractor and comparing them with the global prototype. and locally unbiased prototypes The cosine similarity is: （12） （13） （14） In equations (12), (13), and (14), For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by the personalized feature extractor during the inference phase and corresponding categories global prototype Cosine similarity between them yes Standardized metrics yes Standardized metrics yes Standardized metrics The number of global cluster prototypes representing category m. This represents the t-th global cluster prototype for category m.

8. The personalized federated learning method for asynchronous classifier optimization and prototype-aware reasoning according to claim 6, characterized in that, Step S73 specifically involves: The weights between the predicted outputs of the shared classifier and the individualized classifier are calculated using formula (15): （15） In equation (15), and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. + =1, For the client Features generated by personalized feature extractor and corresponding categories Local unbiased prototype Cosine similarity between them For the client Features generated by personalized feature extractor and corresponding categories global prototype Cosine similarity between them; Based on the weights of the prediction outputs of the individualized classifier and the shared classifier, the final inference prediction output is: （16） in, For inference and prediction output, and represent the predicted output weights of the shared classifier and the predicted output weights of the personalized classifier for client k, respectively. The predicted output of the personalized classifier for client k. This is the predicted output of the shared classifier.

9. The application of the classifier asynchronous optimization and prototype-aware reasoning personalized federated learning method as described in any one of claims 1-8 in medical image analysis, for personalized model optimization tasks in medical image analysis.

Images