Federal consistency representation learning system and method based on prototype guidance

Abstract

The invention belongs to the technical field of federal learning, and particularly relates to a federal consistency representation learning system and method based on prototype guidance. The system comprises two main modules: an in-source characterization calibration module and a cross-source consistent characterization learning module. The method comprises the following steps: firstly, improving local training by using an in-source representation correction module, and correcting feature distribution on unbalanced data; this helps mitigate significant differences in the feature space due to dataset deviations. Meanwhile, it provides prototype information to a server, including cluster prototypes, cluster variances, and attention scores. The cross-source consistent representation learning module then learns generalized projections and classifiers using prototype information obtained from all customers. According to the algorithm, firstly, statistical knowledge is utilized to generate extended features, feature space is refined, and diversity is improved; then, the features of different sources are mapped to a unified space for comparison and classification, and interference of isolated points is eliminated according to attention scores;

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and in particular relates to a prototype-guided federated consistency representation learning system and method. Background Art

[0002] Federated learning supports collaborative modeling using data from different sources, which shares model parameters instead of raw data between data sources to ensure privacy and security. This significantly improves the effective use of isolated data, enabling them to contribute to collaborative decision-making and learn a universal model. However, existing studies have emphasized that the heterogeneity of data distribution among clients may lead to reduced effectiveness of collaborative modeling. This is mainly because it becomes challenging to learn a consistent feature space when dealing with data that is unbalanced within the client and inconsistently distributed between clients, which makes it very difficult to integrate multiple learners with inconsistent goals into a significant model.

[0003] The inventors found that the prior art has the following technical defects: the existing federated learning method cannot guide the client model to use private data to learn a consistent feature space without sharing user privacy data, and ignores the imbalance of training data, which may hinder the effective representation learning and calibration of categories with limited sample sizes, resulting in low image classification accuracy of the aggregated model and lack of universality. Summary of the invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention provides a prototype-guided federated consistency representation learning system and method, which significantly improves the image classification effect of the existing methods in the federated learning scenario.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: In a first aspect, the present invention provides a prototype-guided federated consistent representation learning system, comprising: an in-source representation calibration module, which is used to model the representation distribution of the client, improve local training, and correct the feature distribution on imbalanced data; at the same time, it provides prototype information to the server, including cluster prototypes, cluster variances, and attention scores; The cross-source consistent representation learning module is used to learn generalized projections and classifiers. It first uses statistical knowledge to generate enhanced features, refine the feature space, and improve diversity. Subsequently, it maps features from different sources to a unified space for comparison and classification, and eliminates the interference of outliers based on attention scores.

[0006] In a typical implementation, the in-source representation calibration module includes two processes: knowledge-guided representation calibration and clustering-driven class pattern modeling.

[0007] Furthermore, knowledge-guided representation calibration specifically uses fixed category-aware text features learned from pre-trained CLIP (Contrastive Language-Image Pre-training) as the optimization target for local feature learning to normalize client-side feature learning. It performs supervised prototype contrast learning to maximize the image features in the latent space. and text prototype The consistency between them, the target loss is defined as: , in, Represents the client Medium The image features, , is a text encoder, represents the number of classes, is the image feature Tags, is the indicator function, , , For Clients The number of training data in Represents the temperature parameter.

[0008] At the same time, empirical loss is used to further ensure the classification ability of the model, that is, , in, Represents the first elements, is the label of the image.

[0009] In a typical embodiment, the cluster-driven class pattern modeling is specifically: using a fixed model learned from the source representation calibration module Features are extracted from all training data and k-means clustering method is used to mine different patterns in the latent space, i.e., , in, Represents the client Medium Category data, is a hyperparameter that represents the number of clusters. In addition, in order to obtain a more accurate distribution in the feature space, this module calculates the clusters The average value of the features within and variance ,Right now, , , in, Indicates the size of the cluster.

[0010] In addition, considering the limited ability caused by the imbalance of data distribution, in order to learn the discriminative representation of minority sample categories, the in-source representation calibration module further evaluates the importance of all clusters to reduce the interference of abnormal features on model calibration, which contains three factors, including cluster size , cluster compactness and the minimum distance to the cluster center of other categories For cluster , , , ,in, is with cluster Cluster centers of different categories, Indicates that it belongs to a cluster data characteristics. Essentially, the larger and more compact a cluster is, and the farther it is from the center of other clusters, the more important it is. The importance score can be expressed as Finally, the client The triple and local models Upload to the server, where Is the client It is worth noting that the cluster center is also called the local prototype.

[0011] In a typical implementation, the cross-source consistent representation learning module obtains all local models and local prototype sets uploaded from the client, and aligns the prototype features from the heterogeneous space, which mainly includes two processes: class-aware region calibration and cross-source feature alignment.

[0012] Furthermore, the class perception region calibration is specifically as follows: using knowledge transfer technology to calibrate the class perception region, using the Gaussian model Generate extended features based on variance. Variance with high importance score and other variances to transfer important knowledge to other categorical features in the client ,Right now, , in, is a local prototype, It is an enhanced feature The number of represents the fusion variance, Indicates the cluster variance corresponding to the client with the highest score.

[0013] It is worth noting that compared with the point-to-point approach in existing methods, the generated set of enhanced features forms regions, which helps the cross-source feature alignment module to achieve region-to-region alignment.

[0014] Furthermore, cross-source feature alignment is as follows: for the original global model , global feature extractor No retraining is required, and the projection head and classifier Need to calibrate, that is , The locally learned prototypes and enhanced features from the class-aware region calibration module are mapped to a new space designed for cross-source collaborative classification. At the same time, it refines representation learning using a two-level regularization method, including local consistency matching and complementary consistency matching, which can more effectively emphasize the commonality of intra-class features and the difference of inter-class features, eliminating customer-specific information.

[0015] For the local consistency matching level, it facilitates the learning process by imposing constraints on the consistency of the relationships between local representations, thereby guiding the model to obtain features that remain unchanged across different clients. This can be expressed in the following way: , in, Is a calibrated projection head After mapping, if Is the client Medium The local prototype of ;if is an enhanced feature, then , , Represents the dot product.

[0016] ,in, is the Euclidean distance.

[0017] In general, the local matching loss is defined as: .

[0018] For the complementary consistency matching level, it exploits the complementarity of features from different sources to promote the model to learn consistent features across customers, allowing the model to go beyond the limitations of a single perspective and achieve a more comprehensive level of learning. This can be defined as: .

[0019] Furthermore, to enhance the robustness of model correction and maintain a clear decision boundary, this module utilizes the importance scores of all features in a weighted manner To shift the focus of the model away from lower-quality features, a weighted supervised classification loss is designed, which is defined as follows: , in, is the attention score learned from the in-source representation calibration module, It is a classifier Learned predictions.

[0020] Furthermore, the cross-source consistent representation learning module calibrates the global model Sent to all clients.

[0021] In the second aspect, the present invention provides a federated consistent representation learning method based on prototype guidance, including: using private data in the client to train a private model, and using the class-aware text representation output by the pre-trained CLIP model to constrain the representation learning of the local model, the client stores a private data set and a private model, after the training, the trained private model is used to extract features from the private data set, the extracted features are clustered, the feature means within the cluster are regarded as prototypes, and the variance of the features within the cluster in each dimension and the importance score of the cluster prototype are calculated. The client sends the private model and prototype related information to the server; the server receives the private model and prototype uploaded by the client, sums the parameters of the private model, and then takes the average value to obtain the global model; the server uses the prototypes of all parties to calibrate the global projection head and the global classifier in the global model, and sends the calibrated global model to the client.

[0022] In a typical implementation, at least two clients and one server are included.

[0023] In a typical implementation, the private models in all clients have the same structure.

[0024] In a typical implementation, the training goal of the client is to calibrate the local distribution of the client to alleviate the significant differences in feature space between clients caused by imbalanced data distribution. The overall optimization goal of the client training strategy is: , in, is the weighting parameter; The server further reduces the differences in feature distribution in different spaces and optimizes the following objective functions: , in, is the weight parameter.

[0025] In a typical implementation, the relevant information is cluster prototype, intra-cluster variance, and importance score of cluster prototype.

[0026] In a typical implementation, in the intra-source representation calibration module, the weight parameter is 0.1-5, the temperature parameter is 0.5, and the number of clusters is 1-3; in the cross-source consistent representation learning module, the weight parameter is 0.1-5, and the number of extended features is 1-8.

[0027] Furthermore, in the source characterization calibration module, the weighted parameters , temperature parameters , the number of clusters .

[0028] In the cross-source consistent representation learning module, the weight parameter , extended feature number .

[0029] The present invention achieves the following technical effects: The present invention proposes a new prototype-guided federated consistent representation learning method, called FedCRL, which contains two main modules: an intra-source representation calibration module and a cross-source consistent representation learning module. In order to promote the alignment of features across different spaces, the intra-source representation calibration module uses pre-trained text representations to guide the local learning of each client to reduce the huge differences in the feature spaces of different sources; then, clustering is used to model the class patterns, and prototype information is provided to the server to assist model calibration. Subsequently, in order to enhance the robustness of the calibration, the present invention develops a class-aware region calibration method in the cross-source consistent representation learning module, which reconstructs the feature space of each source by utilizing knowledge transfer technology, alleviates the adverse consequences of unbalanced data distribution, and increases the diversity of samples in each category. At the same time, the present invention uses two layers of constraints to achieve effective alignment of cross-source features.

[0030] It is worth noting that the cross-source consistent representation learning module is a highly adaptable tool that can be easily integrated into various algorithms. Experiments were conducted on four datasets, including performance comparison, ablation study of key components, in-depth analysis of the effectiveness of cross-source consistent representation learning, and case studies. The experimental results show that the present invention is able to calibrate heterogeneous representations of different data sources into a unified space, and its performance is better than existing methods. Without the need to share user privacy data, the present invention calibrates heterogeneous features between clients into a unified space. The calibrated model has high image classification accuracy and is universal. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a prototype-guided federated consistency representation learning system in Embodiment 1 of the present invention; Figure 2This is a schematic diagram of poor performance of a federated learning model caused by poor client representation learning in federated learning in Example 1 of the present invention; Figure 3 A framework description of a prototype-guided federated consistency representation learning method in Example 1 of the present invention; Figure 4 Schematic diagram of robustness analysis of FedCRL under different hyperparameters in Example 1 of the present invention; Figure 5 A schematic diagram of the relationship between different numbers of enhanced features generated and classification results in Example 1 of the present invention; Figure 6 Schematic diagram of the effectiveness of the in-source characterization calibration module in Example 1 of the present invention; Figure 7 This is an error analysis diagram of the cross-source consistent representation learning method proposed in Example 1 of the present invention; Figure 8 Schematic diagram of characteristic distribution of FedCSPC in Example 1 of the present invention; Fig. 9 This is a schematic diagram of feature distribution after cross-source feature alignment in FedCRL in Example 1 of the present invention. DETAILED DESCRIPTION

[0032] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] The present invention is further described below in conjunction with embodiments.

[0035] Example 1 like Figure 1As shown, the present invention provides a prototype-guided federated consistent representation learning system to enhance the generalization ability of the global model, including: an intra-source representation calibration module, which is used to model the representation distribution of the client and provide prototype information to the server; a cross-source consistent representation learning module, which is used to learn generalization projections and classifiers, which first uses statistical knowledge to generate enhanced features, refine the feature space, and improve diversity; then, the features from different sources are mapped to a unified space for comparison and classification, and the interference of outliers is eliminated according to the attention score, which enables the global model to generalize to all clients.

[0036] 1. In-source characterization and calibration module The in-source representation calibration module aims to leverage pre-trained knowledge to help reduce the differences in feature space between clients and provide the server with prototypical information about representations, which helps with model calibration. It has two main processes: knowledge-guided representation calibration and modeling prototypical representations for data through clustering.

[0037] (1) Knowledge-guided representation calibration Intuitively, the change of the target is not conducive to optimizing the representation of categories with fewer samples. Therefore, the Knowledge Guided Representation Alignment (KGRA) module utilizes a fixed set of prototypes to regularize the feature learning of the client. Inspired by the pre-trained visual language model, the KGRA module uses the fixed category-aware text features learned from the pre-trained CLIP model as the optimization target for local feature learning, ensuring the consistency of all training rounds, promoting cooperation to make up for one's own shortcomings, and avoiding error accumulation. Specifically, it performs supervised prototype contrast learning to maximize the image features in the latent space. and text prototype The consistency between them, the target loss is defined as: , in, Represents the client Medium The image features, , is a text encoder, represents the number of classes, is the image feature Tags, is the indicator function, , , For Clients At the same time, the empirical loss is used to further ensure the classification ability of the model, that is, , in, Represents the first elements, is the label of the image.

[0038] (2) Clustering-driven class pattern modeling It is understandable that despite efforts to enhance feature learning, intra-class differences and inter-class overlaps still occur in the feature space, which generates noisy information that interferes with prototype modeling. Therefore, the clustering-driven class pattern modeling (CPM) module aims to explore the consistency and diversity of representations in each class and evaluate the importance of each prototype. Specifically, the CPM module utilizes a fixed model learned from the KGRA module Features are extracted from all the data and the k-Means clustering method is used to mine different patterns in the latent space, i.e., , in, Represents the client Medium Category data, is a hyperparameter that represents the number of clusters. In addition, in order to obtain a more accurate distribution in the feature space, this module calculates the clusters The average value of the features within and variance ,Right now, , , in, Indicates the size of the cluster.

[0039] In addition, considering the limited ability caused by the imbalance of data distribution, in order to learn the discriminative representation of minority sample categories, the in-source representation calibration module further evaluates the importance of all clusters to reduce the interference of abnormal features on model calibration, which contains three factors, including cluster size , cluster compactness and the minimum distance to the cluster center of other categories For cluster , , , ,in, is with cluster Cluster centers of different categories, Indicates that it belongs to a cluster data characteristics. Essentially, the larger and more compact a cluster is, and the farther it is from the center of other clusters, the more important it is. The importance score can be expressed as Finally, the client The triple and local models Upload to the server, where Is the client It is worth noting that the cluster center is also called the local prototype.

[0040] 2. Cross-source consistent representation learning module In general, simply training a model with a local prototype set that contains only a few samples and is partially noisy may degrade its generalization ability. Therefore, a cross-source consistent representation learning module is designed to improve the learning process by focusing on enriching the latent space and reducing the attention to noise. Specifically, it has two main processes: class-aware region calibration and cross-source feature alignment based on relation consistency.

[0041] (1) Class-aware region calibration In order to increase the diversity of the latent space, the Class-Aware Region Calibration (CRC) module uses knowledge transfer technology to calibrate the class-aware regions, thereby further alleviating the adverse consequences of imbalanced data distribution. Specifically, the CRC module uses a Gaussian model Generate enhanced features based on variance, and use variance with high importance score and other variances to transfer important knowledge to other categorical features in the client ,Right now, , in, is a local prototype, It is an enhanced feature The number of represents the fusion variance, Indicates the cluster variance corresponding to the client with the highest score.

[0042] It is worth noting that compared with the point-to-point approach in existing methods, the generated set of enhanced features forms regions, which helps the cross-source feature alignment module to achieve region-to-region alignment.

[0043] (2) Cross-source feature alignment based on relational consistency Obviously, as the imbalance of data distribution increases, it becomes challenging to completely eliminate the feature heterogeneity in the client. Therefore, a cross-source feature alignment (CSFA) module based on relation consistency is used to generalize the projection and classifier Retraining is performed so that the global model can achieve unified feature learning for samples with the same labels across data sources, that is, for the original global model , global feature extractor No retraining is required, and the projection head and classifier Need to calibrate, that is , Specifically, the CSFA module maps the locally learned prototypes and enhanced features from the CRC module to a new space designed for cross-source collaborative classification. At the same time, it refines the representation learning using a two-layer regularization method, including local consistency matching and complementary consistency matching, which can more effectively emphasize the commonality of intra-class features and the difference of inter-class features, eliminating customer-specific information.

[0044] For the local consistency matching level, it facilitates the learning process by imposing constraints on the consistency of the relationships between local representations, thereby guiding the model to obtain features that remain unchanged across different clients. This can be expressed in the following way: , in, Is a calibrated projection head After mapping, if Is the client Medium The local prototype of ;if is an enhanced feature, then , , Represents the dot product.

[0045] ,in, is the Euclidean distance.

[0046] In general, the local matching loss is defined as: .

[0047] For the complementary consistency matching level, it exploits the complementarity of features from different sources to promote the model to learn consistent features across customers, allowing the model to go beyond the limitations of a single perspective and achieve a more comprehensive level of learning. This can be defined as: .

[0048] Furthermore, to enhance the robustness of model correction and maintain a clear decision boundary, this module utilizes the importance scores of all features in a weighted manner To shift the focus of the model away from lower-quality features, a weighted supervised classification loss is designed, which is defined as follows: , in, is the attention score learned from the in-source representation calibration module, It is a classifier Learned predictions.

[0049] The global model calibrated by the cross-source consistent representation learning module Sent to all clients.

[0050] Example 2 like Figure 3 As shown, this embodiment provides a prototype-guided federated consistency representation learning method, including: The private model is trained using the private data in the two clients, and the representation learning of the local model is constrained by the class-aware text representation output by the pre-trained CLIP model. The client stores the private data set and the private model, and the private models in all clients have the same structure. After the training, the trained private model is used to extract features from the private data set, the extracted features are clustered, the feature mean within the cluster is regarded as the prototype, and the variance of the features within the cluster in each dimension and the importance score of the cluster prototype are calculated. The client sends the private model and prototype related information (cluster prototype, intra-cluster variance and cluster prototype importance score) to a server; the server receives the private model and prototype uploaded by the client, sums the parameters of the private model, and then takes the average value to obtain the global model; the server uses the prototypes of all parties to calibrate the global projection head and global classifier in the global model, and sends the calibrated global model to the client.

[0051] This method works synergistically at the client and server levels to alleviate the challenges of class imbalance across clients. It has the following training strategies.

[0052] In the client, the goal is to calibrate the local distribution of the client to alleviate the significant differences in the feature space between clients due to the imbalanced data distribution. The optimization target loss is defined as: , in, is the weighting parameter.

[0053] On the server side, FedCRL aims to further reduce the distribution differences in the feature space among different clients and optimize the following objective function: , in, is the weight parameter.

[0054] In summary, the present invention includes three main contributions: A new cross-source consistent representation learning framework FedCRL in federated learning is proposed, which can alleviate the negative impact of data imbalance and improve feature alignment between data sources. The proposed cross-source consistent representation learning module represents an orthogonal enhancement to the client-based approach. Its plug-and-play feature enables it to effectively combine multiple strategies and enhance the generalization performance of the corresponding models. Experimental results show that unbalanced training data weakens the benefits of cross-silo feature alignment to federated learning. This finding ensures the effectiveness of the proposed intra-silo representation calibration and cross-source consistent representation learning modules, which enhance the representation learning on the client and the cross-source feature alignment on the server, respectively.

[0055] In order to alleviate the negative impact of the differences between different feature spaces on model aggregation in joint learning, existing research can be roughly divided into two categories, including knowledge distillation-based methods and model calibration-based methods.

[0056] It should be understood that the knowledge distillation-based method is a promising strategy to alleviate data heterogeneity in federated learning, aiming to guide clients to learn consistent knowledge and construct similar feature spaces. Generally, in this research field, traditional methods need to use additional information as regularization to regulate local updates. In this context, regularization techniques play an important role. For example, Moon uses contrastive regularization to penalize the inconsistency between local and global feature spaces. FedProc, FedProto, and FPL construct prototypes for each class based on sample representations to represent the center of the intra-class representation. Then, it guides the local training process by constraining the representations of all clients to converge to these prototypes. In addition, using classifiers to guide the calibration of feature space is also an effective strategy, such as FedETF uses a fixed simplex equiangular tight frame classifier to encourage all clients to learn unified and optimal feature representations. FedFA uses feature anchors to simultaneously optimize the feature space and calibrate the classifier, promoting a virtuous cycle between feature space and classifier updates. Although the results of these methods are positive, further exploration of imbalanced data is needed because imbalance usually accumulates errors in training iterations.

[0057] It should be understood that in order to improve the performance of model aggregation in federated learning, methods based on model correction have become the focus of many studies. Unlike knowledge distillation, they focus on improvements on the server side, aiming to alleviate the bias problem caused by the weighted average of model parameters. Traditional methods along this line include global classifier calibration, projection head retraining, and global model fine-tuning. They all hope to obtain a universal model to adapt to all data from different sources. For example, CCVR fuses the mean and variance of sample features obtained from customers and uses a Gaussian mixture model to generate virtual features for retraining the global classifier. Creff and CLIP2FL generate a series of features whose gradients are consistent with the actual data to fine-tune the classifier. FedFTG fine-tunes the entire global model by using a generator to explore the input space. In addition, the goal of FedCSPC is to map different features into a common space for alignment and classification. From the previous analysis, their performance is closely related to the quality of local feature information, such as Figure 2 Therefore, alleviating the imbalance factor to enhance representation learning is a promising approach.

[0058] 1. Performance comparison In order to verify the effectiveness of the algorithm, the present invention uses three datasets in the experiment, including CIFAR10, CIFAR100 and TinyImageNet datasets. Table 1 below gives the statistical details of the datasets and uses The data set is divided using the Dirichlet distribution when .

[0059] Table 1 Statistics of the dataset used .

[0060] The present invention uses the Top-1 accuracy to evaluate the performance of the method, and the formula is as follows: , in, , are the number of correct predictions and the total number of samples, respectively.

[0061] For all methods, set the number of clients and ， Sampling ratio , the number of local training epochs , batch size , communication rounds on CIFAR10 and CIFAR100 datasets , communication rounds on the TinyImageNet dataset , and the learning rate of the SGD optimizer , the weight decay is set to .

[0062] In the in-source characterization calibration module, the weighting parameters , temperature parameters , the number of clusters ; In the cross-source consistent representation learning module, the weight parameter , extended feature number .

[0063] The present invention is compared with nine state-of-the-art methods, including FedAvg, MOON, CCVR, FedNTD, FedProc, FedDecorr, FedETF, FedCSPC, and CLIP2FL. The network architecture used for all methods includes an image encoder, a projection head, and a classifier. For all datasets, the present invention adopts a 2-layer multi-layer perceptron (MLP) as the projection head, and the classifier is a 1-layer fully connected layer. For the CIFAR10 dataset, the present invention adopts a convolutional neural network including two 5x5 convolutional layers followed by 2x2 maximum pooling, and two fully connected layers with ReLU functions as the image encoder. For other datasets, the present invention uses ResNet18 (excluding its last fully connected layer) as the image encoder. The following can be observed from Table 2 below: FedCRL is a general framework that can combine various knowledge extraction based methods such as FedAvg and FedETF, bringing them performance improvements, which shows its model-independent ability; Model calibration based methods usually outperform knowledge distillation based methods, which is demonstrated by this invention because it strives to use information from multiple sources to train a general model; FedETF uses a unified single-shaped equiangular tight frame classifier and often produces better results than data-driven knowledge based methods (FedProc, FedNTD, MOON). This may be because they avoid the problem of poor knowledge quality caused by data differences and inherent limitations of the model itself. As the number of data sources increases, the performance tends to decline. This is caused by the widening gap between data distributions. FedCRL maintains its advantage in performance, fully demonstrating the effectiveness of its calibration mechanism.

[0064] Table 2 Performance comparison of FedCRL and existing methods on CIFAR10, CIFAR100 and TinyImageNet datasets .

[0065] 2. Ablation Experiment Table 3. Ablation effect of different components of FedCRL on CIFAR10 and CIFAR100 datasets .

[0066] From Table 3 above, we can see that the Intra-Source Representation Calibration (ISRC) module plays a key role and contributes an average performance improvement of 1.2% to the baseline method, which verifies that providing unified guidance to different customers helps improve their collaborative results; the cooperation between the Class-Aware Region Calibration (CRC) module and the ISRC module improves the accuracy of the baseline method by about 3% in all cases; in general, the Cross-Source Representation Alignment (CSFA) module can also produce good results without CRC support because it reduces outlier interference in a weighted manner compared to existing methods; combining the ISRC, CRC, and CSFA modules produces the best performance. This is reasonable because the ISRC module provides reliable feature information, and the CSFA module optimizes the feature space and uses this information to reduce the distribution differences across different spaces.

[0067] 3. Robustness of FedCRL to Hyperparameters like Figure 4 As shown in Figure 2, FedCRL shows highly consistent results in all cases. This suggests that FedCRL is relatively robust and shows little sensitivity to the choice of hyperparameters over a wide range. Figure 4 The worst result shown in still outperforms the baseline. In addition, it is found that the best results are obtained when the number of clusters is small, set to 2. This is because a single cluster cannot model the variability within the class, and therefore ignores the interference of outliers. As the number of clusters increases, the significance of important features may decrease, which may shift the attention to noisy features.

[0068] 4. Generate different numbers of enhanced samples and classification results like Figure 5 As shown in Figure 2, in most cases, the greater the number of enhanced features generated, the greater the performance gain. This is because enriching the feature space through knowledge transfer can effectively simulate the true feature distribution. This can increase the information required to train the generalized model to alleviate the overfitting problem caused by the insufficient number of features. It is worth noting that using fewer extended samples can still improve the performance by about 3%. In addition, the inventors observed that when The performance of CIFAR100 decreases when . This decrease is partly due to the fact that CIFAR100 has much fewer samples per class than CIFAR10, making feature learning difficult. In addition, more classes in CIFAR100 increase the chance of overlapping with the class distribution, thus generating misleading information. Therefore, enhanced feature learning is of considerable importance in complex tasks.

[0069] 5. In-source characterization calibration This section aims to evaluate the impact of spatial distribution calibration on representation learning, prototype modeling, and model performance. The inventors selected two clients with different data distributions and class missingness, and used t-SNE to visualize the feature distribution of the two classes in the training test set. Figure 6 As shown, FedProc and FedCRL learn more discriminative representations under the guidance of knowledge compared to FedAvg, which is particularly evident for classes with most samples (e.g., sandy brown class). However, FedProc faces challenges with classes characterized by a limited number of samples due to error accumulation caused by the differences in the target space between different training rounds. It is worth noting that FedCRL uses consistent features to guide local training, ensuring similar representations for shared classes even if classes are missing. These factors lead to FedCRL's better performance than other methods. In addition, FedCRL also evaluates the importance of the generated prototypes. The results show that FedCRL assigns lower weights to prototypes in the overlapping areas of different classes. This helps the CSCRL module reduce the interference of outliers during model calibration.

[0070] 6. Error Analysis of FedCRL This section presents a case study based on GradCAM visualization to deeply investigate the working mechanism of FedCRL. Figure 7 As shown in (a), incorporating ISRC and CSCRL modules further enhances the discriminative ability of the basic method. This is reasonable because they can utilize a large number of samples to improve representation learning and cross-source feature alignment. Figure 7 (b) shows that limited sample size may cause local model failure, thus undermining the effectiveness of cooperation. The ISRC module enhances representation learning through distribution calibration, helping to align heterogeneous features within the CSCRL module to correct prediction errors. In addition, it finds that model calibration may fail. Although it makes accurate predictions before calibration, its reliability is questionable. It cannot accurately focus on the target, and the low-quality representation it learns hinders the improvement of subsequent calibration performance, such as Figure 7 Finally, Figure 7 (d) illustrates the case where these methods always use minority samples to make incorrect predictions for the class. It is worth noting that the CSCRL module promotes the model to focus on the target region and reduces the prediction difference between the ground truth and TOP-1. These findings verify the negative impact of data imbalance on joint learning and confirm the effectiveness of the proposed framework.

[0071] 7. Learning consistent representations across sources Two clients were randomly selected and two shared categories were selected from them (bird with fewer samples and airplane with more samples). Then, 200 samples were extracted from the test set of each category. Figure 8 and Fig. 9 The representation distributions of FedCSPC and FedCRL for a given sample are illustrated, as well as their CKA similarities and model performance. The results show that the representation distribution learned by FedCRL is more compact within the class and more discriminative between classes compared to FedCSPC. At the same time, FedCRL is able to reduce the heterogeneity of the feature space across clients before calibration, laying a solid foundation for cross-source feature alignment. This is also reflected in the similarity of CKA. In contrast, FedCSPC achieves better aircraft class alignment between the two clients compared to birds. This is because the limited representation quality learned from a small number of samples leads to significant differences that hinder feature alignment. In addition, the inventors note that distribution differences may lead to a decrease in collaborative performance (see Figure 8 ). Model calibration often enhances the personalization capabilities of local models by leveraging the knowledge of other customers to make up for their own shortcomings.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A prototype-guided federated consistent representation learning system, characterized in that: include: In-source representation calibration module, used to model the representation distribution of the client, improve local training, and correct feature distribution on imbalanced data; At the same time, it provides prototype information to the server, including cluster prototype, cluster variance, and attention score; A cross-source consistent representation learning module is used to learn generalized projections and classifiers. It first uses statistical knowledge to generate enhanced features and refine the feature space. , improve diversity; then, the features from different sources are mapped to a unified space for comparison and classification, and the interference of outliers is eliminated according to the attention score.

2. A prototype-guided federated consistent representation learning system according to claim 1, characterized in that: The in-source representation calibration module includes two processes: knowledge-guided representation calibration and clustering-driven class pattern modeling.

3. A prototype-guided federated consistent representation learning system according to claim 2, characterized in that: The knowledge-guided representation calibration specifically uses fixed category-aware text features learned from pre-trained CLIP as the optimization target for local feature learning to normalize the client's feature learning; it performs supervised prototype contrastive learning to maximize the image features in the latent space. and text prototype The consistency between them, the target loss is defined as: ， in, Represents the client Medium The image features, , is a text encoder, represents the number of classes, is the image feature Tags, is the indicator function, , , For Clients The number of training data in represents the temperature parameter; at the same time, the empirical loss is used to further ensure the classification ability of the model, that is, ， in, Represents the first elements, is the label of the image.

4. A prototype-guided federated consistent representation learning system according to claim 2, characterized in that: The cluster-driven class pattern modeling is specifically: using a fixed model learned from the source representation calibration module Features are extracted from all training data and k-means clustering method is used to mine different patterns in the latent space, i.e., ， in, Represents the client Medium Category data, is a hyperparameter that represents the number of clusters.

5. The prototype-guided federated consistent representation learning system according to claim 1, characterized in that: The cross-source consistent representation learning module obtains all local models and local prototype sets uploaded from the client, and aligns the prototype features from heterogeneous spaces, which mainly includes two processes: class-aware region calibration and cross-source feature alignment.

6. A prototype-guided federated consistent representation learning system according to claim 5, characterized in that: The class perception area calibration is specifically: using knowledge transfer technology to calibrate the class perception area, using Gaussian model Generate extended features based on variance, and use variance with high importance score and other variances to transfer important knowledge to other categorical features in the client ,Right now, ， in, is a local prototype, It is an enhanced feature The number of represents the fusion variance, Indicates the cluster variance corresponding to the client with the highest score.

7. The prototype-guided federated consistent representation learning system according to claim 5, characterized in that: The cross-source feature alignment is specifically as follows: for the original global model , global feature extractor No retraining is required, and the projection head and classifier Need to calibrate, that is , .

8. The prototype-guided federated consistent representation learning system according to claim 7, characterized in that: The cross-source feature alignment adopts a two-level regularization method to refine representation learning, including local consistency matching and complementary consistency matching.

9. The prototype-guided federated consistent representation learning system according to claim 8, characterized in that: The local consistency matching is expressed in the following way: ， in, Is a calibrated projection head After mapping, if Is the client Medium The local prototype of ;if is an enhanced feature, then , , represents the dot product; ， in, is the Euclidean distance; The local matching loss is defined as: ； The complementary consistency match is defined as: 。 10. A prototype-guided federated consistency representation learning method, characterized in that: The system according to any one of claims 1 to 9 comprises: training a private model using private data in a client, and using a class-aware text representation output by a pre-trained CLIP model to constrain the representation learning of a local model, wherein a private data set and a private model are stored in the client, and after the training, the trained private model is used to extract features from the private data set, the extracted features are clustered, the feature means within the cluster are regarded as prototypes, and the variance of the features within the cluster in each dimension and the importance score of the cluster prototype are calculated; the client sends information related to the private model and the prototype to the server; the server receives the private model and prototype uploaded by the client, sums the parameters of the private model, and then takes the average value to obtain a global model; the server uses the prototypes of all parties to calibrate the global projection head and the global classifier in the global model, and sends the calibrated global model to the client.

Images