Generalizable federated learning method and system based on consistency of perception heterogeneity features and base matrix

Abstract

This invention proposes a generalizable federated learning method and system based on perceptual heterogeneity features and consistency of the fundamental matrix. The system includes a central server and multiple clients. The central server has an unlabeled public image dataset, and each client holds a labeled private image dataset and deploys a local image classification model that can be decomposed into an image feature extractor and a classifier. Each client is equipped with a feature mapping layer that projects the image features output by the image feature extractor onto a unified high-dimensional feature space. This invention solves the heterogeneity and catastrophic forgetting problems through the FBCL federated learning framework.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a generalizable federated learning method and system based on the consistency of perceptual heterogeneity features and fundamental matrices. Background Technology

[0002] In recent years, the rapid development of artificial intelligence (AI) in computer vision, especially in image classification, has profoundly impacted many important fields such as medical image diagnosis, autonomous driving environmental perception, industrial visual quality inspection, and security monitoring. However, these applications often involve large amounts of sensitive or proprietary image data, and their collection, sharing, and centralized training face severe data privacy and security challenges. Federated learning (FL), as a novel decentralized training paradigm, has emerged to address this need. It allows multiple participants (clients) to collaboratively train models without sharing the original data, providing an innovative solution for building high-performance image classification models while protecting image data privacy. For example, multiple hospitals can collaboratively train a more powerful and robust disease diagnosis model without disclosing patients' original medical images (such as X-rays and CT slices); multiple intelligent transportation systems can jointly optimize traffic sign or pedestrian recognition models without uploading the original video streams from vehicle cameras.

[0003] Despite the significant advantages of federated learning in protecting image data privacy, it still faces many core challenges stemming from the data and the model itself in practical applications of building image classification models:

[0004] (1) Image data feature heterogeneity: In image classification tasks, image data collected by different clients often exhibit significant distributional differences. This image feature heterogeneity stems from various factors, such as: different medical imaging equipment (e.g., CT, MRI) models and imaging parameters used by different medical institutions; changes in lighting, weather, and viewing angles faced by surveillance cameras deployed in different cities or environments; and differences in background and resolution between product images taken by different factory quality inspection lines. This heterogeneity can be formally expressed as... ,in This represents image data. It causes each client's local model to tend towards the optimal solution for its local image data, which can lead to slow convergence, decreased accuracy, and even model divergence when using the classic FedAvg algorithm to collaboratively train image classification models.

[0005] (2) Image classification model structural heterogeneity: Participants in federated learning may adopt different image classification model architectures based on their own hardware computing power, real-time requirements, or historical technology choices. For example, some may use the lightweight MobileNet to meet mobile deployment needs, while others may use the deep residual network ResNet to pursue higher accuracy. This model structural heterogeneity, i.e. This makes it difficult to average and aggregate traditional model parameters, directly affecting the efficiency and effectiveness of collaborative training across heterogeneous models.

[0006] (3) Catastrophic Forgetting in Image Classification Tasks: In the dynamic environment of federated learning, when the model is updated to adapt to the image data distribution of newly added clients or new scenes, it may severely forget the image features and classification knowledge previously learned from other clients, leading to a sharp decline in performance on old image classification tasks. This is the "catastrophic forgetting" problem. Existing mitigation methods (such as parameter isolation and replay buffers) often significantly increase the complexity and storage overhead of image classification models. Currently, research on the heterogeneity problem in federated learning mainly focuses on personalized federated learning, which aims to customize a model for each client. However, this sacrifices the model's cross-domain generalization ability to some extent, meaning that the trained model may perform poorly on unseen image data distributions or clients. In addition, many methods rely on directly sharing model parameters or category prototypes between clients, which still poses certain privacy risks or high communication costs.

[0007] To address the complex technical challenges of federated learning in image classification scenarios, including intertwined issues such as feature and model heterogeneity, communication efficiency, privacy protection, and catastrophic forgetting, this invention proposes FBCL (Federated Feature and Basic-matrix Consistency Learning)—a novel, heterogeneous-aware, generalizable federated learning framework for image classification tasks. The core of this invention lies in utilizing publicly available unlabeled general image data (such as CIFAR and subsets of ImageNet) as a "bridge." By implementing instance-level distribution alignment and basic matrix consistency constraints in the feature space, the representational capabilities of heterogeneous client image feature extractors are implicitly aligned, thereby learning a shared feature space with strong discriminative and generalizable properties. Simultaneously, by designing targeted local training constraint mechanisms, global knowledge is effectively preserved when adapting to local image data, mitigating catastrophic forgetting. Summary of the Invention

[0008] The purpose of this invention is to propose a generalizable federated learning method and system based on the consistency of perceived heterogeneity features and fundamental matrices, and to solve the problems of heterogeneity and catastrophic forgetting through the FBCL federated learning framework.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows:

[0010] A generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix is ​​applied to a federated learning system for image classification tasks. The system includes a central server and multiple clients. The central server holds an unlabeled public image dataset, while each client holds a labeled private image dataset and deploys a local image classification model that can be decomposed into an image feature extractor and a classifier. Each client is equipped with a feature mapping layer that projects the image features output by the image feature extractor onto a unified high-dimensional feature space. Each iteration of model training in this method includes the following steps:

[0011] S1. The central server collects public images from the unlabeled public image dataset and sends them to all clients participating in this round of training. Each client uses the public images in this batch to update the parameters of the feature extractor and the feature mapping layer by minimizing the joint loss function of feature instance-level alignment loss and feature basis matrix consistency loss, and obtains an aligned local image classification model.

[0012] S2. Each client samples labeled images from its private image dataset and uses the local image classification model obtained from the previous training round or the aggregated global model issued by the central server as the teacher model. Using the labeled images of this batch, the parameters of the complete local image classification model are updated by minimizing the joint loss function, which includes cross-entropy loss, correct non-target distillation loss, and feature relation regularization loss, to obtain the local image classification model after this training round.

[0013] S3. Each client determines the degree of difference between the local image classification model trained in the current round and the previous round. If the degree of difference is less than the preset threshold, the parameters of the local image classification model are frozen and no longer updated; otherwise, return to step S1.

[0014] Preferably, after obtaining the local image classification model after this round of training, the central server aggregates the local image classification models of each client, generates a global model, and distributes it to the client.

[0015] Preferably, for clients that have already participated in training, the trained local image classification model is used directly for image classification tasks; for new clients that have not participated in training, the aggregated global model is deployed for image classification tasks.

[0016] Preferably, the calculation of the feature instance-level alignment loss is as follows:

[0017] Each client calculates the Gaussian distribution parameters of the feature vectors obtained from the feature extractor and feature mapping layer of the public images in this batch:

[0018]

[0019]

[0020] in, It is a mean vector. It is the covariance matrix. It is the size of the feature map. It is the aligned feature vector obtained after the feature extractor and feature mapping layer process the sample. It is the feature vector processed by the feature extractor of the i-th client;

[0021] Each client calculates the loss due to the difference between its own feature distribution and the average feature distribution of all clients. :

[0022]

[0023] in, and Let be the mean vector and the average of the covariance matrix of all client feature distributions, respectively. It is the L2 norm, used to measure the similarity between two feature outputs.

[0024] Preferably, the consistency loss of the feature basis matrix is ​​calculated by each client calculating the orthogonality loss between feature vectors of different samples within the same batch. :

[0025]

[0026] in, This is the first in this batch The first sample and the first The degree of difference between individual samples It is the feature vector of the u-th sample in this batch. It is the feature vector of the v-th sample in this batch. This represents the absolute value operation. The L2 norm (or L2 norm) represents the magnitude of a vector. It refers to the batch size.

[0027] Preferred, correct non-target distillation loss The calculation is as follows:

[0028]

[0029] in, These are sample images from the labeled images in this batch. For the true labels of the sample images, For category indexing, This indicates that all categories without actual labels are traversed; Teacher model prediction sample image Category The distribution of soft tags, The aligned local image classification model prediction sample image obtained in step S2 Category The distribution of soft labels; It is the KL divergence; This is the confidence parameter for historical knowledge. If the teacher model's prediction is correct, that is, the maximum predicted probability output corresponds to the label... ,but ;otherwise, .

[0030] Preferably, the feature relation regularization loss The calculation is as follows:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] in, , , These are the local model's output features for this batch of samples. The Middle The, the The, the The output features of each sample , , The global model outputs features for this batch of samples. The Middle The, the The, the The output features of each sample The distance difference between two feature vectors. , These are all weight parameters for the distance scale, and N is the number of samples in this batch; For two eigenvectors with respect to the anchor vector or The difference in the included angle, It is the dot product of vectors. It is a 2-norm.

[0037] A generalizable federated learning system based on perceptual heterogeneity features and consistency of the fundamental matrix is ​​disclosed. The system is implemented using any of the above-mentioned federated learning methods and includes a central server and multiple clients. The central server has an unlabeled public image dataset for the image classification task. Each client holds a labeled private image dataset and is equipped with a local image classification model that can be decomposed into an image feature extractor and a classifier. Each client is equipped with a feature mapping layer that projects image features onto a unified high-dimensional feature space.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention proposes a novel federated learning method that perceives heterogeneity, utilizing a Gaussian distribution to characterize the features of samples from different channels on unrelated, unlabeled public data. By aligning cross-domain features, an original cross-domain consistency learning approach is designed to facilitate communication between heterogeneous clients and learn feature representations with stronger generalization capabilities.

[0040] This invention designs a novel feature basis matrix consistency learning mechanism to establish a well-structured feature space, which encourages features from different categories to be distributed along orthogonal directions, thereby reducing interference between features and improving feature discriminability and generalization potential.

[0041] This invention introduces an innovative dual consistency constraint method to preserve the global perspective of the old model during local updates. Combined with label supervision, it balances the knowledge of different models, mitigates the catastrophic forgetting problem, and improves both inter-domain and intra-domain performance.

[0042] Extensive experiments using real-world unlabeled public datasets validated the superior performance of the proposed FBCL both intra- and inter-domain, outperforming state-of-the-art methods across various image classification tasks. Furthermore, ablation studies verified the effectiveness of the core components of FBCL, demonstrating its robustness and practicality in diverse scenarios. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the FBCL federated learning framework of the present invention;

[0044] Figure 2 This is an example diagram of the Digits and Office-Caltech datasets in an embodiment of the present invention;

[0045] Figure 3 This is a comparison chart of the average intra-domain and inter-domain accuracy of Digits and Office-Caltech with CIFAR-100 in an embodiment of the present invention. Detailed Implementation

[0046] The following is in conjunction with the appendix Figure 1-3 The technical solution of the present invention will be described in detail below.

[0047] In federated learning for image classification tasks, there are... Each participating client. Having a private image dataset ,in For an image, It is its corresponding category label. This represents the total number of samples. An image classification model is deployed locally on the client side. The model is for images The predicted output is denoted as . Indicates client The joint distribution of image data and its labels. This invention defines the core challenges in federated learning for image classification as follows:

[0048] (1) Image Feature Heterogeneity: Due to differences in acquisition equipment, shooting environment, lighting conditions, and resolution among image data from different clients (such as CT equipment in different hospitals or surveillance cameras in different cities), the distribution of deep visual features extracted from these images is inconsistent. This heterogeneity in image feature distribution can be formally expressed as... ,in Represents the image data space.

[0049] (2) Image classification model heterogeneity: Participants may adopt image classification models with different structures and capacities based on their own computing resources, storage limitations, or specific needs (for example, some clients use the lightweight MobileNet, while others use the deeper ResNet). This heterogeneity in model structure can be described as follows: .

[0050] This invention proposes a generalizable federated learning method based on the consistency of perceptual heterogeneity features and the fundamental matrix, such as... Figure 1As shown, the proposed FBCL operates through two integration phases. During collaborative updates, feature instance alignment is performed using public data, and features are represented by a unified fundamental matrix through Feature Instance-level Alignment (FIA) and Feature Basic-matrix Consistency (FBM), reducing model redundancy while enhancing cross-domain adaptability. This approach minimizes the privacy risks and communication overhead inherent in traditional federated learning by enabling secure knowledge transfer between clients. When transitioning to local updates, FBCL employs Correct Not-target Distillation (CND) combined with Feature Relationship Regularization (FRR) to balance intra-domain / inter-domain information. Specifically, under data heterogeneity, it mitigates catastrophic forgetting by preserving previous cross-domain knowledge while adapting to the new local distribution.

[0051] 1. Feature instance-level alignment

[0052] Inspired by autoencoders, a suitable feature distribution contains rich information and can replace the direct transmission of model parameters, thereby reducing communication overhead. This applies to image classification models for each client. Split into image feature extractor (Typically, convolutional neural networks (CNNs) and classifiers) Unlike prototype-based methods (which learn domain-invariant representations that reveal category-level statistics), this invention utilizes public data. This invention approximates cross-model feature extraction capabilities. To address the issue of model heterogeneity due to different output dimensions of feature extractors, this invention provides a client-side solution. Add a mapping branch (Right now Convolution (or convolution) transforms features into the same high-dimensional space. For public data... Features extracted from different clients should be mapped to similar locations:

[0053] (1)

[0054] in , It is a sample The average value after being mapped by all clients. This invention measures the similarity between two feature outputs. Considering that traditional pooling operations discard a significant amount of spatial information, this invention aligns feature distributions by channel to capture complex activation patterns across the entire feature map. Directly calculating Euclidean distance is limited to comparing feature offsets in a single channel and lacks a mechanism for cross-sample comparison. Furthermore, image depth features are typically high-dimensional and sparse, making KL divergence inapplicable when distributions do not overlap. Therefore, a Gaussian distribution is used to model the feature vectors extracted from the high-dimensional space, defined as:

[0055] (2)

[0056] in It is the covariance matrix. It is a mean vector. It is the size of the feature map. It is a matrix determinant. By representing features in this probabilistic manner, features from different clients can be better aligned and compared, contributing to the formation of more consistent and robust feature representations. Therefore, the loss of FIA is defined as:

[0057] (3)

[0058] 2. Consistency of the feature basis matrix

[0059] Relying solely on average features for model updates can lead to convergence to outliers (e.g., like...). This risks mapping all inputs to trivial solutions at the same location. To capture diverse invariant features and mitigate feature redundancy, the discriminative power between features of different samples is maximized while minimizing the distance between features of the same samples. Since public data is unlabeled, directly evaluating the distance between different samples in a high-dimensional space is infeasible. Instead, this invention measures the positional relationship of samples through feature orientation. The feature vectors of each batch of samples can form a feature matrix. The difference between features is maximized if the feature vectors are orthogonal to each other and constitute a subset of the solution to the fundamental matrix. The first feature vector in each batch... The and the first The difference between samples is denoted as . The value ranges from 0 (completely unrelated) to 1 (completely identical). Since the normal vector of each feature is not unique in high-dimensional space, when the batch size... Less than dimension When calculating the direction of the feature vector, a direction should be set to approximate the true direction of the feature vector. For simplicity, the average direction of the feature vector is taken as the preset direction. The difference between the mapping features of each sample and the preset direction is quantified as follows: Since this optimization direction is consistent with equation (3), it can be ignored. Therefore, the loss of FBM is defined as:

[0060] (4)

[0061] FBM loss encourages the orthogonality and diversity of features, enabling FBCL to learn comprehensive and robust representations. Furthermore, by combining... and The overall loss for collaborative updates is updated as follows:

[0062] (5)

[0063] 3. Correct non-target distillation

[0064] While soft labels from older models provide richer inter-class relationship information than hard labels, enhancing knowledge transfer under complex distributions, their inconsistencies and potential misalignment with true labels propagate errors during distillation. To address this issue, this invention proposes a novel CND method that calculates the probability distribution using only "non-target class" information during training, thereby minimizing the negative impact of erroneous labels. Weight parameters are then set for the predicted probabilities of the corresponding labels. Therefore, the samples... The CND loss is defined as:

[0065] (6)

[0066] in This is the confidence parameter for historical knowledge. If the old model's prediction is correct, that is, the maximum predicted probability output exactly corresponds to the label... ,but ;otherwise, . It's about the old model. The distribution of soft tags, Is the current model about The distribution of soft tags, It is the KL divergence.

[0067] 4. Feature Relationship Regularization

[0068] Relying solely on locally supervised learning carries the risk of overfitting the current data distribution, thereby reducing inter-domain performance. To prevent feature representation drift, this invention incorporates the corresponding global model. The global model serves as the teacher model to guide local updates. Since the global model acts as a feature extractor and does not output specific logits, classic logits-based knowledge distillation strategies cannot be directly applied. To achieve effective knowledge transfer between the global and local models, this invention utilizes a feature relation matrix... To constrain changes in the local model, it is defined as:

[0069] (7)

[0070]

[0071]

[0072]

[0073]

[0074] in, , , These are the local model's output features for this batch of samples. The Middle The, the The, the The output features of each sample , , The global model outputs features for this batch of samples. The Middle The, the The, the The output features of each sample The distance difference between two feature vectors. , These are all weight parameters for the distance scale, and N is the number of samples in this batch; For two eigenvectors with respect to the anchor vector or The difference in the included angle, It is the dot product of vectors. It is a 2-norm.

[0075] This consistency constraint ensures that the distribution of feature representations in high-dimensional space remains unchanged between the source and target domains, thus avoiding over-reliance on domain-specific data. It enhances the model's generalization ability across different domains while mitigating the effects of catastrophic forgetting, thereby enabling more efficient knowledge transfer and sharing among clients.

[0076] 5. In addition to ensuring the similarity between local and global features, the classification features of the local model also need to have sufficient discriminative power to distinguish different categories in the feature space. Therefore, this invention retains the label loss, which is defined as:

[0077] (8)

[0078] Therefore, the total loss update for local updates is:

[0079] (9)

[0080] 6. Proof of Global Generalization Error Bound

[0081] Consider a target domain that is a collection of all client data (i.e.) If the generalization error of the global model has an upper bound, then the generalization error of the global model has an upper bound. In the classic federated learning (FL) scenario, there are K clients. The local model of client k consists of a feature extractor. and a classifier These models are composed of local private data. Training is performed. The system maintains a global feature space. And learn a feature transformer The target model consists of a feature extractor. and a classifier Composition. Our goal is to utilize publicly available unlabeled data. Align the features of each client and minimize the global error. To simplify the problem, we first make the following assumptions: Assumption 1: Loss function yes -Lipschitz continuous, and upper bounded Assumption 2: Feature Transformer yes -Lipschitz continuous, i.e.

[0082] Assume a 3-dimensional global classifier yes -Lipschitz, i.e.

[0083] Assume 4 global feature space The diameter is ,Right now

[0084] Assumption 5: Public Data As the target domain The agent has a divergence of Based on the above assumptions, the upper bound of the global generalization error of the K local models can be defined as:

[0085]

[0086] in It is a constant complexity. It is the minimum number of samples per client, and .

[0087] Proof: The generalization error of the global model consists of client-specific error, alignment error, domain difference, and statistical terms, defined as:

[0088]

[0089] According to the triangle inequality, for client k, we have:

[0090]

[0091] according to of -Lipschitz property, we have: terms

[0092] Alignment loss Applying Jensen's inequality, we have:

[0093]

[0094] Combination Lipschitz constant and diameter ,have:

[0095] Item II measures the global classifier With local classifier In transforming features Differences in performance. Assuming local-global classifier consistency (i.e.) If ), then term II is 0; otherwise, it is absorbed into the client error. middle:

[0096]

[0097] According to the multi-source domain adaptation theory, the proxy domain With the target domain The divergence between them is bounded:

[0098]

[0099] Using Rademacher complexity in federated learning, the statistical terms are bounded:

[0100]

[0101] in Here are the confidence parameters. Considering all the above, the upper bound of the generalization error of the global model is:

[0102]

[0103] 7. Specific Implementation Steps of the Algorithm

[0104] Phase 1: System Initialization and Client-Side Information Preparation: Prepare a task-related, unlabeled public image dataset. (For example, a subset of images randomly selected from CIFAR-10 or ImageNet). This dataset does not contain any client-private category information and is used only for feature alignment. Determine the total number of communication rounds. Participating client set (total) (number of items), hyperparameters () ), optimizers (such as Adam), and learning rates Initialize or define a basic global image classification model architecture. Client preparation: Each client... They hold their private, labeled image datasets. For example, a medical image dataset from a hospital. Each client independently selects or initializes its local image classification model based on its own capabilities (computing resources, latency requirements). This allows for model heterogeneity; for example, client A can use ResNet-18 while client B uses MobileNetV2. Each model can be broken down into image feature extractors. and classifier Each client has its own feature extractor. Add a learnable feature mapping layer (such as one) Convolutional layers are used to project extracted image features into a unified, shared high-dimensional feature space.

[0105] Phase Two: Iterative Model Training for Each Round of Communication Perform the following steps:

[0106] Step 1: Collaborative Update (Global Knowledge Alignment and Generalization Enhancement): This stage aims to utilize public image data to align the feature representations of each client and learn a shared feature space with strong generalization capabilities. 1) Server Distribution of Public Data: The central server distributes public data from... A batch of samples (size is) is taken from the middle. (public image) And send it to all clients participating in this round of training. 2) Client feature extraction and mapping: Each client Using its current local model, via image feature extractor and mapping layer Processing public images The mapped feature set is obtained. ,in 3) Calculate the characteristic distribution parameters (corresponding to formula (2)): for each client Calculate the Gaussian distribution parameters of the feature vectors of this batch: mean vector The covariance matrix is ​​the average of all eigenvectors at a given location in the feature map space. : Correlation matrix between each dimension of the feature vector. 4) Calculate the feature instance-level alignment loss (FIA) (corresponding to formula (3)): Each client calculates the difference loss between its own feature distribution and the average feature distribution of all clients. This step encourages different clients to produce similar deep visual feature statistics for the same public image. 5) Calculate the feature basis matrix consistency loss (FBM) (corresponding to formula (4)): Each client calculates the orthogonality loss between feature vectors of different samples within its batch. This step encourages the model to map different input images onto directions in the feature space that are as orthogonal as possible, enhancing the discriminativeness and diversity of features and building a more clearly structured feature space for image classification.

[0107] 6) Update the feature extractor and mapping layer (corresponding to formula (5)): for each client To minimize joint loss For the target, only update its image feature extractor. and feature mapping layer Parameters (classifier) (No updates are performed in this phase). The update formula is: .

[0108] Output: Aligned and updated local model It has better feature generalization ability.

[0109] Step Two: Local Update (Adapting to Private Data and Retaining Historical Knowledge): This stage aims to optimize model classification accuracy using private image data, while preventing the forgetting of general feature representations learned in the collaborative stage through constraints. 1) Local Data Sampling: Each client From its private image dataset A batch of samples (size is) is taken from the middle. ) marked image 2) Calculate the correct non-target distillation loss (CND) (corresponding formula (6)): Client The model obtained from the previous training round (or the aggregated global model distributed by the server) is used as the "teacher model". Using the teacher model and the current model Predict the images in the current batch. Only select samples with high confidence in the teacher model's prediction of the true labels, and distill their soft probability distribution for the "non-true label" category, calculating the distillation loss. This effectively preserves valuable global historical knowledge.

[0110] 3) Calculate the feature relation regularization loss (FRR) (corresponding to formula (7)): Client Features of the same batch of private images are extracted using both the teacher model (representing a global perspective) and the current model, and their feature relationship matrices (e.g., sample similarity matrices) are calculated. The difference loss between the two relationship matrices is minimized. This constrains the current model's understanding of the relationships between samples in the feature space to remain consistent with the global consensus, thereby stabilizing the feature representation and mitigating catastrophic forgetting. 4) Calculate the domain-specific discriminative learning loss (corresponding to formula (8)): Client Using standard cross-entropy loss Train on private image data to ensure the model has strong local image classification capabilities. 5) Update the complete local model (corresponding to formula (9)): Client To minimize joint loss To achieve this, update the complete parameters of its local image classification model (including the feature extractor, mapping layer, and classifier). The update formula is: Output: Updated local model It adapts to local image data while retaining global generalization knowledge.

[0111] Step 3: Determine if the iteration terminates

[0112] Each client determines the model for this round. Compared with the previous model If the degree of difference is less than a threshold, it is considered to be in dynamic equilibrium, and the corresponding model parameters are frozen and no longer updated. Otherwise, proceed to step one.

[0113] Phase Three: Deployment and Reasoning

[0114] go through After one round of training, a series of improved client-side local models can be obtained. A strongly generalized global model (if aggregation is performed) During inference, the appropriate model can be selected based on the scenario. Personalized model: On the client side. In the local environment, directly use its trained... Classifying new images yields optimal results. Global model: Deploy the global model on new devices or clients that were not trained on. Because it learns generalized feature representations, it also performs well on cross-domain image classification tasks.

[0115] 8. Method Evaluation

[0116] The proposed FBCL is evaluated on two real-world datasets: Digits and Office. The Digits dataset contains four domains: MNIST, USPS, SVHN, and SYN, each containing digits in 10 categories from 0 to 9. In each domain, 100 samples are randomly selected from each category to form the training set, while the test set remains unchanged. For the Office dataset, 30% of the samples in each category of each domain are randomly selected as the test set, and the remaining 50% are randomly selected as the training set. Based on these settings, each client is assigned data from one domain, with no overlap with data from other domains. Figure 2 As shown, datasets from different domains exhibit significant feature heterogeneity. For public datasets, CIFAR and ImageNet datasets are used, with 5,000 images randomly selected. To verify the robustness and generalization of FBCL, various mainstream model architectures are considered as backbone networks, including ResNet, EfficientNet, and MobileNet. Table 1 summarizes the model architecture settings for different datasets.

[0117] Table 1: Model structure settings for different datasets

[0118]

[0119] To ensure fairness, the same parameter settings were used for the proposed FBCL and existing state-of-the-art methods. Specifically, the number of clients was set to 4 for all datasets. The Adam optimizer was used with a learning rate of 0.001. Hyperparameters were set. , , For CIFAR-100, the number of unlabeled public data points is 5,000. During local updates, the batch size is set to 128 and the number of iterations is 40. During collaborative updates, the batch size is set to 256. For preprocessing, all input images are adjusted to... To ensure compatibility, no data augmentation was performed, and the baseline model was trained based on 100 iterations of optimization using only private data. The following commonly used performance metrics were used to evaluate model accuracy.

[0120] (11)

[0121] (12)

[0122] in It is the first Number of test samples on each client.

[0123] 8.1 The experimental results and analysis are as follows:

[0124] Intra-domain performance: As shown in Table 2, the Baseline algorithm, optimized only using local data and without benefiting from collaborative learning with other domains, performs worst on the simple Digits dataset but achieves comparable performance on the complex Office dataset. This suggests that other algorithms may fail to effectively extract domain-invariant features, implying that blindly pursuing cross-domain joint communication may be counterproductive. In contrast, the proposed FBCL achieves best or near-best performance on all datasets. Figure 3 As shown, FBCL converges with fewer communication rounds and exhibits minimal oscillations thereafter, demonstrating its efficiency and stability.

[0125] Table 2: Intra-domain performance comparison with CIFAR-100 on the Digits and Office-Caltech datasets

[0126]

[0127] Inter-domain performance: To evaluate the effectiveness of different methods in addressing catastrophic forgetting, this invention analyzes their inter-domain performance on different datasets. As shown in Table 3, the baseline performs poorly in all cases, indicating that communication generally has a positive impact. FCCL+ and FBCL significantly outperform other methods, which is attributed to their unique design for catastrophic forgetting, demonstrating that preserving global model information significantly enhances cross-domain performance. Notably, except for the Office-31 dataset, FBCL slightly outperforms FCCL+, especially achieving approximately 1.46% higher performance on the Office-Caltech dataset. These results highlight the robustness of FBCL in maintaining high performance across different domains and mitigating the impact of catastrophic forgetting.

[0128] Table 3: Cross-domain performance comparison with CIFAR-100 on different datasets (e.g., "mnist+" indicates that "mnist" is private data and the model is tested in all other domains).

[0129]

[0130] Performance under model homogeneity: FBCL is compared with other methods under model homogeneity. The shared model is set to ResNet-18, and a parameter averaging operation is added between collaborative and local updates. Table 4 shows the inter-domain and intra-domain performance on Office-Caltech and CIFAR-100.

[0131] Table 4: Performance comparison with existing state-of-the-art methods on Office-Caltech and CIFAR-100 under model homogeneity.

[0132]

[0133] 8.2 Ablation Experiment:

[0134] Component Effectiveness Analysis: Ablation experiments were conducted to evaluate the effectiveness of the core components in the proposed FBCL, including Feature Instance-Level Alignment (FIA), Feature Basis Matrix Consistency (FBM), Correct Non-Target Distillation (CND), and Feature Relation Regularization (FRR). Table 5 shows the intra-domain and inter-domain performance with these components included. Specifically, the first row refers to models trained independently on private data. Both FIA and FBM contribute to better intra-domain and inter-domain performance. This is because FIA ​​aligns feature instances, while FBM ensures consistency of the feature basis matrix. CND enhances intra-domain accuracy but leads to a decrease in inter-domain performance in some scenarios. This is because the teacher model focuses on preserving the local private data distribution rather than providing guidance based on the global data distribution. Therefore, CND is effective for intra-domain generalization but may introduce bias when the model is applied across domains. In contrast, FRR performs exceptionally well in improving inter-domain performance. By constraining changes in the model representation, FRR effectively mitigates domain bias, thereby achieving better generalization to other domains. When CND and FRR are combined, FBCL achieves improvements in both intra-domain and inter-domain performance, exceeding the gains obtained solely through feature swapping.

[0135] Table 5: Ablation study of FBCL core components on Digits and CIFAR-100

[0136]

[0137] Communication overhead: In FBCL, communication is achieved by transmitting the feature N(μ,δ) of the common data, enabling interaction between heterogeneous models. The communication overhead is positively correlated with the feature length and the number of samples, but does not grow linearly with the model parameter size. According to the FBCL settings, the model parameter size to be transmitted in each round of communication is 2.56M, calculated as: number of common data samples × feature dimension length × 2. In comparison, the number of parameters in ResNet20 is approximately 9M. In addition, the robustness of FBCL ensures that the loss of a single sample feature has minimal impact on the overall model update, maintaining the integrity and effectiveness of the learning process. As the number of iterations increases, the features within each batch gradually become sparse and orthogonal to each other. At this point, the feature set can be represented by two compact vectors: (1) value vector: encoding the magnitude of non-zero elements; (2) index vector: marking the position of each non-zero element in the batch. This representation reduces the communication overhead to: number of common data samples × feature dimension length × 4 / global batch size. Table 6 shows the computational cost of other algorithms, where the number of public samples N=5,000, the feature dimension F=256, the feature map size M=16, the global batch size B=256, the local batch size B0=128, and the number of model parameters P (11 million for ResNet18).

[0138] Table 6: Comparison of Computational Costs

[0139]

[0140] Privacy Conservation Discussion: Unlike model aggregation-based methods, the framework of this invention only exchanges features extracted from public datasets, and no client can access the model parameters of other clients. Compared to prototype calibration-based methods, the public dataset used in this invention does not overlap with private data in terms of categories. Therefore, third parties can only infer a shared feature space but cannot know the specific location and distribution of private samples. Furthermore, unlike knowledge distillation methods that expose category-level logits, this invention forces a uniform distribution of features in the embedding space without mapping them to soft predictions, thus not revealing any information about the underlying category distribution.

[0141] Diverse model structures: As shown in Table 7, different model structures were used on the client side. The results show that on the Digits dataset, FBCL consistently demonstrates superior performance in both intra-domain and cross-domain scenarios.

[0142] Table 7: Same-domain and cross-domain performance of FBCL on the Digits dataset with different model architectures

[0143]

[0144] Performance under different feature dimensions: As shown in Table 8, the intra-domain performance of FBCL remains stable across different feature dimensions, with accuracy maintained between 92% and 93%. In contrast, the cross-domain performance of FBCL is more sensitive to changes in dimensionality. When constructing the feature basis matrix, insufficient dimensionality can lead to multiple inputs in the same batch mapping to the same basis vector, making it difficult to accurately calculate the loss; while excessive dimensionality can cause one input to map to multiple basis vectors, resulting in conflicting optimization directions and hindering model convergence. Cross-domain performance is optimal when the dimensionality is slightly greater than or equal to the batch size (i.e., 256). This configuration achieves a balanced representation, ensuring that each input is fully mapped to a unique basis vector, guaranteeing accurate loss calculation and promoting convergence.

[0145] Table 8: Intra-domain and cross-domain performance of FBCL on the Digits dataset with different feature dimensions.

[0146]

[0147] Impact of different amounts of public data: Table 9 shows the impact of public data on FBCL performance from the perspective of data volume. As the amount of public data increases, the accuracy of FBCL improves, and its generalization ability also strengthens. However, it is worth noting that when the data volume exceeds a certain scale, the performance improvement tends to saturate.

[0148] Table 9: FBCL performance on the Digits dataset with different amounts of public data in both intra-domain and cross-domain scenarios.

[0149]

[0150] Impact of Different Public Data Types: As shown in Table 10, FBCL performs well across all domains. While intra-domain accuracy remains stable across different datasets, the model's generalization ability is affected by the public data type. Accuracy is higher using the simpler Fashion-MNIST dataset than using the more complex Tiny-ImageNet dataset. Counterintuitively, results using noisy data are comparable to using CIFAR data, while performance is worst when using the same Digits dataset as the public data. This is because the FBCL algorithm is fundamentally different from other knowledge transfer methods: it transfers feature extraction capabilities rather than aligning class output logits. Specifically, FBCL maps public samples to different locations in the feature space, requiring the public data to have significant attribute features, rather than class correlation with private data.

[0151] Table 10: FBCL's performance on the Digits dataset for intra-domain and cross-domain use with different types of public data.

[0152]

[0153] In summary, this invention proposes a novel, generalizable, and heterogeneity-aware federated learning framework, FBCL, to address the problems of heterogeneity and catastrophic forgetting. FBCL first aligns instance feature distributions and constructs a fundamental matrix to extract client-invariant features from unlabeled public data. Next, it balances inter-domain and intra-domain performance by preserving accurate historical knowledge and feature relationship information. Extensive experiments on real-world image classification datasets demonstrate that FBCL significantly outperforms existing state-of-the-art methods in both discriminative and generalization capabilities across various scenarios. Furthermore, ablation studies validate the effectiveness of the core components designed in FBCL.

[0154] This invention addresses the core challenges of federated learning, including client-side data and model heterogeneity, and catastrophic forgetting caused by continuous learning. It proposes a generalizable, heterogeneity-aware federated feature and foundation matrix consistency learning framework (FBCL). First, a lightweight feature instance-level alignment mechanism based on Gaussian distribution modeling is designed, and a feature foundation matrix consistency constraint is introduced. A structured shared feature space is constructed using unrelated public unlabeled data to efficiently bridge representation differences between heterogeneous clients and enhance generalization ability. Next, to mitigate knowledge forgetting during local updates, a dual consistency constraint method is designed. Valuable global historical knowledge is preserved through correct non-target distillation, and feature relation regularization is combined to stabilize feature representations, thereby balancing intra-domain discriminability and inter-domain generalization when adapting to new local data. Extensive experiments using real image classification datasets (Digits, Office-Caltech) and the public dataset (CIFAR) validate the effectiveness of the proposed FBCL framework. Compared with various state-of-the-art federated learning methods, FBCL demonstrates significant superiority in intra-domain accuracy, cross-domain generalization performance, and mitigation of catastrophic forgetting.

[0155] The specific usage process or method of this invention is as follows:

[0156] (1) During the system initialization phase, the central server prepares an unlabeled public dataset related to the task and determines each participating client. Each client holds its own private labeled data and can independently select or initialize its local model architecture (feature extractor and classifier).

[0157] (2) During the collaborative update phase of each training cycle, the server distributes a batch of common data to each client. Each client processes the batch of data using its local feature extractor and projects the features to a unified high-dimensional space through a learnable mapping layer. Subsequently, each client calculates the feature instance-level alignment loss (based on the difference between the Gaussian distribution parameters of the feature vectors and the global average distribution) and the feature basis matrix consistency loss (encouraging the orthogonality of feature vectors from different samples), and uses these to jointly update its mapping layer and feature extractor parameters, aiming to align feature representations among all clients and improve the discriminativeness and diversity of their features.

[0158] (3) During the local update phase, each client uses its private labeled data for training. The training objective combines three parts: standard cross-entropy classification loss to ensure the model's discriminative ability on the local task; correct non-target distillation loss, which uses the previous or global model as a teacher to guide the current model to learn the soft label distribution of non-target classes to retain historical knowledge; and feature relation regularization loss, which constrains the consistency between the current local model and the global model (or the previous model) on the feature relation matrix to prevent feature representation drift. The client updates its entire local model by optimizing this joint loss.

[0159] (4) After completing the local update for a specified number of rounds, the client may choose to upload the model update (such as classifier parameters) to the server. The server executes aggregation algorithms such as federated averaging to generate a new global model.

[0160] (5) During operation, the system continuously collects information such as model updates, performance metrics, and the distribution of aligned public data features from each client. The improved FBCL framework dynamically adjusts the alignment and regularization strength based on the new global model and local training feedback, and can update the public data batch or client participation set in subsequent rounds, forming a closed-loop learning mechanism of feature alignment-local optimization-global aggregation-continuous evolution, so as to continuously improve the overall performance, robustness and generalization ability of the federated learning system in heterogeneous environments.

[0161] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A generalizable federated learning method based on the consistency of perceptual heterogeneity features and the fundamental matrix, characterized in that, A federated learning system for image classification tasks is provided. The system comprises a central server and multiple clients. The central server holds an unlabeled public image dataset, while each client holds its own labeled private image dataset and deploys a local image classification model that can be split into an image feature extractor and a classifier. Each client is equipped with a feature mapping layer that projects the image features output by the image feature extractor onto a unified high-dimensional feature space. Each iteration of model training in this method includes the following steps: S1. The central server collects public images from the unlabeled public image dataset and sends them to all clients participating in this round of training. Each client uses the public images in this batch to update the parameters of the feature extractor and the feature mapping layer by minimizing the joint loss function of feature instance-level alignment loss and feature basis matrix consistency loss, and obtains an aligned local image classification model. S2. Each client samples labeled images from its private image dataset and uses the local image classification model obtained from the previous training round or the aggregated global model issued by the central server as the teacher model. Using the labeled images of this batch, the parameters of the complete local image classification model are updated by minimizing the joint loss function, which includes cross-entropy loss, correct non-target distillation loss, and feature relation regularization loss, to obtain the local image classification model after this training round. S3. Each client determines the degree of difference between the local image classification model trained in the current round and the previous round. If the degree of difference is less than the preset threshold, the parameters of the local image classification model are frozen and no longer updated; otherwise, return to step S1.

2. The generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix as described in claim 1, characterized in that, After obtaining the local image classification model after this round of training, the central server aggregates the local image classification models of each client, generates a global model, and distributes it to the client.

3. The generalizable federated learning method based on the consistency of perceptual heterogeneity features and the fundamental matrix as described in claim 2, characterized in that, For clients that have already participated in training, the trained local image classification model is used directly for image classification tasks; for new clients that have not participated in training, the aggregated global model is deployed for image classification tasks.

4. The generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix as described in claim 1, characterized in that, The calculation of the feature instance-level alignment loss is as follows: Each client calculates the Gaussian distribution parameters of the feature vectors obtained from the feature extractor and feature mapping layer of the public images in this batch: in, It is a mean vector. It is the covariance matrix. It is the size of the feature map. It is a high-dimensional feature matrix of size m×c obtained after the feature extractor and feature mapping layer process the samples, where c is the number of channels. It is a feature at spatial location i; Each client calculates the loss due to the difference between its own feature distribution and the average feature distribution of all clients. : in, and Let be the mean vector and the average of the covariance matrix of all client feature distributions, respectively. It is the L2 norm, used to measure the similarity between two feature outputs.

5. The generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix as described in claim 1, characterized in that, The calculation of the consistency loss of the feature basis matrix is ​​performed by each client calculating the orthogonality loss between the feature vectors of different samples within the same batch. : in, This is the first in this batch The first sample and the first The degree of difference between individual samples It is the feature vector of the u-th sample in this batch. It is the feature vector of the v-th sample in this batch. This represents the absolute value operation. The L2 norm of a vector is represented by the magnitude of the vector. It refers to the batch size.

6. The generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix as described in claim 1, characterized in that, Correct non-target distillation loss The specific calculation is as follows: in, These are sample images from the labeled images in this batch. The true labels for the sample images. For category indexing, This indicates that all categories without actual labels are traversed; Teacher model prediction sample image Category The distribution of soft tags, The aligned local image classification model prediction sample image obtained in step S2 Category The distribution of soft labels; It is the KL divergence; This is the confidence parameter for historical knowledge. If the teacher model's prediction is correct, that is, the maximum predicted probability output corresponds to the label... ,but ;otherwise, .

7. The generalizable federated learning method based on perceptual heterogeneity features and consistency of the fundamental matrix as described in claim 1, characterized in that, Feature relation regularization loss The specific calculation is as follows: in, , , These are the local model's output features for this batch of samples. The Middle The, the The, the The output features of each sample , , The global model outputs features for this batch of samples. The Middle The, the The, the The output features of each sample The distance difference between two feature vectors. , These are all weight parameters for the distance scale, and N is the number of samples in this batch; For two eigenvectors with respect to the anchor vector or The difference in the included angle, It is the dot product of vectors. It is a 2-norm.

8. A generalizable federated learning system based on the consistency of perceptual heterogeneity features and fundamental matrices, characterized in that, The system is implemented using the federated learning method described in any one of claims 1-7, and includes a central server and multiple clients; wherein the central server has an unlabeled public image dataset for the image classification task, each client holds a labeled private image dataset and is equipped with a local image classification model that can be split into an image feature extractor and a classifier, and each client is equipped with a feature mapping layer that projects image features onto a unified high-dimensional feature space.

Images