Federated learning system and method based on prototype-guided cross-training mechanism

Abstract

The present invention discloses a federated learning system and method based on a prototype-guided cross-training mechanism, wherein a client clusters extracted image features and selects the image features in the largest cluster as a class-aware prototype; the client sends the model parameters corresponding to the class-aware prototype and the intermediate image classification model to a server; the client obtains the model parameters corresponding to the class-aware prototype and the intermediate image classification model uploaded by other clients from the server; the local intermediate image classification model is updated according to the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; the updated intermediate image classification model is trained using a local training set to obtain an optimized image classification model; during the training of the updated intermediate image classification model, the class-aware prototype is used to constrain the loss function; and the client then sends the model parameters corresponding to the optimized image classification model to the server.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a federated learning system and method based on a prototype-guided cross-training mechanism. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Federated learning is an emerging distributed learning paradigm that allows multiple parties to build a shared model that works for them. Existing federated learning methods allow model-level interaction between clients and servers without sharing local data. This makes federated learning effective in avoiding the risk of privacy leaks. However, recent research has revealed the vulnerability of federated learning models when exposed to non-IID scenarios. This is primarily due to the misalignment between local and global optimization objectives, and the difficulty in aggregating multiple biased learners into a high-performance global model.

[0004] Existing methods for alleviating data heterogeneity can be roughly divided into two categories: regulating local processes under the constraints of global knowledge and improving the versatility of local models. The former uses global outputs as knowledge to guide multiple clients to learn a unified goal. Traditional algorithms along this research line include feature-based, parameter-based, and prediction-based constraints. For example, FedDC and MOON align the outputs of local and global models in parameter and feature space, respectively. The latter approach focuses on knowledge expansion, which enables local models to fit different data distributions. For example, FedExg and FedMe apply a cross-training strategy, retraining local models on different clients to learn comprehensive knowledge. It can be seen that the cross-training strategy can automatically adjust the optimization objectives of local models. However, the inconsistency of data distribution between clients also leads to the problem of knowledge forgetting, resulting in limited performance improvement.

[0005] The inventors discovered that the existing technology has the following technical defects: the existing cross-training and federated learning cannot fully utilize the client's private data to train the model without leaking user privacy. The image classification accuracy of the trained model is not high and lacks universality. Summary of the Invention

[0006] In order to address the shortcomings of the existing technology, the present invention provides a federated learning system and method based on a prototype-guided cross-training mechanism. The present invention fully utilizes the client's private data to train the model without leaking user privacy. The trained model has high image classification accuracy and universal applicability.

[0007] In a first aspect, the present invention provides a federated learning system based on a prototype-guided cross-training mechanism;

[0008] A federated learning system based on a prototype-guided cross-training mechanism includes: at least two clients and one server, where all clients store the same initial image classification model;

[0009] The client is configured to: determine a local training set and an initial image classification model; use the local training set to train the initial image classification model to obtain an intermediate image classification model; use the intermediate image classification model to extract features from each class of images in the training set, cluster the extracted image features, and select the image features in the largest cluster in the clustering result as the class perception prototype; the client sends the model parameters corresponding to the class perception prototype and the intermediate image classification model to the server;

[0010] The server is configured to: receive model parameters corresponding to the class perception prototype and the intermediate image classification model uploaded by all clients;

[0011] The client is further configured to: obtain from the server the class-aware prototypes and model parameters corresponding to the intermediate image classification model uploaded by other clients; update the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; train the updated intermediate image classification model using a local training set to obtain an optimized image classification model; during the training of the updated intermediate image classification model, constrain the loss function using the class-aware prototype; and the client then sends the model parameters corresponding to the optimized image classification model to the server;

[0012] The server is further configured to: receive model parameters corresponding to the optimized image classification models uploaded by all clients; sum the model parameters corresponding to the optimized image classification models of all clients, then take the average value, use the average value corresponding to each model parameter as the final model parameter value, and then distribute the final model parameter value to all clients;

[0013] The client is further configured to: update the local optimized image classification model according to the final model parameter value to obtain a final image classification model; and classify the image to be classified based on the final image classification model to obtain an image classification result.

[0014] In a second aspect, the present invention provides a federated learning method based on a prototype-guided cross-training mechanism;

[0015] Federated learning methods based on prototype-guided cross-training mechanisms include:

[0016] The client determines a local training set and an initial image classification model; uses the local training set to train the initial image classification model to obtain an intermediate image classification model; uses the intermediate image classification model to extract features from each class of images in the training set, clusters the extracted image features, and selects the image features in the largest cluster in the clustering result as the class perception prototype; the client sends the model parameters corresponding to the class perception prototype and the intermediate image classification model to the server;

[0017] The server receives all class perception prototypes and model parameters corresponding to the intermediate image classification model uploaded by the client;

[0018] The client obtains the class-aware prototypes and model parameters corresponding to the intermediate image classification model uploaded by other clients from the server; updates the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; trains the updated intermediate image classification model using the local training set to obtain an optimized image classification model; during the training of the updated intermediate image classification model, the class-aware prototype is used to constrain the loss function; the client then sends the model parameters corresponding to the optimized image classification model to the server;

[0019] The server receives the model parameters corresponding to the optimized image classification model uploaded by all clients; sums the model parameters corresponding to the optimized image classification model of all clients, then takes the average value, takes the average value corresponding to each model parameter as the final model parameter value, and then distributes the final model parameter value to all clients;

[0020] The client updates the local optimized image classification model according to the final model parameter values ​​to obtain the final image classification model; based on the final image classification model, the client classifies the image to be classified to obtain the image classification result.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] A new prototype-guided cross-training mechanism, PGCT, is proposed to regularize the variation of class-level data representations between clients. It consists of two main modules, where the prototype-guided representation learning module uses client-aware prototypes of data patterns learned through clustering to guide the learning of consistent representations across feature spaces. This maintains similar decision boundaries between different clients. The prototype-based feature enhancement module uses prototypes as soft attention rules to further aggregate rich information to enhance the discriminability of historical features. Performance comparison, ablation study and case study experiments were conducted on four datasets. PGCT is able to learn discriminative features of different categories under the guidance of prototypes, and its performance is better than existing methods. The present invention fully utilizes the client's private data to train the model without leaking user privacy. The trained model has high image classification accuracy and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a framework description of Example 1;

[0025] FIG2( a ) shows that the cross-training process of FedExg in Example 1 may cause significant changes in the decision boundary of the local model;

[0026] FIG2( b ) shows that PGCT of Example 1 is able to regularize the learned data representation to be similar to the data representation learned from the previous client when performing cross processing;

[0027] Figure 3(a)-Figure 3(d) This is a schematic diagram of the cross-training method for correcting errors according to Example 1;

[0028] Figure 4(a)-Figure 4(f) Schematic diagram of the PGCT learning the discriminative feature distribution under the guidance of the prototype in Example 1. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Cross-training has become a common strategy for addressing data heterogeneity in federated learning. It aims to improve generalization by retraining local models on different clients in a privacy-preserving manner. The key idea is to ensure that local models are adequate for all client data. However, heterogeneity between data sources can cause local models to quickly forget knowledge learned through multiple rounds of cross-training.

[0032] Example 1

[0033] This embodiment provides a federated learning system based on a prototype-guided cross-training mechanism;

[0034] A federated learning system based on a prototype-guided cross-training mechanism includes: at least two clients and one server, where all clients store the same initial image classification model;

[0035] The client is configured to: determine a local training set and an initial image classification model; use the local training set to train the initial image classification model to obtain an intermediate image classification model; use the intermediate image classification model to extract features from each class of images in the training set, cluster the extracted image features, and select the image features in the largest cluster in the clustering result as the class perception prototype; the client sends the model parameters corresponding to the class perception prototype and the intermediate image classification model to the server;

[0036] The server is configured to: receive model parameters corresponding to the class perception prototype and the intermediate image classification model uploaded by all clients;

[0037] The client is further configured to: obtain from the server the class-aware prototypes and model parameters corresponding to the intermediate image classification model uploaded by other clients; update the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; train the updated intermediate image classification model using a local training set to obtain an optimized image classification model; during the training of the updated intermediate image classification model, constrain the loss function using the class-aware prototype; and the client then sends the model parameters corresponding to the optimized image classification model to the server;

[0038] The server is further configured to: receive model parameters corresponding to the optimized image classification models uploaded by all clients; sum the model parameters corresponding to the optimized image classification models of all clients, then take the average value, use the average value corresponding to each model parameter as the final model parameter value, and then distribute the final model parameter value to all clients;

[0039] The client is further configured to: update the local optimized image classification model according to the final model parameter value to obtain a final image classification model; and classify the image to be classified based on the final image classification model to obtain an image classification result.

[0040] Furthermore, the local training set includes images with known image classification labels, such as cat, dog, horse, cow, car, airplane, ship, etc.

[0041] Furthermore, the initial image classification model is, for example, a support vector machine classifier, a deep residual network ResNET or a convolutional neural network CNN.

[0042] Furthermore, the local training set is used to train the initial image classification model to obtain an intermediate image classification model. The training process includes:

[0043] The local training set is input into the initial image classification model to train the model. When the loss function value of the initial image classification model no longer decreases or the training reaches the set number of iterations, the training is stopped to obtain the trained initial image classification model; the trained initial image classification model is used as the intermediate image classification model.

[0044] Furthermore, the intermediate image classification model is used to extract features from each class of images in the training set, the extracted image features are clustered, and the image features in the largest cluster in the clustering result are selected as the class perception prototype, specifically including:

[0045] Using the intermediate image classification model, we extract features from each class of images in the training set and cluster all image features under the same class label to obtain several clusters.

[0046] The feature values ​​of all image features in the largest cluster are summed up, and then the average is taken as the class perception prototype.

[0047] Furthermore, the client is also configured to obtain from the server the model parameters corresponding to the class perception prototype and the intermediate image classification model uploaded by other clients, where the other clients here refer to clients other than the client.

[0048] Exemplarily, client i obtains from the server the model parameters corresponding to the class-aware prototype and the intermediate image classification model uploaded by client j; wherein i and j represent the client numbers respectively.

[0049] Furthermore, the updating of the local intermediate image classification model based on the model parameters uploaded by other clients and obtained from the server to obtain the updated intermediate image classification model means that all the model parameters of the local intermediate image classification model are replaced by the model parameters uploaded by other clients and obtained from the server.

[0050] Furthermore, in the process of training the updated intermediate image classification model, the class-perceptual prototype is used to constrain the loss function, wherein the total loss function used in the training process includes: the cross-entropy loss function of image feature classification, the contrast loss function based on the class-perceptual prototype, and the attention-weighted feature enhancement loss function based on the class-perceptual prototype.

[0051] Furthermore, the total loss function is expressed as:

[0052]

[0053] in, is the cross entropy loss for image feature classification, is a prototype-based contrast loss, is the enhancement loss, α and γ are the loss weights.

[0054] Furthermore, the cross entropy loss of the image feature classification The formula is:

[0055]

[0056] Among them, I(y n =) is a sign function, which takes 1 if the true category of sample n is equal to m, otherwise it takes 0; n represents the true label of sample n, p n,mIt represents the predicted probability of sample n in category m. N is the number of samples and M is the number of categories.

[0057] Furthermore, the contrast loss function based on class perception prototype The formula is:

[0058]

[0059] Among them, p + and p - Respectively represent and feature f x Prototypes with the same and different labels, sim() is the cosine similarity function, τ is the temperature parameter, and exp() is the exponential function.

[0060] Furthermore, the attention-weighted feature enhancement loss function based on class-aware prototypes is The formula is:

[0061]

[0062] in, is the updated intermediate image classification model, Represents enhanced features The classification loss, CE represents the cross entropy function, Represents enhanced sample features The corresponding label.

[0063]

[0064]

[0065]

[0066] Among them, ε is the scale parameter, and softmax() normalizes the scores of all enhanced prototypes. Represents enhanced features The classification loss function is represents the enhanced sample features, f represents the sample features, w n represents the attention weight, represents the enhanced prototype, f T Indicates the transposition of feature f, represents the enhanced prototype, p represents the prototype, Represents a Gaussian distribution.

[0067] Furthermore, the model parameters corresponding to the optimized image classification models of all clients are summed up and then averaged, the average value corresponding to each model parameter is used as the final model parameter value, and the final model parameter value is distributed to all clients, which means summing and averaging the same model parameters.

[0068] Furthermore, the updating of the local optimized image classification model according to the final model parameter values ​​to obtain the final image classification model; and classifying the image to be classified based on the final image classification model to obtain the image classification result, refers to:

[0069] Replace all model parameter values ​​of the local optimized image classification model with the final model parameter values.

[0070] like Figure 1 As shown, compared with the traditional cross-training method, the PGCT (prototype-guided cross-training mechanism) proposed in the present invention realizes prototype knowledge extraction to learn consistent representations across clients and maintain similar decision boundaries. Specifically, PGCT has two main modules: a prototype-guided representation learning (PGRL) module and a prototype-based feature enhancement (PFA) module. Among them, the contrast loss function based on class-aware prototypes is implemented by the prototype-guided representation learning (PGRL) module, and the attention-weighted feature enhancement loss function based on class-aware prototypes is implemented by the prototype-based feature enhancement (PFA) module.

[0071] Considering that private data cannot be shared, PGCT uses class-aware prototypes instead of class-level representations. The PGRL module uses these prototypes to guide the learning of consistent representations to maintain the discriminability of corresponding features. The PFA module focuses on stabilizing the decision stage. It uses representative prototypes as soft-attention regularizers to refine and enhance image features, and utilizes feature-level linear combinations to fuse information from the client's intra-class representations.

[0072] Performance comparisons, ablation studies of key PGCT components, and case studies of the effectiveness of representation learning were conducted on four common datasets. The experimental results verify that prototypes can serve as effective knowledge to guide representation learning, and that prototype-guided cross-training can expand the learnable knowledge of local models, thereby reducing classification bias.

[0073] In summary, this paper has two main contributions: 1. It proposes a model-agnostic cross-training mechanism, PGCT, to mitigate the problem of knowledge forgetting. To our knowledge, this is the first method to use data prototypes to guide local models in learning consistent representations across clients in federated learning. Research has found that knowledge forgetting primarily stems from inconsistencies in representation distribution and a loss of discernibility for historical features. PGCT also demonstrates its effectiveness in addressing these issues.

[0074] It should be understood that in order to address the data heterogeneity problem in federated learning, there are different training strategies: 1) local training + global aggregation and 2) local training + random exchange + cross-training + global aggregation. The former method usually aligns local and global optimization objectives. For example, FedProx and FedUFO aim to align the outputs of local and global models in parameter and feature space, respectively. The latter method utilizes a cross-training mechanism to retrain local models across different clients. This enables local models to be trained on more data to learn comprehensive knowledge, such as FedExg and FedMe. Cross-training is orthogonal to the previous methods, and it can be combined with these techniques in the local training stage. Therefore, cross-training is an effective strategy to improve the generalization ability of local models.

[0075] It should be understood that knowledge extraction has also been widely used to deal with data heterogeneity in federated learning. Existing methods usually rely on proxy datasets, and they aggregate local predictions of proxy datasets rather than model parameters or gradients. However, it is observed that the correlation between proxy data and local data determines the effectiveness of the decision. Inspired by prototype learning, many studies have shown that global prototypes can serve as effective knowledge to guide the update of local models. The prototype is derived from the average features of all classes, which is easy to implement and does not involve privacy leakage, but the averaging may lose some representative information.

[0076] like Figure 1 As shown in the figure, the prototype-guided cross-training mechanism (PGCT) in federated learning has three main stages, including local training, cross-training, and global aggregation. In the first stage, PGCT can use any federated learning algorithm to optimize the model and store a class prototype for each class. Then, PGCT randomly shuffles the local models and prototypes and broadcasts them anonymously on the server side, expanding the trainable dataset of the local model without leaking privacy. In the second stage, each client obtains the model and prototype learned in the first stage from another client for retraining. In the second stage, the algorithm includes two key modules: a prototype-guided representation learning module and a prototype-based feature enhancement module. PGCT obtains all local models from the second stage and aggregates them in the third stage to generate a global model.

[0077] The second stage of prototype-guided representation learning: The prototype-guided representation learning (PGRL) module retrains the model to adapt to different data distributions under the guidance of class-aware prototypes. It performs a generalized version of contrastive learning to align the distribution of prototypes and image features. This enables the local model to learn invariant representations across clients.

[0078] To achieve prototype extraction across clients, the present invention clusters each class to generate a class-aware prototype. It can learn the visual patterns of the class and aggregate similar features of the same class into a cluster. For example, to calculate the prototype on client i, the process can be formulated as:

[0079]

[0080] in represents the kth cluster of class m, Represents the data of class m in client i. is a feature extractor. And the prototype of class m in client j Learning is done through weighting, which is defined as:

[0081]

[0082] like Figure 1 As shown in Figure 2, after random swapping, the PGRL module in client j obtains the local model Ei and class-aware prototype pi from client i. To prevent overlap in representations between different classes in the latent space, PGCT learns unified features by maximizing the consistency between samples and corresponding prototypes. Inspired by contrastive learning in representation learning, this paper defines a prototype-based contrastive loss similar to the NT-Xent loss:

[0083]

[0084] Among them, p + and p - Respectively represent and feature f x Prototypes with the same and different labels. sim() is the cosine similarity function, and τ is the temperature parameter.

[0085] like Figure 1 As shown, PGCT can optimize local models using any algorithm (such as FedAvg and MOON) in the first stage and generate a class-aware prototype for each class. In the second stage, after random swapping, PGCT learns a unified representation for different clients to maintain similar decision boundaries and generates enhanced features through prototype-based attention weighting to enhance classification. Finally, PGCT aggregates all local models in the third stage.

[0086] Prototype-based Attention-Weighted Feature Enhancement: The prototype-based feature enhancement (PFA) module aims to utilize historical features to improve the generalization ability of local models. A practical idea is to reuse class-aware prototypes to refine and enhance image features. Specifically, it uses Gaussian noise to perform prototype enhancement and treats the enhanced prototypes as a soft attention regularizer to generate attention weights, which are defined as:

[0087]

[0088] Among them, ε is the scaling parameter, n is a positive integer greater than or equal to 1. In order to obtain a smoother decision boundary at the feature level, the PFA module implements feature enhancement through attention weighting to help classification, and the attention weight can be defined by the dot product similarity:

[0089]

[0090] Among them, softmax() normalizes the scores of all enhanced prototypes. And the enhanced image features fuse the information of the corresponding prototype and the original image features, which can be expressed as

[0091]

[0092] The present invention optimizes the model by increasing the loss.

[0093]

[0094] in is a classifier, Represents enhanced features .

[0095] PGCT focuses on learning consistent features across clients under the guidance of prototypes and using enhanced features to assist classification. Therefore, the comprehensive goal of PGCT in cross-training is to minimize

[0096]

[0097] in is the cross entropy loss for image feature classification, is a prototype-based contrast loss, is the enhancement loss, α and β are the loss weights.

[0098] Table 1. Algorithm performance comparison.

[0099] For all methods, we ran two experiments and reported the mean and standard deviation.

[0100]

[0101] Table 2. Statistics of the datasets used in the experiments

[0102]

[0103] This paper uses three benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and the medical image dataset PathMNIST, which are commonly used for federated learning in experiments. Their statistics are listed in Table 2.

[0104] For a fair comparison, we used the same network architecture for all methods. The network consists of three modules: an image encoder, a prediction head, and a classifier. For all datasets, we used two fully connected layers as the encoder for the MNIST dataset. We used a two-layer multilayer perceptron (MLP) as the prediction head, and a one-layer fully connected network as the classifier.

[0105] For CIFAR 10 and PathMNIST, we use a CNN network with two 5×5 convolutional layers followed by a 2×2 max pooling layer and two fully connected layers with ReLU activation.

[0106] For CIFAR 100, the encoder adopts ResNet-18 without the last fully connected network.

[0107] Hyperparameter settings: For all methods, the local training epochs in the global round are E = 10, the number of clients N = 10, the sample fraction C = 1.0, the local optimizer is the SGD algorithm, and the communication round T = 100. For local training, we set the weight decay to 1e-05, the batch size to 64, the initial learning rate to 0.01, the Dirichlet parameters β = 0.3 and β = 0.5, the temperature parameter τ = 0.5, α and γ are selected from {0.01, 0.05, 0.1, 0.5, 1.0}, and the scale parameter ε is selected from {0.1, 0.01}.

[0108] The present invention compares PGCT with existing methods in three categories:

[0109] 1) Local training without joint learning (SOLO);

[0110] 2) Federated learning methods without cross-training, including FedAvg, FedProx, SCAFFOLD, FedDyn, MOON, and FedDC;

[0111] 3) Federated learning methods with cross-training, including FedExg and FedMe (a simplified version of the mutual learning idea). The following results can be obtained from Table 1.

[0112] PGCTFedAvg ,PGCT FedProx ,PGCT MOON Significant improvements in classification are achieved compared to the original baseline, which demonstrates the algorithm-independent nature of the PGCT algorithm.

[0113] The PGCT algorithm generally has better performance than other algorithms. This is reasonable because PGCT can expand the training set of local models and alleviate the problem of knowledge forgetting.

[0114] Federated learning methods with cross-training generally achieve better performance than the corresponding baseline (FedAvg). This demonstrates that cross-training algorithms can be combined with other algorithms to achieve performance improvements.

[0115] For different distribution parameters β, the performance of all algorithms improves as β increases. This is mainly because a small β will lead to a highly skewed local dataset. This proves that it is important to balance the categories of local data. The effectiveness of different PGCT procedures is further investigated. The results are summarized in Table 3.

[0116] Table 3. PGCT ablation studies based on MNIST and CIFAR10 datasets.

[0117]

[0118] Using model exchange (Exg) alone may not lead to significant improvements due to the knowledge forgetting problem. Even when combined with the prototype-based feature enhancement module (PFA), the improvement is still limited.

[0119] Assisted by prototype-guided representation learning (PGRL), model exchange (Exg) outperforms the baseline model on both datasets by 1.5% and 2.5% respectively, verifying the effectiveness of representation learning.

[0120] The PGCT algorithm, which combines Exg, PGRL, and PFA, performs best, indicating that consistent representation learning and strengthening classifiers can effectively reduce knowledge forgetting. Figure 2(a) shows that the cross-training process of FedExg in Example 1 may cause significant changes in the decision boundary of the local model; Figure 2(b) shows that PGCT in Example 1 can regularize the learned data representation to be similar to the data representation learned from the previous client when performing cross-processing; Figure 3(a)-Figure 3(d) This is a schematic diagram of the cross-training method for correcting errors according to Example 1;

[0121] Figure 4(a)-Figure 4(f) Schematic diagram of the PGCT learning the discriminative feature distribution under the guidance of the prototype in Example 1.

[0122] Error Analysis of PGCT: We further analyzed the working mechanism of PGCT from the perspective of feature attention and model output. We used GradCAM to generate heatmaps. As shown in Figure 3(a), the first-stage model was unable to focus on small targets. Both FedExg and PGCT were able to learn new knowledge during the cross-training phase, compensating for this deficiency and making correct predictions. When the first-stage model was able to focus on the classification target, PGCT retained this ability and made correct predictions, while FedExg failed to classify due to knowledge forgetting, as shown in Figure 3(b). Figure 3(c) illustrates a case where the first-stage model learned poor attention and provided unreliable guidance to PGCT, while FedExg was able to focus on learning new knowledge and focusing on the "airplane" region, making a correct decision. Figure 3(d) shows a case where both methods made incorrect predictions. However, PGCT better focused on the "airplane" region and reduced the difference between the predictions for the "airplane" and the top-1 prediction. These observations validate the effectiveness of PGCT for cross-joint classification.

[0123] Quality Analysis of Representation Learning: The quality of representation learning is further studied. As shown in Figure 4, the present invention uses TSNE technology to visualize the representation in the feature space of the CIFAR10 test dataset. The present invention randomly selects two local models of customers. Obviously, the local model trained with the FedAvg method learns a poor representation distribution and there is overlap of multiple class features. This is because the local model faces an unbalanced data distribution, resulting in poor generalization ability of the local model to global data. Compared with FedAvg, FedExg and PGCT use cross-training to expand the trainable dataset of the local model, enabling it to learn comprehensive knowledge. However, due to knowledge forgetting, FedExg may achieve limited improvement. Intuitively, PGCT learns more discriminative representation distributions. Experiments show that the prototype-guided representation learning module helps PGCT improve the generalization ability of the local model.

[0124] This paper proposes a new cross-training mechanism, PGCT, to address the problem of knowledge forgetting. PGCT performs prototype-guided representation learning to learn consistent representations across clients. PGCT also employs a prototype-based feature enhancement method to enhance classification performance. Experimental results demonstrate that PGCT effectively learns invariant representations for the same class and discriminative representations for different classes. This improves the generalization capability of local models.

[0125] Example 2

[0126] This embodiment provides a federated learning method based on a prototype-guided cross-training mechanism;

[0127] The federated learning method based on the prototype-guided cross-training mechanism includes:

[0128] The client determines a local training set and an initial image classification model; uses the local training set to train the initial image classification model to obtain an intermediate image classification model; uses the intermediate image classification model to extract features from each class of images in the training set, clusters the extracted image features, and selects the image features in the largest cluster in the clustering result as the class perception prototype; the client sends the model parameters corresponding to the class perception prototype and the intermediate image classification model to the server;

[0129] The server receives all class perception prototypes and model parameters corresponding to the intermediate image classification model uploaded by the client;

[0130] The client obtains the class-aware prototypes and model parameters corresponding to the intermediate image classification model uploaded by other clients from the server; updates the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; trains the updated intermediate image classification model using the local training set to obtain an optimized image classification model; during the training of the updated intermediate image classification model, the class-aware prototype is used to constrain the loss function; the client then sends the model parameters corresponding to the optimized image classification model to the server;

[0131] The server receives the model parameters corresponding to the optimized image classification model uploaded by all clients; sums the model parameters corresponding to the optimized image classification model of all clients, then takes the average value, takes the average value corresponding to each model parameter as the final model parameter value, and then distributes the final model parameter value to all clients;

[0132] The client updates the local optimized image classification model according to the final model parameter values ​​to obtain the final image classification model; based on the final image classification model, the client classifies the image to be classified to obtain the image classification result.

[0133] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A federated learning system based on a prototype-guided cross-training mechanism, characterized by: include: At least two clients and one server, all of which store the same initial image classification model; The client is configured to: determine a local training set and an initial image classification model; use the local training set to train the initial image classification model to obtain an intermediate image classification model; use the intermediate image classification model to extract features from each class of images in the training set, cluster the extracted image features, and select the image features in the largest cluster in the clustering result as the class perception prototype; the client sends the model parameters corresponding to the class perception prototype and the intermediate image classification model to the server; The client is further configured to: obtain class-aware prototypes and model parameters corresponding to the intermediate image classification model uploaded by other clients from the server; update the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; use the local training set to train the updated intermediate image classification model to obtain an optimized image classification model; during the training of the updated intermediate image classification model, use the class-aware prototype to constrain the loss function; the loss function is obtained based on the prototype-guided representation learning PGRL module and the prototype-based feature enhancement PFA module; wherein the contrast loss function based on the class-aware prototype is implemented by the prototype-guided representation learning PGRL module, and the PGRL module uses these prototypes to guide the learning of consistent representations to maintain the distinguishability of corresponding features; the attention-weighted feature enhancement loss function based on the class-aware prototype is implemented by the prototype-based feature enhancement PFA module, and the PFA module uses representative prototypes as a soft attention regularizer to refine and enhance image features and uses linear combinations of feature levels to fuse information from the client's intra-class representation; the client then sends the model parameters corresponding to the optimized image classification model to the server; The server is configured to: receive model parameters corresponding to the optimized image classification model uploaded by all clients; sum the model parameters corresponding to the optimized image classification model of all clients, then take the average value, use the average value corresponding to each model parameter as the final model parameter value, and then distribute the final model parameter value to all clients.

2. The federated learning system based on the prototype-guided cross-training mechanism as claimed in claim 1, characterized in that: The client is further configured to: update the local optimized image classification model according to the final model parameter value to obtain a final image classification model; and classify the image to be classified based on the final image classification model to obtain an image classification result.

3. The federated learning system based on the prototype-guided cross-training mechanism as claimed in claim 1, characterized in that: The local training set is used to train the initial image classification model to obtain an intermediate image classification model. The training process includes: The local training set is input into the initial image classification model to train the model. When the loss function value of the initial image classification model no longer decreases or the training reaches the set number of iterations, the training is stopped to obtain the trained initial image classification model; the trained initial image classification model is used as the intermediate image classification model.

4. The federated learning system based on the prototype-guided cross-training mechanism according to claim 1, characterized in that: The intermediate image classification model is used to extract features from each class of images in the training set, cluster the extracted image features, and select the image features in the largest cluster in the clustering result as the class perception prototype, specifically including: Using the intermediate image classification model, we extract features from each class of images in the training set and cluster all image features under the same class label to obtain several clusters. The feature values ​​of all image features in the largest cluster are summed up, and then the average is taken as the class perception prototype.

5. The federated learning system based on the prototype-guided cross-training mechanism as claimed in claim 1, characterized in that: The updating of the local intermediate image classification model based on the model parameters uploaded by other clients and obtained from the server to obtain the updated intermediate image classification model means that all the model parameters of the local intermediate image classification model are replaced by the model parameters uploaded by other clients and obtained from the server.

6. The federated learning system based on the prototype-guided cross-training mechanism as claimed in claim 1, characterized in that: In the process of training the updated intermediate image classification model, the class-perceptual prototype is used to constrain the loss function, wherein the total loss function used in the training process includes: a cross-entropy loss function for image feature classification, a contrast loss function based on the class-perceptual prototype, and an attention-weighted feature enhancement loss function based on the class-perceptual prototype.

7. The federated learning system based on the prototype-guided cross-training mechanism according to claim 6, characterized in that: The total loss function is expressed as follows: in, is the cross entropy loss for image feature classification, is a contrast loss function based on class-aware prototypes, is an attention-weighted feature enhancement loss function based on class-aware prototypes, and For weight loss.

8. The federated learning system based on the prototype-guided cross-training mechanism as claimed in claim 7, characterized in that: Cross entropy loss for classification of the image features , the formula is: in, Is a sign function, if the true category of sample n is equal to m, it takes 1, otherwise it takes 0; represents the true label of sample n, Represents the predicted probability of sample n in category m; N is the number of samples, M is the number of categories; The contrast loss function based on class-aware prototype , the formula is: in, and Representation and characteristics Prototypes with the same and different tags, is the cosine similarity function, is the temperature parameter, is an exponential function; The attention-weighted feature enhancement loss function based on class-aware prototypes is expressed as: in, is the updated intermediate image classification model, Represents enhanced features Classification loss; represents the cross entropy function, Represents enhanced sample features Corresponding labels; in, is the scaling parameter, Normalize the scores of all enhanced prototypes, Represents enhanced features The classification loss function is represents the enhanced sample features, represents the sample characteristics, represents the attention weight, represents an enhanced prototype, Represents the feature Transpose, represents an enhanced prototype, Represents the prototype, Represents a Gaussian distribution.

9. The federated learning system based on prototype-guided cross-training mechanism as claimed in claim 2, characterized in that: The method of summing up the model parameters corresponding to the optimized image classification models of all clients and then taking the average value, taking the average value corresponding to each model parameter as the final model parameter value, and then distributing the final model parameter value to all clients refers to summing up and taking the average value of the same model parameter; The local optimized image classification model is updated according to the final model parameter values ​​to obtain the final image classification model; Based on the final image classification model, the image to be classified is classified to obtain the image classification result, which means that all model parameter values ​​of the local optimized image classification model are replaced with the final model parameter values.

10. A federated learning method based on a prototype-guided cross-training mechanism, characterized by: include: The client determines the local training set and initial image classification model; Using a local training set to train the initial image classification model to obtain an intermediate image classification model; Using the intermediate image classification model, we extract features from each class of images in the training set, cluster the extracted image features, and select the image features in the largest cluster as the class-aware prototype. The client sends the class-aware prototype and the model parameters corresponding to the intermediate image classification model to the server. The server receives all class perception prototypes and model parameters corresponding to the intermediate image classification model uploaded by the client; The client obtains the model parameters corresponding to the class-aware prototypes and intermediate image classification models uploaded by other clients from the server; updates the local intermediate image classification model based on the model parameters uploaded by other clients obtained from the server to obtain an updated intermediate image classification model; uses the local training set to train the updated intermediate image classification model to obtain an optimized image classification model; during the training of the updated intermediate image classification model, the class-aware prototypes are used to constrain the loss function; the loss function is obtained based on the prototype-guided representation learning PGRL module and the prototype-based feature enhancement PFA module; among them, the contrast loss function based on the class-aware prototype is implemented by the prototype-guided representation learning PGRL module, which uses these prototypes to guide the learning of consistent representations to maintain the discriminability of corresponding features; the attention-weighted feature enhancement loss function based on the class-aware prototype is implemented by the prototype-based feature enhancement PFA module, which uses representative prototypes as a soft attention regularizer to refine and enhance image features and uses feature-level linear combinations to fuse the information of the intra-class representation from the client; the client then sends the model parameters corresponding to the optimized image classification model to the server; The server receives the model parameters corresponding to the optimized image classification model uploaded by all clients; sums the model parameters corresponding to the optimized image classification model of all clients, then takes the average value, takes the average value corresponding to each model parameter as the final model parameter value, and then distributes the final model parameter value to all clients; The client updates the local optimized image classification model according to the final model parameter values ​​to obtain the final image classification model; based on the final image classification model, the client classifies the image to be classified to obtain the image classification result.

Images