A federated semi-supervised learning image classification method based on semantics and instances

Abstract

The application relates to the technical field of machine learning, in particular to a federated semi-supervised learning image classification method based on semantics and instances. The method comprises the following steps: S1, a server sends global model parameters to each client; S2, each client trains a local model deployed locally based on semantic-instance relationship alignment and pseudo-supervised contrast learning, and uploads local model parameters to the server after the training is completed; S3, the server aggregates the local model parameters uploaded by the clients to generate a global model, and updates the global model parameters; S1-S3 are cyclically executed for T rounds. The application introduces semantic-instance relationship alignment, encourages the model to have consistent distribution for similarity prediction of unmarked data and marked data, and introduces pseudo-supervised contrast learning, uses unmarked data to assist the local model of the client to train, and improves the image classification accuracy of the global / local model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a federated semi-supervised learning image classification method based on semantics and instances. Background Technology

[0002] Federated Learning (FL), as an emerging distributed machine learning paradigm, aims to improve model performance while protecting user privacy by collaboratively training a shared global model through multiple clients without centralizing users' original data. Based on the overlap of data features and sample spaces among the participants, federated learning is typically categorized into horizontal federated learning, vertical federated learning, and federated transfer learning. Horizontal federated learning is suitable for scenarios where clients share the same feature space but their samples do not overlap. Its typical process includes: the server distributes the global model to the clients; the clients train the model based on their local data and upload the updated parameters; the server aggregates model parameters from multiple clients to generate a new global model; this process iterates through several communication rounds until the model converges.

[0003] Meanwhile, semi-supervised learning (SSL) aims to address the problem of scarce labeled data. In traditional supervised learning, model performance heavily relies on large amounts of high-quality labeled data, but in practical applications (such as healthcare and finance), obtaining accurate labels is often costly and dependent on domain-specific knowledge. To alleviate this bottleneck, semi-supervised learning improves model generalization ability by effectively utilizing large amounts of unlabeled data. Current mainstream methods are mainly based on two technical paradigms: pseudo-labeling and consistency regularization. The former uses the model's predictions on unlabeled data as artificial labels for retraining, while the latter encourages the model to output a consistent prediction distribution for different augmented views of the same input. Existing semi-supervised methods typically combine these two mechanisms and are optimized for specific tasks.

[0004] In recent years, researchers have combined federated learning with semi-supervised learning, proposing Federated Semi-Supervised Learning (FSSL) to address the problem of insufficient or even completely missing client-side labeled data in real-world scenarios. Based on the distribution of labeled data, FSSL can be divided into three typical settings:

[0005] (1) labels-at-client, meaning each client holds both labeled and unlabeled data simultaneously; (2) labels-at-server: the client only has unlabeled data, while the server maintains a separate labeled dataset; (3) labels-at-partial-client: Some clients contain only labeled data, while others contain only unlabeled data.

[0006] Most current FSSL methods employ the pseudo-labeling strategy from centralized semi-supervised learning, relying on the "manifold assumption"—that semantically similar data should have the same label—to propagate semantic information learned by the model from labeled data to unlabeled samples. For example, Chinese patent CN115526334B discloses a weighted aggregation federated distillation method and system, belonging to the field of federated learning technology. This invention distributes local generators to each client through a central server, and each client performs distributed training on its local generator. Finally, the data is returned to the central server for global aggregation to simulate the global data distribution, thus replacing the direct introduction of external data and improving the usability of the federated distillation method. In addition, this invention introduces a small discriminator in each client, which is trained to output the probability density of the distilled data in the local data. This probability density is then used for weighted averaging to obtain more accurate teacher soft labels, thereby improving distillation efficiency and the accuracy of the trained model. This improves the accuracy of federated learning training models in scenarios where the dataset is not independent and identically distributed.

[0007] However, the effectiveness of this strategy is highly dependent on the model's calibration capabilities. Especially in cases of heterogeneous data distribution between the client and server (non-IID), the quality of pseudo-labels deteriorates significantly, leading to performance limitations and resulting in low accuracy of the trained model when performing image classification. Summary of the Invention

[0008] The purpose of this invention is to overcome the problem of low accuracy in image classification in the prior art and to provide a federated semi-supervised learning image classification method based on semantics and instances.

[0009] In a first aspect, the present invention provides a federated semi-supervised learning image classification method based on semantics and instances, comprising the following steps: S1. At the start of communication round t, the server sends the global model parameters to several clients participating in this round of federated training; a memory queue is set up on the server to store the low-dimensional feature distribution information of each instance of the labeled dataset; S2. Training the local models deployed on each of the clients using the global model parameters, including: S21. For the dataset owned by the unlabeled client, calculate the semantic-instance fusion probability distribution for each unlabeled data point; S22. Construct a pseudo-label dataset using the semantic-instance fusion probability distribution; S23. The local model is optimized and trained using pseudo-supervised contrastive learning combined with the pseudo-label dataset; S24. If the local model has completed training, obtain the parameters of the local model; S3. The server aggregates the parameters of each local model to generate a global model and updates the parameters of the global model. S1~S3 are executed in a loop for T rounds. After S1~S3 have been executed for T rounds, the client uses the local model deployed locally to perform image classification.

[0010] According to a preferred embodiment, in S1, the client initializes the local model deployed locally using the global model parameters; the global model parameters include at least the weight parameters and bias parameters of the global model.

[0011] According to a preferred embodiment, in S21, for the input unlabeled data, the category probability distribution and low-dimensional feature embedding distribution corresponding to its weakly enhanced view are calculated respectively. The cosine similarity between the low-dimensional feature embedding distribution and the center of each class instance is calculated using the cosine distance metric to obtain an instance probability distribution. The semantic-instance fusion probability distribution is generated by combining the semantic probability distribution and the instance probability distribution.

[0012] According to a preferred embodiment, the category instance center is: the low-dimensional feature embedding distribution corresponding to each category in the labeled dataset.

[0013] According to a preferred embodiment, the semantic-instance fusion probability distribution is obtained by weighted summation of the semantic probability distribution and the instance probability distribution.

[0014] According to a preferred embodiment, the optimization loss for training the local model includes: optimization loss for the pseudo-label dataset, semantic-instance alignment loss, and pseudo-supervised contrastive learning loss.

[0015] According to a preferred embodiment, given a global model, the global model is fine-tuned using a labeled dataset; the labeled data is represented using the global model; and the low-dimensional feature embedding distribution in the memory queue is updated with momentum.

[0016] In a second aspect, the present invention also provides an image recognition method that utilizes the semantic and instance-based federated semi-supervised learning image classification method provided by the present invention for image recognition.

[0017] In a third aspect, the present invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the semantic and instance-based federated semi-supervised learning image classification method provided by the present invention.

[0018] In a fourth aspect, the present invention also provides a computer program product. The computer program product includes a computer program. When executed by a processor, the computer program implements the semantic and instance-based federated semi-supervised learning image classification method provided by the present invention.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a federated semi-supervised learning image classification method based on semantics and instances. By utilizing the instance similarity between unlabeled client data and labeled server data, the generated semantic pseudo-labels are corrected, resulting in pseudo-labels that simultaneously contain semantic and instance information. These pseudo-labels are then used to guide the local model training process, mitigating performance losses caused by differences in data distribution. Furthermore, to fully utilize the instance information in the labeled server data, this invention introduces semantic-instance relationship alignment, encouraging the model to have a consistent distribution in its similarity predictions between weakly augmented views of unlabeled data and strongly augmented views of labeled data. Finally, based on contrastive learning, pseudo-supervised contrastive learning is introduced to further utilize unlabeled data to assist the local model training, improving the image classification accuracy of both the global and local models. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the framework of a preferred embodiment of the present invention: a federated semi-supervised learning image classification method based on semantics and instances. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0022] Furthermore, the use of terms such as "first," "second," and "third" in terminology is merely for distinguishing descriptions of identical or similar components and should not be interpreted as emphasizing or implying the relative importance of a particular component.

[0023] Furthermore, in the description of the embodiments of the present invention, "several", "more than", and "a number of" represent at least two. The number can be any number, such as 2, 3, 4, 5, 6, 7, 8, or 9, and can even exceed nine.

[0024] Example 1 This embodiment provides a federated semi-supervised learning image classification method based on semantics and instances, including the following steps: S1. At the start of communication round t, the server sends the global model parameters to several clients participating in this round of federated training; a memory queue is set up on the server to store the low-dimensional feature distribution information of each instance of the labeled dataset; S2. Training the local models deployed on each of the clients using the global model parameters, including: S21. For the dataset owned by the unlabeled client, calculate the semantic-instance fusion probability distribution for each unlabeled data point; S22. Construct a pseudo-label dataset using the semantic-instance fusion probability distribution; S23. The local model is optimized and trained using pseudo-supervised contrastive learning combined with the pseudo-label dataset; S24. If the local model has completed training, obtain the parameters of the local model; S3. The server aggregates the parameters of each local model to generate a global model, and updates the parameters of the global model. The process involves looping S1 through S3 for T rounds, where t ranges from 0 to T, and T is a positive integer. After S1 through S3 has completed T rounds, the client performs image classification using a locally deployed local model.

[0025] The labeled dataset includes several labeled data sets; the labeled data sets include images recorded in a standard format and the corresponding recognition result labels for the images.

[0026] The labeled dataset includes: a number of unlabeled data. Unlabeled data consists of images recorded by the client in a local format without any recognition results.

[0027] Pseudo-labeled dataset: Images recorded by the client in their local format are converted to images recorded in a standard format and then labeled with pseudo-labels. The pseudo-labels are the image classification results.

[0028] After S1~S3 has been executed for T rounds, the client will deploy the image input, which is recorded in local format but has not recorded the image classification results, into a local model. The local model will then output the image classification results.

[0029] The semantic and instance-based federated semi-supervised learning image classification method provided in this embodiment introduces semantic-instance relationship alignment, encourages the model to have a consistent distribution of similarity predictions for unlabeled and labeled data, and introduces pseudo-supervised contrastive learning to use unlabeled data to assist the local model on the client side in training, thereby improving the image classification accuracy of the global / local model.

[0030] Example 2 This embodiment is a further improvement on embodiment 1, and the repeated content will not be described again.

[0031] See Figure 1 The semantic and instance-based federated semi-supervised learning image classification method provided in this embodiment includes: S1, in the new round of communication t Initially, the server will set global model parameters. Sent to participants in this round of federal training n Each client uses this global model parameter to initialize its locally deployed local model. Global model parameter It can include: the weight parameters and bias parameters of the global model, which are generally stored in the form of a matrix.

[0032] S2, For unmarked clients Data sets possessed First, calculate each unlabeled data. Semantic-instance fusion probability distribution Then, construct the pseudo-label dataset using the following method. :

[0033] in, It is a confidence threshold. The optimization loss term for the pseudo-label dataset is:

[0034] By setting an optimization loss term for the pseudo-labeled dataset, the output of local models can be encouraged to approximate the pseudo-labels during training on unlabeled clients. This effectively gives the unlabeled data a manually assigned label, providing supervisory information to optimize the training process. Setting an optimization loss term for the pseudo-labeled dataset can improve the utilization of unlabeled data while helping to obtain a globally better classification model.

[0035] Each client uses pseudo-supervised contrastive learning to optimize and train a locally deployed model. After training, each client uploads the local model parameters. To the server side. Local model parameters. This also includes weight parameters and bias parameters. The local and global models have the same network architecture, but because the datasets used to train the local and global models are different, their parameters are also different.

[0036] S3, the server side, is mainly responsible for aggregating the local model parameters uploaded by the client. Generate a global model and generate new global model parameters, which are calculated as follows:

[0037] In addition, after obtaining the global model, the server will also utilize the labeled dataset. Fine-tuning the generated global model E One epoch, that is, the global model parameters obtained by combining the labeled dataset from the server. Retrain E times, during which the global model parameters... Slight changes will occur, allowing for fine-tuning. The optimization objective of this process is:

[0038] in, It is the input labeled image data of the i-th image. yes The corresponding tags.

[0039] Finally, a global model is used to represent the labeled data, and a memory queue is used. Momentum updates are performed using the low-dimensional feature embedding distribution.

[0040] The above process is repeated T Round communication, that is, S1~S3 executes round T in a loop.

[0041] Self-supervised learning (SSL) has gradually become an important means of improving model representation capabilities. Among them, methods such as contrastive learning construct positive and negative sample pairs to bring similar instances closer together and push away dissimilar instances in the feature space, thereby learning rich instance-level feature representations. Although such methods do not require real labels and can effectively mine structural information in unlabeled data, their optimization objective focuses on distinguishing individual instances rather than category semantics, which has an inconsistency with downstream classification tasks. They usually require additional fine-tuning to adapt to specific tasks.

[0042] Existing federated semi-supervised learning methods generally suffer from a key technical deficiency: the difficulty in effectively utilizing both semantic and instance information within a unified framework. On the one hand, pseudo-label-based methods rely excessively on the model's prediction confidence in unlabeled data, easily introducing noisy labels in heterogeneous data environments, and ignoring fine-grained structural relationships between samples. On the other hand, while self-supervised methods can capture discriminative features at the instance level, they lack explicit modeling of high-level semantic categories, making it difficult to directly serve classification objectives.

[0043] Therefore, how to collaboratively integrate semantic guidance and instance awareness in federated semi-supervised learning to construct a global model that has both strong generalization ability and maintains semantic consistency has become a core technical challenge that urgently needs to be solved.

[0044] This invention improves the pseudo-labeling method for unlabeled clients to optimize the local training process of federated semi-supervised learning, thereby enhancing the performance and robustness of the global model.

[0045] Preferably, in S2, each unlabeled data is calculated. Semantic-instance fusion probability distribution The method is as follows: Based on the pseudo-labeling method used in traditional federated semi-supervised learning algorithms, semantic-instance pseudo-labeling is performed on unlabeled data by combining the semantics and instance similarity of unlabeled data.

[0046] Most existing federated semi-supervised learning algorithms employ the traditional pseudo-labeling method, generating pseudo-labels in unlabeled clients, i.e.:

[0047]

[0048] It is a semantic probability distribution. It's a fake tag. The model parameters represent the number of communications (t). The table represents data augmentation operations, softmax is the normalization function, and argmax is used to represent the input (independent variable) that makes a function reach its maximum value.

[0049] When using formulas (5) and (6) to generate pseudo-labels, only the semantic information of the server-side tag data is considered, ignoring the relevant instance information.

[0050] This invention proposes a novel pseudo-tag method, namely semantic-instance pseudo-tag, from the perspectives of semantics and instance similarity.

[0051] To obtain instance feature information corresponding to the labeled data, a non-linear mapper needs to be added to the backbone network to map the original input data into a... L The low-dimensional feature distribution vector is denoted by . The backbone network is the basic feature extraction network, using WRN-28-2 (SVHN and CIFAR-10 datasets) and WRN-28-8 (CIFAR-100 dataset). For ease of distinction, the entire model (including the global model and local model framework) is represented. It is denoted as the following three parts: the backbone network is denoted as The classifier is denoted as The mapper is denoted as The instance feature information corresponding to the labeled data is transmitted to the client through global model parameters.

[0052] The global model and local models have the same architecture; the global model is deployed on the server, while local models are deployed on each client. The only difference between the global and local models is their parameters, particularly the weight and bias parameters.

[0053] Therefore, for server-side labeled datasets , Indicates labeled data, To represent the labels corresponding to the labeled data, an in-memory queue needs to be maintained to store the labeled dataset. The low-dimensional feature distribution information of each instance's data, this queue is denoted as... ,in, Enc is used to convert high-dimensional image data into low-dimensional feature embedding data. (This is) the backbone network. It is a projection head used to reduce the dimensionality of extracted features. This involves performing a data augmentation operation (rotation, cropping, etc.) on the image.

[0054] For the input unlabeled data We calculate the category probability distribution and low-dimensional feature embedding distribution corresponding to the weakly augmented view, respectively. Categories are distinguished by the specific labels corresponding to the data; the same label indicates the same category, and different labels indicate different categories.

[0055] Category probability distribution (semantic probability distribution):

[0056] Low-dimensional feature embedding distribution:

[0057] Unlike formula (6), semantic-instance pseudo-tags do not use Instead of directly calculating pseudo-labels, we further refine them by combining instance information.

[0058] Memory queues maintained on the server side In the above, the category instance center of each category i can be calculated as follows:

[0059] I(·) is an indicator function.

[0060] Then, use the cosine distance metric formula. calculate The cosine similarity to the center of each category instance yields an instance probability distribution. The specific calculation method is as follows:

[0061] in, Represents unlabeled data The cosine similarity between the center of instance and the center of instance corresponding to category i. K It is the total number of categories. It is the temperature coefficient. This represents unlabeled data. Similarity to instances of category i, because in representation we tend to use ordinal numbers such as 0, 1, 2... to represent category labels. Figure 1 Sim( ) represents the cosine measure.

[0062] Combining semantic probability distribution and instance probability distribution Semantic-instance pseudo-tags can generate pseudo-tags that contain both semantic and instance information for unlabeled data. Therefore, formula (5) can be rewritten as follows:

[0063] in, It is control and Hyperparameters for relative weights.

[0064] Throughout the federated training process, memory queues The feature distribution information is maintained and updated by the server. It does not directly use the representation results of the labeled data from the global model in a specific communication round, but instead uses the exponential moving average of historical representation results, i.e.:

[0065] It is a hyperparameter weight.

[0066] Memory queue Also included in global model parameters In, depending on the global model parameters Transmitted to each client.

[0067] This approach can make the distribution of low-dimensional feature embeddings in labeled data change more smoothly, thus enhancing the robustness of the generated pseudo-labels.

[0068] Similarly, for unlabeled data For a strongly enhanced view, its corresponding low-dimensional feature embedding distribution can be calculated using the following formula:

[0069] Then, combine formulas (9) and (10) to calculate the low-dimensional feature embedding distribution. Similarity distribution with the center of each category instance .

[0070] Based on unlabeled data Weakly enhanced view ( ) Calculated, Then based on unlabeled data Enhanced view ( The result is that two different data augmentation operations were applied to the same data, and the results were not the same.

[0071] However, during training, the low-dimensional features corresponding to the weakly augmented view were not directly embedded into the distribution. As The training is based on the objective of this approach, because it only considers the instance information corresponding to the labeled data, while ignoring the semantic information it provides. Therefore, it is also possible to train based on the prediction probability distribution of the weakly augmented view of unlabeled data using the local model. The probability distribution of its instances The data is then corrected to incorporate the rich semantic information provided by the labeled data. The specific calculation method is as follows:

[0072] Here, ⊙ is the Hadamard product operator.

[0073] Obtain the final target distribution Subsequently, the cross-entropy loss function can be used to optimize the learning process, namely, the semantic-instance alignment loss, which can be expressed as:

[0074] in, This represents the cross-entropy loss function. Semantic-instance alignment loss is a normalization technique introduced during training on unlabeled clients, constraining the model to determine the specific category of data not solely based on semantic information. Semantic-instance alignment loss encourages the model to simultaneously determine the category of unlabeled data based on both its semantic information and its instance similarity to labeled data, thus improving classification robustness.

[0075] The above methods enable the semantic and instance relationships to be mutually transformed and influenced, jointly promoting the optimization process of the local model and thus generating a more robust global model.

[0076] Traditional contrastive learning is conducted in a completely unsupervised manner, meaning that no label information is required during the learning process. It uses each data point itself as an anchor, treating each data point as a category, and the different augmented views corresponding to that category can be seen as latent data within that cluster. Based on this idea, this relationship can be used to provide supervisory signals to guide the learning process. Typically, this process uses the self-supervised contrastive loss InfoNCE or its variants as the objective function, with the following general form:

[0077] in, and These represent the low-dimensional feature embedding distributions of two different augmented views from the same instance data, i.e., positive sample pairs. Negative sample pairs typically originate from different augmented views of other instance data. They can be sampled from other data samples within the same batch, as in SimCLR, or obtained from a maintained in-memory data dictionary, similar to MoCo. While this approach allows for more granular mining of feature information contained in each data sample at the instance level, it may also lead to overfitting to local features and ignoring global semantic information. This means the model may learn minute features of individual samples in the training data, rather than common features of the entire category, thus affecting generalization ability. In supervised learning, all samples belonging to the same category have the same label, so the model can learn the similarity between categories. In contrastive learning, each sample is treated as an independent category, which may lead to excessively large distances between samples within a category, affecting the category aggregation ability. Therefore, in unlabeled clients, this invention proposes a pseudo-supervised contrastive learning method to assist training. It incorporates pseudo-label information to enhance the supervision signal during contrastive learning, encouraging the model to cluster unlabeled data and labeled data whose pseudo-labels and real labels belong to the same category into a cluster in the feature space and then use Avg( The averaging operation processes the embedding features belonging to the same cluster, providing more discriminative embedding features for subsequent classification tasks.

[0078] For unmarked clients dataset Each unlabeled data point is calculated using formulas (6) and (11). Corresponding pseudo tags Simultaneously, obtain its corresponding low-dimensional feature embedding distribution. By introducing monitoring signals Formula (16) can be rewritten in the following form:

[0079] Formula (17) is the loss function for pseudo-supervised contrastive learning in this embodiment. The pseudo-supervised contrastive learning loss setting can not only further utilize the instance information of the server-side labeled data, but also encourage the local model to learn more cohesive features during the training process of the unlabeled client (that is, tend to gather data with the same label into a cluster, which is beneficial for classification), thereby further generating a global model with better classification performance.

[0080] Meanwhile, combining formulas (15) and (17), the overall optimization objective for the unlabeled client is denoted as:

[0081] in, and These are two hyperparameters, which determine the loss term. and The relative weights.

[0082] The training objective of the local models on each client is the same. The only difference is the dataset used for training.

[0083] To evaluate the effectiveness of the semantic and instance-based federated semi-supervised learning image classification method proposed in this invention, comparative experiments were conducted on standard datasets in the fields of computer vision and machine learning (SVHN, CIFAR-10, and CIFAR-100) to compare the proposed semantic and instance-based federated semi-supervised learning (FedSMatch) image classification method with other federated semi-supervised learning algorithms (FedMatch, SemiFL, and CBAFed) in different scenarios. The evaluation metric for the experiments was the final classification accuracy of the global model. The best results were indicated in bold, and the second-best results were indicated in underline. The specific experimental results are shown in Table 1.

[0084] Table 1: Comparative experimental results of various federated semi-supervised learning image classification methods on different datasets with different label amounts.

[0085] Observing the experimental results, it can be found that the classification accuracy of the algorithm of this invention exceeds that of the baseline model on most datasets. For example, for the SVHN dataset, in the Non-IID, K=2 scenario using 250 and 1000 labels, the test accuracy of this invention reached 89.16% and 93.11%, respectively, while the suboptimal CBAFed results were 88.34% and 92.64%, respectively. In comparison, the final classification performance was improved by 0.82% and 0.47%, respectively. For the CIFAR-10 dataset, in the Non-IID, Dir(0.3) scenario using 250 and 4000 labels, the test classification accuracy of this invention was 73.94% and 89.16%, respectively, which is an improvement of 1.99% and 0.26% compared to the suboptimal results of 71.95% and 88.90%. For the most complex CIFAR-100 dataset, in the Non-IID, K=2 scenarios using 2500 and 10000 labels, the test performance of this invention was 42.75% and 62.54%, respectively, while CBAFed, which achieved suboptimal results, achieved 42.52% and 62.38%, respectively. In comparison, the final test performance was 0.23% and 0.16%, respectively. Furthermore, it was observed that the performance improvement of this invention was relatively larger when there was less label information. One possible reason for this is that when there is less labeled data, fusing the instance information provided by the labeled data can generate better pseudo-labels to guide the training process of the local model, ultimately generating a global model with relatively good generalization performance. However, when there is sufficient label information, although the instance information provided by the labeled data can improve the quality of pseudo-labels, it does not play a dominant role, thus the final global model performance improvement is relatively small. Regarding the poor performance of the model on the CIFAR-100 dataset, this paper suggests that the possible reason is that the dataset has a large number of categories and relatively few labels for each category. Although it can provide instance feature information, it is insufficient to enhance the quality of pseudo-labels, thus resulting in relatively poor generalization ability of the final model.

[0086] To verify the effectiveness of the semantic-instance pseudo-label, semantic-instance alignment loss, and pseudo-supervised contrastive loss in this invention, experiments were conducted on the CIFAR-10 dataset using 250 labels. The effectiveness of the semantic-instance pseudo-label, semantic-instance alignment loss, and pseudo-supervised contrastive loss was evaluated using accuracy. The experimental results are shown in Table 2.

[0087] Table 2: Ablation experimental results on the CIFAR-10 dataset

[0088] Table 2 shows that the model's classification performance was improved to varying degrees after introducing semantic-instance pseudo-labels, semantic-instance alignment loss, and pseudo-supervised contrastive loss. For example, in the Non-IID, Dir(0.1) scenario, the model's test classification accuracy was 62.13% without these three elements, but increased to 63.60% after introducing semantic-instance pseudo-labels, a 1.49% improvement. Further, the test classification accuracy increased to 63.93% after introducing semantic-instance alignment loss, a 0.31% improvement. Finally, the test classification accuracy increased to 64.14% after introducing pseudo-supervised contrastive loss, a 0.21% improvement. When all three elements (semantic-instance pseudo-labels, semantic-instance alignment loss, and pseudo-supervised contrastive loss) are introduced (i.e., the proposed FedSIMatch), the model's final classification accuracy is 64.14%, a 2.01% improvement compared to the model without these three elements. Similarly, in the IID scenario, the classification accuracy of the model was also improved to some extent after introducing the above three mechanisms.

[0089] Taking into account the above numerical experimental results, the inventors believe that the semantic and instance-based federated semi-supervised learning image classification method proposed in this invention can produce good application results in some extreme data areas where federated semi-supervised learning is applicable and some high-quality public datasets currently exist.

[0090] Federated learning, as an emerging fundamental machine learning method for solving the "data silo" problem while protecting data privacy, ensures the security of information exchange between computing nodes and protects the privacy of terminal and personal data, without the original data leaving the domain. It enables highly efficient joint machine learning among multiple participants or computing nodes, collaboratively building shared models without sharing private data. It also effectively addresses the problem of insufficient training data for single users, possessing powerful and secure distributed collaborative learning capabilities. Semi-supervised learning can enhance model learning capabilities using minimal labeled information and unlabeled data, even in scenarios with a large number of missing labeled samples. Deeply integrating semi-supervised learning methods into federated learning methods and frameworks can not only discover dynamic relationships in distributed medical big data but also solve problems such as data heterogeneity, scarce labeled samples, and isolated privacy. Especially when facing highly heterogeneous datasets and a large number of collaborating institutions, based on the performance discussion in the preceding numerical experiments, the semantic and instance-based federated semi-supervised learning image classification method proposed in this invention has significant performance advantages.

[0091] This invention modifies the generated semantic pseudo-labels by leveraging the instance similarity between unlabeled client data and labeled server data, producing pseudo-labels that simultaneously contain semantic and instance information. This guides the local model training process and mitigates performance losses caused by differences in data distribution. Furthermore, to fully utilize the instance information from the labeled server data, a semantic-instance relationship alignment loss is introduced, encouraging the model to have a consistent distribution in its similarity predictions for weakly augmented views of unlabeled data, strongly augmented views, and labeled data. Finally, based on contrastive learning, pseudo-supervised contrastive learning is introduced to further utilize unlabeled data to assist the local model training, improving the robustness and generalization of the global model.

[0092] Example 3 This embodiment provides an image recognition method.

[0093] This embodiment utilizes the semantic and instance-based federated semi-supervised learning image classification method described in Embodiment 1 or Embodiment 2 for image recognition, specifically including: S1. At the start of communication round t, the server sends the global model parameters to several clients participating in this round of federated training; a memory queue is set up on the server to store the low-dimensional feature distribution information of each instance of the labeled dataset; the labeled data includes images recorded in a standard format and the corresponding classification result labels of the images; S2. Train the local models deployed on each client using global model parameters, including: S21. For the dataset owned by the unlabeled client, calculate the semantic-instance fusion probability distribution for each unlabeled data point; the unlabeled dataset includes: several unlabeled data points; the unlabeled data points are images recorded by the client in a local format without recording classification results; In S21, for unlabeled data, the pseudo-label category probability distribution and low-dimensional feature embedding distribution corresponding to its weakly augmented view are calculated respectively. The cosine similarity between the low-dimensional feature embedding distribution and the center of each pseudo-label category instance is calculated using the cosine distance metric to obtain an instance probability distribution. A semantic-instance fusion probability distribution is generated by combining semantic probability distribution and instance probability distribution; the semantic probability distribution represents the pseudo-label category probability distribution corresponding to the text of unlabeled data; the instance probability distribution represents the pseudo-label category probability distribution corresponding to the low-dimensional feature distribution information of unlabeled data. S22. Construct a pseudo-label dataset using semantic-instance fusion probability distribution; Pseudo-labeled dataset: Images recorded by the client in their local format are converted into images recorded in a standard format and then labeled with pseudo-labels; the pseudo-labels are the image classification results. S23. A pseudo-supervised contrastive learning approach is used in conjunction with a pseudo-label dataset to optimize and train the local model. S24. Obtain the local model parameters after the local model has completed training; S3. The server aggregates the parameters of each local model to generate a global model and updates the parameters of the global model. S1~S3 are executed cyclically for T rounds.

[0094] After S1~S3 has been executed for T rounds, the client will deploy the image input, which is recorded in local format but has not recorded the image classification results, into the local model. The local model will output the image classification results, thus completing image recognition.

[0095] Preferably, the global model and the local model are image recognition models with the same architecture. Preferably, the image recognition model involved in this embodiment can be an existing image recognition model that requires training through machine learning. Specifically, the image recognition model can be built based on a convolutional neural network (CNN), a visual Transformer architecture, or other architectures.

[0096] Images can be stored in different formats, including bitmaps (such as JPEG, PNG, GIF, WebP, AVIF, TIFF, BMP, HEIC, etc.) and vector graphics (such as SVG, AI, EPS, PDF, etc.).

[0097] A single image can contain a variety of information, such as people, colors, objects, and text.

[0098] Labels and pseudo-labels represent the classification results of an image. Labels are the classification results confirmed by manual marking in the image, while pseudo-labels are the classification results predicted by the model based on the image information.

[0099] The number of categories for labels (pseudo-labels) can be set based on the specific object being identified and the application scenario. For example, when identifying numbers, the categories for labels (pseudo-labels) can be set to: 1, 2, 3, 4, 5, 6, 7, 8, 9, 0; when identifying objects, the categories for labels (pseudo-labels) can be set to: people, animals, cars, buildings, etc.

[0100] Example 4 This embodiment provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When executed by a processor, the computer program implements the semantic and instance-based federated semi-supervised learning image classification method involved in Embodiment 1, Embodiment 2, or Embodiment 3.

[0101] Example 5 This embodiment provides a computer program product. The computer program product includes a computer program. When executed by a processor, the computer program implements the semantic and instance-based federated semi-supervised learning image classification method involved in Embodiment 1, Embodiment 2, or Embodiment 3.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A federated semi-supervised learning image classification method based on semantics and instances, characterized in that, Includes the following steps: S1. At the start of communication round t, the server sends the global model parameters to several clients participating in this round of federated training; a memory queue is set up on the server to store the low-dimensional feature distribution information of each instance of the labeled dataset; S2. Training the local models deployed on each of the clients using the global model parameters, including: S21. For the dataset owned by the unlabeled client, calculate the semantic-instance fusion probability distribution for each unlabeled data point; S22. Construct a pseudo-label dataset using the semantic-instance fusion probability distribution; S23. The local model is optimized and trained using pseudo-supervised contrastive learning combined with the pseudo-label dataset; S24. If the local model has completed training, obtain the parameters of the local model; S3. The server aggregates the parameters of each local model to generate a global model, and updates the parameters of the global model. S1~S3 are executed cyclically for T rounds; After S1~S3 has been executed for T rounds, the client uses the local model deployed locally to perform image classification.

2. The federated semi-supervised learning image classification method based on semantics and instances according to claim 1, characterized in that, In S1, the client initializes the local model deployed locally using the global model parameters; The global model parameters include at least the weight parameters and bias parameters of the global model.

3. The federated semi-supervised learning image classification method based on semantics and instances according to claim 2, characterized in that, In S21, for the input unlabeled data, the category probability distribution and low-dimensional feature embedding distribution corresponding to the weak augmented view are calculated respectively. The cosine similarity between the low-dimensional feature embedding distribution and the center of each class instance is calculated using the cosine distance metric to obtain an instance probability distribution. The semantic-instance fusion probability distribution is generated by combining the semantic probability distribution and the instance probability distribution.

4. The federated semi-supervised learning image classification method based on semantics and instances according to claim 3, characterized in that, The category instance center is the low-dimensional feature embedding distribution corresponding to each category in the labeled dataset.

5. The federated semi-supervised learning image classification method based on semantics and instances according to claim 3, characterized in that, The semantic-instance fusion probability distribution is obtained by weighted summation of the semantic probability distribution and the instance probability distribution.

6. The federated semi-supervised learning image classification method based on semantics and instances according to claim 1, characterized in that, The optimization losses for training local models include: optimization loss for pseudo-label datasets, semantic-instance alignment loss, and pseudo-supervised contrastive learning loss.

7. The federated semi-supervised learning image classification method based on semantics and instances according to claim 1, characterized in that, Given the global model Fine-tune the global model using a labeled dataset; The labeled data is represented using the global model. Momentum update is performed on the low-dimensional feature embedding distribution in the memory queue.

8. An image recognition method, characterized in that, Image recognition is performed using the semantic and instance-based federated semi-supervised learning image classification method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the semantic and instance-based federated semi-supervised learning image classification method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the semantic and instance-based federated semi-supervised learning image classification method as described in any one of claims 1 to 7.

Images