A semi-supervised image classification method based on category prototypes

Abstract

The application relates to a kind of semi-supervised image classification methods based on category prototype, belong to deep learning and semi-supervised learning technical field.Firstly, a kind of feature description category prototype information containing sample information is used, then a weighted semi-supervised method is used to design a classifier, so as to achieve the purpose of image classification.The application uses the feature information of the labeled sample and the feature information of the reliable unlabeled sample when extracting category prototype information, so as to more accurately describe the category prototype information.In order to reduce the damage of distribution outside sample and private sample to the performance of the classifier, the application uses the information of the category prototype to weight different training samples, and applies it to image classification.Experiments show that the method proposed in the application can obtain satisfactory classification accuracy on common image data sets under the condition that there is difference between the label space of the labeled sample and the unlabeled sample.

Description

Technical Field

[0001] This invention relates to a semi-supervised image classification method based on category prototypes, belonging to the field of deep learning and semi-supervised learning technology. Background Technology

[0002] Image classification is a crucial and fundamental research area in deep learning and semi-supervised learning. Typically, training image classification models requires a large number of labeled images, i.e., labeled samples. However, labeling image categories requires significant human resources. Semi-supervised image methods greatly reduce the need for labeled samples by applying unlabeled samples, thus effectively lowering labor costs. Consequently, semi-supervised image classification methods have achieved considerable success in recent years.

[0003] Classical semi-supervised image classification methods can be divided into three types: 1) Pseudo-labeling: This method uses a classifier trained on labeled samples to predict the category of unlabeled samples and uses the category with the highest predicted probability as the pseudo-label for the unlabeled sample, thus transforming the training process into fully supervised learning. 2) Consistency regularization: This method enhances the robustness of the network by requiring different networks to make the same prediction for the same unlabeled sample. 3) Data augmentation: This method enhances the classifier's ability to learn from unlabeled samples by requiring the classifier to maintain consistent prediction results for different augmented versions of the same unlabeled sample.

[0004] However, traditional semi-supervised image classification methods all assume that the label space of the unlabeled sample set is exactly the same as the label space of the labeled sample set, but this is often not met in real-world applications. For example, to investigate the local ecological environment, it is often necessary to use automatic cameras to photograph target species in their wild environment. On the one hand, species outside the target range may be captured during the shooting process; on the other hand, some species within the target range may not be captured by the camera. In this case, the label spaces of the unlabeled sample set and the labeled sample set are different, with overlapping and independent parts. The goal of this invention is to improve the classification accuracy of the classifier in the label space of the labeled sample set under such circumstances. This invention refers to this task as a "semi-supervised image classification task under the intersection class mismatch situation". A class that appears only in the unlabeled sample set is called an out-of-distribution (OOD) class. A class that appears only in the labeled sample set is called a private class. A class that appears in both the unlabeled and labeled sample sets is called a shared class.

[0005] Current methods still fall short in handling semi-supervised image classification tasks with mismatched intersection classes. Recent semi-supervised image classification algorithms focus on reducing the negative impact of OOD (Out of Detail) classes on the classification task. Specifically, these methods select samples that are likely to belong to the OOD class (i.e., OOD samples) and then reduce their weights during training. Key methods in recent papers include:

[0006] Y. Chen et al. (2020) identified potential OOD data by averaging historical predictions from a self-distillation method. L. Guo et al. (2020) developed an instance-weighted strategy to mitigate the impact of OOD samples in unlabeled samples. Q. Yu et al. (2020) proposed distinguishing between ordered sequences of shared and OOD samples, from simple to difficult. K. Saito et al. (2021) proposed using Soft Open Set Consistency Regularization (SOCR) to detect OOD samples. J. Huang et al. (2021) proposed using a self-supervised learning strategy to achieve better performance in OOD category detection.

[0007] In terms of patents, Chinese invention patent application CN202210493913.6 classifies image samples based on nearest neighbor graphs and category indicator matrices. Chinese invention patent application CN202211120458.1 classifies image samples by constructing channel-dimensional attention networks and spatial-dimensional multi-scale self-attention networks. Chinese invention patent application CN201910659388.9 develops an image classification method based on an end-to-end dual-channel feature recalibration densely connected convolutional neural network. Chinese patent application CN202111074306.8 develops an image classification method based on deep neural networks. Chinese patent application CN201711245804.8 develops an image classification algorithm by automatically generating a large number of images. Chinese patent application CN202110519996.7 applies a concept cognition method to the field of image classification. Chinese patent application CN202110136790.6 develops an image classification task based on class center points, using a random batch class center update strategy. Chinese patent application CN201910564028.0 addresses image classification tasks under complex and variable influencing factors by generating new samples and using multiple different classifiers.

[0008] Since private classes also participate in the testing phase, it's crucial to prevent unlabeled samples from being misclassified into private classes during training. Furthermore, the reliance on linear prediction is highly susceptible to the overconfidence issue in cases of mismatched intersection classes, where an OOD sample is predicted with extremely high probability as a known class, thus impairing the classifier's performance in image classification. Therefore, this invention aims to address these issues to improve the classifier's performance in semi-supervised learning image classification tasks with mismatched intersection classes. Summary of the Invention

[0009] To address the aforementioned problems, the purpose of this invention is to provide a semi-supervised image classification method based on category prototypes. By leveraging the effective information contained in the generated category prototypes and sample features, and employing a semi-supervised learning strategy based on contrastive learning and weighted learning, the accuracy of semi-supervised image classification is improved in cases of mismatched intersection classes.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: First, a feature description of category prototype information containing sample information is used, and then a classifier is designed through a weighted semi-supervised method to achieve the purpose of image classification.

[0011] This invention utilizes both labeled and reliable unlabeled sample feature information when extracting category prototype information, thus enabling a more accurate description of category prototype information. To reduce the negative impact of out-of-distribution and private samples on classifier performance, this invention uses category prototype information to weight different training samples and applies it to image classification, thereby achieving better image classification results.

[0012] Specifically, the present invention provides a method for generating category prototypes, comprising the following steps:

[0013] The first step is to perform data augmentation on the unlabeled training images to obtain strongly augmented and weakly augmented versions of the unlabeled samples.

[0014] The second step involves analyzing the labeled samples and the augmented unlabeled samples described in the first step using a classifier to obtain the predicted distribution and 128-dimensional low-dimensional features.

[0015] The third step is to generate a feature queue and category prototype based on the low-dimensional features of the labeled samples described in the second step.

[0016] The fourth step is to calculate the clustering loss based on the low-dimensional features of the labeled samples described in the second step and the category prototypes described in the third step.

[0017] Fifth, based on the feature queue and category prototype described in the third step, calculate the intra-class similarity and similarity ratio;

[0018] Step 6: Based on the low-dimensional features of the unlabeled samples described in Step 2, the category prototypes described in Step 3, and the intra-class similarity described in Step 5, select low-dimensional features of the unlabeled samples to update the feature queue and the category prototypes.

[0019] Step 7: Calculate the contrastive loss based on the updated category prototype described in Step 6 and the low-dimensional features of the unlabeled samples described in Step 2.

[0020] Step 8: Based on the clustering loss described in step 4 and the contrast loss described in step 7, the parameters of the classifier are trained using stochastic gradient descent to obtain the trained classifier.

[0021] A weighted semi-supervised method is also provided, which includes the following steps:

[0022] Step 1: Calculate the private loss based on the predicted distribution of the unlabeled samples described in Step 2 and the similarity ratio described in Step 5.

[0023] Step 2: Calculate the cross-entropy loss based on the low-dimensional features and predicted distribution of the labeled samples described in Step 2.

[0024] Step 3: Calculate the semi-supervised loss based on the low-dimensional features and predicted distribution of the unlabeled samples described in Step 2, the intra-class similarity described in Step 5, and the category prototype described in Step 6.

[0025] Step 4: Based on the private loss described in Step 1, the cross-entropy loss described in Step 2, and the semi-supervised loss described in Step 3, the parameters of the classifier are trained using stochastic gradient descent to obtain the trained classifier.

[0026] Furthermore, the third step specifically involves: first, generating a feature queue based on the low-dimensional features of the labeled samples; the generation of the feature queue includes placing the low-dimensional features of the labeled samples into the corresponding stack according to their annotation information; if the stack is full, the oldest low-dimensional feature is replaced with a new low-dimensional feature.

[0027] Then, a category prototype is generated based on the feature queue; the expression of the category prototype is:

[0028]

[0029] in, Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0030] Furthermore, the fourth step specifically involves calculating the clustering loss, the expression for which is:

[0031]

[0032] in, Represents the clustering loss. Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first Low-dimensional features of a labeled sample, This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0033] Furthermore, the fifth step specifically involves: firstly, calculating the intra-class similarity based on the feature queue and the category prototype; the expression for the intra-class similarity is:

[0034]

[0035] in, Indicates the first Intra-class similarity of known classes Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates belonging to the first The first known class in the feature queue Low-dimensional features of each sample;

[0036] Then, based on the intra-class similarity, the similarity ratio is calculated, and the expression for the similarity ratio is:

[0037]

[0038] in, Indicates the first The similarity ratio of known classes. Indicates the first Intra-class similarity of known classes This represents the index of the known class with the highest intra-class similarity.

[0039] Further, the sixth step specifically involves: firstly, based on the low-dimensional features of the category prototype and the unlabeled samples, calculating the known class number corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample; the expression for calculating the known class number corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample is:

[0040]

[0041] in, Indicates the relationship with the first The known class indices corresponding to the class prototypes most similar to the low-dimensional features of the weakly augmented versions of unlabeled samples. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class;

[0042] Then, based on the known class number corresponding to the category prototype, the intra-class similarity, and the low-dimensional features of the unlabeled samples, it is determined whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue; the expression for determining whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue is:

[0043]

[0044] in, This represents the judgment value regarding whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class. Indicates the first The intra-class similarity of known classes; specifically, when At that time, Add to the In the feature queue of a known class; when At that time, then not Add to any feature queue;

[0045] Finally, the category prototype is calculated based on the updated feature queue.

[0046] Further, the seventh step specifically involves: calculating the contrast loss, the expression for which is:

[0047]

[0048] in, Indicates comparative loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first A strongly augmented version of low-dimensional features from unlabeled samples. This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0049] Further, step 2 specifically involves calculating the cross-entropy loss, the expression for which is:

[0050]

[0051] in, Represents cross-entropy loss, Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first The labeled sample was predicted as the first... The probability of a known class.

[0052] Further, step 1 specifically involves calculating the private loss, the expression for which is:

[0053]

[0054] in, Indicates private loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The index of the class with the highest probability in the predicted distribution of the weakly augmented version of the unlabeled samples. Indicates the first The similarity ratio of known classes. Indicates the first The weakly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0055] Further, step 3 specifically involves calculating the semi-supervised loss, the expression for which is:

[0056]

[0057] in, Indicates semi-supervised loss. Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes Indicates the first The strongly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0058] This invention also provides a semi-supervised image classification method based on category prototypes, comprising the following steps:

[0059] Step 1: Perform data augmentation on the unlabeled training images to obtain new unlabeled samples with different augmentations.

[0060] Step 2: Analyze the labeled samples and the unlabeled samples using a neural network to obtain the predicted distribution and low-dimensional features.

[0061] Step 3: Calculate the cross-entropy loss based on the predicted distribution of the labeled samples.

[0062] Step 4: Generate a feature queue and category prototype based on the low-dimensional features of the labeled samples.

[0063] Step 5: Calculate the clustering loss based on the low-dimensional features and category prototypes of the labeled samples.

[0064] Step 6: Calculate the intra-class similarity and similarity ratio based on the feature queue and category prototype.

[0065] Step 7: Based on the intra-class similarity, category prototype, and low-dimensional features of unlabeled samples, select low-dimensional features of unlabeled samples to update the feature queue and category prototype.

[0066] Step 8: Calculate the contrastive loss based on the low-dimensional features of the class prototype and the unlabeled samples.

[0067] Step 9: Calculate the private loss based on the similarity ratio and the predicted distribution of unlabeled samples.

[0068] Step 10: Calculate the semi-supervised loss based on the category prototype, intra-class similarity, low-dimensional features of unlabeled samples, and predicted distribution.

[0069] Step 11: Using multiple labeled samples and multiple unlabeled samples with different augmented versions as input, low-dimensional features and predicted distribution as output, and the cross-entropy loss, clustering loss, contrastive loss, private loss and semi-supervised loss as loss functions, the parameters of the neural network are trained using the stochastic gradient descent method to obtain a trained classifier.

[0070] Step 12: Input the test image into the trained neural network to obtain the predicted distribution and classification results.

[0071] The beneficial effects of this invention are:

[0072] This invention uses category prototype information to weight different training samples, thus avoiding the overconfidence problem caused by using linear prediction weighting. By analyzing private class samples, this invention prevents unlabeled samples from being misclassified into private classes. By utilizing sample feature information to identify OOD samples and private classes, and using a weighted semi-supervised learning method to control the classifier's training process in cases of mismatched intersection classes, this invention can improve the accuracy of image classification. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart of the semi-supervised image classification method based on category prototypes provided by the present invention;

[0075] Figure 2 A flowchart illustrating the practical application of the semi-supervised image classification method based on category prototypes provided by this invention;

[0076] Figure 3 A schematic diagram of a semi-supervised image classification system based on category prototypes provided by the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] like Figure 1 As shown, the present invention provides a semi-supervised image classification method based on category prototypes, comprising:

[0080] Step 1: Perform data augmentation on the unlabeled training images to obtain new unlabeled samples with different augmentations.

[0081] Step 2: Analyze the labeled samples and the unlabeled samples using a neural network to obtain the predicted distribution and low-dimensional features.

[0082] Step 3: Calculate the cross-entropy loss based on the predicted distribution of the labeled samples. The expression for the cross-entropy loss is:

[0083]

[0084] in, Represents cross-entropy loss, Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first The labeled sample was predicted as the first... The probability of a known class.

[0085] Step 4: Generate a feature queue and category prototypes based on the low-dimensional features of the labeled samples. Step 4 specifically includes:

[0086] A feature queue is generated based on the low-dimensional features of the labeled samples. This generation of the feature queue involves placing the low-dimensional features of the labeled samples into a corresponding stack according to their annotation information. If the stack is full, the oldest low-dimensional feature is replaced with a new one.

[0087] A category prototype is generated based on the feature queue. The expression for the category prototype is:

[0088]

[0089] in, Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0090] Step 5: Calculate the clustering loss based on the low-dimensional features and category prototypes of the labeled samples. The expression for the clustering loss is:

[0091]

[0092] in, Represents the clustering loss. Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first Low-dimensional features of a labeled sample, This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0093] Step 6: Calculate the intra-class similarity and similarity ratio based on the feature queue and category prototype. Step 6 specifically includes:

[0094] Based on the feature queue and category prototype, calculate the intra-class similarity. The expression for the intra-class similarity is:

[0095]

[0096] in, Indicates the first Intra-class similarity of known classes Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0097] Based on the feature queue and category prototype, a similarity ratio is calculated. The expression for the similarity ratio is:

[0098]

[0099] in, Indicates the first The similarity ratio of known classes. Indicates the first Intra-class similarity of known classes This represents the index of the known class with the highest intra-class similarity.

[0100] Step 7: Based on the intra-class similarity, category prototype, and low-dimensional features of unlabeled samples, select low-dimensional features of unlabeled samples to update the feature queue and category prototype. Step 7 specifically includes:

[0101] Based on the low-dimensional features of the category prototypes and unlabeled samples, calculate the known class index corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample. The expression for calculating the known class index corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample is as follows:

[0102]

[0103] in, Indicates the relationship with the first The known class indices corresponding to the class prototypes most similar to the low-dimensional features of the weakly augmented versions of unlabeled samples. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class.

[0104] Based on the known class number corresponding to the category prototype, the intra-class similarity, and the low-dimensional features of the unlabeled samples, it is determined whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. The expression for determining whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue is as follows:

[0105]

[0106] in, This represents the judgment value regarding whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes.

[0107] Specifically, when At that time, Add to the In the feature queue of a known class; when At that time, then not Add to any feature queue.

[0108] The category prototype is calculated based on the updated feature queue.

[0109] Step 8: Calculate the contrastive loss based on the low-dimensional features of the class prototype and the unlabeled samples. The expression for the contrastive loss is:

[0110]

[0111] in, Indicates comparative loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first A strongly augmented version of low-dimensional features from unlabeled samples. This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0112] Step 9: Calculate the private loss based on the similarity ratio and the predicted distribution of the unlabeled samples. The expression for the private loss is:

[0113]

[0114] in, Indicates private loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The index of the class with the highest probability in the predicted distribution of the weakly augmented version of the unlabeled samples. Indicates the first The similarity ratio of known classes. Indicates the first The weakly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0115] Step 10: Calculate the semi-supervised loss based on the category prototype, intra-class similarity, low-dimensional features of unlabeled samples, and predicted distribution. The expression for the semi-supervised loss is:

[0116]

[0117] in, Indicates semi-supervised loss. Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes Indicates the first The strongly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0118] Step 11: Using multiple labeled samples and multiple unlabeled samples with different augmented versions as input, low-dimensional features and predicted distribution as output, and the cross-entropy loss, clustering loss, contrastive loss, private loss and semi-supervised loss as loss functions, the parameters of the neural network are trained using the stochastic gradient descent method to obtain the trained neural network.

[0119] Step 12: Input the test image into the trained neural network to obtain the predicted distribution.

[0120] like Figure 2 As shown, this invention also provides a workflow for a semi-supervised image classification method based on category prototypes in practical applications. It includes the following steps:

[0121] The first step is data preprocessing. Data augmentation is performed on the unlabeled training images to obtain new unlabeled samples with different augmentations.

[0122] Specifically, for the images X1, X2, ..., X in the unlabeled training dataset n The pixel values ​​are standardized to ensure that the distribution of each pixel value follows a standard normal distribution. Then, data augmentation operations such as color transformation and random flipping are performed on the images to enrich the training set samples. Finally, the size of the augmented images is adjusted to the same size as before data augmentation, and these are used as the final training images.

[0123] The second step involves generating a feature queue and category prototypes. Based on the labeled and unlabeled samples, a neural network is used to analyze the data to obtain the predicted distribution and low-dimensional features. A feature queue and category prototypes are then generated based on the low-dimensional features of the labeled samples.

[0124] Specifically, generating a feature queue based on the low-dimensional features of the labeled samples means placing the low-dimensional features of the labeled samples into the corresponding stack according to their annotation information. If the stack is full, the oldest low-dimensional feature is replaced with a new low-dimensional feature.

[0125] Specifically, the expression for the category prototype is:

[0126]

[0127] in, Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0128] The third step is to calculate the intra-class similarity and similarity ratio. Based on the feature queue and category prototype, calculate the intra-class similarity and similarity ratio.

[0129] Specifically, the expression for the intra-class similarity is:

[0130]

[0131] in, Indicates the first Intra-class similarity of known classes Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0132] Specifically, the expression for the similarity ratio is:

[0133]

[0134] in, Indicates the first The similarity ratio of known classes. Indicates the first Intra-class similarity of known classes This represents the index of the known class with the highest intra-class similarity.

[0135] The fourth step is to update the feature queue and category prototypes. Based on the category prototypes and the low-dimensional features of the unlabeled samples, calculate the known class number corresponding to the category prototype whose low-dimensional features are most similar to those of each unlabeled sample. Based on the known class number corresponding to the category prototype, the intra-class similarity, and the low-dimensional features of the unlabeled samples, determine whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Based on the updated feature queue, calculate the category prototypes.

[0136] Specifically, the expression for calculating the known class index corresponding to the class prototype most similar to the low-dimensional features of each unlabeled sample is as follows:

[0137]

[0138] in, Indicates the relationship with the first The known class indices corresponding to the class prototypes most similar to the low-dimensional features of the weakly augmented versions of unlabeled samples. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class.

[0139] Specifically, the expression for determining whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue is:

[0140]

[0141] in, This represents the judgment value regarding whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes.

[0142] Specifically, when At that time, Add to the In the feature queue of a known class; when At that time, then not Add to any feature queue.

[0143] Step 5: Calculate the total model loss. Calculate the cross-entropy loss based on the predicted distribution of the labeled samples. Calculate the clustering loss based on the low-dimensional features and class prototypes of the labeled samples. Calculate the contrast loss based on the class prototypes and the low-dimensional features of the unlabeled samples. Calculate the private loss based on the similarity ratio and the predicted distribution of the unlabeled samples. Calculate the semi-supervised loss based on the class prototypes, intra-class similarity, the low-dimensional features of the unlabeled samples, and the predicted distribution. Finally, calculate the total loss based on the cross-entropy loss, clustering loss, contrast loss, private loss, and semi-supervised loss. .

[0144] Specifically, the expression for the cross-entropy loss is:

[0145]

[0146] in, Represents cross-entropy loss, Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first The labeled sample was predicted as the first... The probability of a known class.

[0147] Specifically, the expression for the clustering loss is:

[0148]

[0149] in, Represents the clustering loss. Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first Low-dimensional features of a labeled sample, This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0150] Specifically, the expression for the contrast loss is:

[0151]

[0152] in, Indicates comparative loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first A strongly augmented version of low-dimensional features from unlabeled samples. This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0153] Specifically, the expression for the private loss is:

[0154]

[0155] in, Indicates private loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The index of the class with the highest probability in the predicted distribution of the weakly augmented version of the unlabeled samples. Indicates the first The similarity ratio of known classes. Indicates the first The weakly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0156] Specifically, the expression for the semi-supervised loss is:

[0157]

[0158] in, Indicates semi-supervised loss. Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes Indicates the first The strongly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0159] Specifically, the expression for the total loss is:

[0160]

[0161] Step 6: Solve the model. Utilize the total loss... The network parameters are updated using the stochastic gradient descent algorithm, and then the process jumps to the first step to process the next batch of training images.

[0162] Step 7: During testing, the test image is processed by the network to obtain the predicted category.

[0163] Compared with existing inventions, this invention has the following advantages: 1) The use of category prototype technology during the training phase makes the model more capable of handling OOD sample detection. 2) The use of category prototypes reduces the negative impact of private categories on unlabeled sample learning, making the network's learning of unlabeled samples less disturbed. 3) The method does not require complex modifications to the model structure and is easily applicable to various existing models.

[0164] Table 1 shows the image classification accuracy of the method of this invention under class mismatch conditions on authoritative datasets in the field, MNIST, SVHN, and CIFAR-10. This method achieves good image classification performance on different datasets under different class mismatch conditions.

[0165] Table 1 shows the image classification accuracy of this method under the case of intersection class mismatch on the MNIST, SVHN, and CIFAR-10 datasets. Specifically, the label space of the labeled sample set and the OOD category are set as follows: and To demonstrate the impact of different numbers of private categories on classification performance, four different shared categories were set up: (1) (2) (3) (4) .

[0166] Table 1

[0167]

[0168] like Figure 3 As shown, the semi-supervised image classification system based on category prototypes provided by the present invention includes:

[0169] Data augmentation module 1 is used to augment unlabeled training images to obtain new unlabeled samples with different augmentations;

[0170] Analysis module 2 is used to analyze the labeled samples and the unlabeled samples using a neural network to obtain the predicted distribution and low-dimensional features;

[0171] The cross-entropy loss determination module 3 is used to calculate the cross-entropy loss based on the predicted distribution of the labeled samples;

[0172] Prototype building module 4 is used to generate feature queues and category prototypes based on the low-dimensional features of the labeled samples;

[0173] Clustering loss determination module 5 is used to calculate clustering loss based on the low-dimensional features and category prototypes of the labeled samples;

[0174] Similarity calculation module 6 is used to calculate intra-class similarity and similarity ratio based on the feature queue and category prototype;

[0175] The prototype optimization module 7 is used to select low-dimensional features of unlabeled samples based on the intra-class similarity, category prototype and low-dimensional features of unlabeled samples to update the feature queue and category prototype.

[0176] The contrast loss determination module 8 is used to calculate the contrast loss based on the low-dimensional features of the category prototype and the unlabeled sample;

[0177] Private loss determination module 9 is used to calculate private loss based on the similarity ratio and the predicted distribution of unlabeled samples;

[0178] The semi-supervised loss determination module 10 is used to calculate the semi-supervised loss based on the category prototype, intra-class similarity, low-dimensional features of unlabeled samples, and predicted distribution.

[0179] Training module 11 is used to train the parameters of the neural network by taking multiple labeled samples and multiple unlabeled samples with different augmented versions as inputs, low-dimensional features and predicted distribution as outputs, and cross-entropy loss, clustering loss, contrastive loss, private loss and semi-supervised loss as loss functions, and using stochastic gradient descent to obtain a trained neural network.

[0180] The test module 12 is used to input the test image into the trained neural network to obtain the predicted distribution and classification results.

[0181] In practical applications, the expression for the cross-entropy loss is:

[0182]

[0183] in, Represents cross-entropy loss, Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first The labeled sample was predicted as the first... The probability of a known class.

[0184] In practical applications, the step of generating feature queues and category prototypes based on the low-dimensional features of the labeled samples specifically includes:

[0185] A feature queue is generated based on the low-dimensional features of the labeled samples;

[0186] Generate category prototypes based on the feature queue.

[0187] In practical applications, generating a feature queue based on the low-dimensional features of the labeled samples means placing the low-dimensional features of the labeled samples into the corresponding stack according to their annotation information. If the stack is full, the oldest low-dimensional feature is replaced with a new low-dimensional feature.

[0188] In practical applications, the expression for the category prototype is:

[0189]

[0190] in, Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0191] In practical applications, the expression for the clustering loss is:

[0192]

[0193] in, Represents the clustering loss. Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first Low-dimensional features of a labeled sample, This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0194] In practical applications, the calculation of intra-class similarity and similarity ratio based on the feature queue and category prototype specifically includes:

[0195] Calculate intra-class similarity based on the feature queue and category prototype;

[0196] Calculate the similarity ratio based on the feature queue and the category prototype.

[0197] In practical applications, the expression for intra-class similarity is:

[0198]

[0199] in, Indicates the first Intra-class similarity of known classes Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

[0200] In practical applications, the expression for the similarity ratio is:

[0201]

[0202] in, Indicates the first The similarity ratio of known classes. Indicates the first Intra-class similarity of known classes This represents the index of the known class with the highest intra-class similarity.

[0203] In practical applications, the step of selecting low-dimensional features of specific unlabeled samples to update the feature queue and category prototypes specifically includes:

[0204] Based on the low-dimensional features of the category prototype and the unlabeled sample, calculate the known class number corresponding to the category prototype whose low-dimensional features are most similar to those of each unlabeled sample;

[0205] Based on the known class number, intra-class similarity, and low-dimensional features of the unlabeled samples corresponding to the category prototype, determine whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue.

[0206] The category prototype is calculated based on the updated feature queue.

[0207] In practical applications, the expression for calculating the known class index corresponding to the class prototype most similar to the low-dimensional features of each unlabeled sample is as follows:

[0208]

[0209] in, Indicates the relationship with the first The known class indices corresponding to the class prototypes most similar to the low-dimensional features of the weakly augmented versions of unlabeled samples. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class.

[0210] In practical applications, the expression for determining whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue is:

[0211]

[0212] in, This represents the judgment value regarding whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes.

[0213] Specifically, when At that time, Add to the In the feature queue of a known class; when At that time, then not Add to any feature queue.

[0214] In practical applications, the expression for the contrast loss is:

[0215]

[0216] in, Indicates comparative loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first A strongly augmented version of low-dimensional features from unlabeled samples. This represents the total number of known classes. Indicates the first The category prototype of a known class.

[0217] In practical applications, the expression for the private loss is:

[0218]

[0219] in, Indicates private loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The index of the class with the highest probability in the predicted distribution of the weakly augmented version of the unlabeled samples. Indicates the first The similarity ratio of known classes. Indicates the first The weakly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0220] In practical applications, the expression for the semi-supervised loss is:

[0221]

[0222] in, Indicates semi-supervised loss. Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes Indicates the first The strongly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

[0223] This invention discloses a method for augmenting each unlabeled training image during the training phase, resulting in new unlabeled samples with varying augmentations. For these input images, cross-entropy loss is calculated based on the predicted distribution of labeled samples. Simultaneously, low-dimensional features of labeled samples are used to generate a feature queue and category prototypes, and clustering loss, intra-class similarity, and similarity ratio are calculated. Low-dimensional features of unlabeled samples are selected to update the feature queue and category prototypes, and contrastive loss is calculated. Private loss is calculated using the similarity ratio and the predicted distribution of unlabeled samples. Semi-supervised loss is calculated using category prototypes, intra-class similarity, low-dimensional features of unlabeled samples, and the predicted distribution. All these losses mitigate the negative impact of OOD (Out of Detail) and private categories, ultimately improving the network's image classification ability. By using category prototypes to guide the network's semi-supervised learning of image classification during training, the model can make more accurate predictions about image categories.

[0224] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0225] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A semi-supervised training method for a classifier, characterized in that, The following steps are included: The first step is to perform data augmentation on the unlabeled training images to obtain strongly augmented and weakly augmented versions of the unlabeled samples. The second step involves analyzing the labeled samples and the unlabeled samples augmented in the first step using a classifier to obtain the predicted distribution and 128-dimensional low-dimensional features. The third step is to generate a feature queue and category prototype based on the low-dimensional features of the labeled samples described in the second step. The fourth step is to calculate the clustering loss based on the low-dimensional features of the labeled samples described in the second step and the category prototypes described in the third step. Fifth, based on the feature queue and category prototype described in the third step, calculate the intra-class similarity and similarity ratio; Step 6: Based on the low-dimensional features of the unlabeled samples described in Step 2, the category prototypes described in Step 3, and the intra-class similarity described in Step 5, select low-dimensional features of the unlabeled samples to update the feature queue and the category prototypes. Step 7: Calculate the contrastive loss based on the updated category prototype described in Step 6 and the low-dimensional features of the unlabeled samples described in Step 2. Step 8: Calculate the private loss based on the predicted distribution of the unlabeled samples described in Step 2 and the similarity ratio described in Step 5. Step 9: Calculate the cross-entropy loss based on the low-dimensional features and predicted distribution of the labeled samples described in Step 2. Step 10: Calculate the semi-supervised loss based on the low-dimensional features and predicted distribution of the unlabeled samples described in Step 2, the intra-class similarity described in Step 5, and the category prototype described in Step 6. In step eleven, the parameters of the classifier are trained using stochastic gradient descent based on the clustering loss from step four, the contrast loss from step seven, the private loss from step eight, the cross-entropy loss from step nine, and the semi-supervised loss from step ten, to obtain the trained classifier.

2. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The third step is as follows: First, a feature queue is generated based on the low-dimensional features of the labeled samples; the generation of the feature queue includes putting the low-dimensional features of the labeled samples into the corresponding stack according to their annotation information. If the stack is full, the oldest low-dimensional feature is replaced with a new low-dimensional feature. Then, a category prototype is generated based on the feature queue; the expression of the category prototype is: ； in, Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates the first The first known class in the feature queue Low-dimensional features.

3. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The fourth step specifically involves calculating the clustering loss, the expression for which is: ； in, Represents the clustering loss. Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first Low-dimensional features of a labeled sample, This represents the total number of known classes. Indicates the first The category prototype of a known class.

4. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The fifth step specifically involves: firstly, calculating the intra-class similarity based on the feature queue and the category prototype; the expression for the intra-class similarity is: ； in, Indicates the first Intra-class similarity of known classes Indicates the first The category prototype of a known class. Indicates the first The number of low-dimensional features in the feature queue of a known class Indicates belonging to the first The first known class in the feature queue Low-dimensional features of each sample; Then, based on the intra-class similarity, the similarity ratio is calculated, and the expression for the similarity ratio is: ； in, Indicates the first The similarity ratio of known classes. Indicates the first Intra-class similarity of known classes This represents the index of the known class with the highest intra-class similarity.

5. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The sixth step specifically involves: firstly, based on the low-dimensional features of the category prototype and the unlabeled samples, calculating the known class number corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample; the expression for calculating the known class number corresponding to the category prototype most similar to the low-dimensional features of each unlabeled sample is as follows: ； in, Indicates the relationship with the first The known class indices corresponding to the class prototypes most similar to the low-dimensional features of the weakly augmented versions of unlabeled samples. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class; Then, based on the known class number corresponding to the category prototype, the intra-class similarity, and the low-dimensional features of the unlabeled samples, it is determined whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue; the expression for determining whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue is: ； in, This represents the judgment value regarding whether the low-dimensional features of the corresponding unlabeled samples should be included in the corresponding feature queue. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first The category prototype of a known class. Indicates the first The intra-class similarity of known classes; specifically, when At that time, Add to the In the feature queue of a known class; when At that time, then not Add to any feature queue; Finally, the category prototype is calculated based on the updated feature queue.

6. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The seventh step specifically involves calculating the contrast loss, the expression for which is: ； in, Indicates comparative loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first A weakly augmented version of low-dimensional features from unlabeled samples. Indicates the first A strongly augmented version of low-dimensional features from unlabeled samples. This represents the total number of known classes. Indicates the first The category prototype of a known class.

7. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The ninth step specifically involves calculating the cross-entropy loss, the expression for which is: ； in, Represents cross-entropy loss, Indicates the number of labeled samples. Indicates the first The labeling categories of each labeled sample. Indicates the first The labeled sample was predicted as the first... The probability of a known class.

8. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The eighth step specifically involves calculating the private loss, the expression for which is: ； in, Indicates private loss, Indicates the number of unlabeled samples. Indicates the relationship with the first The index of the class with the highest probability in the predicted distribution of the weakly augmented version of the unlabeled samples. Indicates the first The similarity ratio of known classes. Indicates the first The weakly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

9. The method for semi-supervised training of a classifier according to claim 1, characterized in that: The tenth step specifically involves calculating the semi-supervised loss, the expression for which is: ； in, Indicates semi-supervised loss. Indicates the number of unlabeled samples. Indicates the relationship with the first The known class number of the class prototype most similar to the low-dimensional features of an unlabeled sample. Indicates the first The category prototype of a known class. Indicates the first Intra-class similarity of known classes Indicates the first The strongly augmented version of the unlabeled sample was predicted as the [missing value]. The probability of a known class.

10. A semi-supervised image classification method based on category prototypes, characterized in that, Includes the following steps: Step 1: Perform data augmentation on the unlabeled training images to obtain strongly augmented and weakly augmented versions of the unlabeled samples; Step 2: Analyze the labeled samples and augmented unlabeled samples using a neural network to obtain the predicted distribution and low-dimensional features; Step 3: Calculate the cross-entropy loss based on the predicted distribution of the labeled samples; Step 4: Generate a feature queue and category prototype based on the low-dimensional features of the labeled samples; Step 5: Calculate the clustering loss based on the low-dimensional features and category prototypes of the labeled samples; Step 6: Calculate the intra-class similarity and similarity ratio based on the feature queue and category prototype; Step 7: Based on the intra-class similarity, category prototype, and low-dimensional features of unlabeled samples, select low-dimensional features of unlabeled samples to update the feature queue and category prototype; Step 8: Calculate the contrastive loss based on the low-dimensional features of the class prototype and the unlabeled samples; Step 9: Calculate the private loss based on the similarity ratio and the predicted distribution of the unlabeled samples; Step 10: Calculate the semi-supervised loss based on the category prototype, intra-class similarity, low-dimensional features of unlabeled samples, and predicted distribution; Step 11: Using multiple labeled samples and multiple unlabeled samples with different augmented versions as input, low-dimensional features and predicted distribution as output, and the cross-entropy loss, clustering loss, contrastive loss, private loss and semi-supervised loss as loss functions, the parameters of the neural network are trained using the stochastic gradient descent method to obtain a trained classifier. Step 12: Input the test image into the trained classifier to obtain the predicted distribution and classification results.

Images