A multi-view semi-supervised image classification method based on contrastive loss

Abstract

The present invention discloses a multi-view semi-supervised image classification method based on contrast loss. The method uses self-supervised contrast loss on unlabeled data to make full use of the potential features of the data. At the same time, based on supervised contrast loss, it uses category supervision information and fuses different view information with the help of multi-view consistency, thereby achieving the improvement of the accuracy of the model in multi-view classification tasks without the need for pseudo labels. This method is aimed at multi-view image classification tasks in semi-supervised scenarios, and can improve classification performance with a small amount of annotation information and with the help of consistency between multiple views. It is suitable for solving multi-view image classification tasks that lack data label information.

Description

Technical Field

[0001] The present invention relates to a multi-view semi-supervised image classification method. The method introduces contrast loss, can make full use of unlabeled data to extract data features and a small amount of data label information, and improves classification performance with the help of multi-view consistency, belonging to the field of computer artificial intelligence image classification technology. Background Art

[0002] In real life, the same thing can be described from many different ways or angles. For example, different people can be identified by face images, irises, fingerprint information, etc. Information from different sources constitutes multiple views of things. For common image data in practical applications, it is necessary to establish a huge image database and manually annotate it, but the cost of manual annotation is high and inefficient. In the era of big data, it is not difficult to collect large-scale data, but the collected data often lacks label information, and it is very difficult and time-consuming to provide labels for these data. Therefore, in practical applications, we often face semi-supervised scenarios where a large amount of data does not have category labels and only a small amount of data has category labels. Traditional semi-supervised learning methods are mostly applied to single-view data. Existing methods such as collaborative training are trained by providing pseudo-labels to unlabeled data. However, this learning method that relies on pseudo-labels can easily bring noise labels and cause a lot of repeated training. Summary of the invention

[0003] Purpose of the invention: Aiming at the problem of multi-view image classification in semi-supervised scenarios, the present invention proposes a novel multi-view semi-supervised image classification algorithm based on contrast loss. The algorithm constructs positive and negative examples through image enhancement on unlabeled data, and uses self-supervised contrast loss to make full use of the potential features of the data. At the same time, based on a small amount of category supervision information, supervised contrast loss is used to help improve the network's ability to extract data features, and different view information is fused with the help of multi-view consistency, so as to achieve the performance improvement of the model in multi-view classification tasks without the help of pseudo labels. This method is aimed at multi-view image classification tasks in semi-supervised scenarios, and can improve classification performance by using a small amount of annotation information and the consistency between multiple views. It is suitable for solving multi-view image classification tasks that lack data label information.

[0004] Technical solution: A multi-view semi-supervised image classification method based on contrast loss, which applies contrast loss at the instance level and category level to improve the network's ability to represent images, and improves the network's classification performance for multi-view images with the help of multi-view consistency, including the following contents:

[0005] First, the user needs to prepare a multi-view image training library, which contains two views of the image. Each view consists of a majority of unlabeled samples and a small number of samples with category labels.

[0006] Next, two different image enhancement techniques are applied to each image sample in the training data of each view, so that each image sample can obtain two different enhanced images after enhancement. The obtained enhanced image is input into the encoder network for training, and the feature representation of the enhanced image can be extracted. Since one image sample can produce two different enhanced images, two different feature representations can be obtained after passing through the encoder network.

[0007] There are many image enhancement techniques, such as rotation, cropping, masking, color distortion (including a collection of one or several attribute adjustment operations such as brightness, contrast, saturation, hue, etc.), adding noise, etc. Multiple enhancement techniques can usually be used in combination.

[0008] The feature representation obtained by the encoder network is passed as input to the projection network, which can map the high-dimensional features to the low-dimensional projection space to obtain the low-dimensional embedding of the feature in the projection space. Then, two contrast losses are calculated in the low-dimensional embedding space, namely the contrast loss at the instance level and the category level, which constitute an important part of the loss function of the entire network model.

[0009] How to calculate these two contrastive losses. Specifically, for each image sample, the two low-dimensional embeddings obtained through the above operations for the two enhanced images can be used as positive examples for each other, and any of them with the low-dimensional embeddings generated by other image samples constitutes a negative example, so the instance-level contrastive loss of each sample can be calculated. In the instance-level contrastive loss, for each sample, there are multiple pairs of negative examples, but only one pair of positive examples. In addition, for samples with category labels in image samples, positive and negative examples can also be constructed according to their category labels, that is, sample pairs with the same category labels are positive examples, and sample pairs with different category labels are negative examples, so the contrastive loss at the category level can be calculated for each labeled sample. Here, for each labeled sample, there can be multiple pairs of positive and negative examples.

[0010] Then, the features obtained by the encoder network of the labeled samples are passed into the classifier to obtain the category prediction and calculate the cross entropy loss with the true label. At the same time, the similarity between the softmax layer outputs obtained by the classifier is calculated using the cross entropy loss. In addition, the norm can be used to calculate the similarity between the low-dimensional embeddings of the samples of the two views. All the above loss functions are weighted summed, the gradient is calculated, and the entire network model is updated iteratively through back propagation.

[0011] After the network is trained, prediction is performed. The user takes the multi-view image to be predicted as input and passes it through the encoder network and classifier in sequence to obtain the predicted category of the object to be tested.

[0012] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-view semi-supervised image classification method based on contrast loss as described above is implemented.

[0013] A computer-readable storage medium stores a computer program for executing the multi-view semi-supervised image classification method based on contrast loss as described above.

[0014] Beneficial effects: Compared with the existing technology, the semi-supervised learning method used in the present invention on multi-view data also needs to consider the correlation between views. Through two contrast losses, it not only extracts the potential features of the data from the unlabeled data, but also makes full use of a small amount of category supervision information to assist the training of the network. At the same time, it uses the consistency of multiple views to fuse the information of different views, and successfully avoids the introduction of label noise, and finally achieves a good classification effect. In addition, the present invention is aimed at multi-view image classification tasks in semi-supervised scenarios, and is suitable for solving multi-view image classification tasks that lack data label information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a method principle diagram of an embodiment of the present invention;

[0016] Figure 2 is a method flow chart of an embodiment of the present invention;

[0017] Figure 3 is a flow chart of multi-view image samples processed through a network in an embodiment of the present invention;

[0018] Figure 4 is a flow chart of calculating contrast loss in low-dimensional space in an embodiment of the present invention;

[0019] Figure 5 This is a flow chart of calculating the consistency loss of two views in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0021] like Figure 1 As shown in FIG, the multi-view semi-supervised image classification method based on contrast loss includes the following contents:

[0022] First, the user needs to prepare a multi-view image library containing two views. The image library needs to contain two views of the image, and the data set of each view consists of a majority of unlabeled image samples and a small number of image samples with category labels. Then, each input image sample is subjected to two different image enhancements and input into the encoder network to obtain two feature representations. These feature representations are mapped into a low-dimensional projection space, and two contrast losses are calculated on this low-dimensional space. In addition, these feature representations are directly input into the classifier for category prediction, and the classification loss between its output and the true label is calculated. Considering the consistency of multiple views, it is also necessary to calculate the similarity between the classification predictions obtained from the two views, as well as the similarity between the embeddings in the low-dimensional space corresponding to the features of the two views, and add them to the overall loss function of the network. The above loss functions are weighted summed, and the encoder network and classifier are iteratively updated through gradient descent back propagation. After the network is trained, when making predictions, the user only needs to input the multi-view image to be predicted into the encoder network, and then pass the features into the classifier, and the classifier will return the predicted category of the object to be tested to the user.

[0023] like Figure 2 As shown in Figure 2, the process of the multi-view semi-supervised image classification method based on contrast loss is as follows:

[0024] Step 100: Establish a multi-view image library as a training data set The total number of samples included is N. The multi-view image library needs to contain two-view datasets. The training set of each view consists of a majority of unlabeled data samples and a small number of data samples with category labels, which are denoted as in It is a sample The one-hot vector of the true category label, when the sample When the category is c∈{0,…,C-1} otherwise The multi-view image dataset of the category to be predicted is denoted as

[0025] Step 101, extract a batch of samples from the training set of each view each time, input them into the network to extract features, and project them into a low-dimensional space. The specific steps are as follows:

[0026] Step 1011, a batch of image samples (batch size is b) from the training dataset of the vth view (v∈V≡{1,2}) Implement two different image enhancements t and t ′ Get two enhanced images and

[0027] Step 1012: the enhanced image and Input to the encoder network f v (v∈{1,2}) to obtain different enhanced feature representations of image samples and

[0028] Step 1013: Feature Representation and After the projection network g v (v∈{1,2}) is mapped to a low-dimensional space, and the normalized embedding of these feature representations in the low-dimensional space is obtained and

[0029] Step 102, calculating two contrast losses in the low-dimensional embedding space, the specific steps are:

[0030] Step 1021, calculate the instance-level contrast loss on the low-dimensional embedding obtained for each training sample of each view. In a batch of training data of size b, for any input image sample Two enhanced images of and They pass through the encoder network and the projection network to obtain the corresponding low-dimensional embedding and The low-dimensional embedding of one of the augmented images As an anchor point, the low-dimensional embedding of another enhanced image of the image sample constitutes a positive example The low-dimensional embeddings generated by other image samples in the batch constitute negative examples So we get the instance-level contrast loss:

[0031]

[0032] Among them, i∈I≡{1,…,b}, M={1,…,2b}, A(i)≡M\{i}, τ1 is the temperature parameter. Here, in a batch, there is one positive example and 2b-2 negative examples.

[0033] Step 1022, recalculate the category-level contrast loss on the low-dimensional embedding obtained from the labeled image sample. in It is a sample The one-hot vector of the true category label, when the sample When the category is c∈{0,…,C-1} otherwise The low-dimensional embeddings generated by image samples with the same class label As a positive example (there may be multiple positive examples), the negative examples include the embedding of image samples of different categories and the embedding of unlabeled samples (there are also multiple negative examples). Therefore, the contrast loss is introduced at the category level, that is, the supervised contrast loss function:

[0034]

[0035] Among them, i∈I≡{1,…,b}, |P(i)| represents the potential of the set P(i), and τ2 is the temperature parameter.

[0036] Step 103, based on the outputs obtained from the two views, calculate the consistency loss between the views, the specific steps are:

[0037] Step 1031, for the input image sample from the vth (v∈V≡{1,2})th view in a batch Embedding obtained by encoder network and projection network and Compute the consistency regularization loss function between the embeddings of these two views

[0038]

[0039] in, The consistency criterion of multiple views is achieved by constructing a regularization term through the l2 norm.

[0040] Step 1032, for a labeled image sample from the vth (v∈V≡{1,2})th view in a batch in It is a sample The one-hot vector of the true category label, when the sample When the category is c∈{0,…,C-1} otherwise Its two enhancements can obtain feature representation after passing through the encoder network Then input the feature representation into the classifier h v The softmax layer output can be obtained in (·)

[0041]

[0042] Therefore, the score of the image sample can be calculated using the following formula: Next, use the cross entropy loss to calculate the similarity of the softmax layer outputs of the two views:

[0043]

[0044] Step 104, weighted sum the above loss functions and update the network through gradient back propagation. Specifically, the consistency regularization loss in step 1031 is Similarity loss in step 1032 Classification Loss And the two contrast losses in steps 1021-1022 and Perform weighted summation and get

[0045]

[0046] The encoder network and classifier are then iteratively updated through back propagation. Among them, λ1, λ2, λ3, λ4, λ5 are weight parameters.

[0047] Step 105, training ends the prediction phase, and the multi-view images of the category to be predicted are They are input into the encoder networks f1 and f2 in step 101 respectively, and the obtained feature representations are input into the classifiers h1 and h2 in step 103 respectively, and two classification predictions are obtained. and Take the average That is the predicted category of the multi-view image.

[0048] Obviously, those skilled in the art should understand that the various steps of the multi-view semi-supervised image segmentation method based on contrast loss of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A multi-view semi-supervised image classification method based on contrast loss, characterized in that: The steps include: Step 100, establishing a multi-view image library as a training data set, the multi-view image library contains two-view data sets, each view data set includes image data without category labels and image data with category labels, wherein the number of image data without category labels is greater than the number of image data with category labels, and C represents the number of categories of samples in the data set; Step 101, extracting a batch of image data from the data set of each view as samples and inputting them into the encoder network for feature extraction, and projecting them into a low-dimensional space to obtain two low-dimensional embeddings; Step 102, calculating two contrast losses for samples of each view in the low-dimensional embedding space; Step 103, based on the outputs obtained from the two views, the consistency loss between the views is calculated; for the feature representation of the sample with the category label obtained in step 101, the classification loss is calculated with the real label of the sample after passing through the classifier, and then the similarity and consistency regularization loss are calculated for the classification outputs generated by different views and the low-dimensional embedding of the feature representation respectively; the loss function is combined with the contrast loss in step 102, and the encoder network and the classifier are iteratively updated through back propagation; Step 104, performing weighted summation on the loss function and updating the network through gradient back propagation; Step 105, when the training is completed and prediction is performed, the multi-view images of the category to be predicted are respectively input into the encoder network in step 101, and the obtained feature representation is input into the classifier in step 103, and the prediction results are averaged to obtain the predicted category of the multi-view image; The step 103 comprises: Step 1031, for an input image sample from the vth view in a batch Embedding obtained by encoder network and projection network and Compute the consistency regularization loss function between the embeddings of these two views in, Step 1032, for a labeled image sample from the vth view in a batch in It is a sample The one-hot vector of the true category label, its two enhancements after the encoder network can obtain feature representation Then input the feature representation into the classifier h v The softmax layer output can be obtained in (·) Therefore, the classification loss of the image sample is calculated using the following formula: Next, use the cross entropy loss to calculate the similarity of the softmax layer outputs of the two views:

2. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: In step 101, two different image enhancements are performed on each image sample in the training data set from the vth view, and the enhanced image is input into the encoder network for training to obtain different enhanced feature representations of the image sample. The obtained features are mapped to a low-dimensional space through a projection network to obtain two low-dimensional embeddings, v∈V≡{1,2}.

3. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: In step 102, two contrast losses are calculated for the image samples of each view in the low-dimensional embedding space; for each image sample, the low-dimensional embedding of its enhanced sample is used as a positive example, and the low-dimensional embedding obtained with the enhancement of other images constitutes a negative example, and the instance-level self-supervised contrast loss is used for optimization; in addition, for image samples with category labels, the low-dimensional embeddings of image samples with the same category labels constitute positive examples, and those with different category labels constitute negative examples, and the supervised contrast loss is used for optimization at the category level.

4. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: The step 101 comprises: Step 1011, a batch of image samples from the training data set of the vth view Implement two different image enhancements t and t ′ Get two batch sizes of augmented images and i∈{1,…,b}; Step 1012, the enhanced images of the two views are forward propagated through encoder networks f1 and f2 respectively to obtain feature representations and Step 1013: The feature representations of the two views are respectively passed into the projection networks g1 and g2 for further propagation to obtain the normalized low-dimensional embeddings of these feature representations. and 5. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: The step 102 includes: Step 1021, calculate the instance-level contrast loss on the low-dimensional embedding obtained by the projection network for each view; in a batch of training data of size b, for any input image sample Two enhanced images of and They pass through the encoder network and the projection network to obtain the corresponding low-dimensional embedding and The low-dimensional embedding of one of the augmented images As an anchor point, the low-dimensional embedding of another enhanced image of the image sample constitutes a positive example The low-dimensional embeddings generated by other image samples in the batch constitute negative examples So we get the instance-level contrast loss: Among them, i∈I≡{1,…,b}, j≠i, M={1,…,2b}, A(i)≡M\{i}, τ1 is the temperature parameter; in the instance-level contrast loss function, in a batch, there is one positive example and 2b-2 negative examples; Step 1022, recalculate the category-level contrast loss on the low-dimensional embedding obtained from the labeled image sample; for the input image sample taken from the category-labeled dataset in It is a sample The one-hot vector of the true category label, when the sample When the category is c∈{0,…,C-1} otherwise The low-dimensional embeddings generated by image samples with the same class label As a positive example, The negative examples include the embedding of image samples of different categories and the embedding of unlabeled samples. Therefore, the contrast loss is introduced at the category level, that is, the supervised contrast loss function: Among them, i∈I≡{1,…,b}, |P(i)| represents the potential of the set P(i), and τ2 is the temperature parameter.

6. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: In step 104, the consistency regularization loss in step 1031 is Similarity loss in step 1032 Classification Loss and the two contrast losses in steps 1021 and 1022 and Perform weighted summation and get The encoder network and classifier are then iteratively updated through back propagation; among them, λ1, λ2, λ3, λ4, λ5 are weight parameters.

7. The multi-view semi-supervised image classification method based on contrast loss according to claim 1, characterized in that: In step 105, the multi-view image of the category to be predicted is They are input into the encoder networks f1 and f2 in step 101 respectively, and the obtained feature representations are input into the classifiers h1 and h2 in step 103 respectively, and two classification predictions are obtained. and Take the average That is the predicted category of the multi-view image.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-view semi-supervised image classification method based on contrast loss as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the multi-view semi-supervised image classification method based on contrast loss as described in any one of claims 1 to 7.

Images