An open set image recognition method based on self-supervised learning

Abstract

The invention discloses an open set image recognition method based on self-supervised learning, and the steps include: obtaining a corresponding training data set according to the category of the image to be recognized to construct an open set data set; initializing all image samples in the open set data set; augmenting the image samples in the open set data set; establishing a self-supervised learning model based on sample similarity; establishing an open set recognition model based on a multi-branch structure; combining the self-supervised learning model with the open set recognition model to construct a self-supervised open set recognition model; and using the self-supervised open set recognition model trained by deep learning to evaluate the test set. The open set image recognition method improves the representation extraction ability of the model by means of self-supervised contrastive learning; takes into account two scenarios of image recognition in practical applications, insufficient labels and open set problems; joint training can promote each other and improve the accuracy of the two tasks at the same time.

Description

Technical Field

[0001] The invention relates to an open set image recognition method, in particular to an open set image recognition method based on self-supervised learning. Background Art

[0002] With the development of information technology, deep learning has achieved remarkable results in computer vision due to its powerful representation ability, especially in the field of image classification, which has been applied to many commercial scenarios, such as face recognition, robots and autonomous driving. Deep learning has already surpassed human level in some fields. However, the reliability of deep learning models depends to a certain extent on the number and quality of sample labels in the application scenario. At present, deep learning, which is mainly based on supervised learning, requires a large amount of labeled data to train the model, so as to continuously improve the accuracy of the model in its target field, and finally achieve a certain generalization ability and put it into practical application scenarios. However, it is worth noting that in many practical application tasks, samples do not have real labels. In addition, in order to obtain high-quality manual labels, a lot of manpower and material resources are required, and even annotation experts in the corresponding field need to be requested. Manual labeling also has some inevitable problems of human bias and randomness in the screening of samples.

[0003] The limited number of labeled samples will limit the performance of the model. In order to solve this problem, many scholars have proposed many methods to alleviate the problem of insufficient sample labels based on incompletely supervised models. Incompletely supervised learning has gradually attracted the attention of researchers, aiming to use fewer labeled samples for training without reducing model performance, thereby reducing the cost of manual labeling. The main methods include active learning, semi-supervised learning, and self-supervised learning.

[0004] In addition, most current models are based on the assumption that the data set is closed, that is, the data category distribution of the model's training set and test set is consistent, but the recognition problem in the real world is obviously the opposite. The model may face a large amount of unknown category data, that is, out-of-distribution samples, which will cause the model performance to drop sharply. Although the impact of unknown data on the recognition model can be alleviated to a certain extent by collecting a certain amount of unknown category data and setting up additional "unknown" classes (that is, known unknown category data) during the training process, the model's behavior is still unpredictable when facing unknown data other than known unknown category data because the categories in nature are difficult to enumerate. Therefore, the model needs a more reasonable way to handle unknown categories than traditional image classification, that is, open set recognition. Therefore, it is necessary to design an open set image recognition method based on self-supervised learning, which can fully consider the problems of insufficient number of labels and open environment in practical applications, can have better innovation and promotion, and can be used in image recognition problems encountered in practice. Summary of the invention

[0005] Purpose of the invention: To provide an open set image recognition method based on self-supervised learning, which can be processed in an open environment through self-supervision and open set recognition techniques, thereby solving the problem of image recognition tasks without large-scale labeled training sets.

[0006] Technical solution: The open set image recognition method based on self-supervised learning described in the present invention comprises the following steps:

[0007] Step 1: construct an open set dataset according to the category of the image to be identified, and the open set dataset includes labeled samples and unlabeled external samples;

[0008] Step 2, determining the number of labeled samples and the number of unlabeled samples in the open set dataset, and initializing all image samples in the open set dataset;

[0009] Step 3, augmenting the image samples in the open set dataset, so that each image sample in the open set dataset is augmented using two different image augmentation methods, and obtaining a positive sample pair and a negative sample pair for each image sample;

[0010] Step 4: Establish a self-supervised learning model based on sample similarity, and use the samples after data augmentation to train the established self-supervised learning model;

[0011] Step 5, establishing an open set recognition model based on a multi-branch structure for classifying and identifying out-of-distribution samples;

[0012] Step 6, combining the self-supervised learning model with the open set recognition model to construct a self-supervised open set recognition model, and using the open set dataset to perform deep learning training on the self-supervised open set recognition model;

[0013] Step 7: Use the self-supervised open set recognition model trained by deep learning to perform an evaluation test on the test set, and use the self-supervised open set recognition model for open set image recognition after the test meets the standards.

[0014] Furthermore, in step 1, the unlabeled out-of-distribution samples include out-of-distribution samples whose categories do not belong to the target classification.

[0015] Furthermore, in step 2, the specific steps for initializing all image samples in the open set dataset are:

[0016] Step 2.1, set the confidence s for all image samples in the open set dataset to indicate the probability that each image sample is an out-of-distribution sample;

[0017] In step 2.2, the confidence s of the labeled samples is initialized to 0, and the confidence s of the unlabeled samples is initialized to 1.

[0018] Furthermore, in step 3, the image augmentation methods used include geometric transformation data augmentation methods and color transformation data augmentation methods; the geometric transformation data augmentation methods include flipping transformation, rotation transformation, cropping transformation, deformation transformation and scaling transformation; the color transformation data augmentation methods include noise transformation, blur transformation, color transformation, erasing transformation and filling transformation.

[0019] Furthermore, in step 3, when obtaining the positive sample pair and the negative sample pair of each image sample, the image f(x) and f(x) obtained after the same image is augmented by the two methods are added. + ) as a positive sample pair, and the images f(x) and f(x) obtained after different images are augmented in two ways are compared. - ) as negative sample pairs.

[0020] Furthermore, in step 4, the self-supervised learning model established is:

[0021]

[0022] In the formula, is the contrast loss, and N is the total number of image samples in the open set dataset.

[0023] Furthermore, in step 5, the open set recognition model based on the multi-branch structure is established as follows:

[0024]

[0025] In the formula, is the loss of out-of-distribution sample detection, N l and N u are the number of labeled samples and unlabeled samples in the open set dataset, s(θ,x li ) and s(θ,x ui ) are respectively for the sample x li and x ui The confidence level output by logistic regression, s li and ui are the true values ​​of the sample confidence s within the distribution and outside the distribution, respectively.

[0026] Furthermore, in step 6, the specific steps of constructing the self-supervised open set recognition model are:

[0027] Step 6.1, establish a multi-classification loss model:

[0028]

[0029] In the formula, is the multi-classification loss, yi For sample x li The label, p i For sample x li The predicted probability of

[0030] Step 6.2, construct the self-supervised open set recognition model as:

[0031]

[0032] In the formula, is the loss function, α and λ are both loss weights.

[0033] Furthermore, in step 6, when the open set dataset is used to perform deep learning training on the self-supervised open set recognition model, the deep learning training method is the Adam gradient descent method, so as to minimize the loss function Thus, the optimal parameters of the self-supervised open set recognition model are obtained.

[0034] Compared with the prior art, the present invention has the following advantages: the representation and extraction capability of the model is improved by means of self-supervised contrastive learning; two scenarios of image recognition in practical applications, namely, insufficient labels and open set problems, are taken into consideration; joint training can promote each other and improve the accuracy of the two tasks at the same time; the present invention has high scalability and can be used in more different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 Define a diagram for the problem of the present invention;

[0037] Figure 3 This is a partial data set diagram of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0039] Embodiment 1:

[0040] like Figure 1-3 As shown, the open set image recognition method based on self-supervised learning disclosed in the present invention includes the following steps:

[0041] Step 1: construct an open set dataset according to the category of the image to be identified, and the open set dataset includes labeled samples and unlabeled external samples;

[0042] Step 2, determining the number of labeled samples and the number of unlabeled samples in the open set dataset, and initializing all image samples in the open set dataset;

[0043] Step 3, augmenting the image samples in the open set dataset, so that each image sample in the open set dataset is augmented using two different image augmentation methods, and obtaining a positive sample pair and a negative sample pair for each image sample;

[0044] Step 4: Establish a self-supervised learning model based on sample similarity, and use the samples after data augmentation to train the established self-supervised learning model;

[0045] Step 5, establishing an open set recognition model based on a multi-branch structure for classifying and identifying out-of-distribution samples;

[0046] Step 6, combining the self-supervised learning model with the open set recognition model to construct a self-supervised open set recognition model, and using the open set dataset to perform deep learning training on the self-supervised open set recognition model;

[0047] Step 7: Use the self-supervised open set recognition model trained by deep learning to perform an evaluation test on the test set, and use the self-supervised open set recognition model for open set image recognition after the test meets the standards.

[0048] Furthermore, due to the problem of insufficient labels, there are a small number of labeled samples and a large number of unlabeled samples in the training data set. The unlabeled samples also contain a certain number of out-of-distribution samples. The true categories of these out-of-distribution samples do not belong to the classification target. Assume that there are K categories in the category space, and the categories of the labeled samples all belong to one of the K categories. The true category of the unlabeled samples does not belong to this K category. Therefore, in step 1, the unlabeled out-of-distribution samples contain out-of-distribution samples whose categories do not belong to the target category. When constructing an open set data set by obtaining the corresponding training data set according to the category of the image to be identified, one method is to use category I of a data set as the in-distribution sample, and the remaining category O as the out-of-distribution sample. The test set does not contain these I+O categories, which do not belong to the target category. Another method is to use multiple data sets for data set construction, with category I of one data set as the in-distribution sample and category O of another data set as the out-of-distribution sample.

[0049] Furthermore, in step 2, the specific steps for initializing all image samples in the open set dataset are:

[0050] Step 2.1, set the confidence s for all image samples in the open set dataset to indicate the probability that each image sample is an out-of-distribution sample;

[0051] In step 2.2, the confidence s of the labeled samples is initialized to 0, and the confidence s of the unlabeled samples is initialized to 1. The confidence represents the probability that the image sample is an out-of-distribution sample. Unlabeled samples with scores close to 1 are more likely to be out-of-distribution samples, and the closer the score is to 0, the less likely it is to be an out-of-distribution sample.

[0052] Furthermore, in step 3, the image augmentation methods used include geometric transformation data augmentation methods and color transformation data augmentation methods; the geometric transformation data augmentation methods include flipping transformation, rotation transformation, cropping transformation, deformation transformation and scaling transformation; the color transformation data augmentation methods include noise transformation, blur transformation, color transformation, erasing transformation and filling transformation.

[0053] Furthermore, in step 3, when obtaining the positive sample pair and the negative sample pair of each image sample, the image f(x) and f(x) obtained after the same image is augmented by the two methods are added. + ) as a positive sample pair, and the images f(x) and f(x) obtained after different images are augmented in two ways are compared. - ) as negative sample pairs.

[0054] Furthermore, in step 4, the self-supervised learning model established is:

[0055]

[0056] In the formula, is the contrast loss, N is the total number of image samples in the open set dataset, and the self-supervised learning model constructs a representation by learning the similarity or dissimilarity of two image samples. After constructing positive and negative samples, self-supervised learning is achieved by measuring the distance between positive and negative samples. The distance between a sample and a positive sample is much larger than the distance between a sample and a negative sample.

[0057] Furthermore, in step 5, the open set recognition model based on the multi-branch structure is established as follows:

[0058]

[0059] In the formula, is the loss of out-of-distribution sample detection, N l and N u are the number of labeled samples and unlabeled samples in the open set dataset, s(θ,x li ) and s(θ,x ui ) are respectively for the sample x li and x ui The confidence level output by logistic regression, s li and uiare the true values ​​of the sample confidence s within the distribution and outside the distribution, respectively. As the model training proceeds, As it decreases gradually, the prediction results of the model will become more and more accurate.

[0060] Furthermore, in step 6, the specific steps of constructing the self-supervised open set recognition model are:

[0061] Step 6.1, establish a multi-classification loss model:

[0062]

[0063] In the formula, is the multi-classification loss, y i For sample x li The label, p i For sample x li The predicted probability of

[0064] Step 6.2, construct the self-supervised open set recognition model as:

[0065]

[0066] In the formula, is the loss function, α and λ are both loss weights.

[0067] Furthermore, in step 6, when the open set dataset is used to perform deep learning training on the self-supervised open set recognition model, the deep learning training method is the Adam gradient descent method, so as to minimize the loss function Thus, the optimal parameters of the self-supervised open set recognition model are obtained.

[0068] The open set image recognition method based on self-supervised learning disclosed in the present invention can effectively alleviate the problem of the classification model's dependence on a large amount of manual annotation and the model's application in an open environment.

[0069] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. An open set image recognition method based on self-supervised learning, characterized in that: The steps include: Step 1: construct an open set dataset according to the category of the image to be identified, and the open set dataset includes labeled samples and unlabeled external samples; Step 2, determining the number of labeled samples and the number of unlabeled samples in the open set dataset, and initializing all image samples in the open set dataset; Step 3, augmenting the image samples in the open set dataset, so that each image sample in the open set dataset is augmented using two different image augmentation methods, and obtaining a positive sample pair and a negative sample pair for each image sample; Step 4: Establish a self-supervised learning model based on sample similarity, and use the samples after data augmentation to train the established self-supervised learning model; Step 5, establishing an open set recognition model based on a multi-branch structure for classifying and identifying out-of-distribution samples; Step 6, combining the self-supervised learning model with the open set recognition model to construct a self-supervised open set recognition model, and using the open set dataset to perform deep learning training on the self-supervised open set recognition model; Step 7, using the self-supervised open set recognition model trained by deep learning to evaluate the test set, and using the self-supervised open set recognition model for open set image recognition after the test meets the standards; In step 3, when the positive sample pair and the negative sample pair of each image sample are obtained, the image f(x) and f(x) obtained after the same image is augmented in two ways are added. + ) as a positive sample pair, and the images f(x) and f(x) obtained after different images are augmented in two ways are compared. - ) as a negative sample pair; In step 4, the self-supervised learning model established is: In the formula, is the contrast loss, N is the total number of image samples in the open set dataset; In step 5, the open set recognition model based on the multi-branch structure is established as follows: In the formula, is the loss of out-of-distribution sample detection, N l and N u are the number of labeled samples and unlabeled samples in the open set dataset, s(θ,x li ) and s(θ,x ui ) are respectively for the sample x li and x ui The confidence level output by logistic regression, s li and ui are the true values ​​of the sample confidence s within and outside the distribution respectively; In step 6, the specific steps of building a self-supervised open set recognition model are: Step 6.1, establish a multi-classification loss model: In the formula, is the multi-classification loss, y i For sample x li The label, p i For sample x li The predicted probability of Step 6.2, construct the self-supervised open set recognition model as: In the formula, is the loss function, α and λ are both loss weights.

2. The open set image recognition method based on self-supervised learning according to claim 1, characterized in that: In step 1, the unlabeled out-of-distribution samples include out-of-distribution samples whose categories do not belong to the target classification.

3. The open set image recognition method based on self-supervised learning according to claim 1, characterized in that: In step 2, the specific steps for initializing all image samples in the open set dataset are: Step 2.1, set the confidence s for all image samples in the open set dataset to indicate the probability that each image sample is an out-of-distribution sample; In step 2.2, the confidence s of the labeled samples is initialized to 0, and the confidence s of the unlabeled samples is initialized to 1.

4. The open set image recognition method based on self-supervised learning according to claim 1, characterized in that: In step 3, the image augmentation methods used include geometric transformation data augmentation methods and color transformation data augmentation methods; the geometric transformation data augmentation methods include flipping transformation, rotation transformation, cropping transformation, deformation transformation and scaling transformation; the color transformation data augmentation methods include noise transformation, blur transformation, color transformation, erasing transformation and filling transformation.

5. The open set image recognition method based on self-supervised learning according to claim 1, characterized in that: In step 6, when using the open set dataset to perform deep learning training on the self-supervised open set recognition model, the deep learning training method is the Adam gradient descent method, so as to minimize the loss function Thus, the optimal parameters of the self-supervised open set recognition model are obtained.

Images