An open set image recognition method and system based on active learning

Abstract

The present application relates to the technical field of machine learning open set image recognition, and provides an open set image recognition method and system based on active learning, which can fully utilize pseudo-labeled information of unlabeled data, reduce the influence of open set samples on the classification model, expand the labeled data set, and improve the accuracy of open set image recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of open set image recognition based on machine learning, and particularly relates to an open set image recognition method and system based on active learning. BACKGROUND

[0002] Image classification is the most basic problem in the field of computer vision, and has a wide range of application scenarios. In real life, for example, in the field of traffic management, by classifying the pictures of vehicles taken, better urban road planning can be achieved; in the field of e-commerce, by classifying the product patterns browsed by customers, more accurate product recommendations can be made to customers; in the field of agriculture, by classifying the insect pictures collected, farmers can be guided to discover pests in time and take preventive measures. The goal of image classification is to divide different images into different categories to achieve the minimum classification error. In recent years, image classification technology based on deep learning often learns a classification model through a large number of labeled image data, and makes a category prediction for unlabeled image data. This process is closed, that is, the images in the labeled data and the unlabeled data belong to the same category. However, the real application scenario is open, and different images from the category of the labeled data often appear in the unlabeled data, the category space between the labeled data and the unlabeled data is different, which leads to a decrease in the prediction performance of the model for the unlabeled samples. The open set image recognition task requires the classifier to not only accurately classify the seen class data, but also effectively handle the new class data.

[0003] The existing open set image recognition methods mainly include sample re-weighting and open set detection scoring. The idea of sample re-weighting is to assign a lower weight to the open set image, so as to reduce its influence on model training. The sample re-weighting method involves a meta-learning-based unlabeled data re-weighting mechanism and a distance-based unlabeled data re-weighting mechanism. The purpose of open set detection scoring is to design a scoring mechanism to calculate an open set score for each unlabeled image data, and detect the open set image according to the score and a predefined threshold. In the field of open set image recognition, the open set instance detection scoring mechanism based on model prediction integration and the open set detection scoring mechanism based on metric do not consider the information of the unlabeled samples, and only learn from limited labeled data, which cannot effectively improve the recognition performance of the open set image. The method based on noise label optimization, the method based on one-versus-all classifier, and the method based on image feature and category label matching score propose to consider mining the information of the unlabeled samples to improve the expression ability of the model. However, these methods are insufficient in evaluating the unlabeled data and utilizing the pseudo-label information, and it is difficult to learn a feature representation with high category discrimination, which leads to a low open set recognition accuracy and affects the accuracy of image classification. SUMMARY

[0004] To solve the problems in the prior art, the application provides an open set image recognition method and system based on active learning.

[0005] The application aims to realize the following technical solutions:

[0006] An open set image recognition method based on active learning comprises the following steps:

[0007] S1: Given a labeled image dataset and an unlabeled dataset Generate the class semantic representation corresponding to the seen class image by using the language model Word2Vec

[0008] S2: Introduce the image rotation prediction framework RotNet, rotate the images in the given image dataset by 0°, 90°, 180° and 270° respectively, perform self-supervised representation enhancement learning, train the image feature extraction model, and construct the self-supervised loss L sef ;

[0009] S3: Use the image feature extraction model to perform feature representation on all images, integrate class semantic knowledge, map the image features and class semantic knowledge to a common space for similarity calculation, calculate the image feature and class semantic matching score by using the cosine similarity, and construct the matching loss L ma ;

[0010] S4: The Otsu algorithm is used to adaptively select a predefined threshold, the matching score is compared with the predefined threshold, the unlabeled image with a matching score lower than the threshold is regarded as an open set image, and the unlabeled image with a matching score higher than the threshold is regarded as a seen class image;

[0011] S5: An active learning strategy is used to select high-confidence open set samples and high-confidence seen class samples, the top-k samples with the highest scores are high-confidence samples, the high-confidence seen class samples and the calculated pseudo-labels are added to the labeled data for training, and the cycle is formed until the open set samples are identified, and the expansion of the labeled dataset is realized at the same time;

[0012] S6: Class prediction is performed for each unlabeled data, and the classification loss L ce .

[0013] Further, step S1 comprises: giving a labeled sample set and an unlabeled sample set wherein and respectively represent the labeled and unlabeled images, represent the true label of the labeled sample The labeled sample set contains K target categories, that is a semantic representation is defined for each category represent the semantic features of the category.

[0014] Further, step S2 comprises: introducing an image rotation prediction framework RotNet, performing self-supervised representation enhancement learning on all labeled data and unlabeled data, generating four corresponding images by rotating each image by 0°, 90°, 180° and 270°, and predicting the category probability, for image feature x i , the feature x i,j after rotation is x i *(j-1)*90°, where j=1,2,3,4, the prediction probability q i,j after rotation is:

[0015] q i,j =h(f(x i,j )),

[0016] wherein h(·) represents a linear layer and a softmax function, and the self-supervised loss function of the training process is:

[0017]

[0018] Further, step S3 comprises: mapping the image features and the category semantic features to a common space to realize the alignment of visual features and semantic features, for the semantic features, using Word2Vec to obtain the semantic representation of the seen categories, for image x and its label y, first extracting the feature z=f(x) of the image x, then extracting the semantic representation a corresponding to the label y, and then mapping z and a to a common space for similarity calculation, using cosine similarity, to obtain the matching score of sample x and label y:

[0019] s(x,y)=s(z,a)=cos(f(x),a), s(x,y) reflects whether y is the correct category label of x by calculating the size of the similarity between the image and the category semantic, wherein the category semantic with large similarity is considered to be the correct category label;

[0020] randomly selecting a label other than the true label and the most difficult to distinguish label The loss function of model training is as follows, and the loss function of labeled data is as follows:

[0021]

[0022] wherein, and respectively represent the most difficult to distinguish label and the randomly selected label,

[0023]

[0024]

[0025] For each unlabeled data Assuming its prediction probability is Its pseudo label The loss function of unlabeled data is as follows:

[0026]

[0027] wherein,

[0028] Thus, the matching loss L is constructed ma :

[0029]

[0030] Further, step S4 includes: the selection of the threshold value is adaptive by the Otsu algorithm, for a batch of samples, first, the histogram is constructed according to the matching score value in s, the score value is normalized, and the forward cumulative sum and the horizontal flip sum are calculated, respectively, to obtain w1 and w2, then the average value of each score in the forward cumulative sum and the horizontal flip sum is calculated, respectively, and m1 and m2 are obtained:

[0031]

[0032]

[0033] Then, the variance v is calculated according to the average value:

[0034] v = w1 * w2 * (m1-m2) 2 ,

[0035] Finally, the variance is maximized, and the score corresponding to the maximum variance is the threshold value t:

[0036]

[0037] Further, step S5 includes: using an active learning strategy, selecting high-confidence open set samples and high-confidence seen class samples, adding high-confidence seen class samples and their calculated pseudo labels to the labeled data for training, forming a cycle and completing the recognition of open set samples, the top-k samples with the highest scores are high-confidence samples, and the matching scores s i are sorted from large to small, the top-k samples with the highest scores and their calculated pseudo labels are added to the labeled data set, the top-k samples with the lowest scores are considered to be open set samples, the number k of high-confidence samples is determined according to the number of open set samples, after completing the selection of open set samples and seen class samples in one round, the next iteration is performed, and the feature extraction network is updated until all open set data is recognized.

[0038] Further, step S6 includes: for each input image x i extracting a feature representation: z i =f(x i ), f(·) represents a calculation function of the feature extraction network, a linear layer and a softmax layer are added to the feature extraction network to obtain a K-dimensional category probability vector p i :

[0039] p i =g(z i )=g(f(x i )),

[0040] wherein g(·) represents a linear layer and a softmax operation, in the training stage, the category probability vector of the labeled image is optimized by using a cross-entropy loss:

[0041]

[0042] wherein, represents the predicted probability of

[0043] An open set image recognition system based on active learning includes the following modules:

[0044] A category semantic generation module: according to the categories corresponding to the labeled images in the given image data set, a language model Word2Vec is used to generate semantic representations of the categories;

[0045] A self-supervised representation enhancement module: an image rotation prediction framework RotNet is introduced, the images in the given image data set are rotated by 0°, 90°, 180° and 270° respectively, and a self-supervised representation enhancement learning is performed to train an image feature extraction model;

[0046] Knowledge-guided open set detection module: using ResNet to perform feature representation on the image, integrating class semantic knowledge, mapping the image features and class semantic knowledge to a common space for similarity calculation, using cosine similarity to calculate the image feature and class semantic matching score;

[0047] Class distinction module: by Otsu algorithm to adaptively select a threshold, compare the matching score with the predefined threshold, and the unlabeled image with a matching score lower than the threshold is considered as an open set image, and the unlabeled image with a matching score higher than the threshold is considered as a seen class image;

[0048] Active learning module: an active learning strategy is adopted to select high-confidence open set samples and high-confidence seen class samples, and the high-confidence seen class samples and the pseudo-labels calculated therefrom are added to the labeled data for training, so as to form a cycle and complete the identification of the open set samples and the expansion of the labeled data set;

[0049] Image classification module: class prediction is performed for each unlabeled data.

[0050] In summary, the present application has the following beneficial effects:

[0051] The present application introduces the semantic knowledge of seen classes, iteratively identifies open set samples using a threshold selection strategy and an active learning method, reduces the influence of open set samples on the classification model, and fully utilizes unlabeled data to expand the labeled data set. BRIEF DESCRIPTION OF DRAWINGS

[0052] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings:

[0053] Figure 1 The present application is a method flowchart;

[0054] Figure 2 It is a self-supervised representation enhanced learning diagram;

[0055] Figure 3 It is an image feature and class knowledge mapping in a common space and similarity calculation diagram;

[0056] Figure 4 It is an active learning framework diagram;

[0057] Figure 5 It is a system structure diagram of the present application. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with reference to the accompanying drawings and examples.

[0059] As Figure 1 shown is a flowchart of an open set image recognition method based on active learning, specifically comprising the following steps:

[0060] S1: Given labeled image dataset and unlabeled dataset Generate class semantic representation corresponding to seen class image pairs using language model Word2Vec

[0061] wherein step S1 is specifically:

[0062] Given labeled sample set and unlabeled sample set wherein and represent labeled and unlabeled images respectively, represent the true label of the labeled sample The labeled sample set contains K target classes, i.e. define a semantic representation for each class represent the semantic features of the class. Open set samples refer to new class samples existing in the unlabeled dataset, i.e. do not belong to any of the K target classes. It is assumed that a single training batch consists of n labeled images and m unlabeled images.

[0063] S2: Introduce image rotation prediction framework RotNet, rotate the images in the given image dataset by 0°, 90°, 180° and 270° respectively, perform self-supervised representation enhancement learning, train the image feature extraction model, and construct the self-supervised loss L sef ;

[0064] As shown in Figure 2 is a schematic diagram of self-supervised representation enhancement learning.

[0065] wherein step S2 is specifically:

[0066] Introduce image rotation prediction framework RotNet to perform self-supervised representation enhancement learning on all labeled data and unlabeled data. Rotate each image by 0°, 90°, 180° and 270° to generate four corresponding images, and predict the classification probability. Specifically, for image feature x i , the rotated feature x i,j = x i *(j-1)*90°, where j = 1, 2, 3, 4. The predicted probability q i,j :

[0067] q i,j = h(f(x i,j )),

[0068] where h(·) denotes a linear layer and a softmax function. The self-supervised loss function of the training process is:

[0069]

[0070] S3: Use the image feature extraction model to perform feature representation on all images, map the image features and category semantic knowledge to a common space for similarity calculation by integrating category semantics, calculate the image feature and category semantic matching score using cosine similarity, and construct a matching loss L ma ;

[0071] As shown in Figure 3 is a schematic diagram of image and category knowledge mapping in a common space and similarity calculation.

[0072] wherein step S3 is specifically:

[0073] Map the image features and category semantic features to a common space to align the visual features and semantic features. For semantic features, use Word2Vec to obtain the seen class semantic representation. For sample x and its label y, first extract the feature z of the image x = f(x), then extract the semantic representation a corresponding to the label y, and then map z and a to a common space for similarity calculation. Using cosine similarity, the matching score of sample x and label y is calculated:

[0074] s(x, y) = s(z, a) = cos(f(x), a), s(x, y) is compared with a predefined threshold value adaptively selected by Otsu algorithm, and s(x, y) reflects whether y is the correct class label of x by calculating the similarity between the image and the category semantic, where a larger similarity is considered to be the correct class label.

[0075] To increase more supervision signals, when training the model, negative labels are constructed for the labeled samples. For each training image the real label is The negative label is constructed in two ways: one is to select the most difficult to distinguish label, i.e. the label that is different from the real label but has the highest prediction score The second is to randomly select a label other than the real label and the most difficult to distinguish label The loss function of the model training is as follows:

[0076]

[0077] wherein, and represent the most difficult to distinguish label and the randomly selected label, respectively.

[0078]

[0079]

[0080] For each unlabeled data Assume its prediction probability is Its pseudo label The loss function of unlabeled data is as follows:

[0081]

[0082] Wherein,

[0083] Thus, the matching loss L is constructed ma :

[0084]

[0085] As Figure 4 shown in the active learning framework diagram.

[0086] S4: The threshold value is adaptively selected by the Otsu algorithm, and the matching score is compared with the predefined threshold value. The unlabeled image with a matching score lower than the threshold value is considered as an open set image, and the unlabeled image with a matching score higher than the threshold value is considered as a seen class image.

[0087] Wherein, step S4 is specifically: the threshold value is adaptively selected by the Otsu algorithm. For a batch of samples, first, construct a histogram according to the matching score value in s, and then perform forward cumulative summation and horizontal flip summation after normalization of the score value, to obtain w1 and w2 respectively. Then, the average values of each score in the forward cumulative summation and the horizontal flip summation are calculated, and m1 and m2 are obtained.

[0088]

[0089]

[0090] Then, the variance v is calculated according to the average value:

[0091] v=w1*w2*(m1-m2) 2 ,

[0092] Finally, the variance is maximized, and the score corresponding to the maximum variance is the threshold value t.

[0093]

[0094] S5: Adopt an active learning strategy, select high-confidence open set samples and high-confidence seen class samples, add the high-confidence seen class samples and their calculated pseudo-labels to the labeled data for training, thereby forming a loop and completing the recognition of open set samples and the expansion of the labeled dataset.

[0095] Specifically, step S5 involves employing an active learning strategy, selecting high-confidence open set samples and high-confidence seen class samples, and adding the high-confidence seen class samples and their calculated pseudo-labels to the labeled data for training. This process forms a loop and completes the identification of open set samples. The sample with the highest top-k score is the high-confidence sample. The matching score s for all unlabeled samples... i The samples are sorted from largest to smallest, and the top-k highest-scoring samples and their calculated pseudo-labels are added to the labeled dataset. Samples with lower scores are not added to the labeled dataset. The top-k lowest-scoring samples are considered open set samples. The number k of high-confidence samples selected is determined based on the number of open set samples. After one round of selecting open set samples and seen class samples, the data is input into the entire model for the next iteration, continuously updating the feature extraction network. This continues until all open set data has been identified, and the model's loss function stabilizes.

[0096] S6: Perform class prediction for each unlabeled data point, and construct the classification loss L. ce .

[0097] Specifically, step S6 involves: for each input image x i Feature representation extracted: z i =f(x) i f(·) represents the computation function of the feature extraction network. Adding a linear layer and a softmax layer to the feature extraction network yields a K-dimensional class probability vector p. i :

[0098] p i =g(z) i )=g(f(x i )),

[0099] Where g(·) represents the linear layer and the softmax operation. During the training phase, cross-entropy loss is used to optimize the class probability vector of the labeled image:

[0100]

[0101] in, express The predicted probability.

[0102] like Figure 5An open set image recognition system based on active learning is shown, and the structure schematic diagram specifically comprises:

[0103] The category semantic generation module generates the semantic representation of the category according to the category of the labeled image in the given image data set, and uses the language model Word2Vec to generate the semantic representation of the category;

[0104] The self-supervised representation enhancement module introduces the image rotation prediction framework RotNet, rotates the images in the given image data set by 0°, 90°, 180° and 270° respectively, and performs self-supervised representation enhancement learning to train a strong image feature extraction model;

[0105] The knowledge-guided open set detection module uses the image feature extraction model to represent the features of the image, integrates the category semantic knowledge, maps the image features and the category semantic knowledge to the public space for similarity calculation, and calculates the matching score of the image features and the category semantics using the cosine similarity;

[0106] The category distinguishing module adaptively selects the threshold value through the Otsu algorithm, compares the matching score with the predefined threshold value, and considers the unlabeled image with a matching score lower than the threshold value as an open set image, and considers the unlabeled image with a matching score higher than the threshold value as a seen category image;

[0107] The active learning module selects high-confidence open set samples and high-confidence seen category samples using the active learning strategy, adds the high-confidence seen category samples and the pseudo-labels calculated therefrom to the labeled data for training, and forms a cycle to complete the recognition of the open set samples and the expansion of the labeled data set;

[0108] The image classification module predicts the category of each unlabeled data.

[0109] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall be considered within the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered within the protection scope of the present application.

Claims

1. An open-set image recognition method based on active learning, characterized in that, The method includes the following steps: S1: Given a labeled image dataset and unlabeled datasets The Word2Vec language model is used to generate category semantic representations for known class images. ; S2: Introduce the image rotation prediction framework RotNet to rotate images in a given image dataset. , , and Self-supervised representation reinforcement learning is performed to train the image feature extraction model, and a self-supervised loss is constructed. ; S3: Utilize an image feature extraction model to represent features of all images, incorporating category semantic knowledge. Map image features and category semantic knowledge to a common space for similarity calculation. Use cosine similarity to calculate the matching score between image features and category semantics, and construct a matching loss. ; Image features and category semantic features are mapped to a common space to align visual and semantic features. For semantic features, Word2Vec is used to obtain the semantic representation of the seen class. For images... and its label First, extract the image. Features Next, extract the labels. Corresponding semantic representation Then and The similarity is calculated by mapping the samples to a common space and then using cosine similarity to obtain the sample data. and tags Match score: This is reflected by calculating the magnitude of semantic similarity between images and categories. Is it? The correct category label is determined by the category with high semantic similarity. Randomly select labels other than the true labels and the most difficult-to-distinguish labels. Below is the loss function for model training, and the loss function for labeled data is as follows: in, and These represent the most difficult-to-distinguish labels and randomly selected labels, respectively. For each unlabeled data Assuming its predicted probability is Its pseudo-labels The loss function for unlabeled data is as follows: in, , Therefore, a matching loss is constructed. : S4: The Otsu algorithm adaptively selects a predefined threshold, compares the matching score with the predefined threshold, and considers unlabeled images with matching scores below the threshold as open set images, and unlabeled images with matching scores above the threshold as seen class images; S5: Employ an active learning strategy, selecting high-confidence open set samples and high-confidence known class samples. The highest-scoring sample is the high-confidence sample. The high-confidence seen class samples and their calculated pseudo-labels are added to the labeled data for training. This process is repeated until all open set samples are identified, thereby expanding the labeled dataset. S6: Perform class prediction for each unlabeled data point and construct a classification loss. .

2. The open set image recognition method based on active learning according to claim 1, characterized in that, Step S1 includes: given a labeled sample set and an unlabeled sample set ,in and These represent labeled and unlabeled images, respectively. Indicates labeled samples The true labels, the labeled sample set contains Target categories, namely Define a semantic representation for each category. , Represents category semantic features.

3. The open set image recognition method based on active learning according to claim 1, characterized in that, Step S2 includes: introducing the image rotation prediction framework RotNet, performing self-supervised representation reinforcement learning on all labeled and unlabeled data, and rotating each image separately. , , and Generate four corresponding images and predict the class probabilities for the image features. Features after rotation ,in Predicted probability after rotation : in This represents a linear layer and a softmax function. The self-supervised loss function during training is: 。 4. The open set image recognition method based on active learning according to claim 1, characterized in that, Step S4 includes: the threshold is adaptively selected using the Otsu algorithm; for a certain batch of samples, firstly according to... A histogram is constructed based on the matching scores. After normalizing the scores, forward cumulative summation and horizontal flip summation are performed to obtain the results. and Next, calculate the average of each score in the positive cumulative summation and the horizontal flip summation respectively. and : Then calculate the variance based on the mean. : Finally, the variance is maximized; the score corresponding to the maximum variance is the threshold. : 。 5. The open set image recognition method based on active learning according to claim 1, characterized in that, Step S5 includes: employing an active learning strategy, selecting high-confidence open set samples and high-confidence seen class samples, adding the high-confidence seen class samples and their calculated pseudo-labels to the labeled data for training, thereby forming a loop and completing the identification of open set samples. The highest-scoring sample is the high-confidence sample, and the matching score across all unlabeled samples is [not specified]. Sort by size from largest to smallest, then select The highest-scoring sample and its calculated pseudo-label are added to the labeled dataset. The lowest-scoring sample is considered an open set sample, and the number of high-confidence samples selected is... k The number of open set samples determines the next iteration after selecting open set samples and known class samples in one round, and the feature extraction network is continuously updated until all open set data is identified.

6. The open set image recognition method based on active learning according to claim 1, characterized in that, Step S6 includes: processing each input image Extracted feature representation: , This represents the computation function of the feature extraction network. Adding a linear layer and a softmax layer to the feature extraction network yields... dimensional class probability vector : in, This represents a linear layer and a softmax operation. During the training phase, cross-entropy loss is used to optimize the class probability vector of the labeled image. in, express The predicted probability.

7. An open set image recognition system based on active learning, implementing the open set image recognition method based on active learning as described in any one of claims 1 to 6, characterized in that, Includes the following modules: Category semantic generation module: Based on the category corresponding to the labeled image in the given image dataset, the module uses the language model Word2Vec to generate the semantic representation of the category; Self-supervised representation enhancement module: Introduces the RotNet image rotation prediction framework to rotate images in a given image dataset. , , and Self-supervised representation reinforcement learning is used to train the image feature extraction model; The knowledge-guided open set detection module uses an image feature extraction model to represent images, incorporates category semantic knowledge, maps image features and category semantic knowledge to a common space for similarity calculation, and uses cosine similarity to calculate the matching score between image features and category semantics. Category differentiation module: The Otsu algorithm adaptively selects a threshold and compares the matching score with a predefined threshold. Unlabeled images with a matching score below the threshold are considered open set images, and unlabeled images with a matching score above the threshold are considered seen class images. Active learning module: Adopting an active learning strategy, it selects high-confidence open set samples and high-confidence seen class samples, adds the high-confidence seen class samples and their calculated pseudo-labels to the labeled data for training, thereby forming a loop and completing the recognition of open set samples and the expansion of the labeled dataset; Image classification module: predicts the category for each unlabeled data point.

Images