An image classification method based on active learning

Abstract

The application discloses an image classification method based on active learning, which mainly comprises the following steps: obtaining a prototype vector of a labeled image; calculating a rejection confidence value between the prototype vector of the labeled image and that of an unlabeled image; calculating an uncertainty value of the unlabeled image; calculating a diversity value of the unlabeled image; performing weighted summation on the rejection confidence value, the uncertainty value and the diversity value to obtain an evaluation scalar of the unlabeled image; selecting a plurality of unlabeled images from an unlabeled image set according to the evaluation scalar to perform labeling, thereby forming new labeled images; and adding the new labeled images to an initial labeled image set to train a classifier. The image classification method selects images with relatively large information amounts to perform labeling and trains a model through a hybrid query strategy formed by three performance indexes of a rejection confidence value between images, uncertainty and diversity of the images, thereby avoiding excessive prediction of the model on out-of-distribution images and improving the classification precision of the classification model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image classification method based on active learning. Background Technology

[0002] Image classification is a core task in computer vision, aiming to distinguish different categories of images based on the different features reflected in the image information. Currently, most active learning methods based on the Softmax classifier train the Softmax classifier using a small set of labeled images, which can generalize well to the complete image set. However, the Softmax-based classifier tends to identify images close to the classification boundary and cannot identify out-of-distribution (OOD) images. Furthermore, the network is prone to overconfident predictions about OOD images, which may cause some valuable OOD images to be missed.

[0003] Existing technologies use classifiers as a foundation, employing rejection confidence and confusion confidence to select images. Rejection confidence selects the image furthest from the labeled image, while confusion confidence selects images from the most chaotic regions. Both methods alleviate the problem of active learning methods based on Softmax classifiers failing to recognize out-of-detail (OOD) images. However, these query strategies primarily consider the distance between unlabeled and labeled images, neglecting image distribution, which may lead to redundancy in the selected images.

[0004] Therefore, in order to address the technical problem of existing technologies being unable to identify OOD images and the redundancy in the selected images, a new classification method needs to be proposed. Summary of the Invention

[0005] The purpose of this invention is to provide an image classification method based on active learning to solve the problem of over-predicting images outside the distribution in existing classification methods.

[0006] The present invention adopts the following technical solution:

[0007] Embodiment 1 of the present invention provides an image classification method based on active learning. The training process of the classifier used in this image classification method includes:

[0008] Obtain the prototype vector of the labeled image;

[0009] Calculate the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image;

[0010] Calculate the uncertainty value of the unlabeled image;

[0011] Calculate the diversity value of unlabeled images;

[0012] The evaluation scalar of the unlabeled image is obtained by weighting and summing the rejection confidence value, uncertainty value, and diversity value.

[0013] Based on the evaluation scalar, multiple unlabeled images are selected from the set of unlabeled images and labeled to form new labeled images;

[0014] The classifier is trained after adding new labeled images to the initial labeled image set.

[0015] Optionally, obtaining the prototype vector of the labeled image includes:

[0016] Multiple images in the unlabeled image set are selected and labeled to obtain an initial labeled image set;

[0017] Obtain the feature map of each labeled image in the initial labeled image set;

[0018] The labeled image is processed by a classifier to obtain the prototype vector of the labeled image.

[0019] Optionally, calculating the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image includes:

[0020] Obtain the unlabeled image feature map of the unlabeled image;

[0021] Calculate the rejection confidence value between the prototype vectors of the unlabeled and labeled images based on the feature maps of the unlabeled images.

[0022] Optionally, the rejection confidence score can be calculated in the following ways:

[0023]

[0024] in, To obtain the classification probability from the prototype vector of the labeled image, C represents the total number of image categories, and x represents the feature map of the unlabeled image. i This represents the feature map of the i-th unlabeled image. This represents the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image.

[0025] Optionally, the uncertainty value of an unlabeled image can be calculated in the following ways:

[0026] F(x i )=|s margin -f(x i )|,

[0027] Where, f(x) i ) represents the output value of the graph convolutional network after processing the unlabeled image, s margin F(x) is a preset constant. i) represents the uncertainty value for the unlabeled image.

[0028] Optionally, calculating the diversity value of unlabeled images includes:

[0029] Filter out several new unlabeled images from the set of unlabeled images;

[0030] Calculate the mutual information between each new unlabeled image and the unlabeled image;

[0031] The redundancy of unlabeled images is calculated based on mutual information.

[0032] The diversity value of unlabeled images is calculated based on redundancy.

[0033] Optionally, redundancy can be calculated in the following ways:

[0034]

[0035] in, x represents the sum of similarity values ​​between all new unlabeled images and unlabeled images; i Let y represent the feature map of the i-th unlabeled image, and let Red(x) represent the new unlabeled image. i ,y) represents the redundancy of the unlabeled image.

[0036] Optionally, the diversity value can be calculated in the following ways:

[0037]

[0038] Among them, Red(x) i，y ) represents the unlabeled image x i Redundancy, Div(x) i ) represents the unlabeled image x i diversity value, This is a set of unlabeled images.

[0039] Embodiment 2 of the present invention provides an image classification method apparatus based on active learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an image classification method based on active learning as described in any of the above embodiments.

[0040] The beneficial effects of this invention are: by using a hybrid query strategy composed of three performance indicators—rejection confidence, uncertainty, and image diversity—to select images with relatively large information content for labeling and training the model, the model avoids over-predicting images outside the distribution and improves the classification accuracy of the classification model. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the steps of an image classification method based on active learning, provided in Embodiment 1 of the present invention.

[0042] Figure 2 This is a schematic diagram of the active learning process of a nearest neighbor embedded network classifier provided in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the active learning process of a nearest neighbor embedded network classifier provided in Embodiment 1 of the present invention;

[0044] Figure 4 This is a schematic diagram of a graph convolutional network provided in Embodiment 1 of the present invention;

[0045] Figure 5 This is a schematic diagram illustrating the simulation verification results of an image classification method based on active learning, provided in Embodiment 1 of the present invention.

[0046] Figure 6 This is a schematic diagram of an image classification method device based on active learning provided in Embodiment 2 of the present invention. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0048] Embodiment 1 of the present invention provides an image classification method based on active learning. The training process of the classifier used in this image classification method is as follows: Figure 1 As shown, the implementation steps of this method include:

[0049] Step 101: Obtain the prototype vector of the labeled image;

[0050] Optionally, obtaining the prototype vector of the labeled image includes:

[0051] Multiple images in the unlabeled image set are selected and labeled to obtain an initial labeled image set;

[0052] Obtain the feature map of each labeled image in the initial labeled image set;

[0053] The feature map of the labeled image is processed by a classifier to obtain the prototype vector of the labeled image.

[0054] In one embodiment, an initial label set is first selected from the unlabeled image set, and the image category of each image sample in the initial label set is manually labeled. For example, a sky image is labeled as sky, a land image as land, and an ocean image as ocean. The initial label image set is obtained and used as input to a convolutional neural network. Feature extraction is performed to obtain the label image feature map of each labeled image. The label image feature map is then classified by a classifier. The classification result contains the prototype vectors of the labeled images of different classes. It should be noted that a preferred classifier type used in this invention is a nearest neighbor embedded network classifier. Prototype vectors refer to the vectors obtained during the training of a classifier using labeled samples. For example, if a classifier is trained using 100 labeled sample images, with 40 labeled as kittens and 60 as puppies, then these 100 labeled samples are used as the input to the classifier, and the output prototype vectors are (cat, dog). The prototype vector for kittens is (1, 0), and the prototype vector for puppies is (0, 1). In one training process, each category can obtain n prototype vectors, and each prototype vector can represent the sample features of that category.

[0055] Step 102: Calculate the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image;

[0056] Optionally, calculating the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image includes:

[0057] Obtain the unlabeled image feature map of the unlabeled image;

[0058] Calculate the rejection confidence value between the prototype vectors of the unlabeled and labeled images based on the feature maps of the unlabeled images.

[0059] In one embodiment, during the initial training of the classifier, in addition to using manually selected labeled images as input to the model, the remaining unlabeled images are also used as input to the model. The unlabeled image feature maps of the unlabeled images are extracted by a convolutional neural network, and the rejection confidence value between the prototype vector of each unlabeled image and the labeled image is calculated based on the unlabeled image feature maps. That is, the distance between the prototype vector of each unlabeled image and the prototype vector of the labeled image is measured. In this way, the image with the closest Euclidean distance to the prototype vector of the labeled image is selected for training the classifier, rather than selecting images that are only located on the classification edge. This avoids the problems of failing to identify images outside the distribution and over-predicting images outside the distribution.

[0060] Step 103: Calculate the uncertainty value of the unlabeled image;

[0061] In one embodiment, it should be noted that the distribution of unlabeled images cannot be determined before training the classifier. By repeatedly training the classifier with images rich in information from the unlabeled image set, the classification accuracy is improved. Therefore, in addition to the rejection confidence level, the distribution of each unlabeled image and the redundancy between images are further considered when selecting images. Specifically, in the feature extraction stage, the model uses a mixture of labeled and unlabeled image samples as input to the feature extraction network. After passing through the feature extraction network, a graph convolutional network is used to identify the labeled sample image features output by the feature extraction network. At this point, the graph convolutional network acts as a binary classifier, outputting the classification probability of the unlabeled images. For example, the graph convolutional network identifies the unlabeled sample images from the labeled sample image features and the unlabeled sample images, and the probability of the unlabeled sample images is 1 / 3. That is, based on this classification probability, the uncertainty value of each unlabeled image is calculated to be 1 / 3. It should be noted that by considering the uncertainty of unlabeled samples, unlabeled sample images with large uncertainty values ​​are selected as sample images that differ significantly from the labeled samples for labeling, thereby improving the richness of the sample images.

[0062] Step 104: Calculate the diversity value of the unlabeled images;

[0063] Optionally, calculating the diversity value of unlabeled images includes:

[0064] Several new unlabeled images are selected from the set of unlabeled images;

[0065] Calculate the mutual information between each new unlabeled image and the unlabeled image;

[0066] The redundancy of unlabeled images is calculated based on mutual information.

[0067] The diversity value of unlabeled images is calculated based on redundancy.

[0068] In one embodiment, redundancy between images is also considered during the image selection process. The current unlabeled image is selected from the image set. To avoid excessive computation, several new unlabeled images with the closest Euclidean distance to the current unlabeled image are selected from the unlabeled image set. The mutual information between the current unlabeled image and each new unlabeled image is calculated. Based on the mutual information, the redundancy of each unlabeled image is calculated, and its diversity is further calculated based on the redundancy of each unlabeled image, thereby avoiding images with high redundancy.

[0069] Step 105: The rejection confidence value, uncertainty value, and diversity value are weighted and summed to obtain the evaluation scalar of the unlabeled image;

[0070] Step 106: Based on the evaluation scalar, select multiple unlabeled images from the unlabeled image set and label them to form new labeled images;

[0071] Step 107: After adding the new labeled images to the initial labeled image set, train the classifier.

[0072] In one embodiment, the outputs of the rejection confidence value calculation module, the uncertainty value calculation module, and the diversity value calculation module are aggregated into a scalar as a criterion for image selection. Specifically, the rejection confidence value, uncertainty value, and diversity value are weighted and summed to obtain an evaluation scalar for unlabeled images. Based on the magnitude of this evaluation scalar, several images with larger aggregated values ​​are selected for labeling, generating new labeled images. The labeled images are added to the initial labeled image set to form a new labeled image set. The classifier is then retrained using this new labeled image set. This avoids the problem that traditional neural networks using only a Softmax classifier cannot identify out-of-distribution (OOD) images, and that the network is prone to making overconfident predictions about OOD images. Furthermore, considering the distribution of images avoids image redundancy, improves image richness, and prevents the loss of valuable out-of-distribution images.

[0073] The specific implementation of the active learning-based image classification method provided in this embodiment is as follows: Figure 2 and Figure 3 As shown.

[0074] The image dimensions are H′ rows × W′ columns, with C′ channels. Taking the CIFAR10 dataset as an example, the CIFAR10 training set contains 5000 images, and the test set contains 1000 images. The image dimensions are 32 rows × 32 columns, with 3 channels. D n U represents the set of labeled images learned in the nth round of active learning. n Let U represent the set of unlabeled images used in the nth round of active learning. The training set for the CIFAR10 dataset is defined as the initial set of unlabeled images, U0.

[0075] Step 1: Select the labeled image set D0:

[0076] Untagged image set U n 1200 images were randomly selected and manually labeled with their categories. The labeled images formed the initial labeled image set D0, which served as the training set for the first round of active learning.

[0077] Step 2, Feature Extraction:

[0078] Using convolutional neural networks for feature extraction, the labeled image set D n and unlabeled image set U nThe input is fed into a convolutional neural network for feature extraction. The feature extraction process can be represented by the following formula:

[0079]

[0080] Where X is the input image, X∈R H′×W′×C′ F is a convolutional neural network. For the extracted feature map, H represents the feature map. The row, W represents the feature map. The column C represents the feature map. The number of channels.

[0081] It should be noted that this example uses the ResNet18 network as the feature extractor for feature extraction, such as... Figure 4 As shown. The labeled image set D is obtained through the above formula (1). n Inputting ResNet18 for feature extraction yields the labeled image feature map for each labeled image; the unlabeled image set U... n Inputting ResNet18 for feature extraction yields the unlabeled image feature map for each unlabeled image.

[0082] Step 3: Initial training of the classifier:

[0083] The labeled image set D obtained by feature extraction n The feature maps of the labeled images are input into the classifier for training. Taking the nearest neighbor embedded network classifier as an example, the prototype vectors m of n labeled images are learned using the labeled images. c，n The classification probability of the prototype vector of the labeled image is calculated as follows:

[0084]

[0085] Where, f(x) i ) represents the feature vector of the labeled image, m c，n is the prototype vector of the nth c-th labeled image, where C represents the total number of image categories, and d(·) represents the calculation of the Euclidean distance. The classification probability of the prototype vector of the labeled image is given by δ(), which represents a non-linear activation function that projects the Euclidean distance into [0, 1].

[0086] During the training process, m c，n Random initialization is performed and the network parameters are jointly updated using gradient descent.

[0087] The loss function of this classifier is the binary cross-entropy, expressed as:

[0088]

[0089] Among them, y x，c This indicates that it belongs to the same set as the currently unlabeled image x. The probability distribution of other unlabeled images, f(x) i ) represents the output value of the graph convolutional network after processing the unlabeled image.

[0090] Step 4: Select an image:

[0091] Using query strategies to unlabeled image set U n To select a batch of images with the richest information for manual labeling, this invention proposes a hybrid query strategy. This strategy consists of three parts: rejection confidence, graph convolutional network, and diversity based on mutual information. Specifically, as follows... Figure 3 As shown. First, the rejection confidence score for the unlabeled image is calculated. The rejection confidence score measures the distance between the unlabeled image and the labeled image. The methods for calculating the rejection confidence score include:

[0092]

[0093] Where, p c (f x Let be the classification probability of the prototype vector of the labeled image, C represent the total number of image categories, and x represent the feature map of the unlabeled image. This represents the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image.

[0094] Simultaneously, the uncertainty of unlabeled images is calculated using a graph convolutional network (Graph Convolutional Network), where a two-layer graph convolutional network is used after the feature extractor to distinguish between labeled and unlabeled images. The loss function of the graph convolutional network can be expressed as:

[0095]

[0096] Where, f(x) i ) represents the output of the graph convolutional network, and λ is used as a weighting factor between the cross-entropy of labeled and unlabeled images. Let N represent the loss function of the graph convolutional network. l This represents the number of labeled images, where N represents the total number of sample images.

[0097] After training the graph convolutional network using labeled and unlabeled images, all unlabeled images are fed into the graph convolutional network to calculate uncertainty. The uncertainty of unlabeled images is calculated as follows:

[0098] F(x i )=|s margin -f(x i (6)

[0099] Where, f(x)i ) represents the output value of the graph convolutional network after processing the unlabeled image, s margin F(x) is a preset constant. i ) represents the uncertainty value of the unlabeled image.

[0100] Simultaneously, the diversity of unlabeled images is calculated. This diversity is measured by the redundancy between images. The calculation method for the diversity value of unlabeled images is as follows:

[0101]

[0102] Among them, Red(x) i Let , y) represent the redundancy of unlabeled images. Redundancy is measured by similarity, where u is a set of unlabeled images. Redundancy is calculated as follows:

[0103]

[0104] Where, sim(x) i (y) represents the currently unlabeled image x. i Similarity between y and other unlabeled images For a set of labeled images, x represents the sum of similarity values ​​between all new unlabeled images and unlabeled images; i Red(x) represents the feature map of the unlabeled image, y represents the new unlabeled image, and Red(x) represents the feature map of the unlabeled image. i ,y) represents the redundancy of the unlabeled image.

[0105] The similarity between images is measured by mutual information, which is calculated as follows:

[0106]

[0107] Where, sim(x) i y) represents the set Similarity of unlabeled images, p(x) i p(y) represents the probability that the image is labeled as belonging to the i-th class, C represents the total number of image classes, and p(y) represents the probability that the image is labeled as belonging to the i-th class. j p(x) represents the probability of an unlabeled image. i y j ) represents the joint entropy between labeled and unlabeled images.

[0108] Because when calculating the mutual information of unlabeled images, we take... The computational cost of handling all images within a given timeframe is too high, therefore... The system selects the 30 images closest to the currently unlabeled image, using Euclidean distance as the distance metric.

[0109] Finally, the outputs of these three modules are aggregated into a scalar, which serves as the criterion for selecting images. The image with the largest aggregated value is then labeled. The labeled images are added to the labeled image set D. n A new set of labeled images D is formed in the middle. n+1 Formula (10) embodies this process.

[0110]

[0111] Specifically, the query strategy used in this example employs three parts to select images: rejection confidence, calculation of uncertainty of unlabeled images using graph convolutional networks, and calculation of diversity of unlabeled images based on mutual information. First, the rejection confidence, uncertainty, and diversity index based on mutual information of unlabeled images are calculated using formulas (4), (6), and (7), respectively; then, the calculation results of formula (10) are used to select the top K images for labeling. The labeled images are then added to the labeled image set D. n A new set of labeled images D is formed in the middle. n+1 .

[0112] Step 5: Update the model:

[0113] Repeat steps two and three to update the classification model and complete a new round of active learning.

[0114] Step 6: Repeat steps 4 and 5 until the expected goal is achieved or all images are labeled, then stop repeating to obtain the results of the final round of active learning.

[0115] Based on steps one through five above, the present invention also provides an experimental simulation result, such as... Figure 5 As shown. From Figure 5As can be seen, the top curve in the figure adopts the active learning-based image classification method of this invention. Compared with other active learning methods, the method in this example achieves the best classification accuracy in most cases. This demonstrates the effectiveness of the method of this invention. When trained with 6600 labeled images, the method of this invention achieves an accuracy of 87.92% on the CIFAR10 dataset, which is 2.45% higher than the recent ADS method; 4.28% higher than the classic CoreSet; 1.8% higher than LLoss; 6.93% higher than VAAL; and 6.46% higher than Random. This is because the rejection confidence used in this invention is measured by calculating the distance between the prototype vectors of unlabeled and labeled images, and by combining graph convolutional neural networks to calculate the difference between labeled and unlabeled images and the difference between unlabeled images and surrounding images based on the diversity of mutual information. Combining these three aspects can filter out images that are highly different from labeled images, far away, and rich in diversity, avoiding the classifier's inability to identify out-of-distribution samples and improving the classifier's recognition rate of samples.

[0116] Embodiment 2 of the present invention provides an image classification method device 600 based on active learning, such as... Figure 6 As shown, it includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. When the processor executes the computer program, it implements an image classification method based on active learning as described in any of the above embodiments.

[0117] It should be noted that the active learning-based image classification method implemented by this device is the same as the method in Embodiment 1 above, and will not be described again here.

Claims

1. An image classification method based on active learning, characterized in that, The training process of the classifier used in the image classification method includes: Obtain the prototype vector of the labeled image; Calculate the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image; Calculate the uncertainty value of the unlabeled image; Calculate the diversity value of the unlabeled image; The rejection confidence value, uncertainty value, and diversity value are weighted and summed to obtain the evaluation scalar of the unlabeled image; Based on the evaluation scalar, multiple unlabeled images are selected from the unlabeled image set and labeled to form new labeled images; The classifier is trained after the new labeled image is added to the initial labeled image set; Obtaining the prototype vector of the labeled image includes: Multiple images are selected from the unlabeled image set and labeled to obtain the initial labeled image set; Obtain the feature map of each labeled image in the initial labeled image set; The labeled image is processed by a classifier to obtain the prototype vector of the labeled image; Calculating the rejection confidence value between the prototype vector of each unlabeled image and the labeled image includes: Obtain the unlabeled image feature map of the unlabeled image; Calculate the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image based on the feature map of the unlabeled image; Calculating the diversity value of the unlabeled image includes: Several new unlabeled images are selected from the set of unlabeled images; Calculate the mutual information between each new unlabeled image and the existing unlabeled images; The redundancy of the unlabeled image is calculated based on the mutual information. The diversity value of the unlabeled image is calculated based on the redundancy. The redundancy is calculated in the following ways: ， in, This represents the sum of similarity values ​​between all the new unlabeled images and the unlabeled images; Indicates the first An unlabeled image feature map, This refers to the new unlabeled image. This indicates the redundancy of the unlabeled image; The calculation method for the diversity value includes: ， in, The maximum redundancy of the unlabeled image. This represents the diversity value of the unlabeled image. For an unlabeled set of images; The calculation method for the rejection confidence value includes: ， in, The classification probability is obtained from the prototype vector of the labeled image, where C represents the total number of image categories. This represents the feature map of an unlabeled image. Indicates the first An unlabeled image feature map, This represents the rejection confidence value between the prototype vectors of the unlabeled image and the labeled image; The uncertainty value of the unlabeled image is calculated using the following methods: ， in, The output value of the graph convolutional network after processing the unlabeled image is shown. As a preset constant, The uncertainty value of the unlabeled image.

2. The apparatus for an image classification method based on active learning as described in claim 1, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements an image classification method based on active learning as described in claim 1.

Images