Image classification-oriented diversified query active learning method and device

Abstract

This invention provides a diversified query active learning method and apparatus for image classification. The method includes: using an auxiliary classification network to generate high-confidence pseudo-labels for the main classification network, and removing redundant samples with similar representations to labeled data from unlabeled data; in each active learning loop, the main classification network and the auxiliary classification network exchange their respective high-confidence pseudo-labeled samples in a collaborative training manner, while adding unlabeled samples predicted by the main classification network and the auxiliary classification network as having different labels to a candidate pool for active learning; using auxiliary learning and multi-level diversity screening strategies to perform human expert annotation on more diverse samples; and using a small number of labeled samples to actively learn and train the classification model to achieve training results close to those using a large number of labeled samples.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a diversified query active learning method and apparatus for image classification. Background Technology

[0002] Deep learning models achieve high classification accuracy in image classification tasks by learning robust and expressive features from a large number of labeled image samples. However, the manual and time costs of labeling a large number of images are very high. In many practical applications, the number of available labeled images is very limited. Therefore, improving classification performance with limited labeled data has become a major challenge in the field of machine learning. Active learning is a very popular technique that can train deep models with strong generalization ability using limited labeled data obtained by iteratively filtering sample images and having them labeled by experts. The sample query strategy is the core of active learning methods. If the query function can select the sample with the most information in each active learning loop, then we can significantly improve the generalization ability of the classification model with a small number of labeled samples, thereby reducing the cost of manual labeling. The query strategy in active learning is mainly designed based on two criteria: uncertainty and representativeness. The uncertainty criterion mainly indicates whether the prediction of a sample has high confidence. Many active learning methods are designed based on this criterion. These methods select the sample with the most information and the lowest classification confidence in each loop, such as marginal sampling and entropy sampling. Although the uncertainty criterion is widely used in active learning, it cannot effectively improve classification performance. For example, if a batch of samples is selected in each iteration, there may be redundancy among samples with similar uncertainties; however, if only the most uncertain samples are selected in each iteration, the number of active learning iterations will increase significantly, leading to unacceptable time costs. To more comprehensively explore the structural information in unlabeled data, a representative method of selecting independent and identically distributed samples in the exponential search space has been proposed. However, on large-scale datasets, the computational cost of selecting a batch of independent and identically distributed samples increases significantly. Therefore, designing an efficient active learning strategy that can improve the generalization ability of deep models and reduce the cost of manual annotation is a problem of significant practical importance. Summary of the Invention

[0003] To address at least one technical problem in the existing technology, the present invention provides a diversified query active learning method and apparatus for image classification.

[0004] In a first aspect, the present invention provides a diversified query active learning method for image classification, the diversified query active learning method for image classification comprising:

[0005] Step 1: Obtain an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

[0006] Step 2: Using the labeled image set L as the training set, train the main classification model f1 and the auxiliary classification model f2 respectively to obtain f1' and f2', and use f1' and f2' to extract features and predict all unlabeled images in the unlabeled image set U;

[0007] Step 3: Based on the prediction results of f1' and f2' for all unlabeled images in the unlabeled image set, obtain image set S consisting of unlabeled images predicted as different labels by f1' and f2' and image set H consisting of unlabeled images predicted as the same label by f1' and f2'.

[0008] Step 4: Cluster and sort the unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled.

[0009] Step 5: Label the unlabeled sample candidate set based on the labeling operation, and remove the labeled samples from the unlabeled image set U and add them to the labeled image set L;

[0010] Step 6: Sort the images in the image set H according to the prediction confidence of f1' and f2' respectively, select the sample sets H1 and H2 with the highest confidence, and assign pseudo labels output by f1' and f2' to the samples in the set H1' and H2' respectively.

[0011] Step 7: Train f1′ using sample set H2′ and labeled image set L to obtain f1″; Train f2′ using sample set H1′ and labeled image set L to obtain f2″.

[0012] Step 8: Using f1” and f2” as the main classification model f1 and the auxiliary classification model f2, return to step 2 until the main classification model f1 reaches the expected classification performance or the number of labeled samples exceeds the set upper limit, then stop iterating and proceed to step 9.

[0013] Step 9: Use the main classification model f1 to perform image classification prediction task.

[0014] Secondly, the present invention also provides a diversified query active learning device for image classification, the diversified query active learning device for image classification comprising:

[0015] The acquisition module is used to acquire an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

[0016] The first training module is used to train the main classification model f1 and the auxiliary classification model f2 respectively using the labeled image set L as the training set to obtain f1' and f2', and to use f1' and f2' to extract features and predict all unlabeled images in the unlabeled image set U.

[0017] The first building module is used to obtain an image set S consisting of unlabeled images predicted by f1' and f2' to all unlabeled images in the unlabeled image set, and an image set H consisting of unlabeled images predicted by f1' and f2' to be the same;

[0018] The second building module is used to cluster and sort unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled.

[0019] The annotation module is used to annotate the candidate set of unlabeled samples based on the annotation operation, and remove the annotated samples from the unlabeled image set U and add them to the labeled image set L;

[0020] The third construction module is used to sort the images in the image set H according to the prediction confidence of f1' and f2' respectively, select the sample sets H1 and H2 with the highest confidence respectively, and assign pseudo labels output by f1' and f2' to the samples respectively to obtain sample sets H1' and H2';

[0021] The second training module is used to train f1' using the sample set H2' and the labeled image set L to obtain f1”, and to train f2′ using the sample set H1' and the labeled image set L to obtain f2”.

[0022] The loop module is used to return to the steps executed by the first training module with f1” and f2” as the main classification model f1 and the auxiliary classification model f2, and stop iterating until the main classification model f1 reaches the expected classification performance or the number of labeled samples exceeds the set upper limit.

[0023] The prediction module is used to perform image classification prediction tasks using the main classification model f1.

[0024] In this invention, the following steps are performed:

[0025] Step 1: Obtain an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

[0026] Step 2: Using the labeled image set L as the training set, train the main classification model f1 and the auxiliary classification model f2 respectively to obtain f1' and f2', and use f1' and f2' to extract features and predict all unlabeled images in the unlabeled image set U;

[0027] Step 3: Based on the prediction results of f1' and f2' for all unlabeled images in the unlabeled image set, obtain image set S consisting of unlabeled images predicted as different labels by f1' and f2' and image set H consisting of unlabeled images predicted as the same label by f1' and f2'.

[0028] Step 4: Cluster and sort the unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled.

[0029] Step 5: Label the unlabeled sample candidate set based on the labeling operation, and remove the labeled samples from the unlabeled image set U and add them to the labeled image set L;

[0030] Step 6: Sort the images in the image set H according to the prediction confidence of f1' and f2' respectively, select the sample sets H1 and H2 with the highest confidence, and assign pseudo labels output by f1' and f2' to the samples in the set H1' and H2' respectively to obtain sample sets H1' and H2'.

[0031] Step 7: Train f1′ using sample set H2′ and labeled image set L to obtain f1″; Train f2′ using sample set H1′ and labeled image set L to obtain f2″.

[0032] Step 8: Using f1” and f2” as the main classification model f1 and the auxiliary classification model f2, return to step 2 until the main classification model f1 reaches the expected classification performance or the number of labeled samples exceeds the set upper limit, then stop iterating and proceed to step 9.

[0033] Step 9: Use the main classification model f1 to perform image classification prediction task.

[0034] This invention reduces the cost of manual annotation significantly. By using a small number of labeled samples to actively learn and iteratively train the classification model, it can achieve training results that are close to those achieved with a large number of labeled samples. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an embodiment of the diversified query active learning method for image classification according to the present invention.

[0036] Figure 2This is a schematic diagram of the functional modules of an embodiment of the active learning device for diversified queries for image classification according to the present invention.

[0037] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0038] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0039] In a first aspect, embodiments of the present invention provide a diversified query active learning method for image classification.

[0040] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the active learning method for diversified queries in image classification according to the present invention. Figure 1 As shown, active learning methods for diverse queries in image classification include:

[0041] Step 1: Obtain an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

[0042] In this embodiment, the first quantity is much larger than the second quantity, that is, the unlabeled image set U includes a large number of unlabeled images, and the labeled image set L includes a small number of labeled images.

[0043] Step 2: Using the labeled image set L as the training set, train the main classification model f1 and the auxiliary classification model f2 respectively to obtain f1′ and f2′, and use f1′ and f2′ to extract features and predict all unlabeled images in the unlabeled image set U.

[0044] In this embodiment, the labeled image set L is used as the training set to train the main classification model f1 and the auxiliary classification model f2 respectively, thereby obtaining f1′ and f2′.

[0045] Further, in one embodiment, the step of using the labeled image set L as the training set to train the main classification model f1 and the auxiliary classification model f2 respectively to obtain f1′ and f2′ includes:

[0046] For the i-th labeled image x in the labeled image set L i , to obtain x i The classification probabilities of being classified into class j by f1 and f2 respectively:

[0047]

[0048]

[0049] Where W1 and W2 are the parameters of f1 and f2, respectively; x is obtained based on f1 i The classification probability of belonging to class j. x is obtained based on f2 i The probability of belonging to class j;

[0050] By minimizing the cross-entropy loss of the main classification model f1 and the auxiliary classification model f2 on the labeled image set L. and The main classification model f1 and the auxiliary classification model f2 are trained to obtain f1′ and f2′, respectively. For labeled images in the labeled image set L, the cross-entropy losses on the main classification model f1 and the auxiliary classification model f2 are as follows:

[0051]

[0052]

[0053] in, and f1 and f2 are the objective functions for training the main classification model f1 and the auxiliary classification model f2, respectively; C is the total number of categories contained in the labeled image set L; l is the number of labeled images contained in the labeled image set L; y i It is the i-th labeled image x i The true label; j is x i Predicted labels; x is obtained based on the main classification model i The predicted probability of belonging to class j. x is obtained based on the auxiliary classification model i The predicted probability of belonging to class j; I{y i =j} is an indicator function, if y i If it equals j, then it is 1; otherwise, it is 0.

[0054] After obtaining f1′ and f2′, feature extraction and prediction are performed on all unlabeled images in the unlabeled image set U using f1′ and f2′. That is, all unlabeled images in the unlabeled image set U are sequentially input into f1′ and f2′ to obtain the prediction results of f1′ and f2′ for each unlabeled image, as well as the feature extraction results of f1′ for each unlabeled image.

[0055] Furthermore, in one embodiment, the step of extracting and predicting features for all unlabeled images in the unlabeled image set U using f1′ and f2′ includes:

[0056] For the i-th unlabeled image z in the unlabeled image set Ui , to obtain z i The classification probabilities of being classified into class j by f1′ and f2′ respectively:

[0057]

[0058]

[0059] Where W1′ and W2′ are the parameters of f1′ and f2′, respectively; z is obtained based on f1′ i The classification probability of belonging to class j. z is obtained based on f2′ i The probability of belonging to class j;

[0060] Depend on:

[0061]

[0062]

[0063] We obtain f1′ and f2′ respectively with respect to z i Predicted pseudo-labels and

[0064] The output of the last pooling layer f1′ is used to obtain the z-axis. i The extracted feature vector.

[0065] Step 3: Based on the prediction results of f1′ and f2′ for all unlabeled images in the unlabeled image set, obtain image set S consisting of unlabeled images predicted as having different labels by f1′ and f2′ and image set H consisting of unlabeled images predicted as having the same label by f1′ and f2′.

[0066] Step 4: Cluster and sort the unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled.

[0067] In this embodiment, based on the feature vector corresponding to each unlabeled image in the image set S, the k-means clustering algorithm is used to divide the image set S into C clusters, represented as U = U1∪U2∪...∪U C For the aforementioned C clusters, each cluster is further divided into several sub-clusters based on the number of class labels predicted by f1′. This iterative clustering operation is continued until each smallest sub-cluster contains only one class label. The clustering operation is then stopped, resulting in m smallest sub-clusters, denoted as... According to the BvSB criterion, the smallest subcluster with the highest uncertainty is selected, resulting in a query dataset F with m samples. The BvSB criterion is expressed as:

[0068]

[0069]

[0070]

[0071] For each minimal subcluster, This refers to the probability that a sample within a cluster is classified as class j by f1′; r max1 This refers to the class index where f1′ has the highest probability of classifying the sample, r max2 This refers to the index of the class with the second highest classification probability for the sample, f1′. Represents the maximum predicted probability of the sample. With the second largest predicted probability The difference between them, for each smallest subcluster, select p. diff The smallest sample forms the query dataset F, p diff The smallest sample is the one with the highest uncertainty.

[0072] If t samples in F have the same pseudo-label k, then the threshold for selecting different samples with the same pseudo-label k is expressed as:

[0073]

[0074] Where Var(·) is the variance operator, and β is a hyperparameter used to adjust the threshold. f1′ represents the classification probability output by f1′ for samples in F that have the same pseudo-label k. f1′ represents the classification probability output by f1′ for samples in F that have the same pseudo-label k.

[0075] Order F according to the uncertainty of the samples, and define it as follows: The uncertainty of the sample comes from Decrease to From F u Samples are selected sequentially and added to the unlabeled sample candidate set Q, where, It is F u If the j-th sample in Q is predicted to be of class k, then if there exists a t sample with pseudo-label k in Q. k For each sample, calculate With t k The similarity of the samples is:

[0076]

[0077] If t k If it is 0, then Add to Q; otherwise, u k When greater than v k At that time, Add to Q; repeat the above filtering process until there are h samples in Q, where h is a hyperparameter controlling the batch size of the active learning filtering. f1′ represents the classification probability output by f1′ for a sample in Q with pseudo-label k.

[0078] Step 5: Label the unlabeled sample candidate set based on the labeling operation, and remove the labeled samples from the unlabeled image set U and add them to the labeled image set L;

[0079] In this embodiment, human experts specifically label the samples in the unlabeled sample candidate set. Based on the expert labeling operation, a label is added to each sample in the unlabeled sample candidate set. Then, the labeled samples are removed from the unlabeled image set U and added to the labeled image set L.

[0080] Step 6: Sort the images in the image set H according to the prediction confidence of f1′ and f2′ respectively, select the sample sets H1 and H2 with the highest confidence respectively, and assign pseudo labels output by f1′ and f2′ to the samples in them respectively to obtain sample sets H1′ and H2′.

[0081] In this embodiment, the images in the image set H are sorted according to the prediction confidence of f1′ and f2′ respectively, resulting in two sequences. The highest confidence samples are selected from sequence 1 to form sample set H1; similarly, the highest confidence samples are selected from sequence 2 to form sample set H2.

[0082] Assign pseudo-labels to each sample in H1 using the output of f1′ to obtain sample set H1′; similarly, assign pseudo-labels to each sample in H2 using the output of f2′ to obtain sample set H2′.

[0083] Further, in one embodiment, the step of sorting the images in the image set H according to the prediction confidence of f1′ and f2′ respectively, and selecting the sample sets H1 and H2 with the highest confidence respectively includes:

[0084] The images in the image set H are sorted according to the prediction confidence of f1′, and the top l×α images with the highest confidence are selected to form the sample set H1;

[0085] The images in the image set H are sorted according to the prediction confidence of f2′, and the top l×α images with the highest confidence are selected to form the sample set H2;

[0086] Where, the prediction confidence of f1′ for the i-th image in the image set H is:

[0087]

[0088] The prediction confidence of f2′ for the i-th image in the image set H is:

[0089]

[0090] Let f1′ be the probability that the i-th image in the image set H is classified into class j. f2′ represents the classification probability of the i-th image in the image set H being classified as class j, l is the number of labeled images in the labeled image set L, and α is a hyperparameter.

[0091] Step 7: Train f1′ using sample set H2′ and labeled image set L to obtain f1″; train f2′ using sample set H1′ and labeled image set L to obtain f2″.

[0092] In this embodiment, the training process is the same as that in step 2, and will not be described again here.

[0093] Step 8: Using f1” and f2” as the main classification model f1 and the auxiliary classification model f2, return to step 2 until the main classification model f1 reaches the expected classification performance or the number of labeled samples exceeds the set upper limit, then stop iterating and proceed to step 9.

[0094] In this embodiment, f1” and f2” are used as the main classification model f1 and the auxiliary classification model f2. The process returns to step 2 until the stopping condition is met, at which point the iteration stops, and the final f1 is used as the image classification model to perform the image classification prediction task.

[0095] Step 9: Use the main classification model f1 to perform image classification prediction task.

[0096] In this embodiment, the final f1 is used for image classification and prediction tasks.

[0097] In this embodiment, assisted learning and a multi-level diversity screening strategy are used to perform human expert annotation on more diverse samples to reduce the cost of manual annotation. Specifically, in assisted learning, this embodiment uses an auxiliary classification network to generate high-confidence pseudo-labels for the main classification network and removes redundant samples with similar representations to the labeled data from the unlabeled data. In each active learning loop, the main classification network and the auxiliary classification network exchange their respective high-confidence pseudo-labeled samples in a collaborative training manner, while unlabeled samples predicted by the main classification network and the auxiliary classification network to have different labels are added to the candidate pool for active learning. Based on the prediction of the main classification network, this embodiment uses an uncertainty criterion to select the most uncertain samples, and then uses a multi-level diversity screening strategy to remove redundant samples from the uncertain samples. The multi-level diversity criterion includes feature-level diversity and class-level diversity. Feature-level diversity can effectively uncover the distribution structure of unlabeled data. Unlabeled samples are clustered based on the pseudo-labels of the main classification model until the pseudo-labels are the same in each sub-cluster. The most uncertain sample is then selected from each sub-cluster, thus filtering out samples that are both uncertain and representative. Building upon feature-level diversity, this embodiment designs a class-level diversity criterion by considering the redundancy within query samples with the same pseudo-label. This effectively removes redundancy from uncertain samples with the same predicted label, ensuring that each query sample contributes to improving the generalization ability of the classification network. Therefore, the multi-level diversity screening criteria adopted in this embodiment can guarantee the representativeness and diversity of the selected samples, thereby reducing the cost of extensive manual annotation. Active learning and iterative training of the classification model using a small number of labeled samples can achieve training results close to those using a large number of labeled samples.

[0098] Secondly, embodiments of the present invention also provide a diversified query active learning device for image classification.

[0099] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of an embodiment of the active learning device for diversified queries in image classification according to the present invention. Figure 2 As shown, the active learning device for diverse queries for image classification includes:

[0100] The acquisition module 10 is used to acquire an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

[0101] The first training module 20 is used to train the main classification model f1 and the auxiliary classification model f2 respectively using the labeled image set L as the training set to obtain f1′ and f2′, and to use f1′ and f2′ to extract features and predict all unlabeled images in the unlabeled image set U.

[0102] The first construction module 30 is used to obtain an image set S composed of unlabeled images predicted by f1′ and f2′ to all unlabeled images in the unlabeled image set, and an image set H composed of unlabeled images predicted by f1′ and f2′ to the same label.

[0103] The second construction module 40 is used to cluster and sort the unlabeled images in the image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled.

[0104] The annotation module 50 is used to annotate the candidate set of unlabeled samples based on the annotation operation, and remove the annotated samples from the unlabeled image set U and add them to the labeled image set L;

[0105] The third construction module 60 is used to sort the images in the image set H according to the prediction confidence of f1′ and f2′ respectively, select the sample sets H1 and H2 with the highest confidence respectively, and assign pseudo labels output by f1′ and f2′ to the samples respectively to obtain sample sets H1′ and H2′.

[0106] The second training module 70 is used to train f1′ using the sample set H2′ and the labeled image set L to obtain f1″, and to train f2′ using the sample set H1′ and the labeled image set L to obtain f2″.

[0107] The loop module 80 is used to return to the steps executed by the first training module with f1” and f2” as the main classification model f1 and the auxiliary classification model f2, until the iteration stops when the main classification model f1 reaches the expected classification performance or the labeled samples exceed the set upper limit.

[0108] The prediction module 90 is used to perform image classification prediction tasks using the main classification model f1.

[0109] Furthermore, in one embodiment, the first training module 20 is used for:

[0110] For the i-th labeled image x in the labeled image set L i , to obtain x i The classification probabilities of being classified into class j by f1 and f2 respectively:

[0111]

[0112]

[0113] Where W1 and W2 are the parameters of f1 and f2, respectively; x is obtained based on f1 i The classification probability of belonging to class j. x is obtained based on f2 i The probability of belonging to class j;

[0114] By minimizing the cross-entropy loss of the main classification model f1 and the auxiliary classification model f2 on the labeled image set L. and The main classification model f1 and the auxiliary classification model f2 are trained to obtain f1′ and f2′, respectively. For labeled images in the labeled image set L, the cross-entropy losses on the main classification model f1 and the auxiliary classification model f2 are as follows:

[0115]

[0116]

[0117] in, and f1 and f2 are the objective functions for training the main classification model f1 and the auxiliary classification model f2, respectively; C is the total number of categories contained in the labeled image set L; l is the number of labeled images contained in the labeled image set L; y i It is the i-th labeled image x i The true label; j is x i Predicted labels; x is obtained based on the main classification model i The predicted probability of belonging to class j. x is obtained based on the auxiliary classification model i The predicted probability of belonging to class j; I{y i =j} is an indicator function, if y i If it equals j, then it is 1; otherwise, it is 0.

[0118] Furthermore, in one embodiment, the first training module 20 is used for:

[0119] For the i-th unlabeled image z in the unlabeled image set U i , to obtain z i The classification probabilities of being classified into class j by f1′ and f2′ respectively:

[0120]

[0121]

[0122] Where W1′ and W2′ are the parameters of f1′ and f2′, respectively; z is obtained based on f1′ i The classification probability of belonging to class j. z is obtained based on f2′ i The probability of belonging to class j;

[0123] Depend on:

[0124]

[0125]

[0126] We obtain f1′ and f2′ respectively with respect to z i Predicted pseudo-labels and

[0127] The output of the last pooling layer f1′ is used to obtain the z-axis. i The extracted feature vector.

[0128] Furthermore, in one embodiment, the second building module 40 is used for:

[0129] Based on the feature vector corresponding to each unlabeled image in the image set S, the k-means clustering algorithm is used to divide the image set S into C clusters, represented as U = U1∪U2∪...∪U C For the aforementioned C clusters, each cluster is further divided into several sub-clusters based on the number of class labels predicted by f1′. This iterative clustering operation is continued until each smallest sub-cluster contains only one class label. The clustering operation is then stopped, resulting in m smallest sub-clusters, denoted as... According to the BvSB criterion, the smallest subcluster with the highest uncertainty is selected, resulting in a query dataset F with m samples. The BvSB criterion is expressed as:

[0130]

[0131]

[0132]

[0133] For each minimal subcluster, This refers to the probability that a sample within a cluster is classified as class j by f1′; r max1 This refers to the class index where f1′ has the highest probability of classifying the sample, r max2 This refers to the index of the class with the second highest classification probability for the sample, f1′. Represents the maximum predicted probability of the sample. With the second largest predicted probability The difference between them, for each smallest subcluster, select p. diff The smallest sample forms the query dataset F, p diff The smallest sample is the one with the highest uncertainty.

[0134] If t samples in F have the same pseudo-label k, then the threshold for selecting different samples with the same pseudo-label k is expressed as:

[0135]

[0136] Where Var(·) is the variance operator, and β is a hyperparameter used to adjust the threshold. f1′ represents the classification probability output by f1′ for samples in F that have the same pseudo-label k. f1′ represents the classification probability output by f1′ for samples in F that have the same pseudo-label k.

[0137] Order F according to the uncertainty of the samples, and define it as follows: The uncertainty of the sample comes from Decrease to From F u Samples are selected sequentially and added to the unlabeled sample candidate set Q, where, It is F u If the j-th sample in Q is predicted to be of class k, then if there exists a t sample with pseudo-label k in Q. k For each sample, calculate With t k The similarity of the samples is:

[0138]

[0139] If t k If it is 0, then Add to Q; otherwise, u k When greater than v k At that time, Add to Q; repeat the above filtering process until there are h samples in Q, where h is a hyperparameter controlling the batch size of the active learning filtering. f1′ represents the classification probability output by f1′ for a sample in Q with pseudo-label k.

[0140] Furthermore, in one embodiment, the third building module 60 is used for:

[0141] The images in the image set H are sorted according to the prediction confidence of f1′, and the top l×α images with the highest confidence are selected to form the sample set H1;

[0142] The images in the image set H are sorted according to the prediction confidence of f2′, and the top l×α images with the highest confidence are selected to form the sample set H2;

[0143] Where, the prediction confidence of f1′ for the i-th image in the image set H is:

[0144]

[0145] The prediction confidence of f2′ for the i-th image in the image set H is:

[0146]

[0147] Let f1′ be the probability that the i-th image in the image set H is classified into class j. f2′ represents the classification probability of the i-th image in the image set H being classified as class j, l is the number of labeled images in the labeled image set L, and α is a hyperparameter.

[0148] The functions of each module in the above-mentioned active learning device for diversified queries for image classification correspond to the steps in the above-mentioned active learning method for diversified queries for image classification. Their functions and implementation processes will not be described in detail here.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0150] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.

[0152] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A diversified query active learning method for image classification, characterized in that, The diversified query active learning method for image classification includes: Step 1: Obtain an unlabeled image set consisting of a first number of unlabeled images. and a set of labeled images consisting of a second number of labeled images. The first quantity is greater than the second quantity; Step 2: Collect the labeled image set As the training set, respectively for the main classification model and auxiliary classification models To train and obtain and and utilize and For unlabeled image sets Feature extraction and prediction are performed on all unlabeled images; Step 3, from and The prediction results for all unlabeled images in the unlabeled image set are obtained from the predictions of the labeled images. and The image set S consists of unlabeled images predicted to have different labels and images predicted to have different labels. and The image set H consists of unlabeled images predicted to have the same label; Step 4: Cluster and sort the unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled. Step 5: Label the unlabeled sample candidate set based on the annotation operation, and then remove the labeled samples from the unlabeled image set. Remove and add tagged images to the set. middle; Step 6: Separate the images in image set H according to... and The prediction confidence scores were ranked, and the sample sets with the highest confidence scores were selected. and And assign each of the samples to a specific method. and The output pseudo-labels yield the sample set. and ; Step 7: Utilize the sample set and labeled image sets right To train and obtain Using sample sets and labeled image sets right To train and obtain ; Step 8, with and As the main classification model and auxiliary classification models Return to step 2 until the main classification model is reached. Stop iterating when the expected classification performance is achieved or the number of labeled samples exceeds the set upper limit, and proceed to step 9; Step 9: Utilize the principal classification model Perform image classification and prediction tasks; The set of labeled images will be As the training set, respectively for the main classification model and auxiliary classification models To train and obtain and The steps include: For labeled image sets The i-th labeled image ,get They were respectively and Classified as Classification probability: in, and They are respectively and Parameters; Based on Received belong Classification probability of a class Based on Received belong Classification probability; By minimizing the main classification model and auxiliary classification models In labeled image sets Cross-entropy loss and For the main classification model and auxiliary classification models To train and obtain and Among them, for labeled image sets Labeled images in the main classification model and auxiliary classification models The cross-entropy losses on are as follows: in, and These are training the main classification model. and auxiliary classification models The objective function; Tag image set The total number of categories included; Tag image set The number of tagged images included; It is the i-th labeled image. The true label; yes Predicted labels; It is obtained based on the main classification model belong Predicted probability of class It is obtained based on the auxiliary classification model belong The predicted probability of a class; It is an indicator function, if equal If the value is 1, then the value is 1; otherwise, the value is 0. The use of and For unlabeled image sets The steps for feature extraction and prediction of all unlabeled images include: For unlabeled image sets The i-th unlabeled image in ,get They were respectively and Classified as Classification probability: in, and They are respectively and Parameters; Based on Received belong Classification probability of a class Based on Received belong Classification probability; Depend on: get and To each Predicted pseudo-labels and ; according to The output of the last pooling layer is obtained from the The extracted feature vector; The steps of clustering and ranking unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled include: Based on the feature vector corresponding to each unlabeled image in the image set S, the k-means clustering algorithm is used to divide the image set S into groups. A cluster, denoted as Regarding the above There are clusters, based on the number of clusters in each cluster. The predicted number of class labels further divides the clusters into several sub-clusters, and the iterative clustering operation described above continues until each smallest sub-cluster contains only one class label. The clustering operation then stops, yielding the desired result. The smallest subclusters are represented as According to the BvSB criterion, the one with the highest uncertainty is selected from each minimum subcluster, resulting in the one with... Query dataset of samples The BvSB criterion is expressed as: For each smallest subcluster, This refers to samples within a cluster being Classified as Classification probability; It refers to The index of the category with the highest classification probability for the sample. It refers to Subscript the class with the second highest classification probability for the sample. Represents the maximum predicted probability of the sample. With the second largest predicted probability The difference between them is selected for each smallest subcluster. The smallest sample forms the query dataset. , The smallest sample is the one with the highest uncertainty. If in If t samples have the same pseudo-label k, then the threshold for selecting different samples with the same pseudo-label k is expressed as: in It is the variance operator. These are hyperparameters used to adjust the threshold. represent against The classification probability output by samples with the same pseudo-label k. , represent against The classification probability output by samples with the same pseudo-label k. ; Will Sort according to the uncertainty of the samples, and define as The uncertainty of the sample is from Decrease to ;from Samples are selected sequentially and added to the unlabeled sample candidate set. In, among them, yes The j-th sample in the dataset is predicted to be of class k. There exists a pseudo-label k in the middle. For each sample, calculate and The similarity of the samples is: like If it is 0, then Add to Middle; otherwise, When greater than At that time, Add to Repeat the above screening process until... There are h samples, where h is a hyperparameter controlling the batch size for active learning selection. represent against The classification probability of the output sample with pseudo-label k.

2. The diversified query active learning method for image classification as described in claim 1, characterized in that, The images in image set H are respectively classified according to and The prediction confidence scores were ranked, and the sample sets with the highest confidence scores were selected. and The steps include: The images in image set H are arranged according to... The prediction confidence scores are sorted, and the top scores with the highest confidence scores are selected. The images constitute the sample set. ; The images in image set H are arranged according to... The prediction confidence scores are sorted, and the top scores with the highest confidence scores are selected. The images constitute the sample set. ; in, The prediction confidence for the images in image set H is: Tag image set It includes the number of tagged images. This is a hyperparameter.

3. A diversified query active learning device for image classification, characterized in that, The image classification-oriented diversified query active learning device includes: The acquisition module is used to acquire an unlabeled image set consisting of a first number of unlabeled images. and a set of labeled images consisting of a second number of labeled images. The first quantity is greater than the second quantity; The first training module is used to train the labeled image set. As the training set, respectively for the main classification model and auxiliary classification models To train and obtain and and utilize and For unlabeled image sets Feature extraction and prediction are performed on all unlabeled images; The first building module is used by... and The prediction results for all unlabeled images in the unlabeled image set are obtained from the predictions of the labeled images. and The image set S consists of unlabeled images predicted to have different labels and images predicted to have different labels. and The image set H consists of unlabeled images predicted to have the same label; The second building module is used to cluster and sort unlabeled images in image set S using multi-level diversity criteria to form a candidate set of unlabeled samples to be labeled. The annotation module is used to annotate the candidate set of unlabeled samples based on annotation operations, and to remove the annotated samples from the unlabeled image set. Remove and add tagged images to the set. middle; The third construction module is used to sort the images in image set H according to... and The prediction confidence scores were ranked, and the sample sets with the highest confidence scores were selected. and And assign each of the samples to a specific method. and The output pseudo-labels yield the sample set. and ; The second training module is used to utilize the sample set. and labeled image sets right To train and obtain Using sample sets and labeled image sets right To train and obtain ; The loop module is used to... and As the main classification model and auxiliary classification models Return to the steps executed in the first training module, and continue until the main classification model is reached. The iteration stops when the expected classification performance is achieved or the number of labeled samples exceeds the set upper limit. The prediction module is used to utilize the main classification model. Perform image classification and prediction tasks; The first training module is used for: For labeled image sets The i-th labeled image ,get They were respectively and Classified as Classification probability: in, and They are respectively and Parameters; Based on Received belong Classification probability of a class Based on Received belong Classification probability; By minimizing the main classification model and auxiliary classification models In labeled image sets Cross-entropy loss and For the main classification model and auxiliary classification models To train and obtain and Among them, for labeled image sets Labeled images in the main classification model and auxiliary classification models The cross-entropy losses on are as follows: in, and These are training the main classification model. and auxiliary classification models The objective function; Tag image set The total number of categories included; Tag image set The number of tagged images included; It is the i-th labeled image. The true label; yes Predicted labels; It is obtained based on the main classification model belong Predicted probability of class It is obtained based on the auxiliary classification model belong The predicted probability of a class; It is an indicator function, if equal If the value is 1, then the value is 1; otherwise, the value is 0. The first training module is used for: For unlabeled image sets The i-th unlabeled image in ,get They were respectively and Classified as Classification probability: in, and They are respectively and Parameters; Based on Received belong Classification probability of a class Based on Received belong Classification probability; Depend on: get and To each Predicted pseudo-labels and ; according to The output of the last pooling layer is obtained from the The extracted feature vector; The second building module is used for: Based on the feature vector corresponding to each unlabeled image in the image set S, the k-means clustering algorithm is used to divide the image set S into groups. A cluster, denoted as Regarding the above There are clusters, based on the number of clusters in each cluster. The predicted number of class labels further divides the clusters into several sub-clusters, and the iterative clustering operation described above continues until each smallest sub-cluster contains only one class label. The clustering operation then stops, yielding the desired result. The smallest subclusters are represented as According to the BvSB criterion, the one with the highest uncertainty is selected from each minimum subcluster, resulting in the one with... Query dataset of samples The BvSB criterion is expressed as: For each smallest subcluster, This refers to samples within a cluster being Classified as Classification probability; It refers to The index of the category with the highest classification probability for the sample. It refers to Subscript the class with the second highest classification probability for the sample. Represents the maximum predicted probability of the sample. With the second largest predicted probability The difference between them is selected for each smallest subcluster. The smallest sample forms the query dataset. , The smallest sample is the one with the highest uncertainty. If in If t samples have the same pseudo-label k, then the threshold for selecting different samples with the same pseudo-label k is expressed as: in It is the variance operator. These are hyperparameters used to adjust the threshold. represent against The classification probability output by samples with the same pseudo-label k. , represent against The classification probability output by samples with the same pseudo-label k. ; Will Sort according to the uncertainty of the samples, and define as The uncertainty of the sample is from Decrease to ;from Samples are selected sequentially and added to the unlabeled sample candidate set. In, among them, yes The j-th sample in the dataset is predicted to be of class k. There exists a pseudo-label k in the middle. For each sample, calculate and The similarity of the samples is: like If it is 0, then Add to Middle; otherwise, When greater than At that time, Add to Repeat the above screening process until... There are h samples, where h is a hyperparameter controlling the batch size for active learning selection. represent against The classification probability of the output sample with pseudo-label k.

4. The active learning device for diversified queries for image classification as described in claim 3, characterized in that, The third building module is used for: The images in image set H are arranged according to... The prediction confidence scores are sorted, and the top scores with the highest confidence scores are selected. The images constitute the sample set. ; The images in image set H are arranged according to... The prediction confidence scores are sorted, and the top scores with the highest confidence scores are selected. The images constitute the sample set. ; in, The prediction confidence for the images in image set H is: Tag image set It includes the number of tagged images. This is a hyperparameter.

Images