Active learning method based on uncertainty and diversity acquisition function

Abstract

This invention presents an active learning method based on uncertainty and diversity acquisition functions, belonging to the field of active learning for object detectors in computer vision. It addresses the problem that existing active learning processes, utilizing initialized object detectors, cannot deeply mine valuable image features. The method includes: using a weakly supervised object detector trained on image-level labels and a fully supervised object detector initialized with partial instance-level labels to detect images in the training set that have been labeled with ground truth or pseudo-ground truth values; calculating instance-level difficulty scores, class-level difficulty scores, and class similarity weights based on the detection results; then detecting images without instance-level labels, calculating image-level difficulty scores based on class relevance weights and entropy, and determining candidate images; calculating the similarity between every two candidate images based on the detection results, determining cluster centers for the candidate images, and identifying valuable images after clustering. This invention further refines fully supervised object detectors.

Description

Technical Field

[0001] This invention relates to an active learning method based on uncertainty and diversity acquisition functions, belonging to the field of active learning for object detectors in computer vision. Background Technology

[0002] Compared to traditional object detectors that perform instance-level annotation on all training set images, active learning object detection technology selects valuable images from the unannotated images in the training set for instance-level annotation during the active learning phase, thus balancing annotation cost and detection performance.

[0003] Active learning object detection technology mainly consists of two stages: the initialization stage and the active learning stage. To ensure that valuable images are discovered during the active learning stage, existing active learning methods use entropy calculations based on an object detector with preliminary detection capabilities during initialization to select valuable images. However, the object detector with preliminary detection capabilities largely depends on the number of image-level or instance-level labels during initialization, and relying solely on a single object detector to select the most valuable image is considered a weak constraint. Therefore, the limitation of existing active learning object detection technology is that the object detector after active learning initialization only has preliminary detection capabilities and cannot deeply mine valuable image features. Summary of the Invention

[0004] To address the problem that existing active learning processes cannot deeply mine valuable image features using initialized object detectors, this invention provides an active learning method based on uncertainty and diversity acquisition functions.

[0005] The present invention provides an active learning method for acquiring functions based on uncertainty and diversity, comprising:

[0006] A weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels are used to detect images in the training set that have been labeled with ground truth or pseudo-ground truth values. The detection results output by the weakly supervised object detector are as follows: The detection result output by the fully supervised object detector is y. f The instance-level difficulty score is calculated by combining the true or false true values ​​accessed during the active learning initialization process; the class-level difficulty score is then calculated by updating the instance-level difficulty score using an exponential moving average; and finally, the class-level difficulty score is processed non-linearly to obtain the class-related weights.

[0007] The unlabeled instance-level images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The image-level difficulty score of the unlabeled instance-level images is calculated based on the class relevance weights and entropy. The image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level images with high uncertainty scores are selected as candidate images. Here, b is a set constant.

[0008] Candidate images are detected using a weakly supervised object detector trained on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The output includes the detection results, which include the object location, object category, and object features. Based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between each pair of candidate images.

[0009] K candidate images with low similarity to each other are selected as cluster centers based on image similarity. All candidate images are then clustered based on cosine similarity and Euclidean distance based on these cluster centers. The cluster center of each cluster is then used as a valuable image to refine the initialized fully supervised object detector. Here, K is a set constant.

[0010] According to the active learning method based on uncertainty and diversity acquisition functions of the present invention, the method for calculating instance-level difficulty scores is as follows:

[0011]

[0012] In the formula, d is the instance-level difficulty score, and y gt For the true or false true value corresponding to the detection result, P(y) f |y gt ) ρ For the test result y f The classification results in the data, where ρ is the exponential hyperparameter. The location information in the detection results is represented by IoU, which is the intersection-union ratio calculation function.

[0013] According to the active learning method based on uncertainty and diversity acquisition functions of the present invention, the method for calculating class-level difficulty scores is as follows:

[0014]

[0015] In the formula The class-level difficulty score g for the i-th object class i The result of the kth training iteration, Let m be the exponential moving average momentum of the i-th type of object. i The result of the (k-1)th training iteration, d represents the number of objects of class i in the current batch during the k-th training iteration. jLet represent the instance-level difficulty score of the j-th instance in the i-th class.

[0016] According to the active learning method of the present invention based on uncertainty and diversity acquisition function, m i The initial value is m 0 If the current batch contains objects of type i, m 0 No change; if the i-th type of object does not exist in the current batch, m i The update formula is:

[0017]

[0018] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, the method for calculating class relevance weights is as follows:

[0019] ω i =1+λ·γlog(1+(e 1 / γ -1)·g i ),

[0020] In the formula ω i Let λ be the class-related weight of the i-th object, λ be the hyperparameter that limits the upper bound of the class-related weight, and γ be the hyperparameter that controls the rate of change of the class-related weight.

[0021] According to the active learning method based on uncertainty and diversity acquisition functions of the present invention, the method for calculating the image-level difficulty score is as follows:

[0022]

[0023] In the formula, U(I) is the uncertainty acquisition function, and N I ω represents the number of objects predicted in the current unlabeled instance-level image. i(j) Let p be the class relevance weight of the j-th object predicted as an i-th object by the fully supervised object detector. ji Let C be the probability of predicting the classification of the j-th object as the i-th object; C is the total number of object categories in the current unlabeled instance-level image.

[0024] Image-level difficulty scores are obtained based on the uncertainty acquisition function U(I).

[0025] According to the active learning method of the present invention based on uncertainty and diversity acquisition function, two candidate images are defined as I a and I b We employ a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels to respectively target I a and I b The detection result output by the weakly supervised object detector is O'. a and Ob The detection result output by the fully supervised object detector is O″. a and O″ b , then I a and I b The image similarity is:

[0026]

[0027] In the formula, S' is the I calculated using a weakly supervised object detector. a and I b The image similarity, S” is the I calculated using a fully supervised object detector. a and I b Image similarity; S' and S” are calculated in the same way:

[0028]

[0029] In the formula t a,j For image I a The classification result of the j-th object, M a For image I a The total number of predicted objects, o′ a,j Image I a The j-th object in the sequence, sim(o′) a,j O b ') represents o′ a,j with I b Cosine similarity of all objects of the same type.

[0030] The beneficial effects of this invention are as follows: By refining the object detector using valuable images, the method of this invention can effectively balance annotation costs and detection performance. In the initialization phase, this invention selects an initialization method based on a large visual model and a weakly supervised object detector; in the active learning phase, it selects valuable images, thus overcoming the limitations of active learning object detection techniques.

[0031] This invention's method employs active learning based on uncertainty and diversity acquisition functions, jointly considering the uncertainties and diversity of both fully supervised and weakly supervised object detectors, and adding strict constraints to traditional acquisition functions. Compared to traditional active learning methods, this invention selects the most valuable images that better cover the data distribution of unlabeled instance-level images across the entire training set. Based on the images / labels from the initialization process and those mined during the active learning process, the fully supervised object detector is further refined.

[0032] This invention addresses the problem that active learning object detection techniques cannot deeply mine valuable image features during the active learning process. It enhances the representation ability of unlabeled images in the training set during active learning, overcoming the limitation of traditional active learning processes relying solely on a single fully supervised detector. This invention belongs to the research on mining valuable images during the active learning process in active learning object detection technology, and to a certain extent promotes the implementation of object detection technology based on deep learning in artificial intelligence, aligning with the development trend of contemporary intelligent manufacturing.

[0033] The method of this invention benefits from the fact that fully supervised object detectors and weakly supervised object detectors can complement each other in terms of precision and recall. Furthermore, the weakly supervised object detector provides strict constraints on the traditional active learning acquisition function. Compared with the traditional active learning acquisition function, it can extract more valuable images and further refine the fully supervised detector, achieving a balance between annotation cost and detection performance. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the active learning method based on uncertainty and diversity acquisition functions as described in this invention.

[0035] Figure 2 Line graph showing the performance comparison of the method of this invention combined with RetinaNet and the existing state-of-the-art methods in each round on the VOC2007 test set;

[0036] Figure 3 Line graph showing the performance comparison of the proposed method combined with Deformable-DETR with existing state-of-the-art methods in each round on the VOC2007 test set;

[0037] Figure 4 This is a visualization of the detection results and ground truth values ​​of a fully supervised object detector trained using the method of this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0041] Specific Implementation Method 1: Combination Figure 1 As shown, this invention provides an active learning method for acquiring functions based on uncertainty and diversity, including:

[0042] A weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels are used to detect images in the training set that have been labeled with ground truth or pseudo-ground truth values. The detection results output by the weakly supervised object detector are as follows: The detection result output by the fully supervised object detector is y. f The instance-level difficulty score is calculated by combining the true or false true values ​​accessed during the active learning initialization process; the class-level difficulty score is then calculated by updating the instance-level difficulty score using an exponential moving average; and finally, the class-level difficulty score is processed non-linearly to obtain the class-related weights.

[0043] The unlabeled instance-level images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The image-level difficulty score of the unlabeled instance-level images is calculated based on the class relevance weights and entropy. The image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level images with high uncertainty scores are selected as candidate images. Here, b is a set constant.

[0044] Candidate images are detected using a weakly supervised object detector trained on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The output includes the detection results, which include the object location, object category, and object features. Based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between each pair of candidate images.

[0045] K candidate images with low similarity to each other are selected as cluster centers based on image similarity. All candidate images are then clustered based on cosine similarity and Euclidean distance based on these cluster centers. The cluster center of each cluster is then used as a valuable image to refine the initialized fully supervised object detector. Here, K is a set constant.

[0046] Combination Figure 1 As shown, the weakly supervised object detector and the fully supervised object detector have learnable parameters that can be trained, while the other parts represent fixed functions and image / label information. The joint weakly supervised object detector M... w and fully supervised object detector M f Entropy is used to measure the classification results of fully supervised object detectors and weak / fully supervised object detectors M. w / M fTo address the difficulty of localization, an uncertainty score is calculated for unlabeled images based on classification confidence and localization quality. Based on the uncertainty score, a subset of images are selected as candidate images, and then valuable images are chosen from these candidate images. This implementation calculates similarity as a proxy task for the diversity acquisition function. Similarly, considering the similarity between the detection results of a weakly supervised object detector and a fully supervised detector, this implementation calculates the similarity of features of objects of the same category between images, and the maximum similarity value is used as the similarity score between the two images. Based on the calculated image similarity, diverse images are selected from the candidate images as valuable images to further refine the fully supervised object detector.

[0047] This implementation method combines a weakly supervised object detector M w and fully supervised object detector M f By increasing the weight of difficult categories and decreasing the weight of easy categories, the object detector focuses on instances corresponding to difficult categories. These labeled images further enhance the detection capabilities of the fully supervised object detector. Assuming there are C categories in the dataset, this invention calculates the instance-level difficulty score d by jointly using the classification confidence p of the fully supervised object detector and the localization quality IoU of the weak / fully supervised detector. The classification confidence of the fully supervised object detector is obtained by performing a softmax operation on the predicted values ​​output by the classification head. The localization quality of the weak / fully supervised object detector is obtained by selecting the intersection-union ratio (IoU) between the detection results of the weak / fully supervised object detector and the ground truth. It is worth noting that, since the ground truth information of unlabeled images is not accessible, the ground truth refers to the ground truth / pseudo-ground truth values ​​already labeled during the active learning initialization phase based on the large visual language model and the weakly supervised object detector. The process of calculating the instance-level difficulty score d not only considers the weakly supervised object detector M... w and fully supervised object detector M f Furthermore, classification and localization quality are also considered. Then, the instance-level difficulty score d is updated using an exponential moving average to calculate the class-level difficulty score g. This invention defines the class-level difficulty score g, and the class-level difficulty score for each class is initialized to 1. Subsequently, the class-level difficulty scores undergo non-linear processing to obtain class-related weights. Combining the entropy calculation method in computer theory, the entropy of objects present in each image is weighted using class-related weights and summed, representing the calculated image uncertainty U. Based on the uncertainty U calculated in the above process, uncertainty scores are calculated for images in the training set that are not labeled with instance-level tags. Next, a candidate pool containing b images is set, and b images with high uncertainty scores are selected from the unlabeled images in the training set and added to the candidate pool.

[0048] Next, after measuring the uncertainty score of each image, a specified number of K valuable images are further extracted from the candidate pool. Consistent with the motivation of the uncertainty function, the fully supervised object detector in the initialization process only has preliminary detection capabilities and is weak in mining deep valuable image features. Thanks to the active learning initialization method that utilizes image-level labels, the weakly supervised object detector M trained using existing image-level labels is utilized. w It exhibits good feature extraction capabilities. For two candidate images, the weakly supervised object detector M... w and fully supervised object detector M f Each image outputs its detection results. Then, the similarity between the two candidate images is calculated. The similarity is determined by matching the maximum similarity between each object and the same type of object in the other image. Based on the calculated similarity, K images are selected from the candidate pool as cluster centers. A clustering process is then performed on all images in the candidate pool until the clustering process is complete. The cluster center of each cluster is considered a valuable image.

[0049] This implementation addresses the problem of traditional active learning methods relying on initialized object detectors that cannot deeply mine valuable image features. By combining the uncertainties and detection results of weakly supervised and fully supervised object detectors, it introduces strict constraints in the process of mining valuable images. The mined valuable images further refine the object detector, balancing annotation costs and detection performance.

[0050] Furthermore, the joint weakly supervised object detector M w and fully supervised object detector M f Considering that entropy measures the degree of uncertainty of information in computer theory, entropy is used to measure the classification results of fully supervised object detectors and weak / fully supervised object detectors M. w / M f The difficulty of localization is addressed by considering the fact that entropy is positively correlated with the number of objects in an image, and the weights for all categories are set to 1 during entropy calculation, without considering the need for different weights for different categories. Therefore, this paper first calculates instance-level difficulty, then calculates class-level difficulty using instance-level difficulty, performs non-linear calculations on class-level difficulty to obtain class-related weights, and combines entropy and class-related weights to calculate image-level uncertainty. For instance-level difficulty, the intersection-union ratio (IU) with the ground truth is obtained from the classification results output by fully supervised and weakly supervised object detectors. The classification results are obtained by applying a softmax function to the classification head output of the fully supervised object detector, and the value with the largest IU between the fully / weakly supervised object detector and the ground truth is selected as the localization quality.

[0051] The method for calculating instance-level difficulty scores is as follows:

[0052]

[0053] In the formula, d is the instance-level difficulty score, and y gt For the true or false true value corresponding to the detection result, P(y) f |y gt ) ρ For the test result y f The classification results in the data, where ρ is the exponential hyperparameter. The location information in the detection results is represented by IoU, which is the crossover ratio function.

[0054] In this embodiment, the constraint of increasing the difficulty of instance-level labels is enhanced by the max operation. Instances that are considered difficult in the traditional active learning process are considered simple in this embodiment. Therefore, the valuable images mined by the active learning acquisition function in this embodiment are more valuable / difficult than those in the past.

[0055] The class-level difficulty score for each class is initialized to 1. The class-level difficulty score is calculated as follows:

[0056]

[0057] In the formula The class-level difficulty score g for the i-th object class i The result of the kth training iteration, Let m be the exponential moving average momentum of the i-th type of object. i The result of the (k-1)th training iteration, d represents the number of objects of class i in the current batch during the k-th training iteration. j Let represent the instance-level difficulty score of the j-th instance in the i-th class.

[0058] m i The initial value is m 0 If the current batch contains objects of type i, m 0 No change; if the i-th type of object does not exist in the current batch, m i The update formula is:

[0059]

[0060] The method for calculating class-related weights is as follows:

[0061] ω i =1+λ·γlog(1+(e 1 / γ -1)·g i ),

[0062] In the formula ω i Let λ be the class-related weight of the i-th object, λ be the hyperparameter that limits the upper bound of the class-related weight, and γ be the hyperparameter that controls the rate of change of the class-related weight.

[0063] Next, calculate the image-level hardness score for each unlabeled instance in the training set, expressed as a weighted sum for each detected target:

[0064]

[0065] In the formula, U(I) is the uncertainty acquisition function, and N I ω represents the number of objects predicted in the current unlabeled instance-level image. i(j) Let p be the class relevance weight of the j-th object predicted as an i-th object by the fully supervised object detector. ji Let C be the probability of predicting the classification of the j-th object as the i-th object; C is the total number of object categories in the current unlabeled instance-level image.

[0066] Image-level difficulty scores are obtained based on the uncertainty acquisition function U(I).

[0067] Furthermore, let's define the two candidate images as I a and I b We employ a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels to respectively target I a and I b The detection result output by the weakly supervised object detector is O'. a and O b The detection result output by the fully supervised object detector is O″. a and O″ b , then I a and I b The image similarity is:

[0068]

[0069] In the formula, S' is the I calculated using a weakly supervised object detector. a and I b The image similarity, S” is the I calculated using a fully supervised object detector. a and I b Image similarity; S' and S" are calculated in the same way:

[0070]

[0071] In the formula t a,j For image I a The classification result of the j-th object, M a For image I a The total number of predicted objects, o′ a,j Image I aThe j-th object in the sequence, sim(o′) a,j O b ') represents o′ a,j with I b The cosine similarity of all objects of the same type in a given set. `sim()` represents the cosine similarity function, where `sim()` calculates the cosine similarity when I = ... a and I b If the object categories do not co-occur, the similarity calculated for the current object is set to 0; otherwise, I is calculated. a and I b Cosine similarity of features of objects of the same category.

[0072] Based on calculated similarity, K images are mined from the candidate pool as cluster centers, where these K images must have low similarity to other images in the candidate pool. Then, with these K images as cluster centers, all images in the candidate pool are clustered into K clusters using cosine similarity and Euclidean distance. This clustering process continues until the cluster centers of each cluster are considered valuable images for further refinement of the fully supervised object detector. Through carefully designed uncertainty and diversified acquisition functions, this implementation ultimately achieves a good balance between annotation cost and detection performance.

[0073] Verification experiment:

[0074] Prepare training samples. The effectiveness of the proposed method is verified using the VOC2007 / 2012 and MS COCO 2017 datasets. The proposed model is further trained using 5011 training images from the VOC2007 dataset and 11540 training images from the VOC2012 dataset. The model accuracy is evaluated using 4952 test images from the VOC2007 test set. Furthermore, the model is trained and validated using COCO2017, with a training set of 118,000 images and a validation set of 5,000 images containing 80 classes of objects to be detected. In the VOC2007 / 2012 and COCO2017 training sets mentioned above, not all images were used to train the model. In the VOC dataset, 2.5% of the images / instance level labels were used in each round of active learning, and 2.0% of the images / instance level labels were used in each round of active learning in the COCO dataset. The experimental results followed the evaluation metrics given by the official documentation. For the VOC dataset, the model's accuracy mAP was evaluated on the test set, and for the COCO dataset, the model's AP, AP50, and AP75 were evaluated on the validation set.

[0075] We employ state-of-the-art active learning initialization methods to enable a fully supervised object detector with preliminary detection capabilities. MiniGPT-v2 is used as the large visual-language model, OD-WSCL as the baseline weakly supervised object detector, and RetinaNet and Deformable-DETR as fully supervised detectors. We set ρ = 0.3 to compute instance-level difficulty, and set the number of active learning epochs r to 6 or 4 on the VOC and COCO datasets, respectively. m0 is set to 0.99, λ to 0.2, and γ to 0.3. The RetinaNet detector is trained for 26 epochs per active learning epoch, with the learning rate decaying to 0.1 times its original value at the 20th epoch. The RetinaNet detector uses the SGD optimizer with an initial learning rate of 0.01. The Deformable-DETR detector is trained for 50 epochs per active learning epoch, with the learning rate decaying to 0.1 times its original value at the 40th epoch. The Deformable-DETR detector uses the AdamW optimizer with an initial learning rate of 0.00002. The OD-WSCL detector was trained for 30,000 iterations using the SGD optimizer with an initial learning rate of 0.01. The code for this method is based on PyTorch and trained using eight RTX 3090 GPUs.

[0076] Experiments demonstrate that the method of this invention improves the constraints for mining valuable images, uncovers deeper levels of valuable images, and enhances the detection performance of active learning object detection techniques. The effectiveness of the proposed method was verified by training on the common VOC2007 / 2012 datasets and testing on the VOC2007 test set. Finally, the test results and ground truth values ​​on the VOC dataset are presented.

[0077] 1) Effectiveness Analysis of Active Learning Process Based on Uncertainty and Diversity Acquisition Functions on the VOC2007 Test Set. As shown in Table 1, using the RetinaNet detector on the VOC2007 test set, a state-of-the-art initialization method achieved a mAP of 59.1%. Based on this, by introducing the uncertainty and diversity acquisition functions designed for the active learning process according to this invention, a mAP of 77.3% was achieved, improving detection performance by 18.2%. Compared to the methods of randomly selecting 15% of image / instance level labels and randomly selecting 15% of image / instance level labels plus pseudo-ground values ​​from the initialization process to train a fully supervised object detector, this invention's method surpasses the detection performance of the above two methods by 13.5% and 1.8%, respectively. Based on the above experimental process, the effectiveness of the active learning process proposed in this invention in mining valuable images is verified. According to the last two rows of Table 1, active learning relying solely on the uncertainty acquisition function achieved a detection performance of 77.0%, while introducing the diversity acquisition function resulted in a detection performance of 77.3%, further illustrating the effectiveness of the uncertainty and diversity acquisition functions designed in this invention.

[0078] Table 1. Effectiveness analysis of uncertainty and diversity acquisition functions for active learning processes on the VOC2007 test set.

[0079]

[0080] 2) The active learning method of this invention was compared with the state-of-the-art weakly supervised object detector (OD-WSCL) and the fully supervised detector (RetinaNet Deformable-DETR) on the VOC and COCO datasets. Table 2 shows that OD-WSCL achieved mAP, AP50, and AP (0.5:0.95:0.05) of 59.7%, 28.1%, and 14.4%, respectively, which is significantly lower than existing fully supervised object detectors. Based on the method of this invention, combined with the RetinaNet fully supervised object detector, mAP, AP50, and AP (0.5:0.95:0.05) of 77.0%, 43.2%, and 26.6% were achieved, respectively, relying on only 15% and 8% of the image / instance level labels. It is worth noting that a%, b%, and c% represent that the initialization process uses a% of the instance level labels, and each round of active learning mines c% of the images as valuable images and labels them with instance level labels, until the active learning process ends, requiring a total of b% of the image / instance level labels. Furthermore, to verify the generalization ability of this invention, a Transformer-based fully supervised object detector (Deformable-DETR) was integrated, achieving a mAP of 72.9% with only 15% instance-level labels, approaching that of the fully supervised method. For a fair comparison, this invention selected 15% and 8% of images / instance-level labels to train RetinaNet and Deformable-DETR, as shown in the last four rows of Table 2. This invention indeed selected valuable images. The active learning method based on uncertainty and diversity designed in this invention outperforms the randomly selected method by approximately 10% in mAP, balancing annotation costs and detection performance, relying only on a small portion of instance-level labels, and approaching the results of fully supervised object detection.

[0081] Table 2 compares the performance of the present invention with that of a weakly / fully supervised object detector on the VOC2007 test set and the COCO2017 validation set using different instance-level label ratios.

[0082]

[0083] 3) The invention was compared with other state-of-the-art (SOTA) methods on the VOC2007 test set and the COCO2017 validation set. As shown in Table 3, the invention outperformed previous SOTA methods. For example, RetinaNet combined with the designed active learning method achieved 77.3% mAP, exceeding existing SOTA methods by 4.7% (72.6% vs 77.3%). Figure 2In this paper, the results of the comparison with existing state-of-the-art methods are visualized. Because the initialization process employs an advanced active learning initialization method, it does not utilize any instance-level labels, relying solely on image-level labels. Therefore, the active learning process of this invention begins with 0% instance-level labels. Figure 2 This invention demonstrates that it surpasses existing methods. For example, it outperforms existing state-of-the-art (SOTA) methods when accessing 2.5% of instance-level tags during active learning.

[0084] To verify the robustness of this invention, test results on the COCO dataset are shown in Table 3. Similar to the experimental conclusions above, this invention surpasses existing methods. Compared to existing state-of-the-art methods, this invention improves AP50 by 38.7% to 45.6% and improves AP(0.5:0.95:0.05) by 23.2% to 27.9%. Figure 3 As shown, the active learning method designed by combining RetinaNet and Deformable-DETR with the present invention relies on only 2% and 4% of instance-level labels, which surpasses the previous active learning method that used 10% of instance-level labels.

[0085] Table 3 compares this invention with other state-of-the-art methods on the VOC2007 test set and COCO2017 validation set.

[0086]

[0087] 4) Visualization of test results and truth values. Figure 4 The prediction results are shown on the VOC and COCO datasets. It can be observed that the active learning object detection technique of this invention can locate complete object regions even when instances are occluded or densely packed. Furthermore, this invention visualizes some failed examples, the main reason being the incorrect category of the object region. Specifically, the pseudo-ground values ​​output by the large visual language model during the initialization process have incorrect categories, affecting the performance of subsequent active learning.

[0088] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. An active learning method for acquiring functions based on uncertainty and diversity, characterized in that, include, A weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels are used to detect images in the training set that have been labeled with ground truth or pseudo-ground truth values. The detection results output by the weakly supervised object detector are as follows: The detection result output by the fully supervised object detector is y. f The instance-level difficulty score is calculated by combining the true or false true values ​​accessed during the active learning initialization process; the class-level difficulty score is then calculated by updating the instance-level difficulty score using an exponential moving average. Then, non-linear processing is applied to the class-level difficulty scores to obtain class-related weights; The unlabeled instance-level images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The image-level difficulty score of the unlabeled instance-level images is calculated based on the class relevance weights and entropy. The image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level images with high uncertainty scores are selected as candidate images. Here, b is a set constant. Candidate images are detected using a weakly supervised object detector trained on image-level labels and a fully supervised object detector initialized with partial instance-level labels. The output includes the detection results, which include the object location, object category, and object features. Based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between each pair of candidate images. K candidate images with low similarity to each other are selected as cluster centers based on image similarity. All candidate images are then clustered based on cosine similarity and Euclidean distance based on these cluster centers. The cluster center of each cluster is then used as a valuable image to refine the initialized fully supervised object detector. Here, K is a set constant. The method for calculating instance-level difficulty scores is as follows: In the formula, d is the instance-level difficulty score, and y gt For the true or false true value corresponding to the detection result, P(y) f |y gt ) ρ For the test result y f The classification results in the data, where ρ is the exponential hyperparameter. The IoU is the location information in the detection results, and it is the intersection-union ratio calculation function. Define two candidate images as I a and I b We employ a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with partial instance-level labels to respectively target I a and I b The detection result output by the weakly supervised object detector is O′. a and O′ b The detection result output by the fully supervised object detector is O″. a and O″ b , then I a and I b The image similarity is: In the formula, S′ is the I calculated using a weakly supervised object detector. a and I b Image similarity, S″ is the I calculated using a fully supervised object detector. a and I b Image similarity; S′ and S″ are calculated in the same way: In the formula t a,j For image I a The classification result of the j-th object, M a For image I a The total number of predicted objects, o′ a,j Image I a The j-th object in the sequence, sim(o′) a,j ,O′ b ) represents o′ a,j with I b Cosine similarity of all objects of the same type.

2. The active learning method based on uncertainty and diversity acquisition function according to claim 1, characterized in that, The method for calculating the difficulty score for each category is as follows: In the formula The class-level difficulty score g for the i-th object class i The result of the kth training iteration, Let m be the exponential moving average momentum of the i-th type of object. i The result of the (k-1)th training iteration, d represents the number of objects of class i in the current batch during the k-th training iteration. j Let represent the instance-level difficulty score of the j-th instance in the i-th class.

3. The active learning method based on uncertainty and diversity acquisition function according to claim 2, characterized in that, m i The initial value is m 0 If the current batch contains objects of type i, m 0 No change; if the i-th type of object does not exist in the current batch, m i The update formula is:

4. The active learning method based on uncertainty and diversity acquisition function according to claim 3, characterized in that, The method for calculating class-related weights is as follows: oh i =1+λ·γlog(1+(e 1 / γ -1)·g i ), In the formula ω i Let λ be the class-related weight of the i-th object, λ be the hyperparameter that limits the upper bound of the class-related weight, and γ be the hyperparameter that controls the rate of change of the class-related weight.

5. The active learning method based on uncertainty and diversity acquisition function according to claim 4, characterized in that, The method for calculating the image-level difficulty score is as follows: In the formula, U(I) is the uncertainty acquisition function, and N I ω represents the number of objects predicted in the current unlabeled instance-level image. i(j) Let p be the class relevance weight of the j-th object predicted as an i-th object by the fully supervised object detector. ji Let C be the probability of predicting the classification of the j-th object as the i-th object; C is the total number of object categories in the current unlabeled instance-level image. Image-level difficulty scores are obtained based on the uncertainty acquisition function U(I).

Images