Active learning method for obtaining function based on uncertainty and diversity

Abstract

The invention discloses an active learning method based on uncertainty and diversity acquisition functions, and belongs to the field of active learning of object detectors in computer vision. The invention aims to solve the problem that an initialized object detector used in the existing active learning process cannot deeply mine the characteristics of valuable images. Comprising the following steps: detecting an image marked with a truth value or a pseudo truth value in a training set by adopting a weak supervision object detector trained based on an image-level label and a full supervision object detector initialized based on a partial instance-level label, and calculating an instance-level difficulty score, a class-level difficulty score and a class same weight according to a detection result; detecting an unlabeled instance-level label image, calculating an image-level difficulty score based on a class correlation weight and entropy, and determining a candidate image; and calculating the similarity of every two candidate images according to the detection result of the candidate images, determining the clustering center of the candidate images, and determining a valuable image after clustering. The invention can further refine a fully supervised object detector.

Description

Technical Field

[0001] The invention relates to an active learning method based on uncertainty and diversity acquisition function, and belongs to the field of active learning of object detectors in computer vision. Background Art

[0002] Compared with the traditional object detector method of performing instance-level annotation on all training set images, active learning object detection technology selects valuable images in the training set without instance-level annotation for instance-level annotation during the active learning stage, balancing the annotation cost and detection performance.

[0003] Active learning object detection technology mainly includes two stages: initialization stage and active learning stage. In order to ensure that valuable images are mined in the active learning stage, the existing active learning method introduces the entropy calculation method to select valuable images based on the object detector with preliminary detection ability in the initialization process. However, the object detector with preliminary detection ability depends to a large extent on the number of image-level labels or instance-level labels in the initialization process, and the strategy of relying solely on a single object detector to select the most valuable images is regarded as a weak constraint. Therefore, the limitation of the existing active learning object detection technology is that the object detector after active learning initialization only has preliminary detection capabilities and cannot deeply mine the features of valuable images. Summary of the invention

[0004] Aiming at the problem that the initialized object detector used in the existing active learning process cannot deeply mine the features of valuable images, the present invention provides an active learning method based on uncertainty and diversity acquisition function.

[0005] The present invention provides an active learning method based on uncertainty and diversity acquisition function, comprising:

[0006] The weakly supervised object detector trained based on image-level labels and the fully supervised object detector initialized with some instance-level labels are used to detect the images with true values ​​or false true values ​​in the training set. The detection results output by the weakly supervised object detector are The detection result output by the fully supervised object detector is y f ; Combine the true value or pseudo-true value accessed during the active learning initialization process to calculate the instance-level difficulty score; Update the instance-level difficulty score by exponential moving average to calculate the class-level difficulty score; Then perform nonlinear processing on the class-level difficulty score to obtain the class-related weight;

[0007] The unlabeled instance-level label images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels; the image-level difficulty score of the unlabeled instance-level label images is calculated based on the class-related weights and entropy, the image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level label images with high uncertainty scores are selected as candidate images; wherein b is a set constant;

[0008] The candidate images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels, and the detection results including object location, object category and object features are output; based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between every two candidate images;

[0009] According to the image similarity, K candidate images with low similarity to each other are selected as clustering centers. Based on the clustering centers, all candidate images are clustered based on cosine similarity and Euclidean distance. The clustering center of each cluster after clustering is used as a valuable image to refine the initialized fully supervised object detector; K is a set constant.

[0010] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, the instance-level difficulty score is calculated as follows:

[0011]

[0012] Where d is the instance-level difficulty score, y gt is the true value or pseudo-true value corresponding to the test result, P(y f |y gt ) ρ is the test result y f The classification results in , ρ is the exponential hyperparameter, is the positioning information in the detection result, and IoU is the intersection over union calculation function.

[0013] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, the calculation method of the class-level difficulty score is:

[0014]

[0015] In the formula is the class-level difficulty score g of the i-th class of objects i The k-th training iteration result is, is the exponential moving average momentum m of the i-th type object i The k-1th training iteration result is, represents the number of objects of the i-th category in the current batch in the k-th training iteration, d jrepresents the j-th instance-level difficulty score in the i-th class.

[0016] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, m i The initial value is m 0 , if there is an object of type i in the current batch, m 0 No change; if there is no object of type i in the current batch, m i The update formula is:

[0017]

[0018] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, the calculation method of class-related weights is:

[0019] ω i =1+λ·γlog(1+(e 1 / γ -1)·g i ),

[0020] Where ω i is the class-related weight of the i-th class of objects, λ is a hyperparameter that limits the upper bound of the class-related weight, and γ is a hyperparameter that controls the rate of change of the class-related weight.

[0021] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, the image-level difficulty score is calculated as follows:

[0022]

[0023] Where U(I) is the uncertainty acquisition function, N I is the number of objects predicted in the current unlabeled instance-level label image, ω i(j) is the class-related weight of the jth object predicted by the fully supervised object detector as the i-th class object, p ji is the classification probability of the jth object predicted as the i-th class object; C is the total number of object categories in the current unlabeled instance-level label image;

[0024] The image-level difficulty score is obtained based on the uncertainty acquisition function U(I).

[0025] According to the active learning method based on uncertainty and diversity acquisition function of the present invention, two candidate images are defined as I a and I b We use a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels to detect I a and I b For detection, the detection result output by the weakly supervised object detector is O' a and Ob ', the detection result output by the fully supervised object detector is O″ a and O″ b , then I a and I b The image similarity is:

[0026]

[0027] Where S' is the I calculated using the weakly supervised object detector a and I b The image similarity of S is calculated using the fully supervised object detector. a and I b The calculation method of S' and S" is the same:

[0028]

[0029] Where t a,j For image I a The classification result of the jth object in M a For image I a The total number of objects predicted in , o′ a,j Represents image I a The jth object in , sim(o′ a,j ,O b ') indicates o′ a,j with I b The cosine similarity of all similar objects in .

[0030] Beneficial effects of the present invention: The method of the present invention can effectively balance the annotation cost and detection performance by selecting valuable images to refine object detectors. The method of the present invention selects an initialization method based on a large visual model and a weakly supervised object detector in the initialization stage, and selects valuable images in the active learning stage, thereby solving the limitations of active learning object detection technology.

[0031] The method of the present invention performs active learning based on uncertainty and diversity acquisition functions, jointly considers the uncertainty and diversity of fully supervised object detectors and weakly supervised object detectors, and adds strict constraints to traditional acquisition functions. Compared with traditional active learning methods, the most valuable images selected by the method of the present invention can better cover the data distribution of unlabeled instance-level label images in the entire training set, and further refine the fully supervised object detector based on the images / labels of the initialization process and the images / labels mined by the active learning process.

[0032] The method of the present invention solves the problem that active learning object detection technology cannot deeply mine the features of valuable images during the active learning process, strengthens the representation ability of unlabeled images in the training set during the active learning process, and breaks through the limitation that the traditional active learning process only relies on a single fully supervised detector. The present invention belongs to the research on mining valuable images in the active learning process in active learning object detection technology, and to a certain extent promotes the implementation of object detection technology based on deep learning of artificial intelligence, which is in line with the development trend of contemporary intelligent manufacturing.

[0033] The method of the present invention benefits from the fact that the fully supervised object detector and the weakly supervised object detector can complement each other in precision and recall, and the weakly supervised object detector provides strict constraints for the traditional active learning acquisition function. Compared with the traditional active learning acquisition function, it mines more valuable images to further refine the fully supervised detector, achieving a balance between labeling cost and detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the active learning method for acquiring functions based on uncertainty and diversity according to the present invention;

[0035] Figure 2 This is a line chart comparing the performance of each round of the proposed method combined with RetinaNet and the existing SOTA method on the VOC2007 test set;

[0036] Figure 3 This is a line chart comparing the performance of each round of the proposed method combined with Deformable-DETR and the existing SOTA method on the VOC2007 test set;

[0037] Figure 4 This is a visualization result diagram of the detection results and true values ​​of the fully supervised object detector trained using the method of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0040] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0041] Specific implementation method 1. Combination Figure 1 As shown, the present invention provides an active learning method based on uncertainty and diversity acquisition function, including:

[0042] The weakly supervised object detector trained based on image-level labels and the fully supervised object detector initialized with some instance-level labels are used to detect the images with true values ​​or false true values ​​in the training set. The detection results output by the weakly supervised object detector are The detection result output by the fully supervised object detector is y f ; Combine the true value or pseudo-true value accessed during the active learning initialization process to calculate the instance-level difficulty score; Update the instance-level difficulty score by exponential moving average to calculate the class-level difficulty score; Then perform nonlinear processing on the class-level difficulty score to obtain the class-related weight;

[0043] The unlabeled instance-level label images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels; the image-level difficulty score of the unlabeled instance-level label images is calculated based on the class-related weights and entropy, the image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level label images with high uncertainty scores are selected as candidate images; wherein b is a set constant;

[0044] The candidate images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels, and the detection results including object location, object category and object features are output; based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between every two candidate images;

[0045] According to the image similarity, K candidate images with low similarity to each other are selected as clustering centers. Based on the clustering centers, all candidate images are clustered based on cosine similarity and Euclidean distance. The clustering center of each cluster after clustering is used as a valuable image to refine the initialized fully supervised object detector; K is a set constant.

[0046] Combination Figure 1 As shown in Figure 1, the weakly supervised object detector and the fully supervised object detector have learnable parameters that can be trained, and the other parts represent fixed functions and image / label information. w and a fully supervised object detector M f , using entropy to measure the classification results of fully supervised object detectors and weak / fully supervised object detectors M w / M fThe difficulty of positioning is taken into account, and the uncertainty score of the unlabeled image is calculated based on the classification confidence and the positioning quality. According to the uncertainty score, some images are selected as candidate images, and then valuable images are selected from the candidate images. This embodiment calculates the similarity as a proxy task for the diversity acquisition function. Similarly, the similarity of the detection results of the weakly supervised object detector and the detection results of the fully supervised detector are jointly considered. This embodiment calculates the similarity of the features of the same category of objects between images, and the maximum similarity value is used as the similarity score between the two images. Based on the calculated image similarity, diverse images are selected from the candidate images as valuable images to further refine the fully supervised object detector.

[0047] This implementation combines a weakly supervised object detector M w and a fully supervised object detector M f Increase the weight of difficult categories and reduce the weight of simple categories, so that the object detector focuses on the instances corresponding to the difficult categories. After these images are annotated, they are more helpful to improve the detection ability of the fully supervised object detector. Assuming that there are C categories in the data set, the present invention combines the classification confidence p of the fully supervised object detector and the positioning quality IoU of the weak / fully supervised detector to calculate the instance-level difficulty score d, wherein the classification confidence of the fully supervised object detector is obtained by the predicted value output by the classification head after a softmax operation, and the positioning quality of the weak / fully supervised object detector is obtained by selecting the intersection of the detection results of the weak / fully supervised object detector and the true value. The ratio IoU. It is worth noting that since the true value information of the unlabeled image cannot be accessed, the true value represents the true value / pseudo-true value that has been annotated in the active learning initialization stage based on the large visual language model and the weakly supervised object detector. The process of calculating the instance-level difficulty score d not only takes into account the weakly supervised object detector M w and a fully supervised object detector M f , and the classification and positioning quality are also taken into account. Then, the instance-level difficulty score d is updated by exponential moving average to calculate the class-level difficulty score g. The present invention defines the class-level difficulty score g, and the class-level difficulty score of each class is initialized to 1. The class-level difficulty score is then processed nonlinearly to obtain the class-related weights. Combined with the entropy calculation method in computer theory, the entropy of each object in the image is weighted and summed using the class-related weights, which is expressed as the uncertainty U of the calculated image. Based on the uncertainty U calculated by the above process, the uncertainty score is calculated for the images in the training set that are not labeled with instance-level labels. Next, a candidate pool containing b images is set, and b images with high uncertainty scores are selected from the images in the training set that are not labeled with instance-level labels and added to the candidate pool.

[0048] Next, after measuring the uncertainty score of each image, we further mine a specified number of K valuable images from the candidate pool. Consistent with the motivation of the uncertainty function, the fully supervised object detector in the initialization process has only preliminary detection capabilities and is weak in mining deep valuable image features. Thanks to the active learning initialization method using image-level labels, the weakly supervised object detector M trained with existing image-level labels is w It has good feature extraction capability. For two candidate images, the weakly supervised object detector M w and a fully supervised object detector M f Each outputs the detection result. Then the similarity of the two candidate images is calculated. The similarity is determined by matching the maximum similarity between each object and the same object in the other image. Based on the calculated similarity, K images are mined from the candidate pool as cluster centers. The clustering process is performed on all images in the candidate pool. When the clustering process is completed, the cluster center of each cluster is regarded as a valuable image.

[0049] This implementation improves the problem that traditional active learning methods cannot deeply mine the features of valuable images by relying on initialized object detectors. By combining the uncertainty and diversity of detection results of weakly supervised object detectors and fully supervised object detectors, strict constraints are introduced in the process of mining valuable images. The mined valuable images further refine the object detector, balancing the annotation cost and detection performance.

[0050] Furthermore, the weakly supervised object detector M w and a fully supervised object detector M f Considering the uncertainty of information measured by entropy in computer theory, entropy is used to measure the classification results of fully supervised object detectors and weak / fully supervised object detectors M w / M f The difficulty of positioning. However, the size of entropy is positively correlated with the number of objects in the image, and the weights of all categories are set to 1 in the process of calculating entropy, without considering the problem that different categories should be given different weights. To this end, the instance-level difficulty is first calculated, and the class-level difficulty is calculated by the instance-level difficulty. The class-related weights are obtained after nonlinear calculation of the class-level difficulty. The image-level uncertainty is calculated by combining entropy and class-related weights. For instance-level difficulty, the classification results output by the fully supervised object detector and the weakly supervised object detector and the intersection-and-union ratio with the true value are obtained. The classification result is obtained by passing the output result of the classification head of the fully supervised object detector through the softmax function, and the value with the largest intersection-and-union ratio of the full / weakly supervised object detector and the true value is selected as the positioning quality.

[0051] The instance-level difficulty score is calculated as:

[0052]

[0053] Where d is the instance-level difficulty score, y gt is the true value or pseudo-true value corresponding to the test result, P(y f |y gt ) ρ is the test result y f The classification results in , ρ is the exponential hyperparameter, is the positioning information in the detection result, and IoU is the intersection over union calculation function.

[0054] Among them, the constraint conditions of the difficulty of instance-level labels are increased through the max operation. The instances that are considered difficult in the traditional active learning process are considered simple in this embodiment. Therefore, the valuable images mined by the active learning acquisition function of this embodiment are more valuable / difficult than previous images.

[0055] The class-level difficulty score of each class is initialized to 1, and the class-level difficulty score is calculated as:

[0056]

[0057] In the formula is the class-level difficulty score g of the i-th class of objects i The k-th training iteration result is, is the exponential moving average momentum m of the i-th type object i The k-1th training iteration result is, represents the number of objects of the i-th category in the current batch in the k-th training iteration, d j represents the j-th instance-level difficulty score in the i-th class.

[0058] m i The initial value is m 0 , if there is an object of type i in the current batch, m 0 No change; if there is no object of type i in the current batch, m i The update formula is:

[0059]

[0060] The class-related weights are calculated as:

[0061] ω i =1+λ·γlog(1+(e 1 / γ -1)·g i ),

[0062] Where ω i is the class-related weight of the i-th class of objects, λ is a hyperparameter that limits the upper bound of the class-related weight, and γ is a hyperparameter that controls the rate of change of the class-related weight.

[0063] Next, we calculate the image-level difficulty score for each unlabeled instance-level label in the training set, expressed as a weighted sum of each detected object:

[0064]

[0065] Where U(I) is the uncertainty acquisition function, N I is the number of objects predicted in the current unlabeled instance-level label image, ω i(j) is the class-related weight of the jth object predicted by the fully supervised object detector as the i-th class object, p ji is the classification probability of the jth object predicted as the i-th class object; C is the total number of object categories in the current unlabeled instance-level label image;

[0066] The image-level difficulty score is obtained based on the uncertainty acquisition function U(I).

[0067] Going further, define two candidate images as I a and I b We use a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels to detect I a and I b For detection, the detection result output by the weakly supervised object detector is O' a and O b ', the detection result output by the fully supervised object detector is O″ a and O″ b , then I a and I b The image similarity is:

[0068]

[0069] Where S' is the I calculated using the weakly supervised object detector a and I b The image similarity of S is calculated using the fully supervised object detector. a and I b The calculation method of S' and S" is the same:

[0070]

[0071] Where t a,j For image I a The classification result of the jth object in M a For image I a The total number of objects predicted in , o′ a,j Represents image I aThe jth object in , sim(o′ a,j ,O b ') indicates o′ a,j with I b The cosine similarity of all similar objects in . sim() represents the cosine similarity function. When I a and I b When the object categories in do not co-occur, the similarity calculated for the current object is set to 0, otherwise the calculation I a and I b The cosine similarity of features of objects of the same category in .

[0072] Based on the calculated similarity, K images are mined from the candidate pool as cluster centers, where the K images need to satisfy a low similarity with other images in the candidate pool. Then, under the premise of taking K images as cluster centers, all images in the candidate pool are clustered into K clusters using cosine similarity and Euclidean distance. Until the clustering process is completed, the cluster center of each cluster is regarded as a valuable image to further refine the fully supervised object detector. Through carefully designed uncertainty and diversified acquisition functions, this implementation ultimately achieves a good balance between annotation cost and detection performance.

[0073] Verification experiment:

[0074] Prepare training samples. VOC2007 / 2012 and MS COCO 2017 datasets are selected to verify the effectiveness of the method of the present invention. The model proposed by the method of the present invention is further trained using a training set of 5011 images in the VOC2007 dataset and a training set of 11540 images in the VOC2012 dataset, and the accuracy of the model is evaluated using 4952 test images in the VOC2007 test set. In addition, the present invention uses COCO2017 to train and verify the model, the training set includes 118,000 images, the verification set includes 5,000 images, and there are 80 types of objects to be detected. Not all images in the VOC2007 / 2012 and COCO2017 training sets are used to train the model. In the VOC dataset, 2.5% of the image / instance-level labels are used for each round of active learning, and in the COCO dataset, 2.0% of the image / instance-level labels are used for each round of active learning. The experimental results follow the official evaluation indicators. The mAP accuracy of the model is evaluated on the test set for the VOC dataset, and the AP, AP50, and AP75 of the model are evaluated on the validation set for the COCO dataset.

[0075] The most advanced active learning initialization method currently available is used to enable the fully supervised object detector to have preliminary detection capabilities. MiniGPT-v2 is used as the visual language large model, OD-WSCL is used as the baseline weakly supervised object detector, and RetinaNet and Deformable-DETR are used as fully supervised detectors. ρ=0.3 is set to calculate the instance-level difficulty, and the number of active learning rounds r is set to 6 or 4 in the VOC dataset and COCO dataset respectively. m0 is set to 0.99, λ is set to 0.2, and γ is set to 0.3. The RetinaNet detector is trained for 26 epochs in each round of active learning process, and the learning rate is decayed to 0.1 times the original at the 20th epoch. The RetinaNet detector uses the SGD optimizer, and the initial learning rate is set to 0.01. The Deformable-DETR detector is trained for 50 epochs in each round of active learning process, and the learning rate is decayed to 0.1 times the original at the 40th epoch. The Deformable-DETR detector uses the AdamW optimizer, and the initial learning rate is set to 0.00002. The OD-WSCL detector is trained for 30K iterations using the SGD optimizer with an initial learning rate of 0.01. The code for this method is based on PyTorch and trained using 8 RTX3090 GPUs.

[0076] Experiments show that the proposed method improves the constraints of mining valuable images, mines more valuable images at a deeper level, and improves the detection performance of active learning object detection technology. The effectiveness of the proposed method is tested on the general VOC2007 / 2012 dataset and the VOC2007 test set to verify the effectiveness of the proposed method. Finally, the test results and true values ​​on the VOC dataset are shown.

[0077] 1) Analysis of the effectiveness of the active learning process based on the uncertainty and diversity acquisition function on the VOC2007 test set. As shown in Table 1, it can be observed that the RetinaNet detector achieves 59.1% mAP on the VOC2007 test set using the most advanced initialization method. On this basis, by introducing the uncertainty and diversity acquisition function designed for the active learning process by the method of the present invention, a mAP of 77.3% is achieved, and the detection performance is improved by 18.2%. Compared with the method of randomly selecting 15% of the image / instance-level labels and randomly selecting 15% of the image / instance-level labels + the pseudo-true value of the initialization process to train the fully supervised object detector, the method of the present invention exceeds the detection performance of the above two methods by 13.5% and 1.8%, respectively. Based on the above experimental process, the effectiveness of the active learning process proposed by the present invention in the process of mining valuable images is verified. According to the last two rows of Table 1, it is observed that the active learning relying only on the uncertainty acquisition function achieves a detection performance of 77.0%. When the diversity acquisition function is introduced, the active learning process achieves a detection performance of 77.3%, which further illustrates the effectiveness of the uncertainty and diversity acquisition function designed by the present invention.

[0078] Table 1 Analysis of the effectiveness of uncertainty and diversity acquisition functions on the active learning process on the VOC2007 test set

[0079]

[0080] 2) The active learning method of the present invention is compared with the SOTA weakly supervised object detector (OD-WSCL) and the fully supervised detector (RetinaNet Deformable-DETR) in the VOC dataset and the COCO dataset. It can be observed from Table 2 that OD-WSCL achieves 59.7%, 28.1% and 14.4% mAP, AP50 and AP (0.5: 0.95: 0.05), which is far behind the existing fully supervised object detector. Based on the method of the present invention, combined with the RetinaNet fully supervised object detector, 77.0%, 43.2% and 26.6% mAP, AP50 and AP (0.5: 0.95: 0.05) are achieved, relying only on 15% and 8% image / instance-level labels. It is worth noting that a%, b%, and c% indicate that the initialization process uses a% of instance-level labels, and each round of active learning mines c% of images as valuable images and annotates instance-level labels until the end of the active learning process, requiring a total of b% image / instance-level labels. In addition, in order to verify the generalization ability of the present invention, a Transformer-based fully supervised object detector (Deformable-DETR) was integrated, and 72.9% mAP was achieved using 15% instance-level labels, which is close to the fully supervised method. For a fair comparison, the present invention selected 15% and 8% image / instance-level labels to train RetinaNet and Deformable-DETR. As shown in the last four rows of Table 2, the present invention did select valuable images. The active learning method based on uncertainty and diversity designed by the present invention exceeded the mAP of the random selection method by about 10%, balancing the annotation cost and detection performance, relying only on a small number of instance-level labels, and close to the results of fully supervised object detection.

[0081] Table 2 Performance comparison of the proposed method and weak / fully supervised object detectors using different instance-level label ratios on the VOC2007 test set and COCO2017 validation set

[0082]

[0083] 3) The present invention is compared with other SOTA methods on the VOC2007 test set and the COCO2017 validation set. As shown in Table 3, it is observed that the present invention exceeds the previous SOTA methods. For example, RetinaNet combined with the designed active learning method achieves 77.3% mAP, which exceeds the existing SOTA method by 4.7% (72.6% vs 77.3%). Figure 2In the figure, the present invention visualizes the results compared with the existing SOTA methods. Since the initialization process uses an advanced active learning initialization method, the initialization process does not use any instance-level labels and only relies on image-level labels. Therefore, the present invention starts the active learning process with 0% instance-level labels. Figure 2 We show that our invention outperforms existing methods. For example, we outperform existing SOTA methods when accessing 2.5% of instance-level labels during active learning.

[0084] In order to verify the robustness of the present invention, the test results on the COCO dataset are shown in Table 3. Similar to the above experimental conclusions, the present invention surpasses the existing methods. Compared with the existing SOTA, the present invention improves AP50 from 38.7% to 45.6%, and improves AP (0.5:0.95:0.05) from 23.2% to 27.9%. Figure 3 As shown, RetinaNet and Deformable-DETR combined with the active learning method designed by the present invention only rely on 2% and 4% of instance-level labels, which exceeds the previous active learning method using 10% instance-level labels.

[0085] Table 3 Comparison of the proposed method with other SOTA methods on the VOC2007 test set and COCO2017 validation set

[0086]

[0087] 4) Visualization of test results and true values. Figure 4 The prediction results are shown. It can be observed in the VOC dataset and the COCO dataset that the active learning object detection technology of the present invention can locate the complete object area when some instances are occluded or dense. In addition, the present invention visualizes some failed examples, the main reason is that the object area has incorrect categories, that is, the pseudo-true values ​​output by the visual language large model in the initialization process have incorrect categories, which affects the performance of subsequent active learning.

[0088] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. An active learning method for acquiring functions based on uncertainty and diversity, characterized in that: include, The weakly supervised object detector trained based on image-level labels and the fully supervised object detector initialized with some instance-level labels are used to detect the images with true values ​​or false true values ​​in the training set. The detection results output by the weakly supervised object detector are The detection result output by the fully supervised object detector is y f ; Combine the true value or pseudo-true value accessed during the active learning initialization process to calculate the instance-level difficulty score; Update the instance-level difficulty score by exponential moving average to calculate the class-level difficulty score; The class-level difficulty scores are then processed nonlinearly to obtain class-related weights; The unlabeled instance-level label images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels; the image-level difficulty score of the unlabeled instance-level label images is calculated based on the class-related weights and entropy, the image-level difficulty score is used as the uncertainty score, and b unlabeled instance-level label images with high uncertainty scores are selected as candidate images; wherein b is a set constant; The candidate images are detected using a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels, and the detection results including object location, object category and object features are output; based on the detection results, the similarity between images is used as a proxy for the diversified acquisition function to calculate the image similarity between every two candidate images; According to the image similarity, K candidate images with low similarity to each other are selected as clustering centers. Based on the clustering centers, all candidate images are clustered based on cosine similarity and Euclidean distance. The clustering center of each cluster after clustering is used as a valuable image to refine the initialized fully supervised object detector; K is a set constant.

2. The active learning method based on uncertainty and diversity acquisition function according to claim 1, characterized in that: The instance-level difficulty score is calculated as: Where d is the instance-level difficulty score, y gt is the true value or pseudo-true value corresponding to the test result, P(y f |y gt ) ρ is the test result y f The classification results in , ρ is the exponential hyperparameter, max(IoU(y f ,y gt ), is the positioning information in the detection result, and IoU is the intersection over union calculation function.

3. The active learning method based on uncertainty and diversity acquisition function according to claim 2, characterized in that: The class-level difficulty score is calculated as: In the formula is the class-level difficulty score g of the i-th class of objects i The k-th training iteration result is, is the exponential moving average momentum m of the i-th type object i The k-1th training iteration result is, represents the number of objects of the i-th category in the current batch in the k-th training iteration, d j represents the j-th instance-level difficulty score in the i-th class.

4. The active learning method based on uncertainty and diversity acquisition function according to claim 3, characterized in that: m i The initialization value is m 0 , if there is an object of type i in the current batch, m 0 No change; if there is no object of type i in the current batch, m i The update formula is:

5. The active learning method based on uncertainty and diversity acquisition function according to claim 4, characterized in that: The class-related weights are calculated as: oh i =1+λ·γlog(1+(e 1 / γ -1)·g i ), Where ω i is the class-related weight of the i-th class of objects, λ is a hyperparameter that limits the upper bound of the class-related weight, and γ is a hyperparameter that controls the rate of change of the class-related weight.

6. The active learning method based on uncertainty and diversity acquisition function according to claim 5, characterized in that: The image-level difficulty score is calculated as: Where U(I) is the uncertainty acquisition function, N I is the number of objects predicted in the current unlabeled instance-level label image, ω i(j) is the class-related weight of the jth object predicted by the fully supervised object detector as the i-th class object, p ji is the classification probability of the jth object predicted as the i-th class object; C is the total number of object categories in the current unlabeled instance-level label image; The image-level difficulty score is obtained based on the uncertainty acquisition function U(I).

7. The active learning method based on uncertainty and diversity acquisition function according to claim 6, characterized in that: Define two candidate images as I a and I b We use a weakly supervised object detector trained based on image-level labels and a fully supervised object detector initialized with some instance-level labels to detect I a and I b For detection, the detection result output by the weakly supervised object detector is O' a and O b ', the detection result output by the fully supervised object detector is O' a ' and O b ”, then I a and I b The image similarity is: Where S' is the I calculated using the weakly supervised object detector a and I b The image similarity of S is calculated using the fully supervised object detector. a and I b The calculation method of S' and S" is the same: Where t a,j For image I a The classification result of the jth object in M a For image I a The total number of objects predicted in o a ' ,j Represents image I a The jth object in , sim(o′ a,j ,O b ') indicates o′ a,j with I b The cosine similarity of all similar objects in .

Images