A low-annotation-cost method for detecting heterogeneous objects

Abstract

The present invention discloses a method for detecting heterogeneous targets with low labeling cost, comprising: constructing an initial labeling set and an unlabeled set; using the initial labeling set to train a target detection model; using the current target detection network to infer the unlabeled set, calculating the uncertainty at the image level and the instance level based on the inference result, and sorting the instances according to the mixed uncertainty; counting the number of instances of each category in the labeled set, and then calculating the sampling rate to sample by category to obtain the sample set selected in the current round; labeling the sample set selected in the current round, updating the labeled set and the unlabeled set; using the current labeled set to train the target detection model; if the number of currently labeled samples is less than the overall labeling budget, returning to continue selecting samples in the unlabeled set according to the mixed uncertainty and category ratio; otherwise completing the target detection network training, the method can be used to detect heterogeneous targets in the detection image. The present invention achieves good target detection accuracy with low labeling cost.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a method for detecting heterogeneous targets with low annotation cost. Background Art

[0002] Urban food waste includes kitchen waste, food residues, and grease. Its high decomposability and potential environmental pollution make it an important challenge for urban environmental management. With the development of artificial intelligence, target detection technology has gradually been applied to the classification and treatment of urban food waste, and most of the current target detection methods rely on deep learning. It has been proven that this method is very effective in solving many computer vision problems when trained with a large amount of data. However, the requirement for the amount of labeled data also brings challenges to this type of algorithm that cannot be ignored. Therefore, using less labeled data to obtain higher detection accuracy has become one of the urgent problems to be solved in the current deep learning field.

[0003] Active learning has been proposed as a method to alleviate the model's demand for labeled samples. It selectively queries unlabeled examples with large amounts of information for labeling to reduce the labeling cost. In the field of object detection, active learning strategies mainly include uncertainty sampling and diversity sampling. Uncertainty sampling selects samples with low model prediction confidence for labeling to improve the model's ability to identify difficult examples. However, simple uncertainty sampling may cause minority class samples to be neglected, affecting the overall performance of the model. In particular, when there is a long-tail distribution problem in the dataset, that is, the number of minority class samples is far less than that of the majority class, this unbalanced distribution leads to the model's poor detection ability for minority classes. Summary of the invention

[0004] The present invention provides a method for detecting heterogeneous objects with low annotation cost, which selects samples with high information content for annotation by evaluating the uncertainty at the image level and instance level, thereby achieving good object detection accuracy at a low annotation cost.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A low-annotation-cost heterogeneous object detection method comprising:

[0007] Step 1: Collect images with special-shaped objects to construct a target detection dataset, and randomly select images with an initial annotation budget size for manual annotation to form an initial annotation set. , the remaining large number of unlabeled images constitute the unlabeled set ;

[0008] Step 2: Use the initial annotation set Train the object detection model;

[0009] Step 3: Use the currently trained target detection network to detect the current unlabeled set The image in the inference is used to calculate the image-level uncertainty and instance-level uncertainty based on the inference results. The instances in are sorted in descending order of the size of the mixed uncertainty;

[0010] Step 4: Count the current labeled set The number of instances of each category in , the category ratio is inverted to obtain the sampling rate of instances of each category, and according to the sampling rate, samples are performed by category from the instances sorted in step 3 to obtain the sample set selected in the current round;

[0011] Step 5: Manually label the sample set selected in the current round to obtain the labeled sample set , and then update the labeled set to , the unlabeled set is ;

[0012] Step 6: Use the current labeled set Train the object detection model;

[0013] Step 7: If the number of currently labeled samples is less than the overall labeling budget, return to step 3; otherwise, end active learning and model training and proceed to step 8;

[0014] Step 8: Use the trained target detection network to detect the heterogeneous targets in the image to be detected.

[0015] Further, the target detection model includes an image encoder, a diffusion model and a detection encoder;

[0016] The image encoder uses Resnet50 as the backbone network to extract the features of the input image, and then combines the feature pyramid network to fuse the features at different levels to generate a multi-scale feature map;

[0017] The diffusion model is used to generate a noise frame;

[0018] The detection encoder crops regions of interest from the multi-scale feature map according to the noise frames generated by the diffusion model, and performs frame regression and classification according to the regions of interest;

[0019] The training method of the diffusion model is:

[0020] First, define a Markov chain of a diffusion forward process, gradually add noise to the sample data, and the forward noise addition process is defined as q:

[0021]

[0022] In the formula, represents the ground-truth bounding box, represents the noise frame at time t, , T is the total number of diffusion steps, through Add noise, Transformed into the noise frame at time t ; is the cumulative proportion of signal retention from time 0 to t, , is the proportion of the signal retained at time step s, represents the noise variance table; I represents the unit matrix, represents Gaussian distribution;

[0023] Then the diffusion model By from the noise box Predict the output bounding box and iteratively train the diffusion model using the following loss function;

[0024]

[0025] In the formula, Represents the loss function value.

[0026] Furthermore, the object detection network's inference results for the input image include the bounding boxes and categories of special-shaped objects. The categories of special-shaped objects include: garbage bags, metals, shoes, plastic bottles, other plastics, cartons, Tetra Pak packaging, and glass.

[0027] Furthermore, the image-level uncertainty is calculated as follows:

[0028]

[0029] in, Represents the unlabeled set The images The image-level uncertainty, is the set threshold, is an image Middle The class prediction probability vector of the instances within the predicted bounding box, Representing images The maximum class confidence score in The index of the predicted bounding box, , For images The number of predicted bounding boxes in the corresponding inference results.

[0030] Furthermore, the calculation formula of instance-level uncertainty is as follows:

[0031]

[0032] in, Represents the unlabeled set The images No. The instance-level uncertainty of the predicted bounding boxes, is an image The The predicted bounding box is The predicted probability of the class, Represents the number of categories of heterogeneous objects.

[0033] Furthermore, the image uncertainty is combined with the instance-level uncertainty to obtain the image Middle Mixed uncertainty of instances within the predicted bounding box :

[0034] , ,

[0035] in, and They represent the unlabeled set The images The image-level uncertainty and instance-level uncertainty of Represents the unlabeled set The number of images in Representing images The number of predicted bounding boxes in the corresponding inference results.

[0036] Furthermore, the statistics of the currently labeled set The number of instances of each category in , and the sampling rate of each category instance is obtained by inverting the category ratio, including:

[0037] (1) Counting the labeled set The number of instances of each category in Indicates The number of class instances, , C represents the number of categories of heterogeneous targets;

[0038] (2) Based on Calculate the distribution probability of each category in the label set, and then calculate the class weight to be sampled:

[0039]

[0040] In the formula, Indicates the number of samples to be sampled class weights of class instances;

[0041] (3) Class weights Perform normalization to obtain the sampling rate of each category instance in the current round .

[0042] The low-annotation-cost heterogeneous target detection method provided by the present invention effectively reduces the dependence of the training target detection model on the amount of annotated data, and achieves good detection accuracy with less annotation cost. The hybrid sampling strategy is adopted to select high-information samples while considering the distribution balance problem between categories. Compared with the existing target detection method based on active learning, the present invention has the following advantages:

[0043] 1. Consider both image-level uncertainty and instance-level uncertainty to obtain high-information images, which contain global and local information respectively, and take into account the spatial information and semantic structure of the image for more comprehensive data evaluation.

[0044] 2. A class distribution balancing strategy is adopted to emphasize instance categories with low occurrence frequency in active queries. In the sampling phase, objects are selectively queried in classes with fewer objects, which solves the long-tail class distribution problem in the dataset.

[0045] 3. It is practical. The present invention has conducted sufficient experiments on the urban kitchen waste target detection dataset OD-KW, which improves the performance of the model while optimizing the annotation budget. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall framework of the heterogeneous target detection method of the embodiment of the present invention.

[0047] Figure 2 The figure is a flow chart of a method for detecting a heterogeneous target according to an embodiment of the present invention.

[0048] Figure 3 The figure shows the AP50 comparison between the method of the embodiment of the present invention and other active learning-based comparison methods on the OD-KW validation set.

[0049] Figure 4 The figure shows the mAP comparison between the method of the embodiment of the present invention and other active learning-based comparison methods on the OD-KW validation set.

[0050] Figure 5 The figure shows the mAP comparison of active learning strategies at different sampling stages on the OD-KW validation set according to the embodiments of the present invention. DETAILED DESCRIPTION

[0051] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0052] The technical solution of the present invention is an active learning strategy based on mixed uncertainty and class distribution balance to realize a target detection method with fewer labeled samples. The Python programming language can be used for experiments, and the C / C++ programming language can also be used for engineering applications.

[0053] The embodiment of the present invention provides a method for detecting heterogeneous objects with low annotation cost. Figure 1 , see the attached flow chart Figure 2 , including the following steps:

[0054] Step 1: Collect images with special-shaped objects to construct a target detection dataset, and randomly select images with an initial annotation budget size for manual annotation to form an initial annotation set. , the remaining large number of unlabeled images constitute the unlabeled set .

[0055] The special-shaped targets in this embodiment are various types of kitchen waste, and the target detection data set OD-KW is constructed. The embodiment collects 15,994 images of actual kitchen waste sorting scenes in different months of a year in the sorting production line of a kitchen waste treatment plant. In order to ensure the richness and representativeness of the time distribution and category distribution of the data set, 3,771, 4,735, 2,130 and 4,358 images of the four seasons of spring, summer, autumn and winter are collected respectively. In order to meet the configuration of the laboratory hardware platform, all images are adjusted from the original resolution of 3256×2724 to 640×480. After investigating the subsequent processing of kitchen waste, the data set is divided into eight categories of targets, including garbage bags, metals, shoes, plastic bottles, other plastics, cartons, Tetra Pak packaging, and glass. The data set is open, allowing new categories to appear in the subsequent active learning process. To optimize the implementation process, 3149 images were fully annotated manually before using the dataset, so that they could be directly called by the manual annotation process in subsequent experiments to improve efficiency. The dataset was divided into a training set and a validation set in a ratio of 8:2, and 2495 images were obtained for the training set and 654 images for the validation set.

[0056] Set the initial annotation budget to 20% of the training set images, that is , the remaining unlabeled images ,The initial annotation set is selected randomly.

[0057] Step 2: Use the initial annotation set Train the object detection model and test its performance on the validation set.

[0058] The target detection model described in the embodiment adopts the DiffusionDet diffusion detection model. The diffusion model generates data samples through iteration, and the model needs to be run multiple times in the inference stage. However, in each iteration, This makes the calculation difficult, so the entire model is divided into two parts, the image encoder and the detection decoder. The former is run only once to extract a deep feature representation from the original input image x, and the latter uses this deep feature as a condition instead of the original image to extract a deep feature representation from the noise frame. Optimize box prediction step by step.

[0059] The image encoder is used for feature extraction and processing. Resnet50 is used as the backbone network to extract image features. It is then combined with the feature pyramid network to fuse features at different levels to generate a multi-scale feature map. The detection decoder takes a set of candidate boxes as input, crops the region of interest from the generated feature map, and sends the features of these regions of interest to the detection head to obtain box regression and classification results.

[0060] The experiment was conducted on Ubuntu 20.04.1LTS system, the hardware platform was GeForce RTX 2080 Ti, the configuration environment was pytorch1.13.1+cuda11.7, python3.9. The parameters in the training process were set as follows: active learning for a total of 5 rounds, the maximum number of iterations per round was 22500, the number of images processed in each batch was 4, the basic learning rate was 0.000025, and the weight decay coefficient was 0.0001.

[0061] Step 3: Use the currently trained target detection network to detect the current unlabeled set The image in the inference is used to calculate the image-level uncertainty and instance-level uncertainty based on the inference results. The instances in are sorted in descending order of the size of the mixed uncertainty.

[0062] The problem definition of using active learning for heterogeneous object detection is to train a good detector by selecting information-rich samples from a large unlabeled set to reduce the labeling cost. It contains three sets of data definitions: a small fully labeled set for initializing the model , a large unlabeled set for data selection , and the updated label set through active learning query , the total tag set is , expressed as:

[0063]

[0064] in, Represents the complete tag set The images, , In the label space The one-hot class label of one of the known classes, For the bounding box labels, is the number of images with labels.

[0065] Unlabeled set It is expressed as:

[0066]

[0067] Refers to the number of images in the unlabeled dataset, in the initial state .

[0068] make For object detection models on unlabeled sets The prediction on is expressed as:

[0069]

[0070] in represents the prediction result of the target detection model, is the class prediction probability vector from the classification head, is the predicted bounding box position from the regression head, which is the output of the Diffusiondet head network.

[0071] The first stage of active learning uses mixed uncertainty for sampling. When an image contains many target instances with high uncertainty, this image should be selected as a candidate for annotation. The uncertainty values ​​predicted by the most reliable model are evaluated and aggregated to represent the uncertainty of the entire image. , image uncertainty The calculation formula is as follows:

[0072]

[0073] in , represents the image The maximum class confidence score in The index of the predicted bounding box, is the set threshold.

[0074] The image of urban kitchen waste contains multiple instances. In order to consider instance-level information in the query, entropy is used to evaluate the uncertainty of each predicted bounding box. The calculation formula is as follows:

[0075]

[0076] in is an image The jth bounding box in The predicted probability of the class.

[0077] Next, the image-level uncertainty and instance-level uncertainty Combined to get the final information score of the instance, that is, mixed uncertainty , expressed as:

[0078]

[0079] According to the mixed uncertainty from high to low, the current unlabeled set Sort the instances in to get the sample set for the current round of active learning.

[0080] Step 4: Count the current labeled set The number of instances of each category in , the category ratio is inverted to obtain the sampling rate of instances of each category, and according to the sampling rate, samples are performed by category from the instances sorted in step 3 to obtain the sample set selected in the current round.

[0081] In order to solve the long-tail data problem in the kitchen waste dataset, a class distribution balance strategy is used to highlight rare category objects. First, the rare categories are identified by counting the instances of each category in the labeled set. Indicates The number of class instances, , by imposing a Inversely proportional preference , to query more objects in categories with fewer instances, is defined as follows:

[0082]

[0083] in, .

[0084] Then based on Calculate the distribution probability of each category in the label set, and then fill it in during the sampling process to get the class weight Finally, in order to ensure that the number of sampled images does not exceed the budget, the class weights are normalized to obtain the sampling rate of each category instance in the current round. .

[0085] Then Multiplying this by the current round’s budget gives us the expected class distribution during sampling. If the annotation budget for a class is exhausted, the remaining objects predicted to belong to that class are discarded to get the final sample set.

[0086] Step 5: Manually label the sample set selected in the current round to obtain the labeled sample set , and then update the labeled set to , the unlabeled set is .

[0087] Step 6: Use the current labeled set Train an object detection model.

[0088] Step 7: If the current number of labeled samples is less than the overall labeling budget, return to step 3; otherwise, end active learning and model training and proceed to step 8.

[0089] In this embodiment, 15% of instance samples are selected in each round. When the overall annotation budget is exhausted, active learning is terminated and the corresponding model parameters are saved.

[0090] Step 8: Use the trained target detection network to detect the heterogeneous targets in the detection image.

[0091] When the sample budget of the active learning cycle is exhausted, all the selected samples are used to train the object detection model, and the model performance is tested on the validation set. Finally, the relevant training parameters and results of the model are saved.

[0092] To facilitate understanding of the technical effects of the present invention, the following is a comparison of the application of the method of the present invention and the traditional method:

[0093] Table 1, Table 2 and Appendix Figure 3 , Attachment Figure 4The AP50 and mAP results of the method of the present invention on the OD-KW validation set are compared with those of other active learning-based methods. The comparison methods based on other active learning include random sampling (Random), sequential sampling (SE), VAAL and CALD. As can be seen from Tables 1 and 2, in the five rounds of active learning cycles, the detection effect of the method of the present invention is better than that of other active learning methods under the same settings. In the first three rounds of selecting samples by active learning strategy, the detection accuracy of the method of the present invention is better than that of other active learning methods, which proves the effectiveness of the present invention. When the amount of labeled data is 35%, the detection index mAP of the method of the present invention on the OD-KW validation set is more than 1% higher than that of other active learning methods. As the amount of labeled data increases, when it reaches 80%, the mAP of the method of the present invention is equal to that of the CALD method and higher than that of the Random, SE and VAAL methods, which shows that the method of the present invention has more obvious advantages when the amount of labeled data is small, and can select more informative samples to effectively improve the performance of the target detection model.

[0094]

[0095]

[0096] In order to further demonstrate the effectiveness of the two active learning strategies, the hybrid uncertainty strategy and the class distribution balance strategy proposed by the method of the present invention, the present invention conducted an ablation experiment on the OD-KW dataset. The experimental results are shown in the attached figure. Figure 5 The results show that the detection effect of using the hybrid uncertainty strategy or the class distribution balance strategy alone is worse than that of using the hybrid uncertainty strategy and the class distribution balance strategy for sample selection at the same time, which illustrates the effectiveness of the combination of the two strategies in the method of the present invention.

[0097] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A low-annotation-cost heterogeneous target detection method, characterized in that: include: Step 1: Collect images with special-shaped objects to construct a target detection dataset, and randomly select images with an initial annotation budget size for manual annotation to form an initial annotation set. , the remaining large number of unlabeled images constitute the initial unlabeled set ; Step 2: Use the initial annotation set Train the object detection model; Step 3: Use the currently trained target detection network to detect the current unlabeled set The image in the inference is used to calculate the image-level uncertainty and instance-level uncertainty based on the inference results. The instances in are sorted in descending order of the size of the mixed uncertainty; Step 4: Count the current labeled set The number of instances of each category in , the category ratio is inverted to obtain the sampling rate of instances of each category, and according to the sampling rate, samples are performed by category from the instances sorted in step 3 to obtain the sample set selected in the current round; Step 5: Manually label the sample set selected in the current round to obtain the labeled sample set , and then update the labeled set to , the unlabeled set is ; Step 6: Use the current labeled set Train the object detection model; Step 7: If the number of currently labeled samples is less than the overall labeling budget, return to step 3; otherwise, end active learning and model training and proceed to step 8; Step 8: Use the trained target detection network to detect the heterogeneous targets in the detection image.

2. According to the low-annotation-cost special-shaped target detection method of claim 1, it is characterized in that: The target detection model includes an image encoder, a diffusion model and a detection encoder; The image encoder uses Resnet50 as the backbone network to extract the features of the input image, and then combines the feature pyramid network to fuse the features at different levels to generate a multi-scale feature map; The diffusion model is used to generate a noise frame; The detection encoder crops regions of interest from the multi-scale feature map according to the noise frames generated by the diffusion model, and performs frame regression and classification according to the regions of interest; The training method of the diffusion model is: First, define a Markov chain of a diffusion forward process, gradually add noise to the sample data, and the forward noise addition process is defined as q: ； In the formula, represents the ground-truth bounding box, represents the noise frame at time t, , T is the total number of diffusion steps, through Add noise, Transformed into the noise frame at time t ; is the cumulative proportion of signal retention from time 0 to t, , is the proportion of the signal retained at time step s, represents the noise variance table; I represents the identity matrix, represents Gaussian distribution; Then the diffusion model By from the noise box Predict the output bounding box and iteratively train the diffusion model using the following loss function; ； In the formula, Represents the loss function value.

3. According to the low-annotation-cost special-shaped target detection method of claim 1, it is characterized in that: The object detection network's inference results for the input image include the bounding boxes and categories of the special-shaped objects. The categories of special-shaped objects include: garbage bags, metals, shoes, plastic bottles, other plastics, cartons, Tetra Pak packaging, and glass.

4. The low-annotation-cost special-shaped target detection method according to claim 1, characterized in that: The image-level uncertainty is calculated as follows: ； in, Represents the unlabeled set The images The image-level uncertainty, is the set threshold, is an image Middle The class prediction probability vector of the instances within the predicted bounding box, Representing images The maximum class confidence score in The index of the predicted bounding box, , For images The number of predicted bounding boxes in the corresponding inference results.

5. The low-annotation-cost special-shaped target detection method according to claim 1, characterized in that: The calculation formula for instance-level uncertainty is as follows: ； in, Represents the unlabeled set The images No. The instance-level uncertainty of the predicted bounding boxes, is an image Middle The predicted bounding box is within the The predicted probability vector on the class, is an image The The predicted bounding box is The predicted probability of the class, C represents the number of classes of heterogeneous targets.

6. The low-annotation-cost special-shaped target detection method according to claim 1, characterized in that: Combining image-level uncertainty with instance-level uncertainty to obtain image Middle Mixed uncertainty of instances within the predicted bounding box : ， ， ； in, and Represent the unlabeled set The images The image-level uncertainty and instance-level uncertainty of Represents the unlabeled set The number of images in Representing images The number of predicted bounding boxes in the corresponding inference results.

7. The low-annotation-cost special-shaped target detection method according to claim 1, characterized in that: The statistics are currently labeled set The number of instances of each category in , and the sampling rate of each category instance is obtained by inverting the category ratio, including: (1) Counting the labeled set The number of instances of each category in Indicates The number of class instances, , C represents the number of categories of heterogeneous targets; (2) Based on Calculate the distribution probability of each category in the label set, and then calculate the class weight to be sampled: ； In the formula, Indicates the number of samples to be sampled class weights of class instances; (3) Class weights Perform normalization to obtain the sampling rate of each category instance in the current round .

Images