Novel object detection method and system based on sparse labeled dataset
By employing an exponential moving average teacher detector and a co-teacher detector in sparse labeled datasets, combined with a fully convolutional single-level object detector FCOS, a pseudo-label set is generated and corrected, thus solving the problem of pseudo-label noise on sparse labeled datasets and achieving efficient and accurate object detection.
Patent Information
- Application Number
- CN202310174614.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-16
- Publication Date
- 2026-05-15
- Estimated Expiration
- 2043-02-16
AI Technical Summary
In sparsely labeled datasets, existing object detection algorithms are susceptible to the influence of false label noise signals, especially erroneous bounding box category and location information, leading to detector overfitting. Furthermore, traditional methods require significant human and time resources, and existing dual-branch methods perform poorly when color enhancement is insufficient.
We employ a teacher detector based on exponential moving average and a co-teacher detector with dual image input ports. By generating and correcting pseudo-label sets and combining different levels of data augmentation, we filter out erroneous pseudo-labels. We then train the model using the fully convolutional single-level object detector FCOS to generate a high-quality pseudo-label set for supervising the model.
It improves object detection performance on sparsely labeled datasets, reduces training and inference time, reduces the impact of pseudo-label noise, enhances small object detection performance, and maintains high performance under varying degrees of missing labels.
Smart Images
Figure CN116342971B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning and computer vision technology, and more specifically, relates to a novel object detection method and system based on sparse labeled datasets. Background Technology
[0002] Object detection, a fundamental task in computer vision, requires detectors to predict the bounding boxes and object categories for all foreground objects in the output image. Recently, with the rapid development of deep convolutional neural networks, numerous object detection methods have emerged, the performance of which largely depends on fully labeled object detection datasets. However, detailed labeling of all possible foreground objects in each image is both difficult and expensive, and incomplete box annotations often lead to a decline in object detector performance.
[0003] Commonly used object detectors typically achieve good results with supervised learning on fully labeled datasets. However, when datasets contain a large number of images and categories, labeling all given object instances (especially in crowded scenes) becomes extremely difficult. In industrial datasets, the cost and difficulty of labeling often significantly impact the quality of the labeled data. For datasets with incomplete labeling, i.e., sparse labeling, the results obtained using general object detection methods are often unsatisfactory because unlabeled objects provide incorrect supervision signals to the detector during training. The detector gradually converges to a suboptimal hyperplane that tends to separate labeled objects from unlabeled objects and the background. Therefore, compared to separating foreground objects from background regions, the detector is more likely to overfit to a suboptimal hyperplane due to the presence of more noise. The traditional approach is to develop more refined labeling strategies, increase the number of labelers and labeling time to improve labeling quality, thereby improving the performance of object detectors. However, such approaches require significant human resources and high time costs, significantly extending the project and engineering cycle.
[0004] While existing techniques can achieve decent object detector performance on sparsely labeled datasets, they all have some limitations. Soft-Sampling methods based on two-level detectors reduce the backpropagation gradient of each region of interest (ROI) in the second level according to the Gomboz function of its overlap with the bounding box. However, this method also reduces gradients from the real background, negatively impacting network learning. PPM introduces a Siamese network into the two-level detector and merges the two region generation networks of the Siamese network into a single region generation network to obtain ROIs. Then, they use ROIs with high objectivity scores and low intersection-union ratios with the bounding boxes as unlabeled instances in the second-stage training process. This method relies on a region generation network that performs relatively well on sparsely labeled datasets and is not suitable for single-level detectors. Although two-level detectors are robust to environments with sparse annotations, they typically require higher-resolution image inputs, making them slower than single-level detectors. Furthermore, they require tuning more predefined and manually designed hyperparameters to achieve optimal performance.
[0005] Background Recalibration Loss (BRL) and Co-mining inherit the high efficiency and simplicity of single-stage detectors. The former redesigns the FocalLoss loss function of single-stage detectors to reduce the negative impact of erroneous supervision. The latter introduces Siamese networks into the sparsely labeled object detection problem of single-stage detectors, extracting features from two different data-augmented input images through two encoders sharing parameters. Each branch generates pseudo-labels, which are then merged with sparse annotations, serving as mutual supervision signals between the two branches. This strategy has achieved good results on the sparsely labeled object detection task of single-stage detectors, but two key issues remain that have not been fully considered: whether simple data augmentation methods such as blurring and color jitter are sufficient to mine more pseudo-labels that can positively influence the model training process by learning different multi-view feature representations; and how to handle erroneous pseudo-labels generated in the sparsely labeled object detection algorithm of single-stage detectors. Summary of the Invention
[0006] This invention primarily concerns object detection on sparsely labeled datasets, specifically training a high-performance object detector on a training dataset with partially missing labels. The objective of this invention is to develop a single-stage object detector algorithm with improved performance when trained on sparsely labeled datasets. This algorithm aims to address the problems of low diversity in pseudo-labels generated by dual-branch generation and the presence of pseudo-labels with incorrect category or location markings in past algorithms, thereby improving the performance of single-stage object detectors in sparsely labeled object detection tasks.
[0007] Problem Description: Previous work used color dithering data augmentation (including random adjustments to brightness, contrast, saturation, and hue) to adjust the color characteristics of the input image in a branch of a Siamese network. They aimed to obtain different pseudo-label sets from two different inputs to recover lost annotations as much as possible. However, data augmentation at the color level alone may not be sufficient; sometimes adjustments to physical properties such as size and shape can also yield good results. Qualitative analysis reveals that feeding similar augmented images into two branches of a network with shared parameters results in highly similar predicted pseudo-label sets. In this case, joint training is almost equivalent to doubling the gradient, which is not very helpful for training the model on sparsely labeled datasets. Furthermore, data augmentation in the color space is not always effective, especially for small objects. For sparsely labeled object detectors based on pseudo-label methods, they are susceptible to noise signals in the pseudo-labels, including incorrect bounding box category information and incorrect bounding box location information, and such effects tend to accumulate and amplify with increasing training iterations.
[0008] According to one aspect of the present invention, a novel object detection system based on a sparsely labeled dataset is provided, comprising a teacher detector based on an exponential moving average and a co-teacher detector with dual image input ports, wherein:
[0009] The teacher detector does not backpropagate gradients during training, but only updates parameters by exponential moving average at the end of each training iteration. The teacher detector updated by exponential moving average is a smoothed version of the dual-input port secondary teacher detector.
[0010] The input images for the teacher detector and the dual-input sub-teacher detector are the original, weakly enhanced, and strongly enhanced images, respectively. The input image for the teacher detector is the original image, which is fed into the teacher detector to generate the teacher pseudo-label set. The weakly enhanced and strongly enhanced images are fed into the sub-teacher detector with two input ports to generate the original pseudo-label set.
[0011] Each original pseudo-label set is corrected and integrated by the pseudo-label set output by the teacher detector and the sparse bounding boxes. Finally, the corrected and integrated bounding boxes are used to supervise the prediction output of the co-teacher detector.
[0012] In one embodiment of the present invention, both the teacher detector and the co-teacher detector are implemented using a fully convolutional single-stage object detector (FCOS).
[0013] In one embodiment of the present invention, the input images of the teacher detector and the dual-input co-teacher detector are the original, weakly enhanced, and strongly enhanced images, respectively, and the three sets of classification score maps, offset maps, and centrality maps are predicted and output by two single-level target detectors FCOS.
[0014] In one embodiment of the present invention, three sets of pseudo-labels with category information and location information are generated based on the prediction graphs output by each branch. Specifically, the final pseudo-label set is obtained by filtering out pseudo-label boxes with low target scores and redundant boxes with high overlap. Based on the pseudo-label set of the teacher network and the existing sparse labeled box set, the pseudo-label sets output by the two branches A and B of the co-teacher network are corrected to filter out pseudo-label boxes with incorrect category information or incorrect location information as much as possible. The supervision signals of the two branches are output to supervise the predicted values output by the co-teacher network.
[0015] In one embodiment of the present invention, the target score is the classification score multiplied by the centrality.
[0016] In one embodiment of the present invention, the parameters of the co-teacher network are updated by the gradient backpropagated from the loss function between the supervision signal and the prediction output, and the parameters of the teacher network are updated by the exponential moving average (EMA).
[0017] According to another aspect of the present invention, a novel object detection method based on a sparsely labeled dataset is also provided, the method comprising the following steps:
[0018] (1) Construct a secondary teacher detector with dual image input ports and a teacher detector whose parameters are updated by exponential moving average. At the start of training, assign the same pre-trained model parameters to both.
[0019] (2) Perform two different levels of data augmentation on sparsely labeled training images;
[0020] (3) Input the original image, weak data augmentation image and strong data augmentation image into the teacher detector branch and the two branches of the assistant teacher detector respectively. Then, predict and output three sets of classification score images, offset images and centrality images through the teacher detector branch and the two branches of the assistant teacher detector respectively.
[0021] (4) Generate pseudo-label boxes for each reference point based on the classification score map and offset map of each group, filter out all pseudo-label boxes whose product of classification score and centrality is less than a preset threshold, and remove redundant boxes with excessive overlap by non-maximum suppression algorithm to obtain three pseudo-label sets.
[0022] (5) Based on the obtained pseudo-label set T of the teacher detector, correct the pseudo-label sets A and B output by the assistant teacher detector, and transform all pseudo-labels in set T into the transform domains of A and B respectively.
[0023] (6) Transform the pseudo-label set Ta / Tb after the transformation domain, take all pseudo-labels in Ta / Tb whose intersection-union ratio with A and B is less than the set threshold as the missing label boxes, and replace all pseudo-labels in A and B whose intersection-union ratio with a certain pseudo-label Pi in Ta / Tb is greater than the preset threshold and whose classification score is smaller than Pi with Pi.
[0024] (7) Merge the corrected pseudo-label sets A` and B` with the sparsely labeled bounding boxes respectively, remove all pseudo-labels whose intersection-union ratio with the bounding boxes is greater than a preset threshold, and obtain two sets of supervision signals. Supervised learning is then performed on the prediction information output by the corresponding two branches.
[0025] (8) After training for a preset period according to the above steps, the obtained detector model is tested on a fully labeled test set to test its detection performance. After reaching the preset performance, the above detector model is used to perform target detection on the image to be detected.
[0026] In one embodiment of the present invention, the data enhancement in step (2) includes:
[0027] One type includes random horizontal flipping and color dithering data augmentation, while the other includes a series of random strong data augmentations such as random horizontal flipping, color dithering data augmentation, random equalization, random sharpening, random translation, random rotation, random shearing, and random erasure.
[0028] In one embodiment of the present invention, both the teacher detector and the co-teacher detector are implemented using a fully convolutional single-stage object detector (FCOS).
[0029] In one embodiment of the present invention, the sparse labeled dataset is an MSCOCO training set with multiple sparse labels of different degrees.
[0030] Compared with the prior art, the technical solution conceived in this invention has the following beneficial effects:
[0031] (1) The present invention is based on a single-level target detector, which has faster training and inference speed and fewer hyperparameter settings that need to be adjusted.
[0032] (2) This invention improves the difference of the pseudo-label set of the dual-branch output during model training and reduces the proportion of noise samples in the pseudo-label set. While maintaining the original inference speed and sacrificing less training speed (about 30%), it can obtain a higher performance target detector in sparse labeled target detection tasks.
[0033] (3) In datasets with varying degrees of missing targets (approximately 20%-65% of foreground targets are missing), the present invention has achieved a superior target detector, and the performance improvement of the algorithm is not diminished by the increase in the percentage of missing targets over a wide range.
[0034] (4) The present invention has a significant improvement in the performance of sparsely labeled target detection for small targets. Attached Figure Description
[0035] Figure 1 This is an overall framework diagram of a novel object detection system based on a sparse labeled dataset in an embodiment of the present invention. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0037] This invention employs a single-stage object detector, FCOS (Fully Convolutional One-Stage), as the basic detector and compares its performance with that of co-mining on several MS COCO datasets with varying degrees of sparse labeling, demonstrating the effectiveness of the invention. Specifically, this invention provides a novel object detection system based on sparsely labeled datasets. This system consists of a teacher detector based on exponential moving average and a co-teacher detector with dual image input ports. We use FCOS as the basic detector. The teacher detector does not backpropagate gradients during training but only updates its parameters through exponential moving average at the end of each training iteration. The teacher detector updated by exponential moving average is a smoothed version of the co-teacher detector with dual input ports.
[0038] The input images for the teacher detector and the dual-input sub-teacher detector are the original, weakly enhanced, and strongly enhanced images, respectively. The input image for the teacher detector is the original image, which is fed into the teacher detector to generate the teacher pseudo-label set. The weakly enhanced and strongly enhanced images are fed into the sub-teacher detector with two input ports to generate the original pseudo-label set.
[0039] Each original pseudo-label set is corrected and integrated from the pseudo-label set output by the teacher detector and sparse bounding boxes. Finally, the corrected and integrated bounding boxes are used to supervise the prediction output of the assistant teacher detector. Specifically, such as... Figure 1As shown, the input images for the three branches from top to bottom are the original, weakly enhanced, and strongly enhanced images, respectively. Two fully convolutional single-stage object detectors (FCOS) predict and output three sets of classification score maps, offset maps, and centrality maps. Based on the predicted maps output by each branch, three sets of pseudo-label sets with category and location information can be generated. This is achieved by filtering out pseudo-label boxes with low target scores (classification score x centrality) and redundant boxes with high overlap. Using the pseudo-label set from the teacher network and the existing sparse bounding box set, the pseudo-label sets output by the two branches A and B of the co-teacher network can be corrected. This involves filtering out pseudo-label boxes with incorrect category or location information, and outputting supervision signals for each branch to monitor the predicted values output by the co-teacher network. The parameters of the co-teacher network are updated by the gradient backpropagated from the loss function between the supervision signal and the predicted output, while the parameters of the teacher network are updated using EMA (Exponential Moving Average).
[0040] Furthermore, this invention provides a novel object detection method based on sparsely labeled datasets, comprising:
[0041] (1) First, construct a dual-image-input-port secondary teacher detector (Fully Convolutional Single-Level Object Detector FCOS) and a teacher detector whose parameters are updated by exponential moving average. At the start of training, assign the same pre-trained model parameters to both.
[0042] (2) Two different levels of data augmentation are applied to the sparsely labeled training images. One includes random horizontal flipping and color jitter data augmentation, and the other includes a series of random strong data augmentations such as random horizontal flipping, color jitter data augmentation, random equalization, random sharpening, random translation, random rotation, random cropping, and random erasure.
[0043] (3) Input the original image, weak data augmentation image and strong data augmentation image into the teacher detector branch and the two branches of the sub-teacher detector respectively. Through the three branches of the FCOS detector (the teacher detector branch and the two branches of the sub-teacher detector), predict and output three sets of classification score images, offset images and centrality images respectively.
[0044] (4) Generate pseudo-label boxes for each reference point based on the classification score map and offset map of each group, filter out all pseudo-label boxes whose product of classification score and centrality is less than a preset threshold, and remove redundant boxes with excessive overlap by non-maximum suppression algorithm to obtain three pseudo-label sets.
[0045] (5) Based on the obtained pseudo-label set T of the teacher detector, correct the pseudo-label sets A and B output by the assistant teacher detector, and transform all pseudo-labels in set T into the transform domains of A and B respectively, because there are data augmentations at different spatial locations among these three.
[0046] (6) Transform the pseudo-label set Ta / Tb after the transformation domain, take all pseudo-labels in Ta / Tb whose intersection-union ratio with A and B is less than the set threshold as the missing label boxes, and replace all pseudo-labels in A and B whose intersection-union ratio with a certain pseudo-label Pi in Ta / Tb is greater than the preset threshold and whose classification score is smaller than Pi with Pi.
[0047] (7) Merge the corrected pseudo-label sets A` and B` with the sparsely labeled bounding boxes respectively, remove all pseudo-labels whose intersection-union ratio with the bounding boxes is greater than a preset threshold, and obtain two sets of supervision signals. Supervised learning is then performed on the prediction information output by the corresponding two branches.
[0048] (8) After training for a preset period according to the above steps, the obtained detector model is tested on a fully labeled test set to test its detection performance. After reaching the preset performance, the above detector model is used to perform target detection on the image to be detected.
[0049] This invention is based on the single-level object detector FCOS and is trained on multiple MS COCO training sets with different degrees of sparse annotation. It achieves the best object detection performance on the fully labeled MS COCO test set. The training time is the total time consumption for training 12 times per image in the same training set, with the only difference being the method. The GPU resources used in the experiment are 4 NVIDIA GeForce RTX 3090s.
[0050] Method (Omission Rate) AP <![CDATA[AP s ]]> <![CDATA[AP m ]]> <![CDATA[AP l ]]> Training time Co-mining (20%) 37.6 -- -- -- 17h This invention (20%) 38.6 23.1 42.2 48.6 22.5h Co-mining (50%) 35.7 19.6 38.9 47.0 17h This invention (50%) 36.7 21.1 40.4 46.8 22.5h Co-mining (65%) 23.1 -- -- -- 17h This invention (65%) 26.1 13.2 28.3 37.2 22.5h
[0051] Table 1 shows the results of training on the MS COCO training set with varying degrees of missing labels and testing on the fully labeled test set.
[0052] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A novel object detection system based on sparsely labeled datasets, characterized in that, This includes a teacher detector based on an exponential moving average and a secondary teacher detector with two image input ports, wherein: The teacher detector does not backpropagate gradients during training, but only updates parameters by exponential moving average at the end of each training iteration. The teacher detector updated by exponential moving average is a smoothed version of the dual-input port secondary teacher detector. The input images for the teacher detector and the dual-input co-teacher detector are the original, weakly enhanced, and strongly enhanced images, respectively. The input image for the teacher detector is the original image, which is fed into the teacher detector to generate the teacher pseudo-label set. The weakly enhanced and strongly enhanced images are fed into the co-teacher detector with two input ports. The single-level target detector FCOS predicts and outputs three sets of classification score maps, offset maps, and centrality maps, and further generates the original pseudo-label set. Each original pseudo-label set is corrected and integrated by combining the pseudo-label set output by the teacher detector with sparse bounding boxes. Finally, the corrected and integrated bounding boxes are used to supervise the prediction output of the assistant teacher detector. The correction and integration specifically involves: Based on the classification score map and offset map of each group, generate pseudo-label boxes for each reference point, filter out all pseudo-label boxes whose product of classification score and centrality is less than a preset threshold, and remove redundant boxes with excessive overlap through non-maximum suppression algorithm to obtain three pseudo-label sets. Based on the obtained pseudo-label set T of the teacher detector, correct the pseudo-label sets A and B output by the assistant teacher detector, and transform all pseudo-labels in set T into the transform domains of A and B respectively. The pseudo-label set Ta / Tb after transformation domain is transformed. All pseudo-labels in Ta / Tb with an intersection-union ratio (IU) of less than a set threshold with A and B are taken as missing label boxes. All pseudo-labels in A and B with an IU of greater than a preset threshold with a certain pseudo-label Pi in Ta / Tb and a classification score smaller than Pi are replaced with Pi. The corrected pseudo-label sets A` and B` are merged with the sparsely labeled bounding boxes, and all pseudo-labels with an intersection-union ratio greater than a preset threshold are removed to obtain two sets of supervision signals. The prediction information output by the corresponding two branches is then subjected to supervised learning.
2. The novel target detection system based on sparse labeled datasets as described in claim 1, characterized in that, Both the teacher detector and the co-teacher detector are implemented using the fully convolutional single-level object detector FCOS.
3. The novel target detection system based on sparsely labeled datasets as described in claim 1 or 2, characterized in that, Based on the prediction maps output by each branch, three sets of pseudo-labels with category and location information are generated. Specifically, the final pseudo-label set is obtained by filtering out pseudo-label boxes with low target scores and redundant boxes with high overlap. According to the pseudo-label set of the teacher network and the existing sparse labeled box set, the pseudo-label sets output by the two branches A and B of the co-teacher network are corrected to filter out pseudo-label boxes with incorrect category or location information as much as possible. The supervision signals of the two branches are output to supervise the predicted values output by the co-teacher network.
4. The novel target detection system based on sparsely labeled datasets as described in claim 3, characterized in that, The target score is the classification score multiplied by the centrality.
5. The novel target detection system based on sparsely labeled datasets as described in claim 1 or 2, characterized in that, The parameters of the co-teacher network are updated by the gradient backpropagated from the loss function between the supervision signal and the predicted output, while the parameters of the teacher network are updated by the exponential moving average (EMA).
6. A novel object detection method based on sparsely labeled datasets, characterized in that, The method includes the following steps: (1) Construct a secondary teacher detector with dual image input ports and a teacher detector whose parameters are updated by exponential moving average. At the start of training, assign the same pre-trained model parameters to both. (2) Two different levels of data augmentation are applied to the sparsely labeled training images. The two different levels of data augmentation include: one includes random horizontal flipping and color jitter data augmentation, and the other includes random horizontal flipping, color jitter data augmentation, random equalization, random sharpening, random translation, random rotation, random cropping and random erasure. (3) Input the original image, weak data augmentation image and strong data augmentation image into the teacher detector branch and the two branches of the assistant teacher detector respectively, and predict and output three sets of classification score images, offset images and centrality images through the teacher detector branch and the two branches of the assistant teacher detector respectively; (4) Generate pseudo-label boxes for each reference point based on the classification score map and offset map of each group, filter out all pseudo-label boxes whose product of classification score and centrality is less than a preset threshold, and remove redundant boxes with excessive overlap by non-maximum suppression algorithm to obtain three pseudo-label sets. (5) Based on the obtained pseudo-label set T of the teacher detector, correct the pseudo-label sets A and B output by the assistant teacher detector, and transform all pseudo-labels in set T into the transform domains of A and B respectively; (6) Transform the pseudo-label set Ta / Tb after the transformation domain, take all pseudo-labels in Ta / Tb whose intersection-union ratio with A and B is less than the set threshold as the missing label boxes, and replace all pseudo-labels in A and B whose intersection-union ratio with a certain pseudo-label Pi in Ta / Tb is greater than the preset threshold and whose classification score is smaller than Pi with Pi. (7) Merge the corrected pseudo-label sets A` and B` with the sparsely labeled bounding boxes respectively, remove all pseudo-labels whose intersection-union ratio with the bounding boxes is greater than a preset threshold, and obtain two sets of supervision signals. Supervised learning is then performed on the prediction information output by the corresponding two branches. (8) After training for a preset period according to the above steps, the obtained detector model is tested on a fully labeled test set to test its detection performance. After the preset performance is achieved, the above detector model is used to perform target detection on the image to be detected.
7. The novel object detection method based on sparsely labeled datasets as described in claim 6, characterized in that, Both the teacher detector and the co-teacher detector are implemented using the fully convolutional single-level object detector FCOS.
8. The novel target detection method based on sparsely labeled datasets as described in claim 6, characterized in that, The sparsely labeled dataset consists of MS COCO training sets with varying degrees of sparseness.