Surrounding frame level active learning target detection method and device and storage medium
By employing an active learning method at the bounding box level, high-value targets are selected for labeling, thus solving the problem of high cost for image-level labeling and achieving efficient target detection model training and cost savings in labeling.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2022-07-15
- Publication Date
- 2026-05-29
AI Technical Summary
Existing object detection algorithms based on active learning suffer from high image-level labeling costs and low labeling efficiency, failing to effectively filter out targets valuable for model training.
We employ an active learning method at the bounding box level. By training the object detection model with a fully labeled dataset, we determine the uncertainty of the predicted bounding box for each object in the unlabeled dataset, label them in descending order of uncertainty, update the dataset with the added labeled bounding boxes, and perform hybrid supervised training, prioritizing the labeling of high-value targets.
It reduces labeling costs, improves model training efficiency, maximizes labeling benefits, reduces labeling difficulty, and optimizes model performance by adjusting pseudo-labels and confidence thresholds.
Smart Images
Figure CN115272818B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing, and more specifically, to a bounding box-level active learning target detection method, apparatus, and storage medium. Background Technology
[0002] Object detection technology typically refers to detecting the location of an object and its corresponding category in an image. In the many technical fields of computer vision, object detection is also a very basic task. Image segmentation, object tracking, key point detection, etc. usually rely on object detection. In machine learning-based object detection algorithms, a large number of images with manual labels are usually required as training samples, so the cost of manual labeling is very high.
[0003] Active learning has attracted increasing attention as a method that can effectively reduce the cost of manual labeling. Compared to providing complete manual labels for all samples, active learning can select objects worthy of being labeled for manual labeling and filter out redundant or outlier samples, thereby achieving the same or better model performance at a lower cost.
[0004] Currently, active learning-based object detection algorithms use a single image as the unit of computational cost for manual labeling. This means that the candidate objects selected by active learning are still images, requiring detailed labeling of the selected images, i.e., labeling all targets contained in the selected images. However, in reality, not all targets in the selected images have the value to be labeled. If detailed labeling is performed on an image-by-image basis, the labeling cost is still relatively high. Summary of the Invention
[0005] This application provides a bounding box-level active learning object detection method, apparatus, and storage medium, aiming to reduce the labeling cost during object detection.
[0006] In a first aspect, embodiments of this application provide a bounding box-level active learning object detection method, the method comprising:
[0007] Obtain a fully labeled dataset from the labeled dataset, wherein each object in each image of the fully labeled dataset carries a labeled bounding box;
[0008] Based on the fully labeled dataset, the object detection model is trained under full supervision.
[0009] The trained object detection model is used to predict the unlabeled dataset, and the uncertainty of the predicted bounding box corresponding to each object in each image of the unlabeled dataset is determined.
[0010] According to the order of decreasing uncertainty, all targets in the unlabeled dataset are sorted, a preset number of targets at the top of the sort are given labeled bounding boxes, and the labeled dataset in the labeled dataset is updated using all images of the targets with the added labeled bounding boxes.
[0011] Based on the labeled dataset, the object detection model is further trained under hybrid supervision, and the model parameters of the object detection model are updated.
[0012] Optionally, the training process of the object detection model and the update process of the partially labeled dataset are carried out alternately and iteratively until the object detection model reaches a preset training state or the number of labels reaches the budgeted number.
[0013] Optionally, during the alternating iterative process of training the object detection model and updating the partially labeled dataset:
[0014] After the target detection model is trained, predictions are made on both the unlabeled dataset and the partially labeled dataset. Bounding boxes with labels are then selected from the unlabeled dataset and the partially labeled dataset. In the unlabeled dataset, all targets do not have labeled bounding boxes.
[0015] Optionally, when predicting the partially labeled dataset after the target detection model has been trained, the method further includes:
[0016] If the overlap between the predicted bounding box of any target in the partially labeled dataset and the bounding box of the labeled target is greater than the overlap threshold, the predicted bounding box of that target is filtered out.
[0017] Determine the uncertainty of the predicted bounding box for each remaining target.
[0018] Optionally, the process of determining the uncertainty of the predicted bounding box corresponding to each target is as follows:
[0019] The highest confidence level among the multiple categories corresponding to the predicted bounding box is selected, and the uncertainty of the predicted bounding box corresponding to each target is determined based on the highest confidence level.
[0020] Optionally, the formula for calculating the uncertainty of the predicted bounding box corresponding to each target is:
[0021]
[0022] In the formula, p k (y = c) represents the confidence scores of the predicted bounding box for the k-th target across multiple categories; c represents the category; θ tThe parameters are the model parameters obtained from the t-th training iteration of the target detection model.
[0023] Optionally, after updating a portion of the labeled dataset in the labeled dataset, the method further includes:
[0024] If, in the partially labeled dataset, the confidence level of any one of the multiple categories of the predicted bounding box of the target is greater than a confidence threshold, and the overlap between the predicted bounding box of the target and the bounding box of any label is less than an overlap threshold, a pseudo-label is added to the target. The predicted bounding box corresponding to the target represents the positional information of the pseudo-label, and the category with the highest confidence level of the target represents the category information of the pseudo-label.
[0025] Optionally, during the process of continuing hybrid supervised training of the object detection model based on the labeled dataset:
[0026] The confidence level corresponding to the pseudo-label is used as a weighting factor;
[0027] The loss function of the object detection model in hybrid supervised training is calculated based on the weighting factors.
[0028] Secondly, embodiments of this application provide a bounding box-level active learning target detection device, the device comprising:
[0029] The acquisition module is used to acquire a fully labeled dataset from the labeled dataset, wherein each target in each image in the fully labeled dataset carries a labeled bounding box;
[0030] The model training module is used to perform fully supervised training of the object detection model based on the fully labeled dataset;
[0031] The uncertainty calculation module is used to make predictions on the unlabeled dataset using the trained target detection model, and to determine the uncertainty of the predicted bounding box corresponding to each target in each image of the unlabeled dataset;
[0032] The label update module is used to sort all targets in the unlabeled dataset in descending order of uncertainty, add labeled bounding boxes to a preset number of targets at the top of the sort, and update a portion of the labeled dataset in the labeled dataset using all images containing the targets with labeled bounding boxes.
[0033] The model update module is used to continue hybrid supervised training of the object detection model based on the labeled dataset and update the model parameters of the object detection model.
[0034] Optionally, the device further includes:
[0035] The alternating update module is used to alternately iterate the training process of the object detection model with the update process of the partially labeled dataset until the object detection model reaches a preset training state or the number of labels reaches the budgeted number.
[0036] Optionally, the alternating update module further includes:
[0037] The filtering unit is used to predict both the unlabeled dataset and the partially labeled dataset after the target detection model has been trained, and to filter out bounding boxes with added labels for targets from the unlabeled dataset and the partially labeled dataset, wherein all targets in the unlabeled dataset do not carry labeled bounding boxes.
[0038] Optionally, the filtering unit further includes:
[0039] The filtering subunit is used to filter out the predicted bounding box of any target in the partially labeled dataset when the overlap between the predicted bounding box of any target and the bounding box of the labeled target is greater than the overlap threshold.
[0040] Determine the sub-units to determine the uncertainty of the predicted bounding box for each remaining target.
[0041] Optionally, the uncertainty calculation module further includes:
[0042] The first calculation unit is used to select the highest confidence level among the confidence levels of multiple categories corresponding to the predicted bounding box, and determine the uncertainty of the predicted bounding box corresponding to each target based on the highest confidence level.
[0043] Optionally, the device further includes:
[0044] The pseudo-label adding module is used to add a pseudo-label to the target when, in the partially labeled dataset, the confidence of any one of the multiple categories of the predicted bounding box of the target is greater than a confidence threshold, and the overlap between the predicted bounding box of the target and the bounding box of any label is less than an overlap threshold. The predicted bounding box corresponding to the target represents the positional information of the pseudo-label, and the category with the highest confidence of the target represents the category information of the pseudo-label.
[0045] Optionally, the alternating update module includes:
[0046] A weighting factor determination unit is used to use the confidence level corresponding to the pseudo-label as a weighting factor.
[0047] The loss function calculation unit is used to calculate the loss function of the target detection model in hybrid supervised training based on the weighting factors.
[0048] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the bounding box-level active learning target detection method described in the first aspect of the embodiments.
[0049] Beneficial effects:
[0050] First, the object detection model is trained in a fully supervised manner on a fully labeled dataset. The trained object detection model then predicts bounding boxes on the unlabeled dataset without labeled bounding boxes, determining the uncertainty of the predicted bounding box for each object in each image of the unlabeled dataset. The uncertainty characterizes the difficulty or training value of the target for the current object prediction model. All targets in the unlabeled dataset are sorted in descending order of uncertainty. A predetermined number of targets at the top of the sort are added with labeled bounding boxes, i.e., targets with greater training value are added with labeled bounding boxes. The images containing targets with labeled bounding boxes are used to update a portion of the labeled dataset. Based on the updated labeled dataset, the object detection model is further trained in a hybrid supervised manner, updating the model parameters.
[0051] By labeling targets at the granularity or unit of each target or a single bounding box, priority is given to labeling targets that are of high value for training the target detection model. This improves the efficiency of model training and saves labeling costs compared to manually labeling all targets in an image. It also reduces the difficulty of labeling. Attached Figure Description
[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a flowchart of the steps of an active learning object detection method at the bounding box level proposed in an embodiment of this application;
[0054] Figure 2 This is a flowchart of the steps of an active learning object detection method at the bounding box level proposed in an embodiment of this application;
[0055] Figure 3 This is a functional block diagram of a bounding box-level active learning target detection device proposed in an embodiment of this application. Detailed Implementation
[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0057] In existing active object detection technologies, due to the influence of classification tasks, a single image is still used as the labeling unit, which means that all objects in an image need to be labeled, such as by manual labeling. However, in reality, the labeling cost for each image is not the same, because the cost is usually calculated based on the number of bounding boxes. This makes the labeling cost of images with a large number of objects and complex scenes significantly higher than that of images with sparse object distribution.
[0058] Meanwhile, image-level active detection is very fragile. Simply selecting the image with the most predicted bounding boxes can achieve detection performance that surpasses most existing algorithms, but such evaluation ignores the differences in annotation costs between images.
[0059] Active classification tasks treat the entire image as the smallest unit for labeling; each candidate is an image, and once selected, exhaustive annotation is performed. In detection tasks, the difficulty of detecting objects in an image varies greatly, and not all objects are valuable for model training. For example, a salient object of a common class is easy for a deep model to learn, while rare classes, distant objects, or objects with severe occlusion may be more challenging and valuable for labeling. If exhaustive labeling is performed on an image-by-image basis, the labeling cost remains relatively high.
[0060] To address these issues, this application proposes a bounding box-level active learning object detection method, which can effectively reduce the difficulty of labeling and maximize labeling benefits.
[0061] Reference Figure 1 The diagram illustrates a flowchart of a bounding box-level active learning object detection method according to an embodiment of the present invention. The method specifically includes the following steps:
[0062] S101: Obtain the fully labeled dataset from the labeled dataset, wherein each object in each image of the fully labeled dataset carries a labeled bounding box.
[0063] First, a fully labeled dataset is obtained for the initial training of the object detection model. The fully labeled dataset contains multiple images, and each object in each image carries a labeled bounding box. The labeled bounding box of any object can represent the position of the object in the image and the correct category of the object. The specific labeling method can be selected according to the actual labeling requirements. In this embodiment, the scheme is described using manually labeled bounding boxes as an example.
[0064] In practice, an unlabeled dataset U can be constructed to train the object detection model. t ={X t U}, at this point t=0. Unlabeled dataset U t ={X t U The images included in}(t=0) are all unlabeled images. A portion of images is randomly selected from the unlabeled dataset, and each object in each randomly selected image is manually labeled to form a fully labeled dataset L. t ={X0 L For example, 5000 images are randomly selected from the fully labeled dataset L0. These randomly selected images are removed from the unlabeled dataset. L B0 represents the fully marked image, and B0 represents the bounding box of the mark.
[0065] S102: Based on the fully labeled dataset, perform fully supervised training on the object detection model.
[0066] Based on the manually labeled fully labeled dataset L0={X0 L The target detection model is trained at t=0, and this training is fully supervised, meaning that each target in all image samples carries a complete manual label during training. The target detection model can be any general detection model M(θ). t ), θ t The parameters are the model parameters obtained after the t-th training of the object detection model. The training process of the object detection model is not limited in this embodiment.
[0067] S103: Use the trained object detection model to predict the unlabeled dataset and determine the uncertainty of the predicted bounding box corresponding to each object in each image of the unlabeled dataset.
[0068] Specifically, when t=0, after the object detection model is trained on the fully labeled dataset, the object detection model is then used on the unlabeled dataset U. t ={X t UThe object detection model performs predictions and outputs the unlabeled dataset U. t ={X t U In the image, each target corresponds to a predicted bounding box. The predicted bounding box represents the location of the target, and each target's predicted bounding box carries multiple categories of confidence. The sum of the multiple categories of confidence of a target's predicted bounding box is 1.
[0069] In one feasible implementation, the process of determining the uncertainty of the predicted bounding box corresponding to each target is as follows: select the highest confidence score among the confidence scores of multiple categories corresponding to the predicted bounding box, and determine the uncertainty of the predicted bounding box corresponding to each target based on the highest confidence score. For example, the uncertainty of the predicted bounding box can be determined by calculating the entropy of the highest confidence score and the confidence scores of other categories, respectively. The calculation formula is as follows:
[0070]
[0071] In the formula, p k (y = c) represents the confidence scores of the predicted bounding box for the k-th target across multiple categories; c represents the category; θ t The parameters are the model parameters obtained from the t-th training iteration of the target detection model.
[0072] In other implementations, other classic value estimation methods can be used to determine the uncertainty of the predicted bounding box, such as other uncertainty measures, richness-based indicators, and indicators that combine uncertainty and richness.
[0073] If the object detection model has a high confidence level for one category in the predicted bounding box of an object, meaning the current object detection model is relatively certain about the category of that object, the calculated uncertainty is a low entropy. If the confidence levels of multiple categories in the predicted bounding box of an object are not significantly different, meaning the object detection model's prediction of that object is still controversial, the calculated uncertainty is a high entropy. This indicates that the object has high training value for the current object detection model, and objects with high training value can be prioritized for manual labeling.
[0074] S104: Sort all targets in the unlabeled dataset according to the order of decreasing uncertainty, add labeled bounding boxes to a preset number of targets at the top of the sort, and update the partial labeled dataset in the labeled dataset using all images of the targets with added labeled bounding boxes.
[0075] After determining the uncertainty of the predicted bounding boxes of all targets in the unlabeled dataset, all targets are sorted in descending order of uncertainty. Targets ranked higher in uncertainty have relatively higher training value. Therefore, based on the actual needs or labeling capabilities of each manual labeling session, a preset number of targets for each manual labeling session is determined, and the preset number of targets ranked higher are selected as the objects for this manual labeling session.
[0076] Then, using all images containing manually labeled bounding boxes, the partially labeled dataset in the labeled dataset is updated. Each image in the partially labeled dataset is an incompletely labeled image. Updating the partially labeled dataset also updates the labeled dataset. The updated labeled dataset is denoted as... For partially labeled images in a partially labeled dataset, B t The bounding box is a partially labeled dataset; that is, the labeled dataset contains both the fully labeled dataset and the partially labeled dataset.
[0077] Compared to existing methods that select images and add manually labeled bounding boxes to all targets in an image, this method adds manually labeled bounding boxes only to selected targets that have more training value, based on the labeling independence of the target bounding boxes, instead of manually labeling all targets in the image. For example, according to the labeling cost of the bounding boxes, such as labeling 5000 bounding boxes each time, the top 5000 targets are selected to add manually labeled bounding boxes, so that the limited manual labeling resources can be used on targets that can make a greater contribution to the training process of the object detection model.
[0078] This method breaks away from labeling individual images as the smallest unit and instead labels each target's bounding box. This allows for prioritizing the labeling of targets that are valuable for training the target detection model, reducing labeling difficulty, and maximizing labeling benefits or efficiency while keeping labeling costs limited.
[0079] In one feasible implementation, considering that the images in the partially labeled dataset at this time only contain bounding boxes with selected targets that have been manually labeled, that is, the image labeling is incomplete, and the targets that have not been selected for labeling are targets with high certainty in the current target detection model, that is, relatively simple targets for the current target detection model. Simple targets will be regarded as negative examples in the training of the target detection model, thus affecting the detection performance of the target detection model. Therefore, in order to reduce the impact of simple targets on the performance of the target detection model, this method also includes knowledge distillation between training rounds during active learning.
[0080] Specifically, in the updated partial dataset, if the confidence of any one of the multiple categories of the predicted bounding box of each target is greater than a confidence threshold, a pseudo-label is added to the target. The predicted bounding box corresponding to the target represents the location information of the pseudo-label, and the category with the highest confidence of the target represents the category information of the pseudo-label.
[0081] In one implementation, the formula for selecting the predicted bounding box as the target of the pseudo-label is as follows:
[0082]
[0083] In the formula, The image obtained by the object detection model after training for the tth time is an incompletely labeled image in the partially labeled dataset. The predicted bounding box; b j ∈B t τ represents the manually labeled bounding boxes on the same image; c represents the multiple categories of the predicted bounding box of any target; τ is the confidence threshold; and δ is the maximum overlap (Intersection over Union, IoU) between the false-label bounding box and the manually labeled bounding box. That is, when the predicted bounding box of a target simultaneously satisfies a confidence level higher than τ and the overlap between the predicted bounding box of the target and any manually labeled bounding box is lower than δ, it is selected as a false label.
[0084] If the confidence level of the object detection model trained so far for a certain category in the predicted bounding box of an object is high, it indicates that the object detection model is confident in its prediction of that object. That is, the object is a simple object for the object detection model, and the predicted bounding box of that object output by the object detection model can be used as a pseudo-label. The labeled dataset in this case is denoted as... in, This is a pseudo-tag.
[0085] After each partial label dataset update, including after adding manually labeled bounding boxes and pseudo-labels, or before each training of the object detection model, let t = t + 1.
[0086] S105: Based on the labeled dataset, continue to perform hybrid supervised training on the object detection model and update the model parameters of the object detection model.
[0087] The current labeled dataset includes not only images where all targets in the image have manually labeled bounding boxes, but also images where some targets in the image have manually labeled bounding boxes, as well as images where some targets have manually labeled bounding boxes and some targets have pseudo-labels with predicted bounding boxes.
[0088] The object detection model is further trained on a labeled dataset with richer labels, and the model parameters are updated.
[0089] Since pseudo-labels are the predicted bounding boxes output by the object detection model itself when predicting images in a partially labeled dataset within the labeled dataset, the accuracy of the predicted bounding boxes as pseudo-labels is not high. Although pseudo-labels bring more knowledge to the training of the object detection model, they also introduce labeled noise into the training process. Therefore, when continuing to train the object detection model based on the labeled dataset, the confidence level corresponding to the pseudo-labels can be used as a weighting factor to reduce the training weight of the object detection model for the pseudo-labeled targets.
[0090] Specifically, the cross-entropy loss function of the object detection model based on the labeled dataset is calculated according to the weighting factor. The formula for the cross-entropy loss function is as follows:
[0091]
[0092] In the formula, As a weighting factor, For pseudo-label category, This represents the predicted category of the object detection model during this training.
[0093] It can be seen that when t=0, the object detection model is trained in a fully supervised manner based on the fully labeled dataset, while when t≥1, the object detection model is trained in a hybrid supervised manner based on the fully labeled dataset and the partially labeled dataset. That is, when t≥1, the object detection model is trained on images with both fully labeled and partially labeled data.
[0094] Reference Figure 2 This document illustrates a flowchart of a bounding box-level active learning object detection method provided in an embodiment of this application. In one feasible implementation, in addition to steps S101-S105, the method further includes:
[0095] S106: The training process of the object detection model and the update process of the partially labeled dataset are carried out alternately and iteratively until the object detection model reaches the preset training state or the number of labels reaches the budgeted number.
[0096] The training process of the object detection model and the updating process of the labeled dataset are carried out alternately and iteratively. For example, after obtaining the object detection model based on the labeled dataset through mixed supervision in step S105, the steps S103-S105 are repeated. However, when t≥1, during the execution of step S103, the object detection model not only predicts the unlabeled dataset, but also predicts the partially labeled dataset. That is, it selects bounding boxes with artificial labels for the target from the unlabeled dataset and the partially labeled dataset.
[0097] For example, the method involves filtering targets by adding manually labeled bounding boxes, or calculating the uncertainty of predicted bounding boxes in the unlabeled dataset and the partially labeled dataset, sorting the targets in the unlabeled dataset and the partially labeled dataset in descending order of uncertainty, selecting the top preset number of targets by adding manually labeled bounding boxes, and updating the partially labeled dataset with the images of the targets in the unlabeled dataset that have added manually labeled bounding boxes.
[0098] Furthermore, during the alternating iterative process of training the target detection model and updating the labeled dataset, when the target detection model makes predictions on a portion of the labeled dataset, if the overlap between the predicted bounding box of any target in the portion of the labeled dataset and the manually labeled bounding box is greater than the overlap threshold, the predicted bounding box of that target is filtered out; and the uncertainty of the predicted bounding box corresponding to each remaining target is determined.
[0099] Since some targets in the partially labeled dataset have already had manually labeled bounding boxes added, if the overlap between the predicted bounding box and the manually labeled bounding box is greater than the overlap threshold, it can be said that the target of interest has already been manually labeled. In fact, it is not necessary to calculate the uncertainty of the predicted bounding boxes of these labeled targets. It is only necessary to determine the uncertainty of the predicted bounding boxes of the remaining targets. By filtering out the predicted bounding boxes with an overlap greater than the overlap threshold, the amount of computation can be reduced and computational resources can be saved.
[0100] Furthermore, during the iterative process, as the object detection model continues to improve, the confidence threshold when adding pseudo-labels will be adaptively adjusted. This is because the object detection model is constantly improving, and the prediction of the bounding box of the target will become more and more accurate, that is, the confidence of a certain category will become higher and higher. Therefore, the confidence threshold will also be adaptively increased.
[0101] Because of the change in confidence threshold, the pseudo-label of a target also changes in real time. At the same time, simple targets with previously added pseudo-labels may make prediction errors for the continuously improving target detection model. Therefore, as the labeled dataset expands and the target detection model improves, the pseudo-labels are self-corrected in a timely manner. For example, the category of a target's pseudo-label may change during the alternating iteration process, or a target's pseudo-label may be removed.
[0102] The training process of the object detection model and the updating process of the partially labeled dataset are alternately iteratively performed. Based on the continuously improving object detection model, manually labeled bounding boxes are added to the targets that have the most training value for the current model. This iterative process continues until the object detection model reaches a preset training state, or the number of manual labeling attempts reaches the target number, i.e., the cost of manual labeling is within budget, at which point the alternating iterative process stops. This application has at least the following beneficial effects:
[0103] 1. By labeling each target as a granularity or unit, priority is given to manually labeling targets that have high value for training the target detection model. This not only improves the efficiency of model training, but also significantly saves labeling costs and reduces labeling difficulty compared to manually labeling all targets in an image.
[0104] 2. Using the predicted bounding boxes with a confidence level greater than the confidence threshold predicted by the current object detection model as pseudo-labels for the object can add knowledge to the training of the next round of object detection model and save labeling costs.
[0105] 3. False labels and confidence thresholds can be corrected or adjusted as the object detection model is trained and the labeled dataset is updated, thereby reducing the impact of false labels on the performance of the object detection model.
[0106] Reference Figure 3 The diagram illustrates a functional block diagram of a bounding box-level active learning target detection device according to an embodiment of the present invention. The device includes:
[0107] The acquisition module 100 is used to acquire a fully labeled dataset from the labeled dataset, wherein each target in each image in the fully labeled dataset carries a labeled bounding box;
[0108] The model training module 200 is used to perform fully supervised training of the object detection model based on the fully labeled dataset.
[0109] Uncertainty calculation module 300 is used to make predictions on the unlabeled dataset using the trained target detection model, and to determine the uncertainty of the predicted bounding box corresponding to each target in each image of the unlabeled dataset;
[0110] The label update module 400 is used to sort all targets in the unlabeled dataset in descending order of uncertainty, add labeled bounding boxes to a preset number of targets at the top of the sort, and update a portion of the labeled dataset in the labeled dataset using all images containing the targets with the added labeled bounding boxes.
[0111] The model update module 500 is used to continue hybrid supervised training of the object detection model based on the labeled dataset and update the model parameters of the object detection model.
[0112] Optionally, the device further includes:
[0113] The alternating update module is used to alternately iterate the training process of the object detection model with the update process of the partially labeled dataset until the object detection model reaches a preset training state or the number of labels reaches the budgeted number.
[0114] Optionally, the alternating update module further includes:
[0115] The filtering unit is used to predict both the unlabeled dataset and the partially labeled dataset after the target detection model has been trained, and to filter out bounding boxes with added labels for targets from the unlabeled dataset and the partially labeled dataset, wherein all targets in the unlabeled dataset do not carry labeled bounding boxes.
[0116] Optionally, the filtering unit further includes:
[0117] The filtering subunit is used to filter out the predicted bounding box of any target in the partially labeled dataset when the overlap between the predicted bounding box of any target and the bounding box of the labeled target is greater than the overlap threshold.
[0118] Determine the sub-units to determine the uncertainty of the predicted bounding box for each remaining target.
[0119] Optionally, the uncertainty calculation module further includes:
[0120] The first calculation unit is used to select the highest confidence level among the confidence levels of multiple categories corresponding to the predicted bounding box, and determine the uncertainty of the predicted bounding box corresponding to each target based on the highest confidence level.
[0121] Optionally, the device further includes:
[0122] The pseudo-label adding module is used to add a pseudo-label to the target when, in the partially labeled dataset, the confidence of any one of the multiple categories of the predicted bounding box of the target is greater than a confidence threshold, and the overlap between the predicted bounding box of the target and the bounding box of any label is less than an overlap threshold. The predicted bounding box corresponding to the target represents the positional information of the pseudo-label, and the category with the highest confidence of the target represents the category information of the pseudo-label.
[0123] Optionally, the alternating update module includes:
[0124] A weighting factor determination unit is used to use the confidence level corresponding to the pseudo-label as a weighting factor.
[0125] The loss function calculation unit is used to calculate the loss function of the target detection model in hybrid supervised training based on the weighting factors.
[0126] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the bounding box-level active learning target detection method described in the embodiments.
[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0129] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.
[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0132] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.
[0133] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0134] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A bounding box-level active learning object detection method, characterized in that, The method includes: Obtain a fully labeled dataset from the labeled dataset, wherein each object in each image of the fully labeled dataset carries a labeled bounding box; Based on the fully labeled dataset, the object detection model is trained under full supervision. The trained object detection model is used to predict the unlabeled dataset, and the uncertainty of the predicted bounding box corresponding to each object in each image in the unlabeled dataset is determined. According to the order of decreasing uncertainty, all targets in the unlabeled dataset are sorted, and a pre-set number of targets at the top of the sort are marked with bounding boxes. This includes: determining the pre-set number of targets to be manually marked each time based on the actual needs or marking capabilities of each manual marking, selecting the pre-set number of targets at the top of the sort as the objects to be manually marked this time, and marking each target's bounding box as a unit. Using all images containing targets with labeled bounding boxes, update the partially labeled dataset in the labeled dataset, where the images in the partially labeled dataset contain only partially labeled bounding boxes for selected targets; Based on the labeled dataset with richer labels, the object detection model is further trained under hybrid supervision, and the model parameters of the object detection model are updated.
2. The method according to claim 1, characterized in that, The training process of the object detection model and the updating process of the partially labeled dataset are carried out alternately and iteratively until the object detection model reaches the preset training state or the number of labels reaches the budgeted number.
3. The method according to claim 2, characterized in that, During the alternating iterative process of training the object detection model and updating the partially labeled dataset: After the target detection model is trained, predictions are made on both the unlabeled dataset and the partially labeled dataset. Bounding boxes with labels are then selected from the unlabeled dataset and the partially labeled dataset. In the unlabeled dataset, all targets do not have labeled bounding boxes.
4. The method according to claim 3, characterized in that, After the target monitoring model has been trained, when making predictions on the partially labeled dataset, the method further includes: If the overlap between the predicted bounding box of any target in the partially labeled dataset and the bounding box of the labeled target is greater than the overlap threshold, the predicted bounding box of that target is filtered out. Determine the uncertainty of the predicted bounding box for each remaining target.
5. The method according to any one of claims 1-4, characterized in that, The process of determining the uncertainty of the predicted bounding box corresponding to each target is as follows: The highest confidence level among the multiple categories corresponding to the predicted bounding box is selected, and the uncertainty of the predicted bounding box corresponding to each target is determined based on the highest confidence level.
6. The method according to any one of claims 1-4, characterized in that, After updating a portion of the labeled dataset in the labeled dataset, the method further includes: If, in the partially labeled dataset, the confidence level of any one of the multiple categories of the predicted bounding box of the target is greater than a confidence threshold, and the overlap between the predicted bounding box of the target and the bounding box of any label is less than an overlap threshold, a pseudo-label is added to the target. The predicted bounding box corresponding to the target represents the positional information of the pseudo-label, and the category with the highest confidence level of the target represents the category information of the pseudo-label.
7. The method according to claim 6, characterized in that, During the process of continuing hybrid supervised training of the object detection model based on the labeled dataset: The confidence level corresponding to the pseudo-label is used as a weighting factor; The loss function of the object detection model in hybrid supervised training is calculated based on the weighting factor.
8. A bounding box-level active learning target detection device, characterized in that, The device includes: The acquisition module is used to acquire a fully labeled dataset from the labeled dataset, wherein each target in each image in the fully labeled dataset carries a labeled bounding box; The model training module is used to perform fully supervised training of the object detection model based on the fully labeled dataset; The uncertainty calculation module is used to make predictions on the unlabeled dataset using the trained target detection model, and to determine the uncertainty of the predicted bounding box corresponding to each target in each image of the unlabeled dataset; The labeling update module is used to sort all targets in the unlabeled dataset according to the order of decreasing uncertainty, and add labeled bounding boxes to a preset number of targets at the top of the sort. Specifically, it is used to determine the preset number of targets to be manually labeled each time based on the actual needs or labeling capabilities of each manual labeling, select the preset number of targets at the top of the sort as the objects to be manually labeled this time, and select to label each target's bounding box as a unit. Using all the images where the targets with labeled bounding boxes are located, the module updates a partial labeled dataset in the labeled dataset, wherein the images in the partial labeled dataset contain only some targets whose bounding boxes have been manually labeled. The model update module is used to continue hybrid supervised training of the object detection model based on the labeled dataset that has become richer in labels, and to update the model parameters of the object detection model.
9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the active learning target detection method at the bounding box level as described in any one of claims 1 to 7.