Automatic labeling with uncertainty quantification

Abstract

This application describes a method for automatically labeling data samples with uncertainty quantification. The proposed method comprises the steps of simultaneously feeding an input data sample into an online object detection module with uncertainty estimation and an off-board object detector with uncertainty estimation; comparing the resulting data from both the online classification and uncertainty estimation module and the off-board classification and uncertainty estimation module; and deciding whether to discard the input data sample, send it for human labeling, or archive it along with the classification and uncertainty estimation from the off-board object detection module with uncertainty estimation.These uncertainty estimates are then used in the training phase of subsequent object detectors to weight different samples differently, with human-labeled samples and automatically labeled samples with high confidence receiving more weight.

Description

Technical area

[0001] The present application describes a method for automatically labeling data samples with uncertainty quantification. State of the art

[0002] Highly and fully automated vehicles (Levels 3-5) will increasingly be seen on public roads in the coming years. All known concepts for automated vehicles require a combination of various perception sensors, such as cameras, radar, ultrasound, and lidar. These perception sensors are used to detect important objects in the scene, such as other road users and stationary objects such as road barriers. In order for these autonomous vehicle systems to be able to detect these objects, a catalog of labeled data is required. This labeled data is used to train models capable of identifying the objects in the scene.The problem is that it currently takes a human more than 10 hours to accurately label a 25-second scene at 10 Hz, and that an enormous amount of diverse labeled data, covering the long tail of real-world scenarios, is required to train the models to be confident enough to be used in an autonomous vehicle.

[0003] Although automatic labeling methods are increasingly being used to support or replace human labelers, these labels automatically generated by hardware computers are mistakenly considered correct, with each label being treated equally when object detection models are trained and deployed. In reality, however, the automatic labeler sometimes generates erroneous labels.

[0004] On the other hand, active learning methods require humans to label all selected samples, which incurs high costs.

[0005] The invention disclosed herein aims to overcome the existing problems of the prior art. Summary

[0006] The present invention describes a method for automatically labeling input data samples with uncertainty quantification, comprising the steps of: simultaneously feeding an input data sample to an online object detection module with uncertainty estimation and to an off-board object detector with uncertainty estimation; comparing a resulting classification and uncertainty estimation data determined by the online object detection module with uncertainty estimation with a resulting classification and uncertainty estimation data determined by the off-board object detection module with uncertainty estimation; and determining whether to discard the input data sample, send it for human labeling, or send it to the off-board object detection module with uncertainty estimation for archiving.

[0007] In a proposed embodiment of the present invention, the input data sample comprises at least one of the set: an image, a frame from a lidar sensor, or even a sequence of images or frames.

[0008] In a further proposed embodiment of the present invention, a resulting input data sample of the online object detection module with uncertainty estimation and the off-board object detector with uncertainty estimation each comprises detected objects of the input data sample.

[0009] In another proposed embodiment of the present invention, the resulting classification and uncertainty estimation data of both the online classification and uncertainty estimation module and the off-board classification and uncertainty estimation module comprise at least one of: classification of an input data sample, and an uncertainty estimation of an input data sample.

[0010] In another proposed embodiment of the present invention, if the comparison between the resulting classification and uncertainty estimation data is true, an uncertainty comparison is performed between both uncertainty estimates of the input data sample; and if the comparison between the resulting classification and uncertainty estimation data is false, an off-board high uncertainty analysis is performed.

[0011] In another proposed embodiment of the present invention, if the off-board high uncertainty analysis is true, the input sample is sent for human labeling, and if the off-board high uncertainty analysis is false, the input sample is stored in the off-board object detection module with uncertainty estimation so that said sample can be used to train an online object detector.

[0012] In another proposed embodiment of the present invention, the input data sample is discarded if the uncertainty comparison between the two uncertainty estimates of the input data sample is low. If the uncertainty comparison between the two uncertainty estimates of the input data sample is not low, an off-board high uncertainty analysis is performed.

[0013] In another proposed embodiment of the present invention, if the off-board high uncertainty analysis is true, the input sample is sent for human labeling, and if the off-board high uncertainty analysis is false, it is stored in the off-board object detection module with uncertainty estimation so that said sample can be used to train an online object detection module.

[0014] In another proposed embodiment of the present invention, the online object detection module with uncertainty estimation includes stored automatically labeled samples with uncertainty estimates and stored human-labeled samples.

[0015] In another proposed embodiment of the present invention, the stored automatically labeled samples and the stored human-labeled samples are combined in a sample weighting module for training.

[0016] In another proposed embodiment of the present invention, the weighted samples are stored, wherein stored automatically labeled samples are deprioritized and human-labeled samples (202) are prioritized to be forwarded to a training module. General description

[0017] Accurate perception models in automated vehicles require precisely labeled data, and manually labeling data is not only expensive and time-consuming, but also not the best solution.

[0018] At the current state of the art, labeling methods are simple and inefficient. Either the unlabeled samples are passed to an automatic labeler, and the resulting automatically labeled samples (assumed to be correct) are used to train an object detection algorithm. Or, in active learning, the keyframes selected for labeling (with high uncertainty) are passed from an object detection algorithm with uncertainty estimation to a human labeler, who labels the samples, which are then fed back into the object detection algorithm's training process.

[0019] Furthermore, existing state-of-the-art automatic labeling methods reduce the need for manually labeled data, but they assume that all automatically generated labels are equally valid and do not account for their uncertainty. In reality, automatically labeled data is typically of lower quality than manually labeled data, and this is not taken into account during the training process. Furthermore, active labeling methods can benefit from greater automation to reduce costs.

[0020] Therefore, it is advantageous to use the uncertainty of the automatically generated labels, estimated by an automatic labeler, when training object recognition models. The disclosure of the invention in this regard is not limited to object recognition models, but can also be used for other models, e.g., semantic segmentation or panoptic segmentation.

[0021] The invention disclosed herein promotes improving the performance of an online object detector by utilizing both its uncertainty estimation and the uncertainty estimation of an off-board object detector. An online object detector is a model or system / device configured to detect objects in real time in the vehicle. An off-board object detector is a model or system / device configured to detect objects but not running in real time in the vehicle (thus enabling the off-board object detector to have higher computational requirements, requiring more complex hardware or more computing time than is available in the vehicle). The uncertainty estimation of both models can be performed using a variety of different uncertainty estimation methods, such as ensembles, Monte Carlo dropout, Bayes by backprop, Laplace approximation, etc.These can also be used with any type of Deep Learning (DL) architecture, such as CNNs, RNNs, Transformers, etc. The required conditions for this solution are. 1. Use the predictions and uncertainty estimated by the online object detector and the off-board object detector to select samples for labeling. Samples with high uncertainty, or where the predictions of the off-board and online models do not agree, should be labeled. 2. Use the uncertainties estimated by the off-board object detector in the selected samples to select which samples should be labeled automatically and which should be labeled manually. 3. Use the uncertainties estimated by the off-board object detector for the samples that were automatically labeled as weights when training the online object detector.

[0022] To support the above conditions, a sample can be an image, a frame from a lidar sensor, or even a sequence or set of images or frames. There are several uncertainty estimation methods that can be used in the context of this invention, such as ensembles, Monte Carlo dropout, Bayes by backprop, Laplace approximation, etc. These can also be used with any type of deep learning (DL) architecture, e.g., CNNs, RNNs, transformers, etc. However, the specific method of uncertainty estimation is not important for the invention; rather, it is the use of said uncertainty estimation to improve the method. If the off-board object detector has a high uncertainty (above a chosen threshold), the result of automatic labeling should not be trusted, and the sample should be forwarded to manual labeling.

[0023] The present invention bridges the gap between automatic labeling and active learning methods for training online object detectors by leveraging a larger, more powerful off-board object detector and uncertainty estimation. It uses an uncertainty estimation method to select the most informative samples in the dataset by choosing those where the online and off-board object detectors disagree or where the greatest uncertainty exists. While the prior art sends all selected samples to a human labeler, here, those that can be confidently classified by the off-board detector are automatically labeled, while samples with high off-board detector uncertainty are sent to a human labeler for review and refinement.In this way, the advantages of active learning methods, which allow only the most useful samples to be included in the training set, are combined with the advantages of automatic labeling methods, which reduce the cost of labeling through automation. Furthermore, by capturing the uncertainty of the off-board detector's predictions, the online object detector's training procedure can mitigate the effects of incorrect auto-labels. The result is a more accurate, informative, and cost-effective dataset, which improves the performance of the online object detector. Short description of the drawings

[0024] To facilitate understanding of the present application, drawings are attached which illustrate preferred embodiments but are not intended to limit the technology disclosed herein. Fig. 1 - shows a possible embodiment of the invention, in particular a computer-implemented method for automatically labeling data samples with uncertainty quantification (100), where the reference numbers stand for the following: 10 input samples; 101 online object detection module with uncertainty estimation; 102 Off-board object detection module with uncertainty estimation; 101a Classification and uncertainty estimation from the online module for object recognition with uncertainty estimation; 102a Classification and uncertainty estimation from the off-board object detection module with uncertainty estimation; 103 Agreement of classifications; 104 high off-board uncertainty; 105 both uncertainties are small; 106 high off-board uncertainty; 11 Storing the prediction and uncertainty estimation of the off-board object detection module with uncertainty estimation; 12 Send for human labeling; 13 Discard input sample; 20 No; 21 Yes. Fig. Figure 2 shows a computer-implemented training procedure for the online object detection module with uncertainty estimation (101), where the reference numbers stand for the following: 201 stored automatically labeled samples with uncertainty estimates; 202 stored human-labeled samples; 203 Sample weighting module; 204 stored weighted samples; 205 training module. Description of the embodiments

[0025] Some embodiments will now be described in more detail with reference to the figures, which, however, do not serve to limit the scope of protection of the present application.

[0026] In a particular embodiment of the present invention, which is described in Fig. As shown, in particular in the flowchart for the proposed method for automatic labeling of data samples with uncertainty quantification (100), an input sample (10) is fed into both an online object detection module with uncertainty estimation (101) and an off-board object detector with uncertainty estimation (102). The classification and uncertainty values ​​(101a, 102a) determined in both modules are then compared to check whether they agree (103), i.e. whether the classification result (101a) of the online classification and uncertainty module (101) agrees with the classification result (102a) of the off-board classification and uncertainty module (102) and therefore the same classification class is predicted.

[0027] If the classification agreement analysis (103) results in no agreement (20) between the results of both modules, a check of the off-board uncertainty results (104) is performed: If the off-board uncertainty (104) is high, yes (21), the input sample (10) is sent for human labeling (12). Otherwise, if the off-board uncertainty (104) is low, no (20), the input sample (10) is stored (11) together with the classification and uncertainty estimation (102a) of the off-board object detection module with uncertainty estimation (102), so that said sample (10) can be used to train an online object detection module (101).The stored sample (11) includes uncertainties that are automatically labeled within said sample (with classification and uncertainty estimation) to be used for training an object detection model by becoming part of the labeled training dataset.

[0028] If there is a match, yes (21), between the classification (103) of the two results of the classification modules, the next step involves checking the uncertainties (105) of the results of both models. If both uncertainties are low, yes (21), then the input sample (10) is discarded (13). If the uncertainties are not both low, no (20), i.e., the uncertainty of at least one of the models is high, then an off-board uncertainty analysis (106) is performed. If the off-board uncertainty (106) is high, yes (21), then the input sample (10) is sent for human labeling (12). Otherwise, in case of a negative result (20), the input sample (10) is stored (11) together with the classification and the uncertainty estimate (102a) of the off-board object detection module with uncertainty estimation (102).

[0029] Fig.Figure 2 illustrates the training procedure of an online object detection module with uncertainty estimation (101). It contains stored auto-labeled samples with uncertainty estimates (201) and stored human-labeled samples (202), which are combined for training in a sample weighting module (203). This weighting module (203) assigns less weight to uncertain auto-labels (201) and more weight to human-labeled samples (202). The weighted samples are then stored in a premise (204).

[0030] An example of how the additional sample-wise uncertainty can be used to improve object detection performance is its use as a scaling factor in the loss calculation during the training process (205), which follows the previously mentioned steps. The stored sample weights (204) define the contribution of the individual samples (10) to the overall loss. When determining object classification, the categorical cross-entropy loss is typically used, given by Equation 1: Loss=1N∑n=1N∑c=1Cyn,c⋅log y^n,c

[0031] The sample weights (204) can be easily integrated into the loss by multiplication and given by equation 2, so that labels that are more secure or were labeled by a human are given a higher weight in the loss calculation. Loss=1N∑n=1N∑c=1Cyn,c⋅log y^n,c⋅wn

[0032] The same principle can also be applied to the bounding-box regression loss. To calculate these weights on a device appropriately configured for this purpose, e.g., a processing unit, the solution is to normalize the uncertainty values ​​across the entire dataset so that they are defined within a fixed range. Or, a theoretical maximum can be determined and the uncertainties scaled accordingly so that the weights are set within a fixed range. For human-labeled samples (202), the maximum of this fixed range is assumed, reflecting maximum confidence in these samples.

[0033] N is the number of samples in the batch, C is the number of classes, y n,c is the one-hot encoding of the true class, ŷ n,c is the output of the model and w nis the weight given to sample n according to the uncertainty in its label.

[0034] The present invention therefore discloses a method for automatically labeling data samples with uncertainty quantification (100) that relies on the use of an online and an off-board object detector with uncertainty estimation (101, 102) to determine which samples should be discarded / rejected (13), which samples should be sent for human labeling (12) or which samples should be sent for automatic labeling (11) in order to reduce the cost and time for labeling the samples (10) required for training the object detectors.

[0035] There are many uncertainty estimation methods that can be used in this invention, such as ensembles, Monte Carlo dropout, Bayes by backprop, Laplace approximation, etc. These can be used with any type of deep learning (DL) architecture, e.g., CNNs, RNNs, transformers, etc.

[0036] This method is not limited to modules and / or algorithms for object detection, but can also be used for semantic segmentation, panoptic segmentation, or other algorithmic approaches. The invention is also applicable to various sensor modalities, such as cameras, radar, ultrasound, and lidar.

[0037] The method (100) for automatically labeling input data samples (10) with uncertainty quantification is intended to be carried out using a data processing apparatus / device / system comprising the technical means for carrying out all the above-mentioned steps as well as for storing data resulting from or supplied to said analysis method.

Claims

[1] Method (100) for automatically labeling input data samples (10) with uncertainty quantification, comprising the following steps: simultaneously feeding an input data sample (10) into an online object detection module with uncertainty estimation (101) and into an off-board object detector with uncertainty estimation (102); Comparing (103) resulting classification and uncertainty estimation data (101a) determined by the online object detection module with uncertainty estimation (101) with resulting classification and Uncertainty estimation data (102a) determined by the off-board object detection module with uncertainty estimation (102); and Determine whether the input sample (10) is discarded (13), sent for labeling by a human (12) or for archiving in the off-board object detection module with uncertainty estimation (102). [2] Method (100) according to the preceding claim, wherein the input data sample (10) comprises at least one of the set: an image, a frame from a lidar sensor, or even a sequence of images or frames. [3] Method (100) according to claim 1, wherein a resulting input data sample of the online object detection module with uncertainty estimation (101) and the off-board object detector with uncertainty estimation (102) each comprises detected objects of the input data sample (10). [4] The method (100) of claim 1, wherein the resulting classification and uncertainty estimation data from both the online classification and uncertainty estimation module (101a) and the off-board classification and uncertainty estimation module (102a) comprise at least one of: classification of an input data sample (10), and an uncertainty estimation of an input data sample. [5] Method (100) according to claims 1 and 4, wherein, if the comparison (103) between the resulting classification and uncertainty estimation data is true (21), an uncertainty comparison (105) is performed between both uncertainty estimates of the input data sample (10); and if the comparison (103) between the resulting classification and uncertainty estimation data is false (20), an off-board analysis (104) for high uncertainty is performed. [6] The method (100) of claim 5, wherein if the off-board analysis (104) for high uncertainty is true (21), the input sample (10) is sent for human labeling (12), and if the off-board analysis (104) for high uncertainty is false (20), the input sample (10) is stored (11) in the off-board object detection module with uncertainty estimation (102) so that said sample (10) can be used to train an online object detector. [7] The method (100) of claim 5, wherein if the comparison (105) of the uncertainties between the two uncertainty estimates of the input data sample (10) is low (21), the input data sample (10) is discarded (13), and if the comparison (105) of the uncertainties between the two uncertainty estimates of the input data sample (10) is not low (20), an off-board analysis (106) for high uncertainty is performed. [8] The method (100) of claim 7, wherein if the off-board analysis (106) for high uncertainty is true (21), the input sample (10) is sent for human labeling (12), and if the off-board analysis (106) for high uncertainty is false (20), it is stored (11) in the off-board object detection module with uncertainty estimation (102) so that said sample (10) can be used to train an online object detection module (101). [9] The method (100) of claims 1, 3 and 4, wherein the online object recognition module with uncertainty estimation (101) contains stored automatically labeled samples with uncertainty estimates (201) and stored human-labeled samples (202). [10] Method (100) according to claims 1, 3, 4 and 9, wherein the stored automatically labeled samples (201) and the stored human-labeled samples (202) are combined in a sample weighting module (203) for training. [11] Method (100) according to one of claims 1, 3, 4, 9 and 10, wherein the weighted samples are stored (204), wherein stored automatically labeled samples (201) are deprioritized and human-labeled samples (202) are prioritized to be forwarded to a training module.

Images