Calibration of object detector uncertainty

Abstract

The invention relates to a method, in particular a computer-implemented method, for calibrating at least one uncertainty ([sigma]) of an object detector neural network (NN), which neural network (NN) is designed to output at least one statement (mu) of a related object (O) and at least one uncertainty ([mu]) assigned to the statement (mu) as a function of input data. The invention also relates to a method, a computer program and a computer program product for detecting at least one object using an object detector comprising a neural network calibrated according to the invention. In the method, a neural network (NN) is provided with calibration data (Ikal) and indications (Ind) with respect to uncertainties (sigma), and the statement (mu) and the corresponding uncertainties (sigma) are determined by processing the calibration data (Ikal) using the neural network (NN) in accordance with parameters of the neural network (NN). Then, at least one error function (Error) is used to compare the normalized generated uncertainty ([sigma] nor) with a deviation of the statement ([mu]) from the indication (Ind), which is also the same normalized, and at least one calibration parameter (pkal) is taken into account for the error function (Error), the calibration parameter (pkal) being optimized on the basis of the comparison.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method, in particular a computer-implemented method, of calibrating at least one uncertainty of a neural network of an object detector, to a method, in particular a computer-implemented method, of detecting at least one object using an object detector comprising a calibrated neural network according to the present invention, to a computer program for performing a method according to the present invention, and to a computer program product having stored thereon a computer program of a method according to the present invention. BACKGROUND

[0002] In many cases, when using a neural network for regression or classification, it is important to determine an uncertainty related to each respective prediction in addition to the respective prediction itself. This is in particular relevant for situations where safety is of utmost importance.

[0003] A prominent example in this regard is an advanced driver assistance system (ADAS) where a neural network can be used for processing input data of various sensors such as radar sensors, lidar sensors or ultrasonic sensors and camera devices. The ADAS functions typically provided by an ADAS system are used on the one hand to assist the driver who still retains control over the driving of the vehicle and on the other hand can also enable fully automated driving depending on the degree of automation. Examples of ADAS functions include various methods of recognizing objects or obstacles on the roadway, methods of recognizing lane boundaries and / or keeping the vehicle in the lane, methods of recognizing rain on the windshield and methods of assisting or performing a parking process.

[0004] Each respective input data used by each respective ADAS sensor can have various problems such as insufficient sensor resolution, bad weather conditions or rare scenarios. In addition, a particular object can be occluded by other objects and thus not visible in the input data. For all these cases, it is important to know the uncertainty of each respective prediction.

[0005] From DE 102 020 215 860 Al a method is known for correcting input image data of multiple camera devices of a surround view system which can be affected by rain, light and / or dirt, the method processing the input image data detected by the camera devices by a trained neural network which also outputs a determined uncertainty of the image correction. A similar method for correcting camera input image data is known from DE 102 020 215 859 Al.

[0006] However, neural networks, in particular neural networks for regression tasks, i.e. neural networks that output continuous values as prediction results, often have the problem that the respective uncertainty is not accurate and thus cannot be used directly. Therefore, in order to improve the accuracy, the uncertainty needs to be calibrated or adjusted. This in particular relates to object localization using an object detector that is arranged before the object classification. Here, there is little or no information about the confidence, i.e. the reliability, of each respective prediction.

[0007] For example, a method for training a neural network for outputting predictions and associated uncertainties is known from DE 102019217300 A1, which takes the determined uncertainty into account by means of a loss function. In this way it can be ensured that the network can determine well-calibrated uncertainties when the distribution of the data to be processed is similar to the distribution of the training data.

[0008] A method for calibrating regression uncertainties exclusively is known from the article "Accurate Uncertainties for Deep Learning Using Calibrated Regression" by V. Kuleshov et al., published in 2018 on arxiv:1807.00263 (DOI: https: / / doi.org / 10.48550 / arXiv.1807.00263). However, the method is at risk of overfitting. Furthermore, it is based on isotonic regression and is therefore relatively computationally complex.

[0009] Another method for calibrating uncertainties is described in the article "Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference" by M-H. Laves et al., published in 2019 on arxiv:1909.13550 (DOI: https: / / doi.org / 10.48550 / arXiv.1909.13550). This method is less computationally intensive, but can be very sensitive to outliers and wrongly assigned variables, which is particularly important for real data sets.

[0010] Therefore, it is desirable that there is a possibility for calibrating object recognition uncertainties that has the features of high precision and high computational efficiency. SUMMARY

[0011] It is therefore the basic task of the present application to provide a possibility of calibrating uncertainties with high precision and high computational efficiency.

[0012] The basic task of the present application is solved by the method according to claim 1, the method according to claim 10, the computer program according to claim 13 and the computer program product according to claim 14.

[0013] With regard to the method, the basic task of the present application is solved by a method, in particular a computer-implemented method, of calibrating at least one uncertainty of an object detector neural network, the neural network being configured to output at least one statement about an object and at least one uncertainty assigned to the statement from input data. The method comprises the following steps:

[0014] providing calibration data for the neural network,

[0015] providing an indication / label related to the uncertainty of the statement for all calibration data,

[0016] generating the statement and the corresponding uncertainty for all calibration data by processing the calibration data using the neural network according to the parameters of the neural network,

[0017] normalizing the generated uncertainty of at least one statement,

[0018] comparing the normalized generated uncertainty with the deviation of the statement from the indication also normalized using at least one error function, wherein at least one calibration parameter is considered for the error function, and

[0019] optimizing the calibration parameter based on the comparison result.

[0020] According to the present application, the calibration of the uncertainty is separate from and subsequent to the training of the neural network. The calibration of the uncertainty is performed according to a calibration data set, in particular a separate calibration data set.

[0021] In an object detector, an object is first localized and then classified. Therefore, an object detector comprises a localization unit and a classification unit. The present application relates in particular to object localization and the resulting uncertainty in object localization.

[0022] The method according to the application has the advantage of being resource-efficient while at the same time being highly precise. It is therefore particularly suitable for use in conjunction with embedded systems, for example in driver assistance systems. Furthermore, the normalization of the uncertainty produced and the deviation from the statement and the indication, i.e. the Ground Truth, has the beneficial effect that the uncertainty of each statement is correctly calibrated independently of the object size and the object class.

[0023] According to an advantageous design, the neural network is a trained neural network. The method according to the application can be part of a training method for training the neural network or can be a separate method which is run independently of the training method.

[0024] The neural network is, for example, a convolutional neural network (English: convolutional neural network, abbreviation: CNN), a hybrid density network, a Transformer network, a recurrent neural network, a multilayer perceptron or a generative adversarial network.

[0025] In one design, the object detector is an anchor-based object detector, a center point-based object detector, a multi-stage object detector, a single-stage object detector, a convolutional neural network (CNN)-based object detector or a Transformer-based object detector. It can be, for example, a YOLO object detector, an SSD object detector or an EfficientDet object detector.

[0026] Advantageously, the statement is a statement about the position of the object, a statement about the properties of the object or a statement about the geometric dimensions of the object. The position of the object can be represented, for example, by one or more coordinates of the object. The properties of the object include, for example, the object type and the geometric dimensions, while the geometric dimensions relate, for example, to the geometric specifications, the size, the cross-sectional area or the volume of the object. It is pointed out that other statements about the object can also be made, which also fall within the scope of the application.

[0027] If a plurality of statements about one object class and / or statements about a plurality of object classes are output, the calibration is preferably carried out separately for each object class and / or for each statement. If the statement is a statement about the position of the object determined by a plurality of coordinates, each coordinate can be considered, for example, as a statement. However, it is also conceivable to combine all coordinates into one statement.

[0028] According to an advantageous design, the deviation between the statements and the indications and the uncertainty is normalized with respect to the geometric dimensions of the object, in particular the length, width, height, area or volume of the object. This approach has the advantage that the influence of different object sizes, aspect ratios, geometric specifications and bounding box sizes on the calibration can be taken into account. Thereby, for example, a negative influence of larger objects or bounding boxes on the calibration of smaller objects or bounding boxes can be avoided.

[0029] It is furthermore advantageous to optimize the model parameters using an optimization method, in particular using isotonic regression, gradient methods, Platt scaling or binning according to a histogram.

[0030] In a design, the error function is a distance measure, in particular a mean of squared errors / mean of squared deviations or an absolute deviation. A distance measure is a metric or distance function for a metric space which assigns a non-negative real number to two elements or points in the space. Distance measures are symmetric and satisfy the triangle inequality. The present invention can also be used in combination with other error functions.

[0031] It is furthermore advantageous that the calibration parameters comprise a scaling factor which is considered in particular for the error function. The scaling factor can be one or more constant scaling coefficients, a scaling vector or a scaling matrix. After the calibration, the scaling of the uncertainty for all predictions can be considered. Using the scaling factor, the calibration of the uncertainty for all predictions can be achieved quickly, efficiently and precisely.

[0032] A further advantageous design comprises determining a calibration rule using which the at least one uncertainty determined using the neural network is calibrated. Thereby, the calibration rule is considered in particular when outputting the statements and the uncertainty using the neural network. The rule considers the at least one calibration parameter, can comprise a multiplication with a scalar value and can also comprise a model.

[0033] Furthermore, the basic task of the present invention is also solved by a method, in particular a computer-implemented method, for detecting at least one object using an object detector comprising a neural network calibrated according to the present invention. The method comprises the following steps:

[0034] - acquiring input data comprising at least one information about at least one object,

[0035] - providing the input data as input to the calibrated neural network which is configured to output at least one statement about the object and at least one uncertainty assigned to the statement from the input,

[0036] - normalizing the uncertainty,

[0037] - calibrating the normalized uncertainty, and

[0038] - outputting the statement and the calibrated, normalized uncertainty for the statement.

[0039] The input data can be detected using one or more sensors. For a driver assistance system, these sensors comprise, for example, different ADAS sensors. The statement about the object is, in particular, the position of the object. The method then involves object localization. The uncertainty is preferably normalized as it is produced in the course of the calibration method according to the calibration data. The uncertainty is normalized, in particular, for the geometric dimensions of the object. The calibration of the uncertainty is, in turn, preferably carried out according to a calibration rule determined in advance.

[0040] In one design variant, a calibrated absolute uncertainty is determined and output. To determine the absolute uncertainty, the calibrated, normalized uncertainty can be multiplied by a normalization factor.

[0041] The method for detecting at least one object using an object detector according to the application is preferably used in a driver assistance system. However, the method can also be used in other fields, for example, computer vision or medical imaging methods. The method for calibrating at least one uncertainty of an object detector neural network is also preferably used in connection with a driver assistance system.

[0042] In a driver assistance system, when an object detector is used for object detection, a statement about at least one object, for example, an object position or an object geometric dimension, is determined from at least one ADAS sensor, for example, a camera. The object can be another road user in the vehicle's surroundings, an animal, a lane boundary, an object in the vehicle's surroundings, or a traffic sign, a traffic light, or a light signal. In addition to the statement, the uncertainty of the statement made is also determined, calibrated, and output.

[0043] The basic task of the application is also solved by a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method for calibrating at least one uncertainty of a neural network or the method for detecting at least one object using an object detector according to the application according to one of the design variants, and by a computer program product having stored thereon the computer program according to the application. BRIEF DESCRIPTION OF DRAWINGS

[0044] The application is explained in detail below with reference to the following drawings. In which:

[0045] Figure 1 a first advantageous design variant of the method according to the application is shown, and

[0046] Figure 2 object detection using a neural network calibrated according to the application is shown.

[0047] In the figures, identical elements are provided with the same reference signs. DETAILED DESCRIPTION

[0048] For Figure 1 The shown advantageous design assumes that in the object detector there is a pre-trained neural network NN. The neural network NN is used to determine a statement μ about the object O and an uncertainty associated with the statement

[0049] The uncertainty output by the neural network NN is typically erroneous and does not correspond to the actually existing uncertainty. In order to also accurately predict the uncertainty , the uncertainty needs to be calibrated, which can be done by the method according to the invention. After calibration, the uncertainty can be interpreted probabilistically, i.e. statements such as "there is a 95% probability that the correct value lies within the predicted uncertainty interval" can be made.

[0050] For the calibration, first calibration data I kal are provided for the neural network NN, and for all calibration data I kal an indication Ind is provided. The calibration data I kal are processed using the neural network and based on the calibration data I kal a statement μ and a corresponding uncertainty are generated. Subsequently, the generated uncertainty is normalized and divided, for example, by a normalization factor n, i.e.

[0051]

[0052] Likewise, the deviation of the statement μ from the indication Ind is normalized using the normalization factor n. Then, the normalized generated uncertainty is compared to the normalized deviation of the statement μ from the indication Ind using an error function Error, and based on the comparison result at least one calibration parameter p kal is optimized.

[0053] If the error function is a distance measure that takes into account a scaling factor s and the scaling factor s is a constant scalar value, a function based on the mean squared error can be chosen as the error function Error, for example, for N detections, the function is given by the following equation:

[0054]

[0055] As an alternative, an error function Error based on the absolute deviation can be used, which is given by the following equation:

[0056]

[0057] Herein, is the absolute deviation, i.e. the difference between the indication Ind and the prediction μ.

[0058] Further, a calibration rule KAL can be determined, which can be used for object localization in the object detector for the design shown here, as Figure 2 is shown.

[0059] For localizing an object, first information about the vehicle's surroundings, in particular about at least one object O in the vehicle's surroundings, is detected using one or more ADAS sensors for the design shown in Figure 2 . These information are used as input Input for the neural network NN. The neural network NN is set up to make a statement about the position P of the at least one object from the input and to output an uncertainty σ corresponding to the position. In addition to the object position P, which can be given, for example, by the coordinates of the object's center point, a statement can also be made about the size of the object O, for example the length, width, height, area or volume of the object O. The uncertainty σ output by the network NN is then normalized σ nor and calibrated according to the calibration rule KAL. The result is a calibrated and normalized uncertainty σ kal,nor from which a calibrated absolute uncertainty can also be determined.

Claims

1. Method, in particular computer-implemented method, for calibrating at least one uncertainty (σ) of a neural network (NN) in an object detector, the neural network (NN) being designed to output at least one statement (μ) about an object (O) and at least one uncertainty (σ) associated with the statement (μ) from input data, the method comprising the following steps: - providing calibration data (I kal ) for a neural network (NN) - providing an indicator (Ind) related to the uncertainty (σ) of the statement for all calibration data, - generating, for all calibration data (I kal ), a statement and a corresponding uncertainty (σ) by processing the calibration data (I kal ) in accordance with the parameters of the neural network (NN) by means of the neural network (NN), kal ), a statement and a corresponding uncertainty (σ) by processing the calibration data (I kal ) in accordance with the parameters of the neural network (NN) by means of the neural network (NN), kal ), a statement - normalizing the generated uncertainty (σ) of at least one statement (μ), - comparing the normalized, generated uncertainty (σ nor ) with the deviation of the statement (μ) from the indication (Ind), which are also normalized identically, using at least one error function (Error), wherein at least one calibration parameter (p kal ) is taken into account for the error function (Error), and - optimizing calibration parameters (p kal ) based on the comparison.

2. The method according to claim 1, wherein the neural network (NN) is a trained neural network (NN).

3. The method according to claim 1 or 2, wherein the object detector is an anchor-based object detector, a center point-based object detector, a multi-stage object detector, a single-stage object detector, a convolutional neural network (CNN)-based object detector or a Transformer-based object detector.

4. The method according to at least one of the preceding claims, wherein the statement (μ) is a statement about a position (P), a property or a geometric dimension of the object (O).

5. The method according to at least one of the preceding claims, wherein the uncertainty (σ) is normalized with respect to a geometric dimension of the object (O), in particular a length, a width, a height, an area or a volume of the object (O).

6. The method according to at least one of the preceding claims, wherein the indicator (Ind) is normalized, in particular the indicator (Ind) is normalized with respect to a geometric dimension of the object (O).

7. The method according to at least one of the preceding claims, wherein the model parameters are optimized using an optimization method, in particular using isotonic regression, gradient method, Platt scaling or histogram binning.

8. The method according to at least one of the preceding claims, wherein the error function (Error) is a distance measure, in particular a mean squared error or an absolute deviation.

9. The method according to at least one of the preceding claims, wherein, a calibration rule (KAL) is determined by means of which at least one uncertainty (σ) determined with the neural network (NN) is calibrated.

10. Method, in particular computer-implemented method, for detecting at least one object (O) with an object detector, the object detector comprising a neural network (NN) calibrated according to at least one of the preceding claims, the method comprising the following steps: - acquiring input data (Input) comprising at least one information about at least one object (O), - providing the input data (Input) as input to the calibrated neural network, the neural network (NN) being configured to output at least one statement (μ) about the object (O) and at least one uncertainty (σ) associated with the statement from the input, - normalizing the uncertainty (σ), - the calibrated normalized uncertainty (σ nor ), and - outputting the statement (μ) and the calibrated, normalized uncertainty (σ kal,nor ) for the statement (μ).

11. The method according to claim 10, wherein the calibrated absolute uncertainty is determined and output.

12. Use of the method according to claim 10 or 11 or of the method according to any one of claims 1 to 9 for a driver assistance system.

13. Computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to claim 10 or 11 or the method according to any one of claims 1 to 9.

14. Computer program product having stored thereon the computer program according to claim 13.

Images