Computer-implemented method for determining an uncertainty of depth information, control device for carrying out such a method and sensor device with such a control device

Abstract

The invention relates to a computer-implemented method for determining an uncertainty (σ ρ ) at least one piece of depth information (p) detected by at least one sensor (5, 5.1, 5.2) and associated with an object (9) in the real world for use in an image processing method, wherein - in a first sensor image (13.1) taken from a first position (11.1) in the real world, a first image (15.1) of the object (9) is determined on an image plane of the at least one sensor (5, 5.1, 5.2), wherein - in a second sensor image (13.2) recorded from a second position (11.2) in the real world that is different from the first position (11.1), a second image (15.2) of the object (9) associated with the first image (15.1) is determined on the or another image plane of the at least one sensor (5, 5.1, 5.2), wherein - at least depending on the first image (15.1) and the second image (15.2), an actual object angle (β) of the object (9) to the at least one sensor (5, 5.1, 5.2), a position distance (L) between the first position (11.1) and the second position (11.2) and a position angle (α0) between the object (9) and the two positions (11.1, 11.2) are determined, characterized in that - at least depending on at least one basic parameter selected from a group comprising the actual object angle (β), the position distance (L), and the position angle (α0), the uncertainty (σ ρ ) is determined.

Description

[0001] The invention relates to a computer-implemented method for determining an uncertainty of at least one piece of depth information detected by at least one sensor and associated with an object in the real world for use in an image processing method.

[0002] Furthermore, the invention relates to a control device for carrying out such a method and a sensor device with such a control device. State of the art

[0003] Estimating the uncertainty of depth information is a well-known problem in sensor and image processing.

[0004] In the publication "Convergence and consistency analysis for extended Kalman filter based slam" by Shoudong Huang and Gamini Dissanayake in IEEE Transactions on Robotics, 23:1036-1049 (2007), it is assumed that the depth information contains Gaussian noise. Therefore, the uncertainty of the depth information is determined using a Kalman filter from the depth information containing Gaussian noise.

[0005] According to Harry H. Ku's publication "Notes on the Use of Propagation of Error Formulas" in the Journal of Research of the National Bureau of Standards - C. Engineering and Instrumentation, 70C:263-273 (1966), error propagation is used to determine the uncertainty of depth information. The disadvantage is that such a determination is extremely computationally intensive.

[0006] Furthermore, it is known from “Semi-supervised depth estimation from a single image based on confidence learning” by Hu Tian and Fei Li in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8573-8577 (2019) that the uncertainty of the depth information can be determined using an artificial neural network.

[0007] Furthermore, the international patent application WO 2020 / 188121 A1 discloses a computer-implemented method for perceiving structures in an environment.

[0008] In addition, a post-optimization method for 3D object detection with uncertainty-aware training from a monocular camera is known from the publication "MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships" by Yongjian Chen et al. in: 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020, pp. 12090-12099.

[0009] The publication "Fast Uncertainty Quantification for Deep Object Pose Estimation" by Guanya Shi et al. in 2021 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2021, pp. 5200-5207, also describes a method for quantifying the prediction uncertainty of position estimators with six degrees of freedom.

[0010] The technical problem to be solved is to provide a computer-implemented method for determining an uncertainty of at least one piece of depth information detected by at least one sensor and assigned to an object in the real world for use in an image processing method.

[0011] In the context of the present technical teaching, an image represents an object of the real world as at least one pixel on an image plane of a sensor.

[0012] In the context of the present technical teaching, an epipolar error angle is an angle between a sensor-object plane and a second line of sight. The sensor-object plane is spanned by a displacement vector of the sensor and a first line of sight.

[0013] In the context of the present technical teaching, a line of sight is a vector from the sensor, in particular a projection center of the sensor, to an image. Thus, the first line of sight is a vector from the sensor to a first image. Furthermore, the second line of sight is a vector from the sensor to a second image.

[0014] In the context of the present technical teaching, a position vector is a vector from a first position of the at least one sensor in the real world to a second position of the at least one sensor in the real world. Furthermore, a position distance is the length of the position vector. Disclosure of the invention

[0015] The computer-implemented method having the features of claim 1 has the advantage that the uncertainty from a plurality of sensors can be determined by means of sensor fusion. Furthermore, the method makes it possible to calculate the uncertainty effectively and simply. In particular, the method is independent of an intrinsic model of the at least one sensor, so that the method can advantageously be used for any sensors. Furthermore, the method is independent of an uncertainty of the images, so that the method advantageously depends only on the geometric conditions. In this case, a first image of the object on an image plane of the at least one sensor is determined in a first sensor image taken from a first position in the real world.Furthermore, in a second sensor recording taken from a second position in the real world that is different from the first position, a second image of the object is determined on the or a different image plane of the at least one sensor, said second image being associated with the first image. In addition, an actual object angle of the object to the at least one sensor, a position distance between the first position and the second position, and a position angle between the object and the two positions are determined, at least as a function of the first image and the second image. The uncertainty is determined, at least as a function of at least one basic parameter selected from a group comprising the actual object angle, the position distance, and the position angle. Preferably, a LiDAR sensor, a radar sensor, or a camera is used as the at least one sensor.In one embodiment, exactly one sensor is used as the at least one sensor, wherein the first image and the second image are then determined at different times. Alternatively, two sensors are used as the at least one sensor, in particular a first sensor in the first position and a second sensor in the second position, wherein the first image and the second image are then determined simultaneously or at different times. Particularly preferably, the first sensor and the second sensor are identical. In particular, the first image and the associated second image are known from a previous method step. Alternatively, the first image and the associated second image are determined during the method. In particular, the uncertainty of the depth information increases when the actual object angle becomes smaller.Alternatively or additionally, the uncertainty of the depth information increases as the position distance decreases. Alternatively or additionally, the uncertainty of the depth information increases as the position angle decreases.

[0016] Furthermore, in the context of the present technical teaching, the actual object angle is an angle between the first visual ray and the second visual ray.

[0017] In the context of the present technical teaching, the position angle is an angle between the position vector and the first line of sight. In particular, the position vector is known. In addition, the position distance is known.

[0018] Particularly preferably, the uncertainty is additionally determined as a function of a base parameter uncertainty of the at least one selected base parameter. Advantageously, this allows the uncertainty of the depth information to be determined efficiently and even more reliably.

[0019] According to a preferred development of the invention, it is provided that a first uncertainty of a first inverse distance from the first position to the object in the real world is determined as the uncertainty. Alternatively or additionally, a second uncertainty of a second inverse distance from the second position to the object in the real world is determined as the uncertainty. Advantageously, the first uncertainty and / or the second uncertainty are thus determined independently of one another. In particular, the depth information is the first inverse distance from the first position to the object in the real world. Alternatively or additionally, the depth information is preferably the second inverse distance from the second position to the object in the real world. In particular, the first inverse distance and / or the second inverse distance is determined by means of a triangulation method at least as a function of the at least one basic parameter.

[0020] Particularly preferably, an epipolar error angle between the first position, the second position, and the object is determined. The at least one base parameter uncertainty is determined at least as a function of an epipolar error angle. Using the epipolar error angle, errors and / or uncertainties in the line of sight can advantageously be taken into account when determining the uncertainty of the depth information. Furthermore, the uncertainty of the depth information is determined effectively and simply.

[0021] According to a further development of the invention, the image processing method is selected from a group comprising a method for probabilistic depth fusion, a method for object recognition, and a method for deep learning. Advantageously, the image processing method takes into account the uncertainty of the depth information in the respective method steps. Particularly preferably, the uncertainty of the depth information is used in an image processing method. In particular, the image processing methods are used in automotive systems—in particular in driver assistance, highly automated driving and parking, and / or driver monitoring—, robotics, photogrammetry, and / or cartography.

[0022] In particular, probabilistic depth fusion combines the uncertainty of the depth information with further depth information determined using stereo disparity, a mono- or multi-image approach based on deep learning, and / or a LiDAR sensor. The uncertainty of the depth information is advantageously used to optimally fuse the respective depth information using the uncertainty, so that good, especially reliable, depth information and poor, especially uncertain, depth information compensate each other.

[0023] In particular, the object detection method uses the uncertainty of the depth information to determine an uncertainty for the detection of an object.

[0024] Particularly preferably, a loss function of an algorithm is optimized by deep learning at least as a function of the uncertainty of the depth information. Advantageously, the algorithm is trained based on the uncertainty of the depth information.

[0025] The control unit according to the invention, with the features of claim 7, is specifically designed to execute the computer-implemented method according to the invention when used as intended. In connection with the control unit, the advantages already explained in connection with the computer-implemented method for determining the uncertainty of depth information arise in particular.

[0026] The sensor device according to the invention with the features of claim 8 has at least one sensor and a control unit according to the invention. In connection with the sensor device, the advantages that have already been explained in connection with the computer-implemented method for determining an uncertainty of depth information and the control unit arise in particular. Preferably, the at least one sensor is a LiDAR sensor, a radar sensor or a camera. In one embodiment, the sensor device has precisely one sensor. Alternatively, the sensor device has two sensors, wherein in particular a first sensor is arranged in a first position and a second sensor in a second position. Particularly preferably, the first sensor and the second sensor are identical.

[0027] Further advantages and preferred features and combinations of features emerge in particular from the above description and from the claims. The invention will be explained in more detail below with reference to the drawings. Fig. 1 a schematic representation of an embodiment of a motor vehicle, Fig. 2 a flowchart of an embodiment of the computer-implemented method for determining an uncertainty of depth information, and Fig. 3 a schematic representation of two embodiments of the motor vehicle in a plan view.

[0028] Fig. 1 shows a schematic representation of an embodiment of a motor vehicle 1 with a sensor device 3. The sensor device 3 has at least one sensor 5 and a control unit 7. The control unit 7 is operatively connected to the at least one sensor 5 in a manner not explicitly shown and is configured to control it. Furthermore, the control unit 7 is specially designed, when used as intended, to implement a computer-implemented method for determining an uncertainty σ ρ at least one depth information ρ detected by the at least one sensor 5 and associated with an object 9 of the real world for use in an image processing method. A preferred embodiment of the method is described in Fig. 2 is explained in more detail.

[0029] Fig. Figure 2 shows a flowchart of an embodiment of the computer-implemented method for determining the uncertainty σ ρthe at least one depth information p detected by the at least one sensor 5 and assigned to an object 9 in the real world.

[0030] Identical and functionally identical elements are provided with the same reference numerals in all figures, so that reference is made to the previous description.

[0031] In a step S1, in a first sensor image 13.1 taken from a first position 11.1 in the real world, a first Fig. .1 of the object 9 on an image plane of the at least one sensor 5. In particular, the first Fig. .1 is known from a previous process step not belonging to the process. Alternatively, the first Fig. .1 is determined in step S1.

[0032] In a step S2, in a second sensor image 13.2 taken from a second position 11.2 in the real world which is different from the first position 11.1, one of the first Fig. .1 assigned second Fig. .2 of the object 9 on the or another image plane of the at least one sensor 5. In particular, the second Fig. .2 is known from a previous process step not belonging to the process. Alternatively, the second Fig. .2 is determined in step S2.

[0033] In a step S3, at least depending on the first Fig. .1 and the second Fig. .2 an actual object angle β of the object 9 to the at least one sensor 5, a position distance L between the first position 11.1 and the second position 11.2 and a position angle α0 between the object 9 and the two positions 11 are determined.

[0034] In a step S4, at least depending on at least one basic parameter selected from a group comprising the actual object angle β, the position distance L, and the position angle α0, the uncertainty σ ρthe depth information ρ. Preferably, the uncertainty σ ρ additionally depending on a basis parameter uncertainty σ B from which at least one selected base parameter is determined. In particular, the uncertainty σ ρ of the depth information ρ can be understood as a standard deviation of the depth information ρ. Thus, taking into account a Gaussian normal distribution, an actual depth information ρ̅ results from the following equation. ρ¯≈N(ρ,σρ2)

[0035] Additionally, it can be assumed that the actual object angle β is based on a Gaussian normal distribution. Alternatively or additionally, it can be assumed that the position angle α0 is based on a Gaussian normal distribution. Alternatively or additionally, it can be assumed that the cross-correlation between the actual object angle β and the position angle α0 is zero. If the position angle α0 is subjected to a position angle error δα0 and the actual object angle β is subjected to an actual object angle error δβ, the depth information ρ also contains an error. In this case, the following equation applies to the depth information ρ. ρ(α0+δα0,β+δβ)≈ρ(α0,α)+∂ρ∂α0⋅δα0+∂ρ∂β⋅δβ

[0036] The uncertainty σ ρ The depth information ρ can be calculated using the derivatives of the depth information ρ and an uncertainty σ α0 of the position angle α0 and an uncertainty σ βof the actual object angle β using the equation σρ2=(∂ρ∂α0)2⋅σα02+(∂ρ∂β)2⋅σβ2 Assuming that the uncertainty σ α0 of the position angle α0 and the uncertainty σ β of the actual object angle β are equal and the equation σα0=σβ=σB applies, the determination of the uncertainty σ is simplified ρ the depth information ρ to σρ=σB⋅(∂ρ∂α0)2+(∂ρ∂β)2 and the uncertainty σ ρ the depth information ρ can be calculated due to the limitations of the L2 norm based on the inequality σρ≤max(|∂ρ∂α0|,|∂ρ∂β|)⋅σB⋅2 be estimated upwards.

[0037] In an optional step S5, an epipolar error angle y between the first position 11.1, the second position 11.2 and the object 9 is determined.

[0038] In a further optional step S6, the at least one basis parameter uncertainty σ B at least as a function of the epipolar error angle y. In particular, at least one basis parameter uncertainty σ B using the equation σB=max(σ^,C⋅γ) where σ̂ and C are two predefined parameters. This allows the uncertainty σ ρ the depth information ρ using the equation σρ≈max(|∂ρ∂α0|,|∂ρ∂β|)⋅max(σ^,C⋅γ) be determined.

[0039] In a further optional step S7, the uncertainty σ ρ the depth information ρ in an image processing method selected from a group comprising a method for probabilistic depth fusion, a method for object recognition, and a method for deep learning.

[0040] In a further optional step S8, the depth information ρ is determined based on the actual object angle β, the position distance L, and the position angle α0. In particular, the depth information ρ is also used in the image processing method in step S7. Alternatively or additionally, in step S4, the depth information ρ is also used to determine the uncertainty σ ρ the depth information ρ is taken into account.

[0041] Fig. 3 shows a schematic representation of two embodiments of the motor vehicle 1 in a plan view.

[0042] In particular, in step S4 Fig. 2 as uncertainty σ ρ a first uncertainty σ ρ1 a first inverse distance ρ1 from the first position 11.1 to the object 9 in the real world is determined. Alternatively or additionally, in step S4 Fig. 2 as uncertainty σ ρ a second uncertainty σ ρ2a second inverse distance ρ2 from the second position 11.2 to the object 9 in the real world.

[0043] In particular, the first inverse distance ρ1 from the first position 11.1 to the object 9 is determined using the equation ρ1=tan βL⋅(sin α0+cos α0⋅tan β) Furthermore, the second inverse distance ρ2 from the second position 11.2 to the object 9 is calculated using the equation ρ2=sin βL⋅sin α0 calculated.

[0044] If the second inverse distance ρ2 is considered as depth information p and thus the second uncertainty σ ρ2 determined, the second uncertainty σ ρ2 the following inequality σρ2≈max(|ρ2tan α0|,|cos βL⋅sin α0|)⋅max(σ^,C⋅γ). or the estimate σρ2≈max(|ρ2tan α0|,|cos βL⋅sin α0|)⋅max(σ^,C⋅γ).

[0045] Taking into account the following estimates for the trigonometric functions cos β≈1−β22 1tanα0≈1α0−α03 results for the second uncertainty σ ρ2 the equation σρ2≈max(ρ2⋅|1α0−α03|,|1−β22L⋅sin α0|)⋅max(σ^,C⋅γ).

[0046] In Fig. 3 a), the sensor device 3 of the motor vehicle 1 has a first sensor 5.1 and a second sensor 5.2. The first sensor 5.1 is arranged at the first position 11.1 and the second sensor 5.2 at the second position 11.2, with the two sensors 5.1, 5.2 having the positional distance L.

[0047] Furthermore, a first line of sight 17.1 from the first sensor 5.1 to the object 9 and a second line of sight 17.2 from the second sensor 5.2 to the object 9 are shown. The first line of sight 17.1 has the length 1ρ1 Furthermore, the second line of sight 17.2 has the length 1ρ2 Furthermore, the first line of sight 17.1 and the second line of sight 17.2 enclose the actual object angle β, and the first line of sight 17.1 and a position vector 19 enclose the position angle α0.

[0048] Furthermore, the first line of sight 17.1 and the position vector 19 define a sensor-object plane with the normal n→ The first line of sight 17.1 is orthogonal to the normal n→ and the position vector 19 is orthogonal to the normal n→. This calculates the epipolar error angle γ as the angle between the sensor-object plane and the second line of sight 17.2.

[0049] In Fig.3 b), the sensor device 3 of the motor vehicle 1 has exactly one sensor 5. Furthermore, the motor vehicle 1 is shown at two different points in time, whereby the sensor 5 is arranged once in the first position 11.1 and once in the second position 11.2. The motor vehicle 1 and the control unit 7 are shown in dashed lines in the second position 11.2.

Claims

[1] Computer-implemented method for determining an uncertainty (σρ) of at least one piece of depth information (ρ) detected by at least one sensor (5, 5.1, 5.2) and associated with an object (9) of the real world for use in an image processing method, wherein - in a first sensor image (13.1) taken from a first position (11.1) in the real world, a first image (15.1) of the object (9) is determined on an image plane of the at least one sensor (5, 5.1, 5.2), wherein - in a second sensor image (13.2) recorded from a second position (11.2) in the real world that is different from the first position (11.1), a second image (15.2) of the object (9) associated with the first image (15.1) is determined on the or another image plane of the at least one sensor (5, 5.1, 5.2), wherein - at least depending on the first image (15.1) and the second image (15.2), an actual object angle (β) of the object (9) to the at least one sensor (5, 5.1, 5.2), a position distance (L) between the first position (11.1) and the second position (11.2) and a position angle (α0) between the object (9) and the two positions (11.1, 11.2) are determined, characterized by , that - at least depending on at least one basic parameter selected from a group comprising the actual object angle (β), the position distance (L), and the position angle (α0), the uncertainty (σ ρ ) is determined. [2] Method according to claim 1, characterized by that the uncertainty (σ ρ ) additionally depending on a basis parameter uncertainty (σ B ) from which at least one selected basic parameter is determined. [3] Method according to one of the preceding claims, characterized bythat the uncertainty (σ ρ ) - a first uncertainty (σ ρ,1 ) a first inverse distance (ρ1) from the first position (11.1) to the object (9) in the real world, and / or - a second uncertainty (σ ρ,2 ) of a second inverse distance (ρ2) from the second position (11.2) to the object (9) in the real world is determined. [4] Method according to one of the preceding claims, characterized by , that a) an epipolar error angle (y) is determined between the first position (11.1), the second position (11.2) and the object (9), and that b) at least one basis parameter uncertainty (σ B ) is determined at least as a function of an epipolar error angle (y). [5] Method according to one of the preceding claims, characterized bythat the image processing method is selected from a group comprising a method for probabilistic depth fusion, a method for object recognition, and a method for deep learning. [6] Method according to one of the preceding claims, characterized by that a loss function of an algorithm can be improved by deep learning at least as a function of the uncertainty (σ ρ ) of the depth information (ρ) is optimized. [7] Control device (7) which is specially adapted to carry out a computer-implemented method according to one of the preceding claims when used as intended. [8] Sensor device (3) with at least one sensor (5, 5.1, 52) and a control unit (7) according to claim 7.

Images