Method and system for automatically marking radar data

Abstract

Method for automatically tagging sensor data, a vehicle comprising radar detection means, a camera and a lidar, the fields of view of which at least partially overlap in an overlapping area, providing a set of radar points, a set of image data and a set of lidar data, automatically assigning a confidence tag to each radar point in such a way that the image data is corrected, the camera-based depth estimate generated by a neural network is calibrated in the overlapping area of the fields of view of the camera and the lidar by means of the lidar data, whereby a three-dimensional point cloud representation is calculated from two-dimensional image information, the three-dimensional point cloud representation is associated with the radar points using the k-nearest neighbor algorithm and the lidar data is associated with the radar points to obtain radar / lidar confidence and radar / camera confidence and these are combined into an optically-based confidence, a radar / tracking confidence is assigned to the radar points by means of tracking with odometry data, this is combined with the optically-based confidence and a confidence tag is assigned. A system for implementing the method is also proposed.

Description

Technical Field

[0001] The present invention relates to a method for automatically labeling radar data or radar points obtained from radar detection means in a vehicle, which has at least auxiliary autonomous driving characteristics or can also be used as a data collector for developing such characteristics. In addition, a system for implementing this method is also proposed. Background Art

[0002] Labeling or annotation is a mark made on data or on objects and environmental features of dynamic and static types represented as data. In machine learning, especially in supervised learning, labeling is a necessary step in analyzing raw data obtained, for example, from a radar. The annotation of conceivable radar signals here is the confidence level corresponding to the detection result, in order to distinguish artifacts from the detection result. Radar artifacts can be understood as effects such as measurement noise, multiple reflections, and false detections. Points with a high confidence level represent reliable radar reflections of, for example, the vehicle's surrounding environment, building structures such as walls, and pillars and other restrictions in the traffic environment. Radar data at the detection level can only be manually labeled as point cloud data in a very complex, time-consuming, and expensive manner (for example, there are hundreds of points per measurement cycle, and each measurement cycle may be in the double-digit millisecond range). Manual operation provides a large margin for errors or inconsistencies here. In addition, there is no automatic labeling process that can achieve this purpose.

[0003] In document DE 10 2017 203 344 A1, the detection of elongated objects relative to the host vehicle is described, in which these objects are detected by a plurality of object detection means and patterns corresponding to these elongated objects are identified in the detection data. The detection data here consists of data that has been merged from at least two of the plurality of object detection means.

[0004] European document EP 3 293 669 A1 discloses an object recognition system having a camera, the output of which provides information about possible objects. The processor, based on the information of at least one additional detector (non-camera), selects the part of the camera output that may contain the possible object and determines the object attachment of that part.

[0005] A method is shown in the document WO 2019 / 161300 A1, in which sensor data from the field of view of at least one sensor is fed to a first neural network and the resulting object data is assigned to the corresponding positions of the detected objects. Based at least in part on these corresponding positions, the cluster characteristics at the detected objects are fed to a second neural network and a confidence value is thereby obtained, where the confidence value represents the likelihood that the cluster corresponds to an object in the surrounding environment. SUMMARY OF THE INVENTION

[0006] In this context, the object of the present invention is to provide a method operating at the detection level for automatically labeling radar points directly obtained from radar detection means. In addition, a system capable of implementing this method should be provided.

[0007] To achieve the above object, a method for automatically labeling sensor data of an observed scene is proposed, where a vehicle includes radar detection means and at least one camera as an optical sensor as well as a LiDAR (Light Detection and Ranging), and the observed scene is, for example, the surrounding environment of the vehicle, where the radar detection means, the camera, and the LiDAR each have at least a part of the surrounding environment of the vehicle as the field of view and the corresponding fields of view overlap at least partially in corresponding overlapping regions. In a sequence of time steps, for each time step t, a set of three-dimensional radar points is provided by the radar detection means, a set of two-dimensional image data is provided by at least one camera, and a set of three-dimensional LiDAR data is provided by the LiDAR. For each time step, a corresponding confidence label is automatically assigned to each radar point in the following manner:

[0008] · The image data is corrected by image correction and subsequent perspective transformation to obtain a rectified view of the scene;

[0009] · In the overlapping region of the fields of view of the camera and the LiDAR, the camera-based depth estimate generated by a neural network is calibrated with the aid of the LiDAR data;

[0010] · With the aid of the depth estimate based on the camera (and taking into account intrinsic camera parameters - such as the focal length), the three-dimensional point cloud representation is calculated from the two-dimensional image information

[0011] · By using the k-nearest neighbor algorithm, the three-dimensional point cloud representation is associated with the radar points and the LiDAR data is also associated with the radar points, thereby obtaining radar / LiDAR confidence and radar / camera confidence for the overlapping region of the field of view, taking into account the Euclidean distance and uncertainty;

[0012] · Combine the radar / lidar confidence and the radar / camera confidence for the corresponding field of view into a combined, optics-based confidence;

[0013] · In parallel therewith, assign a radar / tracking confidence to each radar point by means of tracking while taking into account the odometer data of the vehicle;

[0014] · Combine the optics-based confidence and the radar / tracking confidence and subsequently assign a binary confidence label which characterizes whether the corresponding radar detection means describes an artifact or a true and reliable reflection.

[0015] Conversion to the three-dimensional point cloud representation can be carried out by depth estimation which necessarily does not exist in the two-dimensional image. This conversion also corresponds to a projection.

[0016] The k-nearest neighbor (kNN) algorithm spans a circle in two dimensions and a sphere in three dimensions around a first point capable of obtaining k nearest neighbors, and their respective radii grow until the number k of neighboring points is within the respective radius. By performing the corresponding k-nearest neighbor search when traversing the image data or the three-dimensional point cloud representation and the lidar data pixel by pixel, the corresponding point selection can be associated with the radar detection means.

[0017] The radar signal levels in ascending order are: 1. The raw data in the spectrum. 2. The detection data in the form of 3D points. 3. The list of tracked aggregated objects. By means of the method according to the invention, the labeling of the confidence of the radar points is advantageously carried out already at the detection level (i.e., immediately after obtaining / combining the radar points from the raw data in the spectrum) and before the data is passed on to a higher-level application. Thus, together with the three-dimensional radar points, the correspondingly assigned confidence is provided to the higher-level application as a binary confidence label, for example with a value of 0 (artifact) or 1 (true and reliable reflection). The higher-level application can be, for example, a function for autonomous driving or an autopilot. Thus, the method according to the invention provides an automated pipeline for automatically calculating the most obvious or most likely correlation of the radar information from the corresponding optical sensors (here from a so-called vehicle sensor device having a radar, a lidar and at least one camera). The method according to the invention can also be referred to in English as “Radar Artifact Labeling Framework”, abbreviated as RALF.

[0018] In one embodiment of the method according to the invention, for point tracking in time of the radar / tracking credibility of radar points, the angular velocity of the yaw angle and the translational velocity of the odometer data from the vehicle are used. This radar / tracking credibility is merged with the combined, optically-based credibility. The odometer is an additional in-vehicle sensing device that provides its data via the CAN bus.

[0019] In another embodiment of the method according to the invention, the radar detection means consists of a plurality of radar sensors. Each radar sensor can typically be arranged at the respective corners and on the sides of the vehicle. Similarly, for example, a plurality of cameras in at least one camera can be arranged centrally on all sides of the vehicle.

[0020] In yet another embodiment of the method according to the invention, in each overlapping region of the lidar and the camera, the consistency between the respective credibilities is checked for the corresponding identical points of the respective fields of view. This mutual verification between the information in the image data and the lidar data improves the reliability of the credibility label assignment. In addition, it is advantageously achieved hereby that the depth estimation in the image data of at least one camera is on the same scale as that obtained from the measured lidar distances.

[0021] Furthermore, a system for automatically labeling sensor data of a scene is claimed, wherein the vehicle includes radar detection means and at least one camera as an optical sensor as well as a lidar. The radar detection means, the camera, and the lidar each have at least a part of the vehicle's surroundings as a field of view, wherein the respective fields of view overlap at least partially in respective overlapping regions. In a sequence of time steps, for each time step t, the radar detection means can provide or prepare a set of three-dimensional radar points per se, the at least one camera can provide or prepare a set of two-dimensional image data per se, and the lidar can provide or prepare a set of three-dimensional lidar data per se. For each time step, a corresponding credibility label is automatically assigned or can be assigned to each radar point. The system is configured to:

[0022] · Correct the image data by image correction and subsequent perspective transformation to achieve a straight view of the scene,

[0023] · In the overlapping region of the fields of view of the camera and the lidar, calibrate the camera-based depth estimation generated by a neural network with the aid of lidar data,

[0024] · Calculate a three-dimensional point cloud representation based on two-dimensional image information with the aid of the camera-based depth estimation,

[0025] ·By using the k-nearest neighbor algorithm, not only is the three-dimensional point cloud representation associated with the radar points, but also the lidar data is associated with the radar points, thereby obtaining radar / lidar credibility and radar / camera credibility for the overlapping regions of the field of view, taking into account the Euclidean distance and uncertainty.

[0026] ·The radar / lidar credibility and radar / camera credibility are combined into a combined, optically-based credibility.

[0027] ·In parallel, by means of tracking while considering the odometer data of the vehicle, a radar / tracking credibility is assigned to each radar point.

[0028] ·The optically-based credibility and radar / tracking credibility are combined and then a binary credibility label is assigned, which characterizes whether the corresponding radar detection means describes an artifact or a true and credible reflection.

[0029] In one design of the system according to the present invention, the system is configured to: for point tracking in time of the radar / tracking credibility of the radar points, use the angular velocity of the yaw angle and the translational velocity of the odometer data of the vehicle, and combine the radar / tracking credibility with the combined, optically-based credibility.

[0030] In another design of the system according to the present invention, the radar detection means consists of a plurality of radar sensors distributed around the vehicle.

[0031] In yet another design of the system according to the present invention, the system is configured to: in the respective overlapping regions of the lidar and the camera, check the consistency between the respective credibilities for the corresponding same points of the respective fields of view and thereby improve the reliability of the label assignment.

[0032] Other advantages and designs of the present invention are derived from the description and the drawings.

[0033] It goes without saying that, without departing from the scope of the present invention, the features mentioned above and those still to be described below can be used not only in the respective given combinations, but also in other combinations or individually. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematically shows the structural architecture of an embodiment of the method according to the present invention.

[0035] Figure 2 Shows a view of a camera image of a design of the system according to the present invention and a credibility label superimposed on lidar data. DETAILED DESCRIPTION

[0036] Figure 1 Schematically shows the architecture 100 of an embodiment of the method according to the present invention. For a time step t, an optical observation 110 is performed by an optical sensor (such as a lidar 111 on the one hand and a camera 112 for observing the vehicle surroundings on the other hand). The image data recorded by the camera 112 is accordingly corrected in a correction module 113. According to the lidar data item P lidar,t 101 of the lidar 111, the depth information is used for depth estimation 114 of the image data and for calculating a three-dimensional point cloud representation. In a lidar matching module 115, the lidar data item P lidar,t 101 is associated with a radar data item P radar,t 105 by using a k-nearest neighbor algorithm and a radar / lidar confidence w lm (p i,t )102 is generated. In a camera matching module 116, the three-dimensional point cloud representation is associated with a radar data item P radar,t 105 by using a k-nearest neighbor algorithm and a radar / camera confidence w cm (p i,t )103 is generated. The radar / lidar confidence w lm (p i,t )102 and the radar / camera confidence w cm (p i,t )103 are combined in a combination module 117 into a combined, optically based confidence w opt (p i,t )104, where non-overlapping regions of the fields of view of the lidar 111 and the camera 112 are also considered, and the non-overlapping regions represent the blind spots of the respective other optical sensor. In addition, for the time step t, a signal analysis 120 of the radar sensor 121 in terms of time is performed, and after collecting the raw data of the radar sensor 121 through this signal analysis, a radar point P radar,t 105 is obtained. At the same time, there is odometer data 106 according to the odometer 130 of the vehicle at the angular velocity of the yaw angle and the translational velocity v t . With the aid of the odometer data 106, a time-wise tracking of the radar / tracking confidence w tr (p i,t )107 is performed by a tracking module 123. According to the combination of the combined, optically based confidence w opt (p i,t )104 and the radar / tracking confidence w tr (p i,t )107, a binary confidence flag for differentiating between artifacts and objects is obtained (p i,t ) is used as the result 108.

[0037] Figure 2 Fig. 220 shows a camera image 210 of a design of the system according to the present invention and a view with confidence markers 202, 203 of lidar data 201 superimposed thereon. Here they are respectively the same observation scene.

[0038] List of reference numerals

[0039] 100 Structure architecture of the method

[0040] 101 P lidar,t (Lidar data item at time point t)

[0041] 102 w lm (p i,t )(Radar / lidar confidence)

[0042] 103 w cm (pi,t) (Radar / camera confidence)

[0043] 104 w opt (p i,t )(Combined radar / lidar confidence and radar / camera confidence)

[0044] 105 P radar,t (Radar data item at time point t)

[0045] 106 (Angular velocity of yaw angle and translational velocity)

[0046] 107 w tr (pi,t) (Point tracking confidence at time point t)

[0047] 108 (p i,t )(Confidence marker for point p i,t ) )

[0048] 110 Optical observation

[0049] 111 Lidar

[0050] 112 Surrounding environment camera

[0051] 113 Image correction and perspective transformation

[0052] 114 Depth estimation

[0053] 115 Lidar matching module

[0054] 116 Camera matching module

[0055] 117 Combination of non-overlapping fields of view

[0056] 120 Signal analysis in terms of time

[0057] 121 Radar sensor

[0058] 122 Aggregation

[0059] 123 Tracking module

[0060] 130 Odometer (travel distance measurement / position / orientation)

[0061] 140 Merging and marking

[0062] 210 Camera image 220 View with LiDAR confidence marks superimposed

[0063] 201 LiDAR

[0064] 202 Radar artifacts (confidence marks )

[0065] 203 Genuine and reliable radar detections (confidence marks )

Claims

1. A method (100) for automatically labeling sensor data of a scene, wherein, The vehicle includes a first radar detection means (121), at least one camera (112) as an optical sensor, and a lidar (111), wherein the first radar detection means (121), the camera (112), and the lidar (111) each have at least a part of the vehicle's surrounding environment as a field of view, and the corresponding fields of view overlap at least partially in corresponding overlapping regions. Wherein, in a sequence of time steps, for each time step t, a first radar detection means (121) provides a set of three-dimensional radar points (105), the at least one camera provides a set of two-dimensional image data (210), and the lidar (111) provides a set of three-dimensional lidar data (101, 201), and wherein, for each time step, a corresponding confidence label (108, 202, 203) is automatically assigned to each radar point in such a way that: · The image data (210) is corrected by image correction (113) and subsequent perspective transformation to obtain a bird's-eye view of the scene. · In the overlapping region of the fields of view of the camera (112) and the lidar (111), the camera-based depth estimate (114) generated by a neural network is calibrated with the aid of lidar data (101, 201). · A three-dimensional point cloud representation is calculated from the two-dimensional image information with the aid of the camera-based depth estimate (114). · By using the k-nearest neighbor algorithm, the three-dimensional point cloud representation is associated not only with the radar points (105) but also the lidar data (101, 201) is associated with the radar points (105), whereby, taking into account the Euclidean distance and uncertainty, radar / lidar confidence (102) and radar / camera confidence (103) are obtained for the overlapping region of the field of view. · The radar / lidar confidence (102) and the radar / camera confidence (103) are combined into a combined, optically-based confidence (104). · In parallel, a radar / tracking confidence (107) is assigned to each radar point by means of tracking while taking into account the odometer data (106) of the vehicle. · The optically-based confidence (104) and the radar / tracking confidence (107) are combined and then a binary confidence label (108) is assigned, which binary confidence label characterizes whether the corresponding radar detection means describes an artifact or a credible reflection.

2. The method according to claim 1, wherein For time-wise point tracking of the radar / tracking confidence (107) of the radar points (105), the angular velocity and translational velocity of the yaw angle from the odometer data (106) of the vehicle are used, and the radar / tracking confidence (107) is combined with the combined, optically-based confidence (104).

3. The method according to claim 1 or 2, wherein, The first radar detection means (121) consists of a plurality of radar sensors.

4. The method according to claim 1 or 2, wherein, In each overlapping region of the lidar (111) and the camera (112), the consistency between the corresponding confidences (102, 103) for the corresponding identical points of the corresponding fields of view is checked, thereby improving the reliability of label assignment.

5. A system for automatically tagging sensor data of a scene, wherein, The vehicle includes a first radar detection means (121), at least one camera (112) as an optical sensor, and a lidar (111), wherein the first radar detection means (121), the camera (112), and the lidar (111) each have at least a part of the vehicle's surrounding environment as a field of view, and the corresponding fields of view overlap at least partially in a corresponding overlapping region. Wherein, in a sequence of time steps, for each time step t, the first radar detection means (121) can provide or prepare a set of three-dimensional radar points (105), at least one camera can provide or prepare a set of two-dimensional image data (210), and the lidar (111) can provide or prepare a set of three-dimensional lidar data (101, 201), and wherein, for each time step, a corresponding confidence label (108, 202, 203) is automatically assigned or can be assigned to each radar point, and wherein the system is configured to: · Correct the image data (210) through image correction (113) and subsequent perspective transformation to obtain a direct view of the scene. · In the overlapping region of the fields of view of the camera (112) and the lidar (111), calibrate the camera-based depth estimation (114) generated by a neural network with the aid of the lidar data (101, 201). · Calculate a three-dimensional point cloud representation based on two-dimensional image information with the aid of the camera-based depth estimation (114). · Associate the three-dimensional point cloud representation with the radar points (105) and associate the lidar data (101, 201) with the radar points (105) by using the k-nearest neighbor algorithm, thereby obtaining the radar / lidar confidence (102) and the radar / camera confidence (103) for the overlapping region of the field of view, taking into account the Euclidean distance and uncertainty. · Combine the radar / lidar confidence (102) and the radar / camera confidence (103) into a combined, optically-based confidence (104). · In parallel, assign a radar / tracking confidence (107) to each radar point by means of tracking while considering the odometer data (106) of the vehicle. · Combine the optically-based confidence (104) and the radar / tracking confidence (107) and then assign a binary confidence label (108), which characterizes whether the corresponding radar detection means describes an artifact or a real and reliable reflection.

6. The system according to claim 5, wherein the system is configured to: for time-wise point tracking of the radar / tracking confidence (107) of the radar points (105), use the angular velocity and translational velocity of the yaw angle from the odometer data (106) of the vehicle, and combine the radar / tracking confidence (107) with the combined, optically-based confidence (104).

7. The system according to claim 5 or 6, wherein, The first radar detection means (121) consists of a plurality of radar sensors distributed around the vehicle.

8. The system according to claim 5 or 6, the system being configured to: in respective overlapping regions of the lidar (111) and the camera (112), check the consistency between respective credibility levels (102, 103) for respective identical points of respective fields of view and thereby improve the reliability of label assignment.

Images