Method for annotating data from a vehicle environment and a driver assistance system trained with annotated data

Abstract

The invention relates to a computer-implemented method for assigning at least one annotation to modality-specific data (101, 102, 103) for capturing the vehicle environment of a vehicle. Raw objects are determined from the modality-specific data (101, 102, 103) and fused in a tracking and fusion step (50) to form cross-modality object candidates (104). In a supervision step (20), the object candidates (104) are compared with reference object assignments. If deviations are found, the tracking and fusion step (50) is adjusted. The invention further relates to a driver assistance system with a machine classifier equipped for the identification and / or classification of objects in the vehicle environment.

Description

[0001] The invention relates to a method for assigning annotations to modality-specific data that capture the vehicle environment of a vehicle, according to the preamble of claim 1. Furthermore, the invention relates to a driver assistance system with a classifier that has been trained with such annotated data, according to the preamble of claim 10.

[0002] Vehicles equipped for at least partially autonomous driving perceive their surroundings using typically multiple sensor modalities, such as radar, LiDAR, or ultrasonic sensors. Perception and fusion methods, adapted (trained) using machine learning and / or artificial intelligence techniques, have proven advantageous for identifying and classifying objects (e.g., other vehicles, pedestrians, or other road users) based on the modality-specific data provided by the sensor modalities.

[0003] Such processes require extensive annotated (labeled) modality-specific data, meaning data to which information about objects in the vehicle's environment is assigned. It is foreseeable that the need for such labeled data will increase further with the use of modern artificial intelligence methods in driver assistance systems for at least partially autonomous driving, for example, with the use of transformer networks.

[0004] Labeled data can be generated through manual interaction, whereby human observers (referred to as labelers) mark and classify objects in a captured vehicle environment.

[0005] However, in order to provide the required amounts of annotated modality-specific data quickly, efficiently and flexibly (for example, taking into account new object classifications), procedures are needed that can reduce the manual annotation effort as much as possible.

[0006] Publication US 2021 / 0125004 A1 describes a system and procedure for automatically labeling (annotating) data, in which automatically assigned labels are validated and / or corrected by a user.

[0007] According to a first aspect of the invention, the objective is to provide an improved method for assigning annotations to data used to capture the vehicle environment, in particular a method that is iteratively improved by incorporating manually or automatically acquired reference data. This objective is achieved according to the invention by a method with the features of claim 1.

[0008] Furthermore, the invention is based on the objective of providing a driver assistance system that has been trained with annotated data obtained using a method according to the first aspect of the invention. This objective is achieved according to the invention with a driver assistance system having the features of claim 10.

[0009] Advantageous embodiments of the invention are the subject of the dependent claims.

[0010] In a computer-implemented method for the at least partially automatic assignment of annotations (hereinafter referred to as autolabeling), modality-specific data is provided by at least one sensor of a vehicle that is set up to detect a vehicle environment, for example as radar data, LiDAR data or as camera images from a vehicle environment camera.

[0011] According to the invention, the method comprises at least one modality-specific detection step, a tracking and fusion step, a supervision step and an optimization step.

[0012] In each modality-specific detection step, assigned to the modality-specific data, modality-specific raw objects are determined from the modality-specific data. A trajectory can be assigned to each raw object; that is, a space-time curve that specifies the raw object's position as a function of time with respect to a coordinate system of the respective modality-specific sensor and / or with respect to a vehicle coordinate system that spans multiple sensors. Additionally, a modality-specific annotation can also be assigned to each raw object.

[0013] In a subsequent tracking and fusion step, the identified raw objects are fused into cross-modality object candidates, each of which is assigned a trajectory and at least one annotation.

[0014] In a subsequent supervision step, the assigned trajectory and at least one assigned annotation of the cross-modality object candidates are checked for their correspondence with a reference object assignment. The reference object assignment can be determined manually (for example, by inspection with a labeler), but it can also be determined at least partially automatically by a reference procedure that operates independently of the modality-specific detection step and independently of the tracking and fusion step.

[0015] Correctly identified object candidates (i.e., those consistent with the reference object assignment) are provided as labeled data. This labeled data can be used to train classifiers that continuously and in real time recognize and classify objects in a vehicle environment based on modality-specific data. Machine learning methods using labeled data are well-established.

[0016] For object candidates that are not correctly identified (i.e., those where the reference object assignment does not match the annotation and / or trajectory of the object candidates, or does not match completely), the tracking and fusion step is adjusted in an optimization step to achieve an improved match between the object candidate identified by the adjusted tracking and fusion step and the reference object assignment.

[0017] In particular, the deviation from the reference object assignment can be determined for incorrectly identified object candidates: - as a false positive assignment if an object class or annotation that should not actually be assigned (according to the reference object assignment) was assigned to the object candidate, - as a false negative assignment if an object class or annotation that should actually be assigned (according to the reference object assignment) was not assigned in the object candidate, - as a false association if actually corresponding modality-specific data (e.g. radar and LiDAR data) were not associated or were incorrectly associated or - as an inaccurate value in the case of a quantitative deviation from a value to be assigned according to the reference object assignment.

[0018] Depending on the type of deviation, adjustments can be made to the tracking and fusion steps. For example, in the case of false positive or false negative assignments, detection and clustering process steps can be adjusted. In the case of incorrect associations, data association can be optimized. For inaccurate values, methods for filtering, especially temporal filtering, the modality-specific data can be adapted (for example, the signal statistics underlying a Kalman filter).

[0019] One advantage of the proposed method is that the result of the supervision step, for example, manual labeling, is accumulated. This is achieved not only by continuously providing labeled data, but also by improving procedural steps (modality-specific detection steps and / or the tracking and fusion step) to such an extent that the effort required for subsequent supervision steps is reduced. This ensures that, on the one hand, at least the quality of manual labeling is achieved. On the other hand, with a sufficiently large amount of modality-specific data, the efficiency of the method can be improved to such an extent that only exceptional borderline cases need to be labeled manually.

[0020] Furthermore, by considering object candidates with their assigned trajectories in the supervision step, a point-in-time analysis of objects and their associated labels becomes obsolete. As a result, the effort of the procedure no longer scales with the observation duration during which the vehicle environment is sampled in modality-specific data, but only with the number of objects captured in the modality-specific data. This significantly reduces the effort required.

[0021] Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

[0022] This shows: Fig. 1. A schematic flowchart of a generic autolabeling process, Fig. 2 a schematic flowchart of an autolabeling process taking into account ground truth data as well as Fig. 3 a schematic flowchart of an autolabeling procedure without considering ground truth data.

[0023] Corresponding parts are marked with the same reference symbol in all figures.

[0024] Fig. Figure 1 shows a schematic flowchart for a process for transforming sensor data 100 into labeled data 200, which runs at least partially automatically (without the involvement of a human user) and is therefore called an autolabeling process.

[0025] Sensor data 100 comprises data acquired by a vehicle's surroundings sensors, for example, using a vehicle-side camera, radar, and / or LiDAR. Sensor data 100 may additionally include position data describing the vehicle's geoposition or its time-dependent trajectory. The data acquired by the vehicle's surroundings sensors can be referenced to a specific point in time and / or a specific geoposition.

[0026] The sensor data 100 can also include calibration data, which allows data from various vehicle environment sensors and / or other sensor data 100 to be related to each other. For example, such calibration data can enable the transformation of data from different spatially resolved vehicle environment sensors (e.g., a vehicle environment camera, a radar sensor, and a LiDAR sensor) into a common vehicle-related coordinate system.

[0027] As shown below, based on Fig. As explained in more detail in section 2 for an embodiment, sensor data 100 can also include ground-truth data 110, which are hereinafter also referred to as reference data 110. Ground-truth data 110 refers to data that enables a secure or at least reliable classification of objects in the vehicle's environment. Ground-truth data 110 can, for example, be available as ground-truth LiDAR data, which captures particularly accurate depth information (distance of an object in the vehicle's environment), particularly accurate three-dimensional object geometry, and / or metric (consistent with real Euclidean distances) three-dimensional object geometry. Additionally or alternatively, objects can be automatically assigned (labeled) to an object class using automated processes based on ground-truth data 110.

[0028] The labeled data 200 includes at least parts of the sensor data 100 with associated dynamic and static objects in the vehicle's environment. Each object can be assigned at least one object class.

[0029] In an autolabeling step, 10 objects are extracted from the sensor data (100) and assigned object annotations that describe the type (or class) of an object and / or certain properties characteristic of the object. For autolabeling step 10, methods known from the prior art, such as multi-hypothesis methods, multi-object tracking methods, and / or fusion methods, can be used. Such methods are described, for example, in the publication by S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche, and J. Dickmann, "BAAS: Bayesian Tracking and Fusion Assisted Object Annotation of Radar Sensor Data for Artificial Intelligence Application," 2020 IEEE Radar Conference (RadarConf20), Florence, Italy, 2020, pp. 1–6, doi: 10.1109 / RadarConf2043947.2020.9266698.Unlike in a driver assistance system, the object classification and / or object annotation methods used in step 10 of the autolabeling process do not need to be real-time capable. Therefore, more computationally intensive methods can be used that are unsuitable for driver assistance systems. This allows for comparatively better accuracy, for example, improved specificity and / or sensitivity in determining an object class.

[0030] In a subsequent supervision step 20, the automatically generated object classes and / or object annotations are checked for correctness. This check can be performed manually, in which a human tester or labeler verifies the plausibility of the assigned object classes and / or object annotations. A manually performed supervision step 20 is described below using the following examples: Fig. 3 described in more detail.

[0031] Supervision step 20 can also be performed at least partially automatically by comparing the assigned object classes and / or object annotations with machine-generated reference data 110 (without the involvement of a human tester or labeler). An automated supervision step 20 is described below using the following examples: Fig. 2 described in more detail.

[0032] In a subsequent decision step 30, it is checked whether the supervision step 20 has shown sufficient agreement between the results of the autolabeling step 10 and the specifications of the human tester and / or the reference data 110. If so, along a "yes" branch 31, the object classes and / or object annotations determined in the autolabeling step 10 are confirmed as successfully annotated labeled data 200 and made available.

[0033] If supervision step 20 reveals insufficient agreement between the results of autolabeling step 10 and the specifications of the human tester and / or the reference data 110, the negative result is passed along a "no" branch 32 to optimization step 40. In optimization step 40, the type of error is first determined more precisely. For example, it is determined whether a false positive or a false negative assignment has been identified. A false positive assignment is defined here as the assignment of an object property that does not actually exist (confirmed by the assignment of the human tester and / or the reference data 110). A false negative assignment is defined here as the missing assignment of an object property that does actually exist (confirmed by the assignment of the human tester and / or the reference data 110).Similarly, errors are identified that are based on a false association or on the assignment of inaccurate values.

[0034] By determining the precise nature of an incorrect object classification and / or object annotation, errors can be identified and corrected at the appropriate point in autolabeling step 10. For example, in the case of false positive or false negative assignments, detection and clustering methods must be optimized. In the case of an incorrect association hypothesis, data association must be optimized. If inaccurate values ​​are assigned, filtering methods must be reviewed and optimized. When using neural networks in autolabeling step 10, "optimization" refers to retraining such neural networks. When using model-based tracking and fusion methods, "optimization" refers to adjusting parameters.

[0035] After executing optimization step 40, autolabeling step 10 is repeated with the correspondingly optimized (adjusted) parameters. This allows autolabeling step 10 to be optimized step by step and in parallel with the acquisition of labeled data 200. The procedure takes into account that errors may initially occur when calculating suggestions for the labeled data 200 (for example, in suggested object classifications). The suggestions for the labeled data 200 are improved by a feedback process known as reinforcement learning. In particular, the suggestions are improved by a process known as reinforcement learning from human feedback, which incorporates a manual evaluation of the suggestions for the labeled data 200 and generates a learning error criterion that is reduced during optimization.

[0036] Fig. Figure 2 shows a schematic flow chart for an autolabeling procedure in an embodiment in which ground truth data (also referred to as reference data 110) are used for the evaluation of an autolabeling step 10.

[0037] The method utilizes sensor data 100 in various specific forms (modalities). By way of example, sensor data 100 are presented as camera data 101 from a vehicle surround-view camera, as radar data 102 from a vehicle surround-view radar, and as LiDAR data 103 from a vehicle surround-view LiDAR. Camera data 101, radar data 102, and LiDAR data 103 capture objects in the vehicle's surroundings with spatial resolution (at least two-dimensional, usually also three-dimensional) relative to a sensor coordinate system and with temporal resolution. They are therefore, in principle, suitable for capturing trajectories (spatial-time curves) that describe the movement of objects in the vehicle's surroundings.

[0038] The various sensor coordinate systems (for example, those for the vehicle environment camera, the vehicle environment radar and the vehicle environment LiDAR) can be aligned with each other and / or with a common vehicle coordinate system using calibration procedures so that data from different sensors can be fused.

[0039] The aforementioned sensors (vehicle surround-view camera, vehicle surround-view radar, and vehicle surround-view LiDAR) typically have limitations, for example, in accuracy, reproducibility, metric imaging, and / or signal-to-noise ratio. These limitations make it difficult to reliably detect objects in the vehicle's surroundings and to assign object properties to them.

[0040] In the embodiment described here, 100 high-quality ground-truth LiDAR data points are provided as additional sensor data. Such ground-truth LiDAR data points exhibit, for example, particularly high three-dimensional spatial resolution, a particularly good signal-to-noise ratio, particularly low non-metric distortions, and / or particularly high reproducibility. This particularly high signal quality enables the detection of objects in the vehicle's environment and the assignment of object properties by automated processes with particularly high reliability.

[0041] The sensor data 100 are evaluated in a modality-specific vehicle labeling step 10. In this step, the camera data 101 are evaluated in a camera data detection step 11, the radar data 102 in a radar data detection step 12, and the LiDAR data 103 in a LiDAR data detection step 13. In each of the detection steps 11 to 13, the modality-specific recognizable objects in the vehicle's environment are detected. Static (unmoving with respect to the world coordinate system) and / or dynamic (moving with respect to the world coordinate system) objects can be detected.

[0042] In parallel to the modality-specific detection steps 11 to 13, reference data 110 (in this case, the ground-truth LiDAR data 110) are evaluated in a reference data processing step 14. Automatic methods for identifying, classifying, and annotating objects in the vehicle's environment using such reference data 110 are known from the prior art, for example, from the publication by S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche, and J. Dickmann, "Extended Object Tracking assisted Adaptive Clustering for Radar in Autonomous Driving Applications," 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany, 2019, pp. 1–7, doi: 10.1109 / SDF.2019.8916658.

[0043] The objects detected with spatial and temporal resolution in the modality-specific detection steps 11 to 13 are fused in a tracking and fusion step 50. In this step, a global hypothesis and trajectory are determined for each detected object. It will prove advantageous to use methods for this purpose that go beyond those used in driver assistance systems for tracking and fusion of objects, since the proposed tracking and fusion step 50 is not subject to the real-time requirements and the usual limitations regarding the available computing power of such driver assistance systems. Methods for determining a global hypothesis and trajectory for detected objects are known from the prior art, for example, from the publication by S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche and J.Dickmann, “Extended Object Tracking assisted Adaptive Clustering for Radar in Autonomous Driving Applications,” 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany, 2019, pp. 1-7, doi: 10.1109 / SDF.2019.8916658 and from the publication S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche and J. Dickmann, "BAAS: Bayesian Tracking and Fusion Assisted Object Annotation of Radar Sensor Data for Artificial Intelligence Application," 2020 IEEE Radar Conference (RadarConf20), Florence, Italy, 2020, pp. 1-6, doi: 10.1109 / RadarConf2043947.2020.9266698.

[0044] As a result of tracking and fusion step 50, cross-modality object candidates 104 and their trajectories are identified.

[0045] The ground truth data 110, pre-processed in reference data processing step 14, are transformed into ground truth objects 111 in reference data stabilization step 60. Trajectories are optionally assigned to the ground truth objects 111. Methods for identifying, annotating, and classifying ground truth objects are known from the prior art, for example, from the publication by S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche, and J. Dickmann, “Extended Object Tracking assisted Adaptive Clustering for Radar in Autonomous Driving Applications,” 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany, 2019, pp. 1–7, doi: 10.1109 / SDF.2019.8916658, and from the publication by N. Scheiner et al., “Automated Ground Truth Estimation for Automotive Radar Tracking Applications With Portable GNSS and IMU Devices,” 2019 20th International Radar Symposium (IRS), Ulm, Germany, 2019, pp. 1–10, doi: 10.23919 / IRS.2019.8768169.

[0046] The processing of the modality-specific data 101 to 103 in the tracking and fusion step 50 is compared with the processing of the reference data 110 in the reference data processing step 14 by a cross-calibration step 70, for example by comparing the respective sensor-specific coordinate systems with a common vehicle coordinate system.

[0047] The cross-modality object candidates 104 are compared with the ground-truth objects 111 in a trajectory unification step 80. Specifically, the trajectories assigned to the ground-truth objects 111 are compared with the trajectories assigned to the cross-modality object candidates 104. This establishes a two-stage control and feedback procedure that eliminates or reduces the effort of manual labeling, as examining the objects at different times is no longer necessary, and the labeling check in the subsequent supervision step 20 can be limited to comparing trajectories (which comprise a set of spatial-time coordinates).In particular, by comparing the trajectories of the Ground-Truth objects 111 with the trajectories of the cross-modality object candidates 104, critical object candidates 104 can be preselected from among the proposed labelings.

[0048] The tracking and fusion step 50, the reference data stabilization step 60, the trajectory unification step 80 and the optimization step 40 access vehicle state parameters 300, such as geoposition, direction of movement, speed, timestamp and / or calibration data.

[0049] Therefore, in the subsequent supervision step 20, only a reduced review by a human tester is required. The further course of the procedure proceeds as described below. Fig. 1 already described in decision step 30 and, depending on the evaluation of the unified trajectories in supervision step 20, in case of success by providing the labeled data 200 including the assigned quality attributes (which are obtained from supervision step 20 and / or from the trajectory unification step 80) or in case of failure by executing optimization step 40. Optimization step 40 adapts tracking and fusion step 50, for example by changing parameters, so that a better match is subsequently achieved between the trajectories determined in reference data stabilization step 60 for the ground truth objects 111 and the trajectories determined in tracking and fusion step 50 for the cross-modality object candidates 104.

[0050] In addition to adjusting tracking and fusion step 50, some or all of the modality-specific detection steps 11 to 13 can optionally be adjusted. For this purpose, feedback connections 51 are provided from tracking and fusion step 50 to the modality-specific detection steps 11 to 13.

[0051] The invention further recognizes that a reference sensor (in this case, a ground-truth LiDAR) providing reference data 110 (in this case, ground-truth LiDAR data 110) cannot be assumed in every application. For example, sensor data 100 provided by vehicles during operation is typically provided without corresponding reference data 110. This results in the need for an embodiment of the proposed method that does not require reference data 110. Fig. Figure 3 shows a schematic flowchart for such an embodiment analogous to the one in Fig. 2 shown flowchart, but without including the reference data 110 and its processing in a reference data stabilization step 60 to obtain ground truth objects 111.

[0052] With the modality-specific detection steps 11, 12, 13, the modality-specific sensor data 100 (camera data 101, radar data 102, and LiDAR data 110) are first evaluated independently of one another (i.e., modality-specifically) and then combined in a tracking and fusion step 50. This results in the same results as in the procedure according to Fig. 2 cross-modality object candidates 104 identified.

[0053] However, the omission of reference data 110 makes it impossible to preselect trajectories that would be used in the procedure according to Fig.2. This was possible by comparing the trajectories with those assigned to the Ground-Truth objects 111. Therefore, the trajectories must be manually corrected in supervision step 20.

[0054] However, even in this embodiment, the feedback provided by manual labeling can be used to adjust both the tracking and fusion step 50 and (via the feedback connections 51) the modality-specific detection steps 11 to 13. Such an adjustment can be based on positive (confirming) feedback if the manual labeling in supervision step 20 confirms that cross-modality object candidates 104 proposed by tracking and fusion step 50 are correctly identified. Such an adjustment can also be based on negative (correcting) feedback if the manual labeling in supervision step 20 rejects cross-modality object candidates 104 proposed by tracking and fusion step 50 or corrects their characteristics (for example, in an assigned object class, an annotation, or another assigned object property). Reference symbol list 10 Auto Labeling Steps 11 Camera data detection step, detection step 12 Radar data detection step, detection step 13 LiDAR data detection step, detection step 14 Reference Data Processing Step 20th Supervision Step 30 Decision Step 31 Yes branch 32 No branch 40 Optimization Step 50 Tracking and Fusion Step 51 Feedback connection 60 Reference Data Stabilization Step 70 Cross-calibration step 80 Trajectory Unification Step 100 sensor data 101 camera data, modality-specific data 102 radar data, modality-specific data 103 LiDAR data sets, modality-specific data 104 cross-modality object candidates 110 Ground-Truth LiDAR data sets, reference data, Ground-Truth data 111 Ground-Truth Objects 200 labeled data 300 vehicle condition parameters QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0125004 A1

[0006] Cited non-patent literature

[0000] S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche and J. Dickmann, „BAAS: Bayesian Tracking and Fusion Assisted Object Annotation of Radar Sensor Data for Artificial Intelligence Application,“ 2020 IEEE Radar Conference (RadarConf20), Florence, Italy, 2020, pp. 1-6, doi: 10.1109 / RadarConf2043947.2020.9266698 [0029, 0043] S. Haag, B. Duraisamy, F. Govaers, W. Koch, M. Fritzsche and J. Dickmann, „Extended Object Tracking assisted Adaptive Clustering for Radar in Autonomous Driving Applications,“ 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany, 2019, pp. 1-7, doi: 10.1109 / SDF.2019.8916658 [0042, 0043, 0045] N. Scheiner et al., „Automated Ground Truth Estimation for Automotive Radar Tracking Applications With Portable GNSS And IMU Devices,“ 2019 20th International Radar Symposium (IRS), Ulm, Germany, 2019, pp. 1-10, doi: 10.23919 / IRS.2019.8768169

[0045]

Claims

[1] Computer-implemented method for assigning at least one annotation to modality-specific data (101, 102, 103) provided by sensors of a vehicle set up to detect a vehicle environment, characterized by , that - from the modality-specific data (101, 102, 103) in a respective assigned modality-specific detection step (11, 12, 13) modality-specific raw objects with each assigned trajectory are determined, - the identified raw objects are fused in a tracking and fusion step (50) to cross-modality object candidates (104) each with an assigned trajectory and at least one assigned annotation, - which are checked in a supervision step (20) for their conformity with a manually and / or automatically determined reference object assignment, wherein - correctly identified object candidates (104) are provided as labeled data (200) and - for incorrectly determined object candidates (104) in an optimization step (40) the tracking and fusion step (50) is adjusted in such a way that an improved match of the object candidates (104) determined by the adjusted tracking and fusion step (50) with the reference object assignment is achieved. [2] Method according to claim 1, characterized by , that for incorrectly determined cross-modality object candidates (104) in an optimization step (40) in addition to the adjustment of the tracking and fusion step (50) at least one of the modality-specific detection steps (11, 12, 13) is adjusted in such a way that a better match of the object candidates (104) determined by the adjusted tracking and fusion step (50) with the manually determined object assignment is achieved. [3] Method according to any one of the preceding claims, characterized by , that - Reference data (110) are provided by a reference sensor of the vehicle, which is set up to reliably identify objects in the vehicle environment and to assign annotations to such objects, - from the reference data (110) in a reference data processing step (14) object candidates are determined, each with an assigned trajectory and at least one assigned annotation, - Ground-Truth objects (111) are determined from the identified object candidates in a reference data stabilization step (60), - the cross-modality object candidates (104) are compared with the ground truth objects (111) based on their respective assigned trajectories in a trajectory unification step (80), - object candidates (104) matching the Ground-Truth objects (111) are assigned to the labeled data (200) and - only those cross-modality object candidates (104) that do not match ground-truth objects (111) are passed to the supervision step (20). [4] Method according to claim 3 characterized by , that the reference data (110) include Ground-Truth LiDAR data (110). [5] Method according to claim 3 or 4, characterized by , that the reference data (110) describe the geoposition and / or geoposition change and / or pose of the vehicle using differential geoposition and motion data (dGPS data) and / or absolute geoposition and motion data acquired by a Global Navigation Satellite System (GNSS) and / or position and motion data acquired by an Inertial Measurement Unit (IMU). [6] Method according to any one of the preceding claims, characterized by , that the tracking and fusion step (50) and / or at least one detection step (11, 12, 13) is performed with a machine classifier which is adapted in the optimization step (40) via a reinforcement learning procedure. [7] Method according to claim 6, characterized by that the machine classifier includes a neural network, a clustering method, an association method and / or a filter. [8] Method according to claim 6 or 7, characterized by , that the optimization step (40) includes a process of reinforcement learning from human feedback. [9] Method according to any one of the preceding claims, characterized by, that at least one modality-specific detection step (11, 12, 13) includes a multi-hypothesis procedure and / or a multi-object tracking and fusion procedure. [10] Driver assistance system for a vehicle comprising at least one sensor equipped to detect a vehicle environment using modality-specific data (101, 102, 130) and a machine classifier equipped to identify and / or classify objects in the vehicle environment, characterized by , that the machine classifier is trained with modality-specific data (101, 102, 103) annotated according to a method of the preceding claims.

Images