A method for operating a perception unit for an ADAS and / or an AD of a vehicle, perception unit, and vehicle

Abstract

A method for operating a perception unit 12 for an advanced driver assistance system 16 and / or assisted driving of a vehicle 10, wherein a three-dimensional (3D) representation of an environment 18 of the vehicle 10 is created by the perception unit 12 based on sensor data of a simple environmental sensor 14 (e.g. a camera, radar sensor and / or ultrasonic sensor) by a model, the representation being validated via ground truth data. Further, a generation module generates the ground truth data of the environment 18 in the vehicle 10 in real time from second sensor data of a second (e.g. LIDAR) sensor 22, and provides it so that the perception unit 12 validates the representation using the ground truth data. Models may be based on machine learning, comprising a self-learning algorithm and / or neural network (NN).

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of vehicles, in particular automobiles. More specifically, the present invention relates to a method for operating a perception unit for an advanced driver assistance system (ADAS) and / or assisted driving (AD) of a vehicle, such as a van, car, or truck. Furthermore, the present invention relates to a corresponding perception unit and a vehicle. BACKGROUND INFORMATION

[0002] For advanced driver assistance systems and / or assisted driving an awareness of the environment of the vehicle is necessary. To gain an image or an idea of the environment, a perception unit may be used. This unit could be based on self-learning algorithms for example by using computer vision.

[0003] Learning based models require ground truth (GT) data, which is specifically labeled with the desired output for a given input. The model may learn the underlying distribution of the data by minimizing the loss, for example, the delta between prediction and ground truth. A problem may occur when the ground truth data is not of desired quality. A specific use case may be as follows:

[0004] A perception system or unit in an ADAS / AD aims at generating a three-dimensional geometric environment representation from cost-efficient sensor data such as camera data (only provides a two-dimensional projection), radar and / or ultrasonic data (both with coarse detections). To generate the accurate ground truth labels (the three dimensional geometric environment), data collection vehicles are equipped with LIDAR sensors as an expert sensor. The GT generation process from expert sensor runs offline, for example in an cloud environment..

[0005] In the state of the art, the relevant processors either add more data or work on the model architecture in an attempt to achieve better performance. Furthermore, the state of the art involves automatically building ground truth data from a sensor data from expert sensors such as LIDAR, by automatic labeling. SUMMARY OF THE INVENTION

[0006] It is an object of the present invention to provide a method, a perception unit, and a vehicle with which it is possible to validate a model used by the perception unit for building a three-dimensional map or representation of an environment in a direct manner.

[0007] This object is solved by a method, a perception unit, and a vehicle according to the present invention. Advantageous embodiments are presented in the dependent claims, the description, and the drawings.

[0008] A first step of the present invention relates to a method for operating a perception unit for an advanced driver assistance system (ADAS) and / or assisted driving (AD) of a vehicle, in particular a car, van, or truck. By the method according to the invention a three-dimensional representation of an environment of the vehicle is created or reconstructed by the perception unit on the basis of sensor data of a simple environmental sensor by a model or via a model, wherein the representation is validated via ground truth (GT) data.

[0009] To generate or validate the representation accurately, a further generation module generates the ground truth data of the environment, in particular online, in the vehicle in real time from second sensor data of a second sensor and provides it, in particular, to other downstream components so that the perception unit validates the image or representation using the ground truth data, which is generated in real time in the vehicle.

[0010] In other words, the model predicts a representation from simple sensors. A separate GT data generation pipeline, in particular the generation module, uses an expert sensor as the second sensor to generate a GT representation. The generation module may be a simple data evaluation unit of the second sensor. The GT representation is typically used to train the model. In this invention, the GT representation is generated online in real time to substitute the model and see if the downstream tasks work given expert or "perfect" data. The GT is in particular generated by the generation module, which may be built by a further model using a capable expert sensor (e.g., Lidar) as a real-time process within the vehicle. That allows to use the GT data instead of the potentially faulty model prediction.

[0011] The representation may be a three-dimensional map but could also cover dynamic aspect of the scene. Ground truth data is data that allows the quality of models to be checked. It is therefore known from such data what result it must deliver. Therefore, it could be labeled. The data may be evaluated manually in advance or as shown here via a separate model in real time, which also annotates or labels it. This invention requires that the data-driven model works with simple sensors and is trained by GT data generated from expert sensors.

[0012] In other words, the shown method provides a preprocessing problem, which could execute in real time the creation of ground truth data from sensor data and is integrated in the vehicle in a loop. Generating GT data from an expert sensor is a substitute for what the model typically would do. In this online real time setup, the GT data may directly be fed to downstream components.

[0013] The method provides a validation of data-driven systems using GT data generation in the loop.

[0014] A simple environmental sensor as used by the system is, in particular, a bulk item and / or cost-efficient compared to an expert sensor, which is expensive but may deliver more accurate data.

[0015] An advantage of the method according to the invention is that it is possible even before a model is designed and trained that ground truth data can alre’ady be verified with respect to fulfilling the task. If it turns out that the overall system cannot perform the task even with the labeled ground truth data, then there is no point in training a model that aims at fitting that data. Instead, the overall system design may be changed or the ground truth data generation pipeline may be refined.

[0016] In an advantageous embodiment of the present invention, a camera and / or a radar sensor and / or an ultrasonic sensor are used as the simple environmental sensor.

[0017] In another embodiment of the present invention, LIDAR-like data are generated from the second sensor data by the further model and these are provided to the model.

[0018] In still another embodiment of the present invention, a LIDAR sensor is used as the second sensor, so the further model is using the LIDAR data to generate the ground truth data.

[0019] In yet another embodiment of the present invention, the model and / or the further model are based on machine learning methods and comprise a self-learning algorithm and / or a neural network and, in particular, ground truth data form part of the training data.

[0020] In still another embodiment of the present invention, training of the model is aborted in the event of insufficient validation.

[0021] A second aspect of the present invention relates to a perception unit for an advanced driver assistance system and / or assisted driving of a vehicle comprising a simple environmental sensor, a second sensor and is configured to perform a method according to the first aspect of the present invention.

[0022] Advantages and advantageous embodiments of the first aspect are to be regarded as advantages and advantageous embodiments of the second aspect of the present invention and vice versa.

[0023] A third aspect of the present invention relates to a vehicle configured to perform a method according to the first aspect of the present invention and / or comprising a perception unit according to the second aspect of the present invention.

[0024] Advantages and advantageous embodiments of the first and second aspects of the present invention are to be regarded as advantages and advantageous embodiments of the third aspect of the present invention, and vice versa.

[0025] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figure.

[0027] The drawing shows in:

[0028] Fig. 1 a schematic side view of a vehicle comprising a perception unit and an advanced driver assistance system configured to perform a method for operating the perception unit. DETAILED DESCRIPTION

[0029] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0031] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0032] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0033] Fig. 1 shows a schematic side view of a vehicle 10, which is for example a car. The vehicle 10 comprises a perception unit 12 with a simple environmental sensor 14 and an advanced driver assistance system (ADAS) 16. Via the ADAS 16 it is possible that the vehicle 10 performs autonomous or assisted driving so that the perception unit 12 is for ADAS and / or assisted driving (AD) of the vehicle 10.

[0034] The perception unit 12 as well as the vehicle 10 are able to perform a method for operating the perception unit 12, wherein a three-dimensional representation of an environment 18 of the vehicle 10 is created or reconstructed by the perception unit 12 on the basis of sensor data of the simple environmental sensor 14 by a model which may be deployed on a computational unit 20 of the vehicle.

[0035] The computational unit may contain memory and process units to run an algorithm or a neural network which is underlying the model or a generation module built as a simple data evaluation unit or a further model. The representation is validated via ground truth data in particular in combination with the sensor data, in particular raw sensor data.

[0036] A further model generates the ground truth data of the environment in the vehicle 10 in real time from second sensor data, in particular, raw sensor data of a second sensor 22 and provides it, so that the perception unit 12 validates the representation using the real-time ground truth data. As the, in particular, at least one simple sensor 14 a camera and / or radar sensor and / or an ultrasonic sensor may be used.

[0037] In particular, to generate the ground truth data or provide data besides or aside from the ground truth data for a favorable creation or reconstruction of the three-dimensional representation of the environment 18 LIDAR-like data are generated from the sensor data by the further model and provided to the model.

[0038] An expert sensor such as a LIDAR sensor may be used a the second sensor 22 to further provide ground truth data, wherein the LIDAR sensor may be part of the vehicle 10 or could be part of a separate data collection vehicle.

[0039] Both models may be in particular based on a machine learning method and are therefore data-driven and need to be trained, wherein supervised or unsupervised learning may be used. The machine learning method may be based on a self-learning algorithm and / or a neural network. The generation module is in particular a simple data evaluation unit of the second sensor 22.

[0040] In other words, a solution is presented for learning-based systems with automated ground truth data generation, specifically for simple environmental sensors 14 and an expert sensors. Therefore the generation module or further model provides in particular a LIDAR-like presentation of GT in the loop of the vehicle. In this way, even before a model is designed or trained, the ground truth data may already be verified with respect to fulfilling the task of creating a three-dimensional representation of the environment 18.

[0041] The model and the further model may be integrated in a single model.

[0042] With the shown method, the vehicle, and / or the perception unit 12 first feedback of suitability of the ground truth data is possible. One advantage of the shown method is the early validation of the overall solution. It may reveal which part of the process needs improvement, for example modern architecture, and / or data and thereby guides the development process.

[0043] Furthermore, it guides the ground truth data generation process to reach sufficient quality. In addition, it is suitable for systems with automated ground truth data generation, specifically with expert sensors and allows parallelization of the system development because downstream models may already be implemented based on the ground truth data. Therefore, a solution for validation of data driven system using ground truth data generation in a loop is provided. Reference Signs 10 vehicle 12 perception unit 14 simple environmental sensor 16 advanced driver assistance system 18 environment 20 computational unit 22 second sensor

Claims

1. A method for operating a perception unit (12) for an advanced driver assistance system (16) and / or assisted driving of a vehicle (10), wherein a three-dimensional representation of an environment (18) of the vehicle (10) is created by the perception unit (12) on the basis of sensor data of a simple environmental sensor (14) by a model, wherein the representation is validated by via ground truth data, characterized in thata generation module generates the ground truth data of the environment (18) in the vehicle (10) in real time from second sensor data of a second sensor (22) and provides it, t so that the perception unit (12) validates the representation using the ground truth data.

2. The method according to claim 1, characterized in thata camera and / or a radar sensor and / or an ultrasonic sensor are used as the simple environmental sensor (14).

3. The method according to claim 1 or 2, characterized in thatLIDAR-like data are generated from the second sensor data by a further model and these are provided to the model.

4. The method according to any one of claims 1 to 3, characterized in thata LIDAR sensor is used as the second sensor (22).

5. The method according to any one of the preceding claims, characterized in thatthe model and / or the further model are based on machine learning methods and comprise a self-learning algorithm and / or a neural network and, in particular, ground truth data forming part of the training data.

6. The method according to any one of the preceding claims, characterized in thattraining of the model is aborted in the event of insufficient validation.

7. A perception unit (12) for an advanced driver assistance system (16) and / or assisted driving of a vehicle (10) comprising a simple environmental sensor (14), a second sensor (22), and configured to perform a method according to any one of the preceding claims.

8. A vehicle (10) configured to perform a method according to any one of claims 1 to 6 and / or comprising a perception unit (12) according to claim 7.12

Images