Creating training data, creating data processing models, and controlling vehicle behavior

Abstract

The present invention relates to a method (10) for creating training data (24) for a data processing model (12) for controlling the operation of a vehicle (14). The present invention further relates to a method (68) for creating a data processing model (12) and a method (72) for controlling the operation of a vehicle (14).

Description

[Technical Field]

[0001] The present invention relates to a method for generating training data according to claim 1. Furthermore, the present invention relates to a method for generating a data processing model and a method for controlling the operation of a vehicle. [Background technology]

[0002] DE 10 2011 085 976 A1 describes an apparatus for operating a vehicle, which transmits control signals to a control device depending on sensor signals of a plurality of vehicle sensors for capturing the vehicle's surrounding environment. Summary of the Invention

[0003] To make vehicle driver assistance systems more cost-effective, high-resolution vehicle sensors can be replaced with cheaper vehicle sensors with lower resolution. If the software of the assistance system is based on machine learning methods (e.g., deep neural networks), exchanging a vehicle sensor for a lower-resolution sensor does not require fundamental changes to the algorithm. However, it is essential that there is a sufficient amount of data for training and testing the model, so that the model existing for the high-resolution sensor can be newly trained and adapted to process the data of the low-resolution vehicle sensor.

[0004] According to the invention, a method for generating training data is proposed having the features of claim 1, whereby training data for a data processing model can be provided relatively inexpensively. The training data can be generated synthetically from existing further first sensor data of high resolution. This allows the data processing model to be trained relatively inexpensively to process low-resolution sensor data.

[0005] The vehicle may be a motor-driven vehicle, preferably a car or a motorcycle. The vehicle may be an assisted, partially autonomous, or autonomous vehicle. The control of the vehicle's operation may take into account a driver assistance system. The operation of the vehicle, in particular the driver assistance system, may depend on input data formed from sensor data, which is passed to a data processing model. The data processing model may then calculate output data on which the operation of the vehicle, in particular the driver assistance system, depends.

[0006] The sensor data of the vehicle sensors may optionally be compiled, i.e. further processed, to form measurement data. The surrounding scene may be the surrounding situation that can be captured by each sensor. The surrounding scene may be the surrounding situation of the area around the vehicle that is attributed to the sensor at a certain point in time or over a certain period of time. The first and second measurement data may each present the same surrounding scene in perspective. The only difference in the mutual comparison between the first and second measurement data may be the resolution of the measurement data.

[0007] The at least one sensor providing the first measurement data and / or the at least one sensor providing the second measurement data may be a vehicle sensor. The first measurement data may be provided by one or more sensors of the sensor class. The further first measurement data may capture a further ambient scene that differs from the ambient scene of the first measurement data. The second measurement data may be provided by one or more sensors of the sensor class.

[0008] A sensor class, also known as a sensor modality or sensor type, includes sensors with the same measurement principle, i.e. a radar sensor belongs to a different sensor class than a lidar sensor or a camera.

[0009] The data transformation model is preferably a computer-implemented processing algorithm. The data transformation model may be trained by deep learning. The data transformation model may take into account PointNet, Pointnet++, graph neural networks, continuous convolutions, kernel-point convolutions, or other neural networks.

[0010] The method for generating training data and / or the method for generating a data processing model are preferably computer-implemented methods. In a preferred embodiment of the invention, it is advantageous if the further first measurement data have a similar or the same resolution as the first measurement data. The further first measurement data can originate from at least one sensor capturing the first measurement data or from a further sensor of this sensor class. The further sensor can be a vehicle sensor.

[0011] In a preferred embodiment of the invention, the first and second measurement data are present as point clouds representing the respective surrounding scenes, and the point cloud of the second measurement data may have a smaller number of points than the point cloud of the first measurement data.

[0012] In a preferred embodiment of the invention, it is advantageous if measurements of a first and a second sensor of the same surrounding scene are linked to one another as attributed first and second measurement data, the measurements being carried out during at least one measurement journey by a vehicle having the first and second sensors, from which the first and second measurement data can be compiled.

[0013] In a preferred embodiment of the present invention, the data transformation model is trained to reduce the discrepancy between output data calculated from the first measurement data and second measurement data associated with the first measurement data as target data by applying at least one loss function. This allows unsupervised training of the data transformation model. The imputed second measurement data can be used as target data and as a measure of the calculation accuracy and abstraction performance of the data transformation model.

[0014] In a particular embodiment of the invention, it is advantageous if the output data and / or the sensor data have a similar or the same resolution as the second measurement data, so that the vehicle can be reliably operated using sensor data with a lower resolution compared to the resolution used for the first measurement data.

[0015] In a preferred embodiment of the present invention, it is contemplated that the sensor class includes a radar sensor, the vehicle sensor is a radar sensor, and the measurement data is radar measurement data. The vehicle sensor may be a camera, an ultrasonic sensor, or a microphone. The sensor class may include a lidar sensor, the vehicle sensor may be a lidar sensor, and the measurement data may be lidar measurement data.

[0016] According to the present invention, there is further proposed a method for creating a data processing model for controlling the operation of a vehicle depending on sensor data of at least one vehicle sensor of the vehicle as input data of the data processing model, wherein the data processing model is trained together with output data formed by a method having at least one of the above-mentioned characteristics as training data, whereby the data processing model can be trained using training data that is relatively simple and relatively fast to generate.

[0017] The data processing model may be trained by deep learning, and may take into account PointNet, Pointnet++, graph neural networks, continuous convolutions, kernel-point convolutions, or other neural networks.

[0018] In a particular embodiment of the invention, it is advantageous if the data processing model is trained not only with the output data but also with further second measurement data of at least one sensor of this sensor class as training data, which may be provided by at least one sensor that also provided the second measurement data or by another sensor of this sensor class.

[0019] The present invention further provides a method for controlling the operation of a vehicle with a data processing model trained according to a method having at least one of the above-mentioned features, depending on sensor data of at least one vehicle sensor of the vehicle as input data for the data processing model, The operation of the vehicle depends on the sensor data calculated by the data processing model and may include the operation of a driver assistance system, a partially autonomous driving system and / or an autonomous driving system of the vehicle.

[0020] Furthermore, a computer program is proposed which is machine-readable and executable on at least one computer and which comprises instructions which, when executed, cause a method having at least one of the above mentioned characteristics to proceed.

[0021] Furthermore, a memory unit is proposed that is made machine-readable and accessible by at least one computer and that has stored thereon a computer program as described above.

[0022] Further advantages and advantageous forms of the invention will become apparent from the description and drawings of the figures. The present invention will be described in detail below with reference to the drawings. [Brief explanation of the drawings]

[0023] [Figure 1] 1A-1C illustrate a method for creating training data, a method for creating a data processing model, and a method for controlling the operation of a vehicle, each according to a specific embodiment of the present invention. [Figure 2] FIG. 2 illustrates a process for training a data transformation model in accordance with a particular embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating a process of calculating output data according to a data transformation model in a specific embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] 1 illustrates a method for generating training data in one particular embodiment of the present invention. The method 10 for generating training data for a data processing model 12 may be performed prior to applying the data processing model 12 in a vehicle 14. The method 10 is preferably applied to generate a new data processing model 12 or to adapt an existing data processing model 12.

[0025] The data processing model 12 is used to control the operation of the vehicle 14 depending on sensor data 16 from at least one vehicle sensor 18 of the vehicle 14. From the sensor data 16, input data 20 for the data processing model 12 is formed, from which the data processing model 12 calculates output data 22 that influences the operation of the vehicle 14.

[0026] The data processing model 12 is based, inter alia, on deep learning. Training data 24 for the data processing model 12 is generated by the following steps: First, first measurement data 30 are provided 26 by at least one sensor 32 of a sensor class 34 that also belongs to the vehicle sensors 18, each capturing an ambient scene 28. The first measurement data 30 can be provided by one or more sensors of the sensor class 34 and preferably exists as a point cloud 35. If the sensor 32 is, for example, a radar sensor 36, the sensor class 34 exclusively includes sensors based on the same measurement principle, i.e., radar sensors 36. The first measurement data 30 can, for example, represent the ambient scene 28 around the vehicle and exist as radar measurement data.

[0027] Next, second measurement data 40 are provided 38 by at least one sensor 42 of the sensor class 34, which has a lower resolution than the first measurement data 30, is temporally related to the first measurement data 30, and captures the same surrounding scene 28 as the first measurement data 30. Both sensors 32, 42 belong to the same sensor class 34, so if sensor 32 is a radar sensor 36, sensor 42 is also a radar sensor 36. The second measurement data 40 can be provided by one or more sensors of the sensor class 34 and is preferably present as a point cloud 35 with a lower resolution. The first measurement data 30 and the second measurement data 40 each present the same surrounding scene 28 in perspective. The only difference in the mutual comparison between the first and second measurement data 30, 40 can be the resolution, which is higher for the first measurement data 30 than for the second measurement data 40.

[0028] For training 44 of the data conversion model 46, the first measurement data 30 is used as input data 48, and the second measurement data 40 is used as target data 50. Measurements of the same surrounding scene 28 by the first and second sensors 32, 42, respectively, are linked to each other as attributed first and second measurement data 52. The data conversion model 46 is trained to reduce the discrepancy between output data 60 calculated from the first measurement data 30 by the data processing model 12 during training 44 and the second measurement data 40 attributed to the first measurement data 30 of the attributed first and second measurement data 52 as target data 50 by applying at least one loss function 54. The data conversion model 46 is trained to calculate lower-resolution measurement data having a similar or the same resolution as the second measurement data 40 from measurement data having a similar or the same resolution as the first measurement data 30.

[0029] Upon provision 56 of at least one sensor 32′ of the sensor class 34 with further first measurement data 58, which augments the first measurement data 30 and has the same resolution as the first measurement data 30 and captures a further surrounding scene 28′, output data 62 is calculated 59 using the data conversion model 46 and the further first measurement data 58 as input data 63, whereby the output data 62 has a similar or the same resolution as the second measurement data 40.

[0030] Finally, the output data 62 is provided 64 as training data 24 for the data processing model 12, which may be trained together with the output data 62, thereby enabling the trained data processing model 12 to operate in dependence on the sensor data 16.

[0031] 1 further shows a method 68 for creating the data processing model 12 in a particular embodiment of the invention, which method 68 is preferably executed temporally after the method 10 for creating the training data 24, since the learning 69 of the data processing model 12 is carried out using the training data 24 calculated by at least the data transformation model 46. The data processing model 12 is learned not only with the output data 62 of the data transformation model 46 but also with further second measurement data 70 of at least one sensor 42′ of the sensor class 34 as training data.

[0032] 1 further illustrates a method 72 for controlling 73 the operation of a vehicle 14 in a particular embodiment of the invention. The sensor data 16 are comparable or identical in resolution to the second measurement data 40, so that the calculation results, by inference of the data processing model 12, can be more accurately and more reliably presented as output data 22.

[0033] 2 illustrates the training process of a data transformation model 46 in a particular embodiment of the present invention. During the training process of the data transformation model 46, output data 60 calculated from the first measurement data 30 is iteratively compared with the second measurement data 40 as target data 50, and the differences are backpropagated under the application of a loss function 54.

[0034] 3 illustrates a calculation process of output data by a data conversion model in one particular embodiment of the present invention. This calculation process is a component of the inference of the data conversion model 46. In this calculation process, the additional first measurement data 58 is converted into output data 62 having a lower resolution than the additional first measurement data 58.

Claims

1. A method (10) for generating training data (24) for a data processing model (12) for controlling the operation of a vehicle (14) in dependence on sensor data (16) of at least one vehicle sensor (18) of said vehicle (14), comprising the steps of: providing (26) first measurement data (30) of at least one sensor (32) of a sensor class (34) that also belongs to the vehicle sensor (18), each capturing a surrounding scene (28); providing (38) second measurement data (40) of at least one sensor (42) of the sensor class (34), the second measurement data (40) having a lower resolution than the first measurement data (30), being temporally related to the first measurement data (30) and capturing the same surrounding scene (28) as the first measurement data (30); training (44) a data transformation model (46) with the first measured data (30) as input data (48) and the second measured data (40) as target data (50); providing (56) further first measurement data (58) of at least one sensor (32') of said sensor class (34) that augments said first measurement data (30); calculating (59) output data (62) using the data transformation model (46) and the further first measurement data (58) as input data (63); providing (64) said output data (62) as training data (24) for said data processing model (12).

2. 2. The method (10) for generating training data (24) according to claim 1, wherein the further first measurement data (58) has a similar or the same resolution as the first measurement data (30).

3. 3. A method (10) for generating training data (24) according to claim 1 or 2, characterized in that the first and second measurement data (30, 40) exist as point clouds (35) representing the respective surrounding scenes (28).

4. 4. A method (10) for creating training data (24) according to any one of claims 1 to 3, characterized in that measurements of the first and second sensors (32, 42) of the same surrounding scene (28) are linked to each other as attributed first and second measurement data (52).

5. 5. The method for generating training data according to claim 4, wherein the data transformation model is trained to reduce discrepancies between output data calculated from the first measurement data during the training step by applying at least one loss function, and the second measurement data as target data associated with the first measurement data.

6. 6. The method (10) for generating training data (24) according to any one of claims 1 to 5, characterized in that the output data (62) and / or the sensor data (16) have a similar or the same resolution as the second measurement data (40).

7. 7. The method (10) for creating training data (24) of claim 1, wherein the sensor class (34) includes a radar sensor (36), the vehicle sensor (18) is a radar sensor (36), and the first and second measurement data (30, 40) are radar measurement data.

8. 8. A method (68) for creating a data processing model (12) for controlling the operation of a vehicle (14) depending on sensor data (16) of at least one vehicle sensor (18) of the vehicle (14) as input data (20) of the data processing model (12), wherein the data processing model (12) is trained together with at least the output data (62) formed by the method (10) of any one of claims 1 to 7 as training data (24).

9. 9. A method (68) for creating a data processing model (12) according to claim 8, characterized in that the data processing model (12) is trained not only with the output data (62) but also with further second measurement data (70) of at least one sensor (42') of the sensor class (34) as training data.

10. A method (72) for controlling the operation of a vehicle (14) using a data processing model (12) trained based on the method (68) of claim 8 or 9, depending on sensor data (16) of at least one vehicle sensor (18) of the vehicle (14) as input data (20) of the data processing model (12).

Images