Generation of training data, generation of a data processing model, and vehicle operation control
Patent Information
- Application Number
- EP2024700257
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-01-24
- Filing Date
- 2024-01-09
- Publication Date
- 2025-12-03
AI Technical Summary
Existing vehicle driver assistance systems require extensive retraining when switching from high-resolution to lower-resolution sensors, which is costly and inefficient, as they lack sufficient data for adapting machine learning models to process data from lower-resolution sensors effectively.
Generating synthetic training data from high-resolution sensor data to create a data processing model that can handle lower-resolution sensor data using deep learning techniques, such as PointNet or Graph Neural Networks, allowing for cost-effective adaptation and reliable vehicle operation.
Enables the creation of a data processing model that can operate reliably with lower-resolution sensor data, reducing the need for extensive retraining and lowering costs by using synthetic training data, thus enhancing the efficiency of vehicle driver assistance systems.
Smart Images
Figure EP2024050369_02082024_PF_FP
Abstract
Description
[0001] Creation of training data, creation of a data processing model and control of vehicle operation
[0002] The invention relates to a method for creating training data according to claim 1. Furthermore, the invention relates to a method for creating a data processing model and a method for controlling the operation of a vehicle.
[0003] State of the art
[0004] DE 102011 085 976 A1 describes a device for operating a vehicle which sends control signals to control units depending on sensor signals from several vehicle sensors for detecting the vehicle's surroundings.
[0005] To make driver assistance systems more cost-effective, high-resolution vehicle sensors can be replaced with cheaper, lower-resolution vehicle sensors. If the assistance system software is based on a machine learning method (e.g., deep neural networks), no fundamental changes to the algorithms for replacing the vehicle sensors are required to switch to the lower-resolution sensors. However, it is crucial that sufficient data is available for training and testing the existing model for the high-resolution sensors in order to retrain it and adapt it to processing data from low-resolution vehicle sensors.
[0006] Disclosure of the invention
[0007] According to the present invention, a method for creating training data is proposed, having the features of claim 1. This allows the training data for the data processing model to be provided more cost-effectively. The training data can be synthetically generated from existing high-resolution, additional first sensor data. This allows the data processing model to be trained more cost-effectively to process low-resolution sensor data.
[0008] The vehicle may be a motor-driven vehicle, preferably a motor vehicle or two-wheeler. The vehicle may be an assistance-assisted, semi-autonomous, or autonomous vehicle. The control of the vehicle's operation may include a driver assistance system. The operation of the vehicle, in particular of the driver assistance system, may depend on input data formed from the sensor data, which is transferred to the data processing model. The data processing model can calculate output data from this data, on which the operation of the vehicle, in particular of the driver assistance system, depends.
[0009] The sensor data from the vehicle sensor can be prepared, i.e. further processed, to generate measurement data.
[0010] The ambient scene can be a detectable ambient situation of the respective sensor. The ambient scene can be an ambient situation of an area surrounding a vehicle assigned to the sensor at a point in time or over a period of time. The first and second measurement data can each indicate the same ambient scene from a perspective perspective. The only difference between the first and second measurement data compared to each other can be the resolution of the measurement data.
[0011] The at least one sensor that provides the first measurement data and / or the at least one sensor that provides the second measurement data may be a vehicle sensor.
[0012] The first measurement data can be provided by one or more sensors of the sensor class. The additional first measurement data can capture additional environmental scenes that differ from the environmental scenes of the first measurement data. The second measurement data can be provided by one or more sensors of the sensor class.
[0013] A sensor class, also referred to as a sensor modality or sensor type, includes sensors with the same measurement principle. Radar sensors, for example, are assigned to a different sensor class than lidar sensors or cameras.
[0014] The data conversion model is preferably a computer-implemented processing algorithm. The data conversion model can be trained using deep learning. The data conversion model can include PointNet, PointNet++, graph neural networks, continuous convolutions, kernel-point convolutions, or other neural networks.
[0015] The method for creating training data and / or the method for creating a data processing model is preferably a computer-implemented method.
[0016] In a preferred embodiment of the invention, it is advantageous if the additional first measurement data have a similar or identical resolution to the first measurement data. The additional first measurement data can originate from the at least one sensor that acquires the first measurement data or from another sensor of the sensor class. The additional sensor can be a vehicle sensor.
[0017] A preferred embodiment of the invention is advantageous in which the first and second measurement data are present as point clouds describing the respective environmental scenes.
[0018] The point clouds of the second measurement data may have a smaller number of points than the point clouds of the first measurement data.
[0019] In a preferred embodiment of the invention, it is advantageous if the measurements of the first and second sensors of the same environmental scene are linked to each other as associated first and second measurement data. The measurements can be performed during at least one test run with a vehicle equipped with the first and second sensors, and the first and second measurement data can be processed from these measurements.
[0020] In a preferred embodiment of the invention, the data conversion model is trained as target data by applying at least one loss function to reduce deviations between the output data of the data conversion model calculated from the first measurement data during training and the second measurement data linked to the first measurement data. This allows for unsupervised learning of the data conversion model. The associated second measurement data can serve as target data and a benchmark for the calculation accuracy and abstraction performance of the data conversion model.
[0021] In a specific embodiment of the invention, it is advantageous if the output data and / or the sensor data have a similar or identical resolution to the second measurement data. This allows the vehicle to operate reliably even with sensor data of lower resolution compared to the resolution used for the first measurement data.
[0022] In a preferred embodiment of the invention, the sensor class includes radar sensors, the vehicle sensor is a radar sensor, and the measurement data is radar measurement data. The vehicle sensor can also be a camera, an ultrasonic sensor, or a microphone. The sensor class can include lidar sensors, the vehicle sensor can be a lidar sensor, and the measurement data can be lidar measurement data.
[0023] According to the present invention, a method is further proposed for creating a data processing model for controlling the operation of a vehicle based on sensor data from at least one vehicle sensor of the vehicle as input data of the data processing model, which is trained at least with the output data generated by a method having at least one of the previously described features as training data. This allows the data processing model to be trained with training data that is generated more simply and quickly.
[0024] The data processing model can be trained using deep learning. The data processing model can include PointNet, PointNet++, Graph Neural Network, Continuous Convolutions, Kernel-Point Convolutions, or other neural networks.
[0025] In a specific embodiment of the invention, it is advantageous if the data processing model is trained with additional second measurement data from at least one sensor of the sensor class as training data, in addition to the output data. The additional second measurement data can be provided by the at least one sensor that also provided the second measurement data or by another sensor of the sensor class.
[0026] According to the present invention, a method for controlling the operation of a vehicle is further proposed, using a data processing model trained according to a method having at least one of the previously described features, depending on sensor data from at least one vehicle sensor of the vehicle as input data of the data processing model. Depending on the sensor data, the operation of the vehicle can include the operation of a driver assistance system, a semi-autonomous driving system, and / or an autonomous driving system of the vehicle via the calculation with the data processing model.
[0027] Furthermore, a computer program is proposed which has machine-readable instructions which can be executed on at least one computer and upon execution of which a method with at least one of the features specified above is carried out.
[0028] Furthermore, a storage unit which is machine-readable and accessible by at least one computer and on which the said computer program is stored is proposed.
[0029] Further advantages and advantageous embodiments of the invention will become apparent from the description of the figures and the illustrations.
[0030] The invention is described in detail below with reference to the figures. They show in detail:
[0031] Figure 1: A method for creating training data, a method for creating a data processing model and a method for controlling operation of a vehicle, each in a specific embodiment of the invention.
[0032] Figure 2: A learning process of the data conversion model in a special embodiment of the invention.
[0033] Figure 3: A calculation process of output data with the data conversion model in a specific embodiment of the invention.
[0034] Figure 1 shows a method for creating training data in a specific embodiment of the invention. The method 10 for creating training data for a data processing model 12 can be performed before the data processing model 12 is applied in a vehicle 14. Preferably, the method 10 is used to create a new data processing model 12 or to adapt an existing data processing model 12.
[0035] The data processing model 12 serves to control operation of the vehicle 14 depending on sensor data 16 from at least one vehicle sensor 18 of the vehicle 14. From the sensor data 16, input data 20 is formed for the data processing model 12, which calculates output data 22 therefrom, which influence the operation of the vehicle 14.
[0036] The data processing model 12 is based in particular on deep learning. The training data 24 for the data processing model 12 are created through the following steps. First, first measurement data 30, each capturing environmental scenes 28, from at least one sensor 32 of a sensor class 34, to which the vehicle sensor 18 is also assigned, is provided 26. The first measurement data 30 can be provided by one or more sensors of the sensor class 34 and are preferably present as point clouds 35. If the sensor 32 is, for example, a radar sensor 36, then the sensor class 34 exclusively comprises sensors based on the same measurement principle, in this case radar sensors 36. The first measurement data 30 can, for example, depict environmental scenes 28 of vehicle environments and be present as radar measurement data.Furthermore, second measurement data 40 from at least one sensor 42 of sensor class 34 are provided 38, said second measurement data having a lower resolution than the first measurement data 30, being temporally related to the first measurement data 30 and capturing the same environmental scenes 28 as the first measurement data 30. If the sensor 32 is a radar sensor 36, then the sensor 42 is also a radar sensor 36, since both sensors 32, 42 are assigned to the same sensor class 34. The second measurement data 40 can be provided by one or more sensors of sensor class 34 and are preferably present as lower-resolution point clouds 35. The first measurement data 30 and second measurement data 40 each indicate the same environmental scene 28 in perspective. The only difference between the first and second measurement data 30, 40 compared to one another can be the resolution, which is higher for the first measurement data 30 than for the second measurement data 40.
[0037] For training 44 of a data conversion model 46, the first measurement data 30 is used as input data 48 and the second measurement data 40 is used as target data 50. The measurements of the first and second sensors 32, 42 of the respective identical environmental scene 28 are linked to one another as associated first and second measurement data 52. The data conversion model 46 is trained as target data 50 by applying at least one loss function 54 to reduce deviations between the output data 60 of the data processing model 12 calculated on the basis of the first measurement data 30 during training 44 and the second measurement data 40 of the associated first and second measurement data 52 assigned to the first measurement data 30. The data conversion model 46 is trained to calculate from measurement data having a similar or equal resolution to the first measurement data 30, those with lower resolution that have a similar or equal resolution to the second measurement data 40.
[0038] By providing 56 further first measurement data 58 of at least one sensor 32' of the sensor class 34, which supplement the first measurement data 30 and have the same resolution as the first measurement data 30 and record further environmental scenes 28', a calculation 59 of output data 62 takes place with the data conversion model 46 and the further first measurement data 58 as input data 63. The output data 62 have a similar or the same resolution as the second measurement data 40.
[0039] Finally, the output data 62 is provided 64 as training data 24 for the data processing model 12, which can be trained with the output data 62 in order to enable operation of the vehicle 14 as a trained data processing model 12 depending on the sensor data 16. Furthermore, Figure 1 shows a method 68 for creating a data processing model 12 in a specific embodiment of the invention, which is preferably carried out chronologically after the method 10 for creating training data 24, because the training 69 of the data processing model 12 takes place at least with the training data 24 calculated by the data conversion model 46. The data processing model 12 is trained with additional second measurement data 70 from at least one sensor 42' of the sensor class 34 as training data, in addition to the output data 62 of the data conversion model 46.
[0040] Furthermore, Figure 1 illustrates a method 72 for controlling 73 the operation of a vehicle 14 in a specific embodiment of the invention. The sensor data 16 are comparable or identical in resolution to the second measurement data 40. As a result, the calculation result can be provided as output data 22 with inference from the data processing model 12 in a more accurate and reliable manner.
[0041] Figure 2 shows a training process of the data conversion model in a specific embodiment of the invention. During the training process of the data conversion model 46, the output data 60 calculated from the first measurement data 30 are iteratively compared with the second measurement data 40 as target data 50, and the deviation is backpropagated using the loss function 54.
[0042] Figure 3 shows a calculation process for output data using the data conversion model in a specific embodiment of the invention. The calculation process is part of the inference of the data conversion model 46. During the calculation process, the additional first measurement data 58 are converted into the output data 62, which has a lower resolution than the additional first measurement data 58.
Claims
Patent claims 1. A method (10) for creating training data (24) for a data processing model (12) for controlling the operation of a vehicle (14) depending on sensor data (16) of at least one vehicle sensor (18) of the vehicle (14), comprising the following steps: Providing (26) first measurement data (30) capturing respective environmental scenes (28) from at least one sensor (32) of a sensor class (34) to which the vehicle sensor (18) is also assigned, Providing (38) second measurement data (40) of at least one sensor (42) of the sensor class (34) which have a lower resolution than the first measurement data (30), are temporally related to the first measurement data (30) and record the same environmental scenes (28) as the first measurement data (30), Teaching (44) a data conversion model (46) with the first measurement data (30) as input data (48) and the second measurement data (40) as target data (50), providing (56) further first measurement data (58) supplementing the first measurement data (30) of at least one sensor (32') of the sensor class (34), Calculation (59) of output data (62) with the data conversion model (46) and the further first measurement data (58) as input data (63) and Providing (64) the output data (62) as training data (24) for the data processing model (12).
2. Method (10) for creating training data (24) according to claim 1, characterized in that the further first measurement data (58) have a similar or identical resolution to the first measurement data (30).
3. Method (10) for creating training data (24) according to claim 1 or 2, characterized in that the first and second measurement data (30, 40) are present as point clouds (35) describing the respective environmental scenes (28).
4. Method (10) for creating training data (24) according to one of the preceding claims, characterized in that the measurements of the first and second sensors (32, 42) of the respective same environmental scene (28) are linked to one another as associated first and second measurement data (52).
5. Method (10) for creating training data (24) according to claim 4, characterized in that the data conversion model (46) is trained by applying at least one loss function (54) to reduce deviations between the output data (60) of the data conversion model (46) calculated on the basis of the first measurement data (30) during the training (44) and the second measurement data (40) linked to the first measurement data (30) as target data (50).
6. Method (10) for creating training data (24) according to one of the preceding claims, characterized in that the output data (62) and / or the sensor data (16) have a similar or identical resolution to the second measurement data (40).
7. Method (10) for creating training data (24) according to one of the preceding claims, characterized in that the sensor class (34) comprises radar sensors (36), the vehicle sensor (18) is a radar sensor (36) and the first and second measurement data (30, 40) are radar measurement data.
8. Method (68) for creating a data processing model (12) for controlling operation of a vehicle (14) depending on sensor data (16) of at least one vehicle sensor (18) of the vehicle (14) as input data (20) of the data processing model (12), which is trained at least with the output data (62) formed by a method (10) according to one of the preceding claims as training data (24).
9. Method (68) for creating a data processing model (12) according to claim 8, characterized in that the data processing model (12) is trained as training data in addition to the output data (62) with further second measurement data (70) of at least one sensor (42') of the sensor class (34).
10. Method (72) for controlling operation of a vehicle (14) with a data processing model (12) learned according to a method (68) according to one of claims 8 to 9 depending on sensor data (16) of at least one vehicle sensor (18) of the vehicle (14) as input data (20) of the data processing model (12).