Method for sensor data creation and method for vehicle simulation
By calculating the data of virtual objects in the environmental model around the virtual vehicle, the data of the virtual vehicle sensor is used to use the training algorithm to solve the problem of the high cost of creating sensor models in the existing vehicle simulation method. Essence
Patent Information
- Application Number
- CN202410965374.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-18
- Filing Date
- 2024-07-18
- Publication Date
- 2025-01-21
Smart Images
Figure CN119337492A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method according to the invention for generating sensor data. Furthermore, the invention relates to a method for vehicle simulation. Background Art
[0002] Vehicle simulations performed in known vehicle-in-the-loop techniques assume a sensor model for each vehicle sensor of the vehicle, which physically maps the corresponding vehicle sensor in the vehicle model in order to obtain sensor data for the vehicle simulation from the corresponding sensor model. For example, to simulate a radar sensor, an accurate sensor model is assumed, taking into account the physical properties of the radar sensor, in particular the emission, reflection and reception of the radar signal. If a plurality of vehicle sensors are to be considered in the vehicle simulation, an accurate physical sensor model is required for each individual vehicle sensor. Summary of the invention
[0003] According to the invention, a method according to the invention for creating sensor data is proposed. The method has:
[0004] providing first sensor measurement data of at least one first vehicle sensor,
[0005] calculating virtual object data of a virtual vehicle surroundings model, which includes a virtual second vehicle sensor, wherein in the vehicle surroundings model the sensor detection range of the first vehicle sensor overlaps spatially and / or temporally with the sensor detection range of the second vehicle sensor,
[0006] calculating at least second model sensor data of a virtual second vehicle sensor using a trained first sensor data algorithm based on the virtual object data, wherein the first sensor data algorithm is based on a training process using training data that takes into account selected first sensor measurement data of the first vehicle sensor,
[0007] A sensor data set is output, which comprises at least first sensor measurement data and second model sensor data. This allows the costs for creating a sensor model to be reduced. Vehicle simulations can be performed more cost-effectively, more simply and more quickly. The computational costs of vehicle simulations can be reduced and sensor data of vehicle sensors can be provided with the dynamics required in vehicle simulations. The simulated sensor data can preferably be provided in real time.
[0008] The vehicle may be a motor vehicle, in particular a (partially) autonomous motor vehicle or a two-wheeled vehicle. The vehicle may be a movable robot.
[0009] The sensor measurement data may be digital and / or analog sensor signals of corresponding vehicle sensors. Model sensor data may be data that simulate these sensor signals or components of these sensor data.
[0010] The vehicle surroundings model is a virtual representation of a simulated or imaginary vehicle surroundings of a vehicle with at least one virtual object. The virtual object can be an object, such as a traffic sign, roadway unevenness, creatures, plants, buildings or vehicles in the vehicle surroundings.
[0011] Virtual object data may be data that constitute, describe and / or identify virtual objects in a model of the vehicle's surroundings. The virtual object data may be limited to the dimension of the sensor data or may be higher dimensional. For example, the virtual object data may specify spatial information, distance information and / or reflection properties of at least one virtual object. The virtual object data may be implemented in the sensor signal as a salient feature, such as an amplitude, that identifies the virtual object.
[0012] The spatial field of view of a vehicle sensor that can be covered by the corresponding vehicle sensor is understood as a sensor detection area, also called field of view (FoV). The sensor detection area can be an actual sensor detection area or a virtual sensor detection area.
[0013] The vehicle sensor may be a lidar sensor, a radar sensor, a camera, an acceleration sensor or an ultrasonic sensor. The first and second vehicle sensors may be assigned to the same or different sensor modalities. For example, the first vehicle sensor may be a camera and the second vehicle sensor may be a radar sensor. Alternatively, the first and second vehicle sensors may be radar sensors.
[0014] The first and second virtual vehicle sensors may have sensor detection areas that overlap each other at least partially in time and / or space. Thus, a virtual object may be at least partially detected by two virtual vehicle sensors.
[0015] In addition to the second model sensor data, the sensor data set may also have third model sensor data of a virtual third vehicle sensor. Further model sensor data of further virtual vehicle sensors may also be formed and added to the sensor data set. For this purpose, the vehicle surroundings model may also take virtual third or further vehicle sensors into account. The calculation of the third and further model sensor data may be performed similarly to the described calculation of the second model sensor data.
[0016] The training process of the first sensor data algorithm can be based on machine learning. During the training process, the training data can be used for machine learning. Machine learning can include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, transfer learning and / or evolutionary algorithms. The training data includes not only input information but also expected output values.
[0017] In a preferred embodiment of the invention, it is advantageous to calculate the second model sensor data as input data of the first sensor data algorithm from first model sensor data of a virtual first vehicle sensor that is calculated from virtual object data and that maps the first vehicle sensor in a vehicle surroundings model. The first model sensor data can be calculated from the virtual object data by a physical sensor model that is assigned to the first vehicle sensor. The physical sensor model of the first vehicle sensor can be the only physical sensor model that is used to create the first and second model sensor data, preferably all model sensor data.
[0018] In a special embodiment of the invention, it is advantageous to enrich the first sensor measurement data with the first model sensor data, and the sensor data set preferably includes these enriched first sensor measurement data. This makes it possible to make the surrounding information more comprehensive. The sensor data set can only have the first sensor measurement data as sensor measurement data, while the second and third sensor measurement data are not necessary.
[0019] In a preferred embodiment of the invention, it is advantageous to calculate the second model sensor data from the virtual object data directly as input data for the first sensor data algorithm. This makes it possible to avoid the use of a physical sensor model.
[0020] In a special embodiment of the invention, it is advantageous if the selected first sensor measurement data are selectively formed from components of the first sensor measurement data that are associated with at least one object in the sensor detection range of the first vehicle sensor. The selected first sensor measurement data can correspond to components of the first sensor measurement data that identify at least one object or can be generated from these components, in particular by filtering and / or subsequent processing.
[0021] In a preferred embodiment of the invention, it is provided that the training process takes into account the training of the first sensor data algorithm directly with the aid of the training data including the selected first sensor measurement data. The first sensor data algorithm may include a neural network, the parameters of which have been trained according to the training data. These parameters may be trained by deep learning. The first sensor data algorithm may be trained by supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, transfer learning or an evolutionary algorithm.
[0022] In a special embodiment of the invention, it is advantageous that the training process takes into account the indirect training of the first sensor data algorithm with the aid of training data including the selected first sensor measurement data, in that the selected first sensor measurement data form the training data for the upstream sensor data algorithm, which in turn generates further virtual sensor data based on the virtual object data of the virtual vehicle surroundings model during the application of the training process, which further virtual sensor data in turn form the training data for the first sensor data algorithm. The upstream sensor data algorithm can include a neural network, the parameters of which have been trained based on the training data of the upstream sensor data algorithm. These parameters can be trained by deep learning. The upstream sensor data algorithm can be trained by supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, transfer learning or evolutionary algorithms.
[0023] In a preferred embodiment of the present invention, the second model sensor data of the training data has been calculated by a trained upstream sensor data algorithm. The upstream sensor data algorithm may include a neural network, the parameters of which have been trained. These parameters may be trained by deep learning. The upstream sensor data algorithm may be trained by supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, transfer learning or an evolutionary algorithm.
[0024] In a special embodiment of the invention, it is advantageous if, during use in a training process, the input data of the upstream sensor data algorithm include first model sensor data of a virtual first vehicle sensor mapped into a vehicle surroundings model, calculated from virtual object data.
[0025] In a special embodiment of the invention, it is advantageous if the training data of the first sensor data algorithm directly include the virtual object data.
[0026] According to the invention, a method according to the invention for vehicle simulation is also proposed. The method comprises:
[0027] Provides vehicle models for simulating vehicle dynamics (Fahrzeugdynamik),
[0028] sensor data processing, in which a sensor data set created by the method for sensor data creation described above is processed, and
[0029] At least one simulated vehicle actuator assigned to the vehicle model is actuated as a function of the sensor data processing. The vehicle dynamics may also take the vehicle kinematics into account. The vehicle dynamics may be vehicle movements, in particular vehicle speeds, vehicle accelerations in at least one direction. The vehicle simulation may also be an actual vehicle operation, in which the sensor data are enriched with virtual sensor data.
[0030] Further advantages and advantageous embodiments of the invention emerge from the description of the figures and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The present invention is described in detail below with reference to the accompanying drawings.
[0032] Figure 1 : A method for sensor data creation in a special embodiment of the present invention.
[0033] Figure 2 : A method for creating sensor data in another special embodiment of the present invention.
[0034] Figure 3 : A method for creating sensor data in another special embodiment of the present invention.
[0035] Figure 4 : A method for vehicle simulation in a special embodiment of the present invention. DETAILED DESCRIPTION
[0036] Figure 1 A method for generating sensor data in a special embodiment of the invention is shown. The method for generating sensor data 10 comprises providing 12 first sensor measurement data 14 of at least one first vehicle sensor 16 and calculating 17 virtual object data 18 of a virtual vehicle surroundings model 20. The virtual object data 18 describe virtual objects in the virtual vehicle surroundings.
[0037] The virtual vehicle surroundings model 20 includes a virtual second vehicle sensor 22 ′ and a virtual third vehicle sensor 24 ′, wherein in the vehicle surroundings model 20 , the sensor detection range of the first vehicle sensor 16 , the sensor detection range of the second vehicle sensor 22 ′ and the sensor detection range of the third vehicle sensor 24 ′ overlap spatially and / or temporally.
[0038] In addition, through the trained first sensor data algorithm 34, first model sensor data 28 of a virtual first vehicle sensor 16' that is mapped in the vehicle surroundings model 20 is calculated 26, second model sensor data 30 of the virtual second vehicle sensor 22' is calculated 29, and third model sensor data 32 of the virtual third vehicle sensor 24' is calculated 31, respectively, based on the virtual object data 18 that directly serves as input data of the first sensor data algorithm 34.
[0039] The first sensor measurement data 14 of the first vehicle sensor 16 are completed with the first model sensor data 28 by means of enrichment 36. Furthermore, the second sensor measurement data 40 of the second vehicle sensor 22, which actually corresponds to the virtual second vehicle sensor 22′, are completed with the second model sensor data 30 by means of enrichment 38, the third sensor measurement data 44 of the third vehicle sensor 24, which actually corresponds to the virtual third vehicle sensor 24′, are completed by means of enrichment 42, and a sensor data set 46 including the enriched sensor measurement data 48 is created.
[0040] It is also possible to omit second and third sensor measurement data 40 , 44 from sensor data set 46 , so that sensor data set 46 from second and third vehicle sensors 22 , 24 or virtual second and third vehicle sensors 22 ′, 24 ′ only has second and third model sensor data 30 , 32 .
[0041] It is also conceivable that sensor data set 46 associated with first vehicle sensor 16 includes only first sensor measurement data 14 and not first model sensor data 28 .
[0042] The output of sensor data record 46 can take place for a subsequent sensor data processing 50 of a vehicle simulation.
[0043] The first sensor data algorithm 34 is based on a training process 52 using training data that takes into account selected first sensor measurement data 14 ′ of the first vehicle sensor 16 . The trained first sensor data algorithm 34 is trained with the aid of training data, wherein the training data includes first model sensor data 28 ′ of the virtual first vehicle sensor 16 ′ calculated from virtual object data 18 ′ of the vehicle surroundings model 20 , second model sensor data 30 ′ of the virtual second vehicle sensor 22 ′, and third model sensor data 32 ′ of the virtual third vehicle sensor 24 ′. The training process 52 of the first sensor data algorithm 34 ′ is located temporally before the execution of the method for sensor data generation 10 .
[0044] With the aid of the first model sensor data 28' of the virtual first vehicle sensor 16' as input data, the second and third model sensor data 30', 32' of the training data are calculated again by the trained upstream sensor data algorithm 54. The upstream sensor data algorithm 54' is temporally pre-trained with the aid of the training data formed by the selected first sensor measurement data 14' of the first vehicle sensor 16, the second sensor measurement data 40' of the second vehicle sensor 22, the third sensor measurement data 44' of the third vehicle sensor 24 and the virtual object data 18'.
[0045] Thus, the training process 52 of the first sensor data algorithm 34 includes indirectly training the first sensor data algorithm 34' with the aid of training data including the selected first sensor measurement data 14', in that the selected first sensor measurement data 14' form the training data for the upstream sensor data algorithm 54', wherein the upstream sensor data algorithm 54 in turn generates further virtual sensor data, here second and third model sensor data 30', 32', based on the virtual object data 18' of the virtual vehicle surroundings model 20 during the use in the training process 52, which in turn form the training data for the first sensor data algorithm 34'. In this case, during the use in the training process 52, the input data of the upstream sensor data algorithm 54 include the first model sensor data 28' of the virtual first vehicle sensor 16' calculated from the virtual object data 18'.
[0046] Selected first sensor measurement data 14 ′ are preferably limited to identified objects 58 of the vehicle surroundings and are therefore present as labeled data. In particular, selected first sensor measurement data 14 ′ are selectively formed from components of first sensor measurement data 14 that are associated with at least one object in the sensor detection range of first vehicle sensor 16 .
[0047] Figure 2 A method for creating sensor data in another specific embodiment of the present invention is shown. Figure 1 The method in is the same except for the following differences: the first sensor data algorithm 34 calculates the second and third model sensor data 30, 32 with the aid of the virtual object data 18 as input data, while the first model sensor data 28 is calculated directly from the virtual object data 18. Accordingly, the first model sensor data 28' is also excluded from the training data of the first sensor data algorithm 34'.
[0048] The first model sensor data 28 are calculated from the virtual object data 18 . This calculation therefore takes place in parallel with the calculation of the second and third model sensor data 30 , 32 by means of the first sensor data algorithm 34 .
[0049] The first sensor data algorithm 34' has been trained in a training process 52 using training data including virtual object data 18' of the vehicle surroundings model 20, second model sensor data 30' of the virtual second vehicle sensor 22' and third model sensor data 32' of the virtual third vehicle sensor 24'. The training process 52 of the first sensor data algorithm 34' is located temporally before the execution of the method for sensor data generation 10.
[0050] The first model sensor data 28' of the virtual first vehicle sensor 16' is used as input data, and the second and third model sensor data 30', 32' of the training data are calculated by the trained upstream sensor data algorithm 54. In addition, the unlabeled symbols shown are the same as Figure 1 The corresponding in.
[0051] Figure 3 A method for sensor data creation in another specific embodiment of the present invention is shown. The method for sensor data creation 10 comprises: providing 12 first sensor measurement data 14 of at least one first vehicle sensor 16; calculating first model sensor data 28 of a virtual first vehicle sensor 16' from virtual object data 18; and, using the first model sensor data 28 calculated from the virtual object data 18 as input data, calculating 29 second model sensor data 30 of a virtual second vehicle sensor 22' and calculating 31 third model sensor data 32 of a virtual third vehicle sensor 24' by means of a trained first sensor data algorithm 34, respectively.
[0052] By enriching 36 the first sensor measurement data 14 with the first model sensor data 28, enriching 38 the second sensor measurement data 40 with the second model sensor data 30, and enriching 42 the third sensor measurement data 44 with the third model sensor data 32, a sensor data set 46 is created which includes enriched sensor measurement data 48. It is also possible to omit the second and third sensor measurement data 40, 44 from the sensor data set 46, so that the sensor data set 46 from the second and third vehicle sensors 22, 24 only has the second and third model sensor data 30, 32.
[0053] The first sensor data algorithm 34′ is trained by a training process 52, which takes into account the training of the first sensor data algorithm 34′ directly with the aid of training data comprising the selected first sensor measurement data 14′, the second sensor measurement data 40′ of the second vehicle sensor 22 and the third sensor measurement data 44′ of the third vehicle sensor 24. The training process 52 is located temporally before the method for sensor data generation 10. The selected first sensor measurement data 14′ preferably include identified objects 58 of the vehicle surroundings and are therefore present as labeled data.
[0054] Alternatively, as indicated here by the dashed connecting arrow, there can also be an embodiment in which the enriched first sensor measurement data 48.1 are used as input data for the first sensor data algorithm 34 instead of the first model sensor data 28. In this special embodiment, the sensor data set 46 can then only have the first sensor measurement data 14 as sensor measurement data, while the second and third sensor measurement data 40, 44 are not necessary. Therefore, the sensor data set 46 can include the enriched first sensor measurement data 14 and the second and third model sensor data 30, 32.
[0055] Figure 4 A method for vehicle simulation in a special embodiment of the invention is shown. The vehicle simulation 60 is performed in a vehicle-in-the-loop application and includes providing 62 a vehicle model 64 for simulating the vehicle dynamics of the vehicle. The sensor data processing 50 processes the previously acquired, in particular Figure 1 1 and 2. The sensor data set 46 is created by the method for sensor data creation 10 described in . At least one simulated vehicle actuator assigned to the vehicle model 64 is actuated 70 as a function of the sensor data processing 50. The actuation 70 of the vehicle actuator in turn influences the vehicle dynamics, changes of which are taken into account in the vehicle model 64 and in the surroundings model 20.
Claims
1. A method for sensor data creation (10), the method comprising: providing (12) first sensor measurement data (14) of at least one first vehicle sensor (16), Virtual object data (18) of a virtual vehicle surroundings model (20) are calculated (17), the virtual vehicle surroundings model also taking into account a virtual second vehicle sensor (22'), wherein: In the vehicle surroundings model (20), the sensor detection area of the first vehicle sensor (16) overlaps with the sensor detection area of the second vehicle sensor (22') in space and / or in time, At least second model sensor data (30) of the virtual second vehicle sensor (22′) are calculated (29) based on the virtual object data (18) by means of a trained first sensor data algorithm (34), wherein the first sensor data algorithm (34) is based on a training process (52) using the following training data: the training data takes into account selected first sensor measurement data (14′) of the first vehicle sensor (16), A sensor data set (46) is output, the sensor data set (46) comprising at least the first sensor measurement data (14) and the second model sensor data (30).
2. The method for sensor data creation (10) according to claim 1, characterized in that The second model sensor data (30) are calculated as input data for the first sensor data algorithm (34) based on first model sensor data (28) of a virtual first vehicle sensor (16') calculated from the virtual object data (18), the virtual first vehicle sensor mapping the first vehicle sensor (16) in the vehicle surroundings model (20).
3. The method for sensor data creation (10) according to claim 2, characterized in that The first sensor measurement data (14) are enriched (36) with the first model sensor data (28), and the sensor data set (46) includes the enriched first sensor measurement data (48.1).
4. The method for sensor data creation (10) according to claim 1, characterized in that The second model sensor data (30) is calculated from the virtual object data (18) directly as input data to the first sensor data algorithm (34).
5. The method for sensor data creation (10) according to any of the preceding claims, characterized in that The selected first sensor measurement data (14') are selectively formed from components of the first sensor measurement data (14) that are associated with at least one object (58) in a sensor detection range of the first vehicle sensor (16).
6. The method for sensor data creation (10) according to any of the preceding claims, characterized in that The training process (52) takes into account the training of the first sensor data algorithm (34') directly with the aid of training data including the selected first sensor measurement data (14').
7. The method for generating sensor data (10) according to any one of claims 1 to 5, characterized in that The training process takes into account the indirect training of the first sensor data algorithm (34') with the aid of training data including the selected first sensor measurement data (14'), in that the selected first sensor measurement data (14') form training data for an upstream sensor data algorithm (54'), which in turn generates further virtual sensor data (30', 32') based on virtual object data (18') of the virtual vehicle surroundings model (20) during use in the training process (52), which in turn form training data for the first sensor data algorithm (34').
8. The method for sensor data creation (10) according to claim 7, characterized in that In an application in the training process (52), input data of the upstream sensor data algorithm (54) includes first model sensor data (28') calculated from the virtual object data (18') of a virtual first vehicle sensor (16') mapping the first vehicle sensor (16) in the vehicle surroundings model (20).
9. Method for sensor data creation (10) according to claim 7 or 8, characterized in that The training data of the first sensor data algorithm (34') directly includes the virtual object data (18').
10. A method for vehicle simulation (60), the method comprising: providing a vehicle model (64) for simulating vehicle dynamics of the vehicle, Sensor data processing (50), in which a sensor data set (46) created by the method for sensor data creation (10) according to any of the preceding claims is processed, and At least one simulated vehicle actuator assigned to the vehicle model (64) is actuated (70) as a function of the sensor data processing (50).