METHOD AND SYSTEM FOR AUGMENTING LIDAR DATA

Description

[0001] The invention relates to a computer-implemented method for generating driving scenarios based on LIDAR raw data, a computer-readable data carrier and a computer system.

[0002] Autonomous driving promises an unprecedented level of comfort and safety in everyday traffic. Despite enormous investments by various companies, existing approaches can only be used under limited conditions or only enable a subset of fully autonomous behavior. One reason for this is the lack of a sufficient number and variety of driving scenarios. Training and testing autonomous driving functions requires large numbers of kilometers to ensure safe operation. For example, it is not possible to statistically prove, based on practical road tests, that an autonomous vehicle is safer than a human driver in terms of fatalities.

[0003] By using simulations, the number of kilometers driven can be significantly increased. However, modeling suitable driving scenarios in a simulation environment is cumbersome, and the replay of recorded sensor data is limited to previously encountered driving scenarios.

[0004] From the generic US 2018 / 0364717 A1, a voxel-based method for estimating a ground plane and for identifying static and dynamic objects is known. The determined information can be provided to a tracker or a planner for operating an autonomous vehicle. US 2019 / 0228571 A1 discloses a method for generating realistic virtual 3D worlds based on geographical map data. The generated models can be used to train autonomous vehicle systems. US 2020 / 160598 A1 discloses a method for synthesizing LIDAR point clouds, which can be used to generate driving simulations. In contrast to the present invention, however, dynamic road users are filtered out of the raw data when creating the point clouds.

[0005] The object of the present invention is therefore to provide improved methods for generating sensor data for driving scenarios; in particular, it would be desirable to be able to easily extend existing driving scenarios with variations.

[0006] This object is achieved by a method for generating a simulation scenario for a land vehicle according to claim 1, a computer-readable data carrier according to claim 11, and a computer system according to claim 12. Advantageous further developments are the subject of the dependent subclaims.

[0007] A static object does not change its position over time, whereas the position of a road user can change dynamically. The term "dynamic road user" preferably also includes temporarily static road users, such as a parked car—that is, a road user that can be moving at a given time but can also remain stationary for a certain period of time.

[0008] A simulation scenario preferably describes a continuous driving maneuver, such as an overtaking maneuver, that takes place in an environment defined by road information and static objects. Depending on the behavior or trajectories of the dynamic road users, this may be a safety-critical driving maneuver, for example, if there is a risk of collision with an oncoming vehicle during the overtaking maneuver.

[0009] The specific region can be a geographical area defined by a range of GPS coordinates. It can also be an area defined by the recorded sensor data, which, for example, includes a sub-area of ​​the environment detected by the environmental sensors.

[0010] Exporting the simulation scenario can involve saving one or more files to a storage medium and / or storing information in a database. The files or information in the database can then be read as often as required, for example, to generate sensor data for a virtual driving test. Thus, the existing driving scenario can be used to test various autonomous driving functions and / or to simulate various environmental sensors. It can also be possible to directly export simulated sensor data for the existing driving scenario.

[0011] The method according to the invention focuses on LIDAR data and integrates the scenario generation step, whereby the simulation scenario is not limited to fixed sensor data, but rather relative coordinates for the objects and road users are available. This allows for an intelligent pre-selection of interesting scenarios, but also the supplementation of existing ones.

[0012] In a preferred embodiment of the invention, the raw data comprises synthetic sensor data generated in a sensor-realistic simulation environment. Sensor data recorded from a real vehicle, synthetic sensor data, or a combination of recorded and synthetic sensor data can be used as input data for the method according to the invention.

[0013] A preferred embodiment of the invention provides an end-to-end pipeline with defined interfaces for all tools and operations. This allows for the exploitation of synergies between the various tools, such as the use of scenario-based tests to enrich the simulation scenarios or the simulated sensor data generated from them.

[0014] In a preferred embodiment, the invention further comprises the step of modifying the simulation scenario, in particular by modifying at least one trajectory and / or adding at least one additional dynamic road user, prior to exporting the simulation scenario. The modifications can be arbitrary or adapted to achieve a desired property of a scenario. This has the advantage that by adding simulation scenarios, in particular for critical situations, sensor data for a multitude of scenarios can be simulated and thus the amount of training data can be increased. This allows a user to train their models with a larger amount of relevant data, leading, for example, to improved perception algorithms.

[0015] In a preferred embodiment, the steps of modifying the simulation scenario and exporting the simulation scenario are repeated, with a different modification being applied each time before exporting the simulation scenario, so that a set of simulation scenarios is compiled. The individual simulation scenarios can have metadata that, for example, indicates how many pedestrians occur in a simulation scenario and / or cross the road. The metadata can be derived, in particular, from the localization and identification of the static objects and / or road users. In addition, information about the road course, such as curve parameters, or an environment can also be added to the simulation scenario as metadata. The final dataset preferably contains both the raw data and the extended synthetic point clouds.In a preferred embodiment, the computer used for scenario generation is connected to a database server and / or comprises a database, wherein existing simulation scenarios are stored in the database so that existing scenarios can be used to supplement the set of simulation scenarios.

[0016] In a preferred embodiment, at least one property of the set of simulation scenarios is determined, and modified simulation scenarios are added to the set of simulation scenarios until the desired property is met. The property can, in particular, be a minimum number of simulation scenarios that have a specific feature. For example, a property of the set of simulation scenarios can be required to ensure that simulation scenarios of different types, such as inner-city scenarios, highway scenarios, and / or scenarios in which specified objects occur, occur with at least a specified frequency. It can also be required as a property that a specified proportion of simulation scenarios in the set lead to specified traffic situations, for example, describing an overtaking maneuver and / or resulting in a risk of collision.

[0017] In a preferred embodiment, determining the property of the set of simulation scenarios comprises analyzing each modified simulation scenario using at least one neural network and / or executing at least one simulation of the modified simulation scenario. By executing the simulation, it can be ensured, for example, that the modified simulation scenarios lead to a collision hazard.

[0018] In a preferred embodiment, the property is related to at least one feature of the simulation scenarios, in particular represents a characteristic property of the statistical distribution of simulation scenarios, and the set of simulation scenarios is expanded to obtain a desired statistical distribution of the simulation scenarios. The feature can, for example, indicate whether and / or how many objects of a given class occur in the simulation scenario. It can also be specified as a characteristic property that the set of simulation scenarios is sufficiently large to be able to conduct tests with a given confidence. For example, a predefined number of scenarios can be provided for different object or road user classes in order to deliver a sufficient amount of data for machine learning.

[0019] Preferably, the method comprises the steps of receiving a desired sensor configuration, generating simulated sensor data based on the simulation scenario and the desired sensor configuration, and exporting the simulated sensor data. The simulation scenarios include the spatial relationships of the scene and thus contain sufficient information to generate sensor data for any environmental sensors.

[0020] Particularly preferably, the method further comprises the step of training a neural network for perception using the simulated sensor data and / or testing an autonomous driving function using the simulated sensor data.

[0021] In a preferred embodiment, the received raw data has a lower resolution than the simulated sensor data. By first extracting the abstract scenario from the raw data, driving situations recorded with an older, low-resolution sensor can also be reused for new systems with a higher resolution.

[0022] In a preferred embodiment, the simulated sensor data comprises a plurality of camera images. Alternatively or additionally, scenarios recorded with a LIDAR sensor can be converted into images from a camera sensor.

[0023] The invention further relates to a computer-readable data carrier containing instructions which, when executed by a processor of a computer system, cause the computer system to carry out a method according to the invention.

[0024] Furthermore, the invention relates to a computer system comprising a processor, a human-machine interface and a non-volatile memory, wherein the non-volatile memory contains instructions which, when executed by the processor, cause the computer system to carry out a method according to the invention.

[0025] The processor may be a general-purpose microprocessor typically used as the central processing unit of a workstation, or it may include one or more processing elements capable of performing specific computations, such as a graphics processing unit. In alternative embodiments of the invention, the processor may be replaced or supplemented by a programmable logic device, e.g., a field-programmable gate array, configured to perform a specific number of operations and / or include an IP core microprocessor.

[0026] The invention is explained in more detail below with reference to the drawings. Similar parts are labeled with identical designations. The illustrated embodiments are highly schematic, meaning that the distances and the lateral and vertical dimensions are not to scale and, unless otherwise stated, do not have any deducible geometric relationships to one another.

[0027] It shows: Figure 1 shows an exemplary diagram of a computer system, Figure 2 shows a perspective view of an exemplary LIDAR point cloud, Figure 3 shows a schematic flow chart of an embodiment of the method according to the invention for generating simulation scenarios, and Figure 4 shows an exemplary synthetic point cloud from a bird's eye view.

[0028] Fig. 1 shows an exemplary embodiment of a computer system.

[0029] The embodiment shown comprises a host computer PC with a monitor DIS and input devices such as a keyboard KEY and a mouse MOU.

[0030] The host computer PC comprises at least one processor CPU with one or more cores, a random access memory RAM, and a number of devices connected to a local bus (such as PCI Express) that exchanges data with the CPU via a bus controller BC. The devices include, for example, a graphics processor GPU for driving the display, a USB controller for connecting peripherals, a non-volatile memory such as a hard disk or a solid-state disk, and a network interface NC. The non-volatile memory can include instructions that, when executed by one or more cores of the processor CPU, cause the computer system to perform a method according to the invention.

[0031] In one embodiment of the invention, indicated by a stylized cloud in the figure, the host computer may comprise one or more servers comprising one or more computing elements such as processors or FPGAs, wherein the servers are connected via a network to a client comprising a display device and input device. Thus, the method for generating simulation scenarios may be executed partially or entirely on a remote server, for example, in a cloud computing setup. Alternatively to a PC client, a graphical user interface of the simulation environment may be displayed on a portable computing device, in particular a tablet or a smartphone.

[0032] Fig. 2 shows a perspective view of an example point cloud, as generated by a conventional LIDAR sensor. The raw data has already been annotated with bounding boxes around detected vehicles. On the side of objects facing the LIDAR sensor, the density of measurement points is high, while on the back side, there are hardly any measurement points due to occlusion. Even objects further away may consist of only a few points.

[0033] In addition to LIDAR sensors, vehicles often also have one or more cameras, a receiver for satellite navigation signals (such as GPS), speed sensors (or wheel speed sensors), acceleration sensors, and yaw rate sensors. These are preferably stored during the journey and can thus be taken into account when generating a simulation scenario. In addition to higher resolution, camera data usually also provides color information, making it a good complement to LIDAR data or point clouds.

[0034] Fig. 3 shows a schematic flow chart of an embodiment of the method according to the invention for generating simulation scenarios.

[0035] The input data for scenario generation is point clouds captured at consecutive points in time. Optionally, additional data, such as camera data, can be used to enrich the information in the dataset, for example, if the GPS data is not sufficiently accurate. For this purpose, well-known algorithms for simultaneous positioning and mapping can be used.

[0036] In step S1 (merge LIDAR point clouds), the LIDAR point clouds of a specific region are merged or fused into a common coordinate system.

[0037] To construct a temporally valid / consistent scene, the scans from different points in time are correlated, and the relative 3D translation and 3D rotation between the point clouds are determined. For this purpose, information such as vehicle odometry, consisting of 2D translation and yaw or 3D rotation information, as determined from the vehicle sensors, and satellite navigation data (GPS), consisting of 3D translation information, are used. These are supplemented by lidar odometry, which provides relative 3D translations and 3D rotations using an iterative closest point (ICP) algorithm. Known ICP algorithms can be used for this, such as the algorithm described in the paper "Sparse Iterative Closest Point" by Bouaziz et al. at the Eurographics Symposium on Geometry Processing 2013.

[0038] This information is then fused using a graph-based optimization algorithm, which weights the given information (using its covariances) and calculates the resulting odometry. An example algorithm for graph-based optimization is described in the article "A Tutorial on Graph-Based SLAM" by Grisetti et al., Intelligent Transportation Systems Magazine, IEEE, 2(4):31-43, 2010. The calculated odometry can then be used to fuse the given sensor data (such as lidar, camera, etc.) into a common coordinate system. For the annotation of static objects, it is useful to combine the various single-image point clouds into a registered point cloud.

[0039] In step S2 (localize and classify static objects), static objects within the registered or merged point cloud are annotated, i.e., localized and identified.

[0040] Static object data includes buildings, vegetation, road infrastructure, and the like. Each static object in the registered point cloud is annotated either manually, semi-automatically, automatically, or through a combination of these methods. In a preferred embodiment, static objects in the registered or merged point cloud are automatically identified and filtered using conventional algorithms. In an alternative embodiment, the host computer can receive annotations from a human annotator.

[0041] Using a registered or merged, and therefore dense, point cloud brings many advantages when annotating static objects. With many more points available on an object, it is much easier to determine the correct position and size of each individual object. In addition, an object can be viewed from different angles while driving, providing us with additional points on the object in the lidar point cloud from all directions. Overall, this allows for much more accurate annotation of an object's boundaries. With a point cloud from a single viewpoint, only the points from that one viewpoint would be available for annotation. For example, when viewing an object from the front, it is difficult to determine the boundary at the rear of the object because there is no information to help estimate this boundary.In the composite point cloud, which includes images from multiple angles, static objects are clearly defined.

[0042] In step S3 (Generate road information), road information is generated based on the registered point cloud and camera data.

[0043] To generate the road information, the point cloud is filtered to identify all points that describe the road surface. These points are used to estimate the so-called ground plane, which represents the ground surface for the respective LIDAR point cloud or, in the case of a registered point cloud, for the entire scene. In the next step, the color information is extracted from the images generated by one or more cameras and projected onto the ground plane using the intrinsic and extrinsic calibration information, specifically the camera's lens focal length and angle of view.

[0044] The road is then created using this top-down image. First, the road boundaries are detected and labeled. In a second step, so-called segments and intersections are identified. Segments are parts of the road network with a constant number of lanes. In the next step, obstacles on the road, such as traffic islands, are merged. The next step is the labeling of road markings. The individual elements are then combined with the appropriate links to assemble everything into a road model that describes the geometry and semantics of the road network for this specific scene.

[0045] In step S4 (localize and classify road users), dynamic objects or road users are annotated in the consecutive point clouds.

[0046] Due to their dynamic behavior, road users are annotated separately. Since road users are moving, it is not possible to use a registered or composite point cloud; instead, the annotation of dynamic objects or road users takes place in single-image point clouds. Each dynamic object is annotated, i.e., localized and classified. The dynamic objects or road users in the point cloud can be cars, trucks, delivery vans, motorcycles, cyclists, pedestrians, and / or animals. In principle, the host computer can receive results of manual or computer-assisted annotation. In a preferred embodiment, the annotation is carried out automatically using known algorithms, in particular trained deep learning algorithms.

[0047] For annotation purposes, it is useful to consider images from at least one camera recording parallel to the LIDAR sensor, which, due to temporal coincidence, must show the same objects (assuming appropriate overlap of viewing angles). Compared to a sparse LIDAR point cloud, camera images contain more information, particularly due to their higher resolution. For example, a distant pedestrian (>100 meters) could be represented by only a single LIDAR point in the LIDAR point cloud, but be clearly visible in the camera image. Therefore, object detection in the camera images is expediently performed using well-known algorithms such as YOLO and a correlation with the corresponding region of the LIDAR point cloud.The camera information is therefore very helpful for identifying and classifying an object; moreover, classification based on camera information can also help determine the size of an object. Depending on the classification, predefined standard sizes are then used or specified for the various road users. This facilitates the reproduction of those road users who are never close enough to the detection vehicle to allow size determination from a dense point distribution on the object within the LIDAR point cloud.

[0048] After identifying and classifying the road users in the single-image LIDAR point clouds and, where available, camera images, the temporal chains for the individual objects are identified. Each road user is assigned a unique ID for all individual images, i.e., point clouds and images, in which they appear. In a preferred embodiment, the first image in which the road user appears is used to identify and classify the object, and then the corresponding field or tags are transferred to the subsequent images. In an alternative preferred embodiment, tracking, i.e., algorithmic tracking, of the road user takes place across successive images.Here, the detected objects are compared across multiple frames. If the overlap of the areas exceeds a specified threshold, meaning a match is detected, they are assigned to the same road user, ensuring that the detected objects have the same ID or unique identification number. These two techniques enable the generation of consistent temporal chains across the consecutive camera images and LIDAR point clouds. This creates temporal and spatial trajectories for each dynamic object, which can be used to describe the road user's behavior in the simulation. Thus, dynamic objects are correlated along the temporal course to obtain road user trajectories.

[0049] In step S5 (Create simulation scenario), a playback scenario for a simulation environment is created from the static objects, the road information and the trajectories of the road users.

[0050] Preferably, the information obtained during the annotation of the raw data in steps S2 to S4 is automatically transferred to a simulation environment. This information includes, for example, the sizes, classes, and attributes of static objects, as well as their trajectories for road users. It is expediently stored in a suitable file exchange format, such as a JSON file. JSON stands for JavaScript Object Notation, a common language for specifying scenarios.

[0051] First, the information about the road contained in the JSON file is transferred into a suitable road model in the simulation environment. This includes the road geometry and all semantics determined or introduced during annotation, such as information about which lanes merge into which other lanes at a boundary between road segments, or from which lane to which lane one is permitted to drive when crossing an intersection.

[0052] Next, all annotated static objects are placed in the scene. To do this, the classification for each object, including some additional attributes, is read or derived from the JSON file. Based on this information, a suitable 3D object is selected from a 3D asset library and placed at the appropriate location in the scene.

[0053] Finally, the road users are placed in the scene and move according to the annotated trajectory. To do this, waypoints derived from the trajectory are used and placed in the scene with the necessary temporal information. The driver models in the simulation then reproduce the vehicle's behavior as recorded during the test drive.

[0054] Based on the road information, the static objects, and the trajectories of the road users, a "replay scenario" is generated in the simulation environment. In this scenario, all road users behave exactly as in the recorded scenario, and the replay scenario can be replayed as often as required. This enables a high degree of reproducibility of the recorded scenarios within the simulation environment, for example, to test new versions of driving functions for similar malfunctions as those observed during the test drive.

[0055] In a further step, these replay scenarios are abstracted into "logical scenarios." Logical scenarios are derived from the replay scenarios by abstracting the concrete behavior of the various road users into maneuvers in which individual parameters can be varied within predefined parameter ranges for these maneuvers. Parameters of a logical scenario can include, in particular, relative positions, speeds, accelerations, starting points for certain behaviors such as lane changes, and relationships between different objects. By deriving or inserting maneuvers with meaningful parameter ranges, it is possible to execute variations of the recorded scenarios within the simulation environment. This forms the basis for a later expansion of the set of simulation scenarios.

[0056] In step S6 (Property OK?), a property is determined for the existing set of one or more previously created scenarios and compared with a target value. Depending on whether the desired property is met, execution continues in step S7 or step S9. Here, one or more characteristics of the individual scenario can be considered, for example, requiring that the set only includes scenarios that meet the required characteristic, or determining a characteristic property of the set of simulation scenarios, such as a frequency distribution. These can be formulated into a simple or combined criterion that the data set must meet.

[0057] The analysis of the dataset with regard to the desired specification ("delta analysis") can include, but is not limited to, the following questions: What is the distribution of the various objects in the dataset? What is the target distribution of the dataset for the desired application? In particular, a minimum frequency of occurrence for different classes of road users may be required. This could be verified using metadata or parameters of a simulation scenario.

[0058] Conveniently, during the annotation of the raw data in steps S2 to S4, features can already be identified that can be used in the analysis of the dataset. Preferably, the raw data is analyzed using neural networks to identify the distribution of features within the real lidar point cloud. For this purpose, a series of object detection networks, object trackers, and attribute detection networks are used, which automatically detect objects within the scene and assign attributes to these objects—preferably specific to the application. Since these networks are required anyway to create the simulation scenario, only minimal additional effort is required. The identified features can be stored separately and assigned to the simulation scenario. The automatically detected objects and their attributes can then be used to analyze features of the originally recorded scenario.

[0059] If the raw data already contains a set of multiple scenarios, the properties of the raw data set can be compared with the distribution specified for the application. These properties can include, in particular, the frequency distribution of object classes, lighting and weather conditions (attributes of the scene). The comparison result can be used to determine a specification for data enrichment. For example, it can be determined that certain object classes or attributes are underrepresented in the given data set and therefore corresponding simulation scenarios with these objects or attributes must be added. An insufficient frequency can be attributed to the fact that the objects or attributes in question only rarely occur in the specific region in which the data was recorded and / or happened to occur only rarely at the time of recording.

[0060] When analyzing the dataset, however, it may also be required that a number of simulation scenarios be present in which a pedestrian crosses the street and / or a collision hazard occurs. Therefore, a simulation of the respective scenario can optionally be performed to determine further specification and selection of useful data for enrichment based on scenario-based testing. As a result, a specification for enriching the data is preferably defined, specifying which scenarios are required.

[0061] Optionally, scenario-based testing can be used to identify suitable scenarios to refine the specification for data augmentation. For example, if critical scenarios in inner cities are of particular interest, scenario-based testing can be used to identify scenarios with specific key performance indicators (KPIs) that meet all requirements. Accordingly, the augmented dataset can be restricted to selected and therefore relevant scenarios, rather than simply permuting by KPIs.

[0062] If the property is not given (No), in step S9 (Add simulation scenario) the data set is extended by varying the simulation scenario.

[0063] During this expansion of the dataset in step S9, which is expediently carried out in the simulation environment, a user can define any statistical distribution they wish to achieve in their expanded dataset based on the automatically determined distribution within the raw data. This distribution can preferably be achieved by generating a digital twin from the existing, at least partially automatically annotated scenario. This digital twin can then be permuted by adding road users with a specific object class and / or with different behavior or a modified trajectory. A virtual data collection vehicle with any sensor equipment is placed in the added permuted simulation scenarios and is then used to generate new synthetic sensor data.The sensor configuration can differ in any way from that used to record the raw data. This is useful not only for supplementing existing data, but also when the sensor configuration of the vehicle under consideration changes and the recorded scenarios serve as the basis for generating sensor data from the new / different sensor configuration.

[0064] Within the simulation environment, the set of simulation scenarios can be expanded not only by additional road users, but also by changing contrasts, weather and / or lighting conditions.

[0065] Optionally, existing scenarios from a scenario database can be included in the expansion of the set of simulation scenarios; these can have been created using previously recorded raw data. Such a scenario database increases the efficiency of data expansion; scenarios for various use cases can be stored in the scenario database and annotated with properties. By filtering based on the stored properties, suitable scenarios can be easily retrieved from the database.

[0066] If the condition is met (Yes), simulated sensor data for the simulation scenario or the complete set of simulation scenarios is exported in step S7 (Export sensor data to simulation scenarios). Based on a sensor configuration, such as a height above the ground and a specified resolution, simulated sensor data for one or more environmental sensors can be generated. In addition to LIDAR data, camera images can also be generated based on the camera parameters from the simulation scenario.

[0067] In the illustrated embodiment, the export of simulated sensor data is performed for the entire set of scenarios; in general, alternatively or additionally, each individual scenario could be exported independently after creation. A prior export is necessary, for example, if the determination of a feature requires the simulation of the scenario.

[0068] By applying neural networks, all scenarios can be exported as simulated sensor data or sensor-realistic data. Neural networks, such as generative adversarial networks (GANs), can mimic the properties of various sensors, including sensor noise. This allows both the properties of the sensor originally used to record the raw data and the properties of completely different sensors to be simulated. Therefore, a scenario that largely replicates the raw data can be used for training or testing an algorithm. Furthermore, a virtual recording of the driving scenario can be generated and used using various LIDAR sensors, as well as other imaging sensors, such as a camera.

[0069] In step S8 (Test autonomous driving function using sensor data), the exported one or more scenarios, i.e., self-contained sets of simulated sensor data, are used to test an autonomous driving function. Alternatively, they can also be used to train the autonomous driving function.

[0070] The invention enables augmentation of a dataset of lidar sensor data by intelligently supplementing data missing from the original dataset. By exporting the additional data as realistic sensor data, it can be used directly for training and / or testing an autonomous driving function. It is also possible to create or use a dataset that includes both the original raw data and the synthetic sensor data.

[0071] As a specific use case, the pipeline described above can be used to convert data from one sensor type into a synthetic point cloud of the same environment and with the scenario of another sensor type. For example, data from old sensors can be converted into synthetic data representing a modern sensor. Accordingly, old sensor data can be used to augment the data from current recordings with a new sensor type. For this use case of outputting "recycled" sensor data, a processing pipeline is preferably used, which in particular includes steps S1, S2, S3, S4, S5, and S7.

[0072] Abb. 4 shows a bird's-eye view of a synthetic point cloud resulting from the export of sensor data from the simulation scenario. Individual road users are marked by bounding boxes.

[0073] The method according to the invention enables the supplementation of measured sensor data from a test scenario with simulated sensor data to create varied simulation scenarios. This allows for better training of perception algorithms and more comprehensive testing of autonomous driving functions.

Claims

1. A computer-implemented method for generating a simulation scenario for a vehicle, comprising the steps of receiving raw data, the raw data comprising a plurality of consecutive LIDAR point clouds, a plurality of consecutive camera images, and a plurality of consecutive velocity and / or acceleration data, localizing and classifying (S4) one or more dynamic road users within the plurality of consecutive LIDAR point clouds and generating trajectories for the road user(s), creating (S5) a simulation scenario, and exporting (S7) the simulation scenario, characterized by the steps of merging (S1) the plurality of consecutive LIDAR point clouds of a specified region in a common coordinate system to generate a composite point cloud, localizing and classifying (S2) one or more static objects within the composite point cloud, generating (S3) road information based on the composite point cloud, one or more static objects, and at least one camera image, wherein the simulation scenario is created based on the one or more static objects, the road information, and the trajectories created for the one or more road users.

2. The method according to claim 1, further comprising the step of modifying (S9) the simulation scenario, in particular by modifying at least one trajectory and / or adding at least one further dynamic road user, before exporting (S7) the simulation scenario .

3. The method according to claim 2, wherein the steps of modifying (S9) the simulation scenario and exporting (S7) the simulation scenario are repeated, and wherein before exporting (S7) the simulation scenario, a different modification is applied in each case so that a set of simulation scenarios is compiled.

4. The method according to claim 3, wherein at least one property of the set of simulation scenarios is determined (S6), and wherein modified simulation scenarios are added to the set of simulation scenarios until the desired property is satisfied.

5. The method according to claim 4, wherein determining (S6) the property of the set of simulation scenarios comprises analyzing each modified simulation scenario using at least one neural network and / or performing at least one simulation of the modified simulation scenario.

6. The method according to claim 4 or 5, wherein the property is related to at least one feature of the simulation scenarios, and wherein the set of simulation scenarios is extended to obtain a desired statistical distribution of the simulation scenarios.

7. The method according to any one of the preceding claims, wherein exporting (S7) the simulation scenario comprises receiving a desired sensor configuration, generating simulated sensor data based on the simulation scenario and the desired sensor configuration, and exporting the simulated sensor data.

8. The method according to claim 7, further comprising the step of training a neural network for perceiving using the simulated sensor data, and / or testing (S8) an autonomous driving function using the simulated sensor data.

9. The method according to claim 7 or 8, wherein the received raw data has a lower resolution than the simulated sensor data.

10. The method according to any one of claims 7 to 9, wherein the simulated sensor data comprises a plurality of camera images.

11. A computer readable medium comprising commands which, when executed by a processor of a computer system, cause the computer system to perform a method according to any one of the preceding claims.

12. A computer system comprising a processor, a human-machine interface, and a non-volatile memory, wherein the non-volatile memory contains commands which, when executed by the processor, cause the computer system to perform a method according to any one of claims 1 to 10.