Method for Modeling a Motor Vehicle Sensor in a Virtual Test Environment

Abstract

The present invention relates to a method for modeling a motor vehicle sensor in a virtual test environment by defining and using a sensor support (1), a light projection distribution shape (2), a set of light projection attributes (3, 4, 5), a light projection reflection factor (6), and a light projection echo (7).

Description

Technical Field

[0001] The present invention relates to a method for modeling a motor vehicle sensor in a virtual test environment, in particular a three-dimensional test environment. Background Art

[0002] Virtual testing and virtual development have become the required methods for developing the design features of autonomous driving. A key element of such virtual simulation is the use of models of virtual sensors in a virtual test environment. There are many ways to model virtual sensors (ideal behavior, statistical models, physical models, phenomenological models, etc.).

[0003] The design and testing of related software face new requirements for the appearance of the design features of autonomous and self-driving motor vehicles. Due to the nature of the algorithms used to implement autonomous driving, a large number of tests are required to verify the corresponding software. On the one hand, the preparation of real prototypes and test environments is time-consuming and expensive. On the other hand, due to the high number of kilometers required to verify the algorithms implemented in the autonomous features, the implementation of driving tests is a cumbersome process. For these reasons, methods for virtual testing have become increasingly common. They enable the software at the vehicle level to be tested in a physics-based virtual environment and in an increasing number of realistic environments.

[0004] A key aspect of a virtual test environment is the ability to model sensors with different precisions according to test requirements. The modeling of sensors is essential because they are the eyes and ears of the algorithms for autonomous driving. Various methods have been developed to test camera-based sensors by supplying images from virtual reality to a real camera, which can currently be created with a very high degree of realism. For "wave-based" sensors (ultrasound, radar, lidar, lasers, etc.), there are different modeling methods, such as ideal models, statistical models, up to physical models.

[0005] Due to the complexity of such sensors, the use of physical sensor models in a virtual test environment is very demanding. Physical sensor models require a large amount of computing power and memory, a large amount of experience, and a virtual test environment capable of modeling physical events related to wave propagation, such as different sound speeds depending on atmospheric conditions, interference due to rain, snow, trees, moving grass, etc. In addition, the design features for autonomous driving usually require a large number of sensor information items, which limits the use of such sensor models for virtual testing.

[0006] DE 10 2016 100 416 A1 discloses the use of a virtual environment for testing and training real sensors via virtual sensors. It is proposed to directly upload the calibration processed in the virtual environment to the real sensors. The use of sensor models is mentioned, but no details on how to create these models in a formal and systematic way are given. The virtual sensors can be image-based (including "depth buffer") or ray-casting-based. These are two standard techniques currently used for sensor modeling in a virtual environment. However, this does not provide an indication of the way these techniques are used for sensor modeling. It is also mentioned that raw data of the virtual sensors is provided by ray tracing. For example, where a virtual lidar sensor can output a complete point cloud that a real lidar sensor can output, but no indication is given on how to model the lidar sensor in the virtual environment.

[0007] US 2016 0 236 683 A1 proposes the use of a parametric mathematical model for sensor modeling.

[0008] US 2009 0 300 422 A1 proposes a statistical method for generating virtual sensor data.

[0009] The NIST (National Institute of Standards and Technology) publication 'Research on Safety Control of Manufactured Vehicles towards Standard Test Methods', Bostelman, R. et al., June 7, 2012, http: / / www.mhi.org / downloads / learning / cicmhe / colloquium / 2012 / bostelman.pdf, proposes a method for modeling a 3D vehicle model by ray casting to obtain several spatial features of the 3D model.

[0010] The publication 'Simulation and Regression Testing Framework for Autonomous Vehicles', C.K. Miller et al., August 2007, http: / / engr.case.edu / cavusoglu_cenk / papers / ChristianMillerMS2007.pdf, describes a simulation and test environment for autonomous vehicles that implements regression testing of the corresponding software, i.e., checking whether the modified software still behaves at certain points like the earlier version of the software.

[0011] US 8 705 792 discloses an object tracking mechanism and method using ray-casting technology.

[0012] US 9 176 662 describes how to use ray casting to model behaviors (such as light reflection) for testing camera lenses.

[0013] In the cited prior art, virtual simulation and test environments have been proposed and the use of virtual sensors has been mentioned, but there is no specific description of how to model virtual sensors therein. Summary of the Invention

[0014] The present invention is based on the object of providing a systematic and generally applicable method for modeling virtual sensors in a virtual environment.

[0015] The present invention provides a method for modeling a virtual sensor in a virtual test environment, which is based on the so-called ray casting technique and is particularly suitable for modeling sensors based on wave propagation (such as ultrasonic, radar, lidar, laser, etc.).

[0016] Specifically, a method for modeling a motor vehicle sensor in a virtual test environment includes the definition of a sensor mount, a ray casting distribution shape, a set of ray casting attributes, a ray casting reflection factor, and a ray casting echo.

[0017] The method according to the present invention is based on ray casting, which is common in computer games (and in computer game development environments, the so-called game engines). Its core idea is to describe the line of sight of a three-dimensional sensor and use ray casting distribution to describe variables (such as spatial resolution, temporal resolution, speed, etc.). The ray casting distribution can be obtained by modeling from scratch in a virtual test environment or by an automatic generation method based on the attributes of real sensors, which is a preferred embodiment of the present invention.

[0018] The present invention enables virtual sensors to be modeled as objects in a virtual environment in a formal and generally applicable manner and uses the ray casting technique as a building block.

[0019] Implementing the features of the present invention also means modifying the nature of ray casting, which is essentially an instantaneous object in a virtual environment, i.e., the result is available at any processing stage. Thus, a new type of sensor ray casting is produced, as well as a new algorithm capable of decomposing the data of real sensors into ray casting layouts and ray casting parameters.

[0020] For example, one imagines wanting to model the ultrasonic sensor of a parking assistance system. One can simply send a ray casting perpendicular to a three-dimensional vehicle model into the virtual environment. If an algorithm that evaluates the distance to the closest obstacle is to be evaluated, this will be sufficient. However, if an algorithm that enables the distance to the closest obstacle to be determined based on the propagation of sound waves is to be evaluated, a simple ray casting is not sufficient, and one will need not only a sensor model for a specific sensor, but multiple sensor models depending on the test case. The present invention enables the sensor model to be adapted to the personalized needs of the user.

[0021] In the above-mentioned DE 10 2016 100 416 A1, the sensor behavior model is implemented as part of a post-processing algorithm which accesses the ray casting via a shared memory.

[0022] In the present invention, the sensor model essentially provides the complete data spectrum in time and space, and the post-processing algorithm does not need to obtain the data.

[0023] Thus, the inventors have developed a LIDAR sensor model in a three-dimensional environment that provides a lot of freedom. Thus, for example, one can decide on a bunch of 64 raycasts that rotate in reality at a high frequency, or one can decide on a bunch of N raycasts that cover a 360° area around the vehicle model and are refreshed at a lower frequency. The way of modeling the sensor using raycasts provides the developer with different items of information and therefore results. This shows the importance of describing how to model the sensor.

[0024] Each sensor has its specific properties and each phase of the project requires a different sensor model. Modeling each sensor specifically would take a long time and would force the developer to wonder every time from the beginning how he has to proceed in order to model the sensor. In contrast, the method according to the invention is of a general nature and can be automatic. For example, one can imagine an electronic assistance system that guides people through the process of sensor modeling and creation.

[0025] In a preferred embodiment, the sensor holder is a virtual sensor holder for a virtual sensor model, which forms a three-dimensional or two-dimensional substitute for the sensor in the virtual test environment and has a sensor starting point or a sensor starting surface, which is used as the starting point of the ray projection distribution shape or its ray projection.

[0026] The ray projection distribution shape can be a predefined two-dimensional or three-dimensional shape, the starting point of which is the starting point or starting surface of the sensor holder, wherein the ray projection distribution shape has multiple uniformly distributed ray projections, the starting point of which can be the starting point of the sensor holder or a point in the starting plane of the sensor holder.

[0027] The raycast properties are specifically the damping, propagation speed, and detection accuracy of the raycast.

[0028] The method is particularly suitable for modeling sensors that operate based on electromagnetic or acoustic waves. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The exemplary embodiments are described below based on the accompanying drawings. In the accompanying drawings:

[0030] Figure 1 Shows an illustration of ray casting;

[0031] Figure 2 Shows a system overview; and

[0032] Figure 3 Shows an example of a sensor model. Detailed Description

[0033] To explain ray casting, first refer to Figure 1 .

[0034] In a computer game (or in a computer game development environment, game engine), ray casting, or "visual beam", is represented by at least one three-dimensional vector D in a three-dimensional reference coordinate system xyz, which has a specific length (up to infinity) and an application point or starting point C placed on an object A. Ray casting provides the possibility of finding all objects B located on its trajectory, which particularly enables the coordinates of the contact point E between the ray casting and these objects to be found. Ray casting is typically used to simulate instantaneous projectile trajectories to determine object visibility, etc.

[0035] The sensor model consists of the following parts as shown in Figure 2 :

[0036] 1.) Definition of the sensor support 1.

[0037] The sensor support 1 is the "stand-in" of the sensor in a virtual test environment and has the following properties:

[0038] - A starting point or starting surface (i.e., starting point) for the ray casting distribution shape 2;

[0039] - A time-varying 3D rotational motion that indicates whether the sensor can rotate, for example, at a specific angular velocity or whether the sensor is fixed.

[0040] 2.) Ray casting distribution shape 2:

[0041] - The ray casting distribution shape 2 is a three-dimensional shape (e.g., a cone, a cylinder, or a regular volume), the starting point of which is the starting point or starting surface of the sensor support 1.

[0042] - The dimensions of the ray casting distribution shape 2 simulate the maximum range of the sensor in three-dimensional space.

[0043] - The ray casting distribution contains N ray castings, which are evenly distributed in the ray casting distribution shape 2. The starting point of each ray casting is the starting point of the sensor support 1 or a point in the starting plane of the sensor support 1.

[0044] - The number of ray casts contributes to the spatial resolution of the sensor. They can also be used to simulate the discretization of sensor information items.

[0045] 3.) Definition of ray cast damping 3:

[0046] - The damping 3 can be the same for all ray casts of the ray cast distribution shape 2 or specific for each ray cast.

[0047] - The damping 3 can be a coefficient or formula within the ray cast distribution shape 2 that specifies the percentage of the original signal to be returned in the case where an object is struck by a ray cast at a specific distance from the starting point.

[0048] 4.) Definition of ray cast propagation speed 4:

[0049] - The propagation speed 4 can be the same for all ray casts of the ray cast distribution shape 2 or specific for each ray cast.

[0050] - The propagation speed 4 is used to define the delay after which the echo value of a ray cast becomes available in the sensor at the distance where the ray cast strikes an object.

[0051] - The propagation speed 4 can be used to simulate the properties of waves (such as sound waves, light waves, electromagnetic waves, etc.) emitted by the sensor and environmental effects (such as weather, temperature, etc.).

[0052] 5.) Definition of ray cast detection accuracy 5:

[0053] - The detection accuracy 5 can be the same for all ray casts of the ray cast distribution shape 2 or specific for each ray cast.

[0054] - The detection accuracy 5 is a correction factor that enables the error of the echo value of a ray cast to be simulated. It is the probability that a ray cast returns an incorrect value (e.g., mainly due to the absence of a target, possible false alarms).

[0055] 6.) Definition of reflection factor 6:

[0056] - The definition of the reflection factor 6 is associated with the objects present in the virtual test environment.

[0057] - The reflection factor is a percentage representing the degree to which an object reflects the waves emitted by the sensor.

[0058] - The reflection factor enables materials to be simulated that reflect weakly due to their shape, structure, properties, etc. (e.g., some clothes).

[0059] 7.) Definition of raycast echo 7:

[0060] - Each raycast can send back a batch of values, called echoes, i.e.

[0061] a) a 3D coordinate corresponding to the point of impact with the object (or nothing if no hit occurred), and

[0062] b) an echo value, which lies in the range [0, ..., 1] and depends on the detection accuracy 5, the value of the damping 3, the reflection factor 6 of the object being hit, and whether there is a hit. It can be provided by the following equation:

[0063] Accuracy value = (detection accuracy) × (damping) × (reflection factor) × (1 if hit, otherwise 0)

[0064] - Each value of a raycast echo 7 is available after a propagation time determined by the propagation speed 4 defined for that raycast and the distance of the hit object from the starting point of the raycast.

[0065] If necessary, it is also possible to model the sensor in only two dimensions.

[0066] Figure 3 An example of a sensor model is shown. Figure 3 In , G is the sensor holder 1, which consists of a cylinder and is a 3D stand-in for the sensor, which is placed in the virtual environment and located at the starting point H, which is the starting point of the raycast distribution shape M. The raycast distribution shape M is a cone; the dimensions of the cone provide the maximum range of the sensor in three-dimensional space. The raycast distribution shape M contains four raycasts R, which means that the sensor can detect objects passing through these raycasts R. For each raycast R, the damping 3, the propagation speed 4 and the detection accuracy 5 can be defined.

[0067] It is possible to automatically generate sensor models based on, among other things:

[0068] - Measurements from real-world sensors that are analyzed by a machine learning algorithm trained to extract the parameters needed to model the sensors.

[0069] - Test cases or use cases, which are specified in a machine-interpretable language (e.g., pseudo-natural language) and analyzed by an algorithm that extracts the specific application of the sensor in the use case. This is particularly useful for testing error scenarios or safety-related scenarios.

[0070] - Sensor data tables, which are provided in a standard format (e.g., XML (Extensible Markup Language)) and can be provided by the sensor vendor. Algorithms (e.g., serializers) can extract the parameters needed to model the sensor from such digital data tables.

[0071] In summary, the following methods and / or algorithms are essential:

[0072] A method for modeling a virtual vehicle sensor in a three-dimensional virtual environment by defining and using a sensor mount 1, a raycast distribution shape 2, raycast properties (particularly its damping 3, propagation speed 4 and detection accuracy 5), a raycast reflection factor 6 and a raycast echo 7 (which is the return value of the raycast).

[0073] A method for defining a virtual sensor support of a virtual sensor model having

[0074] - 3D or 2D avatars of sensors shown separately;

[0075] - a sensor origin point or a sensor origin surface that is used as a starting point for the raycast distribution shape or its raycast; and

[0076] - A definition of the sensor mount motion, which describes whether the sensor mount is a fixed mount or can move after being attached to a virtual object (e.g. in the case of LiDAR or rotating radar). If the sensor can move, the associated motion is also described (e.g. rotation speed, etc.).

[0077] A method for defining a raycast distribution shape for a virtual sensor model, wherein:

[0078] - the raycast distribution shape is a three-dimensional (e.g., cone, cylinder, or regular volume) or a two-dimensional shape, the origin of which is the starting point or starting surface of the sensor holder;

[0079] - The size of the raycast distribution shape simulates the maximum range of the sensor in 3D or 2D space;

[0080] - the raycast distribution comprises N raycasts following a specific distribution in the raycast distribution shape, typically a uniform distribution, wherein the starting point of each raycast is a starting point of the sensor holder or a point in a starting plane of the sensor holder; and

[0081] - The number of ray casts leads to the spatial resolution of the sensor and can also be used to model the discretization of sensor information items.

[0082] A method for defining the damping of each ray cast for a ray cast distribution shape, wherein:

[0083] - The damping is the same for all ray casts of the ray cast distribution shape or specific for each ray cast in order to simulate boundary or constraint conditions;

[0084] - The damping is typically expressed as a percentage in the range [0,... 1]; and

[0085] - The damping is defined as a single value or formula that specifies the percentage of the original signal that will be returned in the case where an object is struck by a ray cast within the ray cast distribution shape at a specific distance from the origin.

[0086] A method for defining the propagation speed of a ray cast, wherein:

[0087] - The propagation speed is the same for all ray casts of the ray cast distribution shape or specific for each ray cast;

[0088] - The propagation speed is used to define the delay after which the echo value of the ray cast is available in the sensor at the distance where the ray cast strikes the object; and

[0089] - The propagation speed is used to simulate the properties of waves (acoustic waves, light waves, electromagnetic waves, etc.) emitted by the sensor and to simulate environmental effects (such as weather, temperature, etc.).

[0090] A method for defining the detection accuracy of a ray cast, wherein:

[0091] - The detection accuracy is exactly the same for all ray casts of the ray cast distribution shape or specific for each ray cast; and

[0092] - The detection accuracy is a correction factor that enables the error of the echo value of the ray cast to be simulated. It is the probability that the ray cast sends back an incorrect value (e.g., mainly due to missing targets, possible false alarms).

[0093] A method for defining the detection accuracy of a ray cast, wherein:

[0094] - The detection accuracy is the same for all ray casts of the ray cast distribution shape or specific for each ray cast;

[0095] - The detection accuracy is typically expressed as a percentage in the range [0,... 1];

[0096] - The detection accuracy is a correction factor that enables the error of the echo value of the light projection to be simulated (e.g., it can represent the probability that the light projection sends back an incorrect value (e.g., mainly due to the lack of a target, possible false alarms)); and

[0097] - The detection accuracy is used in the early stages of the project to simulate the errors caused by the signal processing algorithms embedded in the sensors.

[0098] A method for defining an activation time pattern for each light projection of the light projection distribution shape, where the activation time pattern is used to specify when each light projection of the sensor is activated (e.g., together, one after another with a specific delay, etc.).

[0099] A method for defining the reflection factor of a virtual object for virtual sensor testing, where:

[0100] - The definition of the reflection factor is associated with a specific material, which is associated with each object in the virtual test environment;

[0101] - The reflection factor is usually expressed as a percentage within the range [0,... 1];

[0102] - The reflection factor is a percentage representing the degree to which an object reflects the wave emitted by the sensor; and

[0103] - The reflection factor enables the simulation of materials that reflect weakly due to their shape, structure, properties (e.g., some clothes), etc.

[0104] A method for defining the echo for each light projection of a sensor model, where:

[0105] - Process the echo for each light projection of the sensor model;

[0106] - The echo is a value within the range [0,... 1], which is a function of the damping value, detection accuracy, and the reflection factor and presence of the hit of the object struck by the light projection;

[0107] - The echo value is available after a time delay, which is a function of the light projection propagation speed and the distance to the struck object; and

[0108] - The echo value is associated with the three-dimensional or two-dimensional coordinates of the struck object.

[0109] A method for automatically generating a virtual sensor model, where:

[0110] - The method is an algorithm of machine learning that is trained to extract the parameters required for sensor modeling from the measurement values of real sensors;

[0111] - The method is an algorithm that analyzes a test case or application case specified in a machine-interpretable language (e.g., pseudo-natural language) and extracts the specific applications of sensors in the test case or application case (which is particularly useful for test error scenarios or safety-related scenarios); and

[0112] - The method is an algorithm that extracts the parameters required for sensor modeling from a sensor data sheet, which is provided in a standard format (e.g., XML) and is provided by the sensor vendor.

Claims

1. A method, the method comprising extracting parameters required for sensor modeling from measurements of a motor vehicle sensor in reality so as to establish a virtual motor vehicle sensor model in a virtual test environment, the virtual sensor model including a virtual sensor support (1), a virtual ray casting distribution shape (2), a virtual set of ray casting attributes (3, 4, 5), a virtual ray casting reflection factor (6), and a virtual ray casting echo (7); wherein the virtual sensor support (1) forms a three-dimensional or two-dimensional substitute of the sensor in the virtual test environment and has a sensor starting point or a sensor starting surface, the sensor starting point or the sensor starting surface serving as the starting point of the ray casting distribution shape or its ray casting, wherein the ray casting distribution shape (2) is a predefined two-dimensional or three-dimensional shape, the starting point of the predefined two-dimensional or three-dimensional shape being the starting point or the starting surface of the sensor support (1), wherein the ray casting distribution shape (2) has a plurality of evenly distributed ray castings, the starting points of the plurality of evenly distributed ray castings being the starting point of the sensor support (1) or points in the starting surface of the sensor support (1), wherein the ray casting attributes are the damping (3) of the ray casting, the propagation speed (4), and the detection accuracy (5), wherein the damping (3) is the same for all ray castings of the ray casting distribution shape or is specific for each ray casting and is defined as a single value or a formula, wherein the propagation speed (4) is the same for all ray castings of the ray casting distribution shape or is specific for each ray casting and defines a delay after which, at the distance where the ray casting strikes an object, the echo value of the ray casting is available in the sensor, wherein the detection accuracy (5) is exactly the same for all ray castings of the ray casting distribution shape or is specific for each ray casting and is the probability that the ray casting sends back an incorrect value, wherein the reflection factor (6) is associated with the object present in the virtual test environment and is a percentage indicating the degree to which the object reflects the wave emitted by the sensor, wherein the ray casting echo (7) is a set of coordinates and echo values sent back by each ray casting, and wherein the echo value depends on the detection accuracy (5), the damping (3), the reflection factor (6), and the presence of a hit.

2. The method according to claim 1, wherein the single value or formula specifies a percentage of the original signal that will be returned when the object is struck by the ray casting at a specific distance from the starting point.

3. The method according to claim 1, wherein a sensor operating based on electromagnetic waves or sound waves is modeled.

4. A method for automatically generating a virtual sensor model for use in the method according to any one of the preceding claims, wherein: - The method of automatically generating a virtual sensor model is an algorithm of machine learning that is trained to extract the parameters required for modeling the sensor from the measurement values of a real sensor; - The method of automatically generating a virtual sensor model is an algorithm that analyzes test cases or application cases specified in a machine-interpretable language and extracts the specific applications of the sensor in the test cases or application cases; and - The method of automatically generating a virtual sensor model is an algorithm that extracts the parameters required for modeling the sensor from a machine-readable sensor data sheet.

Images