METHOD FOR MODELING A SENSOR IN A TEST ENVIRONMENT

Description

[0001] The invention relates to a method for modeling a sensor, in particular an FMCW LiDAR sensor, in a virtual test environment, in particular for pre-development and / or testing of algorithms or software in an industrial environment and / or in particular for testing software for mobile robots and / or for industrial trucks.

[0002] The development of high-performance, highly integrated sensor technologies, such as FMCW LiDAR sensors or imaging radar sensors, is currently being driven forward. The use of such sensors is crucial for a wide range of applications, particularly with regard to mobile robots and / or industrial trucks. Accordingly, new sensor technologies and their performance in the corresponding applications must be tested before commercial use. WO2021004626A1 describes a method for simulating CW LiDAR sensors. This method involves a complex and time-consuming simulation of the measurement principle in order to realistically recreate the functionality of a real CW LiDAR sensor.

[0003] A disadvantage of this method is that a large amount of sensor-specific information is required to simulate the actual functionality of the CW LiDAR sensor. Furthermore, simulating the CW LiDAR sensor at this level of abstraction is very complex and computationally intensive.

[0004] EMDE MARKUS et al. ("Modelling and Simulation of a Laser Scanner with Adjustable Pattern as Virtual Prototype for Future Space Missions", 2015 3RD INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE, MODELLING AND SIMULATION (AIMS), IEEE, December 2, 2015 (2015-12-02), pages 191-196, XP032983838, DOI: 10.1109 / AIMS.2015.83 [retrieved 2016-10-20]) discloses a simulation of a laser scanner in which a line-of-sight error generator uses Gaussian-distributed error models to generate erroneous azimuth and elevation angle values ​​that are used to reorient the laser scanner. EP 3 637 138 A1 discloses a method for simulating a LiDAR laser beam, in which an origin region of the laser beam is divided into several origins, whereby the propagation of the laser beam can be modeled with a probability.US 2022 / 058309 A1 teaches a method for testing software for an autonomous vehicle, wherein the simulated test environment is generated based on real environmental data, and the direction of movement and speed of a detected object can be determined based on the simulated LiDAR data. US 2018 / 203445 A1 describes a method in which a plurality of sensors are simulated, wherein the simulated sensor output values ​​are used to train a machine learning model, which is used, for example, in a real vehicle for collision avoidance. US 2022 / 269279 A1 relates to a computer system for testing a robot system, wherein detection probabilities are measured during object detection by establishing a detection probability distribution depending on a distance or an angle.

[0005] One object underlying the invention is therefore to provide an improved method for investigating (new) sensors in the context of existing or new applications, as well as a corresponding system. This object is achieved by a method according to claim 1 and a system according to claim 13.

[0006] The invention relates to a computer-implemented method for modeling a sensor, in particular an FMCW LiDAR sensor for distance measurement in a virtual test environment, in particular for testing software for mobile robots and / or for industrial trucks, wherein: a simulation model is defined, wherein the simulation model comprises a virtual sensor and the virtual test environment, wherein in the method: a virtual transmission signal sent by the virtual sensor is simulated, in particular by means of ray tracing, in the virtual test environment; it is determined whether the virtual transmission signal hits a virtual object in the virtual test environment at an impact point; if the determination is positive, a distance traveled by the virtual transmission signal from the virtual sensor to the impact point in the virtual test environment is calculated;and at least one output value of the virtual sensor is determined based on the calculated distance, wherein the output value is further determined based on at least one parameter of the simulation model, wherein the parameter and / or the output value are modeled by a probabilistic distribution, wherein the parameter comprises an azimuth angle of the virtual transmission signal and / or an elevation angle of the virtual transmission signal, wherein the virtual transmission signal is simulated, in particular only, based on the azimuth angle, the elevation angle and a starting point of the virtual transmission signal. ;

[0007] The procedure further includes: the azimuth angle and elevation angle of the virtual transmission signal are modeled by a probabilistic distribution with a predefined reference azimuth angle and a predefined reference elevation angle as expected values; a plurality of different virtual auxiliary transmission signals, whose azimuth angle and elevation angle differ from the reference azimuth angle and elevation angle, are selected based on the probabilistic distribution of the virtual transmission signal; For each virtual auxiliary transmission signal, it is determined whether the virtual auxiliary transmission signal hits the virtual object at the point of impact and / or at least one further virtual object at at least one further point of impact; if this is determined to be positive, a distance traveled by the virtual auxiliary transmission signal from the virtual sensor to the point of impact is calculated in the virtual test environment; and a final distance value is determined based on the plurality of calculated distance values.

[0008] In other words, a sensor is simulated in a virtual, particularly 2- or 3-dimensional, world in order to generate simulated sensor data, i.e., output values, which arise from the virtual sensor's detection of the virtual world. According to the invention, at least one parameter of the simulation model and / or the output value is modeled using a probabilistic distribution, i.e., modified by a probability distribution. By applying the probabilistic distribution to the parameter and / or the output value, a certain "variability" is created for the output value, whereby the fluctuations, for example, of measured values ​​from a real sensor during real operation can be simulated.

[0009] The virtual transmission signal is, in particular, an abstraction of an electromagnetic wave. For example, the virtual transmission signal can be a virtual light beam or a virtual transmission pulse. Alternatively, sound signals are also possible. The simulated sensor can therefore be, for example, an optical sensor (such as the aforementioned "Frequency-Modulated Continuous Wave Light Detection and Ranging" (FMCW-LiDAR) sensor), a radar sensor, or an ultrasonic sensor.

[0010] The application of the probabilistic distribution offers the advantage that many physical effects (e.g., the precise reflection characteristics of the sensor's transmitted signal on non-homogeneous surfaces or the behavior of the transmitted signal in rain) do not need to be simulated, as these effects can be cumulatively incorporated into the probabilistic distribution. The probabilistic distribution can be applied to the parameter and / or the output value with significantly less computational effort, resulting in significant savings in computing power and / or time.

[0011] As already described, the simulated virtual transmission signal is simulated in the simulation model in particular by means of a ray tracing method in which the path of the virtual transmission signals, in particular only, from the virtual sensor or a starting point to the point of impact of the virtual transmission signal on the object is taken into account.

[0012] In addition, a reflection of the virtual transmission signal at the point of impact can be simulated in the same way. Starting from the point of impact, another virtual transmission signal is emitted, which is then simulated up to another point of impact. This can be done recursively, thus allowing any number of reflections to be considered.

[0013] In addition, to simulate multi-echo effects, multiple impact points along a virtual transmission signal in the virtual test environment can be considered, starting from the sensor or an initial point. For example, a virtual transmission signal can have multiple impact points on different virtual objects along its beam direction. The distance traveled can then be calculated for each impact point, with an output value of the virtual sensor being calculated for each impact point based on the distance. The respective output value is also determined based on at least one parameter of the simulation model, with the parameter and / or the output value being modeled by a probabilistic distribution. This allows effects such as those that occur with transparent materials to be taken into account when calculating the output values.For example, an impact point on a pane of glass and at the same time an impact point on an object behind it can be simulated.

[0014] In particular, the return path of such virtual transmission signals after they hit a virtual object is not considered or determined. Rather, a comprehensive simulation of such virtual transmission signals, especially the behavior of the virtual transmission signals after they hit the virtual object, is omitted, thus saving computational effort and further accelerating the simulation. The simulation assumes that a transmission signal that hits an object will also return to the sensor. However, any attenuation or interference in the returning transmission signal is preferably not simulated, but rather modeled globally using the probabilistic distribution or incorporated into the output value.

[0015] Preferably, the simulation of sensor parameters, such as the simulation of an electromagnetic wave with the corresponding properties, such as wavelength modulation, is omitted. In other words, the simulation model is generated, in particular, only based on top-level information, such as the beam direction of the virtual transmission signal and the point of impact. As already mentioned above, the virtual transmission signal is thus simulated in a simplified manner. The simulation model can include, among other things, a sensor model that maps, for example, physical parameters of a real sensor in the simulation model, such as the rotation of a mirror in a laser scanner. The angle of rotation of the mirror can thus become a parameter of the simulation model.

[0016] If the point of impact of the virtual transmitted signal is known, the distance from the sensor to the point of impact can be calculated based on the information about the position of the impact point and the position of the sensor, for example, based on the corresponding coordinates. For example, a position vector from the sensor to the point of impact can be determined, the length of which represents the distance between the virtual sensor and the point of impact or the virtual object. It is understood that the distance traveled does not necessarily have to be calculated explicitly; implicit determination of the distance traveled is also possible.

[0017] As already indicated above, in the real world, when determining the distance to an object or the point of impact of an object, i.e. when a real distance sensor detects a distance to a real object, a multitude of other, particularly external, factors generally play a role, some of which have already been mentioned above. The properties of the emitted light, the weather conditions or the reflective properties of the material of the objects are just a few of many such factors. Since taking such factors into account in the simulation increases the computational effort, such factors are preferably not included in the simulation model according to the invention when determining the output value of the virtual sensor.

[0018] Rather, one or more parameters of the simulation model according to the invention and / or one or more output values ​​of the virtual sensor are modeled by a probabilistic distribution, which indirectly models the influence of the aforementioned factors on the output value of the virtual sensor. A simulation of the aforementioned factors is therefore not necessary. Rather, the use of the probabilistic distribution reduces the computational effort for the simulation model and, at the same time, provides the most accurate and realistic simulation model possible. The parameters of the simulation model can include all parameters used to calculate the output values. For example, the parameters of the simulation model can include properties of the virtual transmission signal and / or properties of the virtual object, e.g., its shape, size, and / or position.

[0019] According to the invention, the output value is further determined based on at least one parameter of the simulation model, wherein the parameter and / or the output value are modeled using a probabilistic distribution. The output value can correspond (at least approximately) to a measured value that a real sensor would have output in the simulated situation. The output value can be, for example, a distance value, a velocity vector, a speed value, e.g., a radial velocity value, an intensity value, and / or the like.

[0020] In particular, a plurality of different output values ​​can be output based on the probabilistic distribution of the output value. For example, output values ​​can be selected from the probabilistic distribution, in particular randomly or weighted according to their probability distribution, whose probability of occurrence assigned by the probabilistic distribution is preferably higher than a predefined threshold. Depending on the application, the various output values ​​can be output sequentially or together.

[0021] In addition to the advantages already mentioned above, the invention has the further advantage that the simulation of the sensor reduces the effort required to generate sensor data, allowing a large amount of data to be generated in a short time. Consequently, the effort and time required to test algorithms or applications that operate using the sensor data are also reduced. Furthermore, the virtual test environment can be modified as desired, and a multitude of different and specific scenarios can be simulated, allowing the sensor data to represent a diverse range of scenarios, particularly in industrial environments, for example, when operating mobile robots and / or industrial trucks.

[0022] Further embodiments of the invention can be found in the description, the subclaims and the drawings.

[0023] According to a first embodiment, the probabilistic distribution generates noise in the output value. More specifically, the noise causes the output value to fluctuate (e.g., when outputting multiple time-spaced output values). This noise thus simulates the real-world effects that would lead to fluctuations in actual measured values.

[0024] The probabilistic distribution can therefore represent mechanical inaccuracies of the (real) sensor (e.g. angular noise of a beam deflection), measurement inaccuracies of the (real) sensor (e.g. distance noise or general measurement noise) and the like.

[0025] According to a further embodiment, the probabilistic distribution comprises a normal distribution. A parameter can thus be modeled, for example, by specifying an expected value and a value for the standard deviation. A value assumed to be an ideal value or a reference value can be used as the expected value. In the case of a virtual transmission signal, as explained above, an ideal direction or reference direction of the normal distribution can be used as the expected value. In the case of a normally distributed distance value, for example, a distance value determined by the simulation model can be used as the expected value. The same applies to the determined radial velocity. The standard deviation of the normal distribution can determine the degree of noise.The standard deviation can, for example, be predefined for each sensor or can itself be a function of other simulation parameters, such as the distance value or the reflection value of the object hit.

[0026] As an alternative to the normal distribution, a log-normal distribution, an exponential distribution or another probability distribution can be used.

[0027] According to a further embodiment, the output value comprises a determined distance value to the virtual object or to the point of impact of the virtual transmission signal on the virtual object and / or a determined radial velocity of the point of impact of the virtual transmission signal on the virtual object with respect to the virtual sensor. The output value can, for example, be a value comparable to a measured value of a real sensor. In particular, the determined distance value and / or the determined radial velocity are modeled by the probabilistic distribution. The determined (ideal) distance value and / or the determined (ideal) radial velocity are thus subjected to noise in order to account for any inaccuracy in the output value.The determined distance value and / or the determined radial velocity can be determined, for example, by a calculation using the point of impact of the virtual transmission signal on the virtual object. In particular, only geometric data from the test environment can be used for the calculation.

[0028] Furthermore, the parameter includes an azimuth angle of the virtual transmission signal and / or an elevation angle of the virtual transmission signal.

[0029] Furthermore, the virtual transmission signal is simulated, in particular only, based on the azimuth angle, the elevation angle and a starting point of the virtual transmission signal.

[0030] For example, the azimuth angle and elevation angle of the virtual transmission signal, together with the origin point, specify a particularly unique beam direction of the virtual transmission signal. The origin point, for example, defines a unique point in the virtual test environment from which the virtual transmission signal is emitted. For example, the origin point can overlap with a position of the virtual sensor. The azimuth angle or horizontal angle specifies the angle of the virtual transmission signal with respect to a reference direction in a horizontal plane, while the elevation angle or vertical angle describes the angle of the virtual transmission signal with respect to a reference direction in a vertical plane.

[0031] In particular, the azimuth angle and / or elevation angle can be modeled using a probabilistic distribution. For example, the probabilistic distribution of the azimuth and / or elevation angle can be used to achieve a deviation of the virtual transmission signal from an intended beam direction, i.e., from a reference beam direction, which may include a reference azimuth angle and / or a reference elevation angle. This deviation can, for example, correspond to the mechanical inaccuracy when transmitting a transmission signal by a real sensor.

[0032] For example, the probabilistic distribution of the azimuth angle and / or elevation angle can be determined based on the reference beam direction by using the reference beam direction or the reference azimuth angle and / or reference elevation angle of the probabilistic distribution as the expected value(s).

[0033] Preferably, the probabilistic distribution of the azimuth angle and / or elevation angle can be incorporated into the calculation of the output value. For this purpose, after changing the azimuth angle and / or elevation angle, the point of incidence of the virtual transmission signal for the changed angles can be determined. This can then, for example, result in the calculated distance being different than for unchanged angles. In particular, data tuples comprising an azimuth angle that has changed compared to the reference azimuth angle and an elevation angle that has changed compared to the reference elevation angle are generated one after the other based on the probabilistic distribution of the azimuth angle and elevation angle in order to calculate corresponding points of incidence in the virtual test environment. In this way, the probabilistic distribution is incorporated into the output value.

[0034] According to a further embodiment, based on the probabilistic distribution of the parameter and / or the output value, a plurality of random values ​​are successively generated for the parameter and / or the output value, which are used to determine the output value, wherein a random value is generated depending on a previously generated random value. The random values ​​can be generated based on predefined or predetermined rules, for example, by introducing additive conditional probability distributions. An additive conditional probability distribution is characterized, for example, in that the probability distribution of one random variable depends on the probability distribution of another random variable.In particular, the probability distribution of the random variable "generating a random value at time t" can depend on the probability distribution of the random variable "generating a random value at time t-1." The generation of a random value can therefore depend on a previously generated random value. For example, the previously described data tuples can be generated sequentially, whereby the generation can be based on predefined rules. In particular, the predefined rules can consider or map a movement of the virtual sensor. This makes it possible, for example, to model the time-dependent deflection angle or jitter of a scanner mirror or the vibration of the sensor or a scanner mount due to environmental influences such as wind.Furthermore, the real temporal course of such parameters can be identified through experiments and used directly in the simulation.

[0035] Furthermore, the calculation of the distance travelled by the virtual sensor to the point of impact on the virtual object of the virtual transmission signal in the virtual test environment further comprises: the azimuth angle and elevation angle of the virtual transmission signal are modeled by a probabilistic distribution with a predefined reference azimuth angle and a predefined reference elevation angle as the expected value; a plurality of different virtual auxiliary transmission signals, whose azimuth angle and elevation angle differ from the reference azimuth angle and reference elevation angle, are selected based on the probabilistic distribution of the virtual transmission signal; for each virtual auxiliary transmission signal, it is determined whether the virtual auxiliary transmission signal strikes the virtual object at an impact point and / or at least one other impact point on at least one other virtual object; if the determination is positive, a distance traveled by the virtual auxiliary transmission signal from the virtual sensor to the impact point is calculated in the virtual test environment;and a final distance value is determined based on the plurality of calculated distance values. ;

[0036] In other words, a plurality of additional virtual auxiliary transmission signals are generated using the probabilistic distribution of the azimuth angle and elevation angle of the virtual transmission signal. In particular, a number N, for example 10, 50, 100, or 200, of virtual auxiliary transmission signals is generated. Based on the probabilistic distribution, the virtual auxiliary transmission signals are emitted in different directions. The distances traveled by these auxiliary transmission signals from the starting point to the point of impact on the virtual object or at least one further point of impact on at least one further virtual object are calculated and incorporated into the calculation of the final distance value, as explained in more detail below. Due to the different directions of the auxiliary transmission signals, the distance values ​​usually also differ; in particular, different virtual objects can be hit.

[0037] According to a further embodiment, the final distance value is determined by: a predefined number of bins of equal size is initialized for a predefined distance range, each bin being assigned to a different sub-range within the predefined distance range; for each auxiliary transmission signal, it is determined whether the corresponding virtual transmission signal has a respective point of impact with the virtual object and / or the at least one further virtual object in the virtual test environment, in particular in the predefined distance range, and if the determination is positive, an associated distance value from the starting point to the respective point of impact is determined, each distance value being assigned to a corresponding bin; an average distance value is calculated based on the distance values ​​associated with the bin having the highest number of associated distance values;the final distance value is modeled using a probabilistic distribution based on the average distance value and is set as the output value. ;

[0038] In other words, a predefined distance range is defined, for example, a specific measuring range of the sensor, which can be defined by a minimum and a maximum distance value. The predefined distance range is divided into different bins. Each bin represents, for example, a sub-range within the predefined distance range and can be defined by a minimum and maximum distance value. Each determined distance value is then assigned to a bin by determining whether the determined distance value of the corresponding auxiliary transmission signal lies within the sub-range of the corresponding bin. Determined distance values ​​that cannot be assigned to a bin can, for example, be ignored.After the assignment of the determined distance values ​​is complete, the bin with the most assigned distance values ​​is selected, for example, and the distance values ​​assigned to this bin are averaged to obtain the average distance value. The final distance value can then be modeled using a probabilistic distribution based on the average distance value and output as the output value.

[0039] For example, the average distance value can be used as the expected value and the standard deviation of the probabilistic distribution can be determined based on the distance values ​​determined from the auxiliary transmit signals.

[0040] For example, an index i of the bin to which a calculated distance value d ray of an auxiliary transmit signal belongs is calculated as follows: i = d ray − d min d max − d min × k − 1 where i represents the index of the bin to which the calculated distance value d ray is assigned, d min and d max represent a minimum and maximum distance of the predefined distance range, and k represents the predefined number of bins. For example, the predefined distance range can be between 20 and 30 m, 40 and 50 m, 70 and 100 m, or 130 and 150 m. Furthermore, the number k of bins can be arbitrarily specified. For example, the number of bins can be 8, 16, 32, 64, 128, 256, 512, 1024, or 2048.

[0041] According to a further embodiment, determining the radial velocity of the point of impact of the virtual transmission signal on the virtual object with respect to the virtual sensor comprises: an impact point i of the virtual transmission signal on the virtual object at a time t is determined, whereby the position of the point of impact v hit (t)on the virtual object in the three-dimensional, virtual test environment; at a time t + Δt, the current position of the impact point v hit (t + Δ t) on the virtual object in the three-dimensional, virtual test environment; and at a time t + Δt based on the two consecutive positions of the impact point v hit (t) and v hit (t + Δ t) the three-dimensional velocity vector of the impact point is determined and the radial velocity of the impact point with respect to the sensor is determined by projection onto the direction of the transmitted virtual transmission signal at time t in the virtual test environment.

[0042] In particular, a grid or mesh is created to determine the point of impact, which may, for example, comprise a plurality of triangles, rectangles, and / or another geometric shape. In particular, the virtual object can be represented or simplified by a corresponding grid. In particular, the grid can be defined by the coordinates of the corner points of the grid. Furthermore, the position of the point of impact within the grid, in particular with reference to corner points of the grid, can be determined and stored.

[0043] In order to determine the speed of the impact point, the position of the impact point is determined at a time t and at a time t + Δt, so that a first position, in particular a first impact vector, and a second position, in particular a second impact vector, of the impact point are determined. Based on the determined first and second positions, a distance between the first and second positions can be determined, wherein the speed of the impact point of the virtual transmission signal on the virtual object in the virtual test environment can be determined based on the distance between the first and second positions and the time Δt required to cover the corresponding distance. Furthermore, the radial speed of the impact point with respect to the sensor can be determined based on the determined speed of the impact point and the calculated position of the impact point at time t.For example, the velocity vector of the impact point can be projected onto the direction of the first virtual transmit signal to determine the radial velocity of the impact point with respect to the sensor.

[0044] What is particularly advantageous here is that the radial velocity is determined by simulating a single virtual transmission signal, in particular a single one. The point of impact on the virtual object, and in particular within the grid, is noted or stored at time t, and the coordinates of the point of impact within the grid are determined or retrieved at time t + Δt. Consequently, a complex simulation of a second transmission signal, which is difficult and computationally intensive to implement due to the high required temporal resolution, can be dispensed with.

[0045] According to a further embodiment, the validity of the output value is determined based on a distance-dependent probability. For example, the expected value of a Bernoulli distribution is selected based on the distance, thus determining the validity of the determined distance value and / or the determined radial velocity.

[0046] Furthermore, distance values ​​within a validity range, i.e., a valid distance range [r min , r max ], can be defined as valid values, while distance values ​​outside the validity range are rejected as invalid values ​​or ignored. The validity range can, for example, be defined sensor-specifically and / or based on the properties attributed to the virtual objects.

[0047] According to a further embodiment, physical sensor data and / or physical position data from a real sensor are evaluated to create the virtual test environment. Information about the sensor and / or its installation location, for example, measured on a real sensor or obtained from data sheets or similar sources, can be incorporated into the virtual test environment. This creates the possibility of simulating a real situation in the virtual test environment.

[0048] According to a further embodiment, information obtained in the virtual test environment, in particular the output values, is used to configure a real sensor or the real sensor for real-world operation. For this purpose, the real sensor can be coupled to a simulation device, for example, via a data connection, in order to transfer settings or parameters to the real sensor.

[0049] For example, the virtual test environment may reveal that certain sensor settings are advantageous for the simulated application. These settings can include a specific scan pattern (i.e., the pattern of transmission signals), advantageous scan frequencies (i.e., the frequency of transmission signals), the transmission power, but also the precise positioning and / or orientation of the sensor, and so on.

[0050] For example, the virtual sensor is placed in the virtual test environment, and after the simulation, a real sensor is adjusted based on data from the virtual sensor. Preferably, by simulating the virtual sensor in the virtual test environment, different positions of the virtual sensor can be tested and an advantageous positioning of the virtual sensor can be determined. The real sensor can thus be positioned according to the position of the virtual sensor and initialized or calibrated using the position data from the virtual sensor.

[0051] According to a further embodiment, annotation data, for example metadata or data on objects in the virtual test environment, is generated based on data from the virtual test environment and / or data from the virtual sensor. The annotation data can, for example, also include 3D bounding boxes for objects within point clouds. A particularly advantageous feature here is that generating the annotation data does not involve additional computational effort, since corresponding annotation data is already provided by the test environment. The annotation data can, for example, indicate whether an object struck by the virtual transmission signal is a person, a vehicle, the ground, or a building.

[0052] According to a further embodiment, AI models for mobile robots and / or for industrial trucks are trained using the data generated by the simulation model and / or the annotation data. For example, the AI ​​models can be trained online, i.e., using data generated in real time, and / or offline, i.e., using existing or previously generated data. When training an AI model, it can be particularly advantageous to have annotation data available, because then it can also be verified whether the AI ​​model correctly recognizes a human, for example.

[0053] According to a further embodiment, a plurality of virtual transmission signals transmitted up to a predefined measurement time are processed together. This creates a snapshot of a current scene in the virtual test environment.

[0054] According to a further embodiment, a plurality of virtual transmission signals transmitted up to a predefined measurement time are divided into a predefined number of subsets, each corresponding to an equal time duration, wherein the virtual transmission signals belonging to a subset are processed together. Thus, a current scene of the virtual test environment is scanned.

[0055] Another subject of the invention is a system for modeling a sensor, in particular an FMCW LiDAR sensor, in a virtual test environment, in particular for testing software for mobile robots and / or for industrial trucks, comprising a computer which is designed to: to define a simulation model, wherein the simulation model comprises a virtual sensor and the virtual test environment, to simulate a virtual transmission signal emitted by the virtual sensor in the virtual test environment; to determine whether the virtual transmission signal hits a virtual object in the virtual test environment at an impact point; if the determination is positive, to calculate a distance traveled by the virtual transmission signal in the virtual test environment from the virtual sensor to the impact point on the virtual object; and to determine at least one output value of the virtual sensor based on the calculated distance, wherein the determination of the output value is further based on at least one parameter of the simulation model, wherein the parameter and / or the output value are modelled by a probabilistic distribution, wherein the Parameter comprises an azimuth angle of the virtual transmission signal and / or an elevation angle of the virtual transmission signal, wherein the virtual transmission signal is simulated, in particular only, based on the azimuth angle, the elevation angle and a starting point of the virtual transmission signal.

[0056] The computer is also designed to: to model the azimuth angle and elevation angle of the virtual transmission signal using a probabilistic distribution with a predefined reference azimuth angle and a predefined reference elevation angle as the expected value; to select a plurality of different virtual auxiliary transmission signals, whose azimuth angle and elevation angle differ from the reference azimuth angle and reference elevation angle, based on the probabilistic distribution of the virtual transmission signal; to determine for each virtual auxiliary transmission signal whether the virtual auxiliary transmission signal hits the virtual object at the point of impact and / or at least one other virtual object at at least one other point of impact; if the determination is positive, to calculate a distance traveled by the virtual auxiliary transmission signal from the virtual sensor to the point of impact in the virtual test environment;and to determine a final distance value based on the multitude of calculated distance values. ;

[0057] In particular, the system can be used to monitor vehicles and pedestrians, for example in urban areas, detect containers, detect production environments, in particular (mobile) robots, driverless transport vehicles, factory automation or logistics.

[0058] According to a further embodiment, ray tracing-specific hardware is used to simulate the virtual transmission signals (e.g., a graphics card) and / or the virtual test environment is simulated using a game engine. This can advantageously accelerate the simulation.

[0059] The statements regarding the method according to the invention apply accordingly to the system, in particular with regard to advantages and embodiments.

[0060] Furthermore, unless otherwise stated, any combination of the above embodiments is possible.

[0061] The invention is illustrated below purely by way of example with reference to the drawings. They show: Fig. 1 shows a virtual test environment; Fig. 2 shows a method for modeling a sensor in the virtual test environment; Fig. 3a shows a set of virtual light beams comprising a reference light beam and a plurality of auxiliary light beams; Fig. 3b shows a diagram of the azimuth and elevation angle noise for a standard deviation of σ azimuth = 0.326° and σ elevation = 0.275° with 5000 data points; Fig. 3c shows a diagram of the azimuth and elevation angle noise for a standard deviation of σ azimuth = 0.326° and σ elevation = 0.275° with 100 data points; Fig. 4a shows a 2-dimensional view of a virtual light beam cone in the virtual test environment with 2 virtual objects; Fig. 4b shows a histogram of determined distance values ​​in a predefined distance range; Fig. 5 shows a movement of a virtual object in the virtual test environment; Fig. 6 an illustration of two data processing variants.

[0062] Fig. 1 shows an exemplary virtual test environment 4, ie a virtual 3-dimensional world, which is created by means of a virtual sensor (not shown), which is part of the virtual test environment, according to the Fig. 2 shown method, wherein the virtual test environment 4 comprises a plurality of virtual objects such as virtual vehicles 6, virtual road infrastructure, road signs 8, road markings 10, pedestrian walkways, and virtual buildings 12. In principle, the virtual test environment 4 can comprise all possible objects, in particular objects other than those shown here, for example people, animals, etc. The test environment in particular replicates a realistic driving scenario of the real world and can be adapted as desired. The test environment is simulated on a computer (not shown) of a simulation facility.

[0063] Fig. 2 shows a method 2 for modeling a sensor in the virtual test environment 4, wherein in the method 2 a simulation model is defined, wherein the simulation model comprises a virtual sensor, in particular a virtual distance sensor, and the virtual test environment 4. The virtual distance sensor is arranged in particular in the virtual test environment 4. The virtual test environment 4 is a virtual 3-dimensional world that has a virtual road infrastructure, ie virtual roads, road signs 8, road markings 10, etc., as well as virtual objects such as virtual people, animals, vehicles, buildings, etc.

[0064] The method begins in method step 100 by sending a virtual transmission signal in the form of a virtual light beam from the virtual distance sensor into the virtual test environment 4, in which a virtual object is located. The virtual light beam has a predefined reference azimuth angle and reference elevation angle, as well as an output point 14, which is determined, for example, by the position of the virtual sensor in the virtual test environment. Since the beam direction of a transmitted light beam in the real world can deviate from the intended beam direction due to interference factors, the azimuth angle and elevation angle of the virtual light beam are modeled with a normal distribution, which simulates the influence of such interference factors (method step 110). In other words, the beam direction of the virtual light beam is provided with a noise component.The reference azimuth angle and the reference elevation angle of the virtual light beam serve as the expected value of the normal distribution, with the standard deviation being predefined by the sensor type.

[0065] Based on the normal distribution of the azimuth angle and the elevation angle, in method step 120 a number n, in this case 100, of virtual auxiliary light beams 18 (see Fig. 3a ) which originate from the same starting point 14 (see Fig. 4a ) and each have an azimuth angle and / or elevation angle, which follows the statistics of a beam profile. Using a normal distribution, a Gaussian beam can be simulated. Using other probability distributions of the auxiliary light beams, any beam profiles can be modeled. For each virtual auxiliary light beam 18, a distance value is then calculated in method step 130, which indicates a corresponding distance to the virtual object or an impact point on the virtual object. Consequently, 100 distance values ​​are calculated in this way. The distance values ​​are calculated using ray tracing, i.e., by tracking a virtual auxiliary light beam 18 along its beam direction, wherein it is determined whether the virtual auxiliary light beam 18 impacts a virtual object along its beam direction. This is described in Fig. 4a presented in more detail.

[0066] If the determination is positive, the point of impact is calculated based on the available information from the test environment, e.g., the Cartesian spatial coordinates. For example, the intersection point of the virtual light beam, e.g., the half-line defined by the starting point 14 and a respective azimuth and elevation angle, with the virtual object, e.g., the coordinates of the virtual object, and in particular, with the point of impact on the virtual object, is calculated.

[0067] After calculating the distance values ​​associated with the auxiliary light beams 18, a number of x bins is initialized in a predefined distance range (method step 140), ie the predefined distance range is divided into x equidistant sub-ranges.

[0068] Subsequently, in method step 150, each of the 100 determined distance values ​​is assigned to a corresponding bin in whose partial distance range the determined distance value lies. Determined distance values ​​that cannot be assigned to a bin are ignored in the further calculation. This results in a histogram of the determined distance values, which indicates a distribution of the determined distance values. Once all 100 determined distance values ​​have been processed into a corresponding bin, the bin with the highest number of assigned distance values ​​is determined in method step 160, and an average distance value of all determined distance values ​​assigned to this bin is calculated (method step 170). This average distance value thus indicates the most probable distance value and is used as the expected value to determine a normally distributed final distance value (method step 180).

[0069] The standard deviation of this normal distribution is predetermined or calculated based on the previously determined distance values.

[0070] Furthermore, in method step 190, the validity of the average distance value is determined using a Bernoulli distribution.

[0071] If the validity check shows that the average distance value is valid, the final normally distributed distance value is output (process step 200). In all other cases, no output value is output.

[0072] Fig. 3a illustrates a reference light beam 16 emanating from an output point 14 and a plurality of auxiliary light beams 18 emanating from the output point 14 with different azimuth and / or elevation angles, wherein the different light beams 16, 18 are represented as arrows and wherein the azimuth and / or elevation angles of the different auxiliary light beams 18 follow a normal distribution. Fig. 3a The auxiliary light beams 18 shown have been generated by adding the azimuth angle of the reference light beam 16 and the elevation angle of the reference light beam 16 with random values ​​of a normally distributed noise function.

[0073] Fig. 3b shows a diagram of the azimuth and elevation angle noise for a respective standard deviation of σ azimuth = 0.326° and σ elevation = 0.275° with 5000 data points. The azimuth and elevation angle noise is added to the azimuth and elevation angle of the reference light beam 16, for example, to generate the plurality of auxiliary light beams 18.

[0074] Fig. 3c shows a diagram of the azimuth and elevation angle noise from Fig. 3b , where 100 data points were randomly selected, which are used to calculate the auxiliary light beams 18.

[0075] Fig. 4a shows a 2-dimensional representation of a virtual light beam cone 22 in the virtual test environment 4 with a first virtual object 24 and a second virtual object 26, wherein a first distance r 1 = 29.5 m lies between the first virtual object 24 and the starting point 14 and a distance r 2 = 29.65 m lies between the second virtual object 26 and the starting point 14. The light beam cone 22 comprises the reference light beam 16 and the 100 selected auxiliary light beams 18, which each have different azimuth and / or elevation angles and emanate from the same starting point 14. A width of the light beam cone 22 defined in this way is determined by the statistics of the beam profile, ie by the azimuth angle and / or elevation angle of the auxiliary light beams.

[0076] According to the Fig. 2 The procedure described in Fig. 4b The histogram shown was determined with a bin size of 7.8 cm. Fig. 4b It can be seen that two bins with an increased number of determined distance values ​​were determined, which clearly correspond to impact points on the first or second virtual object 24, 26. Based on the Fig. 2 However, in the method described above, only the bin with the largest number of determined distance values ​​is considered for the further calculation of the average distance value. In this case, the average distance value is 29.49 m, which serves as the expected value for the normal distribution. Additionally or alternatively, if the histogram is found to have two or more maxima, a warning signal may be issued, indicating that no clear distance value could be determined. In particular, this can be used to detect a multi-echo situation and / or an edge hit.

[0077] Fig. 5 illustrates the method for calculating the radial velocity of an impact point of the virtual transmission signal on the virtual object with reference to the virtual sensor or the starting point 4 of the virtual transmission signal. For this purpose, at a time t = t 0 , an impact point of a first virtual light beam is determined within a virtual object 30 defined by a grid, wherein the exact position of the impact point is determined with reference to the corners V 1 , V 2 and V 3 of the grid. Based on the coordinates of the impact point in the virtual test environment and the starting point, a position vector v hit s t 0 determined.

[0078] At a time t = t 0 + Δt the current position of the impact point relative to the virtual sensor v hit s t 0 + Δt determined based on the information of the positioning of the impact point within the grid.

[0079] Based on the two position vectors, the velocity vector v of the point of impact of the virtual transmission signal on the virtual object 30 is determined. For example, the velocity vector v is determined by the following equation: v = v hit s t 0 + Δt − v hit s t 0 Δt

[0080] The radial velocity is then determined by projecting the velocity vector v onto the position vector v hit s t 0 determined as in Fig. 5 shown.

[0081] Fig. 6 illustrates two different ways to process the data generated by the simulation model. In Fig. 6a A large number of virtual light rays captured up to a measurement time t M are processed simultaneously. This is also referred to as snapshot processing.

[0082] Fig. 6b , however, illustrates a possibility in which a plurality of virtual light rays captured up to the measurement time t M are divided into a predefined number of subsets 28, each corresponding to an equal time duration, with the virtual light rays belonging to a subset 28 being processed together. In this case, one speaks of scanning. Bezugszeichenliste

[0083] 2Method for modeling a sensor 4Virtual test environment 6Virtual vehicles 8Virtual road signs 10Virtual road markings 12Virtual buildings 14Starting point 16Reference light beam 18Auxiliary light beams 22Light beam cone 24First virtual object 26Second virtual object 28Subsets 30Virtual object 100-200 process steps

Claims

1. A computer-implemented method (2) for modeling a sensor, in particular an FMCW LiDAR sensor for distance measurement, in a virtual test environment (4), wherein: a simulation model is defined, wherein the simulation model comprises a virtual sensor and the virtual test environment (4), wherein in the method (2): a virtual transmission signal sent by the virtual sensor is simulated in the virtual test environment (4); it is determined whether the virtual transmission signal impacts a virtual object (30) at a point of impact in the virtual test environment (4); in the event of a positive determination, a distance of the virtual transmission signal covered by the virtual sensor up to the point of impact is calculated in the virtual test environment (4); and at least one output value of the virtual sensor is determined based on the calculated distance, wherein the determination of the output value further takes place based on at least one parameter of the simulation model, wherein the parameter and / or the output value is / are modeled by a probabilistic distribution, wherein the parameter comprises an azimuth angle of the virtual transmission signal and / or an elevation angle of the virtual transmission signal, wherein the virtual transmission signal is simulated, in particular only, based on the azimuth angle, the elevation angle and a starting point (14) of the virtual transmission signal, wherein the method (2) further comprises that: the azimuth angle and elevation angle of the virtual transmission signal are modeled as an expected value by a probabilistic distribution with a predefined reference azimuth angle and a predefined reference elevation angle; a plurality of different virtual auxiliary transmission signals (18), whose azimuth angle and elevation angle differ from the reference azimuth angle and reference elevation angle, are selected based on the probabilistic distribution of the virtual transmission signal; for each virtual auxiliary transmission signal (18), it is determined whether the virtual auxiliary transmission signal impacts the virtual object (30) at the point of impact and / or impacts at least one further virtual object at at least one further point of impact; in the event of a positive determination, a distance of the virtual auxiliary transmission signal covered by the virtual sensor up to the point of impact is calculated in the virtual test environment (4); and a final distance value is determined based on the plurality of calculated distance values.

2. A method (2) according to claim 1, wherein noise is generated in the output value by the probabilistic distribution.

3. A method (2) according to one of the preceding claims, wherein the probabilistic distribution comprises a normal distribution.

4. A method (2) according to claim 1, wherein the final distance value is determined in that: a predefined number of bins of the same size is initialized for a predefined distance range, with each bin being assigned to a different subrange within the predefined distance range; for each auxiliary transmission signal (18), it is determined whether the corresponding virtual transmission signal has a respective point of impact with the virtual object (30) and / or the at least one further virtual object in the virtual test environment (4), in particular in the predefined distance range, and in the event of a positive determination, an associated distance value is determined from the starting point (14) to the respective point of impact, wherein each distance value is assigned to a corresponding bin; an average distance value is calculated based on the distance values that are associated with the bin which has the highest number of assigned distance values; the final distance value is modeled by means of a probabilistic distribution based on the average distance value and is defined as the output value.

5. A method (2) according to any one of the preceding claims, wherein the determination of a radial speed of a point of impact i with respect to the virtual sensor comprises that: the point of impact i of the virtual transmission signal on the virtual object (30) is determined at a point in time t, wherein the position of the point of impact vhit(t) on the virtual object (30) is determined in the three-dimensional virtual test environment (4); at a point in time t + Δt, the current position of the point of impact vhit(t + Δt) on the virtual object (30) is determined in the three-dimensional virtual test environment (4); and at a point in time t + Δt, the three-dimensional velocity vector of the point of impact i is determined based on the two temporally consecutive positions of the point of impact vhit(t) and vhit(t + Δt) and, by projection in the direction of the transmitted virtual transmission signal, the radial speed of the point of impact i is determined with respect to the virtual sensor in the virtual test environment (4) at the point in time t.

6. A method (2) according to any one of the preceding claims, wherein a validity of the output value is determined based on a distance-dependent probability of recognizing an object (30).

7. A method (2) according to any one of the preceding claims, wherein physical sensor data and / or physical position data of a real sensor are evaluated to create the virtual test environment (4).

8. A method (2) according to any one of the preceding claims, wherein information obtained in the virtual test environment (4), in particular the output values, is used to configure a real sensor for real operation.

9. A method (2) according to any one of the preceding claims, wherein annotation data, for example metadata or data on virtual objects (30) in the virtual test environment (4), are generated based on data of the virtual test environment (4) and / or data of the virtual sensor.

10. A method (2) according to claim 9, wherein AI models for mobile robots and / or for fork-lift trucks are trained by means of the data generated by the simulation model and / or the annotation data.

11. A method (2) according to any one of the preceding claims, wherein a plurality of virtual transmission signals acquired up to a predefined measurement time are processed simultaneously.

12. A method (2) according to any one of the preceding claims, wherein a plurality of virtual transmission signals acquired up to a predefined measurement time are divided into a predefined number of subsets (28) which each correspond to an equal time duration, wherein the virtual transmission signals belonging to a subset (28) are processed simultaneously.

13. A system for modeling a sensor, in particular an FMCW LiDAR sensor, in a virtual test environment (4), said system comprising: a computer which is configured: to define a simulation model, wherein the simulation model comprises a virtual sensor and the virtual test environment (4); to simulate a virtual transmission signal sent by the virtual sensor in the virtual test environment (4); to determine whether the virtual transmission signal impacts a virtual object (30) at a point of impact in the virtual test environment (4); in the event of a positive determination, to calculate a distance of the virtual transmission signal covered by the virtual sensor up to the point of impact in the virtual test environment (4); and to determine at least one output value of the virtual sensor based on the calculated distance, wherein the determination of the output value further takes place based on at least one parameter of the simulation model, wherein the parameter and / or the output value is / are modeled by a probabilistic distribution, wherein the parameter comprises an azimuth angle of the virtual transmission signal and / or an elevation angle of the virtual transmission signal, wherein the virtual transmission signal is simulated, in particular only, based on the azimuth angle, the elevation angle and a starting point (14) of the virtual transmission signal, wherein the computer is further configured: to model the azimuth angle and elevation angle of the virtual transmission signal as an expected value by a probabilistic distribution with a predefined reference azimuth angle and a predefined reference elevation angle; to select a plurality of different virtual auxiliary transmission signals (18), whose azimuth angle and elevation angle differ from the reference azimuth angle and reference elevation angle, based on the probabilistic distribution of the virtual transmission signal; for each virtual auxiliary transmission signal (18), to determine whether the virtual auxiliary transmission signal impacts the virtual object (30) at the point of impact and / or impacts at least one further virtual object at at least one further point of impact; in the event of a positive determination, to calculate a distance of the virtual auxiliary transmission signal covered by the virtual sensor up to the point of impact in the virtual test environment (4); and to determine a final distance value based on the plurality of calculated distance values.