System, method, computer program and computer-readable medium for generating annotated data

Abstract

The present invention relates to a system for detecting, classifying and / or segmenting an object and / or a property of an object with a simulation unit and an artificial intelligence unit, wherein the simulation unit is designed to generate simulated measurement data from a simulated sensor unit relating to a virtual object and / or a property of a virtual object in a virtual environment and to annotate the virtual object, the property of the virtual object and / or the virtual environment and to generate simulation data therefrom, wherein the artificial intelligence unit is designed, on the basis of the simulated measurement data and the simulation data, to detect, classify and / or segment an object and / or a property of an object on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement.

Description

[0001] System, method, computer program and computer-readable medium for generating annotated data

[0002] The present invention relates to a system for detecting, classifying and / or segmenting an object and / or a property of an object with a simulation unit and an artificial intelligence unit, wherein the simulation unit is designed to generate simulated measurement data of a simulated sensor unit about a virtual object and / or a property of a virtual object in a virtual environment and to annotate the virtual object, the property of the virtual object and / or the virtual environment and to generate simulation data therefrom.

[0003] For the simulation of radar or radio signals in general, a number of different approaches based on ray tracing algorithms are known from the state of the art.

[0004] A widely used approach, especially for simulating radar data in the automotive environment, is the Shooting and Bouncing Rays (SBR) approach. This is illustrated in Fig. 8. A beam TR emanating from a transmitting antenna T is reflected at the facets F and received by the receiving antenna R. A computationally efficient variant for simulating radar data is to describe the object using so-called scattering targets. These scattering target models can be created through measurements or through a more complex SBR simulation. A significant disadvantage of this technique is that scattering centers must be calculated or measured individually for each object, and effects such as occlusion or multiple reflections cannot be simulated, or can only be simulated to a very limited extent. This makes a realistic simulation difficult overall.

[0005] The increased use of neural networks has also led to the development of data-driven approaches. In this approach, a statistical model or an artificial neural network learns to artificially generate new, unknown radar data from measured radar data. The main disadvantage of this approach is that it can only generate data from existing sensors with existing measurement data. Thus, unlike ray tracing approaches, flexible or random sensor configuration is not readily possible.

[0006] Furthermore, the quality and realism of the data generated by the latter two methods is still noticeably worse compared to a largely physically correct SBR simulation.

[0007] Artificial intelligence is currently being used in the field of object detection and classification in automotive radar applications. The application spectrum ranges from simple object detection to object classification, which often takes place using the Doppler spectrum.

[0008] Machine learning algorithms, i.e. artificial intelligence, are also used to improve angular resolution.

[0009] Artificial intelligence is also used for image enhancement in radar imaging and human movement classification. One of the biggest challenges in using artificial intelligence for radar signals is the generation of so-called ground truth. This requires annotating data, whereby as many individual areas and / or parts of the radar data as possible are assigned labels as precisely as possible, such as "pedestrian," "cyclist," or "car," etc.

[0010] The simplest approach to generating ground truth is manual annotation, but this represents a considerable effort, especially with large data sets.

[0011] Unlike radar data, manual annotation of natural images, such as photographs, is laborious but, with sufficient personnel, in principle possible. This is not always the case with radar data. Radar data can only be effectively classified with expert knowledge, as it differs significantly from natural images and is therefore difficult for humans to interpret. However, even with expert knowledge, not all radar image effects can be correctly classified.

[0012] Compared to optical images based on visible light, this is mainly due to the different reflection behavior of electromagnetic waves in the frequency range of conventional radar systems, the lower (angular) resolution and the different data processing.

[0013] For example, even side lobes in the frequency spectrum of a target reflection still belong to the actual target, even though they are locally distant. Multiple reflections can also cause a single object to represent multiple target detections in the radar image that are spatially distant from each other.

[0014] Another way to generate ground-truth data is to annotate it automatically. This type of learning is often referred to as self-supervised learning. Semi-automatic annotation of processed radar point clouds using a lidar sensor and / or a camera sensor is known.

[0015] GPS or odometry sensors can also be used as a reference, e.g. for lane estimation or road course estimation.

[0016] Furthermore, simulations are already being used to create ground truth data for radar applications.

[0017] However, simulations are used only to a very limited extent for the classification of individual objects. Comprehensive automatic segmentation of complex simulation scenarios based on raw radar data, which is essential for applications in the automotive radar environment, is unknown.

[0018] One reason for this is not only that simulating complex worlds is very complex and computationally intensive. A fundamental problem is also that even in simulations, the 3D environment must be correctly translated into the simulation data. This means that every object, or even every reflection in the radar signal, must be correctly annotated or labeled.

[0019] Against this background, the present invention is based on the object of improving a system described above, in particular with regard to the annotation of objects.

[0020] This object is achieved by the method having the features of independent claim 1. Advantageous developments of the invention are the subject of the dependent claims.

[0021] Accordingly, the invention provides that the artificial intelligence unit is designed to detect, classify and / or segment an object and / or a property of an object on the basis of the simulated measurement data and the simulation data, on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement.

[0022] The system is preferably designed for annotation and subsequent optimal training of the artificial intelligence unit. Annotation preferably takes place in the simulation unit.

[0023] Preferably, it is possible but not required to fully annotate virtual objects in a virtual environment or virtual worlds.

[0024] A property of a virtual object is preferably any type of information associated with the object, such as a vector velocity, preferably of each pixel or ray hitting the object, noise and / or side lobes emanating from the object, temperature, material property, color, etc.

[0025] Annotation preferably refers to the process of giving an object or a pixel or an image area a label or a designation, which can be interpreted by an artificial intelligence unit, particularly in a training process.

[0026] The artificial intelligence unit is preferably designed to recognize common features of objects, pixels or image areas.

[0027] Detection preferably refers to the process of recognizing an object and / or a property of an object. For example, detection can determine whether an object is a person or a car.

[0028] Classification preferably refers to the process of assigning an object and / or a property of an object to a class. This can specify a probability with which an object can be assigned to a class. For example, through classification, an object can be assigned to the class "human" with a probability of 96%.

[0029] Segmentation preferably refers to the process of dividing an object into segments, such as pixels, where each segment is detected and / or classified.

[0030] Preferably, it is provided that the simulation unit is designed to simulate more than one virtual object with more than one property and / or is designed to completely annotate the virtual object and / or the property of the virtual object or the virtual objects and / or the properties of the virtual objects and / or the virtual environment.

[0031] Preferably, the system further comprises means by which the virtual object and / or the virtual environment are generated using 3D information generated from a game engine or based on map information, aerial photographs, earth remote sensing information and / or measurement data from a camera system and / or a laser scanner or other sensor systems.

[0032] It is preferably provided that the simulated sensor unit is modeled on a real sensor unit and / or that the simulated sensor unit is improved compared to a real sensor unit, in particular with regard to resolution, signal-to-noise ratio and / or unambiguousness ranges.

[0033] It is conceivable that the simulated sensor unit simulates a radar, a camera, a lidar, and / or an ultrasound device. Preferably, the simulated, other simulated, or real-measurement data is image data and / or the artificial intelligence unit is configured to enhance raw and processed sensor signals in the measurement data.

[0034] It is conceivable that the environment is a dynamic, three-dimensional environment.

[0035] Preferably, it is provided that the system or the artificial intelligence unit is part of a vehicle and / or that the measurement data generated from the real measurement was generated by a radar, a camera, a lidar and / or an ultrasound device.

[0036] In other words, image recognition can be achieved through annotation.

[0037] In other words, preferably at least one object, preferably a dynamic 3D scene that is as diverse as possible, is simulated, for example in a game engine or on the basis of map information or measurement data from camera systems and / or laser scanners or other sensors of a real scene or a real setup.

[0038] In this object and / or world simulation, radars are preferably subsequently simulated. Based on their position in space and their simulated antennas, they transmit radar signals into this world using a ray tracing approach. They receive and evaluate the reflections, allowing digital radar ADC data to be simulated. The simulated world and the simulated signal are preferably as realistic as possible so that the simulated signal is as similar as possible to a real signal.

[0039] Since the simulated world is known, the labels or annotations for the radar data can be generated directly and comprehensively. This radar data, including the labels, can be used to train an artificial intelligence (AI), such as a neural network, for example, for object detection, object classification, segmentation, or image enhancement.

[0040] Preferably, the simulated object and / or worlds are simulated once with and once without interference. This allows neural networks to be trained that largely suppress these interference effects in reality.

[0041] Preferably, the real-world radar is simulated as a "digital twin" and a significantly improved "advanced digital twin" is simulated. This can, for example, have a significantly larger or more fully populated antenna array to simulate radar images with better angular resolution. If a neural network or a CL is then trained with the simulation data from the "digital twin" and the "advanced digital twin," the CL can improve the real radar measurement data, particularly with regard to resolution, signal-to-noise ratio, unambiguousness ranges, etc.

[0042] Preferably, each ray of the ray tracing approach receives the object ID of the object from which it was reflected as an additional attribute upon reflection. This makes it possible to fully assign each partial echo of an object to that object. This attribute thus preferably corresponds to an optimal label or annotation. This allows for optimal training of AI applications, for example, for object classification, because the network learns to utilize all information from the entire (frequency) spectrum.

[0043] Ideally, the simulated worlds should reflect reality as closely as possible by using existing datasets from remote sensing, map services, or aerial photographs. It is conceivable that the simulated worlds could be generated from real measurement data. For example, data from a measurement run with high-resolution laser scanners could be accumulated in such a way that the real world can be recreated in the form of triangular meshes (or similar).

[0044] It is conceivable that lidar, camera or ultrasound signals could be simulated instead of the radar signal.

[0045] Preferably, efficient and automated learning or training of artificial intelligence for the interpretation or improvement of radar and other sensor systems is enabled.

[0046] Preferably, any objects or more comprehensive scenes or worlds are specifically simulated in a simulation environment. Preferably, radar ADC data or other sensor raw data that are as realistic as possible are then generated in this simulation environment using ray tracing approaches. Preferably, each simulated ray is assigned one or more attributes that describe and identify all objects from which this ray has been reflected or with which it has interacted.

[0047] For example, the movement of an object can be directly captured in maximum detail in the form of a velocity vector for each reflection point or each ray. For example, pedestrians do not have a single velocity, as each point of the body moves individually. Their arms swing, one leg remains stationary, the torso moves relatively constantly, etc. It is precisely this micro-Doppler signature—the interaction of all velocities—that can be well evaluated by a radar and can be annotated or labeled completely accurately.

[0048] Equivalently, we can also pass additional object properties such as temperature (not relevant for radar), material properties, color (not relevant for radar), and information about the transmission channel (rain, fog, etc.), thereby achieving advantages for AI units. Since each object in the raw signal can thus be retroactively and completely identified, the simulated data can then be used directly as automatically annotated, high-quality ground truth data for teaching or training artificial intelligence. By using a simulation environment, it is preferably possible to generate any number of fully annotated training data sets in a short time.

[0049] With the learned or trained artificial intelligence, the sensor data can be interpreted, classified, segmented, or otherwise described. On the other hand, by adapting the simulation environment and the simulated sensors, radar data with improved properties, such as resolution and unambiguousness ranges, as well as with drastically reduced interference effects, can be generated. This allows the parameters of machine learning algorithms to be trained automatically, i.e., self-supervised, in such a way that they significantly improve the output data of real sensors, for example, through super-resolution and noise suppression.

[0050] The invention also relates to a method for detecting, classifying and / or segmenting an object and / or a property of an object using a system according to one of the preceding claims, comprising the following steps: a) generating simulated measurement data from a simulated sensor unit about a virtual object and / or a property of a virtual object in a virtual environment; b) annotating the virtual object, the property of the virtual object and / or the virtual environment and generating simulation data, wherein, on the basis of the simulated measurement data and the simulation data, an object and / or a property of an object is detected, classified and / or segmented on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement.The invention also relates to a method for training an artificial intelligence unit of a system according to one of claims 1 to 8, wherein the artificial intelligence unit is trained on the basis of simulated measurement data and the simulation data to detect, classify and / or segment an object and / or a property of an object on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement.

[0051] An artificial intelligence unit is preferably trained based on simulated data. The artificial intelligence unit can then interpret or improve, preferably, real or simulated measurement data.

[0052] The virtual object and / or the virtual environment can be modeled on a real object and / or a real environment.

[0053] The method preferably involves a complete annotation of the virtual object, the properties of the virtual object, and / or the virtual environment. This is done, for example, using object IDs. Every reflection along the path of a ray or every piece of motion information is preferably annotated. For example, a ray is first reflected off a static house wall and then off a pedestrian's arm swinging at 3 m / s. This information is retained in the annotation.

[0054] The features described herein are, mutatis mutandis, preferably features of the system as well as the methods.

[0055] The invention also relates to a computer program which comprises instructions which cause the system according to the invention to carry out the method steps of a method according to the invention.

[0056] The invention also relates to a computer-readable medium on which the computer program according to the invention is stored. It should be noted here that the terms "a" and "an" do not necessarily refer to exactly one of the elements, although this represents a possible embodiment, but can also refer to a plurality of the elements. Likewise, the use of the plural also includes the presence of the element in question in the singular, and conversely, the singular also encompasses several of the elements in question. Furthermore, all features of the invention described herein can be combined with one another as desired or claimed in isolation from one another.

[0057] Further advantages, features, and effects of the present invention will become apparent from the following description of preferred embodiments with reference to the figures, in which identical or similar components are designated by the same reference numerals. Herein:

[0058] Fig. 1 : a block diagram of an embodiment of a system according to the invention.

[0059] Fig. 2: a block diagram of an embodiment of a method according to the invention.

[0060] Fig. 3: a view of a real measurement scene (left) and a view of a three-dimensional replica of this measurement scene (right).

[0061] Fig. 4: a beam pattern from a known simulation method (left) and a beam pattern from an embodiment of a simulation method according to the invention (right).

[0062] Fig. 5: A view of a three-dimensional replica of a measurement scene (left), a view of a radar image based on it (center), and a view of an augmented radar image based on it (right). Fig. 6: A view of a real measurement scene (left), a view of a radar image based on it (center), and a view of an augmented radar image based on it (right).

[0063] Fig. 7: a view of an antenna arrangement.

[0064] Fig. 8: a schematic view of an antenna arrangement.

[0065] Fig. 1 shows a block diagram of a system according to the invention with a radar system 10, or a “radar system” 10, a digital twin 20, or a “digital twin” 20, an improved digital twin 30, or an “advanced digital twin” 30 and an artificial intelligence 50, or a “deep neural network (DNN)” 50.

[0066] The system outputs an enhanced radar image 40, or an “enhanced radar image”.

[0067] The system's units can be grouped into units in reality 100 and units in virtual space 200. The radar system 10 and the enhanced radar image 40 are located in reality 100, and the digital twin 20 and the enhanced digital twin 30 are located in virtual space 200. The artificial intelligence mediates between reality 100 and virtual space 200.

[0068] The radar system 10 supplies measurement data 15, for example in the form of a radar image, to the artificial intelligence 50 and parameters and information 12 via the antenna array of the radar system 10 to the digital twin 20. The digital twin 20 supplies training data 25 to the artificial intelligence 50 and improved parameters and information 23 via the antenna array of the radar system 10 to the improved digital twin 30. The improved digital twin supplies training data 35 to the artificial intelligence 50. The artificial intelligence 50 processes the measurement data 50, the training data 25 and 35, and outputs an enhanced radar image 40 as output 54. Preferably, advanced radar sensors without undesirable effects are simulated to improve real radar images.

[0069] The artificial intelligence unit can be an attention U-network trained on 9000 images for a regression task.

[0070] The distance resolution of the digital twin 20 and the improved digital twin 30 is preferably as small as possible, preferably less than or equal to that of a real sensor and is, for example, 7.5 cm. The lateral resolution of the digital twin 20 is preferably equal to that of a real sensor and is, for example, 5.3°. The lateral resolution of the improved digital twin 30 is preferably smaller than that of a real sensor and is, in particular, less than 1° and is, for example, 0.4°. The improved digital twin 30 has no or less clutter and noise compared to the digital twin 20. The reconstruction of the digital twin 20 is based on a fast Fourier transform (FFT). The reconstruction of the improved digital twin 30 is based on a suitable filter.

[0071] A Doppler simulation of a running person is conceivable.

[0072] Fig. 2 shows process steps of an exemplary process.

[0073] In a simulation environment, 3D maps can be exported from an open-source simulator, i.e., a simulation unit. This data consists of triangular meshes, which are commonly used in computer graphics, game development, and computer-aided design (CAD).

[0074] Similar to computer graphics, each triangle or object can be assigned a material. This material primarily determines the reflection properties of the radar signal. These triangle meshes can be exported not only from existing simulators, but 3D data from game engines or computer games can also be used directly.

[0075] Furthermore, realistic 3D data can be generated from map services using appropriate software. It is also possible to create an accurate image of the real environment using data from high-resolution lidar sensors or based on photogrammetry. With the appropriate effort, an enormous and theoretically unlimited amount of 3D environmental data can be generated.

[0076] Fig. 3 shows a simulation of a real measurement scene using a 3D graphics program. In Fig. 3, the real measurement scene 101 and a measurement scene 201 simulated by this measurement scene 101 can be seen in a simulation environment.

[0077] Moving objects can also be exported, for example through animations, or created using 3D graphics programs.

[0078] The simulator preferably supports diffuse and metal-like reflections, Doppler simulations and animations, occlusion and multipath effects, MIMO apertures with flexible radar parameters and / or any third-party 3D meshes.

[0079] Thus, a simulative generation of objects and / or scenes takes place with step S1.

[0080] In the simulation, the animation is then moved step by step or discretized for each radar measurement, e.g., for each chirp, and each snapshot is simulated individually. This is covered by step S2, simulation of raw sensor data. The simulator works very similarly to the well-known SBR principle, in which radar beams (rays) are emitted from predefined transmitting antennas, which in turn are reflected by the environment (triangular networks) until they hit a receiving antenna defined as a sphere. From the beams received in this way, an IF signal can be generated based on the beam length and echo energy or amplitude. The following equation (1) describes this using an FMCW signal as an example. This method can be transferred to OFDM, CW, pulse or other radars or modulation types. Equation (1 )

[0081] In contrast to the well-known SBR principle, which is usually based on only a single radar beam hitting a triangle and so-called double counts being avoided as much as possible, the simulator is based on a statistical approach that reflects rays in a statistically distributed manner. These material models are particularly established in ray tracing methods in computer graphics, as they are well suited to describing complex surface properties such as roughness and diffuse scattering. Since the signal propagation in automotive radars increasingly behaves like optical beams due to the high transmission frequencies of approximately 77 to approximately 300 GHz, effects such as diffraction can increasingly be neglected. In the simulator, these material models can therefore depict the environment very realistically.

[0082] Unlike the typical SBR approach, the simulator emits a large number of random beams to create highly realistic radar images. This is evident in Fig. 4, which shows a beam pattern 1 arranged according to a Fibonacci grid and a beam pattern 2 uniformly distributed over a spherical surface.

[0083] An example of a simulator can be found in the following publication: “C. Schüßler, M. Hoffmann, J. Bräunig, I. Ullmann, R. Ebelt and M. Vossiek, "A Realistic Radar Ray Tracing Simulator for Large MIMO-Arrays in Automotive Environments," in IEEE Journal of Microwaves, vol. 1 , no. 4, pp. 962-974, Oct. 2021 , doi: 10.1109 / JMW.2021 .3104722”.

[0084] The simulation of moving objects or the simulation of Doppler or micro-Doppler signatures can be achieved by first scanning the environment with an initial simulation and storing the corresponding beam hits in memory. In subsequent Doppler simulations, radar beams are then sent to exactly the same positions. If the beams were always sent randomly and thus to slightly different positions for each Doppler snapshot, this would lead to phase distortions in the Doppler spectrum, making meaningful use almost impossible.

[0085] This allows even moving targets or objects to be annotated particularly well. The annotation preferably already includes information about the object's movement.

[0086] In particular, it can also be used to directly generate annotations or labels for movement. This can be done in great detail, for example, in the case of a human.

[0087] Each reflection point preferably has its own, particularly vectorial, velocity information. This information is significantly improved over the information that an object is moving at a specific speed.

[0088] Preferably, a perfect annotation of the micro-Doppler signature is generated. This is not possible in the prior art and is particularly valuable for radar systems, since the micro-Doppler signature is typically used primarily in artificial intelligence applications for classification and / or segmentation. Fig. 5 shows a view of a three-dimensional replica of a measurement scene 201, a view of a radar image 215 based on it, and a view of an enhanced radar image 240 based on it.

[0089] Fig. 6 shows a view of a real measurement scene 101, a view of a radar image 115 based thereon and a view of an extended radar image 140 based thereon.

[0090] To avoid multiple simulations, only a single simulation can be performed, and the length of the radar beams can be subsequently adjusted. Since the geometry and movement of all objects are known in advance, only each radar beam is associated with each triangle and object and stored, with its length subsequently adjusted during IF signal generation. This approach has been implemented for radar simulations based on the image approach and is referred to there as dynamic ray tracing.

[0091] This procedure is illustrated in Fig. 7, which schematically shows an efficient simulation of multiple antennas. To avoid phase errors, the transmitted beams from all antennas and in each simulation step (chirp) can be calculated so that they each hit the triangle at the same point. This is shown in Fig. 7. The beam TF emitted by the transmitting antenna T hits the facet F at the same point as the beam TR1 emitted by the transmitting antenna T. Only beam TR1 is further simulated. For further optimization, it is sufficient to simulate only one antenna combination and calculate the beam length for all other antennas.

[0092] A similarly optimized approach can be applied to large antenna arrays. Instead of simulating all antenna combinations, only a single antenna can be simulated, and the beam length for all other combinations can be calculated. With appropriate computer hardware, any number of highly realistic radar simulations with any desired parameters can be generated.

[0093] To train artificial intelligence systems such as neural networks based on simulated data, the first step is to create a digital twin in the simulation environment. Antenna configuration, relevant hardware properties, and signal parameters are ideally adopted for the simulation in a realistic manner. In addition, influences from antenna characteristics or calibration artifacts can also be taken into account.

[0094] During the simulation, individual signal components or beams can be annotated directly, since all simulated objects are known or can be reproduced. Signals can then be generated that serve directly as ground truth.

[0095] It is known to use simulations for automatic annotation in object detection. For example, the results are annotated using bounding boxes to train a YOLO network.

[0096] However, this is not sufficient to classify complex measurement scenarios, as it does not take into account, for example, side lobes, noise and multiple reflections, which can only be inadequately described by simple bounding boxes and cannot yet be annotated manually or automatically.

[0097] The improved digital twin can improve noise reduction by annotating noise components, side lobes, etc.

[0098] Digital twins can be used not only for data classification and segmentation, but also to improve raw data signal processing overall. This includes, among other things:

[0099] - the elimination of unwanted multiple reflections; - the reduction of electronic noise;

[0100] - the artificial improvement of the antenna radiation characteristics;

[0101] - the elimination of ambiguities, e.g. in the angular or Doppler dimension;

[0102] - the reduction of calibration errors due to antenna crosstalk.

[0103] Signal enhancement can be achieved by not only simulating radar data sets that are as close as possible to the real sensor. Preferably, a sensor with a higher resolution, e.g., due to a larger antenna array, is also simulated, and whose simulated hardware and simulated world are free of the aforementioned artifacts and interference.

[0104] Higher resolution can be achieved, for example, by simulating significantly more antennas than in the original sensor. The number of antennas can be higher than would be possible with any real sensor. Interference caused by multipath propagation, such as multipath reflections, can be eliminated by limiting the number of possible beam reflections.

[0105] The other artifacts and disturbances described above can be eliminated in a similar way. The additionally simulated sensor can be referred to as an Advanced Digital Twin.

[0106] If the signals from both virtual sensors are available, any artificial intelligence can be trained to improve the data of the digital twin.

[0107] This is achieved by using the Advanced Digital Twin data as ground truth data. This not only increases the resolution and velocity, distance, and angle uniqueness of the input sensor, but also avoids, suppresses, or at least reduces any or all of the aforementioned artifacts.

[0108] If the radar simulations are sufficiently realistic, the learned or trained artificial intelligence can be applied directly to a real sensor. This can be found for some of the above-mentioned artifacts in the following publication: "C. Schüßler, M. Hoffmann, I. Ullmann, R. Ebelt and M. Vossiek, "Deep Learning Based Image Enhancement for Automotive Radar Trained With an Advanced Virtual Sensor," in IEEE Access, vol. 10, pp. 40419-40431, 2022, doi: 10.1109 / ACCESS.2022.3166227."

[0109] In the simulation, each received beam can be associated with its environment, where it is reflected repeatedly until it hits a receiving antenna. Each beam can carry arbitrary metadata, such as the beam travel time or beam energy.

[0110] Preferably, this metadata is expanded to include a list of object identifiers (object IDs), which indicate, for example, the specific object from which a beam was reflected, and preferably all information required for labeling or annotation, such as vector velocity, preferably of each pixel or beam, noise, temperature, material properties, and / or color. Thus, when the radar signals are generated, it is fully known which beam hit which object type and which entity. The data can thus be annotated in such a way that the labels can be used to distinguish not only between different object classes, such as pedestrians and cyclists, but also between individual objects within the same object class, such as individual pedestrians.

[0111] Since the simulated IF signal, as in equation (1) for FMCW signals, is created by adding up the signal component for each individual beam length, IF signals can be created for each individual object before the actual processing. These signals can then be automatically classified or annotated in the subsequent signal processing. This also means that every side lobe, every multiple reflection and every other associated signal component, such as the speed, can be clearly assigned to an object. Conventional problems in the correct assignment of signal components and objects, which arise, for example, due to occlusion, can be directly solved. In the example case of a pedestrian behind a wall, there would be no pedestrian in the beam data and the segmentation of a pedestrian would correctly be excluded from the ground truth data.Even much more complex scenarios can be solved in this way, for example if a pedestrian is walking or standing behind a car and can therefore only be detected by multiple reflections in the radar signal.

[0112] Problems of this kind cannot be solved using other reference sensors, such as lidar and camera systems, as these are subject to different beam physics or employ different data processing. Even simple annotation, in which the 3D geometry is directly superimposed on the radar images, is insufficient in this case, as not all signal components in the spectrum can be assigned, as spatial affiliation is not possible with side lobes and multipath propagation.

[0113] On the one hand, the annotation of the radar signal or spectral data can be carried out completely for the first time, since every single beam contributing to the radar signal contains all object information and can be used as a perfect label. On the other hand, the annotation can be implemented fully automated and computationally efficiently as part of the simulation process chain, thus eliminating the need for additional manual effort.

[0114] Preferably, a virtually unlimited amount of realistic, perfectly annotated data can be generated. This fulfills the three most important criteria for teaching or training artificial intelligence systems such as neural networks—data quality and quantity, as well as annotation quality—to an unprecedented degree. The resulting artificial intelligence systems can therefore significantly differentiate themselves from the current state of the art and resolve significantly more complex scenarios, for example, implementing significantly more complex segmentation, such as complete panoptic segmentation. The presented approach can be applied not only to radar sensors, but also to a large number of other sensor systems, such as camera, lidar, or ultrasound systems.

[0115] For camera data, superresolution algorithms are typically trained by reducing the resolution of an existing sensor and using the corresponding images as input data. The data at the original resolution is then used as ground truth. However, existing ray tracing simulations in computer graphics are already very sophisticated and realistic. Therefore, it makes sense to simulate camera images with higher resolutions than those of conventional cameras. This approach allows the presented principle of the Advanced Digital Twin to be directly transferred.

[0116] Lidar systems can also be simulated using ray tracing approaches. Ray tracing is very similar to simulating radar data. Since optical lidar beams have a shorter wavelength than radar beams, only the material properties need to be adjusted. Furthermore, multiple reflections play a minor role in lidar data. The resolution of a lidar system can be improved virtually, for example, by increasing the number of beams or the rotation rate and / or measurement rate of the system, thereby generating a denser point cloud. This data can also be used directly as ground truth data using the process described above.

[0117] Unlike radar and lidar systems, ultrasonic sensors do not operate with electromagnetic waves, but with acoustic sound. However, since the signals are also described as waves, the signal processing for ultrasonic data is very similar and often even identical. Ultrasonic signals can therefore be simulated in a similar way to radar signals. Depending on the wavelength, other effects such as diffraction or transmission must be considered, and the simulation environment must be adapted to the respective parameters. However, the approach presented can be adopted directly after adapting the simulation environment.

[0118] Any sensor fusion of different or identical sensor systems can also be implemented. This allows for the direct simulation of more comprehensive and complex scenarios, for example, to directly train applications for autonomous driving.

[0119] The operating principle of the simulation unit is preferably based on the SBR approach. The description of virtual objects and / or the virtual environment is preferably done using triangular meshes. The simulation unit is preferably accelerated using a modern ray tracing engine. Virtual objects and / or the virtual environment can be set up quickly and easily.

[0120] Diffuse and metallic reflections can be simulated, preferably on virtual objects and / or in the virtual environment. Doppler simulation and animations can also be performed. The simulation unit preferably supports the simulation of occlusions and / or multi-path effects. MIMO apertures can be simulated with flexible radar parameters. Any third-party networks are conceivable in the simulation.

[0121] Particularly relevant areas of application for the invention include:

[0122] - Automotive: Autonomous driving and driver assistance systems (including signal enhancement, extensive training, classification and segmentation of static and dynamic objects and road users);

[0123] - Smart home applications (including human and movement detection, fall and presence detection, energy saving techniques);

[0124] - Medical technology and medical applications (including vital sign monitoring, movement analysis (gait, injury, etc.), stress detection, palliative care monitoring, sleep analysis, general diagnostics and monitoring); - Traffic monitoring (including road, transit, and parking monitoring);

[0125] - Industrial Applications (including logistics or robotics applications);

[0126] - Industrial automation and 6G applications;

[0127] - Military technology (including autonomous robotics, drones, target recognition).

Claims

System, method, computer program and computer-readable medium for generating annotated data Patent claims .System for detecting, classifying and / or segmenting an object and / or a property of an object with a simulation unit and an artificial intelligence unit, wherein the simulation unit is designed to generate simulated measurement data of a simulated sensor unit about a virtual object and / or a property of a virtual object in a virtual environment and to annotate the virtual object, the property of the virtual object and / or the virtual environment and to generate simulation data therefrom, characterized in that the artificial intelligence unit is designed to detect, classify and / or segment an object and / or a property of an object on the basis of the simulated measurement data and the simulation data, on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement. System according to claim 1, characterized in that the simulation unit is designed to simulate more than one virtual object with more than one property and / or is designed to fully annotate the virtual object and / or the property of the virtual object or the virtual objects and / or the properties of the virtual objects and / or the virtual environment. System according to claim 1 or 2, characterized in that the system further comprises means by means of which the virtual object and / or the virtual environment is generated using 3D information generated from a game engine or based on map information, aerial photographs, earth remote sensing information and / or measurement data from a camera system and / or a laser scanner or other sensor systems.System according to one of the preceding claims, characterized in that the simulated sensor unit is modeled after a real sensor unit and / or that the simulated sensor unit is improved compared to a real sensor unit, in particular with regard to resolution, signal-to-noise ratio and / or unambiguousness ranges. System according to one of the preceding claims, characterized in that the simulated sensor unit simulates a radar, a camera, a lidar and / or an ultrasound device. System according to one of the preceding claims, characterized in that the simulated measurement data, other simulated measurement data, or measurement data generated from a real measurement are image data and / or that the artificial intelligence unit is designed to improve raw and processed sensor signals in the measurement data. System according to one of the preceding claims, characterized in that the environment is a dynamic, three-dimensional environment. System according to one of the preceding claims, characterized in that the artificial intelligence unit is part of a vehicle and / or that the measurement data generated from the real measurement was generated by a radar, a camera, a lidar, and / or an ultrasound device.Method for detecting, classifying and / or segmenting an object and / or a property of an object using a system according to one of the preceding claims, comprising the following steps: a) generating simulated measurement data from a simulated sensor unit about a virtual object and / or a property of a virtual object in a virtual environment; b) annotating the virtual object, the property of the virtual object and / or the virtual environment and generating simulation data, characterized in that, on the basis of the simulated measurement data and the simulation data, an object and / or a property of an object is detected, classified and / or segmented on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement.Method for training an artificial intelligence unit of a system according to one of claims 1 to 8, characterized in that the artificial intelligence unit is trained on the basis of simulated measurement data and the simulation data to detect, classify and / or segment an object and / or a property of an object on the basis of the simulated measurement data or on the basis of other simulated measurement data and / or on the basis of measurement data generated from a real measurement. A computer program comprising instructions that cause the system of any one of claims 1 to 8 to perform the method steps of any one of claims 9 or 10. A computer-readable medium on which the computer program of claim 11 is stored.

Images