Increasing and processing radar data with machine learning

Abstract

A method for converting source radar data of a source configuration of a radar system into target radar data of a target configuration of the radar system includes the steps of: providing a two-dimensional or more-dimensional source array of grid cells for spatial coordinates of source reflection locations; determining and assigning the probability and / or frequency of one or more source reflection locations being located in the grid cell from the spatial coordinates of all source reflection locations for each grid cell; constructing a source tensor comprising at least a source array occupying the probability and / or frequency; transforming the source tensor into a target tensor comprising at least one target array of grid cells for spatial coordinates of target reflection locations, wherein the target array specifies for each grid cell the probability and / or frequency of a target reflection location being located in the corresponding grid cell when observing a scene using the target configuration; and sampling the spatial coordinates of the target reflection locations according to the probability and / or frequency in the target array to form target radar data.

Description

Technical Field

[0001] This invention relates to generating real radar data from existing radar data, and more generally to processing radar data using machine learning. Background Technology

[0002] In order for vehicles to move at least partially autonomously in road traffic, it is necessary to detect the vehicle's environment and introduce countermeasures if there is a risk of collision with objects in the vehicle's environment. Creating an environmental representation and localization are also necessary for safe autonomous driving.

[0003] Radar detection is independent of lighting conditions and is possible, even at considerable distances at night, without dazzling oncoming traffic with high beams. Furthermore, the distance and speed of the object are directly known from the radar data. This information is crucial for assessing the likelihood of a collision. However, the type of object cannot be directly identified from the radar signal. This identification is currently achieved by calculating attributes from digital signal processing.

[0004] Trained machine learning models, such as neural networks, can make a major contribution, particularly for object recognition. To train these models, training data is needed, often recorded from test drives and subsequently annotated (“labeled”) with objects actually present in the observed scenes. Labeling requires considerable manual work and is therefore expensive. DE 10 2018 204 494 B3 discloses a generator that can store extended synthetic radar data for a given training dataset. Summary of the Invention

[0005] Within the scope of this invention, a method for converting source radar data into target radar data has been developed. Source radar data is radar data obtained by observing a scene using a source configuration of a radar system. Target radar data is radar data that should be expected when observing the same scene using a target configuration of the radar system. Both the source and target configurations of the radar system include at least one transmitter and one receiver for radar radiation, and can additionally be characterized by other parameters such as transmitter and / or receiver settings.

[0006] Here, the source configuration can be the same as the target configuration. In this case, real synthetic radar data can be extended to a given radar data reserve for the configuration of the radar system, as can be achieved, for example, using the generator mentioned at the beginning.

[0007] However, using this method, based on source radar data recorded using the source configuration of the radar system, it is also possible to obtain target radar data that should actually be expected when observing the same scene using completely different target configurations of the same or different radar systems. In this way, the same radar data reserve can, for example, be reused as training data for object recognition using any new target configuration of the radar system, and the same radar data reserve may also have been tagged. This saves the enormous expense of physically recording other radar data. As long as the radar system is carried by a vehicle, the additional costs for performing other test drives can be significantly reduced, for example.

[0008] For example, after numerous test drives using radar sensors installed at specific locations on a vehicle, it can be decided that the radar sensors can be moved to another location on the vehicle in the future. This alters the recording characteristics of the radar sensors. In particular, the observable field of view and angle of view change. The radar data recorded during previous test drives can no longer be used to train object recognition or other evaluations based on radar data recorded from the new sensor location. Instead, the recording of training data must begin from scratch again.

[0009] To date, installing radar sensors in vehicles from other manufacturers may render previously recorded radar data for training purposes unusable. Radar sensors are not visually conspicuous to each end customer and are therefore often mounted behind the vehicle manufacturer's logo. Radar radiation is affected when passing through such a logo, and this effect changes if a logo from one manufacturer is replaced with that of another. In this respect, radar data recorded behind the first logo is only conditionally comparable to radar data recorded behind the second logo. Using this method, such signal changes can be artificially generated, allowing the reserve of training data recorded behind the first logo to continue to be used even after the second logo is replaced.

[0010] Typically, source radar data or target radar data includes at least the spatial coordinates of the source reflection location or target reflection location, and radar radiation reflected from said source reflection location or target reflection location is incident on the radar system. Within the scope of this method, a two-dimensional or more-dimensional source array of grid cells is first provided for the spatial coordinates of the source reflection locations. For each grid cell, the probability and / or frequency of one or more source reflection locations being located within the grid cell is determined from the spatial coordinates of all source reflection locations and assigned to the corresponding grid cell. A source tensor is constructed, which at least includes the source array occupied by the probability and / or frequency.

[0011] For this purpose, for example, an occupancy probability of 1 can be assigned to a circle or sphere with a pre-given radius around the source reflection location. This occupancy probability can then be distributed across all grid cells of the source array according to the area share or volume share of the circle or sphere.

[0012] It has been recognized that the specific physical characteristics of observations using radar radiation lead to uncertainties in radar data. For example, optical observations using cameras or lidar employ radiation with extremely short wavelengths compared to the structure being imaged, thus providing very clear images. In contrast, radar radiation has significantly longer wavelengths, making the wave characteristics of the radiation readily apparent, and the measured reflections are largely random.

[0013] Specifically, this means radar reflection

[0014] • It fluctuates over time, that is, it appears and disappears again in successive single measurements, and

[0015] • It fluctuates in space, that is, it oscillates and is noisy in the spatial coordinates of the location from which the radar radiation appears to originate.

[0016] Therefore, a true picture of the surrounding environment can typically only be obtained from radar measurements as an average and / or a superposition of individual measurements from a large number of radar measurements.

[0017] Because the source tensor contains parameters characterizing the probability distribution of the radar data, this uncertainty is largely suppressed within the source tensor. Therefore, the source tensor essentially contains the result of processing spatial information from the observed scene using a transfer function that is also spatially variable and time-independent.

[0018] The source tensor is now transformed into a target tensor, which includes at least one target array of grid cells representing the spatial coordinates of the target reflection location. This target array specifies, for each grid cell, the probability and / or frequency that the target reflection location, obtained when observing the scene using the target configuration, lies within the corresponding grid cell. Based on the probability and / or frequency in the target array, the spatial coordinates of the target reflection location are sampled to form target radar data.

[0019] In the simple case mentioned at the beginning, where only a single configuration is used and the existing radar data reserve for that configuration should be expanded, the source configuration is the same as the target configuration, and the source tensor is adopted in the same way as the target configuration. New samples are then drawn from the probability distribution determined precisely from the original radar data.

[0020] If the source and target configurations are different, the target reflection position is then correlated with the target tensor via a spatially independent, time-independent transfer function. The relationship between the source and target tensors is thus arising from the difference between their configurations and is also time-independent. However, a closed-formula is typically not available for this relationship. Nevertheless, the path from the source tensor to the target tensor is accessible for machine learning. This means that a trained machine learning model can be used to transform the source tensor into the target tensor. This machine learning model can then be trained using radar data associated with both the source and target configurations.

[0021] Here, the amount of training data required for such training is small compared to the amount of source radar data typically used in practical applications for converting into target radar data. If the amount of source radar data is available and at least the same amount of target radar data is required, the cost of acquiring training data and subsequently training the machine learning model that transforms the source tensor into the target tensor and thus ultimately guides the flow from source radar data to target radar data is significantly lower than the cost of directly acquiring the desired target radar data from the measurement techniques. Furthermore, machine learning models can be repeatedly used to transform other radar data from source configurations to target configurations.

[0022] As explained above, the source configuration and the target configuration can differ, in particular, for example, in at least the following ways:

[0023] • The two configurations include different radar sensors, and / or

[0024] • The radar sensors are arranged spatially differently in the two configurations, and / or

[0025] • In both configurations, the radar radiation is affected in different ways by the materials arranged between at least one radar sensor and the observed scene.

[0026] The method described here therefore has the following effect: this configuration switch does not invalidate the radar data reserves acquired before the switch. Instead, only a machine learning model specifically trained for this switch can be acquired, or such a model can be trained.

[0027] Conversely, this means that rendering previously acquired radar data "invalid" is no longer a key argument for the desired configuration switch. If, for example, it is shown that the new radar sensor for the vehicle provides radar data of significantly better quality than the radar sensors used to date, then repeating all the test drives performed so far using the old radar sensors could be just as costly (Umstieg) for the replacement, making the replacement ultimately uneconomical. If that cost is drastically reduced, then nothing can prevent the replacement.

[0028] In a particularly advantageous extension, the machine learning model includes an encoder-decoder device with an encoder mapping a source tensor to a representation with reduced dimensionality, and the decoder mapping that representation to a target tensor. Such a device can be trained such that the encoder, when compressing the source tensor into the representation, substantially ignores the details of the source configuration and obtains the most important information about the observed scene, information that is difficult to fit into explicitly expressed conditions. The decoder then, conversely, adds the details of the target configuration to the compressed representation, thereby producing the target tensor.

[0029] In another particularly advantageous extension, the machine learning model includes a generator of a Generative Adversarial Network (GAN). Such a generator can be trained to produce a target tensor from a source tensor associated with the source configuration, the target tensor being indistinguishable, or very difficult to distinguish, from a target tensor generated from actual measured target radar data along the same path as the source tensor generated from the source radar data. This training can be performed cooperatively with a discriminator trained to distinguish the target tensor generated from the source tensor from the target tensor generated from the actual measured target radar data. The generator and discriminator together constitute the GAN. If CycleGAN is involved here, it can be trained even if many source radar data measured using the source configuration and many target radar data measured using the target configuration are available, but pairs of source radar data and target radar data associated with the same scene are unavailable.

[0030] Constructing a source tensor from source radar data in the manner previously described is not only suitable for generating new radar data, but can also be used in the same way as radar data actually measured using the source or target configuration of the radar system. More specifically, it is generally advantageous to represent the source radar data as a distribution of probabilities and / or frequencies in the source tensor in order to shield the evaluation of the source radar data (such as object identification or other assignments to classes) from the random fluctuations of radar reflections.

[0031] Therefore, the present invention also relates to a method for processing source radar data using a neural network, the source radar data being obtained by observing a scene using a source configuration of a radar system. The source radar data includes at least the spatial coordinates of a source reflection location, from which radar radiation reflected is incident on the radar system.

[0032] In this method, a two-dimensional or more-dimensional source array of grid cells is provided for the spatial coordinates of the source reflection locations. For each grid cell, the probability and / or frequency of one or more source reflection locations being located within the grid cell is determined from the spatial coordinates of all source reflection locations and assigned to the corresponding grid cell. This constitutes a source tensor, which at least includes a source array occupying the position with probability and / or frequency. This source tensor is fed into a neural network.

[0033] In this way, the uncertainties inherent in the source radar data no longer affect the results provided by the neural network. Thus, the overall system—which first records the source radar data, processes it into results, and then uses those results to control the vehicle—becomes more reliable.

[0034] For example, a source tensor can be mapped by a neural network to one or more classes with a pre-given classification. These classes can, for example, represent objects, with source radar data indicating the presence of said objects. However, these classes can also, for example, represent the traffic conditions in which a vehicle is detected, from source radar data of said vehicle.

[0035] Therefore, in particular, control signals can be generated for a vehicle from the output provided by a neural network, and the vehicle can be controlled using these control signals.

[0036] Regardless of whether the source tensor should ultimately produce new real target radar data or other processing results, the source tensor and, if necessary, the target tensor may each include the allocation of at least one other additional parameter, which can be derived from the source radar data or the target radar data, to the grid cells of the source array or the target array.

[0037] Additional parameters, especially, may include, for example,

[0038] • The spacing between the source radar system and the source reflection location or between the target radar system and the target reflection location, and / or

[0039] • The angle at which radar radiation is incident on the source or target radar system, and / or

[0040] • The velocity of the object reflecting radar radiation, and / or

[0041] • The attribution of objects reflecting radar radiation to one or more pre-given categories, and / or

[0042] • The signal strength of the reflected radar radiation.

[0043] If the processing result is determined directly from the source tensor using a neural network, the values ​​of additional parameters can be used, for example, to resolve ambiguity in object identification. If, for example, one of the additional parameters indicates that an object to which a particular source radar reflection belongs moves independently at a specific speed, then that object cannot be a traffic sign or a similar fixed object.

[0044] If the target tensor is determined from the source tensor, and the target radar data is determined from the target tensor, then we can also learn, along with transforming the source array to the target array, how the values ​​of the additional parameters change in the allocation of grid cells when switching from the source configuration to the target configuration.

[0045] If, for example, a radar sensor views the scene from a different angle in the target configuration than in the source configuration, the target radar data provides information about object velocity components that differ from the source radar data. For instance, the angle at which the radar sensor views the target reflection location in the target configuration also changes compared to the angle at which the radar sensor sees the source reflection location in the source configuration.

[0046] This can be mapped, for example, in a particularly advantageous extension scheme, by taking one or more values ​​of the additional parameter and constructing new values ​​for the additional parameter for one or more grid cells of the target array, and then adding them to the target reflection location in the target radar data, the grid cells facilitating the sampling of the specific target reflection location.

[0047] Typically, combining the source array with additional parameters in the source tensor makes it easy to continue using machine learning models originally designed for processing images. Images with multiple color channels can also be represented as tensors, which have a form very similar to the source tensor.

[0048] In particular, these methods can be implemented entirely or partially by a computer. Therefore, the present invention also relates to a computer program having machine-readable instructions that, when executed on one or more computers, cause the computers to perform one of the described methods. In this sense, control devices for vehicles and embedded systems for technical devices capable of executing machine-readable instructions can also be considered as computers.

[0049] Similarly, the present invention also relates to machine-readable data carriers and / or downloadable products having computer programs. Downloadable products are digital products that can be transmitted via a data network, i.e., downloaded by a user of the data network, and such digital products may, for example, be sold in online stores for immediate download.

[0050] In addition, computers may be equipped with computer programs, machine-readable data carriers, or downloadable products. Attached Figure Description

[0051] Other measures to improve the invention are shown in more detail below with reference to the accompanying drawings, together with the description of preferred embodiments of the invention.

[0052] Figure 1 An embodiment of a method 100 for converting source radar data 3 into target radar data 5 is shown;

[0053] Figure 2 Exemplary scenarios are shown for observing vehicles 50 with different configurations 2 and 4 of radar systems;

[0054] Figure 3 An embodiment of a method 200 for processing source radar data 3 using a neural network 7 is shown. Detailed Implementation

[0055] Figure 1 A schematic flowchart illustrating one embodiment of a method 200 for converting source radar data 3 into target radar data 5 is shown. Source radar data 3 is obtained by observing scene 1 using the source configuration 2 of a radar system. Target radar data 5 is the radar data that should be expected when observing the same scene 1 using the target configuration 4 of the radar system.

[0056] The source radar data 3 includes at least the spatial coordinates of the source reflection locations 31, from which radar radiation reflected is incident on the radar system. In step 110, a two-dimensional or more-dimensional source array 32 is provided with grid cells 32a for these spatial coordinates. In step 120, for each grid cell 32a, the probability and / or frequency 32b for one or more source reflection locations 31 to be located in the grid cell 32a is determined from the spatial coordinates of all source reflection locations 31 and assigned to the corresponding grid cell 32a.

[0057] Therefore, for example according to block 121, the occupancy probability of 1 can be assigned to a circle or sphere with a pre-given radius surrounding the source reflection position 31. Then, according to block 122, this occupancy probability can be distributed to all grid cells 32a of the source array 32, based on the area share or volume share of the circle or the volume share of the sphere.

[0058] In step 130, a source tensor 33 is constructed, which includes at least a source array 32 occupied by probability and / or frequency 32b. The source tensor 33 may also include, for example, values ​​of additional parameters 8, which are assigned to the grid cells 32a of the source array 32.

[0059] In step 140, the source tensor 33 is transformed into a target tensor 53, which includes at least one target array 52 of grid cells 52a for the spatial coordinates of the target reflection position 51. The target array 52 specifies, for each grid cell 52a, the probability and / or frequency 52b of the target reflection position 51 obtained when observing scene 1 using target configuration 4 being located in the corresponding grid cell 52a.

[0060] If source configuration 2 is the same as target configuration 4, then for this purpose, for example according to block 141, the same source tensor 33 as target tensor 53 can be used.

[0061] If the target configuration 4 is different from the source configuration 2, then the source tensor 33 can be transformed into the target tensor 53, for example, by using the trained machine learning model 6 according to block 142.

[0062] In step 150, the spatial coordinates of the target reflection position 51 are sampled according to the probability and / or frequency 52b in the target array 52, thereby forming target radar data 5. In this case, especially according to block 151, for one or more grid cells 52a of the target array 52 that facilitate the sampling of a specific target reflection position 51, a new value 8' of the additional parameter (8) can be formed by one or more values ​​of the additional parameter (8). This new value 8' can then be added to the target reflection position 51 according to block 152.

[0063] Figure 2 This schematically illustrates how and why the source radar data 3 of scenario 1 obtained using source configuration 2 of the radar system differs from the target radar data 5 obtained using target configuration 4 of the same scenario 1, whether using the same or different radar systems.

[0064] exist Figure 2 In the example shown, scenario 1 illustrates vehicle 50. Using source configuration 2 including transmitter 2a and receiver 2b, radar reflection is recorded at source reflection location 31. For clarity, ... Figure 2 Only three of the aforementioned source reflection locations are shown in the diagram. However, using target configuration 4, which includes transmitter 4a and receiver 4b at another spatial location, radar reflection is recorded at target reflection location 51. For clarity, [the following text is incomplete and likely refers to a separate section:] Figure 2 Only three of the target reflection positions are shown in the drawing.

[0065] Figure 3 This is a schematic flowchart of one embodiment of a method 200 for processing source radar data 3. The method begins in a manner similar to that of method 100 described above.

[0066] As in method 100 described above, the source radar data 3 includes at least the spatial coordinates of the source reflection locations 31, from which radar radiation reflected is incident on the radar system. In step 210, a two-dimensional or more-dimensional source array 32 is provided with grid cells 32a for these spatial coordinates. In step 220, for each grid cell 32a, the probability and / or frequency 32b for one or more source reflection locations 31 to be located in the grid cell 32a is determined from the spatial coordinates of all source reflection locations 31 and assigned to the corresponding grid cell 32a.

[0067] Therefore, for example according to block 221, the occupancy probability of 1 can be assigned to a circle or sphere with a pre-given radius surrounding the source reflection position 31. Then, according to block 222, this occupancy probability can be distributed to all grid cells 32a of the source array 32, based on the area share or volume share of the circle or the volume share of the sphere.

[0068] In step 230, a source tensor 33 is constructed, which includes at least a source array 32 occupied by probability and / or frequency 32b. The source tensor 33 may also include, for example, values ​​of additional parameters 8, which are respectively assigned to the grid cells 32a of the source array 32.

[0069] In step 240, the source tensor 33 is fed into the neural network 7 and processed by the neural network 7 into output 240a. Here, the source tensor 7 can be mapped, for example, to one or more classes with a pre-given classification.

[0070] exist Figure 3 In the example shown, in step 250, output 240a constitutes a control signal 250a for vehicle 50, and in step 260, vehicle 50 is controlled using control signal 250a.

Claims

1. A method (100) for converting source radar data (3) obtained by observing a scene (1) using a source configuration (2) of a radar system into target radar data (5) that should be expected when observing the same scene (1) using a target configuration (4) of a radar system, wherein the source radar data (3) or the target radar data (5) includes at least the spatial coordinates of a source reflection location (31) or a target reflection location (51), radar radiation reflected from the source reflection location or the target reflection location is incident on the radar system, and wherein the method (100) comprises the following steps: • A two-dimensional or more-dimensional source array (32) that provides (110) grid cells (32a) for the spatial coordinates of the source reflection location (31); • For each grid cell (32a), determine the probability and / or frequency (32b) of one or more source reflection locations (31) being located in the grid cell (32a) from the spatial coordinates of all source reflection locations (31) and assign (120) to the corresponding grid cell (32a). • Construct (130) a source tensor (33), the source tensor comprising at least a source array (32) occupied by probability and / or frequency (32b). • Transform the source tensor (33) (140) into a target tensor (53), the target tensor including at least one target array (52) of grid cells (52a) for the spatial coordinates of the target reflection position (51), wherein the target array (52) describes for each grid cell (52a) the probability and / or frequency (52b) that the target reflection position (51) obtained when observing the scene (1) using the target configuration (4) is located in the corresponding grid cell (52a); • Based on the probability and / or frequency (52b) in the target array (52), the spatial coordinates of the target reflection position (51) are sampled (150) to form the target radar data (5).

2. The method (100) according to claim 1, wherein the source configuration (2) is the same as the target configuration (4), and wherein the source tensor (33) is adopted in the same way as the target tensor (53) (141).

3. The method (100) according to claim 1, wherein the source configuration (2) is different from the target configuration (4) and wherein the source tensor (33) is transformed (142) into the target tensor (53) using a trained machine learning model (6).

4. The method (100) according to claim 3, wherein the source configuration (2) differs from the target configuration (4) in at least the following ways: • The two configurations (2, 4) include different radar sensors, and / or • The radar sensors are arranged spatially differently in two configurations (2, 4), and / or • In both configurations (2, 4), the radar radiation is affected in different ways by the material arranged between at least one radar sensor and the observed scene (1).

5. The method (100) according to any one of claims 3 to 4, wherein the machine learning model (6) comprises an encoder-decoder device having an encoder and a decoder, the encoder mapping the source tensor (33) to a representation having a reduced dimension, and the decoder mapping the representation to the target tensor (53).

6. The method (100) according to any one of claims 3 to 4, wherein the machine learning model includes a generator of a generative adversarial network (GAN).

7. The method (100) according to any one of claims 1 to 4, wherein the source tensor (33) and the target tensor (53) respectively include the allocation of at least one other additional parameter (8) derived from the source radar data (3) or the target radar data (5) to the grid cells (32a, 52a) of the source array (32) or the target array (52).

8. The method (100) according to claim 7, wherein one or more grid cells (52a) of the target array (52) that help to sample the specific target reflection position (51) constitute (151) a new value (8') of the additional parameter (8) from one or more values ​​of the additional parameter (8) and add (152) the target reflection position (51) to the target radar data (5).

9. The method (100) according to claim 7, wherein the additional parameter (8) includes • The distance between the source radar system and the source reflection location (31) or between the target radar system and the target reflection location (51), and / or • The angle at which the radar radiation is incident on the source radar system or the target radar system, and / or • The velocity of the object reflecting the radar radiation, and / or • The attribution of the object reflecting the radar radiation to one or more pre-given categories, and / or • The signal strength of the reflected radar radiation.

10. The method (100) according to any one of claims 1 to 4, wherein an occupancy probability (121, 221) of 1 is assigned to a circle or sphere having a pre-given radius around the source reflection position (31) and wherein the occupancy probability is distributed (122, 222) to all grid cells (32a) of the source array (32) according to area share or volume share, wherein the area of ​​the circle or the volume of the sphere is distributed to the grid cells.

11. A computer program product having a computer program, the computer program comprising machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 10.

12. A machine-readable data carrier or downloadable product having a computer program, the computer program comprising machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 10.

13. A computer equipped with a computer program and / or a machine-readable data carrier or downloadable product having the computer program, the computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 10.

Images