Radar data augmentation and processing through machine learning

Abstract

To provide a method for converting radar data on a source configuration to radar data on a target configuration.SOLUTION: A method includes: a step 110 of providing a source array 32 of a grid cell 32a with respect to a coordinate of a source reflection position 31; a step 120 of allocating a probability / frequency 32b in which the source reflection position 31 is located within the grid cell 32a from all of the source reflection position 31; a step 130 of forming a source tensor 33 including the source array 32; a step 140 of converting the source tensor 33 to a target tensor 53, in which a target array 52 indicates a probability / frequency 52b in which a target reflection position 51 to be obtained when observing a scene is located with respect to each grid cell 52a; and a step 150 of sampling a coordinate of the target reflection position 51 on the basis of the probability / frequency 52b in the target array 52, which in turn target array data 5 is generated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to generating real radar data from pre-existing radar data and generally to processing radar data by machine learning. [Background technology]

[0002] Prior art In order to be able to drive a vehicle at least partly automatically in road traffic, it is necessary to detect the vehicle's surroundings and to take countermeasures if there is a risk of a collision with an object in the vehicle's surroundings. A description and localization of the surroundings is also necessary for safe automated driving.

[0003] Radar detection of objects can be performed regardless of lighting conditions, for example, at night, and even from a relatively large distance, without blinding oncoming traffic with high beams. Furthermore, radar data directly reveals the distance to the object and the object's speed. This information is important for determining whether a collision with the object is likely to occur. However, the radar signal does not directly identify the type of object. Today, this identification is achieved by calculating attributes using digital signal processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] German Patent Invention No. 102018204494 Summary of the Invention [Problem to be solved by the invention]

[0005] Trained machine learning models, such as neural networks, play an important role here, especially for object identification. To train such models, training data is required. Often, training data is recorded during test drives, and then the training data is used to annotate ("label") the objects actually contained in each observed scene. This labeling requires a lot of manual work and is therefore expensive. DE 102018204494 discloses a generator that can expand a given stock of training data to include synthetic radar data. [Means for solving the problem]

[0006] Disclosure of the Invention Within the scope of the present invention, a method has been developed for converting source radar data into target radar data. The source radar data is radar data obtained by observing a scene with a source configuration of a radar system. The target radar data is radar data expected when observing a similar scene with a target configuration of the radar system. Both the source and target configurations of a radar system each include at least one transmit and receive side of a radar beam and may additionally be characterized by further parameters, for example, by the settings of the transmit and / or receive sides.

[0007] In this case, the source configuration may be identical to the target configuration. In such a case, the given stock of radar data for the configuration of the radar system can be expanded to include actual synthetic radar data, which can be done, for example, by the generator mentioned at the beginning.

[0008] However, in the method according to the present invention, target radar data can also be obtained starting from source radar data recorded by a source configuration of the radar system. The target radar data is what would actually be expected when observing a similar scene with a completely different target configuration of an equivalent or different radar system. In this way, the same given stock of radar data, which may also have already been labeled, can be reused by any new target configuration of the radar system, for example, as training data for object identification. This saves considerable costs associated with the physical recording of additional radar data. In particular, if the radar system is installed on a vehicle, the additional costs of, for example, performing additional test runs can be significantly reduced.

[0009] For example, when a large number of test drives have already been performed with a radar sensor mounted at a specific location on a vehicle, it can be determined whether this radar sensor should be mounted at another location on the vehicle in the future. This changes the recording characteristics of the radar sensor. In particular, the observable field of view and viewpoint change. Up until now, the radar data recorded in previous test drives cannot be used to train object identification or other evaluations based on radar data recorded from a new position of the sensor; the recording of training data must be started again from scratch.

[0010] Up until now, installing radar sensors in vehicles from other manufacturers has also resulted in the unavailability of radar data previously recorded for training purposes. Because radar sensors do not offer a good visual impression to all end customers, they are often installed behind the vehicle manufacturer's emblem. Passing through such an emblem affects the radar beam, and this effect changes when the manufacturer's emblem is replaced with another manufacturer's emblem. Therefore, radar data recorded behind a first emblem can only be compared to radar data recorded behind a second emblem to a limited extent. The method of the present invention allows for the artificial generation of such signal changes, allowing the stock of training data recorded behind the first emblem to be reused even after the first emblem has been replaced with a second emblem.

[0011] Generally, the source radar data or target radar data includes at least the location coordinates of source reflection locations or target reflection locations from which the reflected radar beam enters the radar system. Within this method, a two- or more-dimensional source array of grid cells is first provided relative to the location coordinates of the source reflection locations. From the location coordinates of all source reflection locations, a probability and / or frequency that one or more source reflection locations are located within this grid cell is determined and assigned to each grid cell. A source tensor is formed that includes at least the source array populated by this probability and / or frequency.

[0012] For this purpose, for example, a circle or sphere having a given radius centered on the source reflection location can be assigned an occupancy probability of 1. Then, such occupancy probability can be distributed to all grid cells of the source array into which the area of ​​the circle or the volume of the sphere is distributed according to the area ratio or volume ratio.

[0013] It has been found that the special properties of radar beam observations lead to uncertainties in radar data. For example, optical observations by cameras or lidars provide very clear images because they use beams that are very short compared to the structures to be imaged. In comparison, radar beams are much longer in wavelength, so the wave characteristics of the beam are more pronounced and the measured reflections are primarily stochastic in nature.

[0014] Specifically, this means that the radar reflection is fluctuating over time, i.e. appearing in successive individual measurements and then disappearing again, Spatially varying, i.e. the position coordinates from which the radar beam appears to emanate fluctuate and change.

[0015] Therefore, a real picture of the surrounding environment can only be obtained from radar measurements, and thus typically via a large number of radar measurements, as an average and / or superposition of the individual measurements.

[0016] Such uncertainties are largely contained in the source tensor because it contains parameters that characterize the probability distribution of the radar data. Thus, the source tensor essentially contains the result of processing spatial information from the observed scene by a time-independent transfer function that also varies only spatially.

[0017] The source tensor is then transformed into a target tensor that includes at least one target array of grid cells relative to the location coordinates of the target reflection locations. Such target array indicates, for each grid cell, the probability and / or frequency that a target reflection location obtained when observing the scene with the target configuration is located within that grid cell. Based on this probability and / or frequency in the target array, the location coordinates of the target reflection locations are sampled to generate target radar data.

[0018] In the simple case mentioned above, where only one configuration is used and the existing stock of radar data for this configuration is expanded, the source configuration is identical to the target configuration, and the source tensor is taken to be identical to the target tensor. In this case, new samples are obtained strictly from the probability distribution determined based on the original radar data.

[0019] On the other hand, if the source configuration and the target configuration are not identical, the target reflection position to be determined is similarly related to the target tensor via a spatial, not time-related, transfer function. The relationship between the source tensor and the target tensor is in this case justified by the difference between the source configuration and the target configuration and is similarly time-related. Nevertheless, there is usually no unified definition of such a relationship. However, the path from the source tensor to the target tensor is open to machine learning. That is, the source tensor can be transformed into the target tensor by a trained machine learning model. The machine learning model is similarly trained by radar data related to the source configuration and radar data related to the target configuration.

[0020] However, the amount of training data required for such training is typically small compared to the amount of source radar data to be converted to target radar data in a particular application. That is, if such a quantity of source radar data is provided and at least an equivalent quantity of target radar data is required, the cost of procuring training data and subsequently training a machine learning model that converts source tensors to target tensors, and thus ultimately derives target radar data from the source radar data, is significantly less than the cost of procuring the required target radar data via direct measurement techniques. Furthermore, the machine learning model can be used multiple times to further convert radar data from source configurations to target configurations.

[0021] As noted above, the source and target configurations may differ, among other things, with respect to at least the following: The two configurations include different radar sensors, and / or These radar sensors are spatially arranged differently in the two configurations, and / or In the two configurations, the radar beam is affected differently by materials placed between the at least one radar sensor and the observed scene.

[0022] Thus, the method presented here has the effect that such a change in configuration does not "invalidate" the stock of radar data already acquired prior to the change. Instead, a machine learning model specifically trained for such a change may be procured or trained.

[0023] This means, conversely, that the "invalidation" of previously acquired radar data is no longer an important argument for the intended change of configuration. For example, if it is clear that a new radar sensor for a vehicle provides radar data of significantly better quality than the radar sensor used up until now, repeating all test drives that have been performed up until now with the old radar sensor may nevertheless be a high price for such a changeover, making the changeover uneconomical, all things considered. If such a price can be significantly reduced, this does not preclude the changeover.

[0024] In a particularly advantageous configuration, the machine learning model includes an encoder-decoder device with an encoder and a decoder, where the encoder maps a source tensor to a reduced-dimensional representation and the decoder maps this representation to a target tensor. Such a device is trained so that, when compressing the source tensor to this representation, the encoder essentially omits details of the source configuration and captures crucial information about the observed scene that is difficult to capture in explicitly formulated terms. The decoder, in turn, adds details of the target configuration to the compressed representation, thus generating the target tensor.

[0025] In another particularly advantageous configuration, the machine learning model includes a generator of a generative adversarial network (GAN). Such a generator is trained to generate a target tensor from a source tensor for a source configuration. This target tensor is indistinguishable or difficult to distinguish from a target tensor generated from actually measured target radar data in the same way that the source tensor is generated from the source radar data. Such training can be performed in conjunction with a discriminator, which is trained to distinguish the target tensor generated from the source tensor from the target tensor generated from actually measured target radar data. The generator and discriminator together form a GAN. In the case of a CycleGAN, training is also performed when many source radar data measured by the source configuration and many target radar data measured by the target configuration are provided, but when no source radar data and target radar data pairs for similar scenes are provided.

[0026] The formation of a source tensor from source radar data, as described above, is not only suitable for generating new radar data that can be used similarly to radar data actually measured by a radar system's source or target reconfiguration. Rather, representing source radar data as probability and / or frequency distributions in a source tensor is generally advantageous for shielding the evaluation of source radar data, such as object identification or other assignment to classes, from stochastic variations in radar returns.

[0027] The present invention therefore also relates to a method for processing, by a neural network, source radar data obtained by observing a scene with a source configuration of a radar system, the source radar data including at least the position coordinates of a source reflection location from which a reflected radar beam enters the radar system.

[0028] In this method, a two- or more-dimensional source array of grid cells relative to the location coordinates of source reflection locations is provided. From the location coordinates of all source reflection locations, a probability and / or frequency that one or more source reflection locations are located within this grid cell is determined for each grid cell and assigned to each grid cell. A source tensor is formed that includes at least the source array populated by the probabilities and / or frequencies. This source tensor is then fed to a neural network.

[0029] In this way, uncertainties associated with the source radar data do not affect the results provided by the neural network, resulting in a more robust overall system that first records the source radar data, processes the source radar data into results, and then, for example, controls the vehicle's operation based on the results.

[0030] For example, a neural network can map the source tensor into one or more classes of a predetermined classification. Such classes may represent, for example, objects whose presence indicates source radar data. However, these classes may also represent, for example, traffic situations in which vehicles are present and from which source radar data is detected.

[0031] This makes it possible, in particular, for example, to generate drive control signals for a vehicle from the output provided by the neural network, and to drive the vehicle with the aid of these drive control signals.

[0032] Regardless of whether new actual target radar data or other processing results are ultimately generated from the source tensor, the source tensor and possibly the target tensor may each include an assignment of at least one other additional parameter derivable from the source radar data or target radar data to grid cells of the source array or target array.

[0033] These additional parameters are, in particular, e.g. the spacing between the source radar system and the source reflection location or the target radar system and the target reflection location, and / or the angle at which the radar beam is incident on the source radar system or the target radar system, and / or the speed of the object from which the radar beam is reflected, and / or the attribution of the objects from which the radar beam is reflected to one or more classes of a predetermined classification, and / or - Signal strength of the reflected radar beam may include:

[0034] When the processing result is determined by a neural network directly from the source tensor, the value of the additional parameter can be used, for example, to resolve ambiguity in identifying the object: for example, if one of the additional parameters indicates that the object to which a particular source radar return belongs is moving independently at a particular speed, then such an object cannot be a traffic sign or a similar fixedly mounted object.

[0035] If a target tensor is derived from a source tensor, and target radar data is derived from the target tensor, then along with the transformation of the source array to the target array, it can also be learned how the assignment of values ​​of additional parameters to grid cells changes when changing from a source configuration to a target configuration.

[0036] For example, if a radar sensor in the target configuration observes the scene from a different perspective than in the source configuration, the target radar data will provide information on the velocity components of objects that differs from the source radar data. For example, the angle at which the radar sensor in the target configuration views the target return location will also be changed relative to the angle at which the radar sensor in the source configuration views the source return location.

[0037] This is in particular achieved, for example, by forming, in a particularly advantageous configuration, new values ​​of the additional parameters from the value(s) of the additional parameter(s) for one or more grid cells of the target array that contributed to the sampling of a specific target return location, and assigning them to the target return location in the target radar data.

[0038] Overall, the packaging of source arrays with additional parameters in source tensors facilitates the reuse of machine learning models originally designed to process images: images with several color channels are similarly represented as tensors with a very similar design to the source tensors.

[0039] The method may in particular be fully or partly computer-implemented. The invention therefore also relates to a computer program comprising machine-readable instructions, which, when executed on one or more computers, cause the one or more computers to carry out one of the described methods. In this sense, control devices for vehicles and embedded systems for technical devices, which may also execute machine-readable instructions, are also considered as computers.

[0040] The invention likewise relates to a machine-readable data carrier and / or a download product comprising such a computer program, which is a digital product that can be transmitted over a data network, i.e. downloaded by a user of the data network, and which can be sold for immediate download in, for example, an online shop.

[0041] Furthermore, the computer may be equipped with a computer program, a machine-readable data carrier or a downloadable product.

[0042] Further measures for improving the invention will be explained in detail below with reference to the drawings together with a description of preferred embodiments of the invention. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is an embodiment of a method 100 for converting source radar data 3 into target radar data 5. [Figure 2] 1 is an exemplary situation in which a vehicle 50 is observed by different configurations 2, 4 of a radar system. [Figure 3] 2 is an embodiment of a method 200 for processing source radar data 3 by a neural network 7. DETAILED DESCRIPTION OF THE INVENTION

[0044] Example 1 is a schematic flow chart of an embodiment of a method 100 for converting source radar data 3 to target radar data 5. The source radar data 3 is obtained by observing a scene 1 with a source configuration 2 of a radar system. The target radar data 5 is the radar data expected when observing the same scene 1 with a target configuration 4 of the radar system.

[0045] The source radar data 3 includes at least the location coordinates of source reflection locations 31 from which the reflected radar beam enters the radar system. In step 110, a two- or more-dimensional source array 32 of grid cells 32a for such location coordinates is provided. In step 120, from the location coordinates of all source reflection locations 31, for each grid cell 32a, a probability and / or frequency 32b that one or more source reflection locations 31 are located within that grid cell 32a is determined and assigned to each grid cell 32a.

[0046] To this end, for example, in block 121, an occupancy probability of 1 can be assigned to a circle or sphere having a predetermined radius centered on the source reflection location 31. Then, in block 122, such occupancy probability can be distributed to all grid cells 32a of the source array 32 into which the area of ​​the circle or the volume of the sphere is distributed, according to the area ratio or volume ratio.

[0047] In step 130, a source tensor 33 is formed that includes at least the source array 32 populated with probabilities and / or frequencies 32b. The source tensor 33 may further include, for example, values ​​of additional parameters 8 that are assigned to each grid cell 32a of the source array 32.

[0048] The source tensor 33 is transformed in step 140 into a target tensor 53. The target tensor 53 includes at least a target array 52 of grid cells 52a relative to the position coordinates of the target reflection locations 51. Such target array 52 indicates, for each grid cell 52a, the probability and / or frequency 52b that the target reflection location 51 obtained when observing the scene 1 with the target configuration 4 is located within this grid cell 52a.

[0049] For this purpose, for example, in block 141, source tensor 33 can be taken over as being identical to target tensor 53 if source configuration 2 is identical to target configuration 4.

[0050] On the other hand, if the target configuration 4 is different from the source configuration 2, the source tensor 33 can be transformed into the target tensor 53, for example, by a trained machine learning model 6 in block 142.

[0051] In step 150, location coordinates of target reflection locations 51 are sampled based on the probabilities and / or frequencies 52b in the target array 52 to generate target radar data 5. Specifically, in block 151, new values ​​8' of the additional parameters 8 are formed from the one or more values ​​of the additional parameters 8 for one or more grid cells 52a of the target array 52 associated with the sampling of a particular target reflection location 51. These new values ​​8' may then be assigned to the target reflection locations 51 in block 152.

[0052] FIG. 2 shows schematically how and why source radar data 3 of a scene 1 acquired by a radar system source configuration 2 is distinguished from target radar data 5 acquired of a similar scene 1 by the same or another radar system target configuration 4.

[0053] In the example shown in Figure 2, scene 1 shows a vehicle 50. A source configuration 2 including a transmitter 2a and a receiver 2b causes a radar return to be recorded at source reflection location 31. For clarity, only three of these are shown in Figure 2. In contrast, a target configuration 4 including a transmitter 4a and a receiver 4b at another spatial location causes a radar return to be recorded at target reflection location 51. For clarity, only three of these are shown in Figure 2.

[0054] 3 is a schematic flow chart of an embodiment of a method 200 for processing source radar data 3. The method initially begins similarly to method 100 described above.

[0055] As in method 100 described above, the source radar data 3 includes at least the location coordinates of source reflection locations 31 from which the reflected radar beam enters the radar system. In step 210, a two- or more-dimensional source array 32 of grid cells 32a for such location coordinates is provided. In step 220, from the location coordinates of all source reflection locations 31, for each grid cell 32a, a probability and / or frequency 32b that one or more source reflection locations 31 are located within that grid cell 32a is determined and assigned to each grid cell 32a.

[0056] To this end, for example, in block 221, an occupancy probability of 1 can be assigned to a circle or sphere having a predetermined radius centered on the source reflection location 31. Then, in block 222, such occupancy probability can be distributed to all grid cells 32a of the source array 32 into which the area of ​​the circle or the volume of the sphere is distributed, according to the area ratio or volume ratio.

[0057] In step 230, a source tensor 33 is formed that includes at least the source array 32 populated with probabilities and / or frequencies 32b. The source tensor 33 may further include, for example, values ​​of additional parameters 8 that are assigned to each grid cell 32a of the source array 32.

[0058] The source tensor 33 is fed to the neural network 7 in step 240 and processed by the neural network 7 to an output 240a, whereby the source tensor 33 can in particular be mapped, for example, to one or more classes of a predetermined classification.

[0059] In the example shown in FIG. 3, in step 250, a drive control signal 250a for the vehicle 50 is generated from the output 240a, and in step 260, the vehicle 50 is driven and controlled by this drive control signal 250a.

Claims

1. 1. A method (100) for converting source radar data (3) into target radar data (5), wherein the source radar data (3) is radar data obtained by observing a scene (1) with a source configuration (2) of a radar system, and the target radar data (5) is radar data expected when observing the same scene (1) with a target configuration (4) of the radar system, the source radar data (3) or the target radar data (5) including at least position coordinates of a source reflection location (31) or a target reflection location (51) from which a reflected radar beam is incident on the radar system, providing (110) a two or more dimensional source array (32) of grid cells (32a) relative to the location coordinates of the source reflection locations (31); - determining (120) for each grid cell (32 a) from the position coordinates of all source reflection positions (31), a probability and / or frequency (32 b) that one or more source reflection positions (31) are located within said grid cell (32 a), and assigning this to said each grid cell (32 a); forming (130) a source tensor (33) comprising at least said source array (32) populated by probabilities and / or frequencies (32b); - transforming (140) the source tensor (33) into a target tensor (53) comprising at least one target array (52) of grid cells (52 a) relative to the position coordinates of the target reflection locations (51), the target array (52) indicating, for each grid cell (52 a), the probability and / or frequency (52 b) of the target reflection location (51) being located within said grid cell (52 a) as obtained when observing the scene (1) with said target configuration (4); - sampling (150) position coordinates of target reflection locations (51) based on the probability and / or frequency (52b) in the target array (52), thereby generating the target radar data (5); A method (100) comprising:

2. 2. The method (100) of claim 1, wherein the source configuration (2) is identical to the target configuration (4) and inherits (141) the source tensor (33) as identical to the target tensor (53).

3. 2. The method (100) of claim 1, wherein the source configuration (2) differs from the target configuration (4), and the source tensor (33) is transformed (142) into the target tensor (53) by a trained machine learning model (6).

4. The source configuration (2) and the target configuration (4) differ in at least the following respects: The two configurations (2, 4) contain different radar sensors, and / or the radar sensors are arranged spatially differently in two configurations (2, 4), and / or In two configurations (2, 4), the radar beam is affected differently by materials placed between at least one radar sensor and the observed scene (1); The method (100) of claim 3.

5. 5. The method of claim 3, wherein the machine learning model comprises an encoder-decoder device having an encoder and a decoder, the encoder mapping the source tensor to a reduced-dimensional representation, and the decoder mapping the representation to the target tensor.

6. The method (100) of any one of claims 3 to 5, wherein the machine learning model comprises a generator of a generative adversarial network (GAN).

7. 7. The method of claim 1, wherein the source tensor and the target tensor each include an assignment of at least one other additional parameter derivable from the source radar data or the target radar data to the grid cells of the source array or the target array.

8. 8. The method of claim 7, further comprising forming a new value of the additional parameter from one or more values ​​of the additional parameter for one or more grid cells of the target array that contributed to sampling a particular target reflection location, and assigning the new value to the target reflection location in the target radar data.

9. The additional parameter (8) is: the spacing between the source configuration (2) and the source reflection location (31) or between the target configuration (4) and the target reflection location (51), and / or the angle at which the radar beam is incident on the source configuration (2) or the target configuration (4), and / or the speed of the object from which the radar beam is reflected, and / or - attribution of the object from which the radar beam is reflected to one or more classes of a predetermined classification, and / or - the signal strength of the reflected radar beam 9. The method (100, 200) of claim 7 or 8, comprising:

10. 10. A method (100, 200) according to any one of claims 1 to 9, wherein an occupancy probability of 1 is assigned (121, 221) to a circle or sphere having a predetermined radius centered on the source reflection location (31), and the occupancy probability is distributed (122, 222) to all grid cells (32 a) of the source array (32) into which the area of ​​the circle or the volume of the sphere is distributed, in accordance with the area ratio or volume ratio.

11. A computer program comprising machine-readable instructions, 11. A computer program product, the machine-readable instructions of which, when executed on one or more computers, cause the one or more computers to perform the method (100, 200) of any one of claims 1 to 10.

12. A machine-readable data carrier comprising a computer program according to claim 11.

13. A computer equipped with a computer program according to claim 11 and / or a machine-readable data carrier according to claim 12.

Images