Method and apparatus for providing radar data, computer program, and computer-readable storage medium

Abstract

The present invention relates to a computer-implemented method for providing radar data, the method comprising the steps of: receiving input data, wherein the input data includes satellite images; generating radar data by using a trained machine learning algorithm, applying the trained machine learning algorithm to the input data; and outputting the generated radar data.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for providing radar data, a device for providing radar data, a computer program, and a non-transitory computer-readable storage medium. Background Art

[0002] In the field of autonomous driving, it is necessary to accurately locate a vehicle, whereby the positions of vehicle components can be estimated. Radar data can be used for positioning. The radar data can include dense point clouds, which are generated based on radar measurements. The vehicle can drive through the area to be detected, wherein a radar map is generated based on the radar measurements of the vehicle. However, generating the radar map is time-consuming for the following reasons: all routes in the area to be detected must be driven through. Similarly, generating the radar map is expensive because high costs can be incurred for the vehicle, fuel, and driver.

[0003] Localization based on 3D maps is known from "Global rover localization by matching lidar and orbital3D maps" by Carle et al. (IEEE International Conference on Robotics and Automation, 2010).

[0004] A method for improving an existing map is known from "Exploiting Building Information from PubliclyAvailable Maps in Graph-Based SLAM" by Vysotska et al. (IEEE / RSJ International Conference on Intelligent Robots and Systems, 2016).

[0005] A method for road segmentation is known from "A review of road extraction from remote sensing images" by Wang et al. (Journal of Traffic and Transportation Engineering, March 3, 2016, pages 271-282). Another method of this type can be found in "Learning to Detect Roads in High-Resolution Aerial Images" by Mnih et al. (European Conference on Computer Vision (ECCV), 2010).

[0006] A method based on neural network is known from "Road Extraction from High Resolution Image with Deep Convolution Network—A Case Study of GF-2 Image" by Xia et al. (International Remote Sensing Electronic Conference, 2018). Summary of the Invention

[0007] According to a first aspect, the present invention relates to a computer-implemented method for providing radar data. Input data is received, wherein the input data includes satellite images. Radar data is generated by using a trained machine learning algorithm, wherein the trained machine learning algorithm is applied to the input data. The generated radar data is output.

[0008] According to a second aspect, the present invention relates to a device for providing radar data, the device having an input interface, a computing device and an output interface. The input interface is configured to receive input data, wherein the input data includes satellite images. The computing device is configured to: generate radar data by using a trained machine learning algorithm, wherein the trained machine learning algorithm is applied to the input data. The output interface is configured to output the generated radar data.

[0009] According to a third aspect, the present invention relates to a computer program which, when implemented on a computer, causes the computer to control the implementation of the steps of the computer-implemented method according to the first aspect.

[0010] According to a fourth aspect, the present invention relates to a non-volatile computer-readable storage medium which stores an executable computer program which, when implemented on a computer, causes the computer to control the implementation of the steps of the computer-implemented method according to the first aspect.

[0011] Preferred embodiments are the subject of corresponding extension schemes.

[0012] Advantages of the Invention

[0013] Satellite images are very suitable for detecting large areas. In addition, satellite images can usually be provided cost-effectively. The advantage of the present invention is that certain structures can be well identified both in satellite images and in radar measurements. Road structures such as columns, crash barriers or traffic signs can be detected particularly well by means of both methods. Thereby, the satellite images can be transferred to radar data in a computer-implemented manner.

[0014] In addition, radar data can be generated without a large time overhead, because it is not necessary to drive over the roads in the area to be detected.

[0015] According to the present invention, "the generated radar data" should be understood as synthetic radar data generated based on satellite images. These generated radar data should correspond as precisely as possible to the real radar data generated by radar measurements.

[0016] According to one embodiment of the computer-implemented method, radar data is generated by a radar sensor of a motor vehicle. The radar data generated by the radar sensor of the motor vehicle is compared with the radar data generated based on satellite images. The motor vehicle is positioned based on this comparison. Here, good scalability is advantageous. If satellite images are available in an area, positioning can be performed directly there. Thereby, positioning can be performed in many areas.

[0017] According to one embodiment of the computer-implemented method, applying a trained machine learning algorithm to input data includes: performing semantic segmentation on satellite images. In particular, certain structures such as lane markings, road signs, crash barriers, etc. can be automatically recognized.

[0018] According to one embodiment of the computer-implemented method, radar partial images are generated based on semantic segmentation, in which radar measurements can be expected. Radar cross section (RCS) values are assigned to the pixels in the radar partial images by a machine learning algorithm.

[0019] According to one embodiment of the computer-implemented method, the pixels of the satellite image are initially two-dimensional. The radar cross section values assigned to the pixels are converted into a point cloud of radar measurements in a three-dimensional world coordinate system.

[0020] According to one embodiment of the computer-implemented method, height information is taken into account when converting two-dimensional coordinates into three-dimensional coordinates.

[0021] According to one embodiment of the computer-implemented method, the generated radar data includes a point cloud. Alternatively or additionally, the generated radar data can include a Gaussian distribution.

[0022] According to one embodiment of the computer-implemented method, a radar map is generated based on the point cloud and / or Gaussian distribution by extracting features.

[0023] According to one embodiment of the computer-implemented method, a machine learning algorithm is trained by supervised learning. The machine learning algorithm can in particular be based on a deep learning model for semantic segmentation.

[0024] According to one embodiment of the computer-implemented method, machine learning algorithms are trained based on training data through supervised learning, wherein the training data includes satellite images as input data and real radar data as output data. The real radar data corresponds to the following radar partial images: these radar partial images correspond to the projections of the radar cross-section values measured by real radar. The radar data detected in real radar measurements usually exists in the radar coordinate system. The radar data is converted from the radar coordinate system to the world coordinate system. Thereby, the radar data can be directly mapped onto the satellite image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings illustrate:

[0026] Figure 1 A schematic block diagram of a device for providing radar data according to one embodiment of the present invention.

[0027] Figure 2 A schematic block diagram for illustrating vehicle positioning based on the following radar data: generating the radar data based on satellite images;

[0028] Figure 3 A schematic block diagram for illustrating generating radar data based on satellite images;

[0029] Figure 4 A flowchart of a computer-implemented method for providing radar data according to one embodiment of the present invention;

[0030] Figure 5 A schematic block diagram of a computer program according to one embodiment of the present invention;

[0031] Figure 6 A schematic block diagram of a non-volatile computer-readable storage medium according to one embodiment of the present invention.

[0032] The numbers of the method steps are for clarity and generally should not imply a definite time sequence. In particular, multiple method steps can also be executed simultaneously. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Figure 1 A schematic block diagram of a device 1 for providing radar data according to one embodiment of the present invention is shown. The device 1 includes an input interface 11 for providing input data. The input interface can be connected to at least one external device, in particular to a satellite system or a server, by a cable connection or a wireless connection to receive input data. The input data includes satellite images. Here, the satellite images can be understood as a single satellite image or multiple satellite images. The input data can include additional information, such as altitude information or semantic labels of the satellite images.

[0034] Device 1 also has a memory 12 in which the received input data is stored. The following data can also be stored in the memory 12: This data is necessary for implementing the trained machine learning algorithm.

[0035] Device 1 also has a computing device 13 which is configured to implement the trained machine learning algorithm. The computing device 13 can include at least one of a processor, a microprocessor, an integrated circuit, an ASIC or the like. The computing device 13 accesses the input data stored in the memory 12. By using the trained machine learning algorithm, the computing device 13 identifies the following segments in the satellite image: In these segments, radar measurements may occur. Therefore, these segments correspond to the following objects or structures: The radar beam is reflected on these objects or structures. The computing device 13 can identify the pixels in the satellite image corresponding to these segments. The pixels correspond to the spatial positions where radar reflection may occur.

[0036] Here, the machine learning algorithm can already be trained based on training data, which includes satellite images as input data and real radar data as output data. The computing device 13 can be configured to perform the training of the machine learning algorithm itself. Alternatively, it can also be provided that a trained machine learning algorithm has been provided.

[0037] The computing device 13 can assign a radar cross-section value to the phase pixels corresponding to the spatial positions where radar reflection may occur. These values can also be generated based on the trained machine learning algorithm.

[0038] The computing device 13 can also be configured to convert the radar reflection values assigned to the two-dimensional pixels into a three-dimensional point cloud, where height information can be considered.

[0039] Finally, the computing device 13 can be configured to create a radar map based on the three-dimensional point cloud, where features can be extracted.

[0040] The radar map can be provided in any display manner. Instead of or in addition to the point cloud, for example, a Gaussian distribution can be generated.

[0041] In addition, the input interface 11 can receive real radar data generated by the radar sensor of the motor vehicle. The computing device 13 is configured to compare the received real radar data with the synthetic radar data generated based on the satellite image. Based on this comparison (for example, by registering the real radar data and the synthetic radar data), the motor vehicle can be positioned. The computing device 13 can output the position of the motor vehicle. The driver assistance system can in particular control the functions of the motor vehicle based on the positioning of the motor vehicle.

[0042] The device 1 further includes an output interface 14 for outputting the synthetic radar data or the positioning of the motor vehicle. The output interface 14 can be the same as the input interface 11.

[0043] Figure 2 A schematic block diagram for illustrating the positioning of a motor vehicle based on the following radar data: The radar data has been generated based on a satellite image. A satellite image 21 is provided. A conversion algorithm 22 is applied to the satellite image 21, and the conversion algorithm generates synthetic radar data 23. The conversion algorithm is based on the trained machine learning algorithm described above. Real radar data 24 generated by the radar sensor of the motor vehicle is also provided. The positioning 25 of the motor vehicle is performed based on a comparison. The pose 26 of the motor vehicle can be obtained based on the positioning of the motor vehicle.

[0044] Figure 3 A schematic block diagram for illustrating the generation of radar data based on a satellite image is shown. A satellite image 31 is provided, in which certain structures, such as lane boundaries, can be identified. Through semantic segmentation, pixels or parts corresponding to the structures that reflect the radar beam are identified by means of a trained machine learning algorithm. In addition, radar cross-section values are assigned to the pixels. Thereby, a point cloud 33 is generated. A radar map 34 is generated by extracting features.

[0045] Figure 4 A flowchart of a computer-implemented method for providing radar data according to an embodiment of the present invention is shown. This method can be executed by the above-mentioned device 1. Conversely, the device 1 can be configured to execute the method described below.

[0046] In a first method step S1, input data is received, and the input data includes a satellite image.

[0047] In method step S2, a machine learning algorithm is trained. For this purpose, a certain satellite image is provided as input data and radar data corresponding to the satellite image is provided as output data to perform supervised learning. Thus, the trained machine learning algorithm can be applied to any satellite image. Optionally, the radar data can be preprocessed for training. For example, annotations can be performed by the user. Automatic annotations can also be set. Thereby, good global positioning of the radar data can be achieved with reduced work overhead.

[0048] In method step S3, synthetic radar data is generated by applying the trained machine learning algorithm to the input data. For this purpose, first, the radar parts or pixels corresponding to the objects that reflect the radar radiation can be identified. In addition, radar cross-section values are described for these pixels. For example, three-dimensional radar data in the form of a point cloud and / or a Gaussian distribution can be generated based on height information. A radar map can also be generated by extracting features, and the radar map additionally includes information about certain structures.

[0049] In method step S4, the generated radar data is output.

[0050] This method can also be used to locate a motor vehicle. To this end, in a fifth method step S5, real radar data is generated by a radar sensor of the motor vehicle.

[0051] In method step S6, the real radar data generated based on the radar sensor is compared with the synthetic radar data generated based on satellite images. In particular, registration can be performed, that is, the real radar data is rotated and moved such that the real radar data coincides or overlaps with the synthetic radar data as accurately as possible.

[0052] Thereby, the motor vehicle can be located in a further method step S7. In particular, the pose of the motor vehicle can be calculated. By using the positioning of the motor vehicle, certain driving functions can be automatically controlled.

[0053] Figure 5 A schematic block diagram of a computer program 5 according to an embodiment of the present invention is shown. The computer program 5 includes executable program code 51 which, when implemented on a computer, causes the computer to control or execute the above-described computer-implemented method for providing radar data.

[0054] Figure 6 A schematic block diagram of a non-volatile computer-readable storage medium 6 according to an embodiment of the present invention is shown. The storage medium 6 includes executable program code 61 which, when implemented on a computer, causes the computer to control or execute the above-described computer-implemented method for providing radar data.

Claims

1. A computer-implemented method for providing radar data, the method having the following steps: Receiving (S1) input data, wherein, the input data includes satellite images; Generating (S3) radar data by using a trained machine learning algorithm, wherein the trained machine learning algorithm is applied to the input data; Outputting (S4) the generated radar data, wherein the method further has the following steps: Generating (S5) radar data by a radar sensor of a motor vehicle; Comparing (S6) the radar data generated based on the radar sensor of the motor vehicle with the radar data generated based on the satellite image; Locating (S7) the motor vehicle based on the comparison.

2. The method according to claim 1, wherein, applying the trained machine learning algorithm to the input data includes: performing semantic segmentation on the satellite image.

3. The method according to claim 1 or 2, wherein, the generated radar data includes point clouds and / or Gaussian distributions.

4. The method according to claim 3, wherein, generating a radar map by extracting features based on the point clouds and / or Gaussian distributions.

5. The method according to claim 1 or 2, the method further having the following steps: Training (S2) the machine learning algorithm by supervised learning.

6. The method according to claim 5, wherein, training the machine learning algorithm by supervised learning based on training data, wherein the training data includes satellite images as input data and real radar data as output data.

7. A device (1) for providing radar data, the device having: An input interface (11) configured to receive input data, wherein, the input data includes satellite images; A computing device (13) configured to generate radar data by using a trained machine learning algorithm, wherein the trained machine learning algorithm is applied to the input data; An output interface (14) configured to output the generated radar data, wherein, the input interface (11) is further configured to receive radar data generated by a radar sensor of a motor vehicle; the computing device (13) is further configured to compare the radar data generated based on the radar sensor of the motor vehicle with the radar data generated based on the satellite image; wherein the motor vehicle is located based on the comparison.

8. A computer program (5) which, when implemented on a computer, causes the computer to control the implementation of the steps of the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium (6) storing an executable computer program which, when implemented on a computer, causes the computer to control the implementation of the steps of the method according to any one of claims 1 to 6.

Images