Method for creating a digital map
The combination of aerial images and vehicle trajectory data using a convolutional neural network addresses the limitations of existing digital map creation methods by enhancing lane geometry and topology detection, resulting in accurate and reliable digital maps for autonomous vehicle navigation.
Patent Information
- Application Number
- PCT/EP2024/060635
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-28
AI Technical Summary
Existing methods for creating digital maps for autonomous vehicles are limited by incomplete lane geometry and topology detection due to missing or unclear road markings, especially in complex environments like road intersections, where physical lane markings are not considered, leading to inaccuracies and incomplete representations.
A method that combines aerial images with segmented vehicle trajectories using a trained convolutional neural network to enhance lane geometry and topology detection, allowing for robust lane marking extraction and topology derivation, even in areas with occlusions or ambiguous markings, by aligning and processing both data sources to create a high-resolution digital map.
This approach significantly improves the accuracy and completeness of digital maps, enabling reliable navigation and autonomous vehicle operations by integrating vehicle trajectory data with aerial imagery, reducing errors and enhancing scalability through automated processing.
Smart Images

Figure EP2024060635_28112024_PF_FP_ABST
Abstract
Description
[0001] Procedure for creating a digital map
[0002] The invention relates to a method for creating a digital map according to the preamble of claim 1.
[0003] The invention further relates to a use of such a digital map.
[0004] High-resolution digital maps, also known as high-definition maps, and the information they contain on lane geometry and topology are of great importance for automated, for example, highly automated or autonomous ferry operations, as they provide essential information about relevant roads and the vehicle's surroundings beyond the detection limits of on-board sensors. Such digital maps therefore contribute significantly to the safety and reliability of automated vehicles.
[0005] Lane geometries stored in the digital maps provide the vehicle with the physical layout of a road, as well as the number and actual global positions of specific lanes on the road. Based on this information, a system designed for automated ferry operation of the vehicle can make decisions about future maneuvers, a direction of travel, and necessary lane changes, and plan a specific future trajectory.
[0006] Lane topologies stored in the digital maps describe how the lanes relate to each other and how they change with respect to intersections, junctions, and other road features. This information is also important because it helps the algorithms of the system intended for the vehicle's automated ferry operation navigate safely and efficiently through intersections and other complex traffic situations.
[0007] CN 112418037 A describes a method and system for detecting lane markings in a satellite image. The method comprises the following steps:
[0008] - Inputting a satellite image to be recognized into a trained LaneNet network model to output a satellite image marked with coordinate points of lane lines,
[0009] - Grouping the coordinate points of output lane lines to obtain a coordinate point belonging to each lane line,
[0010] - Adjusting according to the coordinate point belonging to each lane line to obtain each lane line, wherein the LaneNet network model is formed in a training with a training set comprising a plurality of satellite images and lane line coordinate points marked in each satellite image, and
[0011] - Using the LaneNet network model to identify lane lines in an acquired satellite image and create a navigation map.
[0012] From DE 102022 001 030 B3 a method for creating a digital map is known in which sensor data is recorded by a plurality of vehicles while driving in a geographical area and in which a road map is created from the recorded sensor data by means of machine learning methods, which road map contains a description of driving path segments that have been recorded in the past.
[0013] From DE 102019215 522 A1 a method for updating attributes of a digital map based on aerial photographs is known, in which attributes of roadways in the respective segments are determined from segments of the aerial photographs by means of a trained neural network and are used to update or complete attributes stored in the map.
[0014] The invention is based on the object of providing a novel method for creating a digital map and a use of such a digital map.
[0015] The object is achieved according to the invention by a method which
[0016] Claim 1, and by a use having the features specified in claim 10. Advantageous embodiments of the invention are the subject of the dependent claims.
[0017] In a process for creating a digital map, aerial photographs of a given geographical area are captured.
[0018] According to the invention, trajectories with the courses of the geographical positions of the respective vehicles are recorded by a plurality of vehicles during travel in the geographical space, with segmented trajectory groups being formed from the recorded trajectories. The segmented trajectory groups and the aerial photographs are aligned with each other, and lane segments of roadways are determined from the aerial photographs and the segmented trajectory groups aligned with them using a trained convolutional neural network and incorporated into the digital map.
[0019] Lane markings can be extracted from aerial images, such as satellite images or aerial photographs. However, there is a risk that missing, unclear, or ambiguous road markings can result in unclear representations of the road layout using aerial images. This can lead to misjudgments when determining lane geometries and lane topologies. For example, obscurations by bridges, tunnels, buildings, peripheral developments, or marginal vegetation can lead to incompleteness in the derivation of lanes. Deriving a lane topology based solely on actual vehicle trajectories only considers the driving behavior of the vehicles. Physical lane markings, however, are not taken into account. This, in turn, leads to limitations in the recognition of the actual lane geometry and the complete lane topology.
[0020] By combining the aerial photographs with the correlating segmented trajectory groups, this method can achieve a particularly high level of accuracy and correctness in the digital map. This is particularly possible even in the case of missing or ambiguous road markings. This combination can be fully automated through the use of machine learning algorithms and additional processing steps, particularly through the use of trained convolutional neural networks.Particularly in cases where road markings are missing or unclear, such as at road intersections or in poor road conditions, the combination of the aerial images with the correlating segmented trajectory groups in lane detection enables information about the lane geometry and topology to be extracted from the trajectory paths and combined with the aerial images, so that a complete representation of roads with lanes is generated despite the missing information in the aerial images.
[0021] Combining the aerial images with the correlating segmented trajectory groups also makes it possible to bridge shadows and occlusions in the aerial images. The impairments caused by shadows, structural occlusions, or other obscuring objects, such as vehicles, in the aerial images, which can lead to incorrect and incomplete detection of lane markings and road edges, can be remedied by adding the recorded trajectory paths, as the geographical positions of the vehicles are recorded independently of lighting conditions or other influences and can thus be projected into the corresponding aerial image.
[0022] The additional use of trajectory paths also makes it possible to accurately derive lane topologies—that is, compliant lane connections, turning maneuvers at intersections, and lane changes. This is not possible using aerial photographs alone, as restrictions, such as those caused by traffic signs, cannot be detected in bird's-eye views. The combination of these data with trajectory paths, and thus the combinations actually traveled by vehicles, enables robust detection of lane topologies.
[0023] Furthermore, the use of the learning-based convolutional neural network and its generalization capability enables even exceptional cases to be robustly detected and classified or inferred accordingly. Thus, unlike heuristic or classical image recognition methods, not all exceptional cases need to be known a priori, but can be resolved by the method itself based on previously learned rules.
[0024] Furthermore, the present method leads to high scalability. In addition to fully automated execution after successful training, processing data in regions segmented by the segmented trajectory groups enables a decomposition of the subproblem, allowing the map creation process to be parallelized. This significantly reduces the time required compared to conventional mapping processes.
[0025] According to one possible embodiment of the method, lane segments determined for neighboring geographical areas are combined to form lane paths of roadways on the digital map. This allows a global digital map to be created.
[0026] According to another possible embodiment of the method, the digital map is created on a backend server. This enables centralized creation of the digital map, minimizing the computing power required for data processing in individual vehicles. Furthermore, the created digital map can be made available centrally to a large number of vehicles.
[0027] According to another possible embodiment of the method, the geographical positions of the trajectories are optimized using a clustering process. This allows inaccuracies resulting, for example, from so-called multi-path effects in the temporal sequence of geographical positions forming the trajectories to be corrected. This results in increased accuracy of the generated digital map.
[0028] According to another possible embodiment of the method, the geographical positions of the trajectories are aligned with each other in the lateral and longitudinal directions. The alignment is performed based on vehicle observations of localization objects by performing error correction based on identical geographical localization features observed by multiple vehicles. This optimization of trajectory paths allows for a significantly better representation of actual lane paths.
[0029] According to another possible embodiment of the method, the trajectory groups are segmented based on the direction of travel and / or speed of the vehicles. This segmentation increases the performance of the learning-based method implemented using the convolutional neural network. Segmentation can be achieved by highlighting patterns in the input data and reducing the complexity of this input data for the learning-based method. Segmentation can be performed very accurately and efficiently using a clustering method, such as a so-called DBSCAN method (DBSCAN = Density-Based Spatial Clustering of Applications with Noise).
[0030] According to another possible embodiment of the method, the segmentation of the trajectory groups is performed using a regression procedure based on a mathematical model. Using such a regression procedure, the complexity of the input data can be further reduced and a significant data reduction achieved.
[0031] According to another possible embodiment of the method, the aerial photographs are preprocessed using a radiometric calibration, whereby certain features in the aerial photographs are highlighted during the preprocessing. This enables improved processing of the aerial photographs in the learning-based method.
[0032] According to a further possible embodiment of the method, the alignment of the trajectory groups and the aerial photographs to one another is carried out based on a respective geographical location and / or respective elevation information in the trajectories of the trajectory groups and / or localization features of the trajectories of the trajectory groups detected in the aerial photographs. This alignment can improve the combination of both data sources into a single data representation. By using elevation information in the trajectories, for example, optical distortions in the aerial photographs can be compensated for. Furthermore, detected localization features of the trajectories, which can also be found in the aerial photographs, such as lane markings, can be used to align the trajectory groups based on the aerial photographs. This can also prevent orientation deviations, among other things.
[0033] When using a digital map created using the aforementioned method, it is intended to be used in a vehicle to perform a navigation function and / or to perform automated ferry operation. By using a digital map thus created with high accuracy and accuracy, the availability and reliability of vehicle navigation and of automated, in particular highly automated or autonomous, ferry operation of the vehicle can be significantly improved. Exemplary embodiments of the invention are explained in more detail below with reference to drawings.
[0034] Showing:
[0035] Fig. 1 schematically shows an aerial photograph of a road intersection with a plurality of vehicles and trajectories,
[0036] Fig. 2 schematically shows an aerial photograph of the intersection according to Figure 1 with trajectory groups before optimization and the aerial photograph of the intersection according to Figure 1 with trajectory groups after optimization,
[0037] Fig. 3 schematically shows an aerial photograph of the intersection according to Figure 1 with trajectories after a segmentation of trajectory groups and
[0038] Fig. 4 shows a schematic flow of a method for creating a digital map.
[0039] Corresponding parts are provided with the same reference numerals in all figures.
[0040] Figure 1 shows a plan view of a road intersection with a plurality of vehicles F1 to Fn and trajectories T1, T2 driven by several of these vehicles F1 to Fn.
[0041] In one possible embodiment of a method according to the invention for creating a digital map K, shown in more detail in Figure 4, it is provided to combine aerial photographs L of a given geographical area with correlating segmented trajectory groups TG1 to TGy, which are shown in more detail in Figures 2 and 3. This is carried out in a learning-based method, in particular by means of a convolutional neural network (CNN), shown in more detail in Figure 4.
[0042] For this purpose, aerial images L of the specified geographical area are acquired. This acquisition is carried out, for example, using satellite images and / or using at least one camera as aerial photographs. The geographical area is represented from a bird's eye view using an aerial image L. The aerial image L can be available, for example, in an RGB format or another suitable format.
[0043] Furthermore, trajectories T1, T2 with courses of geographical positions of the respective vehicle F1 to Fn are recorded by means of a plurality of vehicles F1 to Fn when driving in the geographical area.
[0044] Subsequently, data processing is performed for both data sources, i.e., for the acquired aerial photographs L and the recorded trajectories T1, T2. Data processing is performed separately for both data sources.
[0045] The data preparation is explained in more detail using Figures 2 and 3.
[0046] Figure 2 shows the aerial image L of the intersection according to Figure 1 with a plurality of trajectories T1 to Tm before optimization (left illustration) and the aerial image L of the intersection according to Figure 1 with trajectories T1 to Tm after optimization (right illustration).
[0047] The trajectories T1 to Tm of the vehicles F1 to Fn consist at least essentially of a temporal sequence of geographical positions, for example GPS positions, of the vehicles F1 to Fn, which together form a trajectory group TG1 to TGy, also referred to as a trajectory track or trace.
[0048] Such trajectory groups TG1 to TGy can be inaccurate for various reasons, such as so-called multi-path effects. To improve this inaccuracy, the geographical positions of the trajectories T1 to Tm of the trajectory groups TG1 to TGy are optimized in an optimization of the trajectory groups TG1 to TGy. This can be performed, for example, using a method from the class of so-called clustering methods. Furthermore, the geographical positions are aligned to one another in the longitudinal and lateral directions. Existing vehicle observations of localization objects are used, and error correction is performed based on identical observed geographical localization features. Through this optimization of the trajectory groups TG1 to TGy, actual lane paths can be represented much more accurately, as illustrated in the right-hand illustration.To increase the performance of the learning-based method, patterns in the input data are highlighted during preprocessing, thus reducing the complexity of the input data. This is achieved by segmenting the trajectory groups TG1 to TGy, as illustrated in Figure 3.
[0049] This segmentation is performed based on defined characteristics, such as the direction of travel and / or speed of the vehicles F1 to Fn, which each record the trajectories T1 to Tm to create the trajectory groups TG1 to TGy. The segmentation can be performed, for example, using additional clustering methods, such as the so-called DBSCAN method. Data reduction achieved through the application of such clustering methods can be supplemented by the application of regression techniques, in which a mathematical model is used to segment the trajectory groups TG1 to TGy.
[0050] Segmentation can provide additional features to a predictive model used in the learning-based method, thus improving its performance. In possible embodiments, such clusters are found in the data itself using the learning-based method.
[0051] Furthermore, the aerial images L are also prepared for use with the learning-based method. To highlight certain features in the aerial images L, they can be preprocessed in a separate preprocessing step using methods such as radiometric calibration to optimize subsequent processing by the learning-based method.
[0052] Following the preprocessing of the input data from both data sources, i.e. following the segmentation of the trajectory groups TG1 to TGy and the possible processing of the aerial photographs L, both data sources are combined into one data representation.
[0053] For this purpose, the trajectory groups TG1 to TGy and the aerial photographs L are first aligned to each other. In addition to a simple alignment based on a respective geographical location, further alignment steps can be used. For example, elevation information from the trajectory groups TG1 to TGy can be used to compensate for optical distortions in the aerial photograph L. Furthermore, similar to the described preprocessing of the trajectory groups TG1 to TGy, recognized localization features of the trajectory groups TG1 to TGy, which can also be found within the aerial photographs L, such as lane markings, can be used to align the trajectory groups TG1 to TGy based on the aerial photographs L. This also helps to avoid orientation deviations, among other things.
[0054] The joint representation thus generated serves later as the data basis for the learning-based procedure, which is intended to interfere with the lane geometries FG and lane topologies FT shown in more detail in Figure 4.
[0055] The combination of both data sources creates a comprehensive dataset of images containing both an aerial photograph L of the streets and the projected trajectory groups TG1 to TGy. Data processing and preparation are completed by dividing the dataset into smaller, homogeneous data points, each of which contains individual street sections.
[0056] To actually derive lane geometries (FG) and lane topologies (FT) from the processed data, a learning-based model from the field of AI-based image understanding, such as a Convolutional Neural Network (CNN), is used. This is shown in Figure 4.
[0057] Here, inference of the desired data is formulated as a so-called instance segmentation problem, in which a convolutional neural network (CNN) is tasked with recognizing each instance of a trajectory group TG1 to TGy in the aerial images L and outputting it accordingly. The choice of the neural network motivates the previously described data processing in the form of processed aerial images L and projected trajectory groups TG1 to TGy as input data. These are used to predict the lane geometries FG and lane topologies FT in the digital map K as output information.
[0058] The use of learning-based methods always requires training the model with suitable, pre-selected data. Image understanding falls into the area of so-called supervised learning, which requires so-called ground truth data for successful model training. This data has previously been attributed by a human and is used to adapt parameters of the neural network. The convolutional neural network (CNN) first solves the problem of detecting lane segments in the input image, i.e., the aerial image L, as well as their actual geographical extent. To fully generate a lane layer, the lane topology (FT), i.e., a relationship between the detected lane segments, must be determined. Two approaches are suitable for this. First, the learning-based model can be supplemented so that the relationship is already solved within the model.Alternatively, by considering the directions of travel of the corresponding trajectory group TG1 to TGy in a subsequent heuristic post-processing step, a topology of the detected lane segments in the aerial image L can be derived. Both variants can be fully automated and integrated into the process.
[0059] A final processing step for generating the digital map K involves the retransformation of the image data into a global geographical representation and the merging of the respective image segments into a global digital map K. In this process, the results of the split aerial images L are recombined at image boundaries. This means that lane segments determined for neighboring geographical areas are combined to form lane paths of the roadways of the digital map K.
Claims
Patent claims 1. A method for creating a digital map (K), wherein aerial photographs (L) of a given geographical area are acquired, characterized in that - by means of a plurality of vehicles (F1 to Fn) during travel in the geographical area, trajectories (T1 to Tm) with the courses of geographical positions of the respective vehicle (F1 to Fn) are recorded, - segmented trajectory groups (TG1 to TGy) are formed from the recorded trajectories (T1 to Tm), - the segmented trajectory groups (TG1 to TGy) and the aerial photographs (L) are aligned with each other and - lane segments of roadways are determined from the aerial photographs (L) and the segmented trajectory groups (TG1 to TGy) aligned to them using a trained convolutional neural network (CNN) and are included in the digital map (K).
2. Method according to claim 1, characterized in that lane segments determined for adjacent geographical areas are combined to form lane courses of roadways of the digital map (K).
3. Method according to claim 1 or 2, characterized in that the creation of the digital map (K) is carried out on a backend server.
4. Method according to one of the preceding claims, characterized in that the geographical positions of the trajectories (T1 to Tm) are optimized by means of a clustering method.
5. Method according to one of the preceding claims, characterized in that - the geographical positions of the trajectories (T 1 to T m) are aligned in lateral and longitudinal directions, whereby - the alignment is carried out as a function of vehicle observations of localisation objects by performing an error correction based on the same geographical localisation features observed by several vehicles (F1 to Fn).
6. Method according to one of the preceding claims, characterized in that the segmentation of the trajectory groups (TG1 to TGy) is carried out based on a direction of travel and / or speed of the vehicles (F1 to Fn).
7. Method according to claim 6, characterized in that the segmentation of the Trajectory groups (TG1 to TGy) based on a mathematical model in a regression procedure.
8. Method according to one of the preceding claims, characterized in that the aerial photographs (L) are pre-processed in a radiometric calibration, wherein certain features in the aerial photographs (L) are highlighted in the pre-processing.
9. Method according to one of the preceding claims, characterized in that the alignment of the Trajectory groups (TG1 to TGy) and the aerial photographs (L) to each other based on a respective geographical location and / or respective height information in the trajectories (T1 to Tm) of the trajectory groups (TG1 to TGy) and / or in the Localization features of the trajectories (T1 to Tm) of the trajectory groups (TG1 to TGy) detected in aerial photographs (L) are used.
10. Use of a digital map (K) created in a method according to one of the preceding claims in a vehicle (F1 to Fn) to carry out a navigation function and / or to carry out an automated ferry operation.
Citation Information
Patent Citations
Lane line identification method and system in satellite picture, electronic equipment and storage medium
CN112418037A
Method for updating attributes in a digital map
DE102019215522A1
Method for determining a navigation route for automated driving of a vehicle
DE102022001030B3
Method and apparatus for generating map geometry based on a received image and probe data
US20160239983A1
Method and apparatus for providing trajectory bundles for map data analysis
US20180066957A1
Cited By
Map processing method, map processing apparatus and computer program product
CN119879960A
