Method for creating high-resolution environmental maps for a vehicle with an autonomous driving function

Abstract

Method for creating high-resolution environmental maps for a vehicle with an autonomous driving function, the method comprising: - Providing (S1) environmental image data of a recognition system of the vehicle, wherein the recognition system has an environmental sensor for detecting a vehicle environment during a journey of the vehicle; - Providing (S2) domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Creating (S3) the high-resolution environment maps of the vehicle environment by supplementing the environment image data using the provided domain knowledge of the trained knowledge graph.

Description

[0001] The invention relates to methods and devices for creating high-resolution environmental maps for a vehicle with an autonomous driving function. State of the art

[0002] Autonomous driving (AD) systems represent the pinnacle of technological innovation in the automotive sector, promising a revolution in mobility and aiming to drastically improve safety and efficiency on our roads. A key component essential for the optimal functioning of these systems is high-definition (HD) maps. These maps provide autonomous vehicles with detailed information about the driving environment, enable precise localization, and support decision-making in complex traffic situations. However, despite their importance, HD maps pose a significant hurdle due to their high creation costs, the challenge of keeping them up-to-date, and their limited availability in certain regions. Furthermore, the accuracy of these maps can be limited in dynamic environments where road conditions change rapidly.

[0003] In this context, the possibility of creating and continuously updating HD maps using the perception systems of autonomous vehicles offers a promising approach. By allowing vehicles to sense their surroundings in real time, collect data, and use this information to refine the maps, many of the existing challenges could be overcome.

[0004] The scientific paper by Y. Liu, Y. Yuan, Yue Wang, Y. Wang, H. Zhao, "VectorMapNet: End-to-End Vectorized HD Map Learning," https: / / arxiv.org / abs / 2206.08920, submitted to ICLR 2023, 2022, introduces an approach that addresses the problem of creating high-resolution (HD) maps from on-board sensors for autonomous driving. The method uses observations from on-board sensors, such as cameras, lidar, or radar, to predict polylines representing various map features such as lanes, pedestrian crossings, or road dividers. The method essentially consists of three steps: feature extraction, in which image features are extracted using a ResNet CNN, followed by mapping the image to a bird's-eye view (BEV) using inverse perceptual mapping (IPM). Furthermore, lidar observations are processed into PointPillars with dynamic voxelization.Image and lidar features are also linked and further processed by a two-layer CNN. A map element detector is also disclosed. A transformer set prediction detector (DETR) detects element keypoints, which later form the polylines of the map elements. A deformable attention module is also used, in which each element query has a unique location. The prediction head has two MLPs that decode element queries into element keypoints and their class labels. Furthermore, a polyline generator is used in a third step. The polyline generator generates detailed geometric shapes of map elements by modeling a distribution across the vertices of the map elements and BEV features.

[0005] Further state of the art is known from the scientific publication L. Moi, et al. HDMapGen: Hierarchical Graph Generative Model of High Definition Maps, IEEE CVPR, 2021.

[0006] The invention is therefore based on the object of specifying at least one improved method and / or device for creating high-resolution environmental maps for a vehicle with an autonomous driving function.

[0007] The problem is solved by a method according to the features of patent claim 1. The problem is solved by a method according to the features of patent claim 2. The problem is solved by a device according to the features of patent claim 10. The problem is solved by a device according to the features of patent claim 11. Disclosure of the invention

[0008] According to a first aspect, a method for creating high-resolution environmental maps (also called HD maps) for a vehicle with an autonomous driving function (also called AD vehicles) is specified, the method comprising: - Providing environmental image data of a recognition system of the vehicle, wherein the recognition system has an environmental sensor for detecting a vehicle environment while the vehicle is traveling; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Creating high-resolution maps of the vehicle environment by supplementing the environmental image data using the domain knowledge provided by the trained knowledge graph.

[0009] According to a second aspect, a method for creating high-resolution environmental maps for a vehicle with an autonomous driving function is provided, the method comprising: - optionally providing the surrounding image data of a recognition system of the vehicle, wherein the recognition system has an environment sensor for detecting a vehicle environment while the vehicle is traveling; - Providing high-resolution maps of the vehicle's surroundings; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Plausibility check and / or supplementation of the high-resolution environment maps and / or optionally the environment image data of the vehicle environment using the provided domain knowledge of the trained knowledge graph.

[0010] It is understood that the steps according to the invention, as well as other optional steps, do not necessarily have to be performed in the order shown, but can also be performed in a different order. Furthermore, further intermediate steps can be provided. The individual steps can also comprise one or more substeps, without thereby departing from the scope of the inventive method according to the first or second aspect.

[0011] According to a third aspect, a device for creating high-resolution environmental maps for a vehicle with an autonomous driving function is provided, the device comprising an evaluation and computing device which is designed to carry out the following steps: - Providing environmental image data of a recognition system of the vehicle, wherein the recognition system has an environmental sensor for detecting a vehicle environment while the vehicle is traveling; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Creating high-resolution maps of the vehicle environment by supplementing the environmental image data using the domain knowledge provided by the trained knowledge graph.

[0012] According to a fourth aspect, a device for creating high-resolution environmental maps for a vehicle with an autonomous driving function is provided, the device comprising an evaluation and computing device which is designed to carry out the following steps: - optionally providing the surrounding image data of a recognition system of the vehicle, wherein the recognition system has an environment sensor for detecting a vehicle environment while the vehicle is traveling; - Providing high-resolution maps of the vehicle's surroundings; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Plausibility check and / or supplementation of the high-resolution environment maps and / or optionally the environment image data of the vehicle environment using the provided domain knowledge of the trained knowledge graph.

[0013] The statements made for the method apply accordingly to the device. It is understood that linguistic modifications of procedurally formulated features can be reformulated for the device according to common linguistic practice, without such formulations having to be explicitly listed here.

[0014] The present invention represents map elements using a knowledge graph (KG) and can thus represent a large number of map elements. Furthermore, the knowledge graph preferably also contains relationships between entities. The knowledge graph can be based on a standardized ontology, e.g., the ASAM OpenX ontology. Therefore, the knowledge graph can represent highly detailed map elements that are important for autonomous driving or autonomous driving functions, e.g., various types of lane dividers, road signs, traffic lights, pedestrian walkways, parking areas, debris, traffic cones, road construction elements, and / or stop lines.

[0015] The knowledge graph learns the distribution of map elements, preferably from a training dataset of labeled map elements. The training dataset can be derived from or provided by autonomous driving datasets. By incorporating the domain knowledge of the knowledge graph, the construction of unrealistic road elements is reduced, e.g., straightness of lanes, lane dividers, road boundaries, maximum road curvature, minimum and / or maximum lane width, etc.

[0016] The knowledge graph is constructed from a training dataset to learn the spatial relationship between map elements and to ensure that only spatially plausible elements are constructed, e.g., that pedestrian crossings start and end at opposite road boundaries, that road dividers are located between two adjacent lanes, and / or that road boundaries are located at the outermost lanes.

[0017] It is advantageous for the creation of high-resolution environmental maps if a vehicle recognition system, for example using a lidar sensor, detects the vehicle's surroundings and segments and classifies these surroundings into individual map elements. The segmentation can be carried out pixel by pixel from the captured images. The classification can be carried out, for example, into classes such as streets, roadsides, buildings, pedestrian walkways, etc. By incorporating domain knowledge, a plausibility check of the classified map elements can be carried out, in particular by assigning element-specific probabilities. For example, contextual knowledge recorded in the knowledge graph, for example about whether the vehicle is in a city, a country road or on a motorway, can also be included. For example, if the vehicle is on the motorway and is detected by the environmental sensor orIf the segmentation and classification algorithm used segments and classifies a pedestrian crossing as a map element, this result can be incorporated into the further creation of the high-resolution environmental map by incorporating domain knowledge from the knowledge graph and assigning a low probability for the actual presence of a pedestrian crossing in the subsequent map creation. The high-resolution environmental map created in this way becomes more accurate. A similar approach can also be achieved, for example, by checking the dimensions of detected road or lane widths.

[0018] At least the method according to the first aspect or the corresponding device, which can be part of a system, can be used for the automatic creation of HD maps within a vehicle with an autonomous driving function (also called an AD vehicle). An AD vehicle creates an HD map of every location the vehicle travels to. The created HD maps can preferably also be transmitted online to a location, e.g., a server or a cloud, where the HD maps of a plurality of AD vehicles are received and collected, in order to create, in particular, a complex and highly informative HD map from the plurality of individual HD maps.

[0019] At least the method according to the first aspect or the corresponding device can be used in automated driving vehicles that have no or only limited access to HD maps. This is because the present method allows the automated driving vehicle to create HD maps immediately or in real time while driving, based on the environmental image data from the recognition or perception system and the description system provided in the form of a knowledge graph. Thus, online HD map creation can be provided for automated vehicles of levels 2 to 5, which may previously only use SD cards.

[0020] At least the method according to the second aspect or the corresponding device can also be used to provide a more current, more accurate, and / or an alternative information source for the final HD map. Preferably, online HD map creation can be specified for automated vehicles from levels 2 to 5, which, for example, use already stored HD maps. In this case, a second or alternative knowledge source with HD map information can be provided, which may be more accurate or more current. Furthermore, the present method can also provide the HD maps for areas not covered by the stored HD maps, or adapt them by supplementing existing HD maps. Thus, online HD map creation is also provided for creating and supplementing area-wide offline HD maps.

[0021] According to one embodiment, the environmental image data are plausibilized and / or supplemented on the basis of standard definition (SD) topology data of the vehicle environment.

[0022] The local topology map can consist of a map typically used in navigation systems, from which road layout and directions of travel can be derived. The map is preferably in the form of standard definition topology data. The SD card topology is preferably used to provide guidance lane topologies, directions of travel, and other information for creating and verifying the HD map.

[0023] According to one embodiment, the environmental image data is preprocessed by at least one of the following steps: - Segmenting image data of the vehicle environment captured by the environment sensor into elements and classifying the elements into predefined classes which correspond in particular to an ontology of the knowledge graph; - Transferring the segmented and classified elements of the vehicle environment into a bird's eye view; - Extracting individual elements of the vehicle environment based on the segmented image data; and / or - Connecting the extracted elements to create environment maps, in particular by checking plausibility using the domain knowledge provided by the knowledge graph.

[0024] The conversion to a bird's eye view can also be done with the captured image data before segmentation and classification.

[0025] In this case, it is proposed to construct an HD map online in an AD vehicle. According to the first and third aspects, this construction is preferably carried out based on environmental image data from the AD perception system, possibly based on an (SD) map of the local topology, and based on the domain knowledge of the knowledge graph of the trained map. The AD perception system is preferably used to detect moving objects, such as vehicles and pedestrians, but also for the detection of map elements, such as lanes, pedestrian crossings, etc. Preferred image segmentation is used to divide all elements into defined classes. A local topology map preferably serves as a guide for the creation of HD maps by containing the most important information about the road topology, direction of travel, curves, etc.The trained map knowledge graph is used to provide information about typical special, relational, topological, area-dependent, and statistical distributions of map elements. It helps construct high-fidelity HD maps.

[0026] The HD map construction process preferably incorporates the individual map element candidates from the recognition or perception system and validates their existence in subsequent steps. This is preferably done by retrospectively connecting map elements and hierarchically verifying their existence based on the current location. Temporal information can be used, if necessary, to filter out moving objects such as vehicles or pedestrians from the image data.

[0027] The individual map elements are preferably linked based on their type and location, which are specified by the perception system, and preferably based on the information provided by the knowledge graph. Methods such as neural networks capable of processing heterogeneous graphs such as knowledge graphs can preferably be used to find a compact vector-based representation of traffic scenes. These can be used to calculate a similarity of the proposed HD map with representations of the stored map knowledge graph.

[0028] According to one embodiment, creating the high-resolution environment maps of the vehicle environment comprises augmenting the environment maps using the domain knowledge provided by the knowledge graph.

[0029] The HD map is preferably supplemented with information that may not be visible in the image data from the environment sensor and / or not provided by the perception system. For example, traffic regulations, traffic lights, or traffic signs typically indicate a stopping zone where a vehicle should stop if it needs to swerve. Information about invisible lane separations can also be added in this way. Even at construction sites with both permanent white and temporary yellow overtaking markings, the method or device can thus determine whether a lane separation is valid.

[0030] According to one embodiment, the environment sensor comprises a lidar sensor and / or a radar sensor and / or a camera.

[0031] The vehicle's perception system may include a video camera, a radar sensor, a lidar sensor, or other sensors. The vehicle's surroundings may also be captured solely by video sensors or video cameras. The perception system detects objects in the captured driving scene while the vehicle is moving and segments the entire image, preferably frame by frame, into predefined classes, particularly according to the ontology described by the knowledge graph. Furthermore, the elements are preferably displayed in a bird's-eye view (BEV), which corresponds to a high-resolution environment map representation.

[0032] According to one embodiment, the knowledge graph is trained based on a training data set of a plurality of driving scenes and / or domain knowledge.

[0033] During training, the knowledge graph preferably learns typical, legal, and / or plausible representations of map elements from a training dataset. This helps create more accurate high-resolution (HD) maps. The map knowledge graph is preferably created from a training dataset with a large number of driving scenes. An ontology of the map is preferably created. The knowledge graph preferably represents a typical distribution of map elements and their topology. It preferably serves as a guide for the online HD map creation process.

[0034] According to one embodiment, the knowledge graph is trained to establish a spatial relationship between map elements and to ensure that only spatially plausible matching elements are used to create and / or supplement the high-resolution environmental maps.

[0035] According to one embodiment, the knowledge graph comprises domain knowledge about a road type and / or a lane type and / or about a road divider and / or about a road boundary and / or about a pedestrian crossing and / or about a stopping area and / or about traffic signs and / or about traffic lights and / or about directional arrows and / or about poles and / or about barriers and / or about traffic cones and / or about buildings and / or about plants and / or about debris.

[0036] By incorporating domain knowledge stored in the trained knowledge graph, the proposed method is capable of considering many more map elements than previous approaches. While previous approaches can often only detect, segment, classify, and thus consider four element types in image data, the proposed method can consider a large number of map elements by incorporating the domain knowledge of the trained knowledge graph when creating, supplementing, or verifying the plausibility of high-resolution environmental maps.Example map elements are different lane types, such as vehicle, bicycle, sidewalk, parking lanes, different lane dividers, such as continuous, double continuous, dashed, dashed-unique, time-limited lane divider or similar, stop areas, such as those not visible but derivable from the ontology of the knowledge graph, traffic signs, traffic lights, directional arrows applied to the road, poles, barriers, traffic cones, buildings, plants, debris, etc. The proposed approach can also derive traffic rules from the recognized map details, e.g. right of way depending on road signs or similar traffic rules.

[0037] According to the invention, a control device is also claimed which is included in a vehicle with an autonomous driving function and / or a robotics system and / or an industrial machine and can be executed according to one of the present methods.

[0038] The invention also claims a computer program with program code for executing at least parts of the method according to the invention in one of its embodiments when the computer program is executed on a computer. In other words, the invention provides a computer program (product) comprising instructions that, when executed by a computer, cause the computer to execute the method / steps of the method according to the invention in one of its embodiments.

[0039] According to the invention, a computer-readable data carrier with program code of a computer program is also proposed for executing at least parts of the method according to the invention in one of its embodiments when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium comprising instructions which, when executed by a computer, cause the computer to execute the method / steps of the method according to the invention in one of its embodiments.

[0040] The described designs and further training courses can be combined as desired.

[0041] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0042] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0043] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0044] They show: Fig. 1 is a schematic flow diagram of a method according to the first aspect; Fig. 2 is a schematic flow diagram of a method according to the second aspect; and Fig. 3 a schematic block diagram of a method according to an embodiment.

[0045] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0046] Fig. 1 shows a schematic flow diagram of a method for creating high-resolution environmental maps for a vehicle with an autonomous driving function according to a first aspect.

[0047] Fig. 2 shows a schematic flow diagram of a method for creating high-resolution environmental maps for a vehicle with an autonomous driving function according to a second aspect.

[0048] The method according to the first and second embodiments can be carried out at least partially by a device 100, which for this purpose can comprise several components not shown in detail, for example, one or more provision devices and / or at least one evaluation and computing device. It is understood that the provision device can be designed jointly with the evaluation and computing device or can be different from it. Furthermore, the system can comprise a storage device and / or an output device and / or a display device and / or an input device.

[0049] The Fig. The computer-implemented method according to the first aspect shown in Figure 1 comprises, according to the invention, at least the following steps: In a step S1, environmental image data of a recognition system of the vehicle are provided, wherein the recognition system has an environmental sensor for detecting a vehicle environment while the vehicle is traveling. In a step S2, domain knowledge of the vehicle environment is provided in the form of a trained knowledge graph. In a step S3, the high-resolution environmental maps of the vehicle environment are created by supplementing the environmental image data using the provided domain knowledge of the trained knowledge graph.

[0050] The Fig. The computer-implemented method according to the second aspect shown in Figure 2 comprises, according to the invention, at least the following steps: In an optional step S10, the surrounding image data of a recognition system of the vehicle are provided, wherein the recognition system has an environment sensor for detecting a vehicle environment while the vehicle is traveling. In a step S11, high-resolution maps of the vehicle's surroundings are provided. In a step S12, domain knowledge of the vehicle environment is provided in the form of a trained knowledge graph. In a step S13, the high-resolution environment maps and / or optionally the environment image data of the vehicle environment are checked for plausibility and / or supplemented using the provided domain knowledge of the trained knowledge graph.

[0051] Fig. 3 shows a schematic block diagram of a method according to an embodiment.

[0052] Image data 300 of a vehicle's surroundings are acquired by an environment sensor, for example a lidar sensor, a radar sensor, and / or a camera. The image data 300 of the vehicle's surroundings acquired by the environment sensor are segmented into (map) elements, and the (map) elements are classified into predefined classes, which correspond in particular to an ontology of a knowledge graph 301. These segmented and classified map elements are preferably converted into a bird's-eye view representation 302. Some characteristic, individual map elements of the vehicle's surroundings are extracted based on the segmented, classified, and bird's-eye view image data, which is denoted by reference numeral 304. The environment image data or the processed image data are plausibility-checked and / or supplemented based on standard definition topology data 306 of the vehicle's surroundings.The extracted map elements are combined to create environmental maps 310. This is preferably done by validating the plausibility using the domain knowledge provided by the knowledge graph. Creating the high-resolution environmental maps 312 (HD maps) of the vehicle's surroundings involves augmenting the environmental maps 310 using the domain knowledge provided by the knowledge graph 301. This augmentation is identified by reference numeral 314.

[0053] The knowledge graph 301 is trained based on a training data set 316 of a plurality of driving scenes and / or domain knowledge 318. The knowledge graph 301 is trained to establish a spatial relationship between map elements and to ensure that only spatially plausible elements are used to create and / or supplement the high-resolution environmental maps 312. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Y. Liu, Y. Yuan, Yue Wang, Y. Wang, H. Zhao. VectorMapNet: End-to-End Vectorized HD Map Learning, https: / / arxiv.org / abs / 2206.08920

[0004] L. Moi, et al. HDMapGen: Hierarchical Graph Generative Model of High Definition Maps, IEEE CVPR, 2021

[0005]

Claims

[1] Method for creating high-resolution environment maps for a vehicle with an autonomous driving function, the method comprising: - Providing (S1) environmental image data of a recognition system of the vehicle, wherein the recognition system has an environmental sensor for detecting a vehicle environment while the vehicle is traveling; - Providing (S2) domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Creating (S3) the high-resolution environment maps of the vehicle environment by supplementing the environment image data using the provided domain knowledge of the trained knowledge graph. [2] Method for verifying the plausibility and / or supplementing high-resolution environmental maps and / or optionally environmental image data for a vehicle with an autonomous driving function, the method comprising: - optionally providing (S10) the surroundings image data of a recognition system of the vehicle, wherein the recognition system has an environment sensor for detecting a vehicle environment during a journey of the vehicle; - Providing (S11) high-resolution maps of the vehicle's surroundings; - Providing (S12) domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Plausibility check and / or supplementation (S13) of the high-resolution environment maps and / or optionally the environment image data of the vehicle environment using the provided domain knowledge of the trained knowledge graph. [3] Method according to claim 1 or 2, wherein the environmental image data are plausibilized and / or supplemented on the basis of standard definition topology data of the vehicle environment. [4] Method according to one of the preceding claims, wherein the environmental image data are preprocessed by at least one of the following steps: - Segmenting image data of the vehicle environment captured by the environment sensor into elements and classifying the elements into predefined classes which correspond in particular to an ontology of the knowledge graph; - Transferring the segmented and classified elements of the vehicle environment into a bird's eye view; - Extracting individual elements of the vehicle environment based on the segmented image data; and / or - Connecting the extracted elements to create environment maps, in particular by checking plausibility using the domain knowledge provided by the knowledge graph. [5] The method of claim 4, wherein creating the high-resolution environment maps of the vehicle environment comprises augmenting the environment maps using the domain knowledge provided by the knowledge graph. [6] Method according to one of the preceding claims, wherein the environment sensor comprises a lidar sensor and / or a radar sensor and / or a camera. [7] Method according to one of the preceding claims, wherein the knowledge graph is trained based on a training data set of a plurality of driving scenes and / or domain knowledge. [8] Method according to claim 7, wherein the knowledge graph is trained to establish a spatial relationship between map elements and to ensure that only spatially plausible matching elements are used for the creation and / or supplementation of the high-resolution environmental maps. [9] Method according to one of the preceding claims, wherein the knowledge graph comprises domain knowledge about a road type and / or a lane type and / or about a road divider and / or about a road boundary and / or about a pedestrian crossing and / or about a stopping area and / or about traffic signs and / or about traffic lights and / or about directional arrows and / or about poles and / or about barriers and / or about traffic cones and / or about buildings and / or about plants and / or about debris. [10] Device (100) for creating high-resolution environmental maps for a vehicle with an autonomous driving function, the device (100) comprising an evaluation and computing device which is designed to carry out the following steps: - Providing environmental image data of a recognition system of the vehicle, wherein the recognition system has an environmental sensor for detecting a vehicle environment while the vehicle is traveling; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Creating high-resolution maps of the vehicle environment by supplementing the environmental image data using the domain knowledge provided by the trained knowledge graph. [11] Device (100) for creating high-resolution environmental maps for a vehicle with an autonomous driving function, the device (100) comprising an evaluation and computing device which is designed to carry out the following steps: - optionally providing the surrounding image data of a recognition system of the vehicle, wherein the recognition system has an environment sensor for detecting a vehicle environment while the vehicle is traveling; - Providing high-resolution maps of the vehicle's surroundings; - Providing domain knowledge of the vehicle environment in the form of a trained knowledge graph; and - Plausibility check and / or supplementation of the high-resolution environment maps and / or optionally the environment image data of the vehicle environment using the provided domain knowledge of the trained knowledge graph. [12] Control device for a vehicle with an autonomous driving function and / or for a robotic system and / or for an industrial machine on which a method according to one of claims 1 to 10 can be carried out. [13] Computer program with program code to carry out at least parts of a method according to one of claims 1 to 10 when the computer program is executed on a computer. [14] Computer-readable data carrier with program code of a computer program for carrying out at least parts of a method according to one of claims 1 to 10 when the computer program is executed on a computer.

Images