Generating a lane level HD map

Abstract

The invention relates to a computing device (10) and a method to generate a lane level HD map, comprising the steps of receiving geometric perception data recorded during single drives of vehicles; aggregating the received geometric perception data; downsampling the location into equally sized areas, wherein each equally sized area comprises a subset of the aggregated geometric perception data; encoding the subset of the aggregated geometric perception data into a respective latent feature space; sampling of center points from driven trajectories that are used as an approximation of the center of a lane; utilizing the center points to query for left and right boundary points from the respective latent feature space to provide lane cross sections; performing link predictions conditioned on the latent feature space to derive a likelihood of two center points being connected, wherein the link predictions define a lane graph; generating a full lane model by using the lane cross sections and their connectivity by the lane graph to provide the lane level HD map.

Description

GENERATING A LANE LEVEL HD MAPFIELD OF THE INVENTION

[0001] The present invention relates to the generation of high-definition (HD) maps, in particular in the field of automobiles. More specifically, the present invention relates to a method to generate a lane level HD map for a motor vehicle. Furthermore, the present invention relates to a corresponding computer program product, as well as to a computing device.BACKGROUND INFORMATION

[0002] HD maps with accurate and up-to-date lane level information are crucial for a plethora of assistance systems and autonomous driving. For example, HD maps, a sensor perception or a combination thereof are often used to localize a vehicle with respect to an HD map in order to plan trajectories.

[0003] However, current HD map creation processes are time consuming and inefficient due to many manual steps. Moreover, HD maps often lack of up-to-dateness due to very long manual driven update cycles. Typically, HD maps are generated using vehicle sensor data and a GNSS system, wherein the vehicle sensor data only comprises geometric information with no topological relation among geometric perceptions and the GNSS accuracy may lead to very noisy and unaligned sensor data, which makes it challenging to identify a single driving lane.SUMMARY OF THE INVENTION

[0004] Therefore, it is the object of the present invention to provide a method, a computer program product as well as a computing device by which a generation of a HD map is improved.

[0005] This object is solved by a method, a computer program product as well as a computing device according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0006] One aspect of the present invention relates to a method to generate a lane level HD map, comprising the following method steps carried out by a computing device. The method comprises the steps of receiving geometric perception data recorded during single drives of vehicles at a location, aggregating the received geometric perception data from the single drives, downsampling the location into equally sized areas, wherein each equally sized area comprises a subset of the aggregated geometric perception data, encoding the subset of the aggregated geometric perception data of each equally sized area into a respective latent feature space and sampling of center points from driven trajectories of the vehicles that are used as an approximation of the center of a lane. The method further comprises utilizing the center points to query for left and right boundary points from the respective latent feature space to provide lane cross sections at the location of the prior point, performing link predictions conditioned on the latent feature space to derive a likelihood of two center points being connected, wherein the link predictions define a lane graph and generating a full lane model of the location by using the lane cross sections and their connectivity by the lane graph to provide the lane level HD map.

[0007] In other words, geometric perception data that is recorded during drives of vehicles may be transmitted to a computing device in order to generate an HD map. The geometric perception data may be measured by a respective sensor device of the vehicles during the drive on a road at a location of which the HD map is to be created. That is, the geometric perception data may be the raw sensor data or features already extracted from the raw sensor data. For example, the geometric perception data may comprise images of the road or location, in particular, camera images and / or lidar images, and / or the geometric perception data may comprise features and / or objects that are extracted from the raw sensor data, for example lane marking points. In particular, the geometric perception data may comprise sparse vehicle observations instead of densesensor measurements. In addition, the geometric perception data of the following vehicles may also be sent to the computing device to further update and improve the HD map.

[0008] The computing device may comprise one or more processors and software to execute the method steps to generate the HD map. The goal is to derive an underlying lane graph as a set of lane pairs (lane cross sections) and their connectivity. For example, the computing device may be personal computer, a laptop, a server or any other data processing device that may be used for HD map generation. In particular, the computing device may be wirelessly connected to the vehicles to receive the geometric perception data, wherein the received geometric perception data from the single drives may be aggregated at the computing device.

[0009] The computing device may then create the HD map by using the aggregated perception data, in particular by employing an artificial intelligence based HD map creation in a fully automated manner. In particular, the computing device may receive a set of driven traces and a set of observed lane boundaries that are derived from the perception data. The set of driven traces and the set of observed lane boundaries may be provided as polylines that are defined as sequences of multiple points. For this purpose, the location or area for which the map is to be created may be divided into equally sized areas or minimaps, so that each equally sized area or minimap comprises a subset of the aggregated geometric perception data. Hereby, the problem may me decomposed into multiple smaller problems, where only the subset of data may be considered that is in a small geospatial area (equally sized area). This can reduce the computational effort, whereby the results of the analyses of the individual equally sized areas can be combined again at the end.

[0010] The subset of the aggregated geometric perception data of each equally sized area may then be encoded into a respective latent feature space. A latent feature space may be a multi-dimensional space where each dimension presents a latent feature or attribute of the geometric perception data. These features may be not directly observed but may be inferred from the geometric perception data through techniques like dimensionally reduction or unsupervised learning. For example, the latent features may be characteristics of the geometric perception data that may represent patterns, like road lane markings or sidewalks, and / or the latent features may be semantic data, like for example a color of a lane marking. In particular, the encoding into the latent feature space may be performed by a pretrained artificial intelligence, wherein the artificial intelligencemay be pretrained with geometric perception data from previous drives, in particular with features or objects extracted from camera images of vehicles travelling on a road.

[0011] Furthermore, center points may be derived from the driven trajectories of the vehicle that may represent an approximation of the center of a lane and / or an approximation where the vehicles drove. These center points are then used as queries for a left and right boundary point respectively via transformer decoding from the respective latent feature space. In particular, the aligned driven traces (center points) may be used as starting points for predicting lane cross sections (lane pairs) defined by the left and right boundary points. For example, a Lane Model Transformer Network (LMT-Net) may be utilized as encoder-decoder neural network architecture that performs polyline encoding and predicts lane cross sections and their connectivity.

[0012] In addition, link predictions or a prediction of the lane connectivity conditioned on the latent feature space may be performed to derive a likelihood of any two center points being connected. This may involve the prediction of a connection or link between two entities, in this case the center points, based on their latent features. These link predictions may define a lane graph that comprises a longitudinal connectivity.

[0013] Finally, the lane level HD map of the location may be provided by the lane cross sections (lane pairs) and the lane graph. In particular, the lane level HD map may comprise the longitudinal connectivity provided by the lane graph and a lateral connectivity provided by the lane cross sections. The advantage of this approach is that the lane level HD maps can be created directly from raw vehicle sensor input without the necessity of geometric alignment, which leads to an increased efficiency. Moreover, a closed map learning loop may be provided to ensure a very high up-to-dateness of the HD map as soon as sufficient new geometric perception data is available for map creation. In addition, a quality of the HD map may be increased by using the geometric perception data of multiple drives of vehicles without the need of ground truth data since labels are created based on the data driven prior selection.

[0014] According to an embodiment, the geometric perception data are extracted features and / or objects from raw sensor data of a sensor device, in particular from a sensor device of the respective vehicle. For example, the sensor device of the vehicle may be a camera or a lidar, wherein the raw sensor data may be analyzed to extract features and / or objects, like for example detected markings. These extracted featuresand / or objects may be transmitted to the computing device for further processing. This has the advantage that the transmitted geometric perception data may be much smaller in size compared to the raw data, which enables an efficient upload from the vehicle and aggregation of the data on the computing device.

[0015] In another embodiment, the geometric perception data are determined from camera data. That is, the camera data may comprise images of the location or road that are analyzed to derive the geometric perception data, in particular extracted features and / or objects from the images.

[0016] In another embodiment, the location is downsampled into equally sized areas of 50 m x 50 m tile size. In other words, the aggregated geometric perception data may be divided in quadrants of 50 m in lateral and longitudinal direction before encoding the downsampled geometric perception data into a latent feature space. The advantage of this downsampling is that the problem is divided into smaller sub-problems, which facilitates the determining of the HD map.

[0017] In another embodiment, the encoding into the latent feature space is performed by a self-attention based transformer architecture. The self-attention based transformer architecture is a mechanism used in machine learning, for example in computer vision tasks. In particular, dependencies and relationships within input sequences of the geometric perception data may be captured by the self-attention transformer. The selfattention transformer transforms the input sequence into vectors, in particular into three vectors by linear transformation. Then a weighted sum of the values based on the similarity between the vectors may be calculated. The resulting weighted sum and the original input may then be passed through a feed-forward neural network to produce the latent feature space that focuses on relevant information and long-range dependencies. In particular, the self-attention based transformer is used to inform individual encoded boundary points (polylines) about others. The advantage of using a self-attention based transformer architecture is that complex patterns and dependencies may be identified, and multiple downsampled areas of the geometric perception data may be computed in parallel making it computationally efficient and scalable for large datasets.

[0018] In another embodiment, the left and right boundary points are queried from the center points via a transformer decoding from the respective latent feature space. The transformer decoding is a neural network architecture that may comprise multiple layersof self-attention and feed-forward neural networks. In particular, the center points may be provided to the decoder as queries, whereby the decoder then may encode the left and right boundary points, for example from polylines that are provided by the geometric perception data. Here the decoder may be mainly used for cross-attention. This enable the geometric perception data to be analyzed fast and automatically.

[0019] In another embodiment, the center points are sampled with predefined distances. In other words, the prior point may be sampled every x meters, for example every 1, 2, 5 or 10 meters, wherein the center points are located on the trajectories driven by the respective vehicles. However, the intervals at which the center points are sampled do not have to be fixed and the sampling may also be performed as a function of curvature. Thereby, these center points provide the approximation of the center of the lane and a desired resolution may be set by the predefined distances.

[0020] In another embodiment, the link predictions of the center points define a lane graph in form of an adjacency matrix. The adjacency matrix may be a square matrix used to represent a graph, wherein the elements of the matrix may indicate if the pairs of nodes, or in this case of center points, are adjacent or not. The advantage of this embodiment is that the connection of the center points can be verified in a simple and efficient way.

[0021] Another aspect of the present invention relates to a computer program product including program code means for performing a method according to the preceding aspect. The computer program product may also be regarded as a computer program. In particular, the computer program product may be stored on a non-transitory computer- readable storage medium. That is, the non-transitory computer-readable storage medium may include at least the computer program product according to the preceding aspect.

[0022] Furthermore, the present invention relates to a computing device, wherein the computing device is configured to perform a method according to the preceding aspect. The advantages and embodiments already described for the method also apply to the computing device.

[0023] Further advantages, features, and details of the invention derive from the following description of embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features andfeature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. In the figures the same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0025] The drawings show in:

[0026] Fig 1 a motor vehicle and a computing device according to an exemplary embodiment;

[0027] Fig. 2 a schematic flow-chart according to an embodiment of a method.

[0028] In the figures the same elements or elements having the same function are indicated by the same reference signs.DETAILED DESCRIPTION

[0029] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0031] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0032] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0033] Fig. 1 shows a schematic system with a computing device 10 and a motor vehicle 12. The computing device 10 may be a server, for example on the internet that may be wirelessly connected to the motor vehicle 12 via a data connection. The computing device 10 may be configured to generate a lane level HD map using geometric perception data, in particular from features and / or objects like lane boundaries or lane markings that are derived from a sensor device 14 of the motor vehicle 12.

[0034] For example, the sensor device 14 is a vehicle camera and the geometric perception data may comprise driven traces and observed lane boundaries. In particular, the geometric perception data may be determined during a drive of the vehicle 12 at a specific location, for example a road that should be mapped.

[0035] The recorded geometric perception data from the sensor device 14 may be transmitted to the computing device 10 via a transceiving device 16 of the motor vehicle 12. For example, the transceiving device 16 may be a wireless modem connected to the internet.

[0036] Preferably, the recorded geometric perception data of multiple vehicles is transmitted to the computing device 10 in this way. The computing device 10 may be configured to aggregate the received geometric perception data from multiple vehiclesand downsample the aggregated geometric perception data into equally sized areas, for example in smaller areas or minimaps of 50 m x 50 m tile size.

[0037] Then, the computing device 10 may encode the downsampled geometric perception data area, for example the perceived lane boundaries, into a latent feature space, in particular via a self-attention based transformer. Moreover, center points may be sampled with predefined distances from driven trajectories that are used as an approximation of a center of the lane. Then, the center points may be used to query for left and right boundary points respectively via transformer decoding from the latent feature space. The two lane boundary points may represent a lane cross section or lane pairs at that specific prior point location. Furthermore, to derive the topology, the computing device 10 may perform link predictions of the center points conditioned on the latent feature space to derive the likelihood of two center points being connected or not. The resulting predictions may define a lane graph in form of an adjacency matrix.

[0038] As a result, the computing device 10 predicts lane cross sections and their connectivity, which allows constructing the lane level HD map for the location with the topological information that comprises longitudinal and lateral connectivity.

[0039] The generated lane level HD map may then be returned to the motor vehicle 12, for example via the transceiver device 16, wherein assistance systems or autonomous driving systems of the motor vehicle 12 may use the generated lane level HD map. By this approach, the lane level HD map can be derived directly from raw vehicle sensor input without the necessity of the geometrically aligned data, which leads to an increased efficiency.

[0040] Fig. 2 shows a schematic flow chart according to an embodiment of a method to generate a lane level HD map. In a step S1, a computing device 10 may receive geometric perception data recorded during single drives of vehicles. The geometric perception data may comprise extracted features and / or objects from raw sensor data of a sensor device 14, for example from a vehicle camera. In particular, the geometric perception data may comprise driven traces and observed lane boundaries that are provided as polylines.

[0041] In a step S2, a received geometric perception data from multiple vehicles may be aggregated in the computing device 10.

[0042] In a step S3, the aggregated geometric perception data is downsampled into equally sized smaller areas, for example into smaller areas of 50 m x 50 m tile size.

[0043] In a step S4, the downsampled equally sized smaller areas are encoded into a latent feature space, e.g. via a self-attention based transformer.

[0044] In a step S5, center points are sampled from driven trajectories of the vehicles that are used as an approximation of the center of a lane. The center points may be sampled every x meters, for example every 1, 2 or 5 meters.

[0045] In a step S6, left and right boundary points of a lane are determined for each center points by using the latent feature space. In particular, a transformer decoding from the latent feature space may be performed to query the respective left and right boundary points. As a result, lane cross sections may be provided.

[0046] In a step S7, a lane graph may be determined by performing link predictions conditioned on the latent feature space to derive a likelihood of two center points being connected. For example, these links predictions may be provided in a form of an adjacency matrix.

[0047] Finally, in a step S8, the lane graph together with the corresponding lane cross sections may provide the lane level HD map at that specific location.

[0048] In other words, sparse vehicle observations are used instead of dense sensor measurements that comprise aligned driven traces (center points) and lane boundaries. A pre-processing step may comprise the generation of polylines by aligning and aggregating observed lane boundaries. The aligned driven traces may be used as starting points for predicting lane pairs (lane cross sections) by the left and right boundary points. For example, a Lane Model Transformer Network may be used as encoder-decoder neural network architecture that performs polyline encoding and prediction of lane pairs and their connectivity. A lane graph may be formed by using the predicted lane pairs as nodes and the predicted lane connectivity as edges.

[0049] In particular, this map learning loop may be repeated as soon as sufficient new geometric perception data gets available. That ensures a very high up-to-dateness of the HD map. The invention thus provides geometric and semantic map learning of lanes from crowd-sourced sensor data.Reference signs10 Computing device12 Motor vehicle14 Sensor device16 Transceiving deviceS1-S8 Method steps

Claims

CLAIMS1. A method to generate a lane level HD map, comprising the steps performed on a computing device (10):- receiving geometric perception data recorded during single drives of vehicles at a location;- aggregating the received geometric perception data from the single drives;- downsampling the location into equally sized areas, wherein each equally sized area comprises a subset of the aggregated geometric perception data;- encoding the subset of the aggregated geometric perception data of each equally sized area into a respective latent feature space;- sampling of center points from driven trajectories of the vehicles that are used as an approximation of the center of a lane;- utilizing the center points to query for left and right boundary points from the respective latent feature space to provide lane cross sections at the location of the prior point;- performing link predictions conditioned on the latent feature space to derive a likelihood of two center points being connected, wherein the link predictions define a lane graph;- providing the lane level HD map of the location by the lane cross sections and the lane graph.

2. The method according to claim 1 , wherein the geometric perception data are extracted features and / or objects from raw sensor data of a sensor device (14).

3. The method according to claim 1 or 2, wherein the geometric perception data are determined from camera data.

4. The method according to any one of claims 1 to 3, wherein the location is downsampled into equally sized areas of 50 m x 50 m tile size.

5. The method according to any one of the preceding claims, wherein the encoding into the latent feature space is performed by a self-attention based transformer architecture.

6. The method according to claim 5, wherein the left and right boundary points are queried form the prior point via a transformer decoding from the respective latent feature space.

7. The method according to any one of the preceding claims, wherein the center points are sampled with predefined distances.

8. The method according to any one of the preceding claims, wherein the link predictions of the center points define the lane graph in form of an adjacency matrix.

9. A computer program product comprising code means for performing a method according to any one of the preceding claims.

10. Computing device (10), wherein the computing device (10) is configured to perform a method according to any of claims 1 to 8.

Images