Method for providing a machine-learning binary classification model for predicting the availability of map-based localization

Abstract

Method for providing a machine learning binary classification model (7) for predicting the availability of a localization map, comprising the following steps: a) Performing a reference run with a real vehicle, recording a reference trajectory (1) and referenced landmark positions (2) surrounding the reference trajectory (1), b) Performing a journey along the reference trajectory (1) with a fictitious vehicle, estimating a trajectory (3) and landmark positions (4) surrounding the trajectory (3) using the localization function, c) Providing a training dataset with data points such that a data point includes first information to describe a map section and second information to indicate the localization availability of that map section, wherein the map section corresponds to the environment of the reference trajectory (1), and the second information is determined based on comparing the reference trajectory (1) with the estimated trajectory (3) and / or on comparing the referenced landmark positions (2) with the estimated landmark positions (4), and d) Training the classification model (7) with the training dataset using artificial intelligence.

Description

State of the art

[0001] The present invention relates to a method for providing a machine-learning binary classification model for predicting the availability of map-based localization and a method for providing a localization map with additional availability information using this machine-learning binary classification model. Furthermore, a localization map with additional availability information is proposed. The invention can be applied in particular to map-based localization for autonomous or semi-autonomous driving.

[0002] Map-based localization relies on comparing sensor data with map data. From this comparison, the position and pose of a vehicle can be estimated. The sensor data can be acquired from at least one sensor installed in the vehicle, such as a camera, radar, lidar, and / or GNSS. The map data can be derived from a localization map and typically includes information about landmarks such as lane markings, traffic signs, traffic lights, etc. Localization in map data using landmarks acquired with sensor data, verification of localization using GNSS with the help of landmarks acquired with sensor data, and, where applicable, supplementation of map data based on acquired sensor data are described in the prior art, for example, in US 2021 / 0310812A1, US 2021 / 0200237A1, and US 2018 / 0088234A1.

[0003] Currently, there are two known approaches to generating such a localization map. One approach uses survey vehicles equipped with special high-end sensors, allowing sensor data to be collected and a localization map to be generated from this data. The other approach is based on crowdsourced data, using only end-customer vehicles instead of survey vehicles. These vehicles collect sensor data while driving and upload it to a cloud, enabling the localization map to be generated from this data in the backend.

[0004] However, with both approaches, it can happen that some sections of the generated localization map do not contain enough landmarks, causing vehicle localization based on this map to fail. This occurs, for example, with the crowdsourced data approach if not enough end-customer vehicles travel on a road, and therefore insufficient data can be collected about that road. This is also generally the case if the road infrastructure lacks sufficient information (e.g., a rural road without lane markings). Furthermore, a faulty localization map (e.g., missing landmarks) can also lead to failed vehicle tracking.

[0005] Therefore, there is a desire to provide a localization map with additional availability information that indicates which map sections can be used for localization and which cannot. This availability information is also crucial for localization systems, as it defines the Operational Domain (ODD), i.e., where the localization function of a localization system can be activated and where it cannot.

[0006] A well-known approach to verifying the availability of map-based localization relies on the use of ground truth. This involves driving a vehicle across an entire localization map to check whether the localization based on that map is functioning correctly. Ground truth generally refers to real-world information gathered through observation and measurement during the data collection phase, rather than through inference. This approach is therefore limited to small-scale localization maps.

[0007] Based on this, the object of the present invention is to alleviate or at least partially solve the problems described with reference to the prior art. In particular, a solution is to be provided with which the availability of a localization map already generated using known approaches can be predicted independently of ground truth and map size, and thus also the availability of large-scale localization maps can be predicted. Disclosure of the invention

[0008] This involves a method for providing a machine-learning binary classification model that can be used to predict the availability of a localization function based on a localization map, the method comprising the following steps: a) Performing a reference run with a real vehicle, recording a reference trajectory and referenced landmark positions surrounding the reference trajectory, b) Performing a journey along the reference trajectory with a fictitious vehicle, estimating the trajectory and landmark positions surrounding the trajectory using the localization function, c) Providing a training dataset with data points such that a data point includes first information to describe a map section and second information to specify a localization availability of the localization function for that map section, wherein the map section corresponds to the environment of the reference trajectory, and the second information is determined based on comparing the reference trajectory with the estimated trajectory and / or on comparing the referenced landmark positions with the estimated landmark positions, and d) Training the classification model with the training dataset using artificial intelligence.

[0009] Using the provided classification model, the availability of a localization map already generated with known approaches and localization functions executed on it can be predicted, whereby only map data (e.g. landmarks) from the localization map are required and the use of ground truth is no longer necessary, so that the prediction can be carried out independently of the map size and thus the availability of large-scale localization maps can be predicted.

[0010] Using the provided classification model, an existing localization map can also be modified to provide additional information about map availability. This is also advantageous in the context of safety-critical automated driving functions.

[0011] The localization map is, for example, an HD map (High Definition Map) or a HAD map (Highly Automated Driving Map) and can be used for autonomous driving. Autonomous driving can be understood as the movement of vehicles that behave largely autonomously, for example, using a GNSS-based localization system and / or sensors for perceiving the environment, such as radar, cameras, or ultrasonic sensors. These vehicles can be motor vehicles such as passenger cars, trucks, or other commercial vehicles, robots, or similar devices.

[0012] A machine-learning binary classification model, as used here, is one that, after training with a training dataset, can output information on whether a map section of a localization map is available for the localization function, given that this map section was input into the classification model. In other words, the classification model assigns a map section of a localization map to either an availability class or a non-availability class.

[0013] A training dataset can contain a large number of data points. A data point is, for example, a labeled piece of information that describes a map section. This can mean that a training dataset consists of a set of map sections with associated features and class memberships (labels).

[0014] It can be provided that a data point of a training dataset includes first information describing a map section and second information indicating the localization availability of this map section. The first piece of information can be generated using a mapping system, and the second piece of information can be determined using ground truth, which can be obtained according to steps a) and b).

[0015] It can be assumed that a data point in the training dataset consists of a pair (X, y), where X corresponds to the first piece of information and y to the second. X is a generic data vector, and y ∈ {0, 1} describes the class membership of X, i.e., the availability of the localization function. The features stored in the data vector X can be generated from a mapping system. A mapping system, in this context, refers to a system that allows the assignment of availability to global coordinates. In particular, information about landmarks such as traffic signs, road boundaries, or traffic lights can be used. Information about such landmarks is regularly available with precise details of their position in a global coordinate system. The assignment of data points to global coordinates is possible through the recognition of such landmarks.Examples of possible features include the number of traffic signs in a map section or the road type. Alternatively or additionally to a global data vector that encodes information about the entire map section, other representations such as a two- or three-dimensional grid, a point cloud, and / or a satellite image can also be used.

[0016] According to step a), a reference trajectory will be recorded by a reference run with a real vehicle. The real vehicle is preferably a vehicle capable of localization, equipped with special high-end sensors such as cameras, radar, lidar, and / or GNSS, so that reference sensor data corresponding to the reference trajectory is also acquired and recorded during the reference run using these high-end sensors. When recording the reference trajectory and the reference sensor data, care must be taken to ensure that the recorded reference trajectory is correct and matches the reference sensor data. The landmarks surrounding the reference trajectory can also be determined from the reference sensor data, including not only their position but also the type of each landmark.The positions of the landmarks surrounding the reference trajectory, determined from the reference sensor data, are called referenced landmark positions.

[0017] According to step b), an estimated trajectory 3 can be obtained by a virtual journey with a fictitious vehicle along the reference trajectory. A map and a localization unit can be used to execute the localization function. The fictitious vehicle can start the localization unit at a given map point, which, for example, also defines the features of the landmarks captured during the reference journey, thus providing an initial position. The fictitious vehicle can then travel along the reference trajectory at a constant speed for a fixed time step, successively feeding new landmarks from the map and sensor data (e.g., the reference sensor data recorded during the reference journey) as input data to the localization unit, thus generating a new estimated position each time. The estimated positions thus constitute the estimated trajectory.The "fictitious" vehicle and the "virtual" drive performed with it are preferably part of a simulation used to test the localization function based on the localization map. This simulation represents the situation in which a real vehicle would perform the localization function during a real drive. The virtual drive and the fictitious vehicle exist within this simulation for the purpose of generating training data.

[0018] According to step c), the second piece of information is determined based on comparing the reference trajectory with the estimated trajectory and / or comparing the referenced landmark positions with the estimated landmark positions. A threshold can be used, and the second piece of information for the data point indicates that the first piece of information for that data point is considered available (i.e., a positive sample) if neither the deviation between the reference trajectory and the estimated trajectory nor the deviation between the referenced landmark positions and the estimated landmark positions reaches the threshold. Otherwise, the second piece of information indicates that the first piece of information is considered unavailable (i.e., a negative sample).

[0019] Preferably, steps a) and b) are repeated to obtain a large number of reference trajectories and estimated trajectories corresponding to the reference trajectories. This allows a large number of data points to be provided in step c).

[0020] According to step d), the classification model is trained using artificial intelligence with the provided training dataset.

[0021] Unlike known approaches where the availability of a localization map is determined using ground truth, for example, based on specific quality criteria of sensor input data such as lighting conditions, the solution presented here predicts the availability of a localization map not with ground truth, but with information extracted from this localization map (e.g., landmarks) and with the help of a previously trained classification model. It is particularly important to distinguish that in this solution, ground truth is only used for training the classification model; it is not used for predicting map availability.This has the particular advantage that the solution presented here can predict the availability of a large-scale map, since determining the availability of a map with ground truth can lead to a vehicle having to drive across the entire map to check if the localization works and is therefore limited to small-scale maps.

[0022] It is preferred if, in step c), the first piece of information includes features of the landmarks contained in this map section.

[0023] It is preferred that the first piece of information includes the landmark position of each landmark contained in this map section.

[0024] It is preferred if the first piece of information is implemented in the form of a data vector, in which the features of the landmarks contained in this map section are stored.

[0025] It is preferred if the first piece of information is represented by a grid with grid cells such that the grid is overlaid on this map section and the features of a landmark contained in this map section are stored in a grid cell if these features are also located in this grid cell.

[0026] The grid can be two-dimensional (2D) or three-dimensional (3D). Additionally, the positions of landmarks in the data can be encoded. Such a multidimensional data representation is commonly referred to as a tensor in the context of machine learning. If the classification model accepts tensor-shaped input data (e.g., a convolutional neural network), this data representation can be used directly. If only vector-based input data is accepted, the tensor simply needs to be unrolled into a one-dimensional vector.

[0027] It is preferred that the initial information be represented by point clouds, such that each landmark within the map area is represented by a point in a point cloud. In this case, each landmark can contain additional information besides its position, such as its type. Processing point cloud data requires specially designed neural networks, such as Kernel Point Convolution (KPConv).

[0028] It is preferred if the first piece of information is additionally provided using a satellite image corresponding to the map section.

[0029] It is preferred if, in step c), the second piece of information is marked with a zero or a one, where one corresponds to availability and zero to non-availability.

[0030] It is preferred if, in step c), the second piece of information indicates the unavailability when the deviation between the reference trajectory and the estimated trajectory reaches a given threshold.

[0031] It is preferred if, in step c), the second piece of information indicates the unavailability when the deviation between the referenced landmark positions and the estimated landmark positions reaches a given threshold.

[0032] It is preferred if, in step d), the classification model is trained using a neural network, a gradient-boosted tree, and / or a support vector machine.

[0033] Furthermore, a method for providing a localization map with additional availability information using a classification model proposed here, and a localization map provided by this method, are proposed.

[0034] The location map is specifically an HD map or a HAD map, whose additional availability information indicates where the location map can and cannot be used for localization. Availability information is crucial for localization products because it defines the Operational Domain (ODD). It's important to note that the accuracy of a location map is generally a different concept than the availability of a location map. A location map can be inaccurate but still be used for localization, and conversely, a location map can be accurate but unavailable for positioning.

[0035] The procedure for providing a location map with additional availability information includes the following steps: i) Extracting landmarks from a map section of a localization map, ii) Inputting the extracted landmarks into the classification model, and iii) Predictions of availability or unavailability for this map section using the classification model, and iv) Repeat steps i) to iii) to predict availability or unavailability on another map section until the entire localization map is annotated with availability information.

[0036] The concepts presented here share the common feature of AI-based prediction of map-based localization availability. First, a machine-learning binary classification model with ground truth is provided. This allows the model to predict the availability of an existing localization map at various map sections, even without ground truth. Only map data (e.g., landmarks) from the localization map are required, thus extending the prediction to large-scale localization maps beyond small-scale ones. In this way, an existing large-scale localization map can also be further processed into a new localization map with additional availability information.In particular, the solution presented here is independent of the approach used to generate the existing location map. Therefore, this solution allows for the flexible further processing of various location maps to supplement the availability information accordingly.

[0037] The solution presented here and its technical context are explained in more detail below with reference to the figures. It should be noted that the invention is not intended to be limited by the illustrated embodiments. In particular, unless explicitly stated otherwise, it is also possible to extract partial aspects of the situations explained in the figures and combine them with other components and / or findings from other figures and / or the present description. The figures show schematically and by way of example: Fig. 1 a virtual journey along a reference trajectory to form an estimated trajectory; Fig. 2. An implementation variant for providing the second piece of information of a data point of a training data set; Fig. 3. Another implementation variant for providing the second piece of information of a data point of a training dataset; and Fig. 4 an architecture of a proposed classification model.

[0038] Fig. Figure 1 schematically and exemplarily shows a virtual journey along a reference trajectory 1 to form an estimated trajectory 3.

[0039] The reference trajectory 1 is in Fig. The reference trajectory is defined by a solid line and can be recorded in process step a) by a reference run with a real vehicle. The real vehicle is preferably a vehicle capable of localization, equipped with special high-end sensors such as a camera, radar, lidar, and / or GNSS, so that reference sensor data corresponding to the reference trajectory are also acquired and recorded during the reference run using the high-end sensors. When recording the reference trajectory 1 and the reference sensor data, care must be taken to ensure that the recorded reference trajectory 1 is correct and corresponds to the reference sensor data. The landmarks surrounding the reference trajectory 1 can also be determined from the reference sensor data, whereby not only the position but also the type of the respective landmarks can be ascertained.The positions of the landmarks surrounding the reference trajectory 1, determined from the reference sensor data, are referred to as referenced landmark positions 2. The different landmark types are described in . Fig. 1 represented as a pentagram, hexagon and rhombus.

[0040] The estimated trajectory 3 is in Fig. Reference trajectory 1 is indicated by a dashed line and can be obtained in process step b) by a virtual journey with a fictitious vehicle along the reference trajectory 1. A map and a localization unit can be used to execute the localization function. The fictitious vehicle can start the localization unit at a given map point, which, for example, also defines the characteristics of the landmarks captured during the reference journey, thus providing an initial position. Subsequently, the fictitious vehicle can travel along the reference trajectory 1 at a constant speed for a fixed time step 6, successively feeding new landmarks from the map and sensor data, such as the reference sensor data recorded during the reference journey, as input data to the localization unit, so that a new estimated position is obtained each time.The estimated positions thus form the estimated trajectory 3.

[0041] Fig. Figure 2 schematically and exemplarily shows one implementation variant for providing the second piece of information for a data point in a training dataset. Fig. 2. It can be seen that the estimated trajectory 3 starts from a starting point 5 and converges towards the reference trajectory 1, and the deviation between the referenced landmark position 2 of an affected landmark and the corresponding estimated landmark position 4 is small. In this case, the second piece of information can indicate that the first piece of information for the data point is considered available (i.e., a positive sample).

[0042] Fig. Figure 3 schematically and exemplarily shows another implementation variant for providing the second piece of information for a data point in a training dataset. Fig. It can be seen that the estimated trajectory 3 originates from a starting point 5 and does not converge towards the reference trajectory 1, and that the deviation between the referenced landmark position 2 of an affected landmark and the associated estimated landmark position 4 is large. In this case, the second piece of information can indicate that the first piece of information for the data point is considered unavailable (i.e., a negative sample).

[0043] Fig.Figure 4 schematically and exemplarily shows the architecture of a proposed classification model 7. A data representation 9 is derived from a map section 8. Optionally, an aerial photograph of this map section 8 can also be used as an information source. The data representation 9 can be in the form of a tensor or a point cloud and is fed into an encoder network 10. The encoder network 10 then processes the data representation 9 into a fixed-size vector. The encoder network 10 can be implemented as a convolutional neural network, a kernel point convolution (KPConv), or a combination of both. This vector can then be fed into a classification network 11. The classification network 11 further processes this vector until it yields a single scalar value that predicts the class membership, i.e., the availability or unavailability of this map section 8.The classification network 11 can include multiple dense layers. Optionally, additional global features 12 can be added to the classification network 11.

Claims

[1] Method for providing a machine learning binary classification model (7) for predicting the availability of a localization function based on a localization map, comprising the following steps: a) Performing a reference run with a real vehicle, recording a reference trajectory (1) and referenced landmark positions (2) surrounding the reference trajectory (1), b) Performing a journey along the reference trajectory (1) with a fictitious vehicle, estimating a trajectory (3) and landmark positions (4) surrounding the trajectory (3) using the localization function, c) Providing a training dataset with data points such that a data point includes first information to describe a map section and second information to specify a localization availability of the localization function for that map section, wherein the map section corresponds to the environment of the reference trajectory (1), and the second information is determined based on comparing the reference trajectory (1) with the estimated trajectory (3) and / or on comparing the referenced landmark positions (2) with the estimated landmark positions (4), and d) Training the classification model (7) with the training dataset using artificial intelligence. [2] Method according to claim 1, wherein in step b) the fictitious vehicle travels along the reference trajectory (1) at a constant speed for a fixed time step (6). [3] Method according to claim 1 or 2, wherein in step c) the first information is generated using a mapping system. [4] Method according to one of the preceding claims, wherein in step c) the first information comprises features of the landmarks contained in this map section. [5] Method according to claim 4, wherein the first information includes the landmark position of each landmark contained in this map section. [6] Method according to claim 4 or 5, wherein the first information is implemented in the form of a data vector in which the features of the landmarks contained in this map section are stored. [7] Method according to any one of claims 4 to 6, wherein the first information is represented by a grid with grid cells such that the grid is placed over this map section and the features of a landmark contained in this map section are stored in a grid cell if these features are also located in this grid cell. [8] Method according to any one of claims 4 to 7, wherein the first information is represented by point clouds such that each landmark contained in this map section is represented by a point in a point cloud. [9] Method according to any one of claims 4 to 8, wherein the first information is additionally provided using a satellite image corresponding to the map section. [10] Method according to one of the preceding claims, wherein in step c) the second piece of information is characterized by a zero or a one, where the one corresponds to availability and the zero to non-availability. [11] Method according to one of the preceding claims, wherein in step c) the second information indicates the unavailability when the deviation between the reference trajectory (1) and the estimated trajectory (3) reaches a given threshold. [12] Method according to any of the preceding claims, wherein in step c) the second information indicates the unavailability when the deviation between the referenced landmark positions (2) and the estimated landmark positions (4) reaches a given threshold. [13] Method according to any of the preceding claims, wherein in step d) the classification model (7) is trained using a neural network, a gradient-boosted tree, and / or a support vector machine. [14] Method for providing a localization map with additional availability information using a binary classification model (7) provided by the method according to any one of the preceding claims 1 to 13, wherein the method comprises the following steps: i) Extracting landmarks from a map section (8) of a localization map, ii) Inputting the extracted landmarks into the classification model (7), and iii) Predictions of availability or unavailability at this map section (8) using the classification model (7), and iv) Repeat steps i) to iii) to predict availability or unavailability on another map section until the entire localization map is annotated with availability information.

Images