AUTOMATED VEHICLE LOCALIZATION BASED ON FILTERED CHARACTERISTICS

Abstract

A method for automated vehicle localization based on filtered feature points comprises at least the following steps: generating sensor data using a vehicle sensor, wherein the sensor data represent the vehicle's environment and a change in the vehicle's position; generating a plurality of feature points based on the sensor data, wherein each feature point represents at least one location point in the vehicle's environment; determining several suitability indicators for at least some of the feature points, wherein each feature point is associated with several suitability indicators, and wherein each suitability indicator represents a suitability of the associated feature point for automated vehicle localization; filtering the feature points based on the suitability indicators; and automatically localizing the vehicle based on the filtered feature points.

Description

[0001] The present disclosure relates to a method for automated vehicle localization based on filtered feature points.

[0002] Driver assistance systems and other automated vehicle control systems can be offered at ever-increasing performance levels thanks to extensive sensor data and high processing power. Automated vehicle localization is a key component for highly automated vehicle control functions. This often relies on a mapping and localization algorithm that creates a map of the vehicle's surroundings based on sensor data and additionally estimates the vehicle's spatial position within this map. Particularly powerful algorithms can determine this information even from features of simple 2D image data, for example, using the Visual SLAM algorithm (SLAM: Simultaneous Localization and Mapping).

[0003] One challenge in sensor-based mapping and localization lies in the large volume of sensor data, the complete and reliable processing of which, especially in real time, can only be guaranteed with high-performance computer hardware. Furthermore, the data and its intermediate products, such as 3D point clouds or other feature points, must be transmitted with high speed and reliability within the vehicle or between the vehicle and a central server (e.g., a cloud server). The performance requirements for the vehicle's communication infrastructure can therefore also be high.

[0004] In light of the challenges described, one objective of the present disclosure is to provide an improved method for automated vehicle localization that reduces the effort required to process the extensive sensor data.

[0005] The problem is solved by the features of the independent claims. The dependent claims contain further developments of the disclosure.

[0006] According to one aspect of the disclosure, the problem is then solved by a method for automated vehicle localization based on filtered feature points, the method comprising at least the following steps: • Generating sensor data using at least one vehicle sensor, wherein the sensor data represent an environment of the motor vehicle and a change in the position of the motor vehicle; • Generating a multitude of feature points based on the sensor data, where each feature point represents at least one location point in the vicinity of the motor vehicle; • Determining several suitability indicators for at least some of the feature points, wherein each feature point is assigned several of the suitability indicators, and wherein the suitability indicators each represent a suitability of the assigned feature point for automated vehicle localization; • Filtering the feature points based on the suitability indicators; and preferably • Automated localization of the vehicle based on the filtered feature points.

[0007] The method is characterized by a filtering of the feature points, which makes it possible to reduce the large amount of data generated by these points and avoid unnecessary data processing. This approach leverages the fact that not all feature points are equally useful or relevant for vehicle localization. Based on suitability indicators, the feature points can be filtered very efficiently without compromising the performance and reliability of the vehicle localization. This also reduces the need for computing and transmission resources.

[0008] Another advantageous aspect of the method is that the filtering is applied to feature points and preferably not to raw sensor data. Feature points, as concrete reference points, are generally better suited for filtering than raw sensor data. A further advantage is that feature points classified as valid by the filtering can generally be used to reduce the number of landmarks representing a vehicle's environment. The landmarks and / or filtered feature points can, for example, be stored and reused, particularly during repeated drives through the environment. This makes it possible to at least partially forgo the computationally intensive and repeated processing of sensor data in a known environment. The need for data processing and transmission resources can thus be further reduced.

[0009] Automated localization preferably comprises determining at least one location parameter (e.g., a coordinate value) that represents the position of the vehicle with respect to a predefined spatial reference system (e.g., a coordinate system, in particular an axis of the coordinate system). In particular, multiple location parameters can also be determined that describe the position of the vehicle in three-dimensional space, e.g., as position data from the GPS system (GPS: Global Positioning System).

[0010] In one embodiment, the method comprises automated control of the motor vehicle based on at least one location parameter determined by automated localization of the vehicle. The automated control of the vehicle can refer to driver assistance systems with the participation of a human driver, as well as to partially and fully automated driving functions.

[0011] Feature points can generally possess a multitude of characteristics, generated based on sensor data, for example, using simple pixel-based operators from the field of image processing. For instance, multiple features based on the pixel brightness value and / or the pixel brightness values ​​of neighboring pixels can be assigned to a single image pixel, thus defining one or more feature points. Each feature point generally represents a location point in the vicinity of the vehicle. This location point can be on a static object (e.g., a building) or a dynamic object (e.g., another vehicle).

[0012] According to one embodiment, the method comprises determining several reliability parameters, each representing the reliability of an associated feature point as a reference point for automated vehicle localization. For this purpose, each reliability parameter is determined based on several suitability indicators of the associated feature point. In other words, each reliability parameter is based on multiple suitability indicators. The reliability parameter can thus be understood as a consolidated suitability of the feature point for automated vehicle localization. Filtering the feature points is preferably, and in particular exclusively, performed based on the reliability parameters. This simplifies the filtering process and requires less computing power.

[0013] Preferably, each reliability indicator is calculated by superimposing the suitability indicators of the assigned feature point. For example, an average of the suitability indicators can be calculated. Preferably, the suitability indicators are each defined by a numerical value that represents the suitability of the respective feature point for vehicle localization on an open or predefined numerical scale. In the case of differing scales, the suitability indicators are preferably normalized before being used for filtering or calculating the reliability indicator.

[0014] According to one embodiment, the filtered feature points are stored and later reintegrated into the automated vehicle localization process when the vehicle has left the environment and is subsequently being controlled within it again. The filtered-out feature points, on the other hand, are preferably not stored and are, for example, discarded. This reduces the vehicle's communication bandwidth, particularly between a vehicle backend (e.g., a central server wirelessly connected to the vehicle) and the vehicle's connected communication units. Vehicle localization can thus be advantageously based, for example, on the most accurate, stable, and easily recognizable feature points.On the other hand, these feature points, or the landmarks determined based on them, ideally only need to be read out when the vehicle is repeatedly localized and / or controlled in the same environment and are then immediately available for localization and / or control. For example, when executing a Visual SLAM, landmarks (e.g., 3D points) in the form of a point cloud can also be generated as a byproduct of the localization information when using feature points. These landmarks can first be reduced to a subset based on the filtered feature points. Then, the remaining landmarks from this subset can be stored as an additional map layer in the vehicle's backend. This allows the landmarks to be reused during subsequent passes through the familiar environment, which is coded according to the landmarks, thus reducing the requirements for vehicle localization.

[0015] According to a further embodiment, the suitability indicators can include a semantic indicator based on a semantic classification of the associated feature point. The semantic indicator preferably specifies the suitability of the feature point as an orientation point depending on a semantic class. For example, a higher semantic indicator is determined for a feature point classified as a point on a building than for another feature point classified as belonging to a tree or other plant. The semantic indicator thus encodes the information that certain very static or unchanging objects, such as...Buildings may be better suited as landmarks for automated vehicle localization than location points or objects that may be considered less reliable for vehicle localization due to altered vegetation (changed size, with or without leaves, pruning) or flexibility (movements in the wind).

[0016] According to another embodiment, the suitability indicators include a distance indicator based on the distance value of the associated feature point. For example, feature points with high distance values ​​are assigned lower distance indicators than feature points with low distance values. This is based on the assumption that feature points representing more distant locations are generally less reliable for automatic vehicle localization than feature points representing locations or objects closer to the vehicle.

[0017] According to another embodiment, the suitability indicators include an observability indicator that represents the frequency of the associated feature point in a temporal sequence of the sensor data. For example, it is assumed that feature points that are repeatedly generated in temporally successive sensor data sets (e.g., image data sets) and are detected as corresponding feature points, i.e., describing the same location, are better suited for vehicle localization than feature points that occur only rarely and can therefore be detected or tracked comparatively unreliably. Accordingly, feature points with a high detection frequency can receive higher suitability indicators than those with a lower frequency.

[0018] According to a further embodiment, the suitability indicators comprise a stochastic indicator that represents a deviation of the assigned feature point over a time course of the sensor data. For example, the stochastic indicator can represent the deviation of temporally adjacent (i.e., preceding and / or subsequent) feature points, particularly between feature points that represent the same location at different times. With large deviations, the reliability or stability of the feature point is statistically lower. Under this assumption, feature points with large deviations between their respective detections can be assigned a lower stochastic indicator than feature points with small deviations. In this way, it is taken into account that feature points that, over time, despite different detection times and, if applicable,spatial displacements with small feature deviations are detected and are statistically more reliable for vehicle localization.

[0019] According to a further embodiment, the suitability indicators include an edge indicator that represents the sharpness level of the assigned feature point and / or the association of the assigned feature point with an object edge and / or an object vertex. Feature points with a higher sharpness level are statistically more suitable as reliable reference points for vehicle localization. Accordingly, feature points with a high sharpness level can be assigned a higher edge indicator and thus preferred over feature points with a lower sharpness level and a correspondingly lower edge indicator. The sharpness level can additionally or alternatively be described by the association with an object edge or an object vertex.

[0020] According to another embodiment, the suitability indicators include a robustness indicator that represents the invariance of the assigned feature point with respect to variable exposure conditions. These variable exposure conditions can be simulated, for example, by systematically varying an image parameter, such as a contrast setting. The influence of the variation on the feature point can then be determined from a correspondingly modified image dataset. The less the feature point is affected by the variation, the more suitable it can be assumed for automatic vehicle localization. Accordingly, feature points with high invariance with respect to exposure-relevant image parameters can be assigned higher robustness indicators than those with low invariance.

[0021] It is important to understand that the exemplary numerical orientation of the suitability indicators (high - low) can be defined differently to convey the same information. For example, the numerical orientation can be reversed (low - high). Furthermore, the suitability indicators do not necessarily have to be formed on a continuous scale using numerical values, but can also be expressed, for example, on categorical scales.

[0022] According to a further embodiment, the filtered feature points and / or landmarks filtered on this basis predominantly, preferably exclusively, represent static objects, e.g., buildings and trees. In particular, the feature points can represent objects that are detectable solely on the basis of the sensor data (non-semantic objects).

[0023] Preferably, the sensor data comprises image data generated by a 2D or 3D video camera on the vehicle. The camera can, for example, be located at the front of the vehicle and essentially correspond to the driver's perspective. In principle, other sensor types can also be used to generate the sensor data, such as radar and / or lidar sensors.

[0024] Furthermore, the sensor data preferably includes position change data. This position change data can be represented by a temporal sequence of image data (video data) and / or by separately acquired inertial sensor data. Inertial sensor data can be generated, for example, by an accelerometer and / or a yaw rate sensor of the vehicle. Preferably, the vehicle sensor for generating the sensor data is mounted on the motor vehicle.

[0025] According to a further embodiment, the method also comprises the following steps: generating three-dimensional landmarks using a mapping and localization algorithm based on the feature points; and filtering the landmarks based on the filtered feature points, wherein the localization of the vehicle is performed based on the filtered landmarks. The filtered feature points can, in particular, be used to determine a subset of all landmarks generated by the mapping and localization algorithm, thus reducing the landmarks to those that, based on the filtered feature points, are considered to have a higher reliability for vehicle localization. Preferably, when applying the mapping and localization algorithm, the sensor data comprises image data that is exclusively two-dimensional, i.e., generated by a 2D video camera of the vehicle.The landmarks represent points of location in the vicinity of the vehicle.

[0026] According to one aspect of the disclosure, a computer program and / or a computer-readable medium is provided. The computer program and / or the computer-readable medium includes instructions that, when executed by a data processing device, cause the device to perform the method according to the disclosure and / or steps thereof. Optionally, the computer program and / or the computer-readable medium includes instructions that, when executed by a data processing device, cause the device to perform the process steps described as advantageous or optional in order to achieve an associated technical effect.

[0027] According to one aspect of the disclosure, a data processing device is provided for a motor vehicle. The data processing device is configured to perform the procedure described above. Optionally, the data processing device is configured to perform a procedure step described as advantageous or optional and / or to implement a procedure feature in order to achieve an associated technical effect.

[0028] According to another aspect of the disclosure, a motor vehicle comprising the data processing device described above is provided. Optionally, the data processing device of the motor vehicle and / or the motor vehicle itself is configured to perform a process step described as advantageous or optional and / or to implement a process feature in order to achieve an associated technical effect.

[0029] The following are further characteristics of the described objects, with reference to the figures, described purely as examples. Fig. Figure 1 schematically shows a flowchart of a procedure according to one aspect of the revelation; Fig. Figure 2 shows a schematic representation of a computer program and / or computer-readable medium according to one aspect of the revelation; and Fig. Figure 3 schematically shows a motor vehicle with a data processing device for executing the procedure of Fig. 1 according to one aspect of the revelation.

[0030] Fig. Figure 1 schematically shows a flowchart of a procedure according to one aspect of the disclosure. The procedure serves for the automated localization of a Fig. The procedure describes the steps involved in generating and filtering the feature points of three schematically represented motor vehicles (50). The procedure is further described in detail below.

[0031] First, sensor data is acquired, which serves as the basis for generating the feature points (see step 61). The sensor data comprises two-dimensional image data, preferably generated using a monocular RGB camera (RGB: Red, Green, Blue), see step 62. Additionally, position change data is generated, describing the spatial change in the position of the vehicle 50, see step 64. The position change data is preferably determined using an inertial measurement unit of the vehicle 50 (not shown).

[0032] The sensor data acquired in step 61 then undergo several preprocessing steps, summarized in block 70. In step 72, image classification is performed, for example, by assigning one or more semantic classes to at least some pixels of the image data. Semantic classes can be, for example, vehicle, building, or roadway. In step 74, the sensor data is preprocessed, for example, by filtering out noise or similar measures to improve data quality. The image data then undergoes semantic segmentation (see step 76), whereby, for example, pixels belonging to a common semantic class are assigned to a common object, such as a vehicle or a building. This creates, for example, coherent and consistently classified pixel groups, each representing an object.

[0033] In a further step (78), a large number of feature points are extracted. Each feature point can be assigned to one or more pixels of the image data and represent one or more features, e.g., a brightness value or a difference between a brightness value and one or more brightness values ​​of spatially adjacent pixels. In a further step (79), the feature points are compared with successive images (e.g., frames) of the image data to identify corresponding feature points that represent the same location in the vicinity of the vehicle (feature matching).

[0034] The image data preprocessed in step 74 undergoes systematic image modification in block 80. For this purpose, the individual images are systematically varied with respect to one or more image parameters, e.g., a contrast parameter, in order to obtain several modified images from each image instance (see step 84). Feature points are then extracted from each of the modified images (see step 86), with the extraction procedure corresponding to that of step 78.

[0035] The data obtained in blocks 70 and 80, in particular the feature points and the semantic classification data, are subjected to an attribute extraction 90. Suitability indicators are determined for at least some feature points, which are described below. The suitability indicators are preferably determined for groups of feature points that were identified as corresponding feature points in step 79.

[0036] In step 92, a semantic indicator is determined that specifies the suitability of a given feature point as an orientation point for vehicle localization, depending on a semantic class. Feature points assigned to optically stable objects, such as buildings, receive higher semantic indicators than feature points assigned to less stable objects, such as plants, especially trees.

[0037] In step 94, a distance indicator is determined that specifies the suitability of a feature point as a reference point for vehicle localization, depending on the distance assigned to that feature point. The distance can, for example, be estimated from the image data in step 78 and be included in the feature points. Feature points with high distance values ​​receive lower distance indicators than feature points with low distance values. This measure is based on the assumption that objects further away are generally less reliable for automatic vehicle localization than feature points that, according to their distance, are closer to the vehicle.

[0038] In step 95, an observability indicator is determined, representing a detection frequency for each relevant feature point. Feature points that are better suited for vehicle localization, based on a high detection frequency determined over a predefined period, are assigned a higher observability indicator than feature points with a low detection frequency. The detection frequency can be determined, for example, by counting the number of detections of a predefined feature point in a predefined number of consecutive image datasets. The predefined feature point corresponds to a predefined location point and is identified across the multiple consecutive image datasets (e.g., using a tracking algorithm, see step 79).The feature point can be determined on the basis of different pixels, especially if the sensor is moved along with the motor vehicle 50 during the recording of the sensor data and a respective location point in the vicinity is recorded by different pixels.

[0039] In step 96, a stochastic indicator is determined that represents a deviation of the assigned feature point from temporally adjacent feature points. High temporal deviations from feature points representing the same location are considered less suitable as reference points for vehicle localization. This means that feature points which, for example, exhibit high deviations between successive image data sets according to a feature-related difference operator, yet still represent the same location on a static or dynamic object, are assigned a lower stochastic indicator than feature points with smaller deviations.

[0040] In step 97, an edge indicator is determined that represents the sharpness level of the assigned feature point and / or the association of the assigned feature point with an object edge and / or an object vertex. Feature points with a higher sharpness level are statistically better suited as reliable reference points. In step 97, higher edge indicators are determined for feature points with a high sharpness level than for feature points with a lower sharpness level.

[0041] In step 98, a robustness indicator is determined that represents the invariance of the assigned feature point under variable exposure conditions. The variable exposure conditions are simulated, as described above in connection with block 90, by systematically varying an image parameter, such as a contrast setting. The influence of the variation on the feature point is determined from a correspondingly modified image dataset. The less a feature point is affected by the variation, the more suitable it can be assumed for automatic vehicle localization. Accordingly, feature points with high invariance under exposure-relevant image parameters can be assigned higher robustness indicators than those with low invariance.

[0042] In step 100, the suitability indicators determined in block 90 are combined to form a reliability index. For this purpose, the suitability indicators, each defined on a standardized numerical scale, are averaged for a given characteristic point or group of characteristic points. Optionally, the suitability indicators can be weighted according to an assigned relevance and included in the average. Furthermore, the reliability index can optionally be determined based on the image classification from step 72.

[0043] In step 102, the characteristic points are filtered using a predefined filter criterion. For example, characteristic points whose reliability index falls below a threshold can be discarded and excluded from further processing. However, other filter criteria based on the reliability index are also conceivable.

[0044] The feature points filtered in step 102 are used in data fusion step 106 to select reliable landmarks for automated vehicle localization. In step 104, the landmarks are determined using a mapping and localization algorithm (e.g., a VSLAM) based on the feature points generated in step 78. The position change data generated in step 64 is also used in step 104. Each landmark describes a location point in the vehicle's vicinity in three spatial dimensions (3D), so that together they form a 3D map of the vehicle's surroundings. The number of landmarks is reduced in data fusion step 106 based on the filtered feature points. Specifically, those landmarks that can be assigned to a filtered feature point are selected.Landmarks that cannot be assigned to any filtered feature point are discarded because they are considered not reliable enough for vehicle localization. The landmarks obtained in this way are transferred to a central server (backend) in step 108 and stored there in non-volatile memory (see step 110). The landmarks can be kept on the server for later use in vehicle localization, particularly if the vehicle 50 re-enters the environment represented by the landmarks. Furthermore, the landmarks can also be used directly, i.e., without prior storage, for localization and / or control of the vehicle 50 (not shown).

[0045] Fig. Figure 2 shows a schematic representation of a computer program and / or computer-readable medium 200 according to one aspect of the disclosure. The computer program and / or computer-readable medium 200 comprises instructions 201 which, when the program or instructions 201 are executed by a data processing device 54, cause it to carry out the process of Fig. 1 and / or at least some steps of this procedure. The instructions 201 can be in the form of program code in any code or language, in particular in code suitable for motor vehicle control systems 50. The computer program and / or computer-readable medium 200 can be or comprise any digital data storage device, such as a USB flash drive, hard drive, CD-ROM, SD card, or SSD card. The computer program need not necessarily be stored on such a computer-readable storage medium, but can also be accessed via the Internet or otherwise.

[0046] The motor vehicle 50 preferably includes the data processing device 54 for executing the computer program and / or computer-readable medium 200. The data processing device 54 comprises, as hardware components, at least a processor 56, a non-volatile memory 58 (e.g., an SD card), and a memory 60 (e.g., main memory, RAM), cf. Fig. 3. The computer program and / or computer-readable medium 200 can be permanently stored in the non-volatile memory 58 and loaded into the memory 60 for execution. Additionally, the data processing device 54 with the sensors (not shown) for recording the sensor data is used in step 60 of the method. Fig. 1 connected. Furthermore, the data processing device 54 has a communication interface for data transmission between a central server (not shown), in particular for the wireless transmission of the filtered landmarks in step 108 of the procedure of Fig. 1. The data processing device 54 enables the computer-implemented execution of the automated vehicle localization method based on the method of Fig. 1 filtered feature points. Reference symbol list 50 motor vehicles 54 Data processing device 56 processor 58 non-volatile memory 60 storage 61 Generating sensor data 62 Generating image data 64 Generating position change data 70 Preprocessing 72 Image classification 74 Image preprocessing 76 Semantic Segmentation 78 Feature Extraction 79 Feature Matching 80 image modifications Modify 82 images 84 modified images are created 86 Feature Extraction 90 Attribute Extraction 92 Determining semantic indicators 94 Determination of distance indicator 95 Determination of the observability indicator 96 Determination of stochastic indicator 97 Determination of edge indicator 98 Determination of Robustness Indicator 100 Determination of reliability indicator 102 Filtering Feature Points 104 Mapping and Localization Algorithm 106 Data Fusion 108 Transmission of filtered landmarks 110 Save the filtered landmarks 200 computer program and / or computer-readable medium 201 commands

Claims

[1] Method for automated vehicle localization based on filtered feature points, wherein the method comprises at least the following steps: Generating sensor data (61) using at least one vehicle sensor, wherein the sensor data represent an environment of a motor vehicle (50) and a change in position of the motor vehicle (50); Generating a plurality of feature points (78) based on the sensor data, wherein each feature point represents at least one location point in the vicinity of the motor vehicle (50); Determining several suitability indicators (90) for at least some of the feature points, wherein several of the suitability indicators are assigned to each feature point, and wherein the suitability indicators each represent a suitability of the assigned feature point for automated vehicle localization; Filtering the feature points based on the suitability indicators (102); and Automated localization of the motor vehicle (50) based on the filtered feature points. [2] Method according to claim 1, further comprising: Determining several reliability parameters (100), each representing a reliability of an assigned feature point as a reference point for automated vehicle localization, wherein each reliability indicator is determined on the basis of several suitability indicators of the assigned characteristic point, and where the filtering of the feature points is based on the reliability indicators (102). [3] Method according to claim 2, wherein each reliability indicator is formed by weighted superposition of the suitability indicators assigned to the feature point. [4] Method according to at least one of the preceding claims, wherein the filtered feature points and / or landmarks determined on the basis of the filtered feature points are stored (110) and are included again at a later time in the automated localization of the motor vehicle when the motor vehicle (50) has left the environment and is again being controlled in the environment at that later time. [5] Method according to at least one of the preceding claims, wherein the suitability indicators include a semantic indicator (92) based on a semantic classification of the assigned feature point; and / or the suitability indicators include a distance indicator (94) which is based on a distance value of the associated feature point. [6] Method according to at least one of the preceding claims, wherein the suitability indicators include an observability indicator (95) representing a frequency of the assigned feature point in a time series of the sensor data, and / or wherein the suitability indicators include a stochastic indicator (96) which represents a deviation of the assigned feature point in the temporal course of the sensor data. [7] Method according to at least one of the preceding claims, wherein the suitability indicators include an edge indicator (97) representing a sharpness level of the associated feature point and / or the association of the associated feature point with an object edge and / or an object vertex; and / or the suitability indicators include a robustness indicator (98) which represents invariance of the associated feature point under variable exposure conditions. [8] Method according to at least one of the preceding claims, further comprising: Generating three-dimensional landmarks using a mapping and localization algorithm based on feature points (104); and Filtering the landmarks based on the filtered feature points (106), wherein the localization of the motor vehicle (50) is based on the filtered landmarks (108). [9] Computer program and / or computer-readable medium (200), comprising instructions (201) which, when executed by a data processing device (54), cause the device to carry out the method and / or the steps of the method according to any one of claims 1 to 8. [10] Data processing device (54) for a motor vehicle (50), wherein the data processing device (54) is configured to perform the method according to any one of claims 1 to 8. [11] Motor vehicle (50) comprising a data processing device (54) according to claim 10.

Images