Method for detecting environmental information of a vehicle using multiple ultrasonic sensors

Abstract

The invention relates to a method, in particular a computer-implemented method, for detecting environmental information of a vehicle (1) using a plurality of ultrasonic sensors (2), a data processing device and a computer program for carrying out the method, and a computer-readable medium on which the computer program is stored. The method comprises: - receiving a measurement signal (USS(t)) by each of the ultrasonic sensors (2) as a function of a recording time (t), - determining two-dimensional object coordinates (x',y') of at least one object (O) in a vehicle environment on the basis of the measurement signal (USS(t)) of each ultrasonic sensor (2), - Transforming the two-dimensional object coordinates (x',y') into a reference coordinate system using data obtained from at least one vehicle sensor (IMU), - expanding the transformed two-dimensional object coordinates (x,y,t) corresponding to different recording times (t) and / or ultrasonic sensors (2) to three dimensions using a dimension expansion function (h) and - Accumulating object points O(x,y,h,t) corresponding to the transformed two-dimensional object coordinates (x,y,h,t) in an environment map (EM) containing spatial information about the at least one object (O) in the vehicle environment.

Description

[0001] The present invention relates to a method, in particular a computer-implemented method, for detecting environmental information of a vehicle using a plurality of ultrasonic sensors, a data processing device and a computer program configured to carry out the method according to the present invention, and a computer-readable medium on which the computer program is stored. The acquired environmental information is preferably used for a driver assistance system, in particular for performing a driver assistance function, in particular for parking assistance.

[0002] However, the invention is not exclusively applicable to the technical field of driver assistance systems. Rather, it can be used for any application in which environmental information needs to be recorded using ultrasonic sensors. Nevertheless, and without limiting its generality, the following description will focus on an application of the invention for driver assistance systems.

[0003] Advanced driver assistance systems (ADAS) for vehicles are based on the processing of various data sensed by various ADAS sensors, such as radar, LiDAR, and ultrasonic sensors, as well as cameras. Using ADAS sensors, information about the vehicle's surroundings can be obtained, which in turn is used to implement various ADAS functions. On the one hand, ADAS functions can provide driver assistance while the driver remains in control of the vehicle. On the other hand, depending on the level of automation, a fully autonomous vehicle can be realized.Well-known ADAS functions include, for example, various methods for detecting and / or classifying objects and / or obstacles in the vicinity of the vehicle, methods for lane detection and / or lane departure, methods for rain detection, or various parking assistance functions.

[0004] In the case of parking assistance functions, various methods are known to assist a vehicle in parking, such as assisting in finding a parking space or assisting in a parking maneuver, as well as autonomously performing the parking maneuver. The higher the degree of automation, the more urgent the need for reliable and detailed information about the vehicle's surroundings at a given time.

[0005] Ultrasonic sensors mounted on a vehicle are typically used to record point information regarding the vehicle's surroundings, whereas acquired measurement points contain two-dimensional coordinates of objects in the vehicle's surroundings. Thus, ultrasonic sensors are often used to indicate the proximity of nearby objects and, for example, to assist with parking maneuvers.

[0006] To obtain more detailed environmental information, additional ADAS sensors are typically considered. One reason is that the point density obtained by ultrasonic sensors is often significantly lower compared to that which can be obtained using other types of environmental sensors, such as LiDAR sensors. Furthermore, ultrasonic sensors can only record a single measurement point at a given time, with no storage for specific past time windows. Another reason is that the measurement points only contain 2D information, neglecting the heights of the detected objects. In summary, the information obtained exclusively from ultrasonic sensors is often insufficient for advanced subsequent processing and, for example, as input for ADAS functions based on machine learning techniques.

[0007] On the other hand, installing a set of multiple sensors, especially including LiDAR sensors, results in significantly higher system costs.

[0008] Based on this, it is an object of the present invention to improve the sensing of the environment by means of ultrasonic sensors in a cost-effective manner.

[0009] This object is achieved by means of the method according to claim 1 and its use in a driver assistance system according to claim 11 as well as by means of the data processing device according to claim 12, the computer program according to claim 13 and the computer-readable medium according to claim 14.

[0010] With reference to the method, the object of the present invention is achieved by means of a method, in particular a computer-implemented method, for detecting environmental information of a vehicle by means of a plurality of ultrasonic sensors, the method comprising the following: - Receiving a measurement signal by each of the ultrasonic sensors depending on a recording time, - Determining two-dimensional object coordinates of at least one object in a vehicle environment based on the measurement signal of each ultrasonic sensor, - Transforming the two-dimensional object coordinates into a reference coordinate system using data obtained from at least one vehicle sensor, - Expanding the transformed two-dimensional object coordinates corresponding to different recording times and / or ultrasonic sensors to three dimensions using a dimension expansion function and - Accumulating object points corresponding to the transformed two-dimensional object coordinates in an environment map containing spatial information about the at least one object in the vehicle environment.

[0011] The plurality of at least two ultrasonic sensors are preferably arranged at different locations of the vehicle and are arranged such that they provide information about at least partially different areas of the vehicle environment.

[0012] Regarding the determination of two-dimensional object coordinates of the at least one object based on the measurement signal from each ultrasonic sensor, there are several possibilities, all of which fall within the scope of the present invention. Preferably, a distance between the vehicle and the object is derived from the measurement signal, and the two-dimensional coordinates are obtained by performing triangulation.

[0013] The reference coordinate system preferably corresponds to a world coordinate system. Accordingly, the environment map is also provided with reference to the world coordinate system. However, any coordinate system can serve as the common reference coordinate system.

[0014] The method of the present invention offers several advantages, namely, detailed information about the vehicle's surroundings can be obtained exclusively using ultrasonic sensors. This, in turn, corresponds to a cost-effective system configuration. Furthermore, by accumulating object points in the environment map, previously recorded or acquired object points also become available. Finally, by utilizing the height expansion function, estimates for the object heights can also be added, resulting in three-dimensional environmental information.

[0015] In a preferred embodiment, a height class of the at least one object is determined based on the measurement signal of at least one ultrasonic sensor. Preferably, the height class is determined based on the amplitude of the measurement signals received by each of the plurality of ultrasonic sensors. In addition, or instead, the height class determination may also comprise an interpolation between height classes associated with one or more neighboring measurement points or object coordinates. The determined height class may be used to expand the transformed two-dimensional object coordinates to three dimensions using the dimension expansion function. The dimension expansion function may, in particular, be based on the determined height classes.

[0016] In another preferred embodiment of the method according to the present invention, the vehicle sensor is a sensor capable of recording information about a pose of the vehicle in a world coordinate system, in particular a position sensor, a GPS sensor or at least one sensor of an inertial measurement unit.

[0017] Preferably, the measurement signals received by each of the ultrasonic sensors and the data acquired by the at least one vehicle sensor are synchronized in time. In this regard, the measurement signal from each of the sensors can be recorded over time, and measurement signals from different sensors can be collated based on their recording times. Alternatively, a frequency corresponding to a rate of accumulation of object points can be selected and used to correlate the recording times of the different sensors.

[0018] Furthermore, the two-dimensional object coordinates are preferably transformed into the reference coordinate system by means of a rigid-body transformation. The rigid-body transformation can, in particular, comprise at least one rotation and / or translation.

[0019] One embodiment of the present invention includes selecting a time interval for the accumulation of object points in the environment map. Accordingly, the environment map includes object points that correspond to the selected time interval and thus include at least the past and present for a selectable period of time.

[0020] In this regard, it is advantageous if the time interval is selected depending on a vehicle speed and / or a yaw rate of the vehicle. For example, the time interval can be selected so that it decreases with increasing vehicle speed and / or yaw rate. In particular, in a more static environment, e.g., during a parking maneuver, a longer time interval can be selected, while relatively short time intervals are more suitable at higher vehicle speeds. The time interval can also be selected depending on the yaw rate, which is particularly useful when mapping the car's surroundings near a corner.

[0021] The selection of the time interval can be automated using a suitable time interval selection function, e.g., a linear function of the vehicle speed and / or yaw rate. However, different functional relationships can also be used to select a suitable time interval for accumulating the object points into the environment map.

[0022] The method of the present invention, in particular the accumulation of object points to obtain the environment map, can be performed online, i.e., in the vehicle and while the vehicle is moving, or offline, i.e., in an external unit. In the latter case, the selected time interval can even be extended to include future points in time.

[0023] A preferred embodiment comprises selecting the dimensional expansion function taking into account the height at which the ultrasonic sensor is arranged relative to the vehicle and a uniform height distribution. The height is thus estimated based on the mounting height of the sensor on the vehicle. The height distribution can preferably be selected as a distribution that at least partially includes constant height intervals.

[0024] It is also advantageous if the dimension extension is selected with additional consideration of the specific height class of at least one object. The specific height class can also be combined with the application of a uniform height distribution.

[0025] In this regard, it is advantageous if the object height is estimated based on a maximum height defined by a cone beam emanating from the sensor and extending to the determined 2D object location.

[0026] Another preferred embodiment of the method comprises applying a filter function to the environment map, in particular a filter for omitting object points with a number of neighboring points below a selected threshold. By applying the filter function, moving objects in the vicinity of the vehicle can be removed from the environment map. Similarly, outliers and / or artifacts can be removed by applying a filter function.

[0027] Regarding filtering using the filter function, it is advantageous if object points corresponding to a predefined time period are not filtered or are added back to the surrounding map after the filter function has been applied to the entire surrounding map. This way, objects currently moving near the vehicle are not omitted.

[0028] The environment map obtained by the method of the present invention according to one of the previously described embodiments is preferably used for a driver assistance system, in particular for performing a driver assistance function, in particular for parking assistance. It is also preferably used for sensor fusion approaches, e.g., for projecting the environment map onto an image obtained by a camera, for curb height estimation, or for a depth estimation method. In the case of depth estimation, the environment map can serve, for example, as ground truth.

[0029] The problem underlying the present invention is also solved by means of a data processing device comprising means for carrying out the method of the present invention according to one of the described embodiments, and by a computer program comprising instructions for causing the computer to carry out the method according to the present invention. Finally, the problem is also solved by means of a computer-readable medium on which the computer program is stored.

[0030] It should be noted that the embodiments described in connection with the method of the present invention are equally applicable to the data processing apparatus, to the computer program and to the computer-readable medium.

[0031] The invention and its preferred embodiments will be further described on the basis of the following figures, in which Fig. 1 shows an exemplary block diagram relating to a preferred embodiment of the method for acquiring environmental information; Fig. 2 shows an exemplary surroundings map of a vehicle; Fig. 3 illustrates a preferred embodiment for object height estimation; and Fig. 4 shows two environment maps before and after applying a filter function.

[0032] In the figures, the same reference numerals are used for the same elements.

[0033] In Fig. 1 shows a block diagram relating to the method according to the present invention for detecting environmental information of a vehicle. The vehicle 1 [in Fig. 1 not shown] comprises several environmental sensors for recording information about the vehicle environment in the form of ultrasonic sensors 2. The vehicle 1 may comprise further environmental sensors, such as cameras, radar or LiDAR sensors.

[0034] In a first step of the method according to the present invention, a measurement signal USS(t) is received from each of the ultrasonic sensors 2 as a function of the recording time t. Based on the measurement signals USS(t), two-dimensional object coordinates x',y',t are determined with respect to at least one object O in the vicinity of the vehicle 1, e.g., by performing triangulation.

[0035] Taking into account additional data IMU(t) from at least one vehicle sensor [not shown], the 2D object coordinates x',y',t are transformed into a reference coordinate system x,y,t. The transformation can be a rigid-body transformation that includes at least one rotation and / or translation. It is advantageous if both the measurement signals USS(t) from the ultrasonic sensors 2 and the data IMU(t) obtained by the at least one vehicle sensor are temporally synchronized for this purpose.

[0036] Furthermore, a dimension expansion function H is used to expand the object coordinates x, y, t to three dimensions, resulting in object points O(x, y, h, t) that correspond to the transformed, height-expanded object coordinates x, y, h, t. There are several ways to expand the object coordinates x, y to three dimensions. For example, the dimension expansion function H can be used to consider a height h. sin which the ultrasonic sensor 2 is arranged with respect to or on the vehicle(s) 1, and can be based on a uniform height distribution X, here in the form of a random variable. A preferred embodiment comprises the selection of a distribution X1 in the form of X1~U(−a, a) with a constant interval a. The height h can then be estimated to h=hs+X1

[0037] Alternatively, the distribution can also be selected as X2~U(0, (tanα2)d) whereas d is the distance between the vehicle 1 and the object O and α is the angle with respect to the cone beam between the ultrasonic sensor 2 and the object O, as in Fig. 3 illustrates.

[0038] If a height class h' of the at least one object O is also determined on the basis of the measurement signals USS of each ultrasonic sensor 1, this can also be taken into account by the dimensional expansion function H. The height h can, for example, be determined such that h={hS+X2, if h'= "high" or neighboring point with "high"hS−X2, if h'= "low" or neighboring point with "low"hS+X1, if h'=? where h scorresponds to a height at which the ultrasonic sensor 2 is arranged with respect to the vehicle 1. The height estimate h depends on the associated height class, whereas a distinction is made between object points O for which the associated height class h' is "high" or "low", for a high or low object O, or for which a majority of a few neighboring points O within a certain radius are associated with the class "high" or "low", and between points O to which no height class h' can be associated.

[0039] From the further course of the Fig. 1, it can be seen that the object points O are accumulated in an environment map EM(O, Δt), which includes several of the determined, transformed object points O, which were recorded, in particular, at different times t. For the embodiment shown, the object points O are accumulated for a predefined time interval Δt. A width of the time interval Δt can be selected depending on a speed v of the vehicle 1, e.g. Δt=A−f(v) where A is a base width of the time interval Δt; and f(v) is a linear function of the vehicle speed v. In addition, or instead, the time interval Δt can be dependent on the yaw rate or angular velocity v a be selected, e.g. Δt=A−g(va)−c where g(v a) is a linear function, and c is a delay. By adding a delay c, it can be taken into account that it is not possible to look around the corner when turning the vehicle 1. The delay c is used for downsampling after turning. In a further preferred embodiment, a turning direction of the vehicle 1 can also be taken into account.

[0040] Optionally, a filter function F can be applied to the environment map, resulting in EM'(O, Δt). This is described with reference to Fig. 4 further illustrates this.

[0041] Fig. Figure 2 shows an exemplary environment map EM(O,Δt) recorded by a moving vehicle 1, for which object points O were accumulated for the predefined time interval Δt. The solid line corresponds to a trajectory t of the vehicle 1, and the dots correspond to recorded object points O.

[0042] Fig. Figure 3 illustrates a preferred embodiment regarding height extension using the dimensional extension function H. A vehicle 1 is equipped with an ultrasonic sensor 2 which is mounted at the mounting height h s is installed. The height h of the object O is estimated using the tangent of the angle α of the cone beam and the distance d between the sensor 2 and the object O.

[0043] Finally, Fig. 4 shows an example of filtering an environment map EM. In Fig. 4a shows an environment map EM before applying the filter function F and in Fig. 4b shows the same EM map after applying the filter function F. For comparison, environmental information obtained from a LiDAR sensor is shown on the EM environmental maps in Fig. 4. The dashed points in the Fig. 4a and Fig. 4b refer to an accumulated environmental map EM generated from measurement points of multiple ultrasonic sensors 2, and the lines correspond to a reference map derived from data acquired by a LiDAR sensor. By comparing the information obtained using the two different sensor types, it can be deduced that the accuracy of the environmental map EM generated according to the present invention and by accumulating measurement data from multiple ultrasonic sensors 2 results in an environmental information quality comparable to that obtainable using a LiDAR sensor.

[0044] The proposed filtering serves to remove outliers or artifacts of temporal accumulation. The filter function F is selected such that it identifies object points O with a number of neighboring object points O below a selected or predefined threshold within a certain radius around the considered object point O. Such object points O typically correspond to moving objects. If a moving object is only present at a certain time, the corresponding environment map EM, when accumulated for the predefined time interval Δt, is blurred and the object points O corresponding to moving objects typically also have a lower density of neighboring points within a certain radius. Such object points O corresponding to a moving object are Fig. 4a by the crossed point surrounded by a circle. After applying the filter function F, the points marked with the circle are removed, as shown on the filtered environment map EM' in Fig. 4b can be seen.

[0045] In a preferred embodiment, a second, particularly short, period of time is selected for which all recorded object points O are reinserted into the filtered environment map EM'. Thus, the most recent information about moving objects O that are currently in the vicinity of the vehicle 1 is still contained in the environment map EM.

Claims

[1] Method, in particular a computer-implemented method, for detecting environmental information of a vehicle (1) by means of a plurality of ultrasonic sensors (2), the method comprising the following: - receiving a measurement signal (USS(t)) by each of the ultrasonic sensors (2) as a function of a recording time (t), - determining two-dimensional object coordinates (x',y') of at least one object (O) in a vehicle environment on the basis of the measurement signal (USS(t)) of each ultrasonic sensor (2), - Transforming the two-dimensional object coordinates (x',y') into a reference coordinate system using data obtained from at least one vehicle sensor (IMU), - expanding the transformed two-dimensional object coordinates (x,y,t) corresponding to different recording times (t) and / or ultrasonic sensors (2) to three dimensions using a dimension expansion function (h) and - Accumulating object points O(x,y,h,t) corresponding to the transformed two-dimensional object coordinates (x,y,h,t) in an environment map (EM) containing spatial information about the at least one object (O) in the vehicle environment. [2] Method according to claim 1, wherein a height class (h') of the at least one object (O) is determined on the basis of the measurement signal (USS(t)) of at least one ultrasonic sensor (2). [3] Method according to claim 1 or 2, wherein the vehicle sensor is a sensor capable of recording information about a pose of the vehicle (1) in a world coordinate system, in particular a position sensor, a GPS sensor or at least one sensor of an inertial measurement unit. [4] Method according to one of the preceding claims, wherein the measurement signals (USS(t)) received by each of the ultrasonic sensors (2) and data (IMU(t)) obtained from the at least one vehicle sensor are synchronized in time. [5] Method according to one of the preceding claims, wherein the two-dimensional object coordinates (x',y',t) are transformed into the reference coordinate system by means of a rigid body transformation. [6] Method according to one of the preceding claims, wherein a time interval (Δt) is selected for the accumulation of the object points (O(x,y,h,t)) in the environment map (EM). [7] Method according to claim 6, wherein the time interval (Δt) is determined as a function of a vehicle speed (v) and / or a yaw rate (v a ) of the vehicle (1) is selected. [8] Method according to one of the preceding claims, wherein the dimension extension function (H) is calculated taking into account a height (h s ), in which an ultrasonic sensor (2) is arranged with respect to the vehicle (1), and a uniform height distribution (X,X1,X2) is selected. [9] Method according to claim 3, wherein the dimensional extension function (H) is selected with additional consideration of the determined height class (h') of the at least one object (O). [10] Method according to one of the preceding claims, wherein a filter function (F) is applied to the environment map (EM), in particular a filter for omitting object points (O) with a number of neighboring points below a selected threshold value. [11] Use of the environment map (EM) obtained by carrying out the method according to one of the preceding claims for a driver assistance system, in particular for carrying out a driver assistance function, in particular for the parking aid. [12] Data processing device comprising means for carrying out the method according to any one of claims 1-10. [13] A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1-10. [14] A computer-readable medium on which the computer program according to claim 13 is stored.

Images