Method for acquiring environmental information of a vehicle by means of multiple ultrasonic sensors
Patent Information
- Application Number
- PCT/EP2025/054414
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-05
- Filing Date
- 2025-02-19
- Publication Date
- 2025-10-02
Smart Images
Figure EP2025054414_02102025_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] Method for acquiring environmental information of a vehicle by means of multiple ultrasonic sensors
[0003] The present invention relates to a method, in particular a computer-implemented method, for acquiring environmental information of a vehicle by means of multiple ultrasonic sensors, a data processing device and computer program embodied to carry out the method according to the present invention, and a computer-readable medium on which the computer program is stored. The obtained environmental information is preferable used for a driver assistance system, in particular for carrying out a driver assistance function, in particular for parking assistance.
[0004] The invention, however, is not solely applicable in the technical field of driver assistance systems. Rather, it can be used for any application in which environmental information needs to be recorded by using ultrasonic sensors. Nonetheless and without loss of generality, the following description will focus on an application of the invention for driver assistance systems.
[0005] Advanced driver assistance systems (ADAS) for vehicles are based on a processing of various data sensed by various ADAS sensors, such as radar-, LiDAR- and ultrasonic sensors as well as cameras. By means of the ADAS sensors information relating to an environment of the vehicle can be obtained which in turn is used to realize various ADAS functions. ADAS functions on the one hand may include an assistance for the driver while control of the vehicle remains with the driver. On the other hand, depending on the level of automation, a full autonomously driving vehicle may be realized. Known ADAS functions for instance are various methods for detecting and / or classifying objects and / or obstacles in the vicinity of the vehicle, methods for lane detection and / or lane departure, methods for rain detection or also various parking assistance functions.
[0006] In case of parking assistance functions, various methods to support parking of a vehicle are known, e. g. support to search for a parking slot or to assist with a parking process, but also for autonomously performing the parking process. The higher the level or automation, the more reliable and detailed information about the vehicle environment at a given time is mandatory.
[0007] Ultrasonic sensors arranged on a vehicle typically are used to record point information relating to an environment of the vehicle, whereas obtained measurement points include two dimensional coordinates of objects in the surrounding of the vehicle. Thus, ultrasonic sensors are frequently used to indicate the proximity of nearby objects and e. g. for assisting a parking process.
[0008] For obtaining more detailed environmental information additional ADAS sensors are typically taken into account. One reason is, that the point density obtained from ultrasonic sensors frequently is significantly lower compared to that which can be obtained by using different types of environmental sensors, e. g. LiDAR sensors. In addition, ultrasonic sensors only able to record one single measurement point at a given time not providing memory for certain time windows back in the past. Another reason is, that the measurement points only contain 2D information neglecting heights of the detected objects. In summary, the information solely obtained from ultrasonic sensors is frequently not sufficient for advanced subsequent processing and e. g. as input for ADAS functions based on machine learning methods.
[0009] On the other hand, installation of a set of multiple sensors, in particular including LiDAR sensors is associated with significantly higher costs of the system.
[0010] Based thereon it is an object of the present invention to improve environmental sensing by means of ultrasonic sensors in a cost efficient manner.
[0011] This object is achieved by means of the method according to claim 1 , and its use in an driver assistance system according to claim 11 , as well as by means of the data processing device according to claim 12, the computer program according to claim 13 and the computer-readable medium according to claim 14. With reference to the method the object of the present invention is achieved by means of a method, in particular a computer-implemented method, for acquiring environmental information of a vehicle by means of multiple ultrasonic sensors, the method comprising: receiving a measurement signal by each of the ultrasonic sensors as a function of a recording time, determining two-dimensional object coordinates of at least one object in a vehicle environment based on the measurement signal of each ultrasonic sensor, transforming the two-dimensional object coordinates to a reference coordinate system by using data obtained from at least one vehicle sensor, extending the transformed two-dimensional object coordinates corresponding to different recording times and / or ultrasonic sensors into three dimensions by using a dimension extension function, and accumulating object points corresponding to the transformed two-dimensional object coordinates in an environmental map containing spatial information about the at least one object in the vehicle environment.
[0012] The multiple, at least two, ultrasonic sensors are preferably arranged at different positions of the vehicle and are arranged such that they provide information about at least partially different areas of the vehicle environment.
[0013] Regarding the determination of two-dimensional object coordinates of the at least one object based on the measurement signal of each ultrasonic sensor there are multiple possibilities which all fall under the present invention. Preferably, a distance between the vehicle and the object is derived from the measurement signal and the two dimensional coordinates are obtained by performing a triangulation.
[0014] The reference coordinate system preferably corresponds to a world coordinate system. Accordingly, also the environmental map is provided with reference to the world coordinate system. However, any coordinate system may serve as a common reference coordinate system. The method of the present invention involves several advantages, i. e. detailed information about the vehicle environment may be obtained solely by using ultrasonic sensors. This in turn corresponds to a cost efficient system set up. In addition, by accumulating object points in the environmental map, previously recorded or obtained object points become available as well, and finally by utilizing the height extension function, estimates for the object heights can be added as well resulting in three-dimensional environmental information.
[0015] In a preferred embodiment, a height class of the at least one object is determined based on the measurement signal of at least one ultrasonic sensor. Preferably, the height class is determined based on the amplitude of the measurement signals received by each of the multiple ultrasonic sensors. In addition, or instead, the height class determination may also include an interpolation between height classes associated to one or multiple neighboring measurement points or object coordinates. The determined height class may be used for the extension of the transformed two-dimensional object coordinates into three dimensions by using the dimension extension function, The dimension extension function in particular may be based on the determined height classes.
[0016] In another preferred embodiment of the method according to the present invention the vehicle sensor is a sensor capable of recording information about a pose of the vehicle in a world coordinate system, in particular a position sensor, a GPS sensor, or at least one sensor of an inertial measurement unit.
[0017] It is preferred, if the measurement signals received by each of the ultrasonic sensors and data obtained from the at least one vehicle sensor are synchronized in time. In this regard, the measurement signal of each of the sensors may be recorded as a function of time and measurement signals of different sensors maybe matched in view of their recording times. Alternatively, a frequency corresponding to a rate of accumulation of object points may be chosen and used for correlating recording times of the different sensors. Further, it is preferred, if the two-dimensional object coordinates are transformed to the reference coordinate system by means of a rigid body transformation. The rigid body transformation in particular may include at least one rotation and / or translation.
[0018] One embodiment of the present invention includes that a time interval for accumulation of the object points in the environmental map is selected. Accordingly, the environmental map comprises object points corresponding to the selected time interval and thus for a selectable time period at least including past and present times.
[0019] In this regard, it is of advantage, if the time interval is chosen in dependence of a vehicle velocity and / or a yaw rate of the vehicle. For instance, the time interval may be selected to be smaller the faster a velocity of the vehicle and / or its yaw rate. In particular, a longer time interval may be chosen in a rather static environment, e.g. during a parking process, while relatively small time intervals are more suitable at higher vehicle velocities. The time interval may also be selected in dependence of the yaw rate which is particular useful when the environment of the car is mapped close to a corner.
[0020] The selection of the time interval may be automated by using an appropriate time interval selection function, e. g. a linear function of the vehicle velocity and / or yaw rate. But also different functional relations can be chosen for selecting an appropriate time interval for accumulation of the object points into the environmental map.
[0021] The method of the present invention, in particular the accumulation of object points to obtain the environmental map may be conducted online, i. e. in the vehicle and during driving of the vehicle, or offline, i. e. in an external unit. In the latter case, the selected time interval may even be extended so as to also take into account points in time relating to the future.
[0022] A preferred embodiment comprises that the dimension extension function is chosen by taking into account a height at which height the ultrasonic sensor is arranged relative to the vehicle, and a uniform height distribution. The height is thus estimated based on the installation height of the sensor on the vehicle. The height distribution may preferably be chosen as a distribution comprising at least partially constant height intervals.
[0023] In addition, it is of advantage, if the dimension extension is chosen by further considering the determined height class of the at least one object. The determined height class may also be combined with applying a uniform height distribution.
[0024] In this regard, it is advantageous, if the object height is estimated based on a maximal height defined by a conical beam starting from the sensor and extending to the determined 2D object location.
[0025] Another preferred embodiment of the method comprises that a filter function is applied to the environmental map, especially a filter to omit object points with a number of neighboring points below a chosen threshold. By applying the filter function, moving objects in the vicinity of the vehicle may be removed from the environmental map. Similarly, outliers and / or artifacts maybe removed by applying a filter function.
[0026] With regards to the filtering by using the filter function it is of advantage if object points corresponding to a predefined time period are not filtered or added again to the environmental map after applying the filter function is applied to the entire environmental map. That way, presently moving objects in the vicinity of the vehicle are not omitted.
[0027] The environmental map obtained by the method of the present invention according to any of the embodiments previously described is preferably used for a driver assistance system, in particular for carrying out a driver assistance function, in particular for parking assistance. It is as well preferably used for sensor fusion approaches, e. g. for projecting the environmental map on an image obtained by a camera, for a curb height estimation or for a depth estimation method. In case of a depth estimation, the environmental map may e. g. serve as a ground truth. The objective problem underlying the present invention is as well solved by means of a data processing device comprising means for carrying out the method of the present invention according to any of the embodiments described and by a computer program comprising instructions to cause the computer to carry out the method according to the present invention. Finally, the objective problem is also solved by means of a computer-readable medium on which the computer program is stored.
[0028] It shall be noted that the embodiments described in connection with the method of the present invention are as well applicable to the data processing device, to the computer program and to the computer-readable medium.
[0029] The invention and its preferred embodiments will be further described based on the following figures, wherein
[0030] Fig. 1 shows an exemplary block diagram referring to a preferred embodiment of the method for acquiring environmental information;
[0031] Fig. 2 an exemplary environmental map of a vehicle;
[0032] Fig. 3 illustrates a preferred embodiment for object height estimation; and
[0033] Fig. 4 shows two environmental maps before and after applying a filter function.
[0034] In the figures the same reference symbols are used for the same elements.
[0035] In Fig. 1 a block diagram referring to the method according of the present invention for acquiring environmental information of a vehicle is shown. The vehicle 1 [not shown in Fig. 1 ] comprises multiple environmental sensors to record information about the vehicle environment in the form of ultrasonic sensors 2. The vehicle 1 may comprise further environmental sensors, such as cameras, radar or LiDAR sensors. In a first step of the method according to the present invention a measurement signal USS(t) is received from each of the ultrasonic sensors 2 as a function of recording time t. Based on the measurement signals USS(t) two-dimensional object coordinates x’,y’,t referring to at least one object 0 in the vicinity of the vehicle 1 are determined, e.g. by performing a triangulation.
[0036] Taking into account additional data IMll(t) from at least one vehicle sensor [not shown] the 2D object coordinates x’,y’,t are transformed to a reference coordinate system x,y , t. The transformation may be a rigid body transformation including at least one rotation and / or translation. It is of advantage, when both the measurement signals USS(t) from the ultrasonic sensors 2 and the data IMll(t) obtained by the at least one vehicle sensor are synchronized in time for this purpose.
[0037] Further, a dimension extension function H is used to extend the object coordinates x,y,t into three dimensions resulting in object points O(x,y,h,t) corresponding to the transformed, height extended object coordinates x,y,h,t. There are several possibilities to extend the object coordinates x,y into three dimensions. The dimension extension function H for instance may be chosen by taking into account a height hsat which the ultrasonic sensor 2 is arranged relative to or on the vehicle 1 , and may be based on a uniform height distribution X, here in the form of a random variable. One preferred embodiment comprises to choose a distribution X1 in the form of with a constant interval a. The height h may then be estimated to h = hs+ XI
[0038] Alternatively, the distribution may also be chosen as a X2~U Q, (tan-)d) whereas d is the distance between the vehicle 1 and object 0 and a is the angle relating to the conical beam between the ultrasonic sensor 2 and the object 0, as illustrated in Fig. 3. If a height class h’ is of the at least one object 0 is determined based on the measurement signals USS of each ultrasonic sensor 1 as well, this may be also taken into account by the dimension extension function H. For instance, the height h may be determined such that
[0039] {hs+ X2 if h' = "high", or neighbour point with "heigh" hs— X2 if h' = "low", or neighbour point with "low" hs+ XI if h' =? wherein hscorresponds to a height at which height the ultrasonic sensor 2 is arranged relative to the vehicle 1 . The height estimation h depends on the associated height class whereas, a distinction is made between object points 0 for which the associated height class h’ is “high” or “low”, for a high or low object 0, or for which a majority of few neighboring points 0 in a given radius are associated with the class “high” or “low”, and between points 0 to which no height class h’ can be associated.
[0040] Following the block diagram shown in Fig. 1 , the object points 0 are accumulated in an environmental map EM(0, At) comprising multiple of the determined, transformed object points 0, especially recorded at different points in time t. For the embodiment shown, the object points 0 are accumulated for a predefined time interval At. A width of the time interval At may be chosen in dependence of a velocity v of the vehicle 1 , e. g.
[0041] At = A - (v) wherein A is a base width of the time interval At; and f(v) is a linear function of the vehicle velocity v. In addition, or instead, the time interval At may be chosen in dependence of the yaw rate or angular velocity va, e.g.
[0042] At = A - g(va) - c wherein g(va) is a linear function c is a delay. By adding a delay c it can be taken into account that when turning the vehicle 1 it is not possible to look behind the corner. The delay c serves for a downsampling after the turn. In a further preferred embodiment, a turning direction of the vehicle 1 may be taken into account as well.
[0043] Optionally, a filter function F may be applied to the environmental map resulting in EM’(O, At). This is further illustrated with reference to Fig. 4.
[0044] Fig. 2 shows an exemplary environmental map EM(O,At) recorded from a moving vehicle 1 for which object points 0 have been accumulated for the predefined time interval At. The solid line corresponds to a trajectory t of the vehicle 1 , and the dots correspond to recorded object points 0.
[0045] Fig. 3 illustrates a preferred embodiment referring to the height extension using the dimension extension function H. A vehicle 1 is equipped with an ultrasonic sensor 2 installed at the installation height hs. The height h of the object 0 is estimated by using the tangent of the angle a of the conical beam and the distance d between the sensor 2 and the object 0.
[0046] Fig. 4 finally exemplarily illustrates a filtering of an environmental map EM. In Fig. 4a an environmental map EM before applying filter function F is shown, and in Fig. 4b the same map EM is shown after applying the filter function F. For comparison, environmental information obtained from a LiDAR sensor is superimposed on the environmental maps EM in Fig. 4. The dotted points in Figs. 4a and 4b refer to an accumulated environmental map EM produced by measurement points of several ultrasonic sensors 2 and the lines correspond to a reference map deduced from data obtained from a LiDAR sensor. Comparing the information obtained by using the two different sensor types it can be deduced that the accuracy of the environmental map EM produced according to the present invention and by accumulating measurement data from multiple ultrasonic sensors 2 yields in an environmental information quality comparable to that obtainable by using a LiDAR sensor.
[0047] The suggested filtering serves to omit outliers or artefacts of the temporal accumulation. The filter function F is chosen such that it identifies object points 0 with a number of neighboring object points 0 below a chosen or predefined threshold in a given radius around the object point 0 regarded. Such object points 0 typically correspond from moving objects. If a moving object is present only at a certain time, the corresponding environmental map EM, when accumulated for the predefined time interval At, is blurred and the object points 0 corresponding to moving objects typically also have a lower density of neighboring points in a given radius. Such object points 0 corresponding to a moving object are indicated in Fig. 4a by the crossed point surrounded by a circle in Fig. 4a. After applying the filter function F, the points marked by the circle are removed as can be seen in the filtered environmental map EM’ in Fig. 4b.
[0048] In a preferred embodiment, a second, especially small, time period is chosen for which all recorded object points 0 are reinserted into the filtered environmental map EM’. That way, latest information about moving objects 0, which are presently in the vicinity of the vehicle 1 , are still contained in the environmental map EM.
Claims
Patent claims1. Method, in particular a computer-implemented method, for acquiring environmental information of a vehicle (1 ) by means of multiple ultrasonic sensors (2), the method comprising: receiving a measurement signal (USS(t)) by each of the ultrasonic sensors 2 as a function of a recording time (t), determining two-dimensional object coordinates (x’y’) of at least one object (0) in a vehicle environment based on the measurement signal (USS(t) of each ultrasonic sensor (2), transforming the two-dimensional object coordinates (x’,y’) to a reference coordinate system by using data (IMU) obtained from at least one vehicle sensor, extending the transformed two-dimensional object coordinates (x,y,t) corresponding to different recording times (t) and / or ultrasonic sensors (2) into three dimensions by using a dimension extension function (h), and accumulating object points O(x,y,h,t) corresponding to the transformed two-dimensional object coordinates (x,y,h,t) in an environmental map (EM) containing spatial information about the at least one object (0) in the vehicle environment.
2. Method according to claim 1 , wherein a height class(h’) of the at least one object (0) is determined based on the measurement signal (USS(t) of at least one ultrasonic sensor (2).
3. Method according to claim 1 or 2, wherein the vehicle sensor is a sensor capable of recording information about a pose of the vehicle (1 ) in a world coordinate system, in particular a position sensor, a GPS sensor, or at least one sensor of an inertial measurement unit.
4. Method according to any of the preceding claims, wherein the measurement signals (USS(t)) received by each of the ultrasonic sensors (2) and data (IMll(t)) obtained from the at least one vehicle sensor are synchronized in time.
5. Method according to any of the preceding claims, wherein the two-dimensional object coordinates (x’y’,t) are transformed to the reference coordinate system by means of a rigid body transformation.
6. Method according to any of the preceding claims, wherein a time interval (At) for accumulation of the object points (O(x,y,h,t)) in the environmental map (EM) is selected.
7. Method according to claim 6, wherein the time interval (At) is chosen in dependence of a vehicle velocity (v) and / or a yaw rate (va) of the vehicle (1 ).
8. Method according to any of the preceding claims, wherein the dimension extension function (H) is chosen by taking into account a height (hs) at which height (hs) an ultrasonic sensor (2) is arranged relative to the vehicle (1 ), and a uniform height distribution (X,X1 ,X2).
9. Method according to claim 3, wherein the dimension extension function (H) is chosen by further considering the determined height class (h’) of the at least one object (0).
10. Method according to any of the preceding claims, wherein a filter function (F) is applied to the environmental map (EM), especially a filter to omit object points (0) with a number of neighboring points below a chosen threshold.11 . Use of the environmental map (EM) obtained by performing the method according to any of the preceding claims for a driver assistance system, in particular for carrying out a driver assistance function, in particular for parking assistance.
12. A data processing device comprising means for carrying out the method of any of claims 1 -10.
13. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 1-10.
14. A computer-readable medium having stored thereon the computer program of claim 13.