Method for identifying speed humps

Abstract

The invention relates to a method (100) for detecting speed humps (30), comprising the following steps: - obtaining (101) input data (111), wherein the input data (111) result at least in part from a sensor detection of a surrounding region of a vehicle (1), the input data (111) comprise at least one scene representation (201) and a corresponding depth map (121), the scene representation (201) represents a scene in the surrounding region and the depth map (121) indicates distances in the scene, - obtaining (102) a road surface estimate (122) in the scene on the basis of the input data (111), - detecting (103) a speed hump (30) in the scene on the basis of an evaluation of the depth map (121) and the road surface estimate (122).

Description

[0001] Description

[0002] title

[0003] Method for detecting road humps

[0004] The present invention relates to a method for detecting road humps. Furthermore, the invention relates to a computer program and a data processing device.

[0005] State of the art

[0006] There are several conventional methods for speed bump detection using video data in the case of autonomous driving.

[0007] Some existing methods focus on processing grayscale images, for example, Devapriya, W., C. Nelson Kennedy Babu, and T. Srihari. "Advance driver assistance system (ADAS) - speed bump detection." 2015 IEEE international conference on computational intelligence and computing research (ICCIC). IEEE, 2015; and Devapriya, W., C. Nelson Kennedy Babu, and T. Srihari. "Real time speed bump detection using Gaussian filtering and connected component approach." 2016 World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave). IEEE, 2016, relying on the fact that the speed bump contains markings and has repeated coloring. These methods typically utilize morphological operations on the grayscale image, labeling of connected components, and signal processing.

[0008] Other methods use deep learning combined with depth processing (stereovision), for example, Ballinas-Hernandez, Ana Luisa, Ivan Olmos Pineda, and J. Arturo Olvera-López. "Speed ​​Bump Detection on Roads using Artificial Vision." Res. Comput. Sci. 148.9 (2019): 71-82, where speed bumps are also detected from 3D features such as the height and gradient of the bump.

[0009] Finally, there are methods that rely solely on artificial intelligence and training a neural network to detect speed bumps. For example, Patil, Sandip Omprakash, VV Sajith Variyar, and KP Soman. "Speed ​​bump Segmentation: an application of conditional generative adversarial network for self-driving vehicles." 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020.

[0010] Disclosure of the invention

[0011] According to aspects of the invention, a method having the features of claim 1, a computer program having the features of claim 9, and a data processing device having the features of claim 10 are provided. Further features and details of the invention are disclosed in the respective dependent claims, in the description, and in the drawings. Features and details described in the context of the method according to the invention also correspond to the computer program and the data processing device, and vice versa.

[0012] According to a first aspect of the invention, a method for detecting road humps comprises obtaining input data, in particular depth input data. The input data may result at least partially from sensor detection of an environment of a vehicle. The input data may comprise at least one scene representation, such as an input image, and a corresponding depth map. The scene representation represents a scene of the environment, which may comprise a representation of a road and objects adjacent to and / or on the road, in particular in the direction of travel of the vehicle. The depth map may indicate distances in the scene, such as the distance to the surface of the road and / or to the objects. The depth map may be obtained from flow estimation and depth calculation (in particular in the case of a mono camera used as a sensor) and / or from stereo vision (in particular in the case of a stereo camera used as a sensor).Therefore, the depth map can be obtained in the form of a three-dimensional or 3D depth map, where each pixel of the depth map contains the estimated depth between the camera and the associated 3D world location in the scene.

[0013] Furthermore, the method may include obtaining a road surface estimate in the scene based on the input data, in particular the depth input data and / or the depth map. The input data may include the road surface estimate in the form of a planar estimate or a non-planar estimate. In the case of a planar model, it may be described by the normal vector and the height of the plane. The road surface estimate may be used to estimate the distance between a 3D point and the road in the scene.

[0014] Furthermore, the input data may optionally include a semantic segmentation of the scene, including in particular the road and / or sidewalk and / or raised structures, which may be used to reject sidewalks and other raised structures that are not part of speed bumps in subsequent processing.

[0015] Furthermore, the input data may optionally include a steering angle, which can be used to deactivate the speed bump detection during left and right turns of the vehicle.

[0016] The method may also include detecting a speed bump in the scene based on an evaluation of the depth map and the road surface estimation. The method according to the invention may thereby allow detection of speed bump objects of various dimensions and colors located on the drivable road surface, in particular based on video inputs. The method steps may be performed by an algorithm that thereby relies primarily on processing 3D data together with a road surface estimation. The location of the speed bump object may be determined in image space using a bounding box. Furthermore, each speed bump object may be considered to have the following 3D attributes: distance and height. The inventive approach can work for both marked and unmarked speed bumps because it processes 3D data instead of 2D image data.The inventive method can thus allow the detection of speed bumps of different heights, colors (textured or textureless), materials, and appearances, especially when their height is elevated relative to the road. Conventional image processing and machine learning approaches, however, rely on 2D image data and may have difficulty detecting unique speed bumps that do not necessarily appear in the training data. According to the inventive method, 3D attributes can be provided for each detected speed bump, in particular distance and height. Based on this information, the detection can accordingly be used in controlling a vehicle, in particular for braking functions and suspension control. The inventive method can also work on mono cameras with noisy depth data.Thus, it has the advantage of working even in the presence of noise in the 3D map. In comparison, stereo vision-based disparity usually provides a better and more accurate 3D map, but mono camera sensors are technically less complex.

[0017] It is also possible for the evaluation to comprise determining a height-above-ground map, in particular as part of a feature accumulation step. The height-above-ground map may indicate a height of surfaces in the scene and may be determined based on the depth map and the road surface estimate, and in particular based on a semantic segmentation of the road. The evaluation and in particular the feature accumulation step may further comprise determining a plurality of height distributions of height values ​​along the road, in particular for different rungs along the road, based on the determined height-above-ground map, in particular for the indicated height of surfaces in the scene. The evaluation may further comprise classifying the determined height distributions into at least one raised threshold segment and one flat road segment based on the height values.The elevation distributions can be determined for a search region of interest in the scene. Each rung of the search region of interest can then be classified as one of (at least or exactly) two classes: raised-threshold segment or flat-road segment. A raised-threshold segment can be a segment that has dense 3D data and a height above the ground that exceeds a threshold. Furthermore, the validity of a threshold segment can be influenced by the standard deviation of depths according to the depth map and / or the height of surfaces.

[0018] It is also possible for the depth map to indicate the distances for various elements corresponding to the scene, in particular pixels, wherein the height-above-ground map is determined based on an element-by-element examination of each of the elements, wherein the element-by-element examination preferably comprises an examination of conditions, in particular whether the respective element represents a static and / or flat structure, in particular at a valid depth, and / or whether the respective element is classified as a road area by the semantic segmentation. Using the height-above-ground map and the above filtering conditions, the distributions of the height values ​​along each rung can be calculated, allowing reliable speed bump detection.

[0019] It is possible to perform classification based on a height error model to consider the height error for the specified height of surfaces in the scene at different distances in the scene. For this purpose, the allowable error can be calculated for each threshold segment candidate. A threshold segment can contain multiple height measurements. The larger the number of measurements, the smaller the theoretical error, which can also be considered using the height error model.

[0020] Furthermore, adjacent threshold segments can be merged into a single threshold object. Clustering can be performed based on the detected threshold segments, whereby adjacent threshold segments can be merged into a single threshold object. To merge two segments, they may need to satisfy the condition of being close to each other in both 2D and 3D space. Furthermore, a threshold object can be obtained based on the detection, wherein the threshold object represents the road threshold by three-dimensional attributes, in particular distance and height. Furthermore, the evaluation can include an evaluation of dimensional features of the input data to thereby detect the road threshold based on the height of the raised road threshold with respect to the road in the scene.

[0021] If necessary, the vehicle can be controlled based on the detected speed bump, with the control including, in particular, braking functions and / or suspension control and / or autonomous driving. This allows for reliable observation of speed bumps while the vehicle is moving.

[0022] In another aspect of the invention, a computer program may be provided, in particular a computer program product, comprising instructions which, when executed by a computer, cause the computer to perform the method according to the invention. The computer program according to the invention may thus have the same advantages as described in detail with reference to a method according to the invention.

[0023] In another aspect of the invention, a data processing device can be provided that is designed to carry out the method according to the invention. For example, a computer that executes the computer program according to the invention can be provided as the device. The computer can comprise at least one processor that can be used to execute the computer program. Furthermore, a non-volatile data memory can be provided in which the computer program can be stored and from which the computer program can be read by the processor for execution.

[0024] According to another aspect of the invention, a computer-readable storage medium can be provided that includes the computer program according to the invention. The storage medium can be formed as a data storage device, such as a hard disk and / or a non-volatile memory and / or a memory card and / or a semiconductor drive. The storage medium can, for example, be integrated into the computer.

[0025] Furthermore, the method according to the invention can be implemented as a computer-implemented method.

[0026] Further advantages, features, and details of the invention will become apparent from the following description, in which embodiments of the invention are described in detail with reference to the drawings. In this connection, the features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show schematically:

[0027] Fig. 1 shows an exemplary architecture of embodiments of the invention.

[0028] Fig. 2 a 3D depth map of a scene with a threshold

[0029] Fig. 3 a visualization of the output.

[0030] Fig. 4 a height-above-ground map for an exemplary given scene.

[0031] Fig. 5 another example scene.

[0032] Fig. 6 a height-above-ground distribution in a street area.

[0033] Fig. 7 shows a height-above-ground distribution in an area containing a threshold.

[0034] Fig. 8 a classification of segments as road and sleeper segments.

[0035] Fig. 9 different threshold segment detections.

[0036] Fig. 10 shows individual threshold segments and a resulting threshold object. Fig. 11 shows another example scene.

[0037] Fig. 12 a street profile.

[0038] Fig. 13 Detection examples.

[0039] Fig. 14 further detection examples.

[0040] Fig. 15 a visualization of a method, a computer program and a device according to embodiments of the invention.

[0041] In the following figures, identical reference numerals are used for the same technical features even in different embodiments.

[0042] Figure 1 shows an exemplary architecture of embodiments of the invention. The method steps shown relate to the algorithm 113 and its inputs 111 and outputs 112, as will be described in more detail below.

[0043] The functional means for providing the method steps related to threshold detection may be referred to as a threshold detection component. According to Fig. 1, the following inputs 111 may be required by the threshold detection component: a 3D depth map 121, a road surface estimate 122 (planar or non-planar), also referred to as a road surface model 122, and in particular, a semantic segmentation 123 of the scene / road and / or a steering angle 124 of the vehicle. Using these inputs 111, the algorithm 113 may first examine 131 the inputs 111 and then perform feature accumulation 132. The feature accumulation 132 may also use results from the road surface model 122 and the semantic segmentation 123. Furthermore, the 3D depth map 121 may be used for feature accumulation 132.Using the resulting height above ground, segment classification 133 can be performed. Threshold object clustering and detection 134 can then be performed using the threshold segments. The resulting threshold objects can be used to generate a threshold object list 141, which specifically contains threshold objects described using a two-dimensional bounding box, the distance, and the height information. Furthermore, a road elevation profile 142 can be generated using the results from clustering and detection 143.

[0044] The 3D depth map can be considered the main input (see Fig. 2). In this map, each pixel contains the estimated depth between the sensor, specifically camera 40 (see Fig. 15), and the associated 3D world location. This can be obtained from flow estimation and depth calculation in the case of mono cameras, and from stereo vision in the case of stereo cameras. It can be used primarily to calculate the height-above-ground map and to estimate the 3D attributes of threshold objects.

[0045] A second necessary input may be the road surface estimate, which can be a planar or non-planar estimate. In the case of a planar model, it can be described by the normal vector and the height of the plane. It can be used to estimate the distance between a 3D point and the road.

[0046] Semantic segmentation of the scene / road is also an optional input that aids feature accumulation by rejecting sidewalks and other raised structures that are not part of the speed bumps. Information about the steering angle is another optional input that can be used to deactivate the detection component during the vehicle's left and right turns.

[0047] The output of the detection algorithm can be threshold objects in the scene. For each threshold object, there can be associated 2D and 3D data. 2D data represents the boundary box of the speed bump in image space. 3D data attributes refer to the distance to the speed bump (between the camera and the threshold) and the height above the ground of the speed bump (relative to the road surface). Additionally, a confidence score between 0 and 100 is calculated for each speed bump object. Fig. 3 shows the visualization of the output.

[0048] A first step of a method according to embodiments of the invention may be a feature accumulation step. In this step, the height-above-ground feature may be calculated and a statistical analysis may be performed on the distribution of height values ​​along each rung in the search region. Fig. 4 shows a height-above-ground map for an exemplary given scene. Along each rung, a height-above-ground distribution may be calculated, in particular based on the pixels that satisfy the following conditions: valid depth, static (not dynamic), relatively flat region (based on the local surface normal), classified as part of the road area by semantic segmentation. Using the height-above-ground map and the above filtering conditions, the distributions of the height values ​​along each rung may be calculated.The mean and standard deviation of any distribution can be calculated, in particular followed by an outlier removal process, resulting in a more accurate mean height estimate.

[0049] Fig. 5 shows another example scene, where rung 501 represents a road rung and rung 502 contains a threshold object. These can each form a sampling region 501, 502, and Fig. 6 and Fig. 7 show the distribution of height values ​​for these sampling regions 501, 502. According to Fig. 6, the height-above-ground distribution in a road area 501 also includes a small portion of a sidewalk 601. The sidewalk region 601 can be considered an outlier and filtered. The mean of the remaining outliers is close to 0 meters, corresponding to a patch on the road (not elevated). According to Fig. 7, the height-above-ground distribution is shown in an area 502 containing a threshold.

[0050] Fig. 8 shows the classification of segments as road and threshold segments. Each rung of the search region of interest 801 can be divided into one of two classes: raised threshold segment and flat road segment. A raised threshold segment can be a segment that has dense 3D data and a height above ground that exceeds a threshold (also taking into account the height-above-ground error at that distance). Furthermore, the validity of a threshold segment can be influenced by the standard deviation of height measurements (a distribution with a high height standard deviation cannot be considered). Fig. 8 shows an example of four individually detected threshold segments 802.

[0051] According to embodiments of the invention, a height error model may be used. When classifying segments into threshold and non-threshold classes based on the mean height, it may be helpful to consider the height error at different distances. Calculating the allowable error for each threshold segment candidate may thus be necessary. In the case of a mono-camera system, the height error may be based on calculating the depth error and propagating this to the height error. The depth error at distance x may where t is related to a metric scale (traveled distance), ro is related to the image distance between 2D point and focus of expansion (measured in pixels), ot is related to a metric scale error, crf oe is related to the focus-of-expansion error and cr F / uss is related to the optical flow error.

[0052] It can be assumed that the depth error along a rung is approximately the same. Thus, only one height error can be calculated (aj:

[0053] = | t?| ■ ff x , where v can be called the vertical normalized coordinate.

[0054] A threshold segment can contain multiple height measurements. The higher the number of measurements, the smaller the theoretical error. This can be modeled by ffs chwefts= where n is the number of height measurements that contributed to the calculation of the mean height. Semantic segmentation masking is described in more detail below. Using only 3D data, it may not be possible in some scenarios to separate road humps from other elevated structures adjacent to them (for example, sidewalks). Fig. 8 and Fig. 9 show an example where the hump 902 and the sidewalk 903 have approximately the same height (see marker 901 in Fig. 9a), making separation difficult using only 3D data. Using semantic segmentation, which labels road, sidewalks, vehicles, and curbs with a different class, it is possible to precisely delineate the hump, as can be seen in Fig. 9.The road mask 904 provided by semantic segmentation can be used in the context of threshold detection to filter elevation values ​​during accumulation, but also when constructing the 2D output bounding box. Figure 9a shows threshold segment detection without a road mask. Figure 9b shows a semantic segmentation output, and the sidewalk is part of the "Other" class and has nothing associated with it. In Figure 9c, the threshold segments 902 are provided after applying the road mask.

[0055] Fig. 10a shows individual threshold segments 1001 that are close to each other in both 2D and 3D. Fig. 10b shows a threshold object 1002 that resulted from merging individual threshold segments. Based on the previously detected threshold segments, clustering can be performed, merging adjacent threshold segments into a single threshold object. Merging two segments may require them to be close to each other in both 2D and 3D space.

[0056] During the processing of height distributions, each rung in the region of interest can be associated with an estimated depth and an estimated height above ground. Based on these filtered depth-height measurements, a longitudinal road profile can be calculated. Fig. 11 shows a detected bump 1101 at a distance of 6 meters, while Fig. 12 shows the associated road elevation profile. Points 1201 represent measurements classified as road. Points 1202 represent elevated points corresponding to the road bump 30. Line 1203 represents the theoretical elevation error model at different distances. Points above line 1203 have a higher probability of being classified as elevated.

[0057] Furthermore, according to embodiments of the invention, the method may include flow-based tracking. In the case of a noisy 3D depth map, false detections are possible. To mitigate the impact of sporadic false detections, flow-based confirmation may be used. Using backward flow, the mean flow along each threshold segment can be calculated. The mean flow vectors provide locations that can be examined in the previous frame. If a detection in the current frame also appears in the previous frame, and both are compared using optical flow, the detection in the current frame is confirmed. Otherwise, it is filtered.

[0058] The method according to embodiments of the invention can be executed by a computer program 20, as shown in Fig. 15, implemented, for example, in C++. The computer program 20 can run on a computer located in a vehicle 1 and / or in real time and / or on the camera 40, such as a mono camera 40, of the vehicle. Furthermore, the method can be integrated into an adaptive cruise control function.

[0059] Furthermore, Figs. 13 and 14 show the detection of a road hump 30 in the case of different scenarios.

[0060] Fig. 15 further shows the method 100 for detecting road humps 30 with a first method step 101 for obtaining the input data 111. The input data 111 can result at least partially from a sensor detection of an environment of the vehicle 1 and comprise at least one scene representation 201, such as an input image 201 and a corresponding depth map 121. The scene representation 201 can represent a scene of the environment, and the depth map 121 can indicate distances in the scene. According to a second method step 102, a road surface estimate 122 in the scene can be obtained based on the input data 111. According to a third method step 103, a road hump 30 can then be detected in the scene based on an evaluation of the depth map 121 and the road surface estimate 122.

[0061] The above explanation of the embodiments describes the present invention in the context of examples. Of course, individual features of the embodiments can be freely combined with each other, as long as this is technically reasonable, without deviating from the scope of the present invention.

Claims

Claims 1. A method (100) for detecting road humps (30), comprising the following steps: Obtaining (101) input data (111), wherein the input data (111) result at least partially from a sensor detection of an environment of a vehicle (1), the input data (111) comprise at least one scene representation (201) and a corresponding depth map (121), the scene representation (201) represents a scene of the environment and the depth map (121) indicates distances in the scene, Obtaining (102) a road surface estimate (122) in the scene based on the input data (111), Detecting (103) a road hump (30) in the scene based on an evaluation of the depth map (121) and the road surface estimate (122).

2. Method (100) according to claim 1, characterized in that the evaluation comprises the following: Determining a height-above-ground map, wherein the height-above-ground map indicates a height of surfaces in the scene, wherein the height-above-ground map is determined on the basis of the depth map (121) and the road surface estimate (122) and in particular on the basis of a semantic segmentation (123) of the scene, Determining multiple height distributions of height values ​​along the road, in particular for different rungs along the road, based on the determined height-above-ground map, in particular for the specified height of surfaces in the scene, Classifying the determined elevation distributions into at least one raised threshold segment and one flat road segment based on the elevation values.

3. Method (100) according to claim 2, characterized in that the depth map (121) indicates the distances for various elements corresponding to the scene, in particular pixels, wherein the height-above-ground map is determined on the basis of an element-by-element examination of each of the elements, wherein the element-by-element examination comprises an examination of conditions, in particular whether the respective element represents a static and / or flat structure, in particular at a valid depth, and / or whether the respective element is classified as a road area by the semantic segmentation.

4. The method (100) of claim 2 or 3, characterized in that the classifying is performed on the basis of a height error field to thereby consider a height error for the specified height of surfaces in the scene at different distances in the scene.

5. Method (100) according to one of claims 2 to 4, characterized in that the evaluation further comprises the following: Merging adjacent threshold segments into a single threshold object.

6. Method (100) according to one of the preceding claims, characterized in that a threshold object is obtained on the basis of the detection (103), wherein the threshold object represents the road threshold (30) by three-dimensional attributes, in particular distance and height.

7. The method (100) according to any one of the preceding claims, characterized in that the evaluation comprises an evaluation of dimensional features of the input data to thereby detect the road hump (30) based on the height of the road hump (30) raised with respect to the road in the scene.

8. Method (100) according to one of the preceding claims, characterized in that a control of the vehicle (1) is carried out on the basis of the detected road hump (30), wherein the control in particular special braking functions and / or suspension control and / or autonomous driving.

9. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to any one of the preceding claims 10. Data processing device (10) comprising means for carrying out the method (100) according to one of claims 1 to 8.

Images