Reliable obstacle detection

Abstract

Reliable obstacle detection The present invention relates to a method for detecting an obstacle, and a corresponding computer program and computer program product, wherein the method generally comprises the following process steps: providing a first image (I1) of a first camera (1) having a first field of view; providing a second image (I2) of a second camera (2) having a second field of view, wherein the first and second field of view areas at least partially overlap; determining a disparity map (D) based on the first (I1) and / or second image (I2); providing the disparity map (D) and at least one of the at least two images (I1, I2) as input for a trained neural network (4) configured to be able to make a statement about the presence or absence of an obstacle (H) in the field of view of at least one of the cameras (1, 2) based on the input; and A step of outputting a statement regarding the presence or absence of an obstacle (H) in the field of view of at least one of the cameras of both (1, 2).

Description

[Technical Field]

[0001] The present invention relates to a method for detecting obstacles, and in particular to a computer-implemented method for obstacle detection. [Background technology]

[0002] Modern vehicles are often equipped with advanced driver-assistance systems (ADAS) to support the driver of the vehicle. In this context, various ADAS functions are known. Primarily, they are used to assist the driver while still retaining control over the vehicle's driving. However, fully automated driving is also possible.

[0003] Among ADAS functions, the detection of obstacles on the road is of particular importance. Obstacles in this context refer to various objects on the road, in particular construction cones, fallen cargo, or similar. However, various obstacles, especially in the area ahead of the vehicle while it is moving, can also be problematic.

[0004] To detect obstacles, various sensor systems integrated into a vehicle can be used, such as radar sensors, lidar sensors, or cameras. One advantage of cameras is that they can achieve high spatial resolution at a relatively low cost. In particular, stereo camera systems or multi-camera systems are increasingly being used in connection with ADAS functions. However, a drawback of such camera systems is the relatively low accuracy of distance measurements, especially at long distances. Distance information is crucial for detecting obstacles in a timely manner and taking appropriate action to avoid them.

[0005] In many cases, camera image analysis for obstacle detection is performed using conventional object recognition techniques and trainable neural networks. However, typical object recognition techniques vary in accuracy depending on the object being detected. Problems with obstacle detection, especially when the obstacle is not a standard object such as another vehicle, can occur. As mentioned above, various objects, such as dropped cargo, with different sizes and geometric dimensions can become obstacles on the road. This is an unavoidable problem when using classification methods, especially machine learning methods. The accuracy of image evaluation depends on the variety of training data available.

[0006] As a method for recognizing various obstacles, P. Pinggera et al., "Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles," pp. 1099-1106 (doi: 10.1109 / IROS.2016.7759186), published at the 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), discloses a road obstacle detection method that integrates semantic segmentation technology, a convolutional neural network (CNN), and image data obtained from a stereo camera. In this way, it is possible to identify structures with height on the road based on stereo camera images.

[0007] Meanwhile, CJ Holder et al. have disclosed a combination of RGB image data and disparity maps in their paper "Depth Not Needed - An Evaluation of RGB-D Feature Encodings for Off-Road Scene Understanding by Convolutional Neural Network," published on arxiv.org, 2018 (https: / / arxiv.org / ftp / arxiv / papers / 1801 / 1801.01235.pdf). However, in this method, the disparity maps are calculated based on calibrated camera image data. Therefore, cumbersome camera calibration is always required. In addition, to reliably recognize obstacles, distortion correction of the image data from each camera must also be performed.

[0008] That is, both of the above-mentioned methods have the disadvantage that they require complex algorithms to recognize obstacles using camera images. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] P. Pinggera et al., Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles, pp. 1099-1106 (doi: 10.1109 / IROS.2016.7759186) [Non-patent document 2] CJHolder et al., arxiv.org, 2018 (https: / / arxiv.org / ftp / arxiv / papers / 1801 / 1801.01235.pdf), “Depth Not Needed - An Evaluation of RGB-D Feature Encodings for Off-Road Scene Understanding by Convolutional Neural Network” Summary of the Invention [Problem to be solved by the invention]

[0010] What is desired is a method for obstacle recognition that is easy and robust. Therefore, the problem that the present invention aims to solve is to provide a method for obstacle recognition that is easy and accurate and that can recognize any object on the road using images taken by one or more cameras. [Means for solving the problem]

[0011] This problem is solved by a method according to claim 1, a computer program according to claim 10 and a computer program product according to claim 11.

[0012] From a method aspect, the problem underlying the present invention is particularly preferably solved by a computer-implemented method for detecting obstacles, which method comprises the following method steps: providing a first image from a first camera having a first field of view; providing a second image of a second camera having a second field of view, wherein the first and second field of view areas at least partially overlap; determining a disparity map based on the first and / or second images; providing the disparity map and at least one of the at least two images as inputs for a trained neural network configured to make a statement regarding the presence or absence of an obstacle in the field of view of at least one of the cameras based on the inputs; and Outputting a statement regarding the presence or absence of an obstacle in the field of view of at least one of the cameras. The first and second cameras may be part of a camera system including at least two cameras, preferably a stereo camera system, or may be two independent cameras mounted on the same vehicle. Preferably, the first and / or second cameras are fixed to one camera. These cameras may be part of a driver assistance system.

[0013] The first and / or second images of the first and / or second cameras can be black and white or color images. In the second case, various kinds of color spaces can be used, for example, the red-green-blue (RGB) color space as an additive color space.

[0014] The disparity map describes the shift of two corresponding pixels in the first and second images of the first and second cameras.

[0015] The neural network can be trained using training data including, for example, images in which obstacles are marked. The network thus trained can identify pixels belonging to the obstacle object class in the first and / or second images from the first and / or second cameras. Advantageously, the method according to the present invention can be used to identify any obstacle on the road.

[0016] The reference to the presence or absence of an obstacle within the field of view of at least one of the cameras may be various descriptions of the obstacle. For example, it is possible to determine whether an obstacle exists within the field of view of at least one of the cameras, the coordinates of the obstacle, etc. It is also possible to determine the type of obstacle, the size of the obstacle, or the distance from the camera. Other references to the obstacle are also possible, and are within the scope of the present invention.

[0017] A feature of the method according to the invention is the low complexity of the algorithm. Nevertheless, the disparity map provides a reference to the distance to each obstacle and their size, allowing accurate obstacle recognition even though such disparity maps are not typically used to determine the distance to an object. In particular, the use of the disparity map significantly reduces the complexity of the algorithm.

[0018] Furthermore, the method according to the invention advantageously does not require calibration of the first and / or second camera, in particular online calibration, or rectification of the camera images.

[0019] In one form of the method, the neural network is a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network.

[0020] In yet another embodiment, the neural network is configured to output an obstacle map corresponding to the first and second images and containing information regarding the presence or absence of obstacles in the first and second images, such that references to obstacles can be appropriately refined. For example, the obstacle map can be used to determine the location of an obstacle or obstacles, particularly relative to the vehicle.

[0021] In this context, it is advantageous if the obstacle map provides for predetermined sub-regions, in particular pixels of the first and second images, whether the sub-region belongs to an obstacle or not.

[0022] Furthermore, each subregion of the obstacle map is preferably assigned one of at least two predefined attribution values, a first attribution value if the subregion is attributed to an obstacle, and a second attribution value if the subregion is not attributed to an obstacle, i.e., the object is in the background, off the road, or similar cases. However, it is also possible to define more than two attribution values.

[0023] For example, a first attribution value can be attributed to the road, a second attribution value to off-road, and a third attribution value to an obstacle within the road area, i.e., further classification beyond that based solely on the presence or absence of an obstacle within the field of view of at least one camera is possible using the images from the first and second cameras.

[0024] One embodiment of the method according to the invention includes a disparity map for the second image taking into account the first image and / or for the first image taking into account the second image.

[0025] A further aspect includes that the disparity map is calculated using non-linear correlation, in particular cross-correlation, preferably mean-removed normalized cross-correlation, in particular by either a two-dimensional block matching algorithm or a semi-global matching algorithm.

[0026] In a particularly preferred embodiment, the disparity map is determined by a trained neural network, i.e., a network configured to determine the disparity map using at least the first and second images, preferably a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network. Neural network derivation of the disparity map is very robust and leak-free, making it particularly suitable for determining mentions of obstacles.

[0027] In particular, with regard to the complexity of the method, it is advantageous if neural networks, particularly preferably of the same type, are used for determining the disparity map and for identifying the mentions of obstacles, in which case it is advantageously possible to select similar architectures for both networks.

[0028] In another particularly preferred embodiment of the present invention, a two-dimensional disparity map is determined, in which case each pixel is assigned two disparity values, for example one for lateral and one for vertical shift.

[0029] Additionally, the problem underlying the present invention is solved by a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the present invention in any one of the above-mentioned forms.

[0030] Furthermore, the problem underlying the present invention is solved by a computer program product on which the computer program according to the present invention is stored.

[0031] The present invention and its advantageous features are further explained below with reference to FIG. [Brief explanation of the drawings]

[0032] [Figure 1] 1 illustrates the method according to the invention. Two cameras 1, 2 are shown whose fields of view partially overlap. A first image I1 from a first camera 1 and a second image I2 from a second camera are provided to a unit 3 for determining a disparity map D in order to derive a statement regarding the presence or absence of an obstacle H. DETAILED DESCRIPTION OF THE INVENTION

[0033] At least one of the two images I1, I2, here the second image I2 of the second camera 2, together with the disparity map, is provided as input to a neural network 4. This neural network 4 is configured to be able to output, based on the input, a statement about the presence or absence of an obstacle H in the field of view of at least one of the two cameras 1, 2, here the second camera 2. Note that instead of the second image I2, it is also possible to provide the neural network 4 with the first image I1 as input (dotted line) or both images I1 and I2 as inputs.

[0034] There is no need to rectify the images I1 and I2 before deriving the disparity map D. The core idea of ​​the present invention is that obstacle recognition using such a disparity map D with the help of a neural network 4 is feasible, whereas according to the prior art, a rectification step must always be carried out before obstacle recognition can be performed.

[0035] There are various conceivable ways of determining the disparity map, which fall within the scope of the present invention. Determining a two-dimensional disparity map is particularly advantageous. In this context, any suitable mathematical correlation function, in particular cross-correlation, can be used. On the other hand, it is also conceivable to employ a neural network, preferably a convolutional neural network (CNN), for determining the disparity map D. Two-dimensional correlations can be particularly well realized using a convolutional neural network. Furthermore, it is particularly advantageous in the implementation of the present invention if the neural network 4 is a convolutional neural network (CNN). In this case, a similar architecture can be selected for both networks.

[0036] In summary, the method, corresponding computer program, and computer program product according to the present invention enable highly robust obstacle recognition, particularly in combination with a driver assistance system for a vehicle. The method is simplified compared to prior art techniques, since no camera image preparation, e.g., rectification, is required. Indeed, it is even possible to directly use the raw data from cameras 1 and 2. Meanwhile, advantageously, otherwise complex two-dimensional correlation functions can be processed very efficiently and accurately, for example, with the aid of neural networks, in particular convolutional neural networks (CNNs).

Claims

1. A method for detecting obstacles, in particular a computer-implemented method, characterized in that it comprises the following process steps: providing a first image (I1) of a first camera (1) having a first field of view; providing a second image (I2) of a second camera (2) having a second field of view, wherein the first and second field of view areas at least partially overlap; determining a disparity map (D) based on the first (I1) and / or second image (I2); providing the disparity map (D) and at least one of the at least two images (I1, I2) as input for a trained neural network (4) adapted to be able to make a statement about the presence or absence of an obstacle (H) in the field of view of at least one of the cameras (1, 2) based on the input; and outputting a statement regarding the presence or absence of an obstacle (H) in the field of view of at least one of the cameras of both (1, 2);

2. The neural network (4) is a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network.

2. The method of claim 1 .

3. The neural network (4) is configured to output an obstacle map corresponding to the first (I1) and second (I2) images and containing information about the presence or absence of obstacles (H) in the first (I1) and second (I2) images.

3. The method according to claim 1 or 2.

4. The obstacle map indicates whether a predetermined sub-region, in particular a pixel of the first and second images, belongs to an obstacle (H) or not.

4. The method of claim 3.

5. Each sub-region of the obstacle map is assigned one of at least two predefined attribution values. If the sub-region is assigned to one obstacle (H), the first attribution value is assigned, and if the sub-region is not assigned to one obstacle (H), the second attribution value is assigned.

5. The method according to claim 3 or 4.

6. It includes a disparity map (D) for the second image (I2) taking into account the first image (I1) and / or for the first image (I1) taking into account the second image (I2).

10. A method according to any one of the preceding claims.

7. The disparity map (D) is calculated using a non-linear correlation, in particular a cross-correlation, particularly preferably a mean-removed normalized cross-correlation, in particular by a two-dimensional block matching algorithm or a semi-global matching algorithm.

10. A method according to any one of the preceding claims.

8. The disparity map (D) is determined by a trained neural network, i.e., a network configured to be able to determine the disparity map (D) using at least the first (I1) and second (I2) images.

10. A method according to any one of the preceding claims.

9. A two-dimensional disparity map (D) is calculated 10. A method according to any one of the preceding claims.

10. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of the preceding claims.

11. A computer program product having stored thereon the computer program of claim 10.

Images