Reliable obstacle detection

Abstract

Reliable obstacle detection The present invention relates to a method for detecting an obstacle, and a corresponding computer program and computer program product, wherein the method generally comprises the following process steps: providing a first image (I1) of a first camera (1) having a first field of view; providing a second image (I2) of a second camera (2) having a second field of view, wherein the first and second field of view areas at least partially overlap; determining a disparity map (D) based on the first (I1) and / or second image (I2); providing the disparity map (D) and at least one of the at least two images (I1, I2) as input for a trained neural network (4) configured to be able to make a statement about the presence or absence of an obstacle (H) in the field of view of at least one of the cameras (1, 2) based on the input; and A step of outputting a statement regarding the presence or absence of an obstacle (H) in the field of view of at least one of the cameras of both (1, 2).

Description

Technical Field

[0001] The present invention relates to a method for detecting an obstacle, particularly a method for detecting an obstacle executed by a computer.

Background Art

[0002] Modern vehicles are often equipped with advanced driver-assistance systems (ADAS) to assist the vehicle driver. In this context, various ADAS functions are known. First, this means that the control related to the running of the vehicle is used to assist the driver in the state where the driver is present. On the other hand, fully automated driving is also possible.

[0003] Among the ADAS functions, the function of detecting obstacles on the road is particularly emphasized. The obstacles in this context mean various objects existing on the road, and in particular, they can be construction cones, fallen loads, or the like. However, various obstacles in the front area of a vehicle, especially during driving, can also be a problem.

[0004] To detect an obstacle, various sensor systems integrated into the vehicle, such as a radar sensor, a lidar sensor, or a camera, can be used. One of the advantages of a camera is that it can achieve a relatively low cost and high spatial resolution. In particular, stereo camera systems or multi-camera systems are increasing in their use in relation to ADAS functions. However, as a disadvantage of such camera systems, the accuracy of distance measurement, especially at long distances, is relatively low. Distance information is extremely important to timely detect an obstacle and perform appropriate operations to avoid it.

[0005] In many cases, camera image analysis for obstacle detection is performed using conventional object recognition methods and learnable neural networks. However, typical object recognition methods vary in accuracy depending on the object being detected. In particular, with regard to obstacle detection, problems tend to arise when the obstacle is not a standard object such as another vehicle. On roads, as mentioned above, various objects such as fallen cargo with varying sizes and geometric dimensions can become obstacles. This is an unavoidable problem when using classification methods, especially when using machine learning methods. Here, the accuracy of image evaluation depends on how much diverse training data can be used.

[0006] As a method for recognizing various obstacles, P. Pinggera et al., in their paper "Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles," pp. 1099-1106 (doi: 10.1109 / IROS.2016.7759186), published at the 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), disclose a road obstacle detection method that integrates semantic segmentation techniques, convolutional neural networks (CNNs), and image data obtained from stereo cameras. In this way, it is possible to identify structures on the road with height based on stereo camera images.

[0007] On the other hand, CJ Holder et al., in their paper "Depth Not Needed - An Evaluation of RGB-D Feature Encodings for Off-Road Scene Understanding by Convolutional Neural Network," published at arxiv.org, 2018 (https: / / arxiv.org / ftp / arxiv / papers / 1801 / 1801.01235.pdf), disclose a combination of RGB image data and disparity images. However, in this method, the disparity image is determined based on calibrated camera image data. Therefore, complex camera calibration is always necessary. In addition, distortion correction must be performed on the image data from each camera in order to reliably recognize obstacles.

[0008] In other words, both of the methods described above have the disadvantage of requiring complex algorithms to recognize obstacles using camera images. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] P. Pinggera et al., Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles, pp. 1099-1106 (doi: 10.1109 / IROS.2016.7759186) [Non-Patent Document 2] CJHolder et al., arxiv.org, 2018 (https: / / arxiv.org / ftp / arxiv / papers / 1801 / 1801.01235.pdf), “Depth Not Needed - An Evaluation of RGB-D Feature Encodings for Off-Road Scene Understanding by Convolutional Neural Network” [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] What is desired is an easy and robust method for recognizing obstacles. Therefore, the problem that the present invention aims to solve is to provide an easy and accurate method for recognizing obstacles that can recognize any object on a road using images captured by one or more cameras. [Means for solving the problem]

[0011] This problem is solved by the method described in claim 1, the computer program described in claim 10, and the computer program product described in claim 11.

[0012] method Regarding The problem underlying this invention is particularly to, This is solved by a method for detecting obstacles implemented in a computer. The method is as follows: of Step 、 The first image of a first camera having a first field of view is provided in step Steps to provide a second image from a second camera having a second field of view region. and The first and second visual field regions overlap at least partially. Step, first image and / also This is the second image Based Steps to determine the disparity map, Based on the input, two cameras It is configured to be able to comment on the presence or absence of obstacles within the field of view of at least one of the cameras. doing , a step of providing a disparity map and at least one of at least two images as input for a trained neural network, Two machines of CameraOutputting a reference regarding the presence or absence of an obstacle within the field of view of at least one camera therein 、 Equipped with . The first and second cameras are at least two cameras Preparation of a camera system, particularly to, can be part of a stereo camera system. Incidentally, these may also be two independent cameras provided on the same vehicle. Preferably, First camera and / or also The second camera is a single vehicle to Attached . These cameras can be part of a driver assistance system.

[0013] The first and / or second images of the first and / or second cameras can be black-and-white images or color images. In the second case, various types of color spaces, for example, the red·green·blue (RGB) color space as an additive color mixing type color space, can be used.

[0014] The disparity map describes the offset of two corresponding pixels in the first and second images of the first and second cameras.

[0015] The neural network can be trained, for example, using training data including an image marked with an obstacle. The network trained in this way can identify pixels belonging to the obstacle object class in the first and / or second images of the first and / or second cameras. By using the method according to the present invention, any obstacle on the road can be advantageously identified.

[0016] two cameras The reference regarding the presence or absence of an obstacle within the field of view of at least one camera therein can be various descriptions regarding the obstacle. For example Two machines of the camera inside at least one CameraIt is possible to determine whether or not an obstacle exists within the field of view, and to determine the coordinates of the obstacle. Furthermore, it is conceivable to determine the type of obstacle, its size, and its distance from the camera. Other references to the obstacle are also conceivable and fall within the scope of this invention.

[0017] A key feature of the method according to the present invention is its minimal algorithmic complexity. Nevertheless, the parallax map allows for information regarding the distance to each obstacle and their size, enabling accurate obstacle recognition even if such a parallax map is not typically used to determine the distance to an object. In particular, using a parallax map significantly reduces the complexity of the algorithm.

[0018] Furthermore, the method according to the present invention is advantageous because it eliminates the need for calibration of the first and / or second cameras, particularly online calibration, or distortion correction of camera images.

[0019] In one embodiment of this method, the neural network is a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network.

[0020] another In terms of form, neural networks are first image and / or Second image to correspondence It is configured to output an obstacle map. , This obstacle map is first image and / or Information regarding the presence or absence of obstacles in the second image include This allows for appropriate and accurate references to obstacles. For example, an obstacle map can be used to determine the location of one or more obstacles, particularly their relative positions to the vehicle.

[0021] In this context, the obstacle map is , the one image and / or Second image Regarding predetermined sub-regions, in particular, pixel Each time, Whether or not the sub-region belongs to an obstacle show This is advantageous.

[0022] Furthermore, it is preferable that each sub-region of the obstacle map be assigned at least one of two pre-configurable assignment values. If the sub-region belongs to a single obstacle, a first assignment value is assigned; if the sub-region does not belong to a single obstacle, i.e., if the object is in the background, outside the road, or similar, a second assignment value is assigned. However, it is also possible to define more than one assignment value.

[0023] For example, the first attribute could be assigned to the road, the second to outside the road, and the third to an obstacle within the road area. In other words, using images from the first and second cameras, further classifications beyond simply the presence or absence of obstacles within the field of view of at least one camera are also possible.

[0024] One embodiment of the method according to the present invention is , the For two images The parallax map is determined by considering the first image. , and / also teeth , the Parallax map for a single image This is calculated taking the second image into consideration. of include .

[0025] In a further form, the disparity map is calculated using nonlinear correlations, particularly cross-correlations, preferably mean-removed normalized cross-correlations, and specifically by either a two-dimensional block-matching algorithm or a semi-global matching algorithm.

[0026] In a particularly preferred form, the disparity map is derived by a trained neural network, i.e., a network configured to derive a disparity map using at least the first and second images. The neural network is preferably a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network. The derivation of the disparity map by a neural network is very robust and omission-free, making it particularly suitable for deriving references to obstacles.

[0027] In particular, regarding the complexity of the method, it is advantageous that the same type of neural network is used, especially preferably, for determining the disparity map and identifying references to obstacles. In this case, it is advantageous that similar architectures are selected for both networks.

[0028] In another particularly preferred embodiment of the present invention, a two-dimensional disparity map is determined. In this case, each pixel is assigned two disparity values, for example, one for horizontal shift and one for vertical shift.

[0029] Furthermore, the fundamental problem underlying the present invention is solved by a computer program that, when executed by a computer, has an instruction to cause the computer to perform one of the methods according to the present invention as described above.

[0030] Furthermore, the fundamental problem underlying the present invention is solved by a computer program product in which the computer program according to the present invention is stored.

[0031] The present invention and its advantageous features will be further explained with reference to Figure 1 below. [Brief explanation of the drawing]

[0032] [Figure 1]Figure 1 illustrates the method according to the present invention. Two cameras 1 and 2 are shown, whose fields of view partially overlap. To derive references to the presence or absence of obstacles H, a first image I1 from the first camera 1 and a second image I2 from the second camera of unit 3 are provided for determining a disparity map D. [Modes for carrying out the invention]

[0033] At least one of the two images I1 and I2—in this case, the second image I2 from the second camera 2—is provided as input to the neural network 4, along with a disparity map. Based on the input, the neural network 4 is configured to indicate and output whether or not there is an obstacle H within the field of view of at least one of the two cameras 1 and 2—in this case, the second camera 2. It is also possible to provide the neural network 4 with the first image I1 as input (dotted line) instead of the second image I2, or with both images I1 and I2 as input.

[0034] Distortion correction of images I1 and I2 before deriving the disparity map D is unnecessary. The core idea of ​​this invention is that obstacle recognition using such a disparity map D with the help of a neural network 4 is feasible. On the other hand, according to conventional techniques, a distortion correction step must always be performed first in order to enable obstacle recognition.

[0035] Various methods can be considered for determining a disparity map, and these fall within the scope of the present invention. Determining a two-dimensional disparity map is particularly advantageous. In this context, any suitable mathematical correlation function, especially cross-correlation, can be used. On the other hand, a neural network, preferably a convolutional neural network (CNN), can also be considered for determining the disparity map D. Using a convolutional neural network allows for particularly good realization of two-dimensional correlations. Furthermore, it is particularly advantageous in implementing the present invention if the neural network 4 is a convolutional neural network (CNN). In this case, similar architectures can be selected for both networks.

[0036] In summary, the method, corresponding computer program, and computer program product according to the present invention enable highly robust obstacle recognition, particularly in combination with driver assistance systems for vehicles. Because it does not require the preparation of camera images, such as distortion correction, the method is simpler than conventional techniques. In fact, it is even possible to directly use the raw data from cameras 1 and 2. On the other hand, the two-dimensional correlation function, which is advantageous but inherently complex, can be processed very efficiently and accurately with the help of, for example, a neural network, particularly a convolutional neural network (CNN). While this application relates to the invention described in the claims, it also includes the following other aspects. 1. disability Methods for detecting harmful substances, in particular, computer-based methods. And, The above method involves the following steps: A step of providing a first image (I1) of a first camera (1) having a first field of view region, The step of providing a second image (I2) of a second camera (2) having a second field of view region. and The first and second visual field regions overlap at least partially. Step, first image (I1) and / or the second image (I2) Based Steps to determine the parallax map (D), Based on the input, two cameras (1,2) is configured to be able to comment on the presence or absence of an obstacle (H) within the field of view of at least one of the cameras. doing , a disparity map (D) and at least two images (I1, I2) as input for the trained neural network (4) inside The step of providing at least one sheet, and, two cameras (1,2) A step to output a reference regarding the presence or absence of an obstacle (H) within the field of view of at least one of the cameras. 、 A method characterized by comprising . 2. A neural network (4) is a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network. The method according to item 1, characterized by the features described above. 3. The neural network (4) is the first image (I1) and / or Second image (I2) to correspondence It is configured to output an obstacle map, and this obstacle map is, first image (I1) and / or Information regarding the presence or absence of obstacles (H) in the second image (I2) include, The method according to 1 or 2 above, characterized by the features described above. 4. Obstacle map, First image and / or second image Predetermined sub-regions About , especially, pixels Each time, Whether or not the subregion belongs to the obstacle (H) show Characterized by 、 The method described in item 3 above. 5. Each subregion of the obstacle map is assigned at least one of two pre-configurable assignment values. If the subregion belongs to an obstacle (H), a first assignment value is assigned; if the subregion does not belong to an obstacle (H), a second assignment value is assigned. The method according to 3 or 4 above, characterized by the features described above. 6. It includes a disparity map (D) for the second image (I2) that takes the first image (I1) into account, and / or for the first image (I1) that takes the second image (I2) into account. A method according to any one of the above 1 to 5, characterized by the following: 7. The disparity map (D) uses nonlinear correlation, particularly cross-correlation, especially mean-removed normalized cross-correlation, and in particular a two-dimensional block matching algorithm or a semi-global matching algorithm. Using Calculated ru A method according to any one of the above 1 to 6, characterized by the following: 8. The disparity map (D) is determined by a trained neural network, i.e., a network configured to determine the disparity map (D) using at least the first (I1) and second (I2) images. A method according to any one of the above 1 to 7, characterized by the following: 9. A two-dimensional parallax map (D) is determined. A method according to any one of the above 1 to 8, characterized by the following: 10. A computer program that, when executed by a computer, has an instruction that causes the computer to perform any one of the methods described in 1 to 9 above. 11. A computer program product in which the computer program described in item 10 above is stored.

Claims

1. A method for detecting an obstacle, more particularly a method implemented in a computer, The above method involves the following steps: A step of providing a first image (I1) of a first camera (1) having a first field of view region, A step of providing a second image (I2) of a second camera (2) having a second field of view region, wherein the first and second field of view regions overlap at least partially, A step of determining a disparity map (D) based on the first image (I1) and the second image (I2), A step of providing a disparity map (D) and at least one of at least two images (I1, I2) as input for a trained neural network (4), wherein the trained neural network (4) is configured to be able to comment on the presence or absence of an obstacle (H) in the field of view of at least one of two cameras (1, 2) based on the input, and A step of outputting a reference regarding the presence or absence of an obstacle (H) within the field of view of at least one of the two cameras (1, 2), Equipped with, The neural network (4) is configured to output an obstacle map corresponding to the first image (I1) and / or the second image (I2), and this obstacle map includes information regarding the presence or absence of obstacles (H) in the first image (I1) and / or the second image (I2). The obstacle map indicates, pixel by pixel, whether a sub-region of the first and / or second image belongs to an obstacle (H). A method characterized by the following.

2. The neural network (4) is a convolutional neural network, a recurrent neural network, a hypernetwork, or a transformer network. The method according to feature 1.

3. Each sub-region of the obstacle map is assigned at least one of two pre-configurable assignment values. If the sub-region belongs to one obstacle (H), a first assignment value is assigned; if the sub-region does not belong to one obstacle (H), a second assignment value is assigned. The method according to feature 1.

4. The disparity map (D) for the second image (I2) is determined considering the first image (I1), and / or the disparity map (D) for the first image (I1) is determined considering the second image (I2). The method according to feature 1.

5. The disparity map (D) is calculated using nonlinear correlation, cross-correlation, or mean-removed normalized cross-correlation, or using a two-dimensional block matching algorithm or a semi-global matching algorithm. The method according to feature 1.

6. The disparity map (D) is determined by a trained neural network, i.e., a network configured to determine the disparity map (D) using at least the first (I1) and second (I2) images. The method according to feature 1.

7. A two-dimensional parallax map (D) is determined. The method according to feature 1.

8. A computer program having an instruction that causes a computer to carry out the method described in any one of Claims 1 to 7 when the computer program is executed by the computer.

9. A computer program product in which the computer program described in claim 8 is stored.

Images