Obstacle detection device

Abstract

The present invention addresses the problem of acquiring an obstacle detection device that can accurately detect a distant obstacle. This obstacle detection device comprises: an image acquisition unit 201 that acquires a captured image from an imaging device 111; a point cloud data acquisition unit 211 that acquires point cloud data from a distance measurement device 112; an image processing unit 202 that recognizes a road surface and a prescribed object on the basis of the image; a point cloud data processing unit 212 that removes, from the point cloud data, road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the prescribed object; and an obstacle detection unit 221 that detects an obstacle on the road surface on the basis of the processed point cloud data. The point cloud data processing unit 212 estimates a road surface model on the basis of the point cloud data and a recognition result of the road surface from the image processing unit, identifies the road surface point cloud data from the point cloud data on the basis of the road surface model, and identifies the object point cloud data by clustering road surface removal point cloud data obtained by removing the road surface point cloud data from the point cloud data.

Description

Obstacle detection device

[0001] The present invention relates to an obstacle detection device.

[0002] ADAS (Advanced Driver Assistance System) / AD (Autonomous Driving) technology has been developed to provide driving assistance / vehicle control by utilizing sensing information around a vehicle. In ADAS / AD technology, there is a need for long-distance obstacle detection to avoid danger early. Patent Document 1 discloses the configuration of an object recognition system that includes an ECU, a distance measurement sensor, and an imaging device, and that, when an emission direction corresponding to an emission wave whose reflected wave cannot be detected by the distance measurement sensor exists between two distance measurement points, sets a threshold based on an image generated by the imaging device to make a determination.

[0003] Japanese Patent Application Laid-Open No. 2022-006716

[0004] While an image capture device alone can detect travel paths and vehicles at long distances, it has difficulty detecting a wide variety of obstacles. A ranging sensor alone has a small amount of point cloud data for objects at long distances, making it difficult to determine whether or not an object is an obstacle.

[0005] In the case of a technology that uses the detection results of an imaging device and a distance measurement sensor, as in Patent Document 1, a positional discrepancy occurs between the detection results of both due to calibration errors or asynchrony, making simple correspondence difficult. For example, there is a problem in that relatively small obstacles, such as objects that have fallen on the road, are buried in part of the road surface and are difficult to detect.

[0006] The present invention has been made in view of the above points, and an object of the present invention is to provide an obstacle detection device that can accurately detect distant obstacles.

[0007] The obstacle detection device of the present invention that solves the above-mentioned problems comprises: an image acquisition unit that acquires images captured by an imaging device; a point cloud data acquisition unit that acquires point cloud data measured by a distance measuring device; an image processing unit that recognizes a road surface and a predetermined object based on the image; a point cloud data processing unit that generates processed point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the predetermined object from the point cloud data; and an obstacle detection unit that detects obstacles present on the road surface based on the processed point cloud data, wherein the point cloud data processing unit estimates a road surface model that indicates the shape of the road surface based on the point cloud data and the road surface recognition result by the image processing unit; identifies the road surface point cloud data from the point cloud data based on the road surface model; and identifies the object point cloud data by clustering road surface-removed point cloud data in which the road surface point cloud data has been removed from the point cloud data.

[0008] According to the present invention, an obstacle detection device capable of detecting distant obstacles with high accuracy can be obtained. Further features related to the present invention will become apparent from the description of this specification and the accompanying drawings. Furthermore, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] 1 is a functional block diagram of a vehicle control device equipped with the obstacle detection device of this embodiment. A functional block diagram of the obstacle detection device of this embodiment. A flowchart showing an obstacle detection method by the obstacle detection device of this embodiment. A flowchart showing the processing content of S313 in FIG. 3 in detail. A schematic diagram showing a specific scene of obstacle detection. A diagram showing an example of a captured image in front of the vehicle and point cloud data. A diagram explaining a method of extracting obstacle candidates in the obstacle detection device of this embodiment. A functional block diagram of another example of the obstacle detection device of this embodiment. A functional block diagram of another example of the obstacle detection device of this embodiment. A functional block diagram of another example of the obstacle detection device of this embodiment. A functional block diagram of another example of the obstacle detection device of this embodiment.

[0010] Next, an embodiment of the obstacle detection device of the present invention will be described. Fig. 1 is a functional block diagram of a vehicle control device equipped with the obstacle detection device of this embodiment.

[0011] The vehicle control device 101 is mounted on a vehicle having an ADAS or AD, and has an imaging device 111 and a distance measuring device 112 connected to its input side, and a steering device 121 and a braking device 122 connected to its output side.

[0012] The vehicle control device 101 is composed of an ECU (Electronic Control Unit) having a CPU, memory, and input / output units, and embodies an obstacle detection device 102, a steering control device 103, and a brake control device 104 by executing software programs stored in the memory with the CPU.

[0013] The obstacle detection device 102 detects an obstacle present in front of the vehicle based on information acquired from the imaging device 111 and the distance measuring device 112, and transmits a detection signal to the steering control device 103 and the brake control device 104. The steering control device 103 and the brake control device 104 control the steering device 121 and the brake device 122 based on the detection signal from the obstacle detection device 102.

[0014] The imaging device 111 is, for example, a monocular camera that is installed on the vehicle to capture an image of the area ahead of the vehicle and generate an image of that image capture range. The ranging device 112 is, for example, a LiDAR (Laser Imaging Detection and Ranging) that is installed on the vehicle so that the imaging range of the imaging device 111 and the ranging range overlap, emitting outgoing waves in front of the vehicle and receiving reflected waves from objects to determine ranging points. The imaging device 111 and ranging device 112 each capture images and measure at a predetermined frame rate.

[0015] 2 is a functional block diagram of the obstacle detection device 102 according to this embodiment. The obstacle detection device 102 includes an image acquisition unit 201, an image processing unit 202, a road surface area model 203, an object model 204, a segmentation result 205, camera parameters 206, a point cloud data acquisition unit 211, a point cloud data processing unit 212, and an obstacle detection unit 221.

[0016] The image acquisition unit 201 acquires an image captured by the imaging device (camera) 111. The image processing unit 202 classifies the captured image into pixel regions of roads and white lines by semantic segmentation, and recognizes predetermined objects such as vehicles, people, bicycles, and motorcycles as rectangular regions by object detection. Semantic segmentation / object detection can be performed based on a model learned by machine learning or deep learning, for example.

[0017] The road surface region model 203 corresponds to semantic segmentation, and the object model 204 corresponds to object detection. In the image processing unit 202 of this embodiment, pixel regions of roads and white lines are classified by semantic segmentation, and object detection is performed by recognizing predetermined objects as rectangular regions through image processing, but the present invention is not limited to these methods. For example, predetermined objects such as vehicles may be recognized as pixel regions by semantic segmentation. Hereinafter, segmentation processing includes semantic segmentation / object detection, and segmentation results include semantic segmentation results / object detection results.

[0018] The image processing unit 202 executes segmentation processing by referring to the road surface area model 203 and the object model 204. Information on the segmentation results by the image processing unit 202 is stored in the segmentation result 205. The information on the segmentation results is stored for each frame. The camera parameters 206 store external parameters (six degrees of freedom corresponding to position and rotation) that indicate the relative positional relationship between the image capturing device 111 and the distance measuring device 112, and internal parameters such as the focal length, image resolution, and distortion coefficient of the image capturing device 111.

[0019] The point cloud data acquisition unit 211 acquires point cloud data measured by the distance measurement device 112. The point cloud data consists of three-dimensional data of multiple distance measurement points. The point cloud data processing unit 212 removes road surface point cloud data and object point cloud data corresponding to predetermined objects from the acquired point cloud data to generate post-removal processing point cloud data. The obstacle detection unit 221 detects obstacles from the post-removal processing point cloud data.

[0020] The obstacle detection unit 221 detects obstacle candidates based on the post-removal processing point cloud data. The obstacle detection unit 221 has a road surface obstacle determination unit that determines whether or not an obstacle candidate exists on the road surface. The road surface obstacle determination unit performs processing to remove point clouds of objects such as buildings, sidewalks, and curbs that are located in places other than the road on which the vehicle is traveling from the post-removal processing point cloud data, and ultimately determines the remaining point cloud data as road surface obstacles.

[0021] FIG. 3 is a flowchart showing an obstacle detection method performed by the obstacle detection device of this embodiment.

[0022] The image acquisition unit 201 acquires a captured image from the imaging device 111 (S301). The image processing unit 202 executes segmentation processing by referring to the road surface area model 203 and the object model 204, and stores the segmentation result 205 (S302).

[0023] Meanwhile, the point cloud data acquisition unit 211 acquires point cloud data from the distance measuring device 112 (S311). Hereinafter, the point cloud data acquired by the point cloud data acquisition unit 211 will be referred to as acquired point cloud data.

[0024] The point cloud data processing unit 212 determines whether there is a segmentation result with a timestamp close to that of the frame of the acquired point cloud data (S312). If there is a segmentation result with a timestamp close to that of the frame of the acquired point cloud data (Yes in S312), the point cloud data processing unit 212 removes the road surface point cloud data and object point cloud data corresponding to a predetermined object from the acquired point cloud data to generate post-removal processing point cloud data (S313). The obstacle detection unit 221 performs processing to detect obstacles from the post-removal processing point cloud data (S314).

[0025] Fig. 4 is a flowchart showing in detail the processing content of S313 in Fig. 3. The point cloud data processing unit 212 extracts road surface candidate point cloud data by associating the segmentation result 205 acquired by the image processing unit through the segmentation processing with the acquired point cloud data acquired by the point cloud data acquisition unit 211 (S401: road surface candidate point cloud data extraction unit). Here, roads and white lines are extracted as road surface candidates from the point cloud data associated with the segmentation results, and the road surface candidate point cloud data is generated.

[0026] Next, a road surface model indicating the shape of the road surface is estimated from the point cloud shape of the road surface candidate point cloud data (S402: road surface model estimating unit). As a method for estimating the road surface model from the point cloud shape, a known technique such as plane estimation using RANSAC is used.

[0027] Here, point cloud data within a predetermined height range from the road surface model may be considered to be included in the road surface. For example, since the vehicle 501 can overcome an obstacle with a height of 10 cm or less, a range of 10 cm or less from the plane may be considered to be included in the road surface as an allowable error range.

[0028] The allowable error range of the road surface model may be set according to distance information included in the acquired point cloud data. For example, the allowable error of the road surface model is set to decrease as the distance from the distance measuring device 112 to the point cloud increases. This makes it possible to prevent distant obstacles from being erroneously determined to be the road surface.

[0029] Then, road surface point cloud data is re-extracted from the acquired point cloud data based on the estimated road surface model to generate re-extracted road surface point cloud data (S403: road surface point cloud data re-extraction unit), and the re-extracted road surface point cloud data is removed from the acquired point cloud data to generate post-road surface removal point cloud data (S404: re-extracted road surface point cloud data removal unit).

[0030] The point cloud data after road surface removal is clustered to identify object point cloud data that is a cluster corresponding to a predetermined object (S405: object point cloud data identification unit). As a result, object point cloud data such as a preceding vehicle, pedestrian, bicycle, or motorcycle is identified.

[0031] The object point cloud data is removed from the road surface removed point cloud data to generate post-removal processing point cloud data (S406: object point cloud data removing unit). As a result, post-removal processing point cloud data is generated in which the road surface point cloud data and the object point cloud data have been removed from the acquired point cloud data.

[0032] 5 is a schematic diagram showing a specific scene of obstacle detection, and FIG. 6 is a diagram showing an example of a captured image of the area in front of the vehicle and point cloud data. In this embodiment, a vehicle 501 is equipped with a vehicle control device 101 including an obstacle detection device 102.

[0033] As shown in FIG. 5, a vehicle 501 is traveling on a three-lane road 504 divided by white lines 505 and 506, and ahead of the vehicle 501 are a leading vehicle 503 and multiple obstacles 502 on the road.

[0034] As shown in Fig. 6A, the captured image 601 captured by the imaging device 111 includes the road surface of the road 504, white lines 505 and 506, the obstacle 502, and the preceding vehicle 503. As shown in Fig. 6B, the point cloud data 611 measured by the distance measuring device 112 includes the point cloud of the road surface of the road 504, the white lines 505 and 506, the obstacle 502, and the preceding vehicle 503.

[0035] As shown in Figure 6(a), the image capture device 111 alone can detect a road or a preceding vehicle 503 at a long distance, but it has difficulty detecting a wide variety of obstacles. Also, as shown in Figure 6(b), the distance measurement device 112 alone has a small amount of point cloud data for objects at a long distance, making it difficult to determine whether or not an object is an obstacle. Furthermore, simply associating the captured image with the point cloud data can result in positional deviations due to calibration errors, asynchrony, and the like, and small obstacles can be buried in part of the road surface, making them difficult to detect.

[0036] 7 is a diagram illustrating a method for extracting obstacle candidates in the obstacle detection device of this embodiment. Point cloud data 701 shown in FIG. 7 represents point cloud data in which segmentation results are associated with acquired point cloud data. In the point cloud data 701, an obstacle 711 is buried in the road surface and is considered to be part of the road. Furthermore, part of the road is also recognized as a vehicle (preceding vehicle) 712.

[0037] In the obstacle detection device 102 of this embodiment, the point cloud data processing unit 212 extracts roads and white lines as road surface candidates from the point cloud data associated with the segmentation results, and extracts road surface candidate point cloud data 702.

[0038] Then, a road surface model indicating the shape of the road surface is estimated from the point cloud shape of the road surface candidate point cloud data, and road surface point cloud data is re-extracted from the point cloud data 701 based on the estimated road surface model to create re-extracted road surface point cloud data 703.

[0039] Next, the re-extracted road surface point cloud data 703 is removed from the point cloud data 701 to generate post-road surface removal point cloud data 704. In the post-road surface removal point cloud data 704, point clouds of an obstacle 711 and a leading vehicle 712 remain as obstacle candidates.

[0040] Here, the post-road surface removal point cloud data 704 is clustered to identify object point cloud data that is a cluster corresponding to a predetermined object (a preceding vehicle 712), and the point cloud data of the preceding vehicle 712 that is the object point cloud data is removed from the post-road surface removal point cloud data 704 to generate post-removal processing point cloud data. Only the point cloud data of the obstacle remains in the post-removal processing point cloud data.

[0041] According to the obstacle detection device 102 of this embodiment, even if a positional deviation occurs between the imaging device 111 and the ranging device 112, it is possible to accurately match the captured image with the point cloud data, and it is possible to accurately detect a wide variety of obstacles on the road surface.

[0042] Therefore, the vehicle control device 101 can determine whether or not avoidance is necessary, and the steering control device 103 and the brake control device 104 can perform control.

[0043] 8 to 11 are functional block diagrams of other examples of the obstacle detection device of this embodiment. A characteristic feature of the example shown in FIG. 8 is that an attribute assigning unit 213 is provided between a point cloud data acquisition unit 211 and a point cloud data processing unit 212. The attribute assigning unit 213 may define one attribute for each distance measurement point of the point cloud data, or may assign a label indicating the attribute (road, white line, etc.) to the point cloud data. The attribute associates the segmentation result with the point cloud, and for example, color may be assigned as an attribute.

[0044] A distinctive feature of the embodiment shown in FIG. 9 is the provision of an image determination unit 222 that acquires point cloud data of the obstacle detection result from the obstacle detection unit 221, reprojects it onto the image of the segmentation result, and performs image determination. The image determination unit 222 determines whether the point cloud data of the obstacle detection result can be reprojected onto the road surface area of ​​the segmentation result. At a distance, the road surface becomes nearly parallel to the LiDAR irradiation direction, so point cloud data corresponding to the road surface is not acquired. Because the obstacle detection unit 221 detects obstacles based on the point cloud data, it cannot determine whether the obstacle of the detection result exists in the road surface area. By providing the image determination unit 222, obstacles in the road surface area can be reliably found.

[0045] 10 is characterized in that a point cloud data integration unit 216 is provided between the point cloud data acquisition unit 211 and the point cloud data processing unit 212, and a point cloud data integration result storage unit 207 is provided to store integrated point cloud data that is the result of integrating the point cloud data. Point cloud data integration can be performed based on a technology known as SLAM (Simultaneous Localization and Mapping), for example.

[0046] The point cloud data integration unit 216 integrates point cloud data for multiple frames and accumulates and stores the integrated point cloud data in the point cloud data integration result storage unit 207. Then, the point cloud data processing unit 212 performs point cloud data processing using the integrated point cloud data, which is the point cloud data integration result. According to this embodiment, the amount of point cloud data increases, allowing for better visibility of distant objects and improved detection performance.

[0047] 11 is characterized in that the vehicle state is acquired from the vehicle sensor 113 and used in the obstacle detection process. The obstacle detection device 102 has a vehicle state acquisition unit 231 and a process control unit 232. The vehicle state acquisition unit 231 acquires information on the vehicle state from the vehicle sensor 113. The vehicle sensor 113 detects the vehicle speed and engine operating state of the vehicle.

[0048] The processing control unit 232 provides information on the vehicle state to the image processing unit 202 and the point cloud data processing unit 212. For example, when the vehicle is stopped, there is no need to perform obstacle detection, so the processing control unit 232 issues an instruction to stop processing to the image processing unit 202 and the point cloud data processing unit 212. This stops the processing of the image processing unit 202 and the point cloud data processing unit 212, reduces the calculation load on the ECU, and allows the remaining calculation resources to be used for other control, etc., thereby improving the processing speed.

[0049] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various design modifications can be made without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0050] DESCRIPTION OF SYMBOLS 101: Vehicle control device, 102: Obstacle detection device, 111: Imaging device, 112: Distance measuring device, 201: Image acquisition unit, 202: Image processing unit, 211: Point cloud data acquisition unit, 212: Point cloud data processing unit, 221: Obstacle detection unit, 501: Vehicle, 502: Obstacle, 503: Leading vehicle, 504: Road, 505: White line, 601: Captured image, 611: Point cloud data, 701: Point cloud data, 702: Road surface candidate point cloud data, 704: Point cloud data after road surface removal, 711: Obstacle, 712: Vehicle (leading vehicle)

Claims

1. An obstacle detection device comprising: an image acquisition unit that acquires an image captured by an imaging device; a point cloud data acquisition unit that acquires point cloud data measured by a distance measuring device; an image processing unit that recognizes a road surface and a specified object based on the image; a point cloud data processing unit that generates processed point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the specified object from the point cloud data; and an obstacle detection unit that detects an obstacle based on the processed point cloud data, wherein the point cloud data processing unit estimates a road surface model indicating the shape of the road surface based on the point cloud data and a result of the road surface recognition by the image processing unit, identifies the road surface point cloud data from the point cloud data based on the road surface model, and identifies the object point cloud data by clustering road surface-removed point cloud data in which the road surface point cloud data has been removed from the point cloud data.

2. The obstacle detection device according to claim 1, characterized in that the point cloud data processing unit sets an allowable range of error for the road surface model in accordance with distance information contained in the point cloud data.

3. The obstacle detection device according to claim 1, further comprising a road surface obstacle determination unit that determines whether or not an obstacle detected by the obstacle detection unit is present on the road surface.

4. The obstacle detection device described in claim 1, characterized in that the image processing unit performs a segmentation process on the image by referring to a road surface area model and an object model, the point cloud data processing unit extracts road surface candidate point cloud data by matching the segmentation result of the segmentation process with the acquired point cloud data acquired by the point cloud data acquisition unit, estimates the road surface model from the point cloud shape of the road surface candidate point cloud data, re-extracts the road surface point cloud data from the acquired point cloud data based on the road surface model to create re-extracted road surface point cloud data, removes the re-extracted road surface point cloud data from the acquired point cloud data to create the post-road surface removal point cloud data, clusters the post-road surface removal point cloud data to identify the object point cloud data which is a cluster corresponding to the specified object, and removes the object point cloud data from the post-road surface removal point cloud data to generate the post-processing point cloud data.

5. The obstacle detection device according to claim 1, characterized in that the point cloud data processing unit generates the processed point cloud data by removing the road surface point cloud data and the object point cloud data from integrated point cloud data accumulated by integrating the point cloud data for multiple frames.

6. The obstacle detection device according to claim 2, wherein the point cloud data processing unit reduces the allowable range of error in the road surface model as the distance to the point cloud increases.

Images