Obstacle detection device

Abstract

To provide an obstacle detection device which can accurately detect a distant obstacle.SOLUTION: An obstacle detection device includes: an image acquisition unit 201 which acquires a captured image from an image-capturing device 111; a point group data acquisition unit 211 which acquires point group data from a distance measurement device 112; an image processing unit 202 which recognizes a road surface and predetermined objects on the basis of the image; a point group data processing unit 212 which removes road surface point group data corresponding to the road surface and object point group data corresponding to the predetermined objects from the point group data; and an obstacle detection unit 221 which detects an obstacle on the road surface on the basis of point group data after processing. The point group data processing unit 212 estimates a road surface model on the basis of the point group data and a result of recognition of the road surface by the image processing unit, identifies the road surface point group data from the point group data on the basis of the road surface model, and clusters point group data after removal of the road surface obtained by removing the road surface point group data from the point group data to identify the object point group data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an obstacle detection device. [Background technology]

[0002] ADAS (Advanced Driver Assistance System) / AD (Autonomous Driving) technology is being developed to provide driving assistance / vehicle control by utilizing sensing information around the vehicle. In ADAS / AD technology, there is a need for long-distance obstacle detection to avoid danger early. Patent Document 1 shows the configuration of an object recognition system that includes an ECU, a distance measurement sensor, and an imaging device, and when an emission direction corresponding to an emission wave that cannot detect the reflected wave of the distance measurement sensor exists between two distance measurement points, sets a threshold based on an image generated by the imaging device and makes a judgment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-006716 A Summary of the Invention [Problem to be solved by the invention]

[0004] Although an image capture device alone can detect paths and vehicles at long distances, it has difficulty detecting a wide variety of obstacles. A distance measurement sensor alone has a small amount of point cloud data for objects at long distances, making it difficult to determine whether or not they are obstacles.

[0005] In the case of technology that uses the detection results of an imaging device and a distance measuring sensor as in Patent Document 1, positional deviations occur between the detection results of the two due to calibration errors and asynchrony, making simple correspondence difficult. For example, there is a problem in that relatively small obstacles such as objects that have fallen on the road are buried in part of the road surface, making them difficult to detect.

[0006] The present invention has been made in consideration of the above-mentioned points, and an object of the present invention is to provide an obstacle detection device that can accurately detect distant obstacles. [Means for solving the problem]

[0007] The obstacle detection device of the present invention that solves the above problems is: an image acquisition unit that acquires an image captured by an imaging device; a point cloud data acquisition unit for acquiring point cloud data measured by a distance measuring device; an image processing unit that recognizes a road surface and a predetermined object based on the image; a point cloud data processing unit that generates processed point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the predetermined object from the point cloud data; an obstacle detection unit that detects an obstacle present on the road surface based on the processed point cloud data, The point cloud data processing unit includes: estimating a road surface model indicating a shape of the road surface based on the point cloud data and a result of the recognition of the road surface by the image processing unit; Identifying the road surface point cloud data from the point cloud data based on the road surface model; The object point cloud data is identified by clustering road surface removed point cloud data obtained by removing the road surface point cloud data from the point cloud data. It is characterized by: Effect of the Invention

[0008] According to the present invention, it is possible to obtain an obstacle detection device capable of detecting distant obstacles with high accuracy. Further features related to the present invention will become apparent from the description of the present specification and the attached drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1]1 is a functional block diagram of a vehicle control device equipped with an obstacle detection device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of the obstacle detection device according to the embodiment. [Diagram 3] 4 is a flowchart showing an obstacle detection method performed by the obstacle detection device of the present embodiment. [Figure 4] 4 is a flowchart showing the details of the process in S313 of FIG. 3. [Diagram 5] FIG. 1 is a schematic diagram showing a specific scene of obstacle detection. [Figure 6] 3A and 3B are diagrams showing an example of a captured image of the area in front of a vehicle and point cloud data. [Figure 7] 3A to 3C are diagrams for explaining a method of extracting obstacle candidates in the obstacle detection device of the embodiment; [Figure 8] FIG. 4 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [Figure 9] FIG. 4 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [Figure 10] FIG. 4 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [Figure 11] FIG. 4 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Next, an embodiment of an obstacle detection device according to the present invention will be described. FIG. 1 is a functional block diagram of a vehicle control device equipped with an obstacle detection device according to this embodiment.

[0011] The vehicle control device 101 is mounted on a vehicle having an ADAS or AD, and has an imaging device 111 and a distance measuring device 112 connected to its input side, and a steering device 121 and a braking device 122 connected to its output side.

[0012] The vehicle control device 101 is composed of an ECU (Electronic Control Unit) having a CPU, memory, and input / output sections, and embodies an obstacle detection device 102, a steering control device 103, and a brake control device 104 by executing software programs stored in the memory using the CPU.

[0013] The obstacle detection device 102 detects an obstacle present in front of the vehicle based on information acquired from the imaging device 111 and the distance measuring device 112, and transmits a detection signal to the steering control device 103 and the brake control device 104. The steering control device 103 and the brake control device 104 control the steering device 121 and the brake device 122 based on the detection signal from the obstacle detection device 102.

[0014] The imaging device 111 is, for example, a monocular camera, and is installed on the vehicle so as to capture an image of the area in front of the vehicle, and generates an image of the captured image range. The distance measuring device 112 is, for example, a LiDAR (Laser Imaging Detection and Ranging), and is installed on the vehicle so that the imaging range of the imaging device 111 and the distance measuring range overlap, and emits an outgoing wave in front of the vehicle and receives a reflected wave from an object to determine a distance measuring point. The imaging device 111 and the distance measuring device 112 each capture images and measures at a predetermined frame rate.

[0015] FIG. 2 is a functional block diagram of the obstacle detection device of the present embodiment. The obstacle detection device 102 has an image acquisition unit 201, an image processing unit 202, a road surface area model 203, an object model 204, a segmentation result 205, camera parameters 206, a point cloud data acquisition unit 211, a point cloud data processing unit 212, and an obstacle detection unit 221.

[0016] The image acquisition unit 201 acquires an image captured by the imaging device (camera) 111. The image processing unit 202 classifies the captured image into pixel regions of roads and white lines by semantic segmentation, and recognizes predetermined objects such as vehicles, people, bicycles, and motorcycles as rectangular regions by object detection. Semantic segmentation / object detection can be performed based on a model learned by machine learning or deep learning, for example.

[0017] The road surface area model 203 corresponds to semantic segmentation, and the object model 204 corresponds to object detection. In the image processing unit 202 of this embodiment, pixel areas of roads and white lines are classified by semantic segmentation, and object detection is performed by recognizing a predetermined object as a rectangular area by image processing, but the method is not limited to these. For example, a predetermined object such as a vehicle may be recognized as a pixel area by semantic segmentation. Hereinafter, segmentation processing includes semantic segmentation / object detection, and segmentation results include semantic segmentation results / object detection results.

[0018] The image processing unit 202 executes segmentation processing by referring to the road surface area model 203 and the object model 204. Information on the segmentation result by the image processing unit 202 is stored in the segmentation result 205. The information on the segmentation result is stored for each frame. The camera parameters 206 store external parameters (six degrees of freedom corresponding to position and rotation) indicating the relative positional relationship between the imaging device 111 and the distance measuring device 112, and internal parameters such as the focal length, image resolution, and distortion coefficient of the imaging device 111.

[0019] The point cloud data acquisition unit 211 acquires point cloud data measured by the distance measurement device 112. The point cloud data consists of three-dimensional data of a plurality of distance measurement points. The point cloud data processing unit 212 removes road surface point cloud data and object point cloud data corresponding to a predetermined object from the acquired point cloud data to generate post-removal processing point cloud data. The obstacle detection unit 221 detects obstacles from the post-removal processing point cloud data.

[0020] The obstacle detection unit 221 detects obstacle candidates based on the post-removal processing point cloud data. The obstacle detection unit 221 has a road surface obstacle determination unit that determines whether or not an obstacle candidate exists on the road surface. The road surface obstacle determination unit performs processing to remove point clouds of objects such as buildings, sidewalks, and curbs that are located in places other than the road on which the vehicle travels from the post-removal processing point cloud data, and ultimately determines the remaining point cloud data as road surface obstacles.

[0021] FIG. 3 is a flowchart showing an obstacle detection method performed by the obstacle detection device of this embodiment.

[0022] The image acquisition unit 201 acquires a captured image from the imaging device 111 (S301). The image processing unit 202 executes a segmentation process by referring to the road surface area model 203 and the object model 204, and accumulates the segmentation result 205 (S302).

[0023] On the other hand, the point cloud data acquisition unit 211 acquires point cloud data from the distance measuring device 112 (S311). Hereinafter, the point cloud data acquired by the point cloud data acquisition unit 211 will be referred to as acquired point cloud data.

[0024] The point cloud data processing unit 212 judges whether there is a segmentation result with a timestamp close to that of the frame of the acquired point cloud data (S312). If there is a segmentation result with a timestamp close to that of the frame of the acquired point cloud data (Yes in S312), the point cloud data after removal process is generated by removing the road surface point cloud data and the object point cloud data corresponding to a predetermined object from the acquired point cloud data (S313). The obstacle detection unit 221 performs a process of detecting an obstacle from the point cloud data after removal process (S314).

[0025] FIG. 4 is a flowchart showing the details of the process in S313 of FIG. The point cloud data processing unit 212 extracts road surface candidate point cloud data by associating the segmentation result 205 acquired by the segmentation process in the image processing unit with the acquired point cloud data acquired by the point cloud data acquisition unit 211 (S401: road surface candidate point cloud data extraction unit). Here, roads and white lines are extracted as road surface candidates from the point cloud data associated with the segmentation result, and are used as road surface candidate point cloud data.

[0026] Next, a road surface model indicating the shape of the road surface is estimated from the point cloud shape of the road surface candidate point cloud data (S402: road surface model estimating unit). As a method for estimating the road surface model from the point cloud shape, a known technique such as plane estimation using RANSAC is used.

[0027] Here, point cloud data within a predetermined height range from the road surface model may be included in the road surface. For example, since the vehicle 501 can overcome an obstacle with a height of 10 cm or less, a range of 10 cm or less from the plane may be included in the road surface as an allowable error range.

[0028] The allowable error range of the road surface model may be set according to distance information included in the acquired point cloud data. For example, the allowable error of the road surface model is set to be smaller as the distance from the distance measuring device 112 to the point cloud increases. This makes it possible to prevent a distant obstacle from being erroneously determined to be a road surface.

[0029] Then, the road surface point cloud data is re-extracted from the acquired point cloud data based on the estimated road surface model to generate re-extracted road surface point cloud data (S403: road surface point cloud data re-extraction unit). The re-extracted road surface point cloud data is removed from the acquired point cloud data to generate road surface removed point cloud data (S404: re-extracted road surface point cloud data removing unit).

[0030] The point cloud data after removing the road surface is clustered to identify object point cloud data that is a cluster corresponding to a predetermined object (S405: object point cloud data identification unit). As a result, object point cloud data such as a preceding vehicle, pedestrian, bicycle, motorcycle, etc. is identified.

[0031] The object point cloud data is removed from the road surface removed point cloud data to generate removed point cloud data (S406: object point cloud data removing unit). As a result, removed point cloud data is generated in which the road surface point cloud data and the object point cloud data are removed from the acquired point cloud data.

[0032] FIG. 5 is a schematic diagram showing a specific scene of obstacle detection, and FIG. 6 is a diagram showing an example of a captured image of the area in front of the vehicle and point cloud data. In this embodiment, a vehicle control device 101 including an obstacle detection device 102 is mounted on a vehicle 501 .

[0033] As shown in FIG. 5, a vehicle 501 is traveling on a three-lane road 504 divided by white lines 505 and 506, and ahead of the vehicle 501 are a leading vehicle 503 and a number of obstacles 502 on the road.

[0034] As shown in Fig. 6(a), a captured image 601 captured by the imaging device 111 includes the road surface of the road 504, white lines 505 and 506, an obstacle 502, and a preceding vehicle 503. As shown in Fig. 6(b), point cloud data 611 measured by the distance measuring device 112 includes a point cloud of the road surface of the road 504, the white lines 505 and 506, the obstacle 502, and the preceding vehicle 503.

[0035] As shown in Fig. 6(a), the imaging device 111 alone can detect a road or a preceding vehicle 503 at a long distance, but it is difficult to detect a wide variety of obstacles. Also, as shown in Fig. 6(b), the distance measuring device 112 alone has a small amount of point cloud data for objects at a long distance, making it difficult to determine whether or not they are obstacles. Furthermore, a simple correspondence between the captured image and the point cloud data causes positional deviations due to calibration errors, asynchronization, etc., and small obstacles are difficult to detect because they are buried in part of the road surface.

[0036] FIG. 7 is a diagram for explaining a method of extracting obstacle candidates in the obstacle detection device of this embodiment. 7 shows point cloud data 701 in which segmentation results are associated with acquired point cloud data. In the point cloud data 701, an obstacle 711 is buried in the road surface and is considered to be part of the road. Part of the road is also recognized as a vehicle (preceding vehicle) 712.

[0037] In the obstacle detection device 102 of this embodiment, the point cloud data processing unit 212 extracts roads and white lines as road surface candidates from the point cloud data associated with the segmentation results, and extracts road surface candidate point cloud data 702.

[0038] Then, a road surface model indicating the shape of the road surface is estimated from the point cloud shape of the road surface candidate point cloud data, and road surface point cloud data is re-extracted from the point cloud data 701 based on the estimated road surface model to create re-extracted road surface point cloud data 703.

[0039] Next, the re-extracted road surface point cloud data 703 is removed from the point cloud data 701 to generate post-road surface removal point cloud data 704. In the post-road surface removal point cloud data 704, point clouds of an obstacle 711 and a preceding vehicle 712 remain as obstacle candidates.

[0040] Here, the post-road surface removal point cloud data 704 is clustered to identify object point cloud data that is a cluster corresponding to a predetermined object (preceding vehicle 712), and the point cloud data of the preceding vehicle 712 that is the object point cloud data is removed from the post-road surface removal point cloud data 704 to generate post-removal processing point cloud data. Only the point cloud data of the obstacle remains in the post-removal processing point cloud data.

[0041] According to the obstacle detection device 102 of this embodiment, even if a positional deviation occurs between the imaging device 111 and the distance measuring device 112, it is possible to accurately match the captured image with the point cloud data, and it is possible to accurately detect a wide variety of obstacles on the road surface.

[0042] Therefore, the vehicle control device 101 can determine whether or not avoidance is necessary, and the steering control device 103 and the brake control device 104 can perform control.

[0043] 8 to 11 are functional block diagrams of other examples of the obstacle detection device of this embodiment. 8 is characterized in that an attribute assigning unit 213 is provided between the point cloud data acquisition unit 211 and the point cloud data processing unit 212. The attribute assigning unit 213 may define one attribute for each distance measurement point of the point cloud data, and may assign a label indicating the attribute (road, white line, etc.) to the point cloud data. The attribute associates the segmentation result with the point cloud, and for example, color may be assigned as the attribute.

[0044] The embodiment shown in FIG. 9 is characterized in that an image determination unit 222 is provided to acquire point cloud data of the obstacle detection result from the obstacle detection unit 221, reproject it onto the image of the segmentation result, and perform image determination. The image determination unit 222 determines whether the point cloud data of the obstacle detection result is reprojected onto the road surface area of ​​the segmentation result. In the distance, the road surface is nearly parallel to the irradiation direction of the LiDAR, so point cloud data corresponding to the road surface is not acquired. Since the obstacle detection unit 221 detects an obstacle based on the point cloud data, it cannot determine whether the obstacle of the detection result exists in the road surface area. By providing the image determination unit 222, it is possible to reliably find an obstacle in the road surface area.

[0045] 10 is characterized in that a point cloud data integration unit 216 is provided between the point cloud data acquisition unit 211 and the point cloud data processing unit 212, and a point cloud data integration result storage unit 207 is provided for storing integrated point cloud data that is the integration result of the point cloud data. The point cloud data integration can be performed based on a technology known as SLAM (Simultaneous Localization and Mapping), for example.

[0046] The point cloud data integration unit 216 integrates the point cloud data for multiple frames and accumulates and stores the integrated point cloud data in the point cloud data integration result storage unit 207. Then, the point cloud data processing unit 212 performs point cloud data processing using the integrated point cloud data that is the point cloud data integration result. According to this embodiment, the amount of point cloud data increases, distant objects can be clearly seen, and detection performance can be improved.

[0047] 11 is characterized in that the vehicle state is acquired from the vehicle sensor 113 and used for the obstacle detection process. The obstacle detection device 102 has a vehicle state acquisition unit 231 and a process control unit 232. The vehicle state acquisition unit 231 acquires information on the vehicle state from the vehicle sensor 113. The vehicle sensor 113 detects the vehicle speed and the engine operating state of the vehicle.

[0048] The processing control unit 232 provides information on the vehicle state to the image processing unit 202 and the point cloud data processing unit 212. For example, when the vehicle is stopped, there is no need to perform obstacle detection, and therefore the processing control unit 232 issues an instruction to stop processing to the image processing unit 202 and the point cloud data processing unit 212. This stops the processing of the image processing unit 202 and the point cloud data processing unit 212, reduces the calculation load on the ECU, and allows the remaining calculation resources to be used for other controls, etc., to improve the processing speed.

[0049] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0050] 101...vehicle control device, 102...obstacle detection device, 111...imaging device, 112...distance measuring device, 201...image acquisition unit, 202...image processing unit, 211...point cloud data acquisition unit, 212...point cloud data processing unit, 221...obstacle detection unit, 501...vehicle, 502...obstacle, 503...preceding vehicle, 504...road, 505...white line, 601...captured image, 611...point cloud data, 701...point cloud data, 702...road surface candidate point cloud data, 704...point cloud data after road surface removal, 711...obstacle, 712...vehicle (preceding vehicle)

Claims

1. an image acquisition unit that acquires an image captured by an imaging device; a point cloud data acquisition unit for acquiring point cloud data measured by a distance measuring device; an image processing unit that recognizes a road surface and a predetermined object based on the image; a point cloud data processing unit that generates processed point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the predetermined object from the point cloud data; an obstacle detection unit that detects an obstacle based on the processed point cloud data, The point cloud data processing unit includes: estimating a road surface model indicating a shape of the road surface based on the point cloud data and a result of the recognition of the road surface by the image processing unit; Identifying the road surface point cloud data from the point cloud data based on the road surface model; The object point cloud data is identified by clustering road surface removed point cloud data obtained by removing the road surface point cloud data from the point cloud data. An obstacle detection device comprising:

2. 2. The obstacle detection device according to claim 1, wherein the point cloud data processing unit sets an allowable range of error for the road surface model in accordance with distance information included in the point cloud data.

3. 2. The obstacle detection device according to claim 1, further comprising a road surface obstacle determination unit that determines whether or not the obstacle detected by the obstacle detection unit is present on the road surface.

4. The image processing unit performs a segmentation process on the image by referring to a road surface area model and an object model; 2. The obstacle detection device according to claim 1, wherein the point cloud data processing unit extracts road surface candidate point cloud data by correlating a segmentation result of the segmentation process with the acquired point cloud data acquired by the point cloud data acquisition unit, estimates the road surface model from a point cloud shape of the road surface candidate point cloud data, re-extracts the road surface point cloud data from the acquired point cloud data based on the road surface model to create re-extracted road surface point cloud data, removes the re-extracted road surface point cloud data from the acquired point cloud data to create the post-road surface-removed point cloud data, clusters the post-road surface-removed point cloud data to identify the object point cloud data which is a cluster corresponding to the specified object, and removes the object point cloud data from the post-road surface-removed point cloud data to generate the post-processed point cloud data.

5. The obstacle detection device according to claim 1, characterized in that the point cloud data processing unit generates the processed point cloud data by removing the road surface point cloud data and the object point cloud data from integrated point cloud data obtained by integrating and storing the point cloud data for multiple frames.

6. 3. The obstacle detection device according to claim 2, wherein the point cloud data processing unit reduces an allowable range of error in the road surface model as the distance to the point cloud increases.

Images