OBSTACLE DETECTION DEVICE

Abstract

The goal is to obtain an obstacle detection device capable of accurately determining an obstacle at a large distance. The obstacle detection device includes an image acquisition unit 201, which receives a captured image from an imaging device 111; a point cloud data acquisition unit 211, which acquires point cloud data from a distance measuring device 112; an image processing unit 202, which recognizes a road surface and a predefined object based on the image; a point cloud data processing unit 212, which extracts road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the predefined object from the point cloud data; and an obstacle detection unit 221, which detects an obstacle on the road surface based on the post-processed point cloud data.The point cloud data processing unit 212 estimates a road surface model based on the point cloud data and the result of the road surface detection by the image processing unit, specifies the road surface point cloud data from the point cloud data based on the road surface model, and specifies the object point cloud data by performing clustering of the point cloud data remaining after the removal of the road surface, in which the road surface point cloud data has been removed from the point cloud data.

Description

Technical field

[0001] The present invention relates to an obstacle detection device. background

[0002] A technology for advanced driver assistance systems (ADAS) and autonomous driving (AD) has been developed to provide driver assistance and vehicle control by using sensing information from around the vehicle. ADAS and AD technology require the ability to detect obstacles from a great distance in order to initiate an evasive maneuver early and avoid hazards.PTL 1 discloses a configuration of an object detection system comprising an ECU, a distance measurement sensor and an imaging device, and is configured such that in a case where there is an emission direction between two distance measurement points corresponding to an emission wave emitted by a distance measurement sensor, from which a reflected wave cannot be detected, a threshold is set based on an image generated by the imaging device to make a determination. Citation list for patent literature

[0003] PTL1: JP 2022-006716 A Overview of the invention Technical problem

[0004] While an imaging device alone can detect a road and a vehicle at a great distance, it is difficult to detect different types of obstacles using only imaging. With a distance sensor alone, the amount of point cloud data for an object at a great distance is small, making it difficult to determine whether the object constitutes an obstacle.

[0005] In the technique that uses the detection results of an imaging device and a distance sensor, as in PTL 1, a positional deviation occurs between the detection results of the imaging device and the distance sensor due to calibration errors, asynchronization, etc., making a simple mapping difficult. For example, this technique has the problem that a relatively small obstacle, such as an object that has fallen onto a road, blends in with part of the road surface, making it difficult to detect the obstacle.

[0006] The present invention was made with regard to the above points, and one objective of the present invention is to provide an obstacle detection device that is capable of accurately detecting an obstacle at a great distance. Solution to the problem

[0007] To solve the above-mentioned problem, an obstacle detection device according to the present invention includes an image acquisition unit that captures an image taken by an imaging device, a point cloud data acquisition unit that captures point cloud data measured by a distance measuring device, an image processing unit that recognizes a road surface and a predefined object based on the image, a point cloud data processing unit that generates post-processed point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the specified object from the point cloud data, and an obstacle detection unit that detects an obstacle present on the road surface based on the post-processed point cloud data, wherein the point cloud data processing unit a road surface model that represents a shape of the road surface, based on the point cloud data and a result of the road surface recognition by the image processing unit, the road surface point cloud data specified from the point cloud data based on the road surface model, and performs clustering of the remaining point cloud data after the removal of the road surface, in which the road surface point cloud data has been removed from the point cloud data, in order to specify the object point cloud data. Advantageous effects of the invention

[0008] According to the present invention, an obstacle detection device can be obtained that is capable of accurately detecting an obstacle at a large distance. Further features related to the present invention will become apparent from the description of this specification and the accompanying drawings. Problems, configurations, and effects other than those described above are illustrated by the following description of embodiments. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a functional block diagram of a vehicle control device which includes an obstacle detection device according to the present embodiment. [ Fig. 2] Fig. Figure 2 is a functional block diagram of the obstacle detection device according to the present embodiment. [ Fig. 3] Fig. Figure 3 is a flowchart illustrating an obstacle detection procedure by the obstacle detection device according to the present embodiment. [ Fig. 4] Fig. 4 is a flowchart that shows processing content in S313 in Fig. 3 illustrated in detail. [ Fig. 5] Fig. Figure 5 is a schematic view illustrating a specific obstacle detection scene. [ Fig. 6] Fig. Figure 6 is a diagram illustrating an example of a captured image and point cloud data of the front of a vehicle. [ Fig. 7] Fig. Figure 7 is a diagram illustrating a method for extracting an obstacle candidate by the obstacle detection device according to the present embodiment. [ Fig. 8] Fig. Figure 8 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [ Fig. 9] Fig. Figure 9 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [ Fig. 10] Fig. Figure 10 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. [ Fig. 11] Fig. Figure 11 is a functional block diagram of another example of the obstacle detection device according to the present embodiment. Description of the embodiments

[0009] Next, an embodiment of an obstacle detection device according to the present invention will be described.

[0010] Fig. Figure 1 is a functional block diagram of a vehicle control device which includes an obstacle detection device according to the present embodiment.

[0011] A vehicle control device 101 is mounted on a vehicle that has ADAS or AD, and an imaging device 111 and a distance measuring device 112 are connected to the input side, and a steering device 121 and a braking device 122 are connected to the output side.

[0012] The vehicle control device 101 is designed with an electronic control unit (ECU) containing a CPU, a memory and an input and output unit and provides the functions of the obstacle detection device 102, a steering control device 103 and a brake control device 104 through the CPU, which executes a software program stored in the memory.

[0013] The obstacle detection device 102 detects an obstacle in front of the vehicle based on information obtained from the imaging device 111 and the distance measuring device 112, and sends a detection signal to the steering control device 103 and the brake control device 104. The steering control device 103 and the brake control device 104 control a steering device 121 and a brake device 122 based on the detection signal from the obstacle detection device 102.

[0014] The imaging device 111, for example, is a monocular camera installed in the vehicle such that it performs imaging of the front of the vehicle and generates an image within its imaging area. The distance measuring device 112, for example, is a laser imaging detection and ranging (LiDAR) device installed in the vehicle such that the imaging area and the distance measuring area of ​​the imaging device 111 overlap. It emits a wave toward the front of the vehicle and receives a reflected wave from the object to obtain a distance measurement point. The imaging device 111 and the distance measuring device 112 perform the imaging and measurement at their respective predefined frame rates.

[0015] Fig. Figure 2 is a functional block diagram of the obstacle detection device according to the present embodiment.

[0016] The obstacle detection device 102 includes an image acquisition unit 201, an image processing unit 202, a road surface area model 203, an object model 204, a segmentation result 205, a camera parameter 206, a point cloud data acquisition unit 211, a point cloud data processing unit 212 and an obstacle detection unit 221.

[0017] The image acquisition unit 201 captures an image taken by the imaging device (camera) 111. The image processing unit 202 classifies parts of a captured image into pixel areas of the road and white lines through semantic segmentation and recognizes predefined objects such as a vehicle, a person, a bicycle, and a motorcycle as rectangular areas through object recognition. The semantic segmentation and object recognition can be performed, for example, based on a model learned through machine learning or deep learning.

[0018] The road surface area model 203 corresponds to semantic segmentation, and the object model 204 corresponds to object recognition. The image processing unit 202 of the present embodiment classifies the pixel areas of the road and the white lines by semantic segmentation and performs object recognition to identify predefined objects as rectangular areas by image processing, but the method is not limited to this. For example, a predefined object such as a vehicle can be recognized as a pixel area by semantic segmentation. In the following, it is assumed that the segmentation processing includes semantic segmentation and object recognition, and that a segmentation result includes both a semantic segmentation result and an object recognition result.

[0019] The image processing unit 202 performs the segmentation processing with reference to the road surface area model 203 and the object model 204. Segmentation result 205 contains information about the segmentation result performed by the image processing unit 202. This segmentation result information is collected for each frame. Camera parameter 206 stores external parameters that specify a relative positional relationship between the imaging device 111 and the distance measuring device 112 (six degrees of freedom corresponding to position and rotation), as well as internal parameters such as focal length, image resolution, and a distortion coefficient of the imaging device 111.

[0020] The point cloud data acquisition unit 211 acquires point cloud data measured by the distance measuring device 112. The point cloud data contains three-dimensional data from a multitude of distance measurement points. The point cloud data processing unit 212 removes road surface point cloud data and object point cloud data corresponding to the specified object from the acquired point cloud data to generate point cloud data for distance-based processing. The obstacle detection unit 221 detects an obstacle from the point cloud data for distance-based processing.

[0021] The obstacle detection unit 221 detects a candidate obstacle based on point cloud data for post-removal processing. The obstacle detection unit 221 includes a road surface obstacle detection unit, which determines whether a candidate obstacle is present on the road surface. This road surface obstacle detection unit performs post-removal processing of point clouds from objects located outside the road surface, such as buildings, sidewalks, and curbs, and identifies the remaining point clouds as road surface obstacles.

[0022] Fig. Figure 3 is a flowchart illustrating an obstacle detection procedure performed by the obstacle detection device according to the present embodiment.

[0023] The image acquisition unit 201 captures an image from the imaging device 111 (S301). The image processing unit 202 performs the segmentation processing with reference to the road surface area model 203 and the object model 204 and collects the segmentation result 205 (S302).

[0024] Meanwhile, the point cloud data acquisition unit 211 acquires point cloud data from the distance measuring device 112 (S311). Hereinafter, the point cloud data acquired by the point cloud data acquisition unit 211 will be referred to as acquired point cloud data.

[0025] The point cloud data processing unit 212 determines whether a segmentation result with a timestamp close to that of the frame of the captured point cloud data exists (S312). If a segmentation result with a close timestamp exists (Yes in S312), the road surface point cloud data and the object point cloud data corresponding to the specified object are removed from the captured point cloud data to generate the point cloud data for post-distance processing (S313). The obstacle detection unit 221 performs obstacle detection processing from the point cloud data for post-distance processing (S314).

[0026] Fig. 4 is a flowchart that shows the processing content in S313 in Fig. 3 presents in detail.

[0027] The point cloud data processing unit 212 extracts point cloud data of a road surface candidate with the segmentation result 205, which was obtained through segmentation processing in the image processing unit associated with the point cloud data acquired by the point cloud data acquisition unit 211 (S401: Road surface candidate extraction unit point cloud data). Here, the road and the white lines are extracted as road surface candidates from the point cloud data associated with the segmentation result to form road surface candidate point cloud data.

[0028] Next, a road surface model representing the shape of the road surface is estimated from the point cloud shape of the point cloud data of the road surface candidate (S402: Road Surface Model Estimation Unit). A well-known technique such as plane estimation using RANSAC is used, for example, to estimate a road surface model from the point cloud shape.

[0029] Here, point cloud data within a predefined height range can be incorporated into the road surface model. For example, since a vehicle 501 can drive over an obstacle 10 cm or less high, an area up to 10 cm high can also be included within a permissible error range and incorporated into the road surface.

[0030] The permissible error range of the road surface model can be determined according to the distance information contained in the acquired point cloud data. For example, the permissible error of the road surface model is set smaller for greater distances between the distance measuring device 112 and the point cloud. As a result, it can be prevented that an obstacle at a great distance is incorrectly identified as the road surface.

[0031] Then, based on the estimated road surface model, the road surface point cloud data are re-extracted from the captured point cloud data to generate re-extracted road surface point cloud data (S403: Unit for re-extracting road surface point cloud data).

[0032] The re-extracted road surface point cloud data are removed from the captured point cloud data to generate remaining point cloud data after road surface removal (S404: Unit for removing re-extracted road surface point cloud data).

[0033] The point cloud data remaining after the removal of the road surface is clustered to specify object point cloud data that represent a cluster corresponding to the specified object (S405: Unit for specifying object point cloud data). As a result, for example, portions of the object point cloud data of a preceding vehicle, a pedestrian, a bicycle, a two-wheeler, etc., are specified.

[0034] The object point cloud data is removed from the point cloud data remaining after the removal of the road surface to generate point cloud data for post-removal processing (S406: Unit for removing object point cloud data). As a result, the point cloud data for post-removal processing is generated in which the road surface point cloud data and the object point cloud data have been removed from the captured point cloud data.

[0035] Fig. Figure 5 is a schematic diagram illustrating a specific obstacle detection scenario, and Fig. Figure 6 is a diagram illustrating an example of captured image and point cloud data of the front of the vehicle.

[0036] In the present embodiment, the vehicle control unit 101, which includes the obstacle detection unit 102, is mounted on the vehicle 501.

[0037] As in Fig. As shown in Figure 5, vehicle 501 is driving on a three-lane road 504, which is divided by white lines 505 and 506, and in front of vehicle 501 there is a vehicle 503 ahead and several obstacles 502 on the road.

[0038] As in Fig. As shown in Figure 6(a), images of the road surface of road 504, the white lines 505 and 506, the obstacles 502, and the vehicle ahead 503 are captured in an image 601 taken by the imaging device 111. As shown in Figure 6(a), images of the road surface of road 504, the white lines 505 and 506, the obstacles 502, and the vehicle ahead 503 are captured in an image 601 taken by the imaging device 111. Fig. Figure 6(b) shows point clouds of the road surface of road 504, the white lines 505 and 506, the obstacles 502 and the vehicle ahead 503 in point cloud data 611 measured by the distance measuring device 112.

[0039] As in Fig. As shown in Figure 6(a), the road and the vehicle 503 ahead can be detected at a great distance solely by the imaging device 111; however, it is difficult to detect different types of obstacles. Furthermore, as shown in Figure 6(a), Fig. As shown in Figure 6(b), using the distance measuring device 112 alone, the amount of point cloud data for an object at a large distance is small, making it difficult to determine whether the object is an obstacle. Furthermore, a simple mapping between a captured image and point cloud data results in positional deviations due to calibration errors, asynchronization, etc., and there is a problem that a small obstacle may be superimposed onto part of the road surface, making it difficult to detect.

[0040] Fig. Figure 7 is a diagram illustrating a method of extracting an obstacle candidate by the obstacle detection device according to the present embodiment.

[0041] In Fig. Point cloud data 701, as shown, depicts point cloud data where a segmentation result is associated with captured point cloud data. In point cloud data 701, an obstacle 711 is superimposed onto the road surface and is considered part of the road. Part of the road is also recognized as a vehicle (a vehicle ahead) 712.

[0042] In the obstacle detection device 102 of the present embodiment, the point cloud data processing unit 212 extracts the road and the white line as road surface candidates from the point cloud data associated with the segmentation result in order to extract point cloud data of the road surface candidate 702.

[0043] Then, from the point cloud shape of the point cloud data of the road surface candidate, a road surface model representing the shape of the road surface is estimated, and the road surface point cloud data are extracted again from the point cloud data 701 based on the estimated road surface model to generate newly extracted road surface point cloud data 703.

[0044] Next, the newly extracted road surface point cloud data 703 are removed from the point cloud data 701, and point cloud data 704 remaining after the removal of the road surface is generated. In the point cloud data 704 remaining after the removal of the road surface, point clouds of the obstacles 711 and the preceding vehicle 712 remain as obstacle candidates.

[0045] Here, the remaining point cloud data 704 after the removal of the road surface are clustered to specify object point cloud data, which represent a cluster corresponding to the predefined object (ahead vehicle 712). The point cloud data of the ahead vehicle 712, which constitutes the object point cloud data, are then removed from the remaining point cloud data 704 after the removal of the road surface to generate point cloud data for post-removal processing. Only the point cloud data of the obstacle remains in the point cloud data for post-removal processing.

[0046] According to the obstacle detection device 102 of the present embodiment, even if a positional deviation occurs between the imaging device 111 and the distance measuring device 112, a recorded image can be accurately assigned to point cloud data, and various types of obstacles on the road surface can be accurately detected.

[0047] Therefore, the vehicle control unit 101 can determine whether an evasive maneuver is necessary and can perform control by means of the steering control unit 103 and the brake control unit 104.

[0048] The Fig. 8 to Fig. 11 are functional block diagrams of other examples of the obstacle detection device according to the present embodiment.

[0049] The in Fig. The example shown in Figure 8 is characterized by the inclusion of an attribute addition unit 213 between the point cloud data acquisition unit 211 and the point cloud data processing unit 212. The attribute addition unit 213 can define an attribute for each distance measurement point in the point cloud data or can add an identifier to the point cloud data that specifies the attribute (a road, a white line, etc.) of the point cloud data. The attribute links a segmentation result to the point cloud, and, for example, a color can be added as an attribute.

[0050] The in Fig. Example 9 is characterized by the inclusion of an image determination unit 222, which obtains the point cloud data of an obstacle detection result from the obstacle detection unit 221 and reprojects the point cloud data onto an image of a segmentation result to perform image determination. The image determination unit 222 determines whether the point cloud data of the obstacle detection result is reprojected onto the road surface area of ​​the segmentation result. Since a road surface runs at a large distance almost parallel to the direction of the LiDAR emission, the point cloud data corresponding to the road surface are not captured. Because the obstacle detection unit 221 detects an obstacle based on point cloud data, it cannot be determined whether the obstacle is present in the road surface area of ​​the detection result.By providing the image determination unit 222, an obstacle in the road surface area can be reliably located.

[0051] The example in Fig. The device 10 is characterized in that a point cloud data integration unit 216 is provided between the point cloud data acquisition unit 211 and the point cloud data processing unit 212, and that a point cloud data integration result storage unit 207 is provided, which accumulates integrated point cloud data that are the result of the integration of point cloud data. The point cloud data integration can be carried out, for example, based on a technique known as simultaneous localization and mapping (SLAM).

[0052] The point cloud data integration unit 216 integrates point cloud data for multiple frames and collects and stores the integrated point cloud data in the point cloud data integration result storage unit 207. The point cloud data processing unit 212 then performs point cloud data processing using the integrated point cloud data resulting from the point cloud data integration. As this example illustrates, the amount of point cloud data increases, enabling a clear view over long distances and improving detection performance.

[0053] The in Fig.The example shown in Figure 11 is characterized in that a vehicle state is detected by a vehicle sensor 113 and used for obstacle detection. The obstacle detection device 102 includes a vehicle state detection unit 231 and a processing control unit 232. The vehicle state detection unit 231 detects vehicle state information from the vehicle sensor 113.

[0054] The vehicle sensor 113 detects the vehicle speed and the engine operating state of the vehicle.

[0055] The processing control unit 232 provides vehicle status information to the image processing unit 202 and the point cloud data processing unit 212. For example, if the vehicle is stationary, there is no need to perform obstacle detection, and therefore the processing control unit 232 sends a command to the image processing unit 202 and the point cloud data processing unit 212 to stop processing. As a result, the processing of the image processing unit 202 and the point cloud data processing unit 212 is stopped, and the processing load of the ECU is reduced so that the unused processing resources can be used for other controls to improve processing speed.

[0056] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments mentioned above, and various design changes can be made without departing from the concept of the present invention as described in the claims. For example, the embodiments described above have been described in detail to facilitate understanding of the present invention, but they are not necessarily limited to those that have all the configurations described. Furthermore, part of the configuration of one embodiment can be replaced by a configuration of another embodiment, and a configuration of one embodiment can be added to the configuration of another embodiment.Furthermore, additions, deletions and replacements with other configurations can be made for part of the configuration of the embodiments. Reference symbol list 101 Vehicle control unit 102 Obstacle detection device 111 Imaging Unit 112 Distance measuring device 201 Image Acquisition Unit 202 Image processing unit 211 Point Cloud Data Acquisition Unit 212 Point Cloud Data Processing Unit 221 Obstacle detection unit 501 vehicle 502 Obstacle 503 preceding vehicle 504 Street 505 white line 601 recorded image 611 point cloud data 701 point cloud data 702 point cloud data sets of road surface candidates 704 point cloud data sets sorted by distance from the road surface 711 Obstacle 712 Vehicle (vehicle ahead) QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2022-006716 A

[0003]

Claims

[1] Obstacle detection device which includes: an image capture unit that captures an image taken by an imaging device; a point cloud data acquisition unit that captures point cloud data measured by a distance measuring device; an image processing unit that recognizes a road surface and a predefined object based on the image; a point cloud data processing unit that generates post-processed point cloud data from the point cloud data by removing road surface point cloud data corresponding to the road surface and object point cloud data corresponding to the specified object; and an obstacle detection unit that detects an obstacle based on the post-processed point cloud data, wherein the point cloud data processing unit a road surface model that represents a shape of the road surface, based on the point cloud data and a result of the road surface recognition by the image processing unit, the road surface point cloud data specified from the point cloud data based on the road surface model, and performs clustering of the remaining point cloud data after the removal of the road surface, in which the road surface point cloud data has been removed from the point cloud data, in order to specify the object point cloud data. [2] Obstacle detection device according to claim 1, wherein the point cloud data processing unit has a permissible range of error of the defines the road surface model according to the distance information contained in the point cloud data. [3] Obstacle detection device according to claim 1, further comprising a unit for determining obstacles on the road surface, which determines whether an obstacle detected by the obstacle detection unit is present on the road surface. [4] Obstacle detection device according to claim 1, wherein The image processing unit performs segmentation processing on the image with reference to a road surface area model and an object model, and The point cloud data processing unit extracts point cloud data from a road surface candidate using a segmentation result from the segmentation processing that is associated with the point cloud data acquired by the point cloud data acquisition unit; estimates the road surface model from a point cloud shape of the point cloud data of the road surface candidate; re-extracts the road surface point cloud data from the acquired point cloud data based on the road surface model to generate re-extracted road surface point cloud data; removes the re-extracted road surface point cloud data from the acquired point cloud data to generate the point cloud data remaining after road surface removal; and performs clustering of the point cloud data remaining after road surface removal to create object point cloud data that represent a cluster corresponding to the specified object.to specify, and removes the object point cloud data from the point cloud data remaining after the removal of the road surface in order to generate the post-processed point cloud data. [5] Obstacle detection device according to claim 1, wherein the point cloud data processing unit generates the post-processed point cloud data by removing the road surface point cloud data and the object point cloud data from integrated point cloud data in which the point cloud data for multiple frames are integrated and accumulated. [6] Obstacle detection device according to claim 2, wherein the point cloud data processing unit reduces a permissible range of error of the road surface model for a greater distance to the point cloud.

Images