Environment recognition device

Abstract

To provide an environment recognition device capable of accurately detecting a road surface having a gradient and determining a division size to improve estimation performance at a remote location where a point group that can be acquired is sparse.SOLUTION: An environment recognition device includes: an area setting information determination unit 134 that determines division area setting information 136 as setting information for division areas in dividing a road surface candidate area into the division areas composed of a plurality of flat surface areas; a road surface model estimation unit 131 that estimates a road surface model indicating a shape of a road surface for each division area on the basis of road surface candidate point group data; and a road surface detection unit 132 that outputs a road surface detection result 133 on the basis of the road surface model. The area setting information determination unit 134 determines the division area setting information 136 on the basis of the road surface detection result 133 by the road surface detection unit 132 and correct answer value data 135 of the road surface. The road surface model estimation unit 131 estimates a road surface model indicating the shape of the road surface for each division area specified by the determined division area setting information 136.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an external environment recognition device. [Background technology]

[0002] Advanced Driver Assistance System (ADAS) / Autonomous Driving (AD) technologies are being developed to provide driver assistance / vehicle control by utilizing sensing information around the vehicle. ADAS / AD technologies require long-distance obstacle detection to avoid danger early and appropriately control driving speed, steering, and obstacle detection. To remove road surface information from sensor information acquired during obstacle detection, it is essential to detect various road surface shapes, including gradient roads, at a distance. Patent Document 1 discloses a road gradient estimation device that calculates the three-dimensional position of a road contact point detected from a captured image using distance information from a distance measuring device and estimates the road plane using this three-dimensional position of the road contact point, thereby estimating the gradient of the road ahead even when there are no specific objects, such as a preceding vehicle or lane boundary lines. Patent Document 2 discloses a point cloud data division method that analyzes road characteristics by dividing large point cloud data acquired about structures using a measuring device into segments of a size suitable for analysis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-225806 [Patent Document 2] Japanese Patent Application Publication No. 2018-141758 Summary of the Invention [Problem to be solved by the invention]

[0004] As in Patent Document 1, there is a road surface detection technology that uses road surface point cloud data acquired by a distance measuring device, but because the road surface model is estimated assuming a flat road, it has the problem that it is difficult to detect road surfaces with slopes.

[0005] As in Patent Document 2, there is a technology for dividing a plane to detect a road surface with a slope. However, the method for determining the division size is not obvious, and if the division size is set improperly, it can result in erroneous detection of obstacles as road surfaces or failure to detect the road surface. Known technologies such as RANSAC (Random Sample Consensus) are used as plane estimation algorithms for road surface point cloud data. For example, in road surface detection using RANSAC, the plane with the most point cloud data present near the plane is output from among multiple estimated road surface plane candidates. Therefore, the estimation result is significantly affected by the dominant point cloud data. For example, if the division size is too small, obstacle point cloud data will dominate within the divided region, resulting in a plane including the obstacle point cloud being estimated and detected as the road surface. If the division size is too large, road surface detection is performed assuming a plane within the divided region, resulting in only a flat road surface or only a portion of a sloped road surface where the road surface point cloud data is dominant being detected as the road surface.

[0006] The present invention has been made in consideration of the above points, and an object of the present invention is to provide an external environment recognition device that can accurately detect a road surface with a slope and can determine a division size that improves estimation performance at a distance where the obtainable point cloud becomes sparse. [Means for solving the problem]

[0007] The external environment recognition device of the present invention that solves the above-mentioned problems comprises: an image acquisition unit that acquires an image captured by an imaging device; a point cloud data acquisition unit that acquires point cloud data measured by a ranging device; an image processing unit that identifies a road surface candidate area where a road surface is imaged based on the image; a point cloud data processing unit that extracts road surface candidate point cloud data corresponding to the road surface candidate area from the point cloud data; an area setting information determination unit that determines divided area setting information that is setting information for the divided areas when dividing the road surface candidate area into divided areas consisting of a plurality of planar areas; a road surface model estimation unit that estimates a road surface model that indicates the shape of the road surface for each divided area based on the road surface candidate point cloud data; and a road surface detection unit that outputs road surface detection results based on the road surface model, wherein the area setting information determination unit determines the divided area setting information based on the road surface detection result by the road surface detection unit and ground truth value data for the road surface, and the road surface model estimation unit estimates a road surface model that indicates the shape of the road surface for each divided area defined by the determined divided area setting information. [Effects of the Invention]

[0008] According to the present invention, by appropriately determining the size of the divided areas, it is possible to obtain an external environment recognition device that can accurately detect road surfaces at close range and on flat road surfaces, as well as road surfaces at long distances where the acquired point cloud becomes sparse, and road surfaces with slopes.

[0009] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a functional block diagram of the external environment recognition device according to the present embodiment. [Figure 2] FIG. 10 is a diagram showing an example of divided area setting information. [Figure 3] 4 is a flowchart showing a road surface model estimation method performed by the external environment recognition device of the present embodiment. [Figure 4]10 is a flowchart showing a method for determining divided area setting information by the external environment recognition device of this embodiment. [Figure 5] FIG. 10 is a functional block diagram of another example of the external environment recognition device of the present embodiment. [Figure 6] FIG. 10 is a functional block diagram of another example of the external environment recognition device of the present embodiment. [Figure 7] FIG. 10 is a functional block diagram of another example of the external environment recognition device of the present embodiment. [Figure 8] FIG. 10 is a functional block diagram of another example of the external environment recognition device of the present embodiment. [Figure 9] 10 is a flowchart showing another road surface model estimation method performed by the external environment recognition device of the present embodiment. [Figure 10] Schematic diagram showing a specific scene of external world recognition. [Figure 11A] A schematic diagram showing a specific scene of external world recognition where the road surface branches off ahead. [Figure 11B] 1 is a schematic diagram showing a specific scene of external world recognition in which the road surface has a lateral gradient; [Figure 12A] FIG. 2 is a diagram showing an example of a captured image of the area ahead of the vehicle. [Figure 12B] FIG. 2 is a diagram showing an example of point cloud data of a front view of a vehicle. [Figure 13] FIG. 10 is a functional block diagram of another example of the external environment recognition device of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Next, an embodiment of the external environment recognition device of the present invention will be described with reference to the drawings. Note that the same components in each drawing will be assigned the same reference numerals, and duplicated descriptions may be omitted as appropriate.

[0012] FIG. 1 is a functional block diagram of the external environment recognition device according to this embodiment.

[0013] The external environment recognition device 101 has an image acquisition unit 111, an image processing unit 112, a road surface area model 113, an object model 114, a segmentation result 115, camera parameters 116, a point cloud data acquisition unit 121, a point cloud data processing unit 122, a road surface model estimation unit 131, a road surface detection unit 132, a road surface detection result 133, an area setting information determination unit 134, correct value data 135, and divided area setting information 136.

[0014] The external environment recognition device 101 detects the road surface ahead of the vehicle based on information acquired from the imaging device 102 and the distance measuring device 103 .

[0015] The imaging device 102 is, for example, a monocular camera that is installed on the vehicle to capture images of the area ahead of the vehicle and generates an image of that image capture range. The ranging device 103 is, for example, a LiDAR (Laser Imaging Detection and Ranging) that is installed on the vehicle so that the imaging range of the imaging device 102 and the ranging range overlap, emitting outgoing waves in front of the vehicle and receiving reflected waves from objects to determine ranging points. The imaging device 102 and ranging device 103 each capture images and measure at a predetermined frame rate. The ranging device 103 may be, for example, a stereo camera that can acquire point cloud data from depth information.

[0016] The external environment recognition device 101 is configured as a computer equipped with an ECU (Electronic Control Unit) having a CPU, memory, input / output units, etc., and embodies a road surface model estimation unit 131, a road surface detection unit 132, and an area setting information determination unit 134 by executing a software program stored in the memory with the CPU.

[0017] The image acquisition unit 111 acquires an image captured by the imaging device (camera) 102. The image processing unit 112 identifies a road surface candidate area where the road surface is captured based on the captured image. The image processing unit 112 classifies the captured image into pixel areas of roads and white lines using semantic segmentation, and also recognizes predetermined objects such as vehicles, people, bicycles, and motorcycles as rectangular areas using object detection. Semantic segmentation / object detection can be performed based on a model learned by machine learning or deep learning, for example.

[0018] The road surface region model 113 corresponds to semantic segmentation, and the object model 114 corresponds to object detection. In the image processing unit 112 of this embodiment, pixel regions of roads and white lines are classified by semantic segmentation, and object detection is performed by recognizing predetermined objects as rectangular regions through image processing, but the method is not limited to these. For example, predetermined objects such as vehicles may be recognized as pixel regions by semantic segmentation. Hereinafter, segmentation processing includes semantic segmentation / object detection, and segmentation result 115 includes semantic segmentation results / object detection results.

[0019] The image processing unit 112 executes segmentation processing by referring to the road surface area model 113 and the object model 114. Information on the segmentation results by the image processing unit 112 is stored in the segmentation result 115. The information on the segmentation results is stored for each frame. The camera parameters 116 store external parameters (six degrees of freedom corresponding to position and rotation) that indicate the relative positional relationship between the image capturing device 102 and the distance measuring device 103, and internal parameters such as the focal length, image resolution, and distortion coefficient of the image capturing device 102.

[0020] The point cloud data acquisition unit 121 acquires point cloud data measured by the distance measurement device 103. The point cloud data consists of three-dimensional data of a plurality of distance measurement points. The point cloud data processing unit 122 extracts road surface candidate point cloud data by associating the segmentation result 115 acquired by the image processing unit 112 through segmentation processing with the point cloud data acquired by the point cloud data acquisition unit 121. Here, the point cloud data processing unit 122 extracts roads and white lines as road surface candidates from the point cloud data associated with the segmentation result 115, and generates road surface candidate point cloud data corresponding to the road surface candidate area.

[0021] The road surface model estimation unit 131 estimates a road surface model based on the road surface candidate point cloud data. In order to detect a road surface having a gradient that cannot be approximated by a single plane, the road surface model estimation unit 131 divides the road surface candidate area into divided areas consisting of a plurality of small areas (planar areas) based on divided area setting information 136 determined by the area setting information determination unit 134, performs road surface model estimation by plane estimation on the road surface candidate point cloud data present in each divided area, and estimates a road surface model that indicates the shape of the road surface for each divided area.

[0022] The road surface detection unit 132 creates a road surface detection result 133 from road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit 121 based on the road surface model estimated by the road surface model estimation unit 131, and point cloud data from which the road surface point cloud data re-extracted from the acquired point cloud data has been removed (point cloud data after road surface removal).

[0023] The area setting information determination unit 134 compares the road surface detection result 133 with correct value data 135 consisting of correct road surface points, and determines divided area setting information 136, which is setting information for the divided areas when dividing the road surface candidate area into divided areas consisting of multiple flat areas. The divided area setting information 136 is information that defines the size of the divided areas when dividing the road surface candidate area into divided areas consisting of multiple flat areas. For example, the size of the divided areas can be set to 2 m, 5 m, 10 m, etc., but the method for defining the size of the divided areas is not limited to this.

[0024] 2, the divided area setting information 136 does not have to be the size of the divided area, but may be, for example, the shape of the divided area. For example, the divided area may be a sector divided radially with the distance measuring device 103 at the center. This makes it possible to make the point cloud density in each divided area uniform, enabling highly accurate road surface detection.

[0025] FIG. 3 is a flowchart showing a road surface model estimation method performed by the road surface model estimation unit 131 of the external environment recognition device of this embodiment.

[0026] The road surface model estimation unit 131 acquires the divided area setting information 136 from the area setting information determination unit 134 .

[0027] The road surface model estimation unit 131 divides the road surface candidate point cloud data (corresponding to the road surface candidate area) acquired from the point cloud data processing unit 122 into multiple areas based on the divided area setting information 136 (S301). The road surface, especially on a road with a slope, is not made up of only one plane, so it is appropriate to represent it by a combination of several planes.

[0028] Then, for each area divided in S301, a road surface model is estimated based on the road surface candidate point cloud data contained in each divided area. The road surface model is estimated from the point cloud shape using a known technique such as plane estimation using RANSAC (S302).

[0029] FIG. 4 is a flowchart showing a method for determining the divided area setting information 136 by the area setting information determination unit 134 of the external environment recognition device according to this embodiment.

[0030] When setting the divided area setting information 136, the area setting information determination unit 134 selects and sets one piece of setting candidate information 502 (FIG. 5) as the divided area setting information 136 (S401). The road surface model estimation unit 131 estimates a road surface model for each area divided based on the selected divided area setting information 136, based on the road surface candidate point cloud data included in each divided area (S402). Based on the set divided area setting information 136, the road surface detection unit 132 outputs a road surface detection result 133 (S403), and the area setting information determination unit 134 compares the output road surface detection result 133 with the correct value data 135 (S404). Steps S401 to S404 are repeated multiple times, and the area setting information determination unit 134 determines the setting candidate information 502 (FIG. 5) relating to the divided area that has the highest degree of agreement with the correct value data 135 among the divided areas defined by the multiple setting candidate information as the final divided area setting information 136 (S405).

[0031] Here, the region setting information determination unit 134 may set the divided region setting information 136 before the vehicle starts traveling.

[0032] The degree of agreement between the road surface detection result 133 and the correct value data 135 is calculated using known techniques such as accuracy rate or F value. The accuracy rate is calculated using the re-extracted road surface point cloud data of the road surface detection result 133 and the correct value data 135. The F value is calculated using the road surface detection result 133 and the correct value data 135.

[0033] Regardless of the road surface shape, if the road surface is not detected or is incorrectly detected, the degree of match decreases, but if the road surface is detected accurately, the degree of match increases. Therefore, according to the external environment recognition device of this embodiment, for example, for a road surface with a large gradient, the divided area setting information 136 is determined to set the size of the divided areas to be small. This makes it possible to detect the road surface with high accuracy, not only on flat roads, but also regardless of the magnitude of the gradient.

[0034] A characteristic feature of the external environment recognition device 101 shown in FIG. 5 is that it includes a setting candidate information storage unit 501 connected to the area setting information determination unit 134. The setting candidate information storage unit 501 stores a plurality of setting candidate information 502, and when the area setting information determination unit 134 sets the divided area setting information 136, the setting candidate information storage unit 501 sequentially sets each of the plurality of setting candidate information 502 as the divided area setting information 136. The setting candidate information 502 may set the size of the divided area to, for example, 2 m, 5 m, or 10 m. The upper and lower limit values ​​of the setting candidate information 502 are set according to the resolution of the point cloud data acquired by the ranging device 103, etc. This makes it possible to appropriately detect the road surface regardless of the resolution of the ranging device 103.

[0035] Here, the setting candidate information 502 does not need to be stored in advance, as in the setting candidate information storage unit 501, and appropriate candidates may be set as appropriate when determining the divided area setting information 136. A known technique such as a binary search method or a Newton method is used to set the candidates. When using the binary search method, the upper and lower limits and the search termination condition are set in advance depending on the resolution of the point cloud data acquired by the distance measuring device 103. When using the Newton method, the accuracy rate and the F-value are set as the objective function.

[0036] A characteristic feature of the external environment recognition device 101 shown in FIG. 6 is that annotated road surface point cloud data 601 extracted visually from point cloud data is provided as the correct value data 135. This road surface point cloud data used as the correct value data 135 is extracted in advance. The annotated road surface point cloud data 601 is created by extracting road surface point cloud data from point cloud data using, for example, an editing tool and saving it as a file. The region setting information determination unit 134 determines, as the divided region setting information 136, setting candidate information for a divided region that has the highest degree of match with the correct value data 135 (annotated road surface point cloud data 601) from the re-extracted road surface point cloud data, among the divided regions defined by multiple pieces of setting candidate information. This correct value data makes it possible to create the correct value data 135 and determine the divided region setting information 136 in advance, thereby reducing the calculation load while the vehicle is traveling.

[0037] A characteristic feature of the external environment recognition device 101 shown in FIG. 7 is that point cloud data obtained by extracting an acquired point cloud corresponding to a pre-generated map 701 is provided as the correct value data 135. The pre-generated map 701 can be created based on a technology known as SLAM (Simultaneous Localization and Mapping), for example. The area setting information determination unit 134 determines, as the divided area setting information 136, setting candidate information related to a divided area that has the highest degree of match with the correct value data (pre-generated map 701) from the re-extracted road surface point cloud data, among the divided areas defined by a plurality of setting candidate information. This correct value data eliminates the need to manually create the correct value data 135, thereby enabling the correct value data 135 to be created in a short period of time. This correct value data can be updated to the latest correct value data 135 while driving using the cloud, for example.

[0038] A characteristic feature of the external environment recognition device 101 shown in Fig. 8 is that road surface candidate point cloud data indicating road surface candidate areas extracted by the point cloud data processing unit 122 is provided as the correct value data 135. The area setting information determination unit 134 determines, from among the divided areas defined by a plurality of pieces of setting candidate information, setting candidate information related to the divided area that has the highest degree of coincidence with correct value data (road surface candidate point cloud data) from the re-extracted road surface point cloud data, as the divided area setting information 136. Using this correct value data, it is possible to create the correct value data 135 while driving, and therefore it is possible to update the divided area setting information 136 to an appropriate value according to the scene during driving.

[0039] FIG. 9 is a flowchart showing another road surface model estimation method performed by the external environment recognition device of this embodiment.

[0040] A distinctive feature of the external environment recognition device 101 shown in FIG. 9 is the road surface model estimation method for each divided area in the road surface model estimation unit 131. The road surface model estimation unit 131 divides the road surface candidate point cloud data (corresponding to the road surface candidate area) acquired from the point cloud data processing unit 122 into multiple areas based on the divided area setting information 136 (S301). The road surface model estimation method of this embodiment first estimates a road surface model based on the road surface candidate point cloud data in the divided area closest to the vehicle where a point cloud can be acquired densely (S901). Thereafter, under the constraint that adjacent road surfaces are continuous, starting from the estimated divided area, road surface models for adjacent divided areas are estimated in order, and road surface models are sequentially estimated starting from the divided area on the vehicle side (S902).

[0041] FIG. 10 is a schematic diagram showing a specific scene of external world recognition.

[0042] As shown in FIG. 10, there is a vehicle 1001 and a sloped road 1002 ahead of the vehicle 1001. The divided area closest to the vehicle 1001 is set as s1, and the road surface model of divided area s1 is set as p1. The road surface model p1 is estimated using only the road surface candidate point cloud data 1003 within divided area s1 (S901). The road surface model p2 of the adjacent divided area s2 is estimated using the road surface candidate point cloud data 1005 within divided area s2 under the constraint that it passes through a straight line 1004 where the road surface model p1 intersects with the boundary between divided area s1 and divided area s2. The road surface models from p2 onwards are estimated in the same way as p2 (S902).

[0043] Typically, three-dimensional plane detection using point cloud data requires a large number of point clouds, but according to this embodiment, a road surface model is estimated with a continuity constraint added so that road surface models of adjacent divided areas are adjacent to each other. Therefore, the adjacent road surface models strongly constrain the road surface model to be estimated next, making it possible to estimate the road surface even in distant locations where the obtainable point clouds are sparse. In other words, road surface estimation is performed using a first road surface model for a first divided area near the vehicle 1001 and a second road surface model for a second divided area adjacent to the first divided area but farther from the vehicle 1001 than the first divided area. The first road surface model and the second road surface model are adjacent to each other at the boundary between the first divided area and the second divided area. Therefore, the adjacent road surface models strongly constrain the road surface model to be estimated next, making it possible to estimate the road surface even in distant locations where the obtainable point clouds are sparse.

[0044] Furthermore, if a large amount of non-road surface point cloud data exists within a divided region and the road surface point cloud data is not dominant, a planar model based on the non-road surface point cloud data is estimated, making it impossible to accurately estimate the road surface model. Such an inaccurate road surface model has a slope angle that deviates from the true road surface model. According to this embodiment, a road surface model is estimated with a continuity constraint added so that the road surface models of adjacent divided regions are in contact. Therefore, the adjacent road surface models strongly constrain the next estimated road surface model, thereby preventing erroneous estimation of the road surface model, such as determining non-road surface point cloud data as the road surface.

[0045] 11A and 11B are schematic diagrams showing a specific scene of external world recognition in which the road surface branches ahead, and in which the road surface has a lateral gradient.

[0046] 11A, a vehicle 1001 is traveling on a road 1102 before the road branches into an inclined road 1002 and a flat road 1101. According to the external environment recognition device of this embodiment, if the area setting information determination unit 134 divides the road surface candidate area in the lateral direction of the vehicle and determines the divided area setting information 136 so that the inclined road 1002 and the flat road 1101 belong to different divided areas, the road surface can be detected with high accuracy even in such a scene.

[0047] The external environment recognition device of this embodiment can also detect a road surface 1103 that has a gradient in the lateral direction relative to the vehicle. Generally, as shown in Fig. 11B, a road surface has a gradient in the lateral direction, with the center of the road surface rising, from the viewpoint of drainage. If the region setting information determination unit 134 determines the divided region setting information 136 so as to divide the road surface candidate region in the lateral direction, it is also possible to detect a road surface that has a gradient in the lateral direction.

[0048] The divided area setting information 136 may set the size and shape of each divided area independently (for each divided area), for example, according to the density of the acquired point cloud data. For example, the size may be set large for areas with low point cloud data density, and small for areas with high point cloud data density. This makes it possible to accurately detect road surfaces even on roads where flat roads and sloped roads coexist.

[0049] 12A and 12B are diagrams showing an example of a captured image of the area ahead of the vehicle and point cloud data, respectively.

[0050] FIG. 13 is a functional block diagram of another example of the external environment recognition device of this embodiment.

[0051] A characteristic feature of the external environment recognition device 101 shown in FIG. 13 is that an obstacle detection unit 1301 is provided.

[0052] The road surface detection unit 132 removes road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit 121 from the road surface detection result 133 (road surface-removed point cloud data), which is used as obstacle candidate point cloud data. The obstacle detection unit 1301 then performs clustering on the obstacle candidate point cloud data, for example, based on Euclidean distance, and re-determines whether or not the obstacle is an obstacle depending on the size and distance from the estimated road surface model, thereby identifying the obstacle point cloud data. This enables robust obstacle detection with respect to the segmentation result 115, time synchronization between the image capture device 102 and the distance measurement device 103, and the accuracy of external parameters.

[0053] As described above, the external environment recognition device 101 of this embodiment includes an image acquisition unit 111 that acquires an image captured by the imaging device 102, a point cloud data acquisition unit 121 that acquires point cloud data measured by the distance measuring device 103, an image processing unit 112 that specifies a road surface candidate area where a road surface is captured based on the image, a point cloud data processing unit 122 that extracts road surface candidate point cloud data corresponding to the road surface candidate area from the point cloud data, and an area setting information decision unit that decides divided area setting information 136 that is setting information for the divided areas when dividing the road surface candidate area into divided areas each consisting of a plurality of planar areas. a road surface model estimation unit 131 that estimates a road surface model that indicates the shape of the road surface for each divided area based on the road surface candidate point cloud data, and a road surface detection unit 132 that outputs a road surface detection result 133 based on the road surface model, wherein the area setting information determination unit 134 determines the divided area setting information 136 based on the road surface detection result 133 by the road surface detection unit 132 and ground truth value data 135 of the road surface, and the road surface model estimation unit 131 estimates a road surface model that indicates the shape of the road surface for each divided area specified by the determined divided area setting information 136.

[0054] The external environment recognition device 101 of this embodiment includes a setting candidate information storage unit 501 that stores a plurality of setting candidate information 502 that are set as the divided area setting information (FIG. 5).

[0055] In the external environment recognition device 101 of this embodiment, the area setting information determination unit 134 determines, as the divided area setting information 136, the setting candidate information 502 relating to the divided area in which the degree of agreement between the road surface detection result 133 by the road surface detection unit 132 and the ground truth value data 135 of the road surface is greatest, from among the divided areas defined by a plurality of setting candidate information 502.

[0056] In the external environment recognition device 101 of this embodiment, the divided area setting information 136 is information that defines the size of the divided area.

[0057] In the external environment recognition device 101 of this embodiment, the road surface detection unit 132 outputs the road surface detection result 133 including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit 121 based on the road surface model estimated by the road surface model estimation unit 131, the ground truth data 135 of the area setting information determination unit 134 is road surface point cloud data extracted visually from the point cloud data, and the area setting information determination unit 134 determines, as the divided area setting information 136, the setting candidate information 502 relating to the divided area in which the degree of coincidence of the re-extracted road surface point cloud data with the ground truth data 135 is greatest among the divided areas defined by a plurality of setting candidate information 502 (FIG. 6).

[0058] In the external environment recognition device 101 of this embodiment, the road surface detection unit 132 outputs the road surface detection result 133 including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit 121 based on the road surface model estimated by the road surface model estimation unit 131, the ground truth data 135 of the area setting information determination unit 134 is point cloud data obtained by extracting an acquired point cloud corresponding to a map generated in advance, and the area setting information determination unit 134 determines, as the divided area setting information 136, the setting candidate information 502 relating to the divided area in which the degree of coincidence of the re-extracted road surface point cloud data with the ground truth data 135 is greatest among the divided areas defined by a plurality of setting candidate information 502 (FIG. 7).

[0059] In the external environment recognition device 101 of this embodiment, the road surface detection unit 132 outputs the road surface detection result 133 including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit 121 based on the road surface model estimated by the road surface model estimation unit 131, the ground truth data 135 of the area setting information determination unit 134 is road surface candidate point cloud data indicating a road surface candidate area extracted by the point cloud data processing unit 122, and the area setting information determination unit 134 determines, as the divided area setting information 136, the setting candidate information 502 related to the divided area in which the degree of coincidence of the re-extracted road surface point cloud data with the ground truth data 135 is greatest among the divided areas defined by a plurality of setting candidate information 502 (FIG. 8).

[0060] In the external environment recognition device 101 of this embodiment, the road surface model estimation unit 131 divides the road surface candidate area into a plurality of divided areas, and estimates a road surface model with a continuity constraint added such that the road surface models of adjacent divided areas are in contact with each other (FIGS. 9 and 10).

[0061] The external environment recognition device 101 of this embodiment includes an obstacle detection unit 1301 that identifies road surface point cloud data from the point cloud data based on the road surface model, and identifies obstacle point cloud data by clustering road surface-removed point cloud data in which the road surface point cloud data has been removed from the point cloud data ( FIG. 13 ).

[0062] According to this embodiment, by appropriately determining the size of the divided regions, it is possible to obtain an external environment recognition device 101 that can accurately detect road surfaces at a distance where the acquired point cloud is sparse, as well as flat road surfaces at close range, and road surfaces with slopes.

[0063] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various design modifications can be made without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0064] 101...External world recognition device 102 Imaging device 103... Distance measuring device 111...Image acquisition unit 112 Image processing unit 113···Road surface area model 114···Object Model 115···Segmentation results 116···Camera parameters 121 Point cloud data acquisition unit 122 Point cloud data processing section 131...Road surface model estimation part 132 Road surface detection unit 133···Road surface detection results 134 Area setting information determination unit 135 Correct answer data 136...Divided area setting information 501: Setting candidate information storage unit 502... Setting candidate information 601 Annotated road surface point cloud data 701... Pre-generated maps 1001... Vehicle 1002... Inclined Road 1003: Road surface candidate point cloud data within the divided area s1 1004: Line where road surface model p1 intersects with the boundary between divided area s1 and divided area s2 1005: Road surface candidate point cloud data within divided area s2 1101...Flat road 1102... The road before it splits into a sloped road and a flat road 1103 Road surface with a lateral slope relative to the vehicle 1301 Obstacle detection unit

Claims

1. an image acquisition unit that acquires an image captured by an imaging device; a point cloud data acquisition unit that acquires point cloud data measured by a distance measuring device; an image processing unit that identifies a road surface candidate area in which the road surface is captured based on the image; a point cloud data processing unit that extracts road surface candidate point cloud data corresponding to the road surface candidate area from the point cloud data; an area setting information determination unit that determines divided area setting information, which is setting information for the divided areas when dividing the road surface candidate area into divided areas each consisting of a plurality of planar areas; a road surface model estimating unit that estimates a road surface model indicating a shape of the road surface for each of the divided areas based on the road surface candidate point cloud data; a road surface detection unit that outputs a road surface detection result based on the road surface model, the region setting information determination unit determines the divided region setting information based on a road surface detection result by the road surface detection unit and correct value data of the road surface; The external environment recognition device, wherein the road surface model estimation unit estimates a road surface model indicating the shape of the road surface for each divided area defined by the determined divided area setting information.

2. The external environment recognition device according to claim 1 , further comprising a setting candidate information storage unit that stores a plurality of setting candidate information set as the divided area setting information.

3. The area setting information determination unit determines, as the divided area setting information, setting candidate information relating to a divided area in which the road surface detection result by the road surface detection unit and the correct value data of the road surface have the highest degree of agreement among the divided areas defined by a plurality of setting candidate information. The external environment recognition device according to claim 1, characterized in that

4. The external environment recognition device according to claim 1 , wherein the divided area setting information is information that defines the size of the divided area.

5. the road surface detection unit outputs the road surface detection result including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit based on the road surface model estimated by the road surface model estimation unit; The correct value data of the area setting information determination unit is road surface point cloud data extracted visually from the point cloud data, and the area setting information determination unit, among the divided areas defined by a plurality of setting candidate information, determines setting candidate information related to the divided area in which the degree of match of the correct value data in the re-extracted road surface point cloud data is greatest as the divided area setting information. The external environment recognition device according to claim 1, characterized in that

6. the road surface detection unit outputs the road surface detection result including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit based on the road surface model estimated by the road surface model estimation unit; 2. The external environment recognition device according to claim 1, wherein the correct value data of the area setting information determination unit is point cloud data obtained by extracting an acquired point cloud corresponding to a map generated in advance, and the area setting information determination unit determines, as the divided area setting information, setting candidate information related to a divided area in which the degree of coincidence of the correct value data in the re-extracted road surface point cloud data is greatest among the divided areas defined by a plurality of setting candidate information.

7. the road surface detection unit outputs the road surface detection result including road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit based on the road surface model estimated by the road surface model estimation unit; The correct value data of the area setting information determination unit is road surface candidate point cloud data indicating the road surface candidate area extracted by the point cloud data processing unit, and the area setting information determination unit determines, as the divided area setting information, setting candidate information related to the divided area in which the degree of agreement of the correct value data in the re-extracted road surface point cloud data is greatest, among the divided areas defined by a plurality of setting candidate information. The external environment recognition device according to claim 1, characterized in that

8. The external environment recognition device according to claim 1, wherein the road surface model estimation unit divides the road surface candidate area into a plurality of divided areas and estimates a road surface model to which a continuity constraint is added such that road surface models of adjacent divided areas are in contact.

9. Identifying road surface point cloud data from the point cloud data based on the road surface model; The external environment recognition device according to claim 1, further comprising an obstacle detection unit that identifies obstacle point cloud data by clustering road surface removed point cloud data in which the road surface point cloud data is removed from the point cloud data.

10. The external environment recognition device according to claim 1, characterized in that the road surface detection unit creates the road surface detection result from road surface point cloud data re-extracted from the point cloud data acquired by the point cloud data acquisition unit based on the road surface model estimated by the road surface model estimation unit, and point cloud data obtained by removing the re-extracted road surface point cloud data from the point cloud data.

11. The external environment recognition device according to claim 1 , wherein the divided area setting information is set based on a resolution or density of the point cloud data.

Images