External recognition device

Abstract

Provided is an external environment recognition device capable of accurately detecting left and right road edges. This external environment recognition device: obtains road edge feature points corresponding to road edges from sensing results output from a vehicle-mounted sensing device for detecting information outside the vehicle; calculates, for each road edge feature point, a line representing the direction of the road edge constituted by the road edge feature point (e.g., a normal to the road edge); groups the road edge feature points into a right road edge group and a left road edge group according to the direction of each line; and identifies each road edge on the basis of the road edge feature points grouped in the same　group corresponding to the road edge, thereby making it possible to accurately detect left and right road edges.

Description

Technical Field

[0001] The present invention relates to an external recognition device mounted on a vehicle or the like.

Background Art

[0002] An external recognition device is a device that recognizes the external environment (the environment around the vehicle) by utilizing sensors mounted on the vehicle. In recent years, advanced driving assistance systems using external recognition devices have become widespread for preventing traffic accidents. In the front sensing of advanced driving assistance systems, there is a technology for detecting the end of the traveling road (hereinafter referred to as the road edge) in addition to detecting the lanes of the traveling road.

[0003] Specifically, in recent years, warning and control functions for suppressing lane departure, which apply the lane detection function of the traveling road, have become widespread. In such an advanced driving assistance system, the development of a road edge detection function for detecting the end of the drivable area (road edge) as a future technology is underway. As a background for the development of the road edge detection function, it can be cited that there are many single-vehicle accidents such as accidents deviating from the road edge in various countries including Japan, Europe, and the United States. In addition, on roads where lane detection cannot be performed due to reasons such as the absence of lane markings such as white lines, there is also a problem that warning and control functions for suppressing lane departure, which apply the lane detection function of the traveling road, cannot be used, and thus driving assistance cannot be received. Against such a background, the development of a function for suppressing not only lane departure but also road edge departure is required. In order to realize such a road edge departure suppression function, it is necessary to develop an external recognition device that enables detection of the road edge portion.

[0004] In the above-described road edge detection technology, there is a technology for extracting road edge feature points corresponding to the road edge from the sensing results output from an in-vehicle sensing device that detects information outside the vehicle, and connecting the road edge feature points to form a road edge portion.

[0005] For example, as a technology for detecting the boundary of a stepped surface in a traveling road as a road edge portion by utilizing an in-vehicle camera as an in-vehicle sensing device, there is a technology described in Patent Document 1.

Prior Art Documents

[0006] [Patent Document 1] JP 2015-184900 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the conventional technology, when connecting the extracted road edge characteristic points, the characteristic point of the left road edge and the characteristic point of the right road edge are mistakenly regarded as the same road edge and connected together. For example, such a problem may occur when, on a left curve road with a large curvature, the right road edge intersects with the extension of the left road edge within the range that the on-board sensing device can sense.

[0008] The present invention has been made in consideration of the above problems, and an object of the present invention is to provide an external environment recognition device capable of accurately detecting left and right road edges. [Means for solving the problem]

[0009] In order to solve the above problems, the external environment recognition device according to the present invention has a processing unit that detects road ends, and the processing unit has a road end feature point extraction unit that determines road end feature points corresponding to road ends from sensing results output from an on-board sensing device that detects information outside the vehicle, a road end direction line calculation unit that calculates a line representing the direction of the road end constituted by the road end feature points for each of the road end feature points, and a road end identification unit that groups the road end feature points into a right road end group and a left road end group according to the direction of the line, and identifies the road end by the road end feature points grouped into the same group. Effect of the Invention

[0010] According to the present invention, in the process of road edge detection, a line indicating the direction of the road edge constituted by the road edge characteristic points is calculated for each road edge characteristic point, and the road edge characteristic points are grouped into right road edges and left road edges according to the direction of the calculated line, thereby enabling more accurate road edge detection. Furthermore, by accurately detecting the left and right road edges, it becomes possible to suppress erroneous control using road edge information (for example, erroneous control of the road edge departure suppression function).

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an external environment recognition device according to an embodiment of the present invention. [Diagram 2] 5 is a process flowchart of a road edge direction line calculation unit. [Diagram 3] Example of road edge normal calculation results (bird's-eye view). [Figure 4] Block diagram of the 3D roadside feature selection unit. [Diagram 5] FIG. 4 is a block diagram of a road edge direction line angle selection unit. [Figure 6] 13 is a process flowchart of a road edge direction line angle selection unit. [Figure 7] An example of left / right grouping according to the direction of the road edge direction line. [Figure 8] An example of grouping when the road edge direction line is oriented on the negative y-axis. [Figure 9] An example where the road edge direction line crosses the center of the image. [Figure 10] FIG. 4 is a block diagram of the roadside height direction sorting unit. [Figure 11] An example of road surface height detection and an example of road edge identification threshold calculation results. [Figure 12] An example of the road edge detection threshold calculation results for off-road and snowy roads. [Figure 13] FIG. 4 is a block diagram of the road edge depth direction sorting unit. [Figure 14] An example of road edge feature point selection based on depth direction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] FIG. 1 shows the overall configuration of an external environment recognition device according to an embodiment of the present invention.

[0015] The external environment recognition device 1 of this embodiment is mounted on a vehicle (the vehicle itself or the vehicle itself), extracts road edge feature points corresponding to road edges from the sensing results output from an on-board sensing device that detects information outside the vehicle, and detects road edges by connecting the extracted road edge feature points. The road edge is an edge of a drivable area including the vehicle's driving path (also referred to as the vehicle's travel path), and the road edge feature point is a feature point that represents the edge of the drivable area (details will be explained later). As shown in FIG. 1, the external environment recognition device 1 of this embodiment mainly includes a stereo camera unit 100 and a processing unit 900. The processing unit 900 includes a parallax generation unit 200, a road edge feature point extraction unit 300, a road edge direction line calculation unit 400, a road edge identification unit 600 having a road edge feature three-dimensional selection unit 500, a road edge detection determination unit 700, and an alarm control unit 800. Each unit of the external environment recognition device 1 will be described below.

[0016] (Stereo camera unit 100) The stereo camera unit 100 is equipped with an in-vehicle stereo camera. The in-vehicle stereo camera constitutes an in-vehicle sensing device that detects information outside the vehicle. The in-vehicle stereo camera acquires images of the outside of the vehicle using left and right cameras, and outputs the acquired (captured) images to the processing unit 900. In this embodiment, the stereo camera is described as an example of a front sensor, but the sensor itself is not limited to a stereo camera, and may be a single camera or Lidar, etc. Also, it may be a fusion sensor of a camera and Lidar.

[0017] (Processing section 900) The processing unit 900 detects the roadside from the sensing result (image in this embodiment) output from the stereo camera unit 100.

[0018] (Parallax generation unit 200) The parallax generating unit 200 identifies the positions on the images where the same object is captured from the images captured by the left and right cameras in the stereo camera unit 100, and generates a parallax image that represents the difference in the positions on the images. By generating a parallax image, it becomes possible to obtain three-dimensional position information of the surrounding environment, such as the road surface, roadside objects, and obstacles. In this embodiment, the stereo camera is described as an example of a front sensor, so the parallax generating unit 200 is provided. However, if the front sensor is a fusion sensor with Lidar or a camera, it is sufficient to use anything that can detect three-dimensional position information of the surrounding environment, such as a three-dimensional point cloud map.

[0019] (Road end characteristic point extraction unit 300) The road end feature point extraction unit 300 utilizes the principle of triangulation to generate a 3D point cloud of each three-dimensional position using the disparity image generated by the disparity generation unit 200. From the calculated 3D point cloud, feature points that are road end candidates (hereinafter also referred to as road end feature points or road end candidate feature points) are extracted.

[0020] The roadside feature point extraction unit 300 will be described in detail. The roadside feature point extraction unit 300 extracts feature points for three-dimensional objects with height, compared with the area corresponding to the road surface. Using the parallax image generated by the parallax generation unit 200, the horizontal direction of the image is the coordinate on the image, and the vertical direction of the image is the parallax value indicating the depth, and by utilizing the principle of triangulation, each three-dimensional position is generated as a 3D point cloud. On the generated image, a voting process is performed for each horizontal coordinate column, and feature points that are candidates for roadsides are extracted. The extracted roadside feature points hold three-dimensional position information on the image. The extracted roadside feature points can also be arranged on a bird's-eye view seen from above the vehicle, and can also be converted into a two-dimensional image.

[0021] The objects to be extracted as road edge feature points include not only typical road edges such as curbs and walls, but also moving objects such as oncoming vehicles and vehicles running parallel to the road, and low steps of about 5 cm. In this embodiment, road edges with height are mainly described as examples, but it is also possible to extract road edge feature points from negative road edges such as side gutters and rice paddy roads, and road edges without steps that are the same height as the road that is traveled, such as grass, gravel, and dirt without steps.

[0022] For road edges such as gutters and rice paddy roads, which are called negative road edges, it is possible to extract characteristic points of the road edge that are lower than the road surface by calculating the height distribution of the area equivalent to the road surface and the vertical height distribution in the same way as for the road edges with height described above.

[0023] Road edges without steps, such as grass or gravel, can be detected using a learning-based detection method that applies a learning algorithm. As an example of learning, an image with a road edge without a step is used to annotate the area corresponding to the road edge, and the image is then learned as a correct answer image. By learning in this way, it becomes possible to extract the feature points of the area corresponding to the road edge without a step, and these can be used in the same way as the feature points of other road edges.

[0024] In other words, the road end feature points extracted by the road end feature point extraction unit 300 in this embodiment include elevated road ends, negative road ends, and step-free road ends, and are feature points that are candidates for areas corresponding to the ends of the drivable area including the vehicle's lane.

[0025] (Roadside direction line calculation unit 400) The road end direction line calculation unit 400 uses the road end feature points extracted by the road end feature point extraction unit 300 to calculate a line representing the direction of the road end constituted by the road end feature points. In this embodiment, a normal line is described as an example of a line representing the road end direction, but the line representing the road end direction is not limited to a normal line, and may be any line that points toward the position or direction of the front sensor, or the direction of the road on which the vehicle is traveling. In other words, the line representing the road end direction calculated by the road end direction line calculation unit 400 in this embodiment is a line that can represent the relative direction (inclination) of the road end with respect to the vehicle (front sensor).

[0026] The road end direction line calculation unit 400 will be described in detail. The road end direction line calculation unit 400 calculates lines representing the directions of the left and right road ends. In this embodiment, the normal line is used as an example of the line representing the direction of the road end. Figure 2 shows a processing flowchart of the road end direction line calculation unit 400. The operation based on the processing flowchart in Figure 2 is as follows.

[0027] In step 401, the road end characteristic points extracted by the road end characteristic point extracting unit 300 are obtained.

[0028] In step 402, of the road end characteristic points acquired in step 401, a search is performed on the left and right sides to determine whether there are any road end characteristic points in the depth direction from the bottom to the top of the image.

[0029] In step 403, surface fitting processing is performed using a point cloud within a certain range centered on the position coordinates of the road end feature points detected in step 402 in the 3D point cloud image generated by the road end feature point extraction unit 300. In this embodiment, surface fitting processing performed using a 3D point cloud generated from three-dimensional position information is described as an example, but in a UD map or an overhead view of two-dimensional position information, it is also possible to perform line fitting from a point sequence of road end feature points.

[0030] In step 404, using the result calculated in step 403, a line in the normal direction of the road edge is calculated from the position of the road edge feature point. The normal line calculates only the direction in which the sensor is attached and does not calculate the normal line in the direction away from the sensor. An example of the calculation result of the road edge normal line is shown in FIG. 3. In this embodiment, the line representing the direction of the road edge is described by taking the normal line as an example, but the line representing the direction of the road edge is not limited to the normal line, and it may be a line in the position or direction where the sensor is attached, or a line toward the direction of the host vehicle's traveling road.

[0031] By the processes of step 403 and step 404, at the road edge feature point detected in step 402, a line representing the direction of the road edge (the road edge normal line in this embodiment) is obtained from the positional relationship with the road edge feature points existing around the road edge feature point.

[0032] In step 405, it is determined whether there are no other acquired road edge feature points. If there are still road edge feature points, the process returns to step 402, and the processes from step 402 to step 404 are repeated. If there are no other road edge feature points, the process ends.

[0033] (Road edge identification unit 600) The road edge identification unit 600 uses the road edge feature three-dimensional selection unit 500 to identify the road edge.

[0034] (Road edge feature three-dimensional selection unit 500) The road edge feature three-dimensional selection unit 500 uses the line representing the direction of the road edge (which may be referred to as the road edge direction line) calculated by the road edge direction line calculation unit 400 to select the road edge feature points extracted by the road edge feature point extraction unit 300 into a right road edge group and a left road edge group.

[0035] The road end feature three-dimensional selection unit 500 will now be described in detail. The block configuration of the road end feature three-dimensional selection unit 500 is shown in Fig. 4. The road end feature three-dimensional selection unit 500 uses the line representing the road end direction calculated by the road end direction line calculation unit 400, and includes a road end direction line angle selection unit 510 that performs selection according to the direction of the line, a road end height direction selection unit 530 that performs selection according to height distribution information from the road surface of the road end direction line, and a road end depth direction selection unit 540 that performs selection according to the sequence of directions of the road end direction lines toward the camera vanishing point on the image, and groups road end feature points into left and right road end groups.

[0036] (Road edge direction line angle selection unit 510) The road end direction line angle selection unit 510 will be described in detail. FIG. 5 shows a block configuration of the road end direction line angle selection unit 510. The road end direction line angle selection unit 510 includes a road end direction line acquisition unit 511 that acquires the road end direction line generated by the road end direction line calculation unit 400, a nearby road end direction line search unit 512 that searches for the road end direction line to be selected in the direction from the bottom to the top of the image, and a target direction line angle selection unit 513 that groups the target road end direction lines to the left and right according to the direction of the road end direction line, and groups the road end feature points into left and right road end groups according to the direction of the road end direction line. FIG. 6 shows a processing flowchart of the road end direction line angle selection unit 510. The operation based on the processing flowchart of FIG. 6 is as follows.

[0037] In step 514, a line (road end direction line) that indicates the direction of the road end calculated by the road end direction line calculation unit 400 is obtained.

[0038] In step 515, for the road end characteristic points acquired in step 514, a search is performed on the left and right sides to determine whether there are any road end characteristic points in the depth direction from the bottom to the top of the image.

[0039] 7 shows an example of selection according to the direction of the road edge direction line in the following steps 516, 518, and 520. Hereinafter, FIG. 7 will be used in detailed explanation of steps 516, 518, and 520. Note that in the coordinate axes in FIG. 7, the y-axis indicates the direction in which the host vehicle (the front sensor of the host vehicle) is pointed, and the x-axis indicates the direction perpendicular to the y-axis in the overhead view (the same applies to other figures other than FIG. 7).

[0040] In step 516, the road edge direction line detected in step 515 is used, and as shown in FIG. 7, the starting point of the line is set as the origin of the coordinate axis, and it is determined whether the angle of the line belongs to the second quadrant, the negative x-axis, or the third quadrant.

[0041] In step 517, if the determination result in step 516 is YES, the target road edge characteristic point is grouped as a right road edge.

[0042] Step 518 operates when the result of the determination in step 516 is NO. Using the road edge direction line detected in step 515, as shown in Fig. 7, the start point of the line is set as the origin of the coordinate axis, and it is determined whether the angle of the line belongs to the first quadrant, the positive x-axis, or the fourth quadrant.

[0043] In step 519, if the determination result in step 515 is YES, the target road edge characteristic point is grouped as a left road edge.

[0044] In step 520, if the result of the determination in step 518 is NO, i.e., if the direction of the road edge direction line is on the negative y-axis, the process operates. A certain area is searched for the surroundings of the target road edge direction line. Figure 8 shows an example of the surrounding search for the target road edge direction line.

[0045] In step 521, the target road edge feature point is grouped into the same group as the orientation of the surrounding road edge direction lines, using the search results from step 520. Taking Fig. 8 as an example, a search is performed within the dashed rectangular area centered on the target road edge direction line, and since the orientation of the detected surrounding road edge direction lines belongs to the third quadrant and is classified into the right road edge group, the target road edge direction line is grouped into the right road edge group, the same as the surrounding road edge direction lines.

[0046] In step 522, it is determined whether there are any other road end characteristic points obtained, and if there are still road end characteristic points, the process returns to step 515 and repeats the processes from step 515 to step 521. If there are no other road end characteristic points, the process ends.

[0047] Step 514 is executed by the road end direction line acquisition unit 511 , step 515 is executed by the nearby road end direction line search unit 512 , and step 516 and subsequent steps are executed by the target direction line angle selection unit 513 .

[0048] We will also describe the case where the road edge direction line crosses the image center. Figure 9 shows an example where the road edge direction line crosses the image center. As shown in Figure 9, the direction of the road edge direction line is in the third quadrant, but in the driving road environment, we will describe an example where it is classified as a left road edge. When the target road edge direction line crosses the image center, in addition to selection according to the direction of the road edge direction line, the grouping result of the peripheral road edge direction lines is also referred to. If the selection results based on the direction of the target road edge direction line and the peripheral road edge direction line differ, the grouping result of the peripheral road edge direction line is given priority, and the target road edge direction line is grouped into the left and right road edge groups. In the case of Figure 9, the angle of the target road edge direction line (A) is in the third quadrant, but the peripheral road edge direction line (B) is in the fourth quadrant and is grouped as the left road edge group, so the grouping result of the peripheral road edge direction line (B) is given priority, and the target road edge direction line (A) is grouped into the same left road edge group as the peripheral road edge direction line (B).

[0049] (Roadside height direction sorting unit 530) The roadside height direction selection unit 530 will now be described in detail. Fig. 10 shows a block configuration of the roadside height direction selection unit 530. The roadside height direction selection unit 530 includes a traveling road surface height detection unit 531 that detects the traveling road surface height from an acquired image, a roadside characteristic point height detection unit 532 that detects the height from the traveling road surface at each of the roadside characteristic points, a roadside identification threshold calculation unit 533 that dynamically calculates a road surface height threshold based on the unevenness of the road surface, and a roadside characteristic point height selection unit 534 that compares the road surface height threshold with height information of the roadside characteristic points to select the roadside characteristic points, and groups the roadside characteristic points based on the height information.

[0050] The road surface height detection unit 531 uses the image generated by the parallax generation unit 200 to calculate the height of the road surface area based on the parallax information. An example of road surface height detection is shown in Fig. 11. Height information of each feature point is detected from the generated image, mainly around the vehicle path. Of the detected feature points, feature points in the vehicle path area are given high priority, and the height information of the feature points is set as a candidate for the height of the road surface. Height information of feature points is detected from the vehicle path area to the left and right. After weighting the feature points in the vehicle path area, the height of the road surface is detected, including the surrounding feature points.

[0051] The road end characteristic point height detection unit 532 detects the height of each of the road end characteristic points from the traveling road surface. The height of the traveling road surface detected by the traveling road surface height detection unit 531 is acquired. After that, the three-dimensional position information of the road end characteristic points and the traveling road surface height are used to detect the height distribution of each of the road end characteristic points from the traveling road surface.

[0052] The road edge identification threshold calculation unit 533 calculates a height threshold for identifying a road edge based on the road surface height detected by the road surface height detection unit 531. This height threshold is calculated based on the flatness of the road surface. In the case of a paved road such as a national highway, the rate of change in height is generally very small over the entire road surface. However, in the case of an off-road or snow-covered road, the rate of change in height varies over the entire road surface. Therefore, if the height threshold is fixed, it may not be possible to respond to slight changes in the road surface height when identifying the road edge, and there is a risk of erroneous detection of the road edge. To solve this problem, the height threshold is dynamically changed depending on the flatness and unevenness of the road surface. FIG. 12 shows an example of the result of calculating the road edge identification threshold (height threshold) depending on the flatness of the road surface assuming an off-road or snow-covered road. The flatness of the road surface is calculated using the road surface feature points detected by the road surface height detection unit 531. As with the road surface height detection unit 531, feature points present in the vehicle travel path area are weighted with a high priority, and then the average height of the road surface is calculated including the surrounding road surface feature points. This calculated average height of the road surface is set as the road edge identification threshold. By dynamically changing the road edge identification threshold according to the flatness of the road surface, it becomes possible to accurately detect road edges even in environments where the height of the road surface is not constant, such as off-road or snow-covered roads.

[0053] The roadend feature point height selection unit 534 uses the height information of the roadend candidate feature points to select them as feature points constituting the left and right road ends, based on the roadend identification threshold calculated by the roadend identification threshold calculation unit 533. For example, when the rate of change in height of a roadend candidate feature point from the traveling road surface at an adjacent roadend candidate feature point is equal to or less than the roadend identification threshold, the roadend candidate feature point height selection unit 534 determines and selects the roadend candidate feature point as being a roadend feature point constituting the same road end as the adjacent roadend candidate feature point.

[0054] (Road edge depth direction sorting unit 540) The road end depth direction selection unit 540 will be described in detail. FIG. 13 shows a block configuration of the road end depth direction selection unit 540. The road end depth direction selection unit 540 includes a road end feature point selection result acquisition unit 541 that acquires the selection results of the road end direction line angle selection unit 510 and the road end height direction selection unit 530, and a road end feature point depth direction selection unit 542 that searches the road end feature points in the depth direction toward the vanishing point of the camera and selects whether the target road end feature point constitutes a road end, and selects the road end feature points based on the sequence in the depth direction. In this embodiment, the vanishing point of the camera is described as an example, but it is also possible to search in the depth direction in the case of LiDAR, etc. In the case of LiDAR, three-dimensional sensing is possible like a camera, so it is possible to detect the traveling road surface area. Therefore, it is possible to search for road end feature points in the depth direction by considering the direction along the detected traveling road surface as the depth direction.

[0055] In addition, in the search in the depth direction, the search is performed from the front side of the vehicle toward the depth direction. A series of road edge feature points is selected from the front side of the vehicle, and while selecting the series in the depth direction, it is possible to predict the direction in which the road edge feature points are connected. Therefore, in the case of a camera, the search starts in the direction toward the vanishing point at the beginning of the search, but in the process of searching in the depth direction, it is also possible to dynamically change the search direction. This makes it possible to select road edge feature points in the depth direction even on road edges whose shapes change.

[0056] A road end characteristic point selection result acquisition unit 541 acquires the results of selecting road end characteristic points in the road end direction line angle selection unit 510 and the road end height direction selection unit 530 .

[0057] The road end feature point depth direction selection unit 542 determines the road end feature points to be selected based on the results acquired by the road end feature point selection result acquisition unit 541. FIG. 14 shows an example of road end feature point selection by the depth direction. The depth direction means the direction from the bottom to the top of the captured image, mainly the direction toward the vanishing point of the camera. In this embodiment, the direction toward the vanishing point of the camera is described as an example, but the direction may be from the bottom to the top of the overhead view. The target road end feature points determined from the results acquired by the road end feature point selection result acquisition unit 531 are selected from the bottom of the image. As shown in FIG. 14, if a target road end feature point exists, a processing area such as a dashed rectangle toward the vanishing point of the camera is set based on the position information of the road end feature point. This processing area does not necessarily have to be toward the vanishing point of the camera, but it may be in a direction from the bottom to the top of the image. Then, this processing area is searched, and if a road end feature point having the same selection result as the target road end feature point is found, the target road end feature point is selected as a road end feature point constituting either the left or right road end. This process is performed for each of the left and right, and selection is performed in the depth direction of the left and right road end group. For example, if a road end feature point is located along the direction from an adjacent road end feature point toward the camera vanishing point on the image, the road end feature point depth direction selection unit 542 determines and selects the road end feature point as a road end feature point constituting the same road end as the adjacent road end feature point.

[0058] The road end height direction selection unit 530 selects road end feature points based on their continuity in the height direction, but this alone is insufficient, and there is a risk that, as shown in FIG. 14, a tire that has fallen and a small amount of snow that is at a certain height from the road surface may be selected as a road end. However, when considering the continuity in the depth direction, many of these feature points are one-off. Therefore, the road end feature points selected by the road end direction line angle selection unit 510 and the road end height direction selection unit 530 are selected as road end feature points constituting the left and right road ends, taking into consideration the continuity in the depth direction. In FIG. 14, the road end feature points of the left and right curb road ends exist continuously in the depth direction, so they are selected as road end feature points constituting the road ends.

[0059] In addition, the method does not only target roadsides that have continuous feature points in the depth direction, such as curbs and walls. For example, roadsides that are made up of pylons installed for construction work can also be targeted. In addition, it is possible to select roadsides that have curbs at regular intervals as the same roadside.

[0060] (Roadside identification unit 600: After processing by roadside feature 3D selection unit 500) The road end identification unit 600 uses the grouping results from the road end feature 3D selection unit 500 to identify the road ends by configuring the road end feature points extracted by the road end feature extraction unit 300 into left road ends and right road ends, respectively.

[0061] The road end identification unit 600 will be described in detail. Based on the results of the selection made by the road end feature three-dimensional selection unit 500, the road end identification unit 600 connects the road end feature points constituting the left and right road ends (more specifically, connects the road end feature points grouped in the same right road end group and left road end group) to identify the left and right road ends, respectively. Based on the left and right grouping results selected by the road end direction line angle selection unit 510, road end feature points that satisfy all three conditions of the road end selection result in the height direction by the road end height direction selection unit 530 and the road end selection result in the depth direction by the road end depth direction selection unit 540 are used. The left and right road ends are identified by these road end feature points.

[0062] This makes it possible to accurately detect the road edges on both the left and right sides without mistaking the characteristic points extracted as road edge characteristic points.

[0063] (Roadside detection and determination unit 700) The road edge detection and determination unit 700 uses the road edge information identified by the road edge identification unit 600 to calculate information required for road edge departure suppression control using road edge information such as the lateral position and yaw angle of the road edge, and determines the reliability of the identified road edge. The road edge detection and determination unit 700 can also use position information of characteristic points identified as road edges to accurately detect the shape of the road edge and predict whether the vehicle will deviate into a road edge in the future.

[0064] (Alarm control unit 800) The warning control unit 800 issues a warning and performs control to prevent the vehicle from deviating from the road edge based on the road edge information calculated by the road edge detection and determination unit 700. By calculating the lateral position and yaw angle of the road edge, it is possible to perform road edge departure prevention control even for complex road edge shapes. In addition, if the vehicle is equipped with a slip angle, etc., it is possible not to activate control using road edge information since there is a risk that the road edge detection result will be inaccurate.

[0065] As described above, the external environment recognition device 1 of this embodiment has a processing unit 900 that detects road ends. The processing unit 900 has a road end feature point extraction unit 300 that determines road end feature points corresponding to road ends from sensing results output from an in-vehicle sensing device that detects information outside the vehicle, a road end direction line calculation unit 400 that calculates a line representing the direction of the road end constituted by the road end feature points (e.g., a normal line to the road end) for each of the road end feature points, and a road end identification unit 600 that groups the road end feature points into a right road end group and a left road end group according to the direction of the line (e.g., 1st and 4th quadrants / 2nd and 3rd quadrants) and identifies the road end by the road end feature points grouped into the same group.

[0066] In other words, the external environment recognition device 1 of this embodiment determines road end feature points corresponding to road ends from sensing results output from an on-board sensing device that detects information outside the vehicle, calculates a line representing the direction of the road end constituted by the road end feature points (e.g., the normal to the road end) for each road end feature point, groups the road end feature points into a right road end group and a left road end group according to the direction of the line, and identifies the road end using the road end feature points grouped into the same group, making it possible to accurately detect the left and right road ends.

[0067] According to this embodiment, in the process of road edge detection, a line indicating the direction of the road edge constituted by the road edge characteristic points is calculated for each road edge characteristic point, and the road edge characteristic points are grouped into right road edges and left road edges according to the direction of the calculated line, thereby enabling more accurate road edge detection. Furthermore, by accurately detecting the left and right road edges, it becomes possible to suppress erroneous control using road edge information (e.g., erroneous control of the road edge departure suppression function).

[0068] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to the embodiment having all of the described configurations.

[0069] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. [Explanation of symbols]

[0070] 1...External world recognition device 100 Stereo camera unit (vehicle-mounted sensing device) 200...Parallax generation unit 300 Road edge feature point extraction unit 400...Road end direction line calculation section 500···3D roadside feature selection section 510 Road edge direction line angle selection unit 511... Roadside direction line acquisition unit 512 Nearby road end direction line search unit 513 Target direction line angle selection unit 530 Roadside height direction sorting section 531 Road surface height detection unit 532: Road edge characteristic point height detection unit 533 Road end specific threshold calculation unit 534...Roadside characteristic point height selection unit 540 Road edge depth direction selection section 541: Roadside feature point selection result acquisition unit 542...Roadside feature point depth direction selection unit 600...Roadside identification section 700 Roadside detection unit 800 Alarm control unit 900 Processing section

Claims

1. It has a processing unit that detects the road end, and the processing unit includes: a road end feature point extraction unit that obtains a road end feature point corresponding to the road end from the sensing result output from an in-vehicle sensing device that detects information outside the vehicle; a road end direction line calculation unit that calculates, for each of the road end feature points, a line representing the direction of the road end constituted by the road end feature points; a road end identification unit that groups the road end feature points into a right road end group and a left road end group according to the direction of the line, and identifies the road end by the road end feature points grouped into the same group; and has: The line representing the direction of the road end is the normal line of the road end; The road end direction line calculation unit is characterized in that, as the line representing the direction of the road end, it calculates only the normal line in the direction in which the in-vehicle sensing device is attached, and does not calculate the normal line in the direction away from the in-vehicle sensing device. An external environment recognition device.

2. In the external environment recognition device according to Claim 1, The line representing the direction of the road end is obtained from the positional relationship with respect to the road end feature points existing around the road end feature point. An external environment recognition device characterized by this.

3. In the external environment recognition device according to Claim 1, further, it has a road end feature point height detection unit that obtains at least the distribution of the height from the road surface at each of the road end feature points, The road end identification unit determines that the road end feature point is a road end feature point that constitutes the same road end as the adjacent road end feature point when the change rate of the height from the road surface at the adjacent road end feature point is equal to or less than a threshold value. An external environment recognition device characterized by this.

4. In the external environment recognition device according to Claim 3, The threshold value is determined according to the flatness of the road surface. An external environment recognition device characterized by this.

5. In the external environment recognition device according to Claim 1, The sensing result is an image, The road end identification unit determines that the road end feature point is a road end feature point that constitutes the same road end as the adjacent road end feature point when the road end feature point is located along the direction from the adjacent road end feature point toward the camera vanishing point on the image. An external environment recognition device characterized by this.

6. In the external environment recognition device according to Claim 1, The roadside identification unit groups the roadside feature points into a right roadside group and a left roadside group according to the direction of the line representing the direction of the roadside, the distribution of the height of the line representing the direction of the roadside from the road surface, and the continuity of the line representing the direction of the roadside in the depth direction. An external recognition device characterized by this.

7. In the external recognition device according to Claim 1, The sensing result is an image, and when the line representing the direction of the roadside crosses the center of the image, When the grouping result by the direction of the line representing the direction of the roadside and the direction of the line representing the direction of the roadside around the line representing the direction of the roadside is different, the roadside identification unit gives priority to the grouping result by the direction of the line representing the direction of the roadside around, and the roadside feature points constituting the line representing the direction of the roadside are grouped into the same right roadside group or left roadside group as the roadside feature points constituting the line representing the direction of the roadside around. An external recognition device characterized by this.

Images