Environment recognition device and environment recognition method

Abstract

To provide an environment recognition device and an environment recognition method capable of performing appropriate vehicle control even in a situation in which the performance in detecting a road surface condition using camera images is degraded.SOLUTION: An environment recognition device with a processing unit recognizes the external environment on the basis of multiple images acquired by multiple imaging devices for capturing images of the outside of a vehicle. The processing unit is configured to: detect a first on-road structure candidate that is a candidate of an on-road structure existing on the road surface which is imaged on the basis of parallax information acquired from multiple images; detect a second on-road structure candidate that is a candidate of an on-road structure existing on the road surface which is imaged on the basis of the parallax information and shading information on the image; determine the position and size of the on-road structure on the basis of the detection results related to the first on-road structure candidate and the second on-road structure candidate; and output information related to the on-road structure whose position and size have been determined to a vehicle control device of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an environment recognition device and an environment recognition method. [Background technology]

[0002] There is a known technology for detecting unevenness on a road surface from an image capturing an area including the road surface. Patent Document 1 describes a transmission amount control device that recognizes unevenness on the road surface on which a vehicle is traveling based on an image captured by a camera device, and controls the vibration absorption characteristics of the suspension based on the result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-55757 A Summary of the Invention [Problem to be solved by the invention]

[0004] The transmission amount control device described in Patent Document 1 has a problem in that it cannot perform appropriate control when the detection performance of the road surface condition is degraded due to, for example, weather conditions, lighting conditions, road markings, and the like.

[0005] An object of the present invention is to provide an environment recognition device that can perform appropriate vehicle control even in a situation where the performance of detecting road surface conditions from images is degraded. [Means for solving the problem]

[0006] An environmental recognition device according to one embodiment of the present invention is an environmental recognition device equipped with a computing device that recognizes the external environment based on multiple images acquired by multiple imaging devices that image the outside of a vehicle, wherein the computing device detects a first road structure candidate indicating a candidate road structure present on the road surface captured in the image based on disparity information obtained from the multiple images, detects a second road structure candidate indicating a candidate road structure present on the road surface captured in the image based on the disparity information and shading information of the image, determines the position and size of the road structure based on the detection results for the first road structure candidate and the second road structure candidate, and outputs information regarding the road structure whose position and size have been determined to a vehicle control device of the vehicle. In addition, an environmental recognition method according to one embodiment of the present invention detects a first road structure candidate indicating a candidate road structure existing on the road surface captured in an image based on disparity information obtained from multiple images acquired by multiple imaging devices that capture images of the outside of the vehicle, detects a second road structure candidate indicating a candidate road structure existing on the road surface captured in the image based on the disparity information and shading information of the image, determines the position and size of the road structure based on the detection results of the first road structure candidate and the second road structure candidate, and outputs information regarding the road structure whose position and size have been determined to a vehicle control device of the vehicle. Effect of the Invention

[0007] According to the present invention, appropriate vehicle control can be performed even in a situation where the performance of detecting road surface conditions from images is degraded. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a vehicle equipped with an environment recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of the environment recognition device according to the first embodiment. [Diagram 3]FIG. 3 is a block diagram illustrating a functional configuration of the environment recognition device according to the first embodiment. [Figure 4] FIG. 4 is a conceptual diagram of a method for determining the road surface height. [Diagram 5] FIG. 5 is an illustration of merging distance points determined at different times. [Figure 6] FIG. 6 is a schematic diagram showing an example of a likelihood map. [Figure 7] FIG. 7 is an image diagram of the case where the bump detection results are applied to a driving assistance or autonomous driving system. [Figure 8] FIG. 8 is a diagram illustrating an example of control contents for each reliability level. [Figure 9] FIG. 9 is a diagram illustrating a table for determining the reliability. [Figure 10] FIG. 10 is a flowchart of the environment recognition process performed by the environment recognition device. [Figure 11] FIG. 11 is a block diagram showing a schematic configuration of a vehicle equipped with an environment recognition device according to the second embodiment. [Figure 12] FIG. 12 is a block diagram illustrating a functional configuration of the environment recognition device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment An environment recognition device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 10.

[0010] 1 is a block diagram showing a schematic configuration of a vehicle 1 equipped with an environment recognition device 3 according to a first embodiment. The vehicle 1 includes a first camera 2a, a second camera 2b, the environment recognition device 3, a vehicle control device 4, a display 5, a speaker 6, an accelerator 7, a brake 8, and a suspension 9.

[0011] The first camera 2a and the second camera 2b capture an image of a range including a field of view in the traveling direction (forward) of the vehicle 1. In the following description, the first camera 2a and the second camera 2b may be collectively referred to as the camera 2.

[0012] The first camera 2a and the second camera 2b each generate a vehicle front image and output it to the environment recognition device 3. The environment recognition device 3 detects a bump on the road surface in front of the vehicle 1 based on the pair of vehicle front images. The environment recognition device 3 calculates the distance to the detected bump, the height of the detected bump, the relative speed between the vehicle 1 and the detected bump, the reliability of the detection, and the like, and outputs the calculation results to the vehicle control device 4. Based on these calculation results, the vehicle control device 4 judges the magnitude of the impact of the bump on the vehicle 1 and the risk of a collision between the bump and the vehicle 1. Depending on the judgment result, the vehicle control device 4 performs control such as displaying a warning message on the display 5, turning on a warning light on the display 5, playing a warning sound from the speaker 6, causing the accelerator 7 to ease or suppress acceleration, causing the brakes 8 to decelerate, and changing the shock absorbing characteristics of the suspension 9.

[0013] FIG. 2 is a block diagram showing a hardware configuration of the environment recognition device 3 according to the first embodiment. The environment recognition device 3 is composed of a computer including a calculation device 11 such as a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), etc., a non-volatile memory 12 such as a read only memory (ROM), a flash memory, a hard disk drive, etc., a volatile memory 13 called a random access memory (RAM), an input / output interface 14, and other peripheral circuits. These hardware components work together to operate software and realize a plurality of functions. The environment recognition device 3 may be composed of one computer or a plurality of computers. In addition, the calculation device 11 may be composed of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0014] The non-volatile memory 12 stores programs capable of executing various calculations. In other words, the non-volatile memory 12 is a storage medium (storage device) capable of reading programs that realize the functions of this embodiment. The volatile memory 13 is a storage medium (storage device) that temporarily stores the results of calculations by the calculation device 11 and signals input from the input / output interface 14. The calculation device 11 is a device that loads the programs stored in the non-volatile memory 12 into the volatile memory 13 and executes calculations, and performs predetermined calculation processing on data taken from the input / output interface 14, the non-volatile memory 12, and the volatile memory 13 in accordance with the programs.

[0015] The environment recognition device 3 is connected to each of the first camera 2a, the second camera 2b, and the vehicle control device 4. An input section of the input / output interface 14 converts signals input from various devices (such as the first camera 2a and the second camera 2b) into data that can be calculated by the arithmetic device 11. In addition, an output section of the input / output interface 14 generates an output signal according to the calculation result in the arithmetic device 11, and outputs the signal to various devices (such as the vehicle control device 4).

[0016] The first camera 2a includes an image sensor 21a and a DSP 22a. Similarly, the second camera 2b includes an image sensor 21b and a DSP 22b. In the following description, the image sensor 21a and the image sensor 21b may be collectively referred to as the image sensor 21, and the DSP 22a and the DSP 22b may be collectively referred to as the DSP 22.

[0017] The imaging element 21 has photodiodes arranged in a grid pattern. The imaging element 21 outputs an imaging signal capturing an image of a subject to the DSP 22. The DSP 22 converts the imaging signal input from the imaging element 21 into an image signal while performing image quality corrections such as contrast correction, gamma correction, and edge correction. The first camera 2a outputs an image signal generated by the DSP 22a to the environment recognition device 3. The second camera 2b outputs an image signal generated by the DSP 22b to the environment recognition device 3.

[0018] Note that the first camera 2a and the second camera 2b may be provided with a function for inputting and outputting a synchronization signal, so that a synchronization signal from the imaging element 21 of one camera 2 can be input to the imaging element 21 of the other camera 2, or the shutter timings of the imaging elements 21 of both cameras 2 can be aligned based on a signal output from the environment recognition device 3. In this way, the calculation accuracy of the parallax calculation unit 31, which will be described later, can be improved.

[0019] Furthermore, image quality correction parameters of the DSP 22 may be adjusted simultaneously for both cameras 2 based on the signal output from the environment recognition device 3. In this way, whiteout and blackout of the image signal are reduced, and the parallax calculation density of the parallax calculation unit 31, which will be described later, is improved.

[0020] 3 is a block diagram showing a schematic functional configuration of the environment recognition device 3 according to the first embodiment. The environment recognition device 3 includes a disparity calculation unit 31, a first detection unit 32, a second detection unit 33, a disparity analysis unit 34, a region analysis unit 35, a measurement processing unit 36, and a reliability calculation unit 37. Each of these functional units is realized in the form of software by the calculation device 11 (FIG. 2) executing a program stored in the non-volatile memory 12 (FIG. 2).

[0021] The parallax calculation unit 31 acquires the captured image from the camera 2 and calculates parallax information (distance image) and an invalid area. The first detection unit 32 detects bumps present on the road surface captured in the captured image based on parallax information obtained from a plurality of captured images. The parallax analysis unit 34 judges whether or not the first detection unit 32 is likely to make a false detection or a non-detection, and generates a flag for the situation. The second detection unit 33 detects bumps present on the road surface captured in the captured image based on the parallax information and the shading information of the captured image. The area analysis unit 35 judges whether or not the second detection unit 33 is likely to make a false detection or a non-detection, and generates a flag for the situation. The measurement processing unit 36 ​​calculates the position and size of the bump based on the detection result of the bump by the first detection unit 32 and the detection result of the bump by the second detection unit 33. The reliability calculation unit 37 calculates the reliability indicating the degree to which the position and size of the bump calculated by the measurement processing unit 36 ​​are reliable.

[0022] (Explanation of parallax calculation process) The parallax calculation process executed by the parallax calculation unit 31 will be described. The parallax calculation unit 31 calculates parallax information (distance image) and an invalid area by a known method using a pair of image signals (captured images) output from the first camera 2a and the second camera 2b. The distance image is an image in which distance information (depth magnitude) is expressed in gray scale, also called a depth map. The invalid area means an area of ​​the entire distance image where the distance could not be calculated. There are various known methods for calculating the distance image and the invalid area from a pair of captured images, such as a method using epipolar geometry or a method using CNN (Convolutional Neural Network), and therefore the description will be omitted.

[0023] (Explanation of the first detection process) The first detection process executed by the first detection unit 32 will be described. The first detection process is a process for detecting bumps on the road surface mainly based on disparity. The first detection unit 32 acquires disparity information and an invalid area from the disparity calculation unit 31, and extracts disparity information of a forward area corresponding to the path of the vehicle 1. The first detection unit 32 calculates the road surface height for each distance based on the extracted disparity information. This calculation method will be described with reference to Figs. 4 and 5.

[0024] FIG. 4 is a conceptual diagram of a method for calculating road surface height. As shown in FIG. 4(a), it is assumed that at a certain time t0, the vehicle 1 is traveling on a substantially flat road surface without running over local unevenness. At this time, the road surface 301 can be estimated from the geometric values ​​of the design of the vehicle 1. The first detection unit 32 uses the fact that the installation positions of the first camera 2a and the second camera 2b mounted on the vehicle 1 are different from each other to detect the positions on the imaging plane when these cameras 2 observe the same subject, and calculates the parallax, which is the difference between the positions. The first detection unit 32 converts the calculated parallax into the distance from the camera 2 to the point of interest, thereby determining the distance points 302, 303, and 304 shown in FIG. 4.

[0025] If there is no bump on the road surface on which the vehicle 1 runs, that is, if the road surface on which the vehicle 1 runs is flat, these distance points 302 to 304 exist on the estimated road surface 301. On the other hand, if there is a bump as shown in FIG. 4(a), there is a distance point determined at a position away from the estimated road surface 301, such as distance point 304, that does not exist on the estimated road surface 301. In the first detection process, when a perpendicular line is drawn vertically downward from distance point 304 to the estimated road surface 301, the length of this perpendicular line is treated as the road surface height at distance point 304. The first detection unit 32 calculates this road surface height for each distance point. For example, the first detection unit 32 estimates a road surface height curve for the road surface height calculated for each distance point by fitting a curve such as a Bezier curve or a spline curve to a point group consisting of a large number of determined distance points.

[0026] As shown in FIG. 4(a), the density of distance points relative to the distance from the vehicle 1 varies depending on the distance from the camera 2 and the road surface shape. The distance from the vehicle 1 here refers to the distance measured parallel to the road surface 301 estimated from directly below a predetermined reference point (for example, the ground contact point when the body of the vehicle 1 is assumed to be placed on a horizontal surface, or the tip of the bumper of the vehicle 1). The density of distance points relative to the distance from the vehicle 1 generally becomes sparser as the distance from the vehicle 1 increases, and conversely, becomes denser at closer distances. This is because the area of ​​a location farther away in a captured image becomes smaller, so that even if distance points are distributed evenly on the captured image, there are fewer distant distance points in three-dimensional space. The density of distance points also changes due to occlusion caused by unevenness on the road surface. For example, if there is a convex part on the road surface, the road surface behind the convex part is hidden by the convex part and cannot be seen from the camera 2 (so-called occlusion). Therefore, the density of distance points decreases behind such a convex part.

[0027] If the density of distance points decreases for the above reasons, the accuracy of road surface height estimation by the first detection unit 32 may decrease. Therefore, distance points determined separately at multiple times may be collected into one point cloud, and the above-mentioned curve may be fitted to the point cloud. In this way, the density of distance points is improved, and the road surface height curve can be estimated with high accuracy.

[0028] FIG. 5 is an explanatory diagram of integrating distance points determined at different times. In each of the diagrams of FIG. 5(a) to (e), the horizontal axis indicates the horizontal distance from the vehicle 1 to the distance point, and the vertical axis indicates the vertical distance from the estimated road surface 301 to the distance point. FIG. 5(a) shows the distance points determined at time t0 shown in FIG. 4(a), FIG. 5(b) shows the distance points determined at time t1 shown in FIG. 4(b), and FIG. 5(c) shows the distance points determined at time t2 shown in FIG. 4(c). Assuming that the shape of the road surface on which the vehicle 1 travels does not change between times t0 and t2, the traveling speed or traveling distance of the vehicle 1 is known, so when the current time is time t2, the distance points determined at past time t0 or the distance points determined at past time t1 can be integrated into the distance points determined at current time t2. For example, FIG. 5(d) shows the result of plotting the distance points at each time by shifting the horizontal positions of the distance points at each time by the difference in the horizontal positions of the vehicle 1 at each time.

[0029] In addition, the estimated road surface 301 in FIG. 4 assumes that the vehicle 1 is on a flat road surface and that no momentary vehicle body sway occurs. However, if there are small irregularities on the road surface, the pitch angle and roll angle of the vehicle 1 may change from moment to moment. In this case, if a road surface estimated based on the geometric values ​​of the design of the vehicle 1 is used, a problem occurs in that an appropriate road surface height cannot be obtained, or a problem occurs in that distance points determined based on disparity information at multiple different times cannot be properly integrated. Therefore, when estimating the road surface, the attitude or angle of the vehicle 1 with respect to the road surface may be taken into consideration. For example, the road surface 301 may be estimated by performing a planar approximation of distance points obtained based on disparity information excluding invalid areas, or the road surface 301 may be estimated by utilizing the fact that the object detection results of the vehicle 1, pedestrians, etc. are grounded on the road surface. Alternatively, the road surface 301 may be estimated by extracting road markings such as white lines indicating road boundaries. By adding such ingenuity, a dense point cloud that follows the shape of the road surface can be obtained, and the road surface height can be detected with high accuracy.

[0030] Furthermore, although it has been described that curves such as Bezier curves and spline curves are fitted to the point group when estimating the road surface height curve, fitting of a curve different from this may be performed. The input point group is p(i), the discrete points on the output curve are y(i), (the approximate straight line slope) x (the length from i-1 to i) is LS(i), and the past discrete points moved according to the progress of the vehicle 1 are Y'(i), and the energy evaluation function Eall is defined as the following equation (1).

[0031]

number

[0032] Curve fitting can be performed by finding Y'(i) that minimizes this energy evaluation function Eall. This makes it possible to obtain a curve fitting result that is close to the input point, has a first derivative value (=slope) close to the road surface slope, has a small second derivative value (change in slope), and is highly consistent with the speed of the vehicle 1. For example, a curve like the fitted curve 401 in FIG. 5(e) can be obtained for the point cloud in FIG. 5(d).

[0033] After estimating the road surface height curve as described above, the first detection unit 32 extracts the distance and height from the reference point of the vehicle 1 to the convex portion (mountain portion) of the road surface height curve based on the road surface height curve. The first detection unit 32 extracts coordinates (i.e., candidate positions of bumps) that have a maximum value on the road surface height curve, and regards these as the distance to the convex portion of the curve. The first detection unit 32 regards the length of a perpendicular line drawn from the coordinates that have the maximum value to the road surface plane as the height of the convex portion of the curve.

[0034] If both the distance and the height are within a predetermined range (i.e., the convex portion is not too far from the vehicle 1 or too low relative to the road surface), the first detection unit 32 sets a tracking point (i.e., a candidate point for a bump) at the coordinates of the road surface height curve that have a maximum value, and tracks the tracking point. For example, the tracking point is tracked in time series using an optical flow method, and the distance change of the tracking point is obtained from the parallax, so that the movement amount of the vehicle 1 can be calculated. Note that, in order to improve stability, multiple tracking points may be tracked and the movement amount of the vehicle 1 may be calculated from the average of the parallax. If the calculated movement amount of the vehicle 1 and the distance change amount of the tracking point are almost equal (if the difference is within a predetermined threshold), a tracking success flag is set, which indicates that the bump has been correctly tracked. If not, the tracking success flag is not set, since it is assumed that the detected candidate point for the bump is not actually a bump.

[0035] The first detection unit 32 may obtain the steering angle, vehicle speed, and the like of the vehicle 1 from a network that communicates vehicle information, such as a controller area network (CAN) (not shown). In this way, the first detection unit 32 can calculate the vehicle's course and movement amount from the steering angle, vehicle speed, and the like, for example, using a well-known dead reckoning method. As a result, it becomes possible to extract disparity information of an area corresponding to the vehicle's course, improving the calculation accuracy of the road surface height and the estimation accuracy of the movement amount of the vehicle 1. Furthermore, calculations such as the optical flow method can be omitted, thereby reducing calculation costs.

[0036] (Description of disparity analysis process by disparity analyzer 34) The following describes the parallax analysis process executed by the parallax analysis unit 34. The parallax analysis unit 34 determines whether or not the first detection unit 32 is in a situation where it is difficult to correctly detect a bump (for example, a situation where it is easy to erroneously detect a non-existent bump or a situation where it is difficult to detect a present bump), and generates a flag indicating the situation.

[0037] The parallax analysis unit 34 generates a flag for determining whether a bump has been erroneously detected based on an erroneous parallax calculation. There is a possibility that an erroneous parallax may occur due to errors in the visual axis or angle of view of each of the multiple cameras 2 functioning as a stereo camera. For example, multiple cameras 2 installed apart in the horizontal direction may calculate an erroneous parallax for a horizontal straight line in an image. Therefore, the parallax analysis unit 34 calculates an edge angle for an image captured by one of the cameras 2, and if an area where the horizontal component is stronger than a predetermined threshold and the vertical component is weaker than a predetermined threshold coincides with the bump area coordinates detected by the first detection unit 32 within the threshold, it is determined that there is a possibility of erroneous detection due to erroneous parallax, and sets a horizontal line presence / absence determination flag.

[0038] In addition, when the cameras 2 are installed at a distance in the vertical direction, there is a possibility that erroneous parallax may occur for vertical line segments as well. In that case, a similar process may be performed for the vertical line segments and a similar flag may be set.

[0039] The parallax analysis unit 34 generates a flag for determining that the road surface is low contrast or low texture. For example, in an environment where the contrast of the image obtained from the camera 2 is reduced, such as a dark environment, a foggy environment, or a uniform paved road surface, the invalid area increases and the valid parallax area decreases when calculating the parallax, so that the detection performance of bumps at a distance decreases, and there is a possibility that the bump will be suddenly detected when the bump approaches the vehicle 1. Therefore, the parallax analysis unit 34 sets multiple small areas for the image captured by at least one of the cameras 2 (for example, small areas of a predetermined size are arranged in the entire image so that they do not overlap each other, and the small areas are set in advance), and calculates statistics such as the average and variance of the brightness values ​​in each small area and the moment. If these statistics are within a predetermined threshold range, the parallax analysis unit 34 sets a low texture road surface determination flag. For example, in a dark country road with few lights such as street lamps, the average and variance of the brightness values ​​of the captured image are small, making it difficult to detect road surface irregularities in the distance. The threshold value used to determine the low texture road surface determination flag is set according to such an environment.

[0040] The parallax analysis unit 34 generates a flag for determining that the specular reflection component of the road surface is strong. For example, in a situation where the paved road surface is wet, such as during or immediately after rain, or when driving on an underpass with poor drainage, the sky or surrounding structures reflected on the road surface, rather than the road surface, may appear in the image captured by the camera 2. In such an environment, the parallax calculated by the parallax calculation unit 31 may not represent the shape of the road surface, so that the bump detection performance of the first detection unit 32 may decrease, and there is a possibility that a bump may be erroneously detected or not detected. Therefore, when the number of distance points below the running plane of the vehicle 1 (i.e., the road surface height is a negative value) obtained from the camera geometry is equal to or greater than a predetermined threshold, the parallax analysis unit 34 considers the reliability of the bump detection result to be low and sets a specular reflection determination flag.

[0041] (Description of the second detection process by the second detection unit 33) The second detection process executed by the second detection unit 33 will be described. The second detection process is a process for detecting bumps on the road surface mainly based on region division. The second detection unit 33 generates a likelihood map of the bump based on the parallax information and the image (grayscale image) captured by at least one of the cameras 2. The likelihood map can be generated, for example, by using a Semantic Segmentation method using CNN. In this case, a model that inputs parallax information of H pixels in the vertical direction and W pixels in the horizontal direction and an image captured by H pixels in the vertical direction and W pixels in the horizontal direction and outputs a likelihood map of H pixels in the vertical direction and W pixels in the horizontal direction may be trained in advance. Such CNN technology and Semantic Segmentation technology are well known, so their description will be omitted. In the following description, the likelihood map is assumed to have a likelihood of 0 to 1 for each pixel position (1 indicates that the likelihood of a bump is the highest).

[0042] FIG. 6 is a schematic diagram showing an example of a likelihood map. FIG. 6(a) is a photographed image (grayscale image), FIG. 6(b) is a positional relationship between the vehicle 1 and the road surface as viewed from the side, FIG. 6(c) is disparity information, and FIG. 6(d) is a likelihood map. As shown in FIG. 6(a) and FIG. 6(b), a flat road surface 501 is present in front of the vehicle 100, and a bump 502 is present between the vehicle 100 and another vehicle 503 traveling in front of it. At this time, as shown in FIG. 6(c), disparity also appears in the area of ​​the flat road surface 501, and the disparity changes smoothly from the front side to the back side in the image. In the likelihood map of FIG. 6(d), an area 504 where the bump 502 exists is the area with the highest likelihood, an area 505 including the front side and the back side is the area with the next highest likelihood, and an area 506 around it is the area with the lowest likelihood. In addition, the likelihood of other areas is zero.

[0043] The input to the second detection unit 33 may be disparity information of a certain size and a grayscale image with the same resolution as the disparity information. The disparity information calculated by a stereo camera consisting of two cameras 2 with the same angle of view and the same resolution and the grayscale image of one of the cameras 2 constituting the stereo camera have the same resolution, but by linking the corresponding areas of the disparity information and the grayscale image and inputting them to the CNN, it is possible to suppress performance degradation for objects that are difficult to separate in the learning data. Examples of objects that are difficult to separate in the learning data include bumps that exist together with a rough road surface and a flat rough road surface. In this way, it is possible to suppress performance degradation in scenes where the textures on the grayscale image are similar but the presence of convex parts is different in the disparity information.

[0044] The second detection unit 33 uses the likelihood map to calculate the activity of whether or not there is a bump for each distance. For example, a likelihood map of H pixels in the vertical direction and W pixels in the horizontal direction is input, and the activity of H pixels in the vertical direction is output. Since bumps are generally installed perpendicular to the road, the stability of bump detection can be improved by accumulating the likelihood map in the horizontal direction to calculate the activity. Specifically, the calculation is as follows. If the likelihood at a position (x, y) on the likelihood map is L(x, y) and the forward area corresponding to the path of the vehicle 1 for each y coordinate is Sy, the activity A(y) is calculated by the following formula (2). If the activity A(y) is equal to or greater than a predetermined threshold, the second detection unit 33 determines that a bump area exists at the y coordinate.

[0045]

number

[0046] Here, f(y) is a correction coefficient for each y coordinate, which is set to 0 in the area above the horizon, small in the area below the horizon and near the vehicle 1 (lower side of the screen), and large in the area far from the vehicle 1 (upper side of the screen). That is, the second detection unit 33 determines that there is a bump area when there are more areas with high likelihood near the vehicle 1, and conversely, it is easy to determine that there is a bump area far away even if there are fewer areas with high likelihood than near. The second detection unit 33 converts the calculated activity from the image coordinate system to a distance in real space based on the camera geometry.

[0047] The second detection unit 33 accumulates the activity level by shifting it according to the movement amount of the vehicle 1, and calculates the accumulated cumulative activity level. If the activity level at a distance d at time t1 is denoted as A(t1, d), and the activity level at a distance d at time t2 is denoted as A(t2, d), and if the vehicle 1 moves by Dt2 from time t1 to time t2, the cumulative activity level IA(t2, d) at the distance d at time t2 is calculated by the following formula (3). Since a bump area usually has a width of about 30 cm to 80 cm in the depth direction, the second detection unit 33 determines that a bump area exists when an area where the cumulative activity level IA(t, d) is greater than a predetermined threshold continues for a predetermined threshold range (for example, a range of 30 cm to 80 cm).

[0048]

number

[0049] (Description of region analysis process by region analysis unit 35) We will now explain the region analysis process executed by the region analysis unit 35. The region analysis unit 35 determines whether or not the second detection unit 33 is in a situation where it is difficult to correctly detect a bump (for example, a situation where it is easy to erroneously detect a non-existent bump or a situation where it is difficult to detect a present bump), and generates a flag indicating the situation.

[0050] The region analysis unit 35 generates a flag for discriminating erroneous detection of bumps based on road markings. Region division by CNN uses a CNN model that is trained in advance by providing pairs of a grayscale image, a disparity image, and teaching information (or a label image), but it is known that there is data that is difficult to separate in this pre-trained data. There are points where a specific road marking is drawn and a bump exists, and points where a specific road marking is drawn but the road surface is flat, and when this road marking exists, the second detection unit 33 may erroneously detect or not detect a bump. By recognizing such road markings, the region analysis unit 35 sets a road marking flag indicating the possibility of erroneous detection or non-detection. For example, the region analysis unit 35 calculates the edge angle for the grayscale image of the bump region output by the second detection unit 33, and activates the road marking flag when the ratio of each element of the histogram of the edge angle is within a predetermined threshold range.

[0051] The region analysis unit 35 generates a flag for determining whether a bump is erroneously detected based on the shadow of a tree. For example, when a complex shadow such as sunlight filtering through the trees falls on the road surface, the shadow may cause an error in the parallax, and such a shadow may not be included in the grayscale image pattern that the CNN model has previously learned, resulting in a false detection or non-detection of a bump. By recognizing such complex tree shadows, the region analysis unit 35 sets a tree shadow flag indicating a possibility of non-detection. The region analysis unit 35 calculates the edge angle for the grayscale image of the bump region output by the second detection unit 33, for example, and activates the tree shadow flag when the histogram of the edge angle is flat within a predetermined threshold value. The edge angle can be obtained, for example, by applying a Sobel filter to each pixel of the grayscale image of the bump region.

[0052] The region analysis unit 35 generates a flag for determining whether a bump is erroneously detected based on a three-dimensional object such as another vehicle or a pedestrian. Even if the road surface on which the vehicle 1 is traveling is flat, if there are three-dimensional objects such as other vehicles or pedestrians in the vicinity, the road surface disparity is affected when calculating the road surface parallax, and unevenness that does not actually exist may be calculated when calculating unevenness on the road surface of the vehicle 1. In order to prevent a bump from being erroneously detected due to the influence of such erroneous disparity, the region analysis unit 35 enables a correlation flag with other recognition logic when the recognition results of other three-dimensional objects such as vehicle detection and pedestrian detection are present in the vicinity within the threshold of the likelihood map calculation region.

[0053] (Explanation of measurement processing by the measurement processing unit 36) The following describes the measurement processing executed by the measurement processing unit 36. The measurement processing unit 36 ​​calculates the distance to the bump, the height of the bump, and the relative speed of the bump as seen from the vehicle 1, based on the bump detection result by the first detection unit 32, the bump detection result by the second detection unit 33, the flag generated by the disparity analysis unit 34, and the flag generated by the region analysis unit 35.

[0054] Due to the principle of a stereo camera, it is difficult to detect parallax at a distance. Therefore, bump detection based on parallax analysis (bump detection by the first detection unit 32) is likely to have a lower detection performance at a distance. Conversely, bump detection based on area analysis (bump detection by the second detection unit 33) can detect bumps accurately even at a distance, but the detection accuracy of the distance to the bump and the height of the bump is likely to decrease. Therefore, when both the first detection unit 32 and the second detection unit 33 detect a bump, if the distance to the detected bump is farther than a predetermined distance Th1, the measurement processing unit 36 ​​adopts the detection result by the second detection unit 33, and if the distance to the detected bump is closer than a predetermined distance Th2, the measurement processing unit 36 ​​adopts the detection result by the first detection unit 32 and the detection result by the second detection unit 33. If the distance to the detected bump is less than Th1 and greater than Th2, the measurement processing unit 36 ​​calculates a weighted average of the detection result by the first detection unit 32 and the detection result by the second detection unit 33, and adopts the calculation result as the distance to the bump and the height of the bump. Here, it is assumed that the relationship Th1>Th2 holds between the two threshold values ​​Th1 and Th2.

[0055] When only one of the first detector 32 and the second detector 33 detects a bump, the measurement processor 36 uses the detection result from that one as is.

[0056] (Description of Reliability Calculation Process by Reliability Calculation Unit 37) The reliability calculation process executed by the reliability calculation unit 37 will be described. The reliability calculation unit 37 calculates the reliability of the bump detection result adopted by the measurement processing unit 36, and determines what kind of control the detection result can be used for. Since the recognition of the environment around the vehicle by image recognition has the possibility of erroneous detection or non-detection due to the driving environment and lighting environment, it is difficult to always perform reliable control. Therefore, by calculating the degree to which the recognition of the environment around the vehicle is performed by image recognition as an evaluation index, the vehicle control device 4 can perform appropriate control according to the reliability of the recognition.

[0057] FIG. 7 is an image diagram of a case where the bump detection result is applied to a driving assistance or automatic driving system. When a bump is detected sufficiently far away, the vehicle control device 4 first (1) displays a warning to the driver or passengers using the display 5. If the vehicle speed is above a certain level, the vehicle control device 4 performs (2) acceleration suppression control on the accelerator 7 to suppress further acceleration. If the vehicle approaches the bump while still at a high speed, the vehicle control device 4 notifies the driver or passengers by (3) sounding an alarm from the speaker 6 that the vehicle will soon pass over the bump and an impact will occur. If the vehicle still approaches the bump while still at a high speed, the vehicle must decelerate, but before actually decelerating, the vehicle control device 4 (4) sounds a deceleration warning sound from the speaker 6 to notify the driver that control will intervene, and then (5) performs deceleration control using the brake 8. When driving over a bump, the vehicle control device 4 performs (6) control of the suspension 9 to absorb shock in order to prevent a decrease in ride comfort. However, in an environment where the reliability of recognition is reduced as described above, by performing only some of these controls, the impact of erroneous recognition or non-recognition on the ride comfort of the driver or passengers can be reduced.

[0058] FIG. 8 is a diagram illustrating an example of the control contents for each reliability. FIG. 8 illustrates the extent to which visual display, audio output, and control of the vehicle 1 are performed for the user (driver or passenger) for each reliability. When the reliability is 0, that is, in an environment where erroneous detection or non-detection is likely to occur, the vehicle control device 4 does not output any display, audio, or control. On the other hand, when the reliability is 4, that is, when a bump is detected without fail, the vehicle control device 4 not only displays a warning light, but also performs all vehicle controls such as audio output such as a bump detection sound, a caution sound, and a deceleration warning sound, suspension control, acceleration suppression control, and deceleration control. Note that the display, audio, and control when the reliability is high may include the display, audio, and control when the reliability is low.

[0059] Fig. 9 is a diagram illustrating a table for determining the reliability. Fig. 9 shows 11 types of combinations of evaluation items as representative examples, but other combinations of evaluation items may be used. The reliability calculation unit 37 calculates the reliability according to the table shown in Fig. 9.

[0060] Case #1 is a case where a bump is detected by both the first detection unit 32 and the second detection unit 33, the initial bump detection is performed farther than a predetermined distance D, tracking is successful for N or more consecutive times, and the disparity analysis unit 34 and the region analysis unit 35 determine that there is no risk of false detection, and in this case the reliability calculation unit 37 outputs a reliability of 4.

[0061] Case #2 and Case #3 are similar to Case #1, but are different in that the parallax analysis unit 34 and the area analysis unit 35 have determined that there is a risk of false detection. In such a case, if strong vehicle control such as deceleration control is performed, the user's ride comfort will deteriorate if there is a false detection. Therefore, the vehicle control device 4 is limited to performing vehicle control such as displaying a warning light, outputting audio such as a bump detection sound and a warning sound, suspension control, and acceleration suppression control.

[0062] Case #4 and case #5 are examples in which a bump is detected by the first detection unit 32 but not by the second detection unit 33. In case #4, although the second detection unit did not detect a bump, the area analysis unit 35 determined that there was a concern about non-detection, and since there is a possibility that a bump actually exists there, the reliability calculation unit 37 outputs a reliability of 3. On the other hand, in case #5, the second detection unit 33 did not detect a bump, and the area analysis unit 35 determined that there was no concern about non-detection, so there is a low possibility that a bump actually exists there, and the reliability calculation unit 37 outputs a reliability of 2, which is lower than in case #4.

[0063] Cases #6 and #7 are examples in which a bump is not detected by the first detection unit 32 but is continuously detected from a distance by the second detection unit 33. As described above, the second detection unit 33 has the characteristic of being able to detect a bump from a distance using disparity information and shading information as input. Although #6 is not detected by the first detection unit 32, the disparity analysis unit 34 determines that there is a concern about non-detection, and there is a possibility that a bump actually exists there, so the reliability calculation unit 37 outputs a reliability of 3. On the other hand, case #7 is not detected by the first detection unit 32 and the disparity analysis unit 34 determines that there is no concern about non-detection, so there is a low possibility that a bump actually exists there, and the reliability calculation unit 37 outputs a reliability of 2, which is lower than the case of case #6.

[0064] Cases #8 and #9, like Case #6 and Case #7, are examples in which a bump is not detected by the first detection unit 32 but is detected by the second detection unit 33; however, they are different in that the bump is not detected at a distance, but is only detected when it is closer than distance D. For this reason, the reliability of detection is lower than in Case #6 and Case #7, and in Case #8 the reliability calculation unit 37 outputs a reliability of 2, which is lower than in Case #6, and in Case #9 the reliability calculation unit 37 outputs a reliability of 1, which is lower than in Case #7.

[0065] Case #10 is an example where the number of tracking times is small, although the object is detected by the second detection unit 33. In this case, the object is not detected multiple times at different times and positions, and the reliability of the detection is not sufficiently improved, so the reliability calculation unit 37 outputs 0 as the reliability.

[0066] Assuming that a certain bump gradually approaches from a distance, initially, as in case #10, the number of tracking attempts is low, so the reliability calculation unit 37 outputs 0 as a reliability, but as detection continues, the reliability increases as in cases #6 to #9, and when the first detection unit 32 also detects a bump, the reliability increases further as in cases #1 to #3. In this way, the output for the same bump may change over time, and actual control is performed based on this output and the distance to the bump.

[0067] FIG. 10 is a flowchart of the environment recognition process by the environment recognition device 3. In step S100, the arithmetic device 11 receives a pair of image signals from the camera 2. In step S110, the parallax calculation unit 31 calculates parallax information and an invalid area. In step S120, the first detection unit 32 executes the first detection process. The first detection process is a process for detecting candidates for bumps (road structures) present on the road surface based on the parallax information. In the first detection process, the first detection unit 32 calculates a road surface height curve that indicates the road surface height for each distance, extracts the distance and height from the vehicle 1 to the convex part (mountain part) of the curve, and outputs the bump area coordinates in the image if the amount of movement of the vehicle 1 and the amount of change in the distance over time match within a threshold value. The bump detected by the first detection unit 32 becomes the first bump candidate.

[0068] In step S130, the disparity analysis unit 34 executes a disparity analysis process. The disparity analysis process is a process for determining whether or not a situation is likely to result in false detection or non-detection by the first detection unit 32. In the disparity analysis process, the disparity analysis unit 34 sets a horizontal line presence / absence determination flag, a low texture road surface determination flag, and a specular reflection determination flag.

[0069] In step S140, the second detection unit 33 executes the second detection process. The second detection process is a process for detecting candidates for bumps (road structures) on the road surface based on the disparity information and image information (image signal). In the second detection process, the second detection unit 33 receives the disparity and image grayscale value as input, generates a likelihood map of the bump, calculates the activity level for each distance as to whether or not it is a bump based on the likelihood map, moves and accumulates the activity level according to the movement amount of the vehicle 1, calculates the accumulated cumulative activity level, and outputs the bump area coordinates in the image if the area where the cumulative activity level is greater than a predetermined threshold continues within the predetermined threshold range. The bump detected by the second detection unit 33 becomes the second bump candidate.

[0070] In step S150, the region analysis unit 35 executes a region analysis process. The region analysis process is a process for determining whether or not a situation is likely to result in false detection or non-detection by the second detection unit 33. In the region analysis process, the region analysis unit 35 sets a road marking flag, a tree shadow flag, and a correlation flag with other recognition logics.

[0071] In step S160, the measurement processing unit 36 ​​executes the measurement process. The measurement process is a process for calculating the distance to the bump, the height of the bump, and the relative speed between the vehicle 1 and the bump. In the measurement process, when only one of the parallax analysis unit 34 and the region analysis unit 35 detects the bump, that is, when only one of the first bump candidate and the second bump candidate is detected, the measurement processing unit 36 ​​adopts the detection result as it is. When both the parallax analysis unit 34 and the region analysis unit 35 detect the bump, that is, when both the first bump candidate and the second bump candidate are detected, the measurement processing unit 36 ​​checks the detected distance. When the detected distance is farther than a predetermined distance Th1 (>Th2), the measurement processing unit 36 ​​adopts the output of the region analysis unit 35, when the detected distance is closer than the predetermined distance Th2, the measurement processing unit 36 ​​adopts the output of the parallax analysis unit 34, and when the detected distance is between Th1 and Th2, calculates the position of the bump based on the weighted average.

[0072] In step S170, the reliability calculation unit 37 executes a reliability calculation process. The reliability calculation process is a process for calculating the reliability of the bump detection result. In the reliability calculation process, the reliability calculation unit 37 calculates the reliability and outputs it to the vehicle control device 4. The vehicle control device 4 performs vehicle control based on the bump detection result and the reliability.

[0073] According to the above-described first embodiment, the following advantageous effects are achieved.

[0074] (1) The arithmetic device 11 detects a first bump candidate (first road structure candidate) indicating a candidate of a bump (road structure) existing on the road surface captured in the captured image based on disparity information obtained from a plurality of images, detects a second bump candidate (second road structure candidate) indicating a candidate of a bump (road structure) existing on the road surface captured in the image based on the disparity information and the shading information of the image, obtains the position and size of the bump (road structure) based on the detection results regarding the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate), and outputs information regarding the bump (road structure) whose position and size have been obtained to the vehicle control device 4 of the vehicle 1. In this way, appropriate vehicle control can be performed even under conditions in which the detection performance of the road surface condition by the camera image is reduced.

[0075] (2) The calculation device 11 calculates the reliability (measurement reliability) of the position and size of the bump (road structure) according to the detection results for the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate), and outputs control information indicating the control content of the vehicle 1 according to the reliability (measurement reliability) to the vehicle control device 4. In this way, it is possible to perform optimal vehicle control according to the reliability.

[0076] (3) The calculation device 11 calculates the reliability (measurement reliability) based on the distance to the location where at least one of the first bump candidate (first road structure candidate) or the second bump candidate (second road structure candidate) is detected. In this way, it is possible to give a low reliability to bumps that are difficult to detect, such as bumps that are too far away or too close, and a high reliability to bumps that are easy to detect.

[0077] (4) The calculation device 11 calculates the reliability (measurement reliability) based on the number of times at least one of the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate) is detected within a predetermined time. In this way, it is possible to give a low reliability to a bump candidate for which tracking is not functioning well, that is, a bump candidate that is likely not a bump.

[0078] (5) The calculation device 11 calculates the reliability (measurement reliability) based on the degree of agreement between the detection results of the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate). In this way, it is possible to give an appropriate reliability whether or not the two detection results are inconsistent.

[0079] (6) The calculation device 11 calculates the reliability (measurement reliability) based on the imaging conditions of the road surface on which the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate) exist. This makes it possible to eliminate the negative effect of the imaging conditions of the road surface on detection.

[0080] (7) When the calculation device 11 detects only one of the first bump candidate (first road structure candidate) and the second bump candidate (second road structure candidate), the calculation device 11 calculates the distance and size of the bump (road structure) based on the detection result of the bump candidate. In this way, even if the bump can be detected by only one of the detection units, appropriate vehicle control can be performed.

[0081] (8) When the calculation device 11 detects a first bump candidate (first road structure candidate) and a second bump candidate (second road structure candidate), it calculates the distance and size of the bump (road structure) based on the distance to the location where at least one of the first bump candidate (first road structure candidate) or the second bump candidate (second road structure candidate) is detected. In this way, it is possible to perform measurements that make use of the characteristics of each of the two types of detection methods.

[0082] <Second embodiment> An environment recognition device according to a second embodiment of the present invention will be described with reference to Figures 11 and 12. Note that the same reference symbols are used for configurations that are the same as or equivalent to those described in the first embodiment, and differences will be mainly described.

[0083] Fig. 11 is a block diagram similar to Fig. 1, which shows a schematic configuration of a vehicle 100 equipped with an environment recognition device 103 according to a second embodiment. An environment recording device 110 and a communication device 111 are additionally provided in the vehicle 100.

[0084] The environment recognition device 3 according to the first embodiment operates on the vehicle 1 alone, and recognizes the ever-changing surrounding environment of the vehicle 1 at any time based on external information obtained by the camera 2 mounted on the vehicle 1. In addition, the environment recognition device 103 according to the second embodiment keeps records and communicates with the outside of the vehicle 100 as necessary.

[0085] The environment recording device 110 can record the recognition result of the environment recognition device 103 together with the vehicle position output by a locator device (not shown). The locator device (not shown) estimates the vehicle position by receiving, for example, latitude, longitude, and altitude obtained by a GNSS receiver. Alternatively, the environment recording device 110 recognizes the environment around the vehicle 100 using the camera 2 or LiDAR, and estimates the vehicle position by comparing it with an internal map (not shown). The environment recording device 110 can also read out the recognition result and the recorded position that exist in the vicinity of the input vehicle position from the contents recorded with the vehicle position as a query.

[0086] Furthermore, the communication device 111 can communicate with one or more of other vehicles, road facilities, and map servers, and can upload the contents of the environment recording device 110 recorded by the vehicle 100 to other vehicles, road facilities, map servers, etc. Conversely, the communication device 111 can communicate with one or more of other vehicles, road facilities, and map servers, and receive recognition results such as the position and height of a bump observed by a party other than the vehicle 100, and adjust parameters, etc., or override the recognition results based on the received information so that the environment recognition device 103 can easily recognize the bump.

[0087] FIG. 12 is a block diagram similar to FIG. 3, and is a schematic diagram showing the functional configuration of the environment recognition device 103 according to the second embodiment. The environment recognition device 103 according to the second embodiment includes a measurement processing unit 136 and a reliability calculation unit 137 instead of the measurement processing unit 36 ​​and the reliability calculation unit 37. The environment recording device 110 associates the distance to the bump, the height of the bump, the reliability, etc., calculated by the measurement processing unit 136 performing the processing described in the first embodiment with the vehicle position estimated by a locator device (not shown) in a storage device (not shown). The environment recording device 110 reads out the information stored in the storage device (not shown), and when the vehicle position is input from a locator device (not shown), outputs the position and height of the bump in the vicinity within a threshold value of the input vehicle position and the reliability to the measurement processing unit 136 and the reliability calculation unit 137.

[0088] If the reliability is greater than a predetermined threshold, the measurement processing unit 136 replaces the calculated bump position, height, and reliability with the information obtained from the environment recording device 110. In this way, instead of performing detection from a distance several times to improve the reliability, it becomes possible to detect the bump from a greater distance with a higher reliability.

[0089] When the reliability calculation unit 137 has the position of the bump obtained from the environment recording device 110 and the reliability is greater than a predetermined threshold, it changes the threshold N of the number of tracking times in Fig. 9 to a smaller value. This makes it possible to respond to the bump from a greater distance.

[0090] The communication device 111 may communicate with one or more of other vehicles, road facilities, and map servers, and upload or download the contents stored in a storage device (not shown) by the environment recording device 110. This allows the vehicle 100 to detect bumps that it has not detected before, such as bumps on a road that the vehicle 100 is traveling on for the first time, from a greater distance and with a greater degree of reliability. It also allows the vehicle 100 to respond to bumps from a greater distance.

[0091] According to the above-described second embodiment, the following advantageous effects are achieved.

[0092] (1) The calculation device 11 acquires the vehicle position of the vehicle 1, records information about the bump (road structure) in association with the vehicle position when the position and size of the bump (road structure) were obtained, transmits the recorded information about the bump (road structure) to a server outside the vehicle 100, receives from the server information about the bump (road structure) previously measured at the vehicle position by the vehicle 100 or another vehicle, and obtains the position and size of the bump (road structure) based on the received information about the bump (road structure).By doing so, even bumps that the vehicle 100 has not detected before, such as bumps on a road on which the vehicle 100 is traveling for the first time, can be detected from a greater distance with a higher degree of reliability.

[0093] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0094] 1, 100...vehicle, 2...camera, 3, 103...environment recognition device, 4...vehicle control device, 5...display, 6...speaker, 7...accelerator, 8...brake, 9...suspension, 11...arithmetic device, 12...non-volatile memory, 13...volatile memory, 14...input / output interface, 31...parallax calculation unit, 32...first detection unit, 33...second detection unit, 34...parallax analysis unit, 35...area analysis unit, 36, 136...measurement processing unit, 37, 137...reliability calculation unit, 110...environment recording device, 111...communication device

Claims

1. An environment recognition device including a computing device that recognizes an external environment based on a plurality of images acquired by a plurality of imaging devices that capture images of the outside of a vehicle, The computing device includes: Detecting a first road structure candidate that indicates a candidate for a road structure present on the road surface captured in the image based on disparity information obtained from the plurality of images; detecting a second road structure candidate indicative of a candidate for a road structure present on the road surface captured in the image based on the parallax information and the shading information of the image; determining a position and a size of a road structure based on detection results regarding the first road structure candidate and the second road structure candidate; outputting information about the road structure, the position and size of which have been obtained, to a vehicle control device of the vehicle; Environment recognition device.

2. The environment recognition device according to claim 1, The computing device calculates measurement reliability of the position and size of the road structure based on the detection results regarding the first road structure candidate and the second road structure candidate, and outputs control information indicating the control content of the vehicle to the vehicle control device based on the measurement reliability.

3. The environment recognition device according to claim 2, The computing device calculates the measurement reliability based on a distance to a location where at least one of the first road structure candidate or the second road structure candidate is detected.

4. The environment recognition device according to claim 2, The computing device calculates the measurement reliability based on the number of times at least one of the first road structure candidate or the second road structure candidate is detected within a predetermined time.

5. The environment recognition device according to claim 2, The computing device calculates the measurement reliability based on a degree of coincidence between detection results for the first road structure candidate and the second road structure candidate.

6. The environment recognition device according to claim 2, The computing device calculates the measurement reliability based on an imaging condition of a road surface on which the first road structure candidate and the second road structure candidate exist.

7. The environment recognition device according to claim 1, The computing device is an environmental recognition device that, when it detects only one of the first road structure candidate and the second road structure candidate, calculates the distance and size of the road structure based on the detection result of the road structure candidate.

8. The environment recognition device according to claim 1, The computing device is an environmental recognition device that, when it detects the first road structure candidate and the second road structure candidate, calculates the distance and size of the road structure based on the distance to the location where at least one of the first road structure candidate or the second road structure candidate is detected.

9. The environment recognition device according to claim 1, The computing device includes: Acquire the vehicle's own position; The information about the road structure is recorded in association with the vehicle position when the position and size of the road structure are obtained; and Transmitting the recorded information about the road structure to a server outside the vehicle, and receiving information about the road structure previously measured by the vehicle or another vehicle at the vehicle position from the server; An environment recognition device that determines the position and size of the road structure based on the received information about the road structure.

10. Detecting a first road structure candidate that indicates a candidate for a road structure present on a road surface captured in the image based on disparity information obtained from a plurality of images captured by a plurality of imaging devices that capture images of the outside of the vehicle; detecting a second road structure candidate indicative of a candidate for a road structure present on the road surface captured in the image based on the parallax information and the shading information of the image; determining a position and a size of a road structure based on detection results regarding the first road structure candidate and the second road structure candidate; outputting information about the road structure, the position and size of which have been obtained, to a vehicle control device of the vehicle; Environmental recognition method.

Images