Environment recognition device and environment recognition method

Abstract

An environment recognition device recognizes an external environment on the basis of a plurality of images acquired by a plurality of image capturing devices for capturing an image of the outside of a vehicle, and includes an arithmetic device that detects a first road structure candidate representing a candidate of a road structure present on a road surface captured in the image on the basis of parallax information obtained from the plurality of images, and that recognizes the external environment on the basis of the detected first road structure candidate. On the basis of the parallax information and gradation information of the image, a second road structure candidate representing a candidate of a road structure present on the road surface captured in the image is detected, and the position and size of the road structure are determined on the basis of the detection results of the first road structure candidate and the second road structure candidate. And outputs, to a vehicle control device of the vehicle, information relating to the road structure for which the position and size have been determined.

Description

Technical Field

[0001] This invention relates to an environmental identification device and an environmental identification method. Background Technology

[0002] A technique for detecting road surface irregularities from an image captured covering a range including the road surface is known. Patent Document 1 discloses a transmission quantity control device that identifies irregularities on the road surface on which a vehicle travels based on an image captured by an imaging device, and controls the vibration absorption characteristics of the suspension based on the result.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-55757 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] The transfer quantity control device described in Patent Document 1 has the following problem: for example, when the detection performance of road surface condition is reduced due to factors such as weather conditions, lighting conditions, road markings, etc., it cannot perform appropriate control.

[0008] The purpose of this invention is to provide an environmental recognition device that can perform appropriate vehicle control even when the detection performance of road surface conditions detected by images deteriorates.

[0009] Technical means for solving technical problems

[0010] An environmental recognition device according to one aspect of the present invention identifies the external environment based on multiple images acquired by multiple imaging devices for capturing images of the exterior of a vehicle. The device includes a computing unit that, based on disparity information acquired from the multiple images, detects a first road structure candidate representing a road structure present on the road surface captured in the images; detects a second road structure candidate representing a road structure present on the road surface captured in the images based on the disparity information and grayscale information of the images; calculates the position and size of the road structure based on the detection results of the first and second road structure candidates; and outputs the relevant information of the road structure, with its position and size determined, to a vehicle control device of the vehicle.

[0011] In addition, one aspect of the environmental recognition method of the present invention is based on parallax information obtained from multiple images acquired from multiple imaging devices outside the vehicle, detecting a first road structure candidate representing a road structure present on the road surface captured in the images; based on the parallax information and the grayscale information of the images, detecting a second road structure candidate representing a road structure present on the road surface captured in the images; based on the detection results of the first road structure candidate and the second road structure candidate, determining the position and size of the road structure, and outputting the relevant information of the road structure with the determined position and size to the vehicle control device of the vehicle.

[0012] Invention Effects

[0013] According to the present invention, appropriate vehicle control can still be performed even when the detection performance of road surface condition detection via images deteriorates. Attached Figure Description

[0014] Figure 1 This is a block diagram schematically illustrating the structure of a vehicle equipped with the environmental recognition device according to the first embodiment.

[0015] Figure 2 This is a block diagram schematically illustrating the hardware structure of the environmental recognition device according to the first embodiment.

[0016] Figure 3 This is a block diagram schematically illustrating the functional structure of the environmental identification device according to the first embodiment.

[0017] Figure 4 This is a conceptual diagram of a method for determining road surface height.

[0018] Figure 5 It is an explanatory diagram that integrates distance points determined at different times.

[0019] Figure 6 This is a schematic diagram illustrating an example of a likelihood diagram.

[0020] Figure 7 This is a schematic diagram illustrating the application of the detection results of protrusions to driver assistance or autonomous driving systems.

[0021] Figure 8 This is a diagram illustrating the control measures for each level of reliability.

[0022] Figure 9 This is a diagram illustrating a table used to determine reliability.

[0023] Figure 10 This is a flowchart of the environmental identification process performed by the environmental identification device.

[0024] Figure 11This is a block diagram schematically illustrating the structure of a vehicle equipped with the environmental recognition device according to the second embodiment.

[0025] Figure 12 This is a block diagram schematically illustrating the functional structure of the environmental identification device according to the second embodiment. Detailed Implementation

[0026] <First Implementation>

[0027] Reference Figures 1-10 This invention describes the environmental identification device according to the first embodiment of the present invention.

[0028] Figure 1 This is a block diagram schematically illustrating the structure of a vehicle 1 equipped with the environmental recognition device 3 according to the first embodiment. The vehicle 1 includes a first camera 2a, a second camera 2b, the environmental recognition device 3, a vehicle control device 4, a display 5, a speaker 6, an accelerator 7, a brake 8, and a suspension 9.

[0029] The first camera 2a and the second camera 2b capture the field of view including the direction of travel (forward) of the vehicle 1. In the following description, the first camera 2a and the second camera 2b may be collectively referred to as camera 2.

[0030] The first camera 2a and the second camera 2b generate images of the area in front of the vehicle and output them to the environment recognition device 3. The environment recognition device 3 detects bumps on the road surface in front of the vehicle 1 based on the pair of images. The environment recognition device 3 calculates the distance to the detected bump, the height of the detected bump, the relative speed between the vehicle 1 and the detected bump, the reliability of the detection, etc., and outputs the calculation results to the vehicle control device 4. The vehicle control device 4 determines the magnitude of the impact of the bump on the vehicle 1 and the degree of danger of a collision between the bump and the vehicle 1 based on these calculation results. According to the determination result, the vehicle control device 4 implements controls such as displaying a warning message on the display 5, turning on the warning lights on the display 5, playing a warning sound from the speaker 6, modulating or suppressing acceleration with the accelerator 7, decelerating with the brakes 8, and changing the shock absorption characteristics of the suspension 9.

[0031] Figure 2This is a block diagram schematically illustrating the hardware structure of the environment identification device 3 according to the first embodiment. The environment identification device 3 is composed of a computer, which includes arithmetic units 11 such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), and DSP (Digital Signal Processor); non-volatile memory 12 such as ROM (Read Only Memory), flash memory, or hard disk drive; volatile memory 13 called RAM (Random Access Memory); input / output interfaces 14; and other peripheral circuits. This hardware works together to enable software operation, realizing various functions. The environment identification device 3 can be composed of a single computer or multiple computers. Furthermore, the arithmetic unit 11 can be an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or similar components.

[0032] The non-volatile memory 12 stores programs capable of performing various operations. That is, the non-volatile memory 12 is a storage medium (storage device) capable of reading programs that implement the functions of this embodiment. The volatile memory 13 is a storage medium (storage device) that temporarily stores the operation results of the arithmetic unit 11 and signals input from the input / output interface 14. The arithmetic unit 11 is a device that expands the program stored in the non-volatile memory 12 into the volatile memory 13 and performs operations, performing prescribed arithmetic processing on data obtained from the input / output interface 14, the non-volatile memory 12, and the volatile memory 13 according to the program.

[0033] The environmental recognition device 3 is connected to the first camera 2a, the second camera 2b, and the vehicle control device 4, respectively. The input section of the input / output interface 14 converts signals input from various devices (first camera 2a, second camera 2b, etc.) into data that can be processed by the computing device 11. In addition, the output section of the input / output interface 14 generates a signal for output based on the processing result of the computing device 11, and outputs the signal to various devices (vehicle control device 4, etc.).

[0034] The first camera 2a includes an image capturing element 21a and a DSP 22a. Similarly, the second camera 2b includes an image capturing element 21b and a DSP 22b. In the following description, image capturing element 21a and image capturing element 21b may be collectively referred to as image capturing element 21, and DSP 22a and DSP 22b may be collectively referred to as DSP 22.

[0035] The capturing element 21 includes photodiodes arranged in a grid pattern. The capturing element 21 outputs a capture signal of the captured image of the subject to the DSP 22. The DSP 22 performs image quality corrections such as contrast correction, gamma correction, and edge correction on the capture signal input from the capturing element 21, while simultaneously converting it into an image signal. The first camera 2a outputs the image signal generated by the DSP 22a to the environment recognition device 3. The second camera 2b outputs the image signal generated by the DSP 22b to the environment recognition device 3.

[0036] By setting input / output synchronization signals on the first camera 2a and the second camera 2b, a synchronization signal from the imaging element 21 of one camera 2 can be input to the imaging element 21 of the other camera 2, or the shutter timing of the imaging elements 21 of the two cameras 2 can be synchronized based on the signal output from the environment recognition device 3. This improves the calculation accuracy of the parallax calculation unit 31, which will be described later.

[0037] Furthermore, the image quality correction parameters of the DSP 22 of both cameras 2 can be adjusted simultaneously based on the signal output from the environmental recognition device 3. This reduces overexposure or underexposure of the image signal and improves the parallax calculation density of the parallax calculation unit 31, which will be described later.

[0038] Figure 3 This is a block diagram schematically illustrating the functional structure of the environmental identification device 3 according to the first embodiment. The environmental identification device 3 includes a parallax calculation unit 31, a first detection unit 32, a second detection unit 33, a parallax analysis unit 34, a region analysis unit 35, a measurement processing unit 36, and a reliability calculation unit 37. These functional units are connected via a computing device 11 ( Figure 2 ) Execution stored in non-volatile memory 12 ( Figure 2 It is implemented in software as a program in ) .

[0039] The disparity calculation unit 31 acquires captured images from the camera 2 and calculates disparity information (distance image) and invalid regions. The first detection unit 32 detects bumps on the road surface captured in the captured images based on the disparity information obtained from multiple captured images. The disparity analysis unit 34 determines whether the first detection unit 32 is in a state prone to false detection or non-detection, and generates a flag indicating this state. The second detection unit 33 detects bumps on the road surface captured in the captured images based on the disparity information and the grayscale information of the captured images. The region analysis unit 35 determines whether the second detection unit 33 is in a state prone to false detection or non-detection, and generates a flag indicating this state. The measurement processing unit 36 ​​calculates the position and size of the bumps based on the bump detection results of the first detection unit 32 and the second detection unit 33. The reliability calculation unit 37 calculates a reliability score, which represents the reliability of the position and size of the bumps calculated by the measurement processing unit 36.

[0040] (Explanation of parallax calculation and processing)

[0041] The disparity calculation process performed by the disparity calculation unit 31 will be explained. The disparity calculation unit 31 uses a pair of image signals (captured images) output from the first camera 2a and the second camera 2b to calculate disparity information (distance image) and invalid regions using known methods. The distance image, also known as a depth map, is an image that uses distance information (depth magnitude) as grayscale. Invalid regions refer to areas in the entire distance image where the distance cannot be calculated. Various methods for calculating the distance image and invalid regions from a pair of captured images are known, such as those using epipolar geometry or CNN (Convolutional Neural Network) methods, and therefore will not be described further.

[0042] (Explanation of the first detection process)

[0043] The first detection process performed by the first detection unit 32 will be described. This first detection process primarily detects bumps on the road surface based on parallax. The first detection unit 32 obtains parallax information and invalid areas from the parallax calculation unit 31, and extracts parallax information for the area ahead corresponding to the travel path of the vehicle 1. Based on the extracted parallax information, the first detection unit 32 calculates the road surface height for each distance. (The text then repeats itself, so the translation will only include the first instance.) Figure 4 and Figure 5 Explain the calculation method.

[0044] Figure 4 This is a conceptual diagram of a method for determining road surface height. Now, as... Figure 4As shown in (a), assume that vehicle 1 is traveling on a generally flat road surface at a certain time t0, without encountering any local bumps or depressions. At this time, the road surface 301 can be estimated based on the design geometry of vehicle 1. The first detection unit 32 utilizes the difference between the placement positions of the first camera 2a and the second camera 2b mounted on vehicle 1 to detect the positions of these cameras 2 when they observe the same subject on the shooting plane, and calculates their positional difference, i.e., parallax. The first detection unit 32 converts this calculated parallax into the distance from the camera 2 to the point of interest, thereby determining... Figure 4 Distance points 302, 303, and 304 are shown.

[0045] If there are no bumps on the road surface where vehicle 1 travels, i.e., the road surface is flat, then these distance points 302-304 exist on the presumed road surface 301. On the other hand, in cases such as Figure 4 In the case shown in (a), where a protrusion exists, there is a distance point like distance point 304, which is not located on the estimated road surface 301, but rather at a position offset from the estimated road surface 301. In the first detection process, when a vertical line is drawn downwards from distance point 304 to the estimated road surface 301, the length of this vertical line is considered as the road surface height at distance point 304. The first detection unit 32 calculates the road surface height at each distance point. The first detection unit 32 estimates the road surface height curve based on the road surface height calculated at each distance point by, for example, fitting a curve such as a Bézier curve or a spline curve to a point cloud composed of multiple determined distance points.

[0046] like Figure 4 As shown in (a), the distance point density relative to the distance to vehicle 1 varies with the distance to camera 2 and the shape of the road surface. The distance to vehicle 1 referred to here is the distance measured parallel to the estimated road surface 301 directly below a predetermined reference point (e.g., the ground contact point assuming vehicle 1 is mounted on a horizontal plane, the front end of vehicle 1's bumper, etc.). The distance point density relative to the distance to vehicle 1 generally becomes sparser as the distance to vehicle 1 increases, and denser as the distance to vehicle 1 decreases. This is because, in the captured image, the farther away a location is, the smaller the displayed area; therefore, even if the distance points are evenly distributed across the captured image, the number of distant distance points in three-dimensional space will decrease. Furthermore, the distance point density also varies due to occlusion caused by road surface irregularities. For example, if there is a convex part on the road surface, the road surface behind the convex part is obscured and cannot be seen by camera 2 (so-called occlusion). Therefore, the distance point density decreases behind such a convex part.

[0047] For the reasons stated above, the accuracy of the road surface height estimation by the first detection unit 32 may decrease when the density of distance points decreases. Therefore, distance points determined at multiple times can be aggregated into a point cloud, and the aforementioned curve can be fitted to this point cloud. This increases the density of distance points, enabling high-precision estimation of the road surface height curve.

[0048] Figure 5 This is an explanatory diagram that integrates distance points determined at different times. Figure 5 In the figures (a) to (e), the horizontal axis represents the horizontal distance from vehicle 1 to the distance point, and the vertical axis represents the vertical distance from the estimated road surface 301 to the distance point. Figure 4 The distance point determined at time t0 shown in (a) is... Figure 5 (a) indicates that, in Figure 4 (b) The distance point determined at time t1 is used Figure 5 (b) indicates that, in Figure 4 The distance point determined at time t2 shown in (c) is... Figure 5 (c) indicates that, assuming the shape of the road surface on which vehicle 1 travels remains constant between times t0 and t2, and the speed or distance traveled by vehicle 1 is known, then at the current time t2, the distance points determined at past times t0 and t1 can be integrated into the distance point determined at the current time t2. For example, by shifting the horizontal position of the distance points at each time by the difference in the horizontal position of vehicle 1 at each time, the distance points at each time can be plotted, as shown in the diagram. Figure 5 As shown in (d).

[0049] In addition, regarding Figure 4 The road surface 301 estimated in the model assumes that vehicle 1 is on a flat road surface and does not experience instantaneous body sway. However, when there are small bumps and dips in the road surface, the pitch angle and roll angle of vehicle 1 may change constantly. In this case, if the road surface estimated based on the geometric values ​​of vehicle 1's design is used, problems may arise such as not obtaining an appropriate road surface height or not properly integrating multiple distance points determined based on disparity information at different times. Therefore, when estimating the road surface, the attitude or angle of vehicle 1 relative to the road surface can be considered. For example, road surface 301 can be estimated by performing a planar approximation on distance points obtained based on disparity information outside of invalid areas, or by using the contact between the detected objects such as vehicle 1 or pedestrians and the road surface. Alternatively, road surface 301 can be estimated by extracting road markings such as white lines representing the boundaries of the driving path. In this way, a high-density point cloud along the road surface shape can be obtained, enabling high-precision detection of road surface height.

[0050] Furthermore, although it has been explained that when estimating the road surface height curve, the point cloud is fitted with curves such as Bézier curves or spline curves, different curve fitting methods can also be implemented. The input point cloud is defined as p(i), the discrete points on the output curve are defined as y(i), the slope of the approximate straight line × the length from i-1 to i is defined as LS(i), the discrete points that have moved in the past according to the travel of vehicle 1 are defined as Y'(i), and the energy evaluation function Eall is defined as Equation (1).

[0051] [Mathematical Expression 1]

[0052] By finding Y'(i) that minimizes the energy evaluation function Eall, curve fitting can be performed. This allows for obtaining curve fitting results that are close to the input point, have a first derivative value (slope) close to the road surface inclination, a small second derivative value (change in slope), and high consistency with the speed of vehicle 1. For example, it allows for obtaining curve fitting results for… Figure 5 The point cloud of (d) was processed. Figure 5 (e) The fitted curve is like curve 401.

[0053] After estimating the road surface height curve as described above, the first detection unit 32 extracts the distance and height from the reference point of vehicle 1 to the convex part (peak) of the road surface height curve based on the road surface height curve. The first detection unit 32 extracts the coordinates with a maximum value in the road surface height curve (i.e., the candidate position of the convexity) and sets it as the distance to the convex part of the curve. A perpendicular line is drawn from the coordinates with the maximum value to the road surface plane, and the first detection unit 32 sets the length of the perpendicular line as the height of the convex part of the curve.

[0054] When the distance and height are both within a specified range (i.e., the protrusion is neither too far from the vehicle 1 nor too close to the road surface), the first detection unit 32 sets the tracking point (i.e., the candidate point of the protrusion) as the coordinate with a maximum value in the road surface height curve and tracks the tracking point. For example, by using optical flow, the tracking point is tracked in a time series, and the distance change of the tracking point is calculated based on the disparity, thus calculating the movement of the vehicle 1. To improve stability, multiple tracking points can be tracked, and the movement of the vehicle 1 can be calculated based on the average disparity of these tracking points. If the movement of the vehicle 1 calculated in this way is substantially equal to the distance change of the tracking point (if the difference is within a specified threshold), a tracking success flag indicating that the protrusion has been correctly tracked is set. If not, it is considered that the detected candidate point of the protrusion is not actually a protrusion, and no tracking success flag is set.

[0055] The first detection unit 32 can also acquire information such as the steering angle and speed of vehicle 1 from a network used for vehicle information communication, such as CAN (Controller Area Network) (not shown). Therefore, for example, the first detection unit 32 can calculate the vehicle's travel path and movement amount from the steering angle, speed, etc., using a known dead reckoning method. As a result, since parallax information of the area corresponding to the vehicle's travel path can be extracted, the accuracy of road height calculation and the accuracy of vehicle 1 movement estimation are improved. Furthermore, since calculations such as optical flow methods can be omitted, computational costs are reduced.

[0056] (Explanation of parallax analysis processing in parallax analysis unit 34)

[0057] The parallax calculation process performed by the parallax analysis unit 34 will be explained. The parallax analysis unit 34 determines whether the first detection unit 32 is in a situation where it is difficult to correctly detect protrusions (for example, it is easy to misdetect non-existent protrusions, or it is difficult to detect existing protrusions), and generates a flag indicating the situation.

[0058] The disparity analysis unit 34 generates a flag to detect false protrusions based on erroneous disparity calculations. Due to errors in the visual axis and viewing angle of each of the multiple cameras 2 functioning as stereo cameras, erroneous disparities may occur. For example, multiple cameras 2 arranged horizontally may calculate incorrect disparities for horizontal lines within an image. Therefore, the disparity analysis unit 34 calculates edge angles for the image captured by one of the cameras 2. If the coordinates of a region where the horizontal component is stronger than a predetermined threshold and the vertical component is weaker than a predetermined threshold coincide with the coordinates of a protrusion region detected by the first detection unit 32 within the threshold, it considers that a false detection may have occurred due to erroneous disparity and sets a flag indicating the presence or absence of a horizontal line.

[0059] When cameras 2 are set up separately in the vertical direction, the same parallax error may occur for vertical line segments. Therefore, in this case, the same processing and the same markings can be applied to the vertical line segments.

[0060] The parallax analysis unit 34 generates a marker to determine whether the road surface is a low-contrast or low-texture surface. For example, in environments where the contrast of the image obtained from camera 2 is reduced, such as dark environments, foggy environments, or uniformly paved roads, the effective parallax area decreases due to the increase in the invalid area when calculating parallax. Therefore, the detection performance for protrusions at a distance decreases, and the protrusion may be detected suddenly when it approaches vehicle 1. Therefore, the parallax analysis unit 34 sets multiple small regions (e.g., pre-set small regions for the entire image, such that small regions of a specified size are arranged without overlapping) for the image captured by at least one camera 2, and calculates the statistical quantities such as the average value, variance, and moments of the brightness values ​​in each small region. When these statistical quantities are within a specified threshold range, the parallax analysis unit 34 sets a low-texture road surface determination marker. For example, on dark rural roads with little lighting such as streetlights, the average value and variance of the brightness values ​​in the captured image become smaller, making it difficult to detect unevenness of the road surface at a distance. The threshold used to determine the low-texture road surface determination marker is set according to such an environment.

[0061] The parallax analysis unit 34 generates a marker to determine if the specular reflection component of the road surface is strong. For example, in situations where the paved road surface is wet, such as during or after rain, or when driving on a poorly drained underground passage, the image captured by camera 2 may show the sky or surrounding structures reflected by the road surface rather than the road surface itself. In such environments, the parallax calculated by the parallax calculation unit 31 may not represent the shape of the road surface, thus reducing the detection performance of the first detection unit 32 for protrusions, potentially leading to false detection or non-detection of protrusions. Therefore, when the number of distance points located below the driving plane of vehicle 1 (i.e., where the road surface height becomes negative), calculated based on the camera geometry model, exceeds a predetermined threshold, the parallax analysis unit 34 considers the reliability of the protrusion detection results to be low and sets a specular reflection determination marker.

[0062] (Explanation of the second inspection process in the second inspection section 33)

[0063] The second detection process performed by the second detection unit 33 will be described. This second detection process primarily detects bumps on the road surface based on region segmentation. The second detection unit 33 generates a likelihood map of the bumps based on disparity information and images (grayscale images) captured by at least one camera 2. The likelihood map can be generated using, for example, semantic segmentation using a CNN. In this case, a model can be pre-learned that takes disparity information of H pixels vertically and W pixels horizontally, and images of H pixels vertically and W pixels horizontally as input, and outputs a likelihood map of H pixels vertically and W pixels horizontally. This CNN technique and semantic segmentation technique are well known, so their description is omitted. In the following description, the likelihood map has a likelihood value of 0 to 1 for each pixel location (1 indicates the highest probability of a bump).

[0064] Figure 6 This is a schematic diagram illustrating an example of a likelihood diagram. Figure 6 (a) is a captured image (grayscale image). Figure 6 (b) is the positional relationship between vehicle 1 and the road surface as viewed from the side. Figure 6 (c) is parallax information. Figure 6 (d) is a likelihood diagram. For example... Figure 6 (a) and Figure 6 As shown in (b), there is a flat road surface 501 in front of vehicle 100, and a protrusion 502 exists between vehicle 100 and another vehicle 503 traveling in front of it. At this time, as... Figure 6 As shown in (c), parallax also appears in the area of ​​flat road surface 501, and the parallax changes smoothly from the front side to the inside side of the image. Figure 6 In the likelihood plot of (d), region 504, which contains the protrusion 502, has the highest likelihood value. Region 505, which includes the area in front of and inside the protrusion, has the second highest likelihood value. The surrounding region 506 has the lowest likelihood value. In addition, the likelihood value of other regions is zero.

[0065] The input to the second detection unit 33 can be disparity information of a certain size and a grayscale image with the same resolution as the disparity information. Although the disparity information calculated by a stereo camera consisting of two cameras 2 with the same viewpoint and resolution has the same resolution as the grayscale image of one camera 2 constituting the stereo camera, by concatenating the corresponding regions of these disparity information and grayscale images and inputting them into the CNN, performance degradation can be suppressed for objects that are difficult to separate in the learning data. As an example of such objects that are difficult to separate in the learning data, there are raised and flat rough surfaces that exist together with rough surfaces. Thus, in scenes where the textures on the grayscale images are similar but the presence of raised parts in the disparity information is different, performance degradation can be suppressed.

[0066] The second detection unit 33 uses a likelihood map to calculate the activity level of whether a protrusion exists for each distance. For example, using a likelihood map of vertical H pixels and horizontal W pixels as input, the activity level of vertical H pixels is output. Since protrusions are usually set orthogonally to the driving path, the activity level can be calculated by accumulating the likelihood map along the horizontal direction, thereby improving the stability of protrusion detection. The specific calculation is as follows. Let L(x, y) be the likelihood value at position (x, y) on the likelihood map, and let Sy be the area in front corresponding to the driving path of vehicle 1 at each y coordinate. Then the activity level A(y) is calculated by the following formula (2). When the activity level A(y) is above a specified threshold, the second detection unit 33 determines that there is a protrusion area at that y coordinate.

[0067] [Mathematical Expression 2]

[0068] Here, f(y) is the correction coefficient corresponding to each y-coordinate. It is set to 0 for areas above the horizon, smaller for areas below the horizon near vehicle 1 (lower part of the image), and larger for areas far from vehicle 1 (upper part of the image). That is, when the second detection unit 33 finds many areas with high likelihood values ​​near vehicle 1, it determines that a protruding area exists. Conversely, when there are fewer areas with high likelihood values ​​in the distance than nearby areas, the second detection unit 33 also easily determines that a protruding area exists. Based on the camera's geometric model, the second detection unit 33 converts the calculated activity from the image coordinate system to a distance in actual space.

[0069] The second detection unit 33 moves and accumulates activity based on the movement of vehicle 1, and calculates the accumulated activity. Assuming the activity at a distance d at time t1 is A(t1, d), the activity at a distance d at time t2 is A(t2, d), and vehicle 1 moves Dt2 from time t1 to time t2, the accumulated activity IA(t2, d) at a distance d at time t2 is calculated by the following formula (3). Generally, since the protruding area has a width of about 30cm to 80cm in the depth direction, when the area where the accumulated activity IA(t, d) is greater than a specified threshold continues within the specified threshold range (e.g., 30cm to 80cm), the second detection unit 33 determines that a protruding area exists.

[0070] [Mathematical Expression 3]

[0071] (Explanation of regional analysis processing in Regional Analysis Department 35)

[0072] The area analysis process performed by the area analysis unit 35 will be explained. The area analysis unit 35 determines whether the second detection unit 33 is in a situation where it is difficult to correctly detect protrusions (e.g., it is easy to misdetect non-existent protrusions, or it is difficult to detect existing protrusions), and generates a flag indicating the situation.

[0073] The region analysis unit 35 generates signs to detect false bumps based on road markings. The CNN-based region segmentation uses a CNN model, which is pre-learned using pairs of grayscale images, disparity images, and training information (or label images). However, it is known that there are data points in this pre-learned data that are difficult to separate. There are locations with specific road markings and bumps, and locations where specific road markings are drawn but the road surface is flat. When such road markings are present, the second detection unit 33 may falsely detect or fail to detect the bump. The region analysis unit 35 identifies such road markings and sets signs indicating road markings that may be falsely detected or failed to detect. For example, the region analysis unit 35 calculates the edge angle of the grayscale image of the bump region output by the second detection unit 33. When the ratio of each element in the histogram of the edge angle is within a specified threshold range, the road marking is set to valid.

[0074] The region analysis unit 35 generates markers based on tree shadows to identify false detections of protrusions. For example, when complex shadows, such as sunlight filtering through leaves, are cast on the road surface, parallax errors may occur due to the influence of these shadows, and the grayscale image patterns learned by the CNN model beforehand may not contain such shadows, potentially leading to false or missed detections of protrusions. The region analysis unit 35 identifies such complex tree shadows and sets markers indicating potentially undetected tree shadows. For example, the region analysis unit 35 calculates edge angles on the grayscale image of the protrusion region output by the second detection unit 33, and sets the tree shadow markers as valid when the histogram of its edge angles is flat within a specified threshold. For example, the edge angles can be calculated by applying a Sobel filter to each pixel in the grayscale image of the protrusion region.

[0075] The region analysis unit 35 generates markers to detect false protrusions based on other three-dimensional objects such as vehicles or pedestrians. Even when the driving path of vehicle 1 is on a flat road surface, for example, if other three-dimensional objects such as vehicles or pedestrians are present nearby, the calculation of road surface parallax may be affected by these three-dimensional objects, and when calculating the unevenness of the driving path of vehicle 1, unevenness that does not actually exist may be calculated. In order to prevent false protrusion detection due to the influence of such erroneous parallax, when the recognition results of other three-dimensional objects such as vehicle detection or pedestrian detection are within the threshold of the likelihood map calculation area, the region analysis unit 35 sets the markers related to other recognition logic to be valid.

[0076] (Explanation of measurement processing in measurement processing unit 36)

[0077] The measurement processing performed by the measurement processing unit 36 ​​will be explained. The measurement processing unit 36 ​​calculates the distance to the protrusion, the height of the protrusion, and the relative speed of the protrusion as observed from the vehicle 1 based on the protrusion detection results performed by the first detection unit 32, the protrusion detection results performed by the second detection unit 33, the mark generated by the parallax analysis unit 34, and the mark generated by the area analysis unit 35.

[0078] Based on the principle of stereo cameras, parallax is difficult to detect at distances. Therefore, the performance of bulge detection based on parallax analysis (bulge detection performed by the first detection unit 32) tends to decrease at distances. Conversely, bulge detection based on region analysis (bulge detection performed by the second detection unit 33) can detect bulges with high accuracy even at distances, but the accuracy of detecting the distance to the bulge and the height of the bulge tends to decrease. Therefore, when both the first detection unit 32 and the second detection unit 33 detect a bulge, if the distance to the detected bulge is greater than a predetermined distance Th1, the measurement processing unit 36 ​​uses the detection result of the second detection unit 33; if the distance to the detected bulge is less than a predetermined distance Th2, the measurement processing unit 36 ​​uses the detection result of the first detection unit 32. When the distance to the detected bulge is less than Th1 and greater than Th2, the measurement processing unit 36 ​​calculates a weighted average of the detection results of the first detection unit 32 and the second detection unit 33, and uses this calculation result as the distance to the bulge or the height of the bulge. Here, the relationship between the two thresholds Th1 and Th2 is Th1 > Th2.

[0079] When only one of the first detection unit 32 and the second detection unit 33 detects a protrusion, the measurement processing unit 36 ​​directly uses the detection result of that detection unit.

[0080] (Explanation of reliability calculation processing in reliability calculation unit 37)

[0081] The reliability calculation process performed by the reliability calculation unit 37 will be explained. The reliability calculation unit 37 calculates the reliability of the bump detection result adopted by the measurement processing unit 36 ​​and determines which type of control the detection result can be used for. Image recognition-based recognition of the vehicle's surrounding environment still has the possibility of false detection or non-detection due to driving or lighting conditions, making it difficult to achieve consistently reliable control. Therefore, by calculating the reliability of image recognition-based recognition of the vehicle's surrounding environment as an evaluation index, the vehicle control device 4 can perform appropriate control corresponding to the reliability of the recognition.

[0082] Figure 7This is a conceptual diagram of applying bump detection results to a driver assistance or autonomous driving system. If a bump is detected at a sufficiently far distance, the vehicle control unit 4 first uses the display 5 to alert the driver or passenger (1). When the vehicle speed reaches a certain level, the vehicle control unit 4 performs acceleration suppression control on the accelerator 7 to suppress further acceleration (2). Furthermore, when the vehicle approaches the bump at high speed, the vehicle control unit 4 notifies the driver of the impending impact by emitting an alarm sound from the speaker 6 (3). When the vehicle is still approaching the bump at high speed and needs to decelerate, before actually decelerating, the vehicle control unit 4 notifies the driver of intervention control by emitting a deceleration warning sound from the speaker 6 (4), and then uses the brakes 8 for deceleration control (5). When driving over the bump, the vehicle control unit 4 controls the shock-absorbing suspension 9 (6) to prevent a decrease in ride comfort. However, as mentioned above, in environments where the reliability of identification is reduced, implementing only a few of these measures can reduce the impact of false or unidentified identification on the ride comfort of the driver or passenger.

[0083] Figure 8 This is a diagram illustrating the control measures for each level of reliability. In Figure 8 The diagram illustrates the stage at which visual displays, audio outputs, and vehicle control are implemented for the user (driver or passenger) at each level of reliability. At a reliability of 0, i.e., in an environment prone to false detection or non-detection, vehicle control unit 4 does not output any of the displays, audio outputs, or controls. On the other hand, when the reliability is 4, i.e., when a bump is reliably detected, vehicle control unit 4 not only displays a warning light, but also outputs audio such as a bump detection sound, an attention-awakening sound, and a deceleration warning sound, as well as performs all vehicle controls including suspension control, acceleration suppression control, and deceleration control. The displays, audio outputs, and controls at higher reliability levels may include those at lower reliability levels.

[0084] Figure 9 This is a diagram illustrating a table used to determine reliability. Figure 9 The document provides representative examples of 11 combinations of evaluation items, but other combinations are also possible. The reliability calculation unit 37 is based on... Figure 9 The table shown calculates reliability.

[0085] Example #1 indicates the following situation: both the first detection unit 32 and the second detection unit 33 detect the protrusion. The protrusion is detected for the first time at a distance greater than the specified distance D. After successful tracking more than N times, the parallax analysis unit 34 and the area analysis unit 35 determine that there is no possibility of false detection. In this case, the reliability calculation unit 37 outputs 4 as the reliability.

[0086] Examples #2 and #3 are similar to Example #1, except that the parallax analysis unit 34 or the area analysis unit 35 determines that false detections may occur. In this case, if strong vehicle control such as deceleration control is implemented, the user's ride comfort will be reduced in the event of a false detection. Therefore, the vehicle control device 4 only implements vehicle control such as warning light display, bulge detection sound and attention-raising sound, suspension control, and acceleration suppression control.

[0087] Examples #4 and #5 are instances where the first detection unit 32 detects a protrusion, but the second detection unit 33 does not. In example #4, although the second detection unit does not detect the protrusion, the area analysis unit 35 determines that there is a possibility of it not being detected, thus implying that a protrusion may actually exist at that location. Therefore, the reliability calculation unit 37 outputs a reliability score of 3. On the other hand, in example #5, the second detection unit 33 does not detect the protrusion, and the area analysis unit 35 determines that there is no possibility of it not being detected. Therefore, the probability that a protrusion actually exists at that location is low, and the reliability calculation unit 37 outputs a reliability score of 2, which is lower than that in example #4.

[0088] Examples #6 and #7 are examples where the first detection unit 32 failed to detect the bump, but the second detection unit 33 continued to detect it from a distance. As described above, the second detection unit 33 has the characteristic of being able to detect bumps from a distance using parallax information and grayscale information as input. In #6, the first detection unit did not detect the bump, but the parallax analysis unit 34 determined that there was a possibility that it was not detected and that a bump might actually exist there, so the reliability calculation unit 37 output 3 as the reliability. On the other hand, in example #7, the first detection unit 32 did not detect the bump, and the parallax analysis unit 34 determined that there was no possibility that it was not detected, so the possibility that a bump actually exists was low, and the reliability calculation unit 37 output 2 as the reliability, which is lower than in example #6.

[0089] Similar to Examples #6 and #7, Examples #8 and #9 are examples where the first detection unit 32 failed to detect the protrusion, but the second detection unit 33 did. However, the difference lies in that the protrusion was not detected at a distance, but was initially detected closer to the distance D. Therefore, the reliability of the detection is lower than that of Examples #6 and #7. In Example #8, the reliability calculation unit 37 outputs a reliability of 2, which is lower than that of Example #6, and in Example #9, the reliability calculation unit 37 outputs a reliability of 1, which is lower than that of Example #7.

[0090] Example #10 is an example where the second detection unit 33 detected a problem but tracked it a limited number of times. In this case, since the detection was not performed multiple times at different times and locations, the reliability of the detection was not sufficiently improved, so the reliability calculation unit 37 outputs 0 as the reliability score.

[0091] Suppose a protrusion gradually approaches from a distance. Due to the initially low number of tracking attempts, as in example #10, the reliability calculation unit 37 outputs 0 as the reliability. However, if it is continuously detected thereafter, the reliability increases as in examples #6 to #9. Furthermore, when the first detection unit 32 also detects the protrusion, as in examples #1 to #3, the reliability increases further. Thus, the output for the same protrusion can vary over time, and actual control can be performed based on this output and the distance to the protrusion.

[0092] Figure 10 This is a flowchart of the environmental recognition process performed by the environmental recognition device 3. In step S100, the computing device 11 receives a pair of image signals from the camera 2. In step S110, the disparity calculation unit 31 calculates disparity information and invalid regions. In step S120, the first detection unit 32 performs a first detection process. The first detection process is a process of detecting candidates for protrusions (road structures) on the road surface based on disparity information. In the first detection process, the first detection unit 32 calculates a road height curve representing the road surface height at each distance, extracts the distance and height from the vehicle 1 to the protrusion (peak) of the curve, and if the time series change of the vehicle 1's movement and distance is consistent within a threshold, the coordinates of the protrusion region in the image are output. The protrusions detected by the first detection unit 32 become first protrusion candidates.

[0093] In step S130, the parallax analysis unit 34 performs parallax analysis processing. Parallax analysis processing determines whether the first detection unit 32 is in a state prone to false detection or non-detection. In parallax analysis processing, the parallax analysis unit 34 sets a horizontal line presence / absence determination mark, a low-texture road surface determination mark, and a specular reflection determination mark.

[0094] In step S140, the second detection unit 33 performs a second detection process. This second detection process detects candidate protrusions (road structures) on the road surface based on disparity information and image information (image signal). In this process, the second detection unit 33 uses disparity and image grayscale values ​​as input to generate a likelihood map of the protrusion. Based on the likelihood map, it calculates the activity level of each distance to determine if it is a protrusion. It moves according to the movement of vehicle 1 and accumulates the activity level, calculating the accumulated activity level. If a region with an accumulated activity level exceeding a predetermined threshold persists within that threshold range, the coordinates of the protrusion region in the image are output. The protrusions detected by the second detection unit 33 become second protrusion candidates.

[0095] In step S150, the area analysis unit 35 performs area analysis processing. Area analysis processing determines whether the second detection unit 33 is in a state prone to false detection or non-detection. During area analysis processing, the area analysis unit 35 sets road markings, tree shadow markers, and markers related to other identification logic.

[0096] In step S160, the measurement processing unit 36 ​​performs measurement processing. Measurement processing involves calculating the distance to the protrusion, the height of the protrusion, and the relative speed between the vehicle 1 and the protrusion. During measurement processing, if only one of the parallax analysis unit 34 or the region analysis unit 35 detects the protrusion (i.e., only one of the first protrusion candidate and the second protrusion candidate is detected), the measurement processing unit 36 ​​directly uses its detection result. If both the parallax analysis unit 34 and the region analysis unit 35 detect the protrusion (i.e., both the first and second protrusion candidates are detected), the measurement processing unit 36 ​​confirms the detected distance. When the detected distance is greater than a predetermined distance Th1 (>Th2), the measurement processing unit 36 ​​uses the output of the region analysis unit 35; when the detected distance is less than the predetermined distance Th2, the measurement processing unit 36 ​​uses the output of the parallax analysis unit 34; and when the detected distance is between Th1 and Th2, the position of the protrusion is calculated based on a weighted average.

[0097] In step S170, the reliability calculation unit 37 performs a reliability calculation process. This process calculates the reliability of the bump detection result. During the reliability calculation process, the reliability calculation unit 37 calculates the reliability and outputs it to the vehicle control device 4. The vehicle control device 4 implements vehicle control based on the bump detection result and the reliability.

[0098] According to the first embodiment described above, the following effects are achieved.

[0099] (1) The computing device 11 detects a first protrusion candidate (first road structure candidate) representing a protrusion (road structure) on the road surface captured in the captured image based on disparity information obtained from multiple images. Based on the disparity information and the grayscale information of the image, it detects a second protrusion candidate (second road structure candidate) representing a protrusion (road structure) on the road surface captured in the image. Based on the detection results of the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate), it calculates the position and size of the protrusion (road structure) and outputs the relevant information of the protrusion (road structure) with the calculated position and size to the vehicle control device 4 of the vehicle 1. Thus, even when the performance of road surface condition detection based on camera images deteriorates, appropriate vehicle control can still be performed.

[0100] (2) The computing device 11 calculates the reliability (measurement reliability) of the position and size of the protrusion (road structure) based on the detection results of the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate), and outputs control information representing the control content of the vehicle 1 based on the reliability (measurement reliability) to the vehicle control device 4. Therefore, optimal vehicle control can be performed based on the reliability.

[0101] (3) The computing device 11 calculates the reliability (measurement reliability) based on the distance to the location where at least one of the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate) is detected. Therefore, a lower reliability can be given to protrusions that are difficult to detect, such as protrusions that are too far away or too close, while a higher reliability can be given to protrusions that are easy to detect.

[0102] (4) The computing device 11 calculates the reliability (measurement reliability) based on the number of times at least one of the first protrusion candidate (first road structure candidate) or the second protrusion candidate (second road structure candidate) is detected within a specified time. Thus, a lower reliability can be given to protrusion candidates that have poor tracking performance, i.e., those that are more likely not to be protrusions.

[0103] (5) The computing device 11 calculates the reliability (measurement reliability) based on the consistency of the detection results of the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate). Thus, an appropriate reliability can be given for both cases where the detection results of the two parties are inconsistent and cases where they are consistent.

[0104] (6) The computing device 11 calculates the reliability (measurement reliability) based on the shooting conditions of the road surface where the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate) are located. Thus, the negative impact of the road surface shooting conditions on the detection can be eliminated.

[0105] (7) When only one of the first protrusion candidate (first road structure candidate) and the second protrusion candidate (second road structure candidate) is detected, the computing device 11 calculates the distance and size of the protrusion (road structure) based on the detection result of the detected protrusion (road structure) candidate. Thus, even if only one of the detection units detects the protrusion, appropriate vehicle control can be performed.

[0106] (8) When a first protrusion candidate (first road structure candidate) and a second protrusion candidate (second road structure candidate) are detected, the computing device 11 calculates the distance and size of the protrusion (road structure) based on the distance to the location where at least one of the first protrusion candidate (first road structure candidate) or the second protrusion candidate (second road structure candidate) is detected. Thus, the measurement can be performed using the characteristics of the two detection methods.

[0107] <Second Implementation Method>

[0108] Reference Figure 11 and Figure 12This section describes the environmental identification device according to the second embodiment of the present invention. The same reference numerals are used for structures that are the same as or equivalent to those described in the first embodiment, and the main differences are explained.

[0109] Figure 11 Is with Figure 1 The same figure is a block diagram schematically showing the structure of a vehicle 100 equipped with the environmental recognition device 103 according to the second embodiment. An environmental recording device 110 and a communication device 111 are additionally provided in the vehicle 100.

[0110] The environmental recognition device 3 of the first embodiment operates independently on the vehicle 1, and identifies the ever-changing environment around the vehicle 1 based on external information obtained from the camera 2 mounted on the vehicle 1. The environmental recognition device 103 of the second embodiment, in addition to this, also maintains recording as needed, or communicates with the outside of the vehicle 100.

[0111] The environment recording device 110 can simultaneously record the recognition results of the environment recognition device 103 and the vehicle's position output by a positioning device (not shown). The positioning device (not shown) estimates the vehicle's position, for example, by receiving latitude, longitude, and altitude obtained from a GNSS receiver. Alternatively, it estimates the vehicle's position by recognizing the environment around the vehicle 100 using a camera 2, LiDAR, or similar means, and comparing it with an internal map (not shown). Furthermore, the environment recording device 110 can use the vehicle's position as a query condition to retrieve recognition results and recorded locations near the input vehicle position from the already recorded data.

[0112] Furthermore, the communication device 111 can communicate with one or more other vehicles, road facilities, and map servers to upload the content recorded by the environmental recording device 110 of the vehicle 100 to other vehicles, road facilities, map servers, etc. Conversely, it can also communicate with one or more other vehicles, road facilities, and map servers to receive identification results such as the position and height of protrusions observed by vehicles or devices other than the vehicle 100, and adjust parameters based on the received information to make it easier for the environmental recognition device 103 to identify the protrusion, or to cover the identification results.

[0113] Figure 12 Is with Figure 3The same diagram is a block diagram schematically illustrating the functional structure of the environmental identification device 103 according to the second embodiment. The environmental identification device 103 according to the second embodiment includes a measurement processing unit 136 and a reliability calculation unit 137, replacing the measurement processing unit 36 ​​and the reliability calculation unit 37. The environmental recording device 110 stores the distance to the protrusion, the height of the protrusion, the reliability, etc., calculated by the measurement processing unit 136 according to the processing described in the first embodiment, in association with the vehicle position estimated by the positioning device (not shown), in a storage device (not shown). The environmental recording device 110 reads the information stored in the storage device (not shown), and when the vehicle position is input from the positioning device (not shown), outputs the position, height, and reliability of the protrusion that are near a threshold relative to the input vehicle position to the measurement processing unit 136 and the reliability calculation unit 137.

[0114] When the reliability exceeds a specified threshold, the measurement processing unit 136 replaces the calculated position, height, and reliability of the protrusion with the information obtained from the environmental recording device 110. Thus, instead of improving reliability through multiple tests from a distance, it is possible to detect the protrusion with higher reliability from a greater distance.

[0115] If the reliability calculation unit 137 obtains the position of the protrusion from the environmental recording device 110 and the reliability is greater than a specified threshold, it will... Figure 9 The threshold N for the number of tracking attempts is changed to be smaller. As a result, bumps can be addressed from a greater distance.

[0116] The communication device 111 can communicate with one or more other vehicles, road facilities, and map servers to upload or download the contents stored in a storage device (not shown) by the environment recording device 110. Therefore, even for bumps that the vehicle 100 has not detected before, such as bumps present on the road during initial travel, it can be detected from a greater distance with higher reliability. Furthermore, bumps can be addressed from a greater distance.

[0117] According to the second embodiment described above, the following effects are achieved.

[0118] (1) The computing device 1 obtains the vehicle position of the vehicle 1, records the information about the protrusion (road structure) in correspondence with the vehicle position when the position and size of the protrusion (road structure) are determined, and sends the recorded information related to the protrusion (road structure) to a server outside the vehicle 100. The device receives information about the protrusion (road structure) previously measured by the vehicle 100 or other vehicles at the vehicle position from the server, and calculates the position and size of the protrusion (road structure) based on the received information about the protrusion (road structure). Thus, even for protrusions that the vehicle 100 has not detected before, such as protrusions on the road when first driven, it is possible to detect them from a greater distance with higher reliability.

[0119] Although the embodiments of the present invention have been described above, the above embodiments only show a part of the application examples of the present invention and are not intended to limit the technical scope of the present invention to the specific structures of the above embodiments.

[0120] Label Explanation

[0121] 1, 100…Vehicle, 2…Camera, 3, 103…Environmental recognition device, 4…Vehicle control device, 5…Display, 6…Speaker, 7…Accelerator, 8…Brake, 9…Suspension, 11…Computing device, 12…Non-volatile memory, 13…Volatile memory, 14…Input / output interface, 31…Parallelism calculation unit, 32…First detection unit, 33…Second detection unit, 34…Parallelism analysis unit, 35…Area analysis unit, 36, 136…Measurement processing unit, 37, 137…Reliability calculation unit, 110…Environmental recording device, 111…Communication device.

Claims

1. An environment recognition device, which identifies the external environment based on multiple images acquired by multiple cameras capturing images of the exterior of a vehicle, and includes a computing device, characterized in that... The computing device performs the following actions: Based on disparity information obtained from multiple images, a first road structure candidate representing a road structure present on the road surface captured in the images is detected. Based on the disparity information and the grayscale information of the image, a second road structure candidate representing the road structure present on the road surface captured in the image is detected. Based on the detection results of the first and second road structure candidates, the location and size of the road structure are determined. The relevant information of the road structure, whose location and size have been determined, is output to the vehicle control device of the vehicle.

2. The environmental identification device as described in claim 1, characterized in that, The computing device calculates the measurement reliability of the position and size of the road structure based on the detection results of the first road structure candidate and the second road structure candidate, and outputs control information representing the control content of the vehicle to the vehicle control device based on the measurement reliability.

3. The environmental identification device as described in claim 2, characterized in that, The computing device calculates the measurement reliability based on the distance to the location where at least one of the first road structure candidate and the second road structure candidate is detected.

4. The environmental identification device as described in claim 2, characterized in that, The computing device calculates the measurement reliability based on the number of times at least one of the first road structure candidate and the second road structure candidate is detected within a specified time.

5. The environmental identification device as described in claim 2, characterized in that, The computing device calculates the measurement reliability based on the consistency between the detection results of the first road structure candidate and the second road structure candidate.

6. The environmental identification device as described in claim 2, characterized in that, The computing device calculates the measurement reliability based on the photographic conditions of the road surfaces where the first road structure candidate and the second road structure candidate are located.

7. The environmental identification device as described in claim 1, characterized in that, When only one of the first road structure candidate and the second road structure candidate is detected, the computing device calculates the distance and size of the road structure based on the detection result of the detected road structure candidate.

8. The environmental identification device as described in claim 1, characterized in that, When the first road structure candidate and the second road structure candidate are detected, the computing device calculates the distance and size of the road structure based on the distance to the location where at least one of the first road structure candidate and the second road structure candidate is detected.

9. The environmental identification device as described in claim 1, characterized in that, The computing device performs the following actions: Obtain the vehicle's current location. Record the relevant information about the road structure in correspondence with the vehicle's position when the location and size of the road structure are determined. The recorded road structure information is sent to a server outside the vehicle, and the vehicle receives road structure information previously measured by the vehicle or other vehicles at its location from the server. Based on the received information about the road structure, the location and size of the road structure are determined.

10. An environmental recognition method, characterized in that, Based on parallax information obtained from multiple images captured by multiple cameras outside the vehicle, a first road structure candidate representing the road structure present on the road surface captured in the images is detected. Based on the disparity information and the grayscale information of the image, a second road structure candidate representing the road structure present on the road surface captured in the image is detected. Based on the detection results of the first and second road structure candidates, the location and size of the road structure are determined. The vehicle control device of the vehicle outputs relevant information about the road structure, which has determined its position and size.

Images