Environment recognition device

The environment recognition device uses a monocular camera and image processing to detect speed bumps by analyzing height changes of three-dimensional objects, addressing accuracy issues in existing parallax-based methods and enhancing vehicle safety.

JP2024158830APending Publication Date: 2024-11-08ASTEMO LTD

Patent Information

Application Number
JP2023074388
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing speed bump detection methods using road surface parallax face accuracy issues, especially at night due to noise and difficulty in calculating parallax, leading to inaccurate detection of irregularities.

Method used

An environment recognition device utilizing a monocular camera to detect speed bumps by analyzing height changes of three-dimensional objects like curbs, employing convolutional neural networks for image processing and curve fitting to identify uneven road surfaces without relying on parallax.

Benefits of technology

Accurately detects speed bumps and other road irregularities with high precision, even at night, reducing the need for additional sensors and lowering costs, and enabling vehicle control for safe passage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158830000001_ABST
    Figure 2024158830000001_ABST
Patent Text Reader

Abstract

To detect an uneven part such as a speed bump with high precision.SOLUTION: The present invention relates to an environment recognition device (1) mounted on a vehicle, and the environment recognition device comprises: an image acquisition part (100) which acquires an image captured by a camera (2); a travel environment recognition part (101) which recognizes a three-dimensional object positioned at the border of a travel path based upon the image; and an unevenness detection part (102) which calculates the height of the three-dimensional object in a depth direction and detects an uneven part of the road surface of the travel path from a change in the calculated height.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an environment recognition device. [Background technology]

[0002] In order to prevent vehicles from exceeding the speed limit, structures such as speed bumps are installed on road surfaces. However, when a vehicle passes a speed bump at a high speed without noticing the speed bump, the driver receives a strong impact. Therefore, it is expected that a preventive safety function will be used to recognize the speed bump and to realize deceleration of the vehicle and suspension control. Patent Document 1 discloses a method of detecting a speed bump using a stereo camera. Patent Document 1 discloses a technology for detecting a speed bump by analyzing the parallax of a stereo camera and detecting unevenness on the road surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-135596 A Summary of the Invention [Problem to be solved by the invention]

[0004] The speed bump detection method in Patent Document 1 utilizes the parallax of the road surface. It is generally known that the accuracy of the road surface parallax decreases at a distance, and that it is difficult to calculate the parallax itself at night due to noise such as blurring. Therefore, the challenge is to detect uneven parts such as speed bumps with high accuracy even when the accuracy of the road surface parallax decreases. The present invention has been made in view of the above-mentioned points, and an object of the present invention is to provide an environment recognition device that detects uneven parts such as speed bumps with high accuracy. [Means for solving the problem]

[0005] The environment recognition device of the present invention that solves the above problems is: An environment recognition device mounted on a vehicle, an image acquisition unit that acquires an image captured by a camera; a driving environment recognition unit that recognizes a three-dimensional object located at a boundary of a road based on the image; an unevenness detection unit that calculates a height of the three-dimensional object in a depth direction and detects unevenness of the road surface of the road from a change in the calculated height; The present invention is characterized by comprising: Effect of the Invention

[0006] According to the present invention, unevenness such as speed bumps can be detected with high accuracy.

[0007] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, the problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a functional block diagram showing a configuration of an environment recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a process flow diagram according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an environment around a vehicle in the first embodiment. [Figure 4] FIG. 4 is a diagram showing the results of the driving environment identification process for FIG. 3; [Diagram 5] A conceptual diagram illustrating how to convert each y coordinate of the curb area into a depth z. [Figure 6] 5A to 5C are diagrams for explaining the height of a curb and curve fitting processing in the first embodiment. [Figure 7] FIG. 4 is a process flow diagram for detecting unevenness on a road surface. [Figure 8] 11A and 11B are diagrams showing results of a driving environment identification process. [Figure 9] FIG. 7 is a diagram corresponding to FIG. 6 for explaining the content of a modified example. [Figure 10]FIG. 11 is a functional block diagram showing the configuration of an environment recognition device according to a second embodiment. [Figure 11] FIG. 11 is a process flow diagram according to the second embodiment. [Figure 12] 13A to 13C are diagrams illustrating the measurement results of the three-dimensional height of a curb and the reference height in the second embodiment. [Figure 13] 13A to 13C are diagrams for explaining a method for calculating distances between projections and recesses in the second embodiment. [Figure 14] FIG. 11 is a functional block diagram showing the configuration of an environment recognition device according to a third embodiment. [Figure 15] FIG. 11 is a process flow diagram according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail.

[0010] [First embodiment] FIG. 1 is a functional block diagram showing the configuration of an environment recognition device 1 according to the first embodiment. The environment recognition device 1 is configured with an on-board ECU having a computer, a memory, a storage device, and the like. One camera 2 is connected to the environment recognition device 1. The camera 2 is a so-called monocular camera, and is configured to be arranged, for example, in the cabin of the vehicle and capture an image of the area in front of the vehicle through the front window. In this embodiment, the environment recognition device 1 is configured to be arranged inside the housing of the camera 2, but may be configured to be provided separately from the camera 2.

[0011] The environment recognition device 1 operates as various functional units by executing a control program stored in a computer's memory, etc. As shown in Fig. 1, the environment recognition device 1 has an image acquisition unit 100, a driving environment recognition unit 101, and an unevenness detection unit 102 as functional units realized by the operation of a computer.

[0012] The image acquisition section 100 acquires images captured by the camera 2. Since the camera 2 captures images at regular time intervals, the image acquisition section 100 acquires images at each time.

[0013] The driving environment recognition unit 101 recognizes the driving environment using the image acquired by the image acquisition unit 100 as an input. The driving environment recognition unit 101 recognizes a three-dimensional object located on the side boundary of the driving road 300 based on the image. Specifically, for each pixel captured in the image, the type of the object, such as a driving road, a sidewalk, or a three-dimensional object such as a curb or a guardrail, is identified. A convolutional neural network can be used for the recognition. Also, a machine learning algorithm such as a random forest may be used to identify the type of each pixel.

[0014] The unevenness detection unit 102 detects unevenness on the road surface including speed bumps based on the recognition result of the driving environment recognition unit 101. The unevenness detection unit 102 calculates the number of pixels on the image as height, fits a curve to the height calculated in the depth direction, and detects points that deviate from the curve as unevenness. Specifically, it calculates the height of a three-dimensional object (such as a curb or guardrail) that contacts the driving road from within the area specified by the driving environment recognition unit 101, and detects unevenness on the road surface based on the change in height between the three-dimensional object and the driving road 300. The unevenness detection unit 102 has a height calculation unit 103 and a height change detection unit 104.

[0015] Hereinafter, an example of the operation of the environment recognition device 1 of this embodiment will be described in detail with reference to the flowchart of FIG.

[0016] FIG. 2 is a processing flow diagram in the first embodiment, FIG. 3 is a diagram showing the environment around the vehicle in the first embodiment, FIG. 3(1) is a plan view showing a schematic state in which a speed bump is present in front of the vehicle, and FIG. 3(2) is a schematic diagram showing an image acquired by the image acquisition unit 100.

[0017] Fig. 3(1) shows a situation in which the road surface of a road 300 on which a vehicle 3 is traveling and a speed bump 303, which is an uneven portion, are arranged on the road surface ahead. Assume that an image 311 shown in Fig. 3(2) is captured by a camera 2 mounted on the vehicle 3. In the image 311, the road 300, a sidewalk 301, a curb 302, and the speed bump 303 are captured.

[0018] A curbstone 302 is placed on the shoulder of the roadway 300, and a sidewalk 301 is provided at a position one step higher than the roadway 300 via the curbstone 302. The curbstone 302 rises almost vertically from the road surface at the end of the roadway 300 on the sidewalk 301 side (hereinafter referred to as the boundary position 304), and bends toward the sidewalk 301 at the upper end, forming a continuous step on the upper surface of the sidewalk 301.

[0019] The speed bump 303 has a height that allows a vehicle 3 to pass over it at a low speed, and is provided as a convex strip extending across the entire width of the road on the road surface of the roadway 300. The end of the speed bump 303 on the sidewalk 301 side is in contact with the curb 302. The boundary position 304 between the roadway 300 and the curb 302 extends continuously in a line in the depth direction of the image 311 with a certain gap between it and the top surface of the sidewalk 301, and the height of the speed bump 303 changes so that the gap between the speed bump 303 and the top surface of the sidewalk 301 becomes narrower at the location where the speed bump 303 is present, that is, so that the speed bump 303 approaches the top surface of the sidewalk 301.

[0020] As shown in FIG. 2, the environment recognition device 1 sequentially executes an image acquisition process P101, a driving environment identification process P102, a three-dimensional object height calculation process P103, a curve fitting process P104, and an unevenness determination process P105.

[0021] The image acquisition process P101 is performed by the image acquisition unit 100. In the image acquisition process P101, an image captured by the camera 2 is acquired.

[0022] The driving environment identification process P102 is performed by the driving environment recognition unit 101. In the driving environment identification process P102, a convolutional neural network is used to perform known semantic segmentation on the image acquired from the camera 2. The output of the convolutional neural network is shown in Figure 4(1) and Figure 4(2).

[0023] Fig. 4 shows the results of the driving environment identification process, where Fig. 4(1) shows the output of the convolutional neural network for the image in Fig. 3(2), and Fig. 4(2) is an enlarged view of the main part A2 in Fig. 4(1). The convolutional neural network executes an identification process for each pixel in the image 311, and recognizes a sidewalk area 401, a curb area 402, and a road area 400 in the image 411 after the identification process, as shown in Fig. 4(1).

[0024] The three-dimensional object height calculation process P103 is performed by the height calculation unit 103 of the unevenness detection unit 102. In the three-dimensional object height calculation process P103, the height of the curb 302, which is a three-dimensional object that contacts the road surface of the road 300, from the road 300 is calculated. Specifically, the height of the curb 302 from the road 300 is the pixel height h of the curb area 402 from the road surface area 400 (h1, h2, h3 in FIG. 4(2)). If a speed bump 303 exists on the road surface of the road 300, the height of the curb 302 is the height from the top surface of the speed bump 303.

[0025] In the three-dimensional object height calculation process P103, the height of the curb 302 on the image in each depth direction, which is the direction away from the vehicle 3, is calculated. At each y coordinate (up and down direction of the image 411) of the curb area 402, the height (pixel height) h on the image is calculated from the image positions of the bottom and top ends of the curb area 402 (arrow in the figure). Fig. 4(2) shows an example of calculating the heights h1, h2, and h3 on the image from the bottom end to the top end of the curb area 402 at the coordinate positions y1 to y3.

[0026] Then, each y coordinate of the curb region 402 is converted to a depth z. FIG. 5 is a conceptual diagram for explaining a method for converting each y coordinate of the curb area into a depth z. If the ground contact height of camera 2 is camh and the angle between the line of sight and the vertical direction of each y coordinate in image 411 is θ(y), then each y coordinate of the curb area can be converted to depth z using z=camh / tan(θ(y)).

[0027] Fig. 6 is a diagram for explaining the height of the curbstone and the curve fitting process in the first embodiment. Fig. 6 shows an example of the height h on the image at the depth z, and in the graph of Fig. 6(1), the horizontal axis indicates the depth z, and the vertical axis indicates the height h on the image. In the three-dimensional object height calculation process P103, the above process is executed, and the height h on the image at each depth z is obtained as shown in Fig. 6(1).

[0028] The curve fitting process P104 and the unevenness determination process P105 are performed by the height change detection unit 104 of the unevenness detection unit 102. In the curve fitting process P104, a curve is fitted to the result calculated in the three-dimensional object height calculation process P103.

[0029] The execution result is shown in Figure 6 (2). The curve is parameterized as h = α × (1 / z), and the parameter α that best fits the height h on the image of the curb region 402 for each depth z is estimated. The well-known RANSAC (Random Sample Consensus) is used to estimate the parameter α. By using RANSAC, it is possible to estimate the parameters after removing outliers.

[0030] In the unevenness determination process P105, unevenness in the road 300 is detected using the curve estimated in the curve fitting process P104 and the height information of the curb area 402 on the image calculated in the three-dimensional object height calculation process P103.

[0031] FIG. 7 is a process flow diagram for detecting unevenness on a road surface. First, the error abs(α / zh(z)) is calculated for the target depth z (F102). Here, α is a parameter obtained in the curve fitting process P104, and h(z) is the height (calculated height value) of the curb area 402 calculated in the solid object height calculation process P103. This error indicates the error between the curve and the curb height.

[0032] Next, this error is compared with a threshold value (e.g., 10) (F103). If the error value is equal to or greater than the threshold value, it is determined that there is a high possibility that a speed bump 303 exists, and the detection count is incremented by 1 (F104). If the error value is smaller than the threshold value, the detection count is reset to 0 (F105), and the process moves to calculating the error in the curb height at the next depth z (F101).

[0033] In F106, the detection count is compared with a threshold (e.g., 3). If the detection count is equal to or greater than the threshold, it is assumed that a speed bump 303 has been detected, and the subsequent process F107 is executed. If the detection count is less than the threshold, the process moves to calculation of the curb height error at the next depth z (F101). In F107, the image coordinates are stored as the detected position of the unevenness. Thus, in the unevenness determination process P105, if the height on the image of the curb region 402 deviates from the curve modeling the height of the curb region 402 multiple times in succession, such as δ1 and δ2 shown in FIG. 6(2), it is determined that a speed bump 303 exists at that position.

[0034] As described above, the environment recognition device 1 of this embodiment detects the speed bump 303 based on the change in height between the road and a three-dimensional object such as a curb 302 that contacts the road surface of the road 300. Since the environment recognition device 1 of this embodiment does not use parallax, it can detect the speed bump 303 with high accuracy even at night when parallax cannot be calculated.

[0035] Furthermore, the environment recognition device 1 of this embodiment assumes a monocular camera as the camera 2, and detects the speed bump 303 based on the change in height on the image of a three-dimensional object such as a curb 302. Since the speed bump 303 can be detected only by the monocular camera without using any other three-dimensional sensor, the cost of the sensor can be reduced.

[0036] In addition, in the environment recognition device 1 of this embodiment, a case has been described in which unevenness such as a speed bump 303 is detected based on the change in height between the upper end of a three-dimensional object such as a curb 302 and the roadway, but on the other hand, in a real environment, there are cases in which the lower end of a three-dimensional object such as a curb 302 can be detected but the upper end cannot be detected. In such a case, instead of detecting the curb 302 based on the change in height between the upper end of the curb 302 and the roadway 300 as shown in Fig. 5, unevenness such as a speed bump 303 can be detected based on the position of the lower end of the curb 302 (boundary position 304) as shown in Fig. 8.

[0037] Fig. 8 shows the results of the driving environment identification process P102, where Fig. 8(1) shows the output of the convolution neural network for the image of Fig. 3(2), and Fig. 8(2) shows an enlarged view of the main part A2 of Fig. 8(1). Fig. 9 is a corresponding view to Fig. 6 for explaining the contents of a modified example.

[0038] In this modification, a straight line 405 is fitted along the bottom end position of the curb region 402 (see FIG. 8(2)). In this case, RANSAC is used to model the portion of the curb region 402 that does not contact the speed bump 303. Then, the height h on the image from the modeled straight line 405 to the bottom end (boundary position 404) of the curb region 402 is obtained for each depth z in the depth direction (see FIG. 9(1)). Then, an upward convex region is detected (see FIG. 9(2)) where multiple candidates for unevenness with a height h on the image equal to or greater than a certain value (δ) are gathered. This region is detected as the speed bump 303. As a result, unevenness such as the speed bump 303 can be detected even in cases where the top end of a three-dimensional object such as the curb 302 cannot be measured.

[0039] According to the environment recognition device 1 of this embodiment, it is possible to detect the presence of unevenness and to grasp the distance to the unevenness. Therefore, using the information on the distance to the unevenness, it is possible to perform vehicle control such as controlling the vehicle speed to decelerate the vehicle before the unevenness, or controlling the vehicle suspension to reduce the impact when going over the unevenness.

[0040] [Second embodiment] Next, a second embodiment of the present invention will be described. FIG. 10 is a functional block diagram showing the configuration of an environment recognition device 1A according to the second embodiment.

[0041] In the configuration of the second embodiment, two cameras 2A and 2B are connected as stereo cameras to an environment recognition device 1A. In the following, the description will be limited to blocks that are functionally different from the first embodiment and blocks that are newly added in the second embodiment.

[0042] The image acquisition unit 100 acquires two images of the front view taken from the vehicle at the same time by two cameras 2A and 2B, which are a pair on the left and right. The three-dimensional information acquisition unit 203 acquires three-dimensional information of the surrounding environment using the two images acquired by the image acquisition unit 100. Specifically, the three-dimensional information can be calculated using a known stereo matching method.

[0043] The driving environment recognition unit 101 may identify a type for each pixel as in the configuration of the first embodiment and estimate a label including three-dimensional objects in the driving environment, or may identify the area of ​​a three-dimensional object standing vertically to the ground by using distance information acquired by the three-dimensional information acquisition unit 203.

[0044] The unevenness distance measuring unit 204 measures the distance from the vehicle 3 to the unevenness detected by the unevenness detection unit 102. Specifically, the distance to the unevenness may be determined using depth information at the time of detection, or the distance to a three-dimensional object such as a curb or guardrail located at a point adjacent to the unevenness may be regarded as the distance to the unevenness.

[0045] Hereinafter, an operation example of the environment recognition device 1A of this embodiment will be described with reference to the flowchart in Fig. 11. It is assumed that the scene shown in Fig. 3 is captured by the cameras 2A and 2B. The processes that differ in content from the flowchart of the first embodiment shown in Fig. 2 are three-dimensional information acquisition process P201, three-dimensional object height calculation process P202, reference height calculation process P203, unevenness determination process P204, and unevenness distance measurement process P205, so these processes will be described below.

[0046] The three-dimensional information acquisition process P201 is performed by the three-dimensional information acquisition unit 203. In the three-dimensional information acquisition process P201, stereo matching is performed using the left and right images acquired by the cameras 2A and 2B. By performing stereo matching, the parallax d is calculated for each pixel of the image. Then, the calculated parallax d is used to calculate the distance z=f*B / (w*d) to the three-dimensional object. Here, f is the focal length of the camera, B is the baseline length of the stereo camera, and w is the actual size of one pixel.

[0047] The three-dimensional object height calculation process P202 and the reference height calculation process P203 are performed by the height calculation unit 103 of the unevenness detection unit 102. In the three-dimensional object height calculation process P202, the three-dimensional height h of the curbstone 302 recognized in the driving environment identification process P102 is calculated. The three-dimensional height h of the curbstone can be calculated using the formula h=(w*hi*z) / f, where hi is the height of the curbstone 302 on the image, and z is the distance calculated in the three-dimensional information acquisition process P201. By performing this calculation for each pixel of the curbstone 302, waveform data with horizontal axis z and vertical axis h is generated, as shown in FIG. 12(1).

[0048] In the reference height calculation process P203, the height of the curbstone 302 is determined using each three-dimensional height of the curbstone 302. In the reference height calculation process P203, the three-dimensional height of the three-dimensional object is calculated in the depth direction as the height, and the reference height of the three-dimensional object is determined from the three-dimensional height in each depth direction. Specifically, the median value of the three-dimensional heights obtained from the curbstone 302 is set as the reference height. The concept of the reference height is shown in Fig. 12(2).

[0049] The unevenness determination process P204 is performed by the height change detection unit 104 of the unevenness detection unit 102. In the unevenness determination process P204, unevenness is detected using the information on the reference height and each three-dimensional height of the curbstone 302. Specifically, when the distance to the target curbstone 302 is z, the difference abs(h'-h(z)) between the reference height h' and the three-dimensional height h(z) of the curbstone 302 is calculated, and it is determined whether or not the value is equal to or greater than a threshold value (for example, 10 cm). If it is equal to or greater than the threshold value, it is determined to be a candidate for an unevenness. Thereafter, abs(h'-h(z')) is similarly calculated for the next distance z', and it is determined whether or not it is equal to or greater than the threshold value. Then, if the difference between the heights at the distance z' is also equal to or greater than the threshold value, it is determined that there is an unevenness at that point.

[0050] The unevenness distance measurement process P205 is performed by the unevenness distance measurement unit 204. In the unevenness distance measurement process P205, the distance from the vehicle 3 to the unevenness part is determined. As described above, in the unevenness determination process P204, if the difference in height between the distance z and the distance z' (z'>z) is equal to or greater than the threshold, the distance z that is closer to the host vehicle is determined to be the distance from the vehicle 3 to the unevenness part.

[0051] From the above, the environment recognition device 1A of this embodiment calculates the three-dimensional height of the curbstone 302, and determines that there is an uneven portion at a point where the three-dimensional height deviates from the reference height of the curbstone 302. Generally, the three-dimensional height of the curbstone 302 is constant in the depth direction, so the reference height of the curbstone 302 can be calculated simply by obtaining the median value of the measured three-dimensional heights of the curbstone 302, and uneven portions can be detected without using calculations with a high processing load, such as curve fitting processing.

[0052] According to the environment recognition device 1A of the present embodiment, it is possible to detect unevenness and grasp the distance to the unevenness. Therefore, using the information on the distance to the unevenness, it is possible to perform vehicle control such as controlling the vehicle speed to decelerate the vehicle before the unevenness, or controlling the vehicle suspension to reduce the impact when going over the unevenness.

[0053] In the unevenness distance measurement process P205 of the second embodiment described above, the distance to the unevenness part is determined using the distance z at which it is determined that an unevenness part exists in the unevenness determination process P204, but it is also possible to search the curb area 402 shown in Fig. 13 in the upward direction (in the direction of the arrow in the figure) and use the median value of the distance in the curb area 402 as the distance to the unevenness part. In this way, the distance to the unevenness part can be determined using multiple distance information, thereby improving the accuracy of the distance.

[0054] [Third embodiment] 14 is a functional block diagram showing the configuration of an environment recognition device 1B of the third embodiment. The environment recognition device 1B of the third embodiment has a configuration in which one camera 2 is connected, similar to the configuration of the first embodiment. The difference from the first embodiment is that the environment recognition device 1B includes a detection determination unit 305.

[0055] The detection determination unit 305 determines whether or not to detect unevenness based on the texture of the three-dimensional object. Specifically, the detection determination unit 305 compares the texture of the three-dimensional object acquired in the past with the texture of the three-dimensional object in the current frame, and if the textures are similar, detects unevenness.

[0056] Hereinafter, an example of the operation of the environment recognition device 1B of this embodiment will be described with reference to the flowchart in Fig. 15. The differences from the flowchart in the first embodiment shown in Fig. 2 are the three-dimensional object texture acquisition process P300, the three-dimensional object texture comparison process P301, and the determination process P302, and therefore these processes will be described below.

[0057] In the three-dimensional object texture acquisition process P300, the texture of a three-dimensional object (such as a curb) is acquired as a template for each frame and saved as the template in a storage area.

[0058] In the three-dimensional object texture comparison process P301, a template of a three-dimensional object acquired in the past is scanned over the three-dimensional object area of ​​the current frame. Then, the area with the highest similarity is identified. The similarity is compared with a threshold value (e.g., 0.5), and if it is equal to or greater than the threshold, the process proceeds to the following three-dimensional object height calculation process P103. On the other hand, if the similarity is less than the threshold, detection of unevenness is not performed in the current frame.

[0059] As described above, the environment recognition device 1B of this embodiment compares the textures of current and past three-dimensional objects (such as curbs), and if the textures are similar, detects unevenness in the current frame, but if they are not similar, does not detect unevenness in the current frame. This makes it possible to reduce the possibility of erroneously determining that unevenness exists on the road surface when there are actually no unevenness such as speed bumps 303 on the road surface of the road 300 due to the influence of small fallen objects or the like at the boundary between the road 300 and the curb 302, by not executing the detection process.

[0060] As another embodiment, in addition to the configurations of the first to third embodiments, a configuration may be adopted in which the system cooperates with map data. Specifically, the position of an uneven portion once recognized is registered on a map, and when the vehicle travels next time around the uneven portion registered on the map, the threshold value in the unevenness determination process of FIG. 2 or FIG. 11 is relaxed. In the embodiments described so far, the presence of an uneven portion is confirmed when consecutive uneven portion candidates exist, but as a relaxation of the threshold value, the presence of an uneven portion can be confirmed when it is determined that an uneven portion candidate exists once. This makes it possible to improve responsiveness to detection of uneven portions and prevent undetection when uneven portions exist on a map.

[0061] Furthermore, the environment recognition device according to the present invention can be applied to uneven parts of the road surface of the roadway 300 other than speed bumps. Specifically, the device can be applied to downwardly concave parts such as potholes and other depressions in the road surface.

[0062] In addition, the environment recognition device according to the present invention can be generalized to use information on the vertical plane of a three-dimensional object that is in contact with an object on the road surface that corresponds to the horizontal plane as seen from the vehicle-mounted camera. Therefore, the present invention can also be applied to the detection of structures other than those mentioned above.

[0063] Although the present invention has been described above with reference to the first to third embodiments, the present invention is not limited to the configurations of the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0064] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0065] 1, 1A, 1B... environment recognition device, 2, 2A, 2B... camera, 3... vehicle, 100... image acquisition unit, 101... driving environment recognition unit, 102... unevenness detection unit, 103... calculation unit, 104... change detection unit, 203... three-dimensional information acquisition unit, 204... unevenness distance measurement unit, 300... driving path, 301... sidewalk, 302... curb, 303... speed bump (uneven portion), 304... boundary position, 305... detection and determination unit, 311... image, 400... road surface area, 401... sidewalk area, 402... curb area

Claims

1. An environment recognition device mounted on a vehicle, an image acquisition unit that acquires an image captured by a camera; a driving environment recognition unit that recognizes a three-dimensional object located at a boundary of a road based on the image; an unevenness detection unit that calculates a height of the three-dimensional object in a depth direction and detects unevenness of the road surface of the road from a change in the calculated height; An environment recognition device comprising:

2. The unevenness detection unit is 2. The environment recognition device according to claim 1, wherein the height is calculated as a number of pixels on the image, a curve is fitted to the height calculated in the depth direction, and points that deviate from the curve are detected as uneven parts.

3. Further comprising a three-dimensional information acquisition unit for acquiring three-dimensional information of a surrounding environment, The environmental recognition device according to claim 1, characterized in that the unevenness detection unit calculates a three-dimensional height of the three-dimensional object in a depth direction as the height, determines a reference height of the three-dimensional object from the three-dimensional height in each depth direction, and detects points that deviate from the reference height as uneven parts.

4. A distance measuring unit for measuring a distance from the vehicle to the uneven portion is further provided, The environment recognition device according to claim 3 , wherein the unevenness distance measurement unit determines the distance acquired from the three-dimensional object as the distance from the vehicle to the unevenness portion.

5. 2. The environment recognition device according to claim 1, further comprising a detection determination unit that stores a past texture of the three-dimensional object, compares a current texture with the past texture, and, if the current and past textures diverge, determines not to detect the unevenness portion by the unevenness detection unit.

Citation Information

Patent Citations

  • Processor

    JP2021135596A

Cited By

  • Method and apparatus for estimating a distance of an object based on a static object and a difference in ground height

    US20240428556A1