Road surface detection device and computer readable storage medium

By using stereo cameras and three-dimensional model correction technology in the road surface detection device, the problem of reducing road surface detection accuracy caused by vehicle shaking is solved, and more accurate road surface state detection is achieved.

CN113165657BActive Publication Date: 2024-11-08AISIN CORP
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN201980079806.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-12-04
Filing Date
2019-11-29
Publication Date
2024-11-08
Estimated Expiration
2039-11-29

AI Technical Summary

Technical Problem

The existing road surface detection technology has failed to effectively deal with the shaking of the shooting part caused by vehicle shaking, resulting in flat road surface being detected as concave and convex pavement, reducing the road surface detection accuracy.

Method used

The road surface detection device is adopted to acquire image data through a stereo camera, generate a three-dimensional model, and estimate the plane normal vector and height position through the correction unit, and match it to the normal value to suppress deterioration of road surface detection accuracy.

Benefits of technology

It effectively suppresses the impact of vehicle shaking on road surface detection accuracy, improves the accuracy of road surface detection, and ensures that the flat road surface is correctly detected as flat.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113165657B_ABST
    Figure CN113165657B_ABST
Patent Text Reader

Abstract

A road surface detection device according to an embodiment comprises: an image acquisition unit that acquires captured image data output from a stereo camera that captures a captured area including a road surface on which a vehicle is traveling; a three-dimensional model generation unit that generates a three-dimensional model of the captured area including the surface shape of the road surface from the viewpoint of the stereo camera based on the captured image data; and a correction unit that estimates a plane from the three-dimensional model and corrects the three-dimensional model so that the direction of the normal vector of the plane and the height position relative to the plane of the stereo camera match the normal value of the direction of the normal vector of the road surface and the normal value of the height position of the road surface relative to the stereo camera, respectively.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to a road surface detection device and a computer-readable storage medium. Background Art

[0002] Conventionally, there is known a technology for detecting the road surface state, such as the height position of the road surface relative to the imaging unit, based on image data output from an imaging unit such as a stereo camera that captures an imaging area including a road surface on which a vehicle is traveling.

[0003] Patent Document 1: Japanese Patent No. 6209648 Summary of the invention

[0004] In the above-mentioned prior art, the shaking of the imaging unit due to the shaking of the vehicle is not considered. If the imaging unit shakes, a flat road surface is detected as a concave and convex road surface in the captured image data, which deteriorates the accuracy of road surface detection.

[0005] As an example, a road surface detection device of an embodiment of the present invention includes: an image acquisition unit, which acquires captured image data output from a stereo camera that captures a shooting area including a road surface on which a vehicle is traveling; a three-dimensional model generation unit, which generates a three-dimensional model of the shooting area including the surface shape of the road surface from the viewpoint of the stereo camera based on the captured image data; and a correction unit, which estimates a plane from the three-dimensional model and corrects the three-dimensional model so that the direction of the normal vector of the plane and the height position of the plane relative to the stereo camera match the normal value of the direction of the normal vector of the road surface and the normal value of the height position of the road surface relative to the stereo camera, respectively.

[0006] Thus, as one example, deterioration in road surface detection accuracy can be suppressed.

[0007] Furthermore, the correction unit, after making the orientation of the normal vector of the plane match the normal value of the orientation of the normal vector of the road surface, corrects the entire three-dimensional model so that the height position of the plane relative to the stereo camera matches the normal value of the height position of the road surface relative to the stereo camera.

[0008] Thus, as an example, correction can be easily implemented.

[0009] Furthermore, the correction unit obtains a normal value of the orientation of the normal vector of the road surface and a normal value of the height position of the road surface relative to the stereo camera based on the values ​​determined during the calibration process of the stereo camera.

[0010] Therefore, as an example, a normal value can be easily obtained.

[0011] A computer-readable storage medium according to an embodiment of the present invention stores a road surface detection program, wherein the road surface detection program causes a computer to execute the following steps: an image acquisition step, which takes into a computer captured image data output from a stereo camera that captures a shooting area including a road surface on which a vehicle is traveling; a three-dimensional model generation step, which generates, based on the captured image data, a three-dimensional model of the shooting area including the surface shape of the road surface from the viewpoint of the stereo camera; and a correction step, which estimates a plane from the three-dimensional model and corrects the three-dimensional model so that the direction of the normal vector of the plane and the height position of the plane relative to the stereo camera match the normal value of the direction of the normal vector of the road surface and the normal value of the height position of the road surface relative to the stereo camera, respectively.

[0012] Thus, as one example, deterioration in road surface detection accuracy can be suppressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a perspective view showing an example in which a part of a vehicle cabin of a vehicle equipped with the road surface detection device according to the embodiment is shown in a see-through state.

[0014] Figure 2 It is a plan view showing an example of a vehicle equipped with the road surface detection device according to the embodiment.

[0015] Figure 3 This is a block diagram showing an example of the configuration of the ECU and its peripheral configuration according to the embodiment.

[0016] Figure 4 It is a diagram illustrating a software configuration realized on the ECU according to the embodiment.

[0017] Figure 5 This is a diagram for explaining an outline of the road surface detection function of the ECU according to the embodiment.

[0018] Figure 6 This is a diagram for explaining the details of the road surface detection function of the ECU according to the embodiment.

[0019] Figure 7 This is a diagram for explaining the details of the road surface detection function of the ECU according to the embodiment.

[0020] Figure 8 This is a diagram for explaining the details of the road surface detection function of the ECU according to the embodiment.

[0021] Fig. 9This is a flowchart showing an example of the procedure of the road surface detection process of the ECU according to the embodiment.

[0022] Fig.10 These are the detection results of a flat road surface obtained by the ECU of the embodiment and the configuration of the comparative example. DETAILED DESCRIPTION

[0023] The following is an exemplary embodiment of the present invention. The structure of the embodiment shown below, and the action, result and effect brought about by the structure are only an example. The present invention can also be implemented by a structure other than the structure disclosed in the following embodiment, and can obtain at least one of the various effects based on the basic structure and the derived effects.

[0024] Implementation

[0025] use Figure 1-Figure 10 The structure of the embodiment will be described.

[0026] Vehicle structure

[0027] Figure 1 This is a perspective view showing an example in which a part of the vehicle cabin 2a of the vehicle 1 equipped with the road surface detection device according to the embodiment is in a see-through state. Figure 2 It is a plan view showing an example of a vehicle 1 equipped with the road surface detection device according to the embodiment.

[0028] The vehicle 1 of the embodiment can be, for example, a car with an internal combustion engine (not shown) as a driving source, i.e., an internal combustion locomotive, a car with an electric motor (not shown) as a driving source, i.e., an electric car or a fuel cell car, etc., a hybrid car with both of the above as driving sources, or a car with other driving sources. In addition, the vehicle 1 can be equipped with various transmission devices, and can also be equipped with various devices such as systems and components required to drive the internal combustion engine or the electric motor. In addition, various settings can be made for the form, number, layout, etc. of the devices related to the driving of the wheels 3 of the vehicle 1.

[0029] like Figure 1 As shown, the vehicle body 2 constitutes a vehicle cabin 2a for passengers (not shown) to sit in. In the vehicle cabin 2a, a steering part 4, an acceleration operating part 5, a brake operating part 6, a speed change operating part 7, etc. are provided in a state facing a seat 2b for a driver who is a passenger. The steering part 4 is, for example, a steering wheel protruding from a dashboard 24. The acceleration operating part 5 is, for example, an accelerator pedal located under the feet of the driver. The brake operating part 6 is, for example, a brake pedal located under the feet of the driver. The speed change operating part 7 is, for example, a shift lever protruding from a center console. In addition, the steering part 4, the acceleration operating part 5, the brake operating part 6, the speed change operating part 7, etc. are not limited thereto.

[0030] In addition, a display device 8 and a voice output device 9 are provided in the vehicle interior 2a. The voice output device 9 is, for example, a speaker. The display device 8 is, for example, an LCD (Liquid Crystal Display) or an OELD (Organic ElectroLuminescent Display). The display device 8 is covered by a transparent operation input unit 10 such as a touch panel. Passengers can observe the image displayed on the display screen of the display device 8 through the operation input unit 10. In addition, passengers can perform operation input by touching, pressing or swiping the operation input unit 10 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 8. These display devices 8, voice output devices 9, operation input units 10, etc. are provided, for example, in a monitoring device 11 located in the center of the instrument panel 24 in the vehicle width direction, that is, the left-right direction. The monitoring device 11 may have an operation input unit such as a switch, a knob, a lever, a button, etc., which is not shown in the figure. In addition, a voice output device which is not shown in the figure may be provided at another position different from the monitoring device 11 in the vehicle interior 2a. Furthermore, voice may be output from the voice output device 9 or other voice output devices of the monitoring device 11. In addition, the monitoring device 11 may also be used as a navigation system or an audio system, for example.

[0031] like Figure 1 , Figure 2 As shown in the example, the vehicle 1 is, for example, a four-wheeled vehicle, and includes two left and right front wheels 3F and two left and right rear wheels 3R. All of these four wheels 3 may be configured to be steerable.

[0032] In addition, the vehicle body 2 is provided with, for example, four imaging units 15a-15d as a plurality of imaging units 15. The imaging unit 15 is, for example, a digital stereo camera having an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The stereo camera simultaneously captures an object through a plurality of cameras, and detects the position or three-dimensional shape of the object from the difference in position on the image of the object obtained by each camera, i.e., parallax. Thus, shape information of the road surface, etc. included in the image can be obtained as three-dimensional information.

[0033] The photographing unit 15 can output captured image data at a specified frame rate. The captured image data can also be dynamic image data. The photographing unit 15 has a wide-angle lens or a fisheye lens, respectively, and can capture a range of, for example, 140°-220° in the horizontal direction. In addition, the optical axis of the photographing unit 15 can also be set to face obliquely downward. Thus, the photographing unit 15 sequentially captures the surrounding environment outside the vehicle 1, including the road surface and objects on which the vehicle 1 can move, and outputs it as captured image data. Here, objects refer to rocks, trees, people, bicycles, and other vehicles that can become obstacles when the vehicle 1 is traveling.

[0034] The photographing unit 15a is, for example, located at the end 2e on the rear side of the vehicle body 2, and is provided on the wall below the rear window on the door 2h of the trunk. The photographing unit 15b is, for example, located at the end 2f on the right side of the vehicle body 2, and is provided on the right door mirror 2g. The photographing unit 15c is, for example, located at the end 2c on the front side of the vehicle body 2, i.e., the front side in the front-to-back direction of the vehicle, and is provided on the front bumper or the front grille. The photographing unit 15d is, for example, located at the end 2d on the left side of the vehicle body 2, and is provided on the left door mirror 2g.

[0035] ECU Hardware Structure

[0036] Next, use Figure 3 An ECU (Electronic Control Unit) 14 and peripheral structures of the ECU 14 according to the embodiment will be described. Figure 3 This is a block diagram showing the configuration of the ECU 14 and its peripheral configuration according to the embodiment.

[0037] like Figure 3 As shown, in addition to the ECU 14 as a road surface detection device, the monitoring device 11, the steering system 13, the braking system 18, the steering angle sensor 19, the accelerator sensor 20, the gear position sensor 21, and the wheel speed sensor 22 are also electrically connected through the in-vehicle network 23 as a telecommunication line. The in-vehicle network 23 is configured as a CAN (Controller Area Network), for example.

[0038] The ECU 14 transmits control signals through the in-vehicle network 23, thereby being able to control the steering system 13, the brake system 18, etc. In addition, the ECU 14 can receive detection results of the torque sensor 13b, the brake sensor 18b, the steering angle sensor 19, the accelerator sensor 20, the gear position sensor 21, the wheel speed sensor 22, etc., and operation signals of the operation input unit 10, etc. through the in-vehicle network 23.

[0039] Furthermore, the ECU 14 can generate an image with a wider viewing angle or a virtual bird's-eye view image of the vehicle 1 viewed from above by performing calculation processing or image processing based on the image data obtained from the plurality of imaging units 15 .

[0040] ECU14 has, for example, a CPU14a (Central Processing Unit), a ROM14b (Read Only Memory), a RAM14c (Random Access Memory), a display control unit 14d, a voice control unit 14e, a flash memory, namely, an SSD14f (Solid State Drive), and the like.

[0041] The CPU 14a can perform various calculations and controls, such as image processing related to the image displayed on the display device 8, determining the target position of the vehicle 1, calculating the moving path of the vehicle 1, determining whether there is interference with an object, automatic control of the vehicle 1, and release of automatic control. The CPU 14a can read a program installed and stored in a non-volatile storage device such as the ROM 14b, and perform calculations according to the program. The program includes a computer program for realizing road surface detection processing on the ECU 14, that is, a road surface detection program.

[0042] The RAM 14c temporarily stores various data used in calculations of the CPU 14a.

[0043] The display control unit 14 d mainly performs image processing using image data obtained by the imaging unit 15 and synthesis of image data to be displayed on the display device 8 among the calculation processes of the ECU 14 .

[0044] The voice control unit 14 e mainly processes the voice data output by the voice output device 9 in the calculation process of the ECU 14 .

[0045] The SSD 14 f is a rewritable nonvolatile storage unit, and can store data even when the power supply of the ECU 14 is turned off.

[0046] In addition, the CPU 14a, ROM 14b, RAM 14c, etc. may be integrated in the same package. In addition, the ECU 14 may also be a structure that uses other logic operation processors or logic circuits such as a DSP (Digital Signal Processor) to replace the CPU 14a. In addition, a HDD (Hard Disk Drive) may be provided to replace the SSD 14f, and the SSD 14f or the HDD may also be provided separately from the ECU 14.

[0047] The steering system 13 has an actuator 13a and a torque sensor 13b, and can steer at least two wheels 3. That is, the steering system 13 is electrically controlled by the ECU 14, etc., so that the actuator 13a works. The steering system 13 is, for example, an electric power steering system or an SBW (Steer by Wire) system. The steering system 13 applies torque, i.e., auxiliary torque, to the steering part 4 through the actuator 13a to supplement the steering force, or steers the wheel 3 through the actuator 13a. In this case, the actuator 13a can steer one wheel 3 or multiple wheels 3. In addition, the torque sensor 13b detects, for example, the torque applied to the steering part 4 by the driver.

[0048] The brake system 18 is, for example, an ABS (Anti-lock Brake System) that suppresses brake lock, an anti-skid device (ESC: Electronic Stability Control) that suppresses the side slip of the vehicle 1 when turning, an electric brake system that enhances the braking force to implement brake assistance, BBW (Brake by Wire), etc. The brake system 18 applies braking force to the wheel 3 and then to the vehicle 1 through the actuator 18a. In addition, the brake system 18 can detect brake lock, idling of the wheel 3, and signs of side slip based on the rotation difference between the left and right wheels 3, and implement various controls. The brake sensor 18b is, for example, a sensor that detects the position of the movable part of the brake operating part 6. The brake sensor 18b can detect the position of the brake pedal as a movable part. The brake sensor 18b includes a displacement sensor.

[0049] The steering angle sensor 19 is a sensor that detects the steering amount of the steering unit 4 such as a steering wheel, for example. The steering angle sensor 19 is formed by, for example, a Hall element, etc. The ECU 14 obtains the steering operation amount of the steering unit 4 caused by the driver, or the steering operation amount of each wheel 3 during automatic steering, etc. from the steering angle sensor 19, and implements various controls. In addition, the steering angle sensor 19 detects the rotation angle of the rotating part included in the steering unit 4. The steering angle sensor 19 is an example of an angle sensor.

[0050] The accelerator sensor 20 is, for example, a sensor that detects the position of a movable portion of the accelerator operation portion 5. The accelerator sensor 20 can detect the position of an accelerator pedal as a movable portion. The accelerator sensor 20 includes a displacement sensor.

[0051] The shift position sensor 21 is, for example, a sensor that detects the position of a movable part of the shift operation part 7. The shift position sensor 21 can detect the position of a movable part such as a lever, an arm, or a button. The shift position sensor 21 may include a displacement sensor or may be configured as a switch.

[0052] The wheel speed sensor 22 is a sensor for detecting the rotation amount and the number of revolutions per unit time of the wheel 3. The wheel speed sensor 22 outputs the number of wheel speed pulses representing the detected number of revolutions as a sensor value. The wheel speed sensor 22 can be constructed, for example, by a Hall element or the like. The ECU 14 calculates the movement amount of the vehicle 1 and the like based on the sensor value obtained from the wheel speed sensor 22 and implements various controls. In addition, the wheel speed sensor 22 can also be provided in the brake system 18. In this case, the ECU 14 can obtain the detection result of the wheel speed sensor 22 through the brake system 18.

[0053] The structures, arrangements, electrical connection forms, etc. of the above-mentioned various sensors and actuators are merely examples, and various settings and changes can be made.

[0054] ECU software structure

[0055] Next, use Figure 4 A software configuration showing the functions of the ECU 14 according to the embodiment will be described. Figure 4 It is a diagram illustrating an example of a software configuration realized in the ECU 14 according to the embodiment. Figure 4 The functions shown are realized by the cooperation of software and hardware. Figure 4 In the example shown, the function of the ECU 14 as the road surface detection device is realized as a result of the CPU 14 a reading and executing a road surface detection program stored in the ROM 14 b or the like.

[0056] like Figure 4 As shown in FIG. 1 , the ECU 14 as a road surface detection device includes an image acquisition unit 401, a three-dimensional model generation unit 402, a correction unit 403, and a storage unit 404. The CPU 14a performs the functions of the image acquisition unit 401, the three-dimensional model generation unit 402, and the correction unit 403 by executing the processing according to the program. The RAM 14c or the ROM 14b functions as the storage unit 404. In addition, at least part of the functions of the above-mentioned units can also be realized by hardware.

[0057] The image acquisition unit 401 acquires a plurality of captured image data from a plurality of imaging units 15 that capture images of the surrounding area of ​​the vehicle 1. The imaging area of ​​the imaging unit 15 includes the road surface on which the vehicle 1 is traveling, and the captured image data includes the surface shape of the road surface and the like.

[0058] The three-dimensional model generation unit 402 generates a three-dimensional model based on the captured image data acquired by the image acquisition unit 401. The three-dimensional model is a three-dimensional point cloud of a plurality of points arranged three-dimensionally according to the road surface state including the surface shape of the road surface on which the vehicle 1 travels.

[0059] The correction unit 403 corrects the slope of the three-dimensional model generated by the three-dimensional model generation unit 402. The three-dimensional model is generated based on the state of the vehicle 1 such as the orientation of the vehicle body 2. Therefore, even when the vehicle 1 is tilted, the correction unit 403 can correct the state to accurately reflect the conditions of the road surface, etc.

[0060] The storage unit 404 stores data used in the calculation processing of each unit, calculation processing result data, etc. The storage unit 404 also stores calibration processing data performed on the imaging unit 15 and the like when the vehicle 1 is shipped.

[0061] ECU Function Example

[0062] Next, use Figure 5 , a functional example of the ECU 14 according to the embodiment will be described. Figure 5 This is a diagram for explaining an outline of the road surface detection function of the ECU 14 according to the embodiment.

[0063] As a prerequisite, when the vehicle 1 leaves the factory, the imaging unit 15 is calibrated in order to correctly obtain the captured image. The calibration process includes, for example, optical axis correction. Through this calibration process, the normal vector perpendicular to the road surface and the height position of the road surface based on the viewpoint of the imaging unit 15 are determined. In the road surface detection of the ECU 14, the normal vector of the road surface based on the viewpoint of the imaging unit 15 and the height position of the road surface can be used as normal values. Based on such normal values, the distance to a specified point on the road surface and the height of the specified point can be correctly calculated from the captured image data of the imaging unit 15.

[0064] That is, for example Figure 5 As shown in (a), the road surface condition is generally detected based on the captured image data captured by the camera 15c in front of the vehicle 1. For example, the road surface condition based on the distance and height to the predetermined point P1-P3 on the road surface closest to the vehicle 1 in the captured image data captured by the camera 15c is regarded as the road surface condition where the vehicle 1 is located at that time, that is, the road surface condition directly below the vehicle 1. Here, a flat road surface parallel to the vehicle 1 is detected.

[0065] Furthermore, the road surface condition slightly ahead of the vehicle 1 is detected based on the distance and height to the predetermined point P4-P6 on the road surface slightly away from the vehicle 1 in the captured image data captured by the imaging unit 15c. Here, a flat road surface parallel to the road surface on which the vehicle 1 is located is detected.

[0066] In addition, for example Figure 5 As shown in (b), based on the distance to and height of predetermined points P4 to P6 on the road surface, an object T of a predetermined height is detected slightly ahead of the flat road surface where the vehicle 1 is located.

[0067] In addition, for example Figure 5 As shown in (c), based on the distance to and height of the predetermined points P4 to P6 on the road surface, a slope such as an uphill slope is detected slightly ahead of the flat road surface where the vehicle 1 is located.

[0068] In addition, for example Figure 5 As shown in (d), when the vehicle 1 is on an inclined road such as a downhill slope, a flat road surface parallel to the vehicle 1 is detected based on the distance and height to the predetermined points P1-P3 on the road surface. In addition, a flat road surface parallel to the road surface where the vehicle 1 is located is detected based on the distance and height to the predetermined points P4-P6 on the road surface. That is, the road surface state detection is performed by treating the road surface at the current position of the vehicle 1 as a road surface parallel to the vehicle 1, regardless of the inclination of the road surface with respect to the gravity direction.

[0069] In this way, the ECU 14 of the embodiment uses the direction of the normal vector Vc of the road surface determined by the calibration process as the normal value, that is, the direction that the normal vector Vr of the road surface should have. That is, it is estimated that the direction of the normal vector Vr perpendicular to the detected road surface this time is consistent with the direction of the normal vector Vc as the normal value. In addition, using the height position Hc of the road surface determined by the calibration process as the normal value, that is, the height that the road surface should have, an object having a height inconsistent with the height position Hc is detected. As described in the above example, an object having a height inconsistent with the height position Hc refers to, for example, objects such as small stones dropped on the road surface or the unevenness of the road surface itself. As Figure 5 As shown in the example of (c), there may be another road surface having a slope different from that of the road surface on which the vehicle 1 is located.

[0070] The road surface information used for detecting the road surface state is not limited to the predetermined points P1 to P6 described above. For example, the ECU 14 acquires road surface information covering the entire area within the range that the imaging unit 15 c can capture, and identifies the road surface state.

[0071] Next, use Figure 6-Figure 8 The functions of the ECU 14 according to the embodiment will be described in more detail. Figure 6-Figure 8 The following processing by the ECU 14 is generally performed based on the captured image data captured by the image capturing unit 15c in front of the vehicle 1. As described above, the captured image data is acquired by the image acquisition unit 401 of the ECU 14.

[0072] First, a case where the vehicle 1 maintains a posture parallel to the road surface will be described.

[0073] like Figure 6As shown in (a), the three-dimensional model generation unit 402 generates a three-dimensional model M based on the captured image data obtained by the image acquisition unit 401. In the three-dimensional model M, various objects such as the road surface, the unevenness of the road surface itself, and objects on the road surface are transformed into a plurality of points arranged in three dimensions according to the distance from the vehicle 1 to the various objects and the heights of the various objects.

[0074] The correction unit 403 estimates the position and orientation of the road surface from the three-dimensional model M generated by the three-dimensional model generation unit 402. Assuming that the road surface is a plane with ideal flatness that does not include bumps and other objects, the position and orientation of the road surface are estimated. This type of plane can be determined using robust estimation such as RANSAC. In robust estimation, when the obtained measurement value includes an outlier, the outlier is removed to estimate the regularity of the observed object. That is, here, the points Px, Py, and Pz representing bumps and other objects included in the three-dimensional model M are removed, and by estimating the flatness of each position and each orientation of the three-dimensional model M, a plane facing a specified direction at a specified position can be obtained.

[0075] like Figure 6 As shown in (b), the correction unit 403 calculates a normal vector Vr perpendicular to the estimated plane R as described above. In addition, after making the direction of the normal vector Vr of the plane R consistent with the direction of the normal vector Vc as a normal value determined by the calibration process, the correction unit 403 corrects the height position of the entire three-dimensional point cloud included in the three-dimensional model M based on the height position Hc as a normal value determined by the calibration process. In the corrected three-dimensional model M, the points Px, Py, and Pz removed as outliers are inconsistent with the height position Hc, and are represented as bumps with a specified height on the road surface.

[0076] exist Figure 6 (b), the direction of the normal vector Vr of the plane R is consistent with the direction of the normal vector Vc which is a normal value, and it can be considered that the correction by the correction unit 403 is not necessary. However, Figure 6 The example of is only an ideal example. In the vehicle 1, there is always a possibility of shaking due to the unevenness of the road surface, and the slope of the normal vector Vr of the plane R with respect to the normal vector Vc always changes.

[0077] Here, a situation in which the vehicle 1 shakes will be described.

[0078] Figure 7This shows a situation where the vehicle 1 is stranded on an object on the road. In this case, although it is the vehicle 1 that is actually tilted, the image data captured by the imaging unit 15 should show that the road surface on the right side of the vehicle 1 is tilted and the road surface on the left side is lowered.

[0079] like Figure 7 As shown in (a), the three-dimensional model generation unit 402 generates a three-dimensional model M based on the captured image data acquired by the image acquisition unit 401.

[0080] like Figure 7 As shown in (b), the correction unit 403 estimates the position and orientation of the plane R from the three-dimensional model M generated by the three-dimensional model generation unit 402. Next, the correction unit 403 calculates the normal vector Vr of the estimated plane R. Figure 7 In the example shown in (b), the normal vector Vr of the plane R does not coincide with the normal vector Vc which is a normal value.

[0081] like Figure 8 As shown in (a), in order to make the normal vector Vr of the plane R consistent with the normal vector Vc which is a normal value, the correction unit 403 corrects the slope of the normal vector Vr of the plane R. In other words, the correction unit 403 performs offset processing on the normal vector Vr of the plane R so that the plane R becomes parallel to the front, rear, left, and right axes of the vehicle 1.

[0082] like Figure 8 As shown in (b), the correction unit 403 matches the height position of the plane R with the height position Hc which is a normal value. At this time, the height position of the entire three-dimensional point cloud included in the three-dimensional model M is corrected. In other words, the height position of the entire three-dimensional point cloud is corrected to the height position with the imaging unit 15c as the viewpoint.

[0083] As a result, the points included in the plane R in the three-dimensional point cloud overlap at the height position Hc with the virtual road surface parallel to the vehicle 1. In addition, the points Px, Py, and Pz representing the bumps and depressions of the road surface and other objects in the three-dimensional point cloud do not overlap with the virtual road surface, but are represented as bumps and depressions with a specified height.

[0084] Example of road surface detection processing by ECU

[0085] Next, use Fig. 9 An example of the road surface detection process performed by the ECU 14 according to the embodiment will be described. Fig. 9 1 is a flowchart showing an example of the procedure of the road surface detection process performed by the ECU 14 according to the embodiment.

[0086] like Fig. 9As shown, the image acquisition unit 401 of the ECU 14 acquires the image data captured by the image capture unit 15 (step S101). In the detection of the road surface state described below, the image data captured by the image capture unit 15c provided in front of the vehicle 1 is generally used. The image data is preferably dynamic image data.

[0087] The three-dimensional model generation unit 402 generally generates a three-dimensional model M with a plurality of points arranged in three dimensions from the image data captured by the imaging unit 15c (step S102). The three-dimensional model M includes the uneven state of the road surface, including the unevenness of the road surface itself and objects on the road surface.

[0088] The correction unit 403 specifies the position and orientation of the plane R from the generated three-dimensional model M (step S103). That is, the correction unit 403 estimates the plane R having a predetermined position and orientation based on calculations such as robust estimation from the three-dimensional model M. The estimated plane R does not include data indicating a concave-convex state.

[0089] The correction unit 403 calculates a normal vector Vr with respect to the identified plane R (step S104 ).

[0090] The correction unit 403 corrects the slope of the normal vector Vr of the plane R (step S105). Specifically, the correction unit 403 performs correction by tilting the normal vector Vr of the plane R by a predetermined angle as required so that the normal vector Vr of the plane R becomes perpendicular to the normal vector Vc which is a normal value.

[0091] The correction unit 403 corrects the height position of the entire three-dimensional model M (step S106). That is, the correction unit 403 corrects the height position of the entire three-dimensional model M based on the height position Hc as a normal value. The height position of the entire three-dimensional model M is corrected by relatively moving all positions of the three-dimensional point cloud included in the three-dimensional model M by a distance corresponding to the movement distance of the three-dimensional point cloud included in the plane R.

[0092] Through the processing from step S103 to step S106 of the correction unit 403, the three-dimensional model is corrected so that the plane R becomes horizontal relative to the vehicle 1 with the normal vector as the reference. As a result, the points represented as the points of the plane R estimated from the three-dimensional point cloud coincide with the height position Hc of the virtual road surface, and the other points are determined as the unevenness of the road surface with a predetermined height. The unevenness with a predetermined height on the road surface may be the unevenness of the road surface itself, an object on the road surface, or other road surfaces such as an inclined road with an inclination different from that of the road surface on which the vehicle 1 is located.

[0093] Through the above-described processing, the road surface detection processing of the ECU 14 according to the embodiment ends.

[0094] Comparative Example

[0095] For example, as a comparative example, assume a structure that does not perform the correction as in the above-mentioned embodiment. In such a comparative example, while the vehicle is moving, the vehicle shakes up and down and left and right based on the road surface conditions or the driver's operation. From the viewpoint of the stereo camera, the road surface is always shaking, resulting in a deterioration in the detection accuracy of the height of a predetermined position on the road surface. There are also cases where bumps and depressions that do not exist are detected as if they exist due to the momentary shaking of the vehicle.

[0096] The ECU 14 of the embodiment estimates the plane R obtained by calculation from the three-dimensional model, takes the normal vector Vc and the height position Hc as normal values, and corrects the three-dimensional model M so that the direction of the normal vector Vr of the estimated plane R and the height position of the plane R are consistent with it. In this way, the influence of the shaking of the vehicle 1 can be suppressed and the detection accuracy of the road surface height can be improved.

[0097] Fig.10 This is a graph showing the detection results of a flat road surface obtained by the ECU 14 of the embodiment and the configuration of a comparative example. Fig.10 The horizontal axis of the graph represents the direction of vehicle travel, and the vertical axis represents the unevenness of the road surface. The road surface is taken as the 0 point, the upper direction sandwiching the 0 point is the convex (positive) side, and the lower direction is the concave (negative) side.

[0098] like Fig.10 As shown, in the configuration of the comparative example in which the three-dimensional model is not corrected, two large convex portions are detected even when the vehicle is traveling on a flat road surface. In contrast, in the ECU 14 of the embodiment, a substantially flat road surface state is detected.

[0099] The ECU 14 of the embodiment can thus detect the height of the unevenness of the road surface with high accuracy. Therefore, the ECU 14 of the embodiment can be applied to, for example, a parking assist system and a suspension control system of the vehicle 1 .

[0100] For example, in the parking assist system of the vehicle 1, by accurately understanding the height of the unevenness of the road surface, it is possible to accurately predict the moving direction of the vehicle 1 when passing through a predetermined path. This allows the vehicle 1 to be guided to the target parking space more reliably.

[0101] Furthermore, by accurately understanding the height of the unevenness of the road surface, the suspension control system of the vehicle 1 can accurately predict the shaking of the vehicle 1 when passing through a predetermined path. This can more reliably suppress the shaking of the vehicle 1.

[0102] Other Implementations

[0103] The road surface detection program executed by the ECU 14 of the above-mentioned embodiment may be provided or distributed via a network such as the Internet. That is, the road surface detection program may be provided in a form of being stored on a computer connected to a network such as the Internet and downloaded via the network.

[0104] The above examples illustrate the embodiments of the present invention, but the above embodiments and variations are only examples and are not intended to limit the scope of the invention. The above embodiments or variations can be implemented in various other ways, and various omissions, substitutions, combinations and changes can be made without departing from the scope of the invention. In addition, the structure or shape of each embodiment or each variation can be partially replaced and implemented.

[0105] Explanation of symbols

[0106] 1…vehicle, 8…display device, 14…ECU, 15…shooting unit, 401…image acquisition unit, 402…three-dimensional model generation unit, 403…correction unit, 404…storage unit, Hc…height position, M…three-dimensional model, R…plane, Vc, Vr…normal vector.

Claims

1. A road surface detection device, characterized in that: have: an image acquisition unit that acquires captured image data output from a stereo camera that captures a capture area including a road surface on which the vehicle is traveling; a three-dimensional model generating unit for generating a three-dimensional model of the photographing area including the surface shape of the road surface with the stereo camera as a viewpoint based on the photographed image data; as well as A correction unit estimates a plane from the three-dimensional model and corrects the three-dimensional model so that the direction of the normal vector of the plane and the height position of the plane relative to the stereo camera match the normal value of the direction of the normal vector of the road surface and the normal value of the height position of the road surface relative to the stereo camera.

2. The road surface detection device according to claim 1, characterized in that: The correction unit, After matching the orientation of the normal vector of the plane with the normal value of the orientation of the normal vector of the road surface, the entire three-dimensional model is corrected so that the height position of the plane relative to the stereo camera matches the normal value of the height position of the road surface relative to the stereo camera.

3. The road surface detection device according to claim 2, characterized in that: The correction unit, Based on the values ​​determined during the calibration process of the stereo camera, a normal value of the direction of the normal vector of the road surface and a normal value of the height position of the road surface relative to the stereo camera are acquired.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a road surface detection program, which enables the computer to execute the following steps: An image acquisition step of acquiring captured image data output from a stereo camera that captures a capture area including a road surface on which the vehicle is traveling; A three-dimensional model generating step, based on the captured image data, generating a three-dimensional model of the captured area including the surface shape of the road surface with the stereo camera as a viewpoint; as well as A correction step is to estimate a plane from the three-dimensional model and correct the three-dimensional model so that the direction of the normal vector of the plane and the height position of the plane relative to the stereo camera match the normal value of the direction of the normal vector of the road surface and the normal value of the height position of the road surface relative to the stereo camera, respectively.

Citation Information

Patent Citations

  • Image processing apparatus, image processing method, and program

    CN102737405A

  • System and method for generating 3D high-precision map based on vehicular multi-sensor fusion

    CN107505644A

  • Drawing device

    JP2003132349A

  • Device and method for detecting plane from image

    JP2011129048A

  • Vehicle image processor and vehicle image processing method

    JP2012123750A