Driving assistance device and vehicle

Abstract

A driving assistance device according to one embodiment of the present disclosure can execute the following three steps. (1) Calculating the number of pixels of a specific region when a bent line is present in a straight line-shaped light pattern included in image data for a travel road surface, said specific region being formed by the bent line and a straight line from if the bending were not present in the straight line-shaped light pattern, and acquiring location data, in the first image data, for the specific region. (2) Estimating the depth or height and / or the size of a recess / protrusion of the travel road surface, on the basis of the number of pixels and the location data. (3) Performing notification control or travel control in accordance with the data for the recess / protrusion of the travel road surface that was obtained as a result of the aforementioned estimation.

Description

Driving aids and vehicles

[0001] The present disclosure relates to a driving assistance device mounted on a vehicle, and a vehicle equipped with such a driving assistance device.

[0002] There may be unevenness on the road surface, such as depressions or fallen obstacles. In this case, when a vehicle goes over the unevenness, depending on the depth and height of the unevenness, not only may the ride be uncomfortable for the occupants, but the vehicle may also get stuck in the depression or collide with an obstacle and run off the road. As a result, driving safety may be reduced.

[0003] For example, Patent Documents 1 and 2 disclose a method of irradiating a road surface ahead of a vehicle with laser light in order to detect the shape of the road surface and obstacles on the road.

[0004] Japanese Patent No. 6962464 Japanese Patent Application Laid-Open No. 2010-18080

[0005] A driving assistance device according to an embodiment of the present disclosure includes an acquisition unit capable of acquiring image data of a road surface ahead of a vehicle, and a control unit capable of deriving the depth or height of unevenness of the road surface based on the image data. The control unit is capable of performing the following four operations: (A1) controlling the irradiation of a linear light pattern onto the road surface, thereby causing the acquisition unit to acquire first image data of the road surface irradiated with the linear light pattern as image data; (A2) when a bend is present in the linear light pattern included in the first image data, calculating the number of pixels of a specific area formed by the straight line and the bend when it is assumed that there is no bend in the linear light pattern, and acquiring position data of the specific area in the first image data; (A3) estimating at least one of the depth or height and size of unevenness of the road surface based on the number of pixels and the position data; and (A4) performing notification control or driving control in accordance with the data on unevenness of the road surface obtained as a result of the above estimation.

[0006] A vehicle according to an embodiment of the present disclosure includes a driving assistance device and a controlled device controlled by the driving assistance device. The driving assistance device includes an acquisition unit capable of acquiring image data of a road surface ahead of the vehicle, and a control unit capable of deriving the depth or height of unevenness on the road surface based on the image data. The control unit is capable of performing the following four operations: (B1) controlling the irradiation of a linear light pattern onto the road surface, thereby causing the acquisition unit to acquire first image data of the road surface irradiated with the linear light pattern as image data; (B2) when a bend is present in the linear light pattern included in the first image data, calculating the number of pixels in a specific area formed by the straight line and the bend when it is assumed that there is no bend in the linear light pattern, and acquiring position data of the specific area in the first image data; (B3) estimating at least one of the depth or height and size of unevenness on the road surface based on the number of pixels and the position data; and (B4) performing notification control or driving control in accordance with the data on unevenness on the road surface obtained as a result of the above estimation.

[0007] The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate one embodiment and, together with the description, serve to explain the principles of the disclosure.

[0008] FIG. 1 is a diagram illustrating an example of the appearance of a front portion of a vehicle according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a view ahead of the vehicle when viewed from the driver's seat of the vehicle of FIG. 1. FIG. 3 is a diagram illustrating an example of a state of the vehicle and a road surface when viewed from above the vehicle of FIG. 1. FIG. 4 is a diagram illustrating an example of a functional block of a cruise control device mounted on the vehicle of FIG. 1. FIG. 5 is a diagram illustrating an example of a network environment capable of communicating with the cruise control device of FIG. 4. FIG. 6 is a diagram illustrating an example of a state in which an image obtained by capturing an image ahead of the vehicle while irradiating a linear visible laser pattern onto the road surface from the vehicle of FIG. 1 is displayed on a display screen of the vehicle of FIG. 1. FIG. 7(A) is a diagram illustrating an example of a linear visible laser pattern included in an image obtained by capturing an image ahead of the vehicle while irradiating a linear visible laser pattern onto the road surface from the vehicle of FIG. 1. FIG. 7(B) is a diagram illustrating an example of a virtual line obtained from the linear visible laser pattern included in the image of FIG. 7(A). 8(A) is a diagram illustrating an example of a linear visible laser pattern included in an image obtained by capturing an image of the area ahead of the vehicle when a linear visible laser pattern is irradiated onto a road surface from the vehicle of FIG. 1 . FIG. 8(B) is a diagram illustrating an example of a virtual line obtained from the linear visible laser pattern included in the image of FIG. 8(A) . FIG. 9 is a diagram illustrating an example of a driving assistance procedure in the cruise control device of FIG. 4 . FIG. 10 is a diagram illustrating an example of a driving assistance procedure subsequent to FIG. 9 . FIG. 11 is a diagram illustrating an example of the appearance of a front portion of a vehicle according to a second embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example of a functional block of a cruise control device mounted on the vehicle of FIG. 11 . FIG. 13 is a diagram illustrating an example of a view ahead of the vehicle when viewed from the driver's seat of the vehicle of FIG. 11 . FIG. 14 is a diagram illustrating an example of a state in which an image obtained by capturing an image of the area ahead of the vehicle when a linear visible laser pattern is irradiated onto a road surface from the vehicle of FIG. 11 is displayed on a display screen of the vehicle of FIG. 11 . FIG. 15 is a diagram illustrating an example of a driving assistance procedure in the cruise control device of FIG. 12 . Fig. 16 is a diagram for explaining an example of the driving assistance procedure following Fig. 15. Fig. 17 is a diagram showing a modified example of the functional blocks of the driving control device of Fig. 4.

[0009] The road surface may be uneven, such as when a depression is formed on the road surface or when an obstacle has fallen on it. In this case, when a vehicle travels over the unevenness, depending on the depth and height of the unevenness, not only may the ride be uncomfortable for the occupants, but the vehicle may also get stuck in the depression or collide with an obstacle and run off the road. As a result, driving safety may be reduced. It is desirable to provide a driving assistance device that can detect unevenness on the road surface in real time and perform notification control or driving control in accordance with the detection results, and a vehicle equipped with such a driving assistance device.

[0010] Some exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following description illustrates one specific example of the present disclosure and should not be construed as limiting the present disclosure. For example, each element, including numerical values, shapes, materials, parts, the position of each part, and the connection method of each part, is merely an example and should not be construed as limiting the present disclosure. Furthermore, in the following exemplary embodiments, components not described in independent claims based on the highest concept of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be drawn to scale. Throughout this specification and the drawings, components having substantially the same function and configuration are designated by the same reference numerals, and redundant description will be omitted. Furthermore, components not directly related to one embodiment of the present disclosure are not shown in the drawings.

[0011] The present disclosure will be described in the following order: 1. First embodiment (FIGS. 1 to 10): An example in which a linear visible laser pattern is projected onto the road surface ahead of the vehicle 2. Second embodiment (FIGS. 11 to 16): An example in which a linear near-infrared laser pattern is projected onto the road surface ahead of the vehicle 3. Modification of the first embodiment (FIG. 17): An example in which a linear visible laser pattern and a linear near-infrared laser pattern are projected onto the road surface ahead of the vehicle

[0012] 1. First Embodiment [Configuration] FIG. 1 illustrates an example of the appearance of a front portion of a vehicle 100 according to a first embodiment of the present disclosure. The drive system of the vehicle 100 is not particularly limited, and the vehicle 100 can be driven, for example, by at least one of an engine and a motor. As shown in FIG. 1 , the vehicle 100 includes a pair of left and right headlights HL and a front windshield FW at its front portion. A visible laser irradiator 120 is provided inside the headlight HL. A stereo camera 110 including a main camera 111 and a sub-camera 112 is provided inside the vehicle interior CR, which is visible from the outside through the front windshield FW. The location of the visible laser irradiator 120 is not particularly limited as long as it is located at the front portion of the vehicle 100. For example, the visible laser irradiator 120 may be provided inside the headlight farther from the sidewalk of the pair of left and right headlights HL. FIG. 1 illustrates an example of the vehicle 100 traveling on a road surface 300.

[0013] Fig. 2 shows an example of a view ahead of the vehicle 100 when the driver looks ahead from the driver's seat while the vehicle 100 is traveling on a road surface 300. Fig. 3 shows an example of the appearance of the vehicle 100 and the road surface 300 when the vehicle 100 and the road surface 300 are viewed from above the vehicle 100. For ease of explanation, broken lines are drawn in Figs. 2 and 3 at locations where the left and right tires TR of the vehicle 100 are expected to pass (expected passage areas 310). Figs. 2 and 3 also show, as an example, the presence of side strips 320 on both sides of the road surface 300.

[0014] As shown in Figures 2 and 3 , a depression 330 exists ahead of the vehicle. However, it may not be easy for the driver to see the depression 330. For example, it is extremely difficult for the driver to see the depression 330 during a snowstorm, at twilight, or at night when there are no street lights. Therefore, as shown in Figures 2 and 3 , the vehicle 100 uses the laser irradiation unit 120 to irradiate a laser beam L onto the road surface 300 ahead of the vehicle, thereby generating a linear visible laser pattern LP on the road surface 300. The linear visible laser pattern LP makes the depression 330 appear three-dimensionally, making it easier for the driver to see the depression 330. Note that Figures 2 and 3 illustrate an example in which the linear visible laser pattern LP is irradiated onto a portion of the road surface 300 where a predicted passage area 310 is likely to exist. When there are no irregularities on the road surface 300, the linear visible laser pattern LP is a straight line without bent lines LPa, LPb or interruptions α and γ (described later). When there are irregularities on the road surface 300, the linear visible laser pattern LP includes, in addition to a straight line, bent lines LPa, LPb and interruptions α and γ (described later).

[0015] The laser pattern generated by irradiating the laser beam L onto the road surface 300 is linear in order to reduce the image processing load. Furthermore, the laser pattern extends in a direction that obliquely intersects with the longitudinal direction of the predicted passage area 310 (i.e., the traveling direction of the vehicle 100) in order to make it easier to detect bends and discontinuities in the laser pattern on the image.

[0016] Fig. 4 shows an example of functional blocks of the cruise control device 1000. The vehicle 100 is equipped with the cruise control device 1000. The cruise control device 1000 corresponds to a specific example of a "driving assistance device" according to an embodiment of the present disclosure. Fig. 5 shows an example of functional blocks of a control device 2000 provided in a network environment NW to which the cruise control device 1000 is connected via wireless communication.

[0017] The control device 2000 is capable of sequentially integrating and updating the road map information transmitted from the cruise control devices 1000 of the respective vehicles, and transmitting the updated road map information to the respective vehicles. The control device 2000 includes, for example, a road map information integration ECU 270 and a transceiver 280.

[0018] The road map boundary information integration ECU 270 is capable of continuously updating road map information surrounding a vehicle on a road by integrating road map information collected from multiple vehicles via the transceiver 280. The road map information may be, for example, a dynamic map, and includes static information and quasi-static information that mainly constitute road information, and quasi-dynamic information and dynamic information that mainly constitute traffic information.

[0019] Static information that makes up road information includes information that needs to be updated within one month, such as roads, road structures, lane information, road surface information, and permanent traffic regulations. "Roads" include, for example, road locations and shapes, intersections, and road attributes (e.g., national roads, prefectural roads, city roads, private roads, priority roads, non-priority roads, general roads, and expressways). "Road structures" include, for example, traffic signs, traffic lights, convex mirrors, and pedestrian bridges.

[0020] The quasi-static information that constitutes the road information is made up of information that needs to be updated every hour, such as traffic regulation information due to road construction or events, wide-area weather information, and congestion forecasts.

[0021] The semi-dynamic information that constitutes traffic information is composed of information that requires updating within one minute, such as the actual traffic congestion situation at the time of observation, driving restrictions, temporary driving obstructions such as fallen objects and obstacles, actual accident conditions, and narrow-area weather information.

[0022] The dynamic information that constitutes the traffic information is composed of information that needs to be updated every second, such as information sent and exchanged between mobile units, information on currently displayed traffic signals, information on pedestrians and bicycles at intersections, information on vehicles traveling on roads, etc. Such road map information is maintained and updated periodically until the next information is received from each vehicle, and the updated road map information is transmitted to each vehicle as appropriate via the transceiver 280.

[0023] 4, the cruise control device 1000 includes, for example, a stereo camera 110, a visible laser irradiation unit 120, a vehicle state quantity sensor 130, a GNSS receiver 140, a transceiver 150, a control flag input unit 160, a notification unit 170, and a high-precision road map DB 180. The cruise control device 1000 may further include components other than those shown in FIG. 4. FIG. 4 illustrates an example of a portion of the configuration of the cruise control device 1000.

[0024] The stereo camera 110 is fixed, for example, to the upper center of the vehicle interior CR and includes, for example, a main camera 111 and a sub-camera 112. The main camera 111 and the sub-camera 112 are autonomous sensors that sense the real space ahead of the vehicle 100. The main camera 111 and the sub-camera 112 are, for example, arranged symmetrically on either side of the central portion in the width direction of the vehicle 100, enabling stereo imaging of the area ahead of the vehicle 100 from different viewpoints. The main camera 111 and the sub-camera 112 are capable of acquiring stereo image data of the area ahead of the vehicle 100 under control of a control unit 210 (described below). The stereo camera 110 is further capable of outputting, to the control unit 210, stereo image data of the area ahead of the vehicle 100 obtained by capturing images with the main camera 111 and the sub-camera 112. The stereo image data is composed of visible area image data Ia obtained by the main camera 111 and visible area image data obtained by the sub-camera 112. The image data obtained by the sub-camera 112 may be image data Ia. The image data Ia corresponds to a specific example of "first image data" in the present disclosure. The stereo camera 110 can further generate distance image data Ib calculated from the amount of deviation between the positions of corresponding objects based on the obtained stereo image data, and output the distance image data Ib to the control unit 210.

[0025] The visible laser irradiator 120 is provided inside the headlight HL, for example, as shown in Fig. 1 . Under control of the control unit 210, the visible laser irradiator 120 scans a laser beam L in the visible range on the road surface (traveling road surface 300) ahead of the vehicle 100, thereby generating a linear visible laser pattern LP on the traveling road surface 300. Under control of the control unit 210, the visible laser irradiator 120 is capable of scanning the laser beam L at a location on the traveling road surface 300 where an expected passage area 310 is likely to exist. Under control of the control unit 210, the visible laser irradiator 120 is capable of scanning the laser beam L in a direction obliquely intersecting the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100). This enables the visible laser irradiation unit 120 to generate a linear visible laser pattern LP that extends in a direction that diagonally intersects the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100) at a location on the road surface 300 where the expected passage area 310 is likely to be present. The visible laser irradiation unit 120 is capable of emitting, as the laser beam L, a single-wavelength laser beam that is included in a wavelength band (e.g., a green wavelength band) that is different from the wavelength band of colors generally used on the road surface 300. Examples of "colors generally used on the road surface 300" include the colors of paved or unpaved roads, and the colors of dividing lines on paved roads.

[0026] The laser irradiation unit 130 includes, for example, a laser emitter capable of emitting visible laser light, an emission control driver capable of controlling the emission of the laser emitter, an optical system capable of scanning the laser light on the road surface 300, and a scan control driver capable of controlling the scanning of the laser light by the optical system. The emission control driver is capable of controlling the emission of the laser emitter under control of the control unit 210. The scan control driver is capable of controlling the operation of the optical system under control of the control unit 210. The laser emitter includes, for example, a semiconductor laser that emits visible laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror reflects the visible laser light emitted from the laser emitter and is capable of scanning the reflected visible laser light on the road surface 300 via the fθ lens.

[0027] The vehicle state quantity sensor 130 is configured to include various sensors such as an acceleration sensor, a vehicle speed sensor, and a gyro sensor. The vehicle state quantity sensor 130 is capable of outputting detection signals obtained by the various sensors to the control unit 210. The GNSS receiver 140 is capable of receiving positioning signals transmitted from a plurality of positioning satellites. The GNSS receiver 140 is capable of outputting the received positioning signals to the control unit 210.

[0028] The control flag input unit 160 is capable of receiving input of a control flag 192 from the driver. The control flag input unit 160 is, for example, a paddle shifter attached to a steering wheel. For example, when the driver simultaneously presses and holds the left and right paddle shifters, the control flag input unit 160 is capable of storing “1” as the control flag 192 in the storage unit 190. For example, when the driver simultaneously presses and holds the left and right paddle shifters again after previously storing “1” as the control flag 192 in the storage unit 160, the control flag input unit 160 is capable of storing “0” as the control flag 192 in the storage unit 190. For example, when the driver simultaneously presses and holds the left and right paddle shifters again after previously storing “0” as the control flag 192 in the storage unit 190, the control flag input unit 160 is capable of storing “1” as the control flag 192 in the storage unit 190.

[0029] When the control flag 192 is "1", it means that the mode is, for example, laser irradiation mode. When the control flag 192 is "0", it means that the mode is, for example, normal mode in which laser irradiation is not performed automatically. Note that the values ​​that the control flag 192 can take are not limited to those mentioned above.

[0030] The notification unit 170 includes, for example, a liquid crystal display panel or an organic EL display panel and a speaker. The notification unit 170 can display an image on the display screen 170A based on a video signal input from the notification control unit 23 (described later), and can output a sound based on a sound signal input from the notification control unit 23 (described later).

[0031] The high-precision road map DB 180 is stored in a large-capacity storage medium such as an HDD. The high-precision road map DB 180 includes high-precision road map information (dynamic map). Similar to the road map information included in the road map information integration_ECU 270, this high-precision road map information includes static information and quasi-static information that mainly constitute road information, and quasi-dynamic information and dynamic information that mainly constitute traffic information.

[0032] The storage unit 190 is configured, for example, by a non-volatile memory. The storage unit 190 stores, for example, drawing data 191, a control flag 192, a resolution table 193, and a front position table 194. The resolution table 193 corresponds to a specific example of a "size data table" in the present disclosure. The drawing data 191 includes data for generating a linear visible laser pattern LP at a predetermined location on the road surface (traveling road surface 300) ahead of the vehicle 100. The control flag 192 includes a flag (for example, "0" or "1") input from the control flag input unit 160.

[0033] The resolution table 193 includes size data for the image data Ia in a predetermined pixel unit (e.g., 1 pixel). The resolution table 193 includes Y-direction resolution for the image data Ia in a direction (Y direction in FIG. 7 described later) corresponding to the traveling direction of the vehicle 100 (Y direction in FIG. 3 ) for each predetermined pixel unit (e.g., 1 pixel). The resolution table 193 further includes X-direction resolution for the image data Ia in a direction (X direction in FIG. 7 described later) corresponding to a direction (X direction in FIG. 3 ) perpendicular to the traveling direction of the vehicle 100 (Y direction in FIG. 3 ). For example, the resolution table 193 stores (2 mm, 3 mm) as the pixel resolution (X-direction resolution, Y-direction resolution) corresponding to a position 20 m ahead from the front end of the vehicle 100. For example, the resolution table 193 stores (1 mm, 2 mm) as the pixel resolution (X-direction resolution, Y-direction resolution) corresponding to a position 5 m ahead from the front end of the vehicle 100.

[0034] The front position table 194 includes position data of the road surface 300 ahead of the vehicle 100 for each pixel in the image data Ia. Assume that the image data Ia is composed of m×n pixels, where m is the number of pixels in the X direction and n is the number of pixels in the Y direction. In this case, the front position table 194 specifies, for example, that the position data of the (m / 2)th pixel in the X direction of the image data Ia and the n-th pixel in the Y direction of the image data Ia is the center position in the width direction of the vehicle 100 and a position 20 m ahead from the front end of the vehicle 100. Furthermore, the front position table 194 specifies, for example, that the position data of the (m / 2)th pixel in the X direction of the image data Ia and the first pixel in the Y direction of the image data Ia is the center position in the width direction of the vehicle 100 and a position 5 m ahead from the front end of the vehicle 100.

[0035] As shown in FIG. 4 , the cruise control device 1000 further includes a control unit 210, a throttle actuator 270, a brake actuator 280, and a steering actuator 290. The control unit 210 corresponds to a specific example of an “acquisition unit” or “control unit” in the present disclosure. The throttle actuator 270, the brake actuator 280, and the steering actuator 290 correspond to a specific example of a “controlled device” in the present disclosure. The control unit 210 is capable of controlling the entire vehicle 100. The control unit 210 is, for example, a so-called ECU (Electronic Control Unit) and includes, for example, one or more processors and one or more memories. The control unit 210 may also include, for example, a CPU (Central Processing Unit). In this case, the control unit 210 may be capable of controlling the entire vehicle 100 by, for example, executing a program stored in the storage unit 190.

[0036] The control unit 210 has, for example, a driving assistance unit 220 as shown in Fig. 4. The driving assistance unit 220 is capable of assisting the driver in driving the vehicle 100. The driving assistance unit 220 has, for example, a laser irradiation control unit 21, a road surface shape estimation unit 22, and a notification control unit 23 as shown in Fig. 4.

[0037] The laser irradiation control unit 21 is capable of controlling the irradiation (drawing) of the laser beam L from the visible laser irradiation unit 120. The laser irradiation control unit 21 is capable of generating a control signal required for the irradiation (drawing) of the linear visible laser pattern LP based on, for example, drawing data 191 (described later) in the storage unit 190, and outputting the control signal to the visible laser irradiation unit 120.

[0038] The laser irradiation control unit 21 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the laser beam L over the road surface ahead of the vehicle 100 on the travel road surface 300, and outputting the control signal to the variable laser irradiation unit 120. The laser irradiation control unit 21 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the laser beam L over a portion of the travel road surface 300 where the expected passage area 310 is likely to be present. The laser irradiation control unit 21 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the laser beam L in a direction obliquely intersecting the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100).

[0039] When the laser irradiation control unit 21 outputs the control signal to the variable laser irradiation unit 120, the control unit 210 outputs a signal to control imaging to the stereo camera 110, thereby enabling the stereo camera 110 to acquire image data Ia including the linear visible laser pattern LP. When the laser irradiation control unit 21 outputs the control signal to the variable laser irradiation unit 120, the control unit 210 outputs a signal to control imaging to the stereo camera 110, thereby enabling the road surface shape estimation unit 22 to acquire image data Ia including the linear visible laser pattern LP obtained by the stereo camera 110. At this time, the control unit 210 outputs a signal to set the resolution to the stereo camera 110, thereby enabling the stereo camera 110 to acquire relatively low-resolution image data Ia or relatively high-resolution image data Ia. The relatively low-resolution image data Ia corresponds to a specific example of "second image data" in the present disclosure. The relatively high-resolution image data Ia corresponds to a specific example of "third image data" in the present disclosure.

[0040] The road surface shape estimation unit 22 is capable of acquiring one piece of image data Ia or multiple pieces of image data Ia in time series, which are obtained by the stereo camera 110 and include the linear visible laser pattern LP. Hereinafter, one piece of image data Ia including the linear visible laser pattern LP obtained by the stereo camera 110 will be referred to as one piece of image data IaLP. Furthermore, multiple pieces of image data Ia in time series, which are obtained by the stereo camera 110 and include at least a portion of the linear visible laser pattern LP, will be referred to as multiple pieces of image data IaLP in time series. The road surface shape estimation unit 22 is capable of detecting unevenness of the road surface (traveling road surface 300) ahead of the vehicle 100, based on one piece of image data IaLP or multiple pieces of image data IaLP in time series, which have relatively low resolution and are obtained by the stereo camera 110. The road surface shape estimation unit 22 is capable of detecting the presence or absence of unevenness on the road surface (traveling road surface 300) ahead of the vehicle 100, for example, based on a single image data IaLP of relatively low resolution obtained by the stereo camera 110 or multiple image data IaLP in a time series.

[0041] The road surface shape estimation unit 22 is capable of detecting, by a predetermined method, the presence or absence of unevenness on the road surface (traveling road surface 300) ahead of the vehicle 100, from a single piece of low-resolution image data IaLP or a plurality of pieces of time-series image data IaLP. For example, when a bent line and a discontinuity are present in the linear visible laser pattern LP included in the single piece of low-resolution image data IaLP or the plurality of pieces of time-series image data IaLP, the road surface shape estimation unit 22 is capable of determining that unevenness is present on the road surface (traveling road surface 300) ahead of the vehicle 100, from the single piece of low-resolution image data IaLP or the plurality of pieces of time-series image data IaLP.

[0042] The characteristics of the "bent line" differ depending on whether the unevenness is a depression 330 or an obstacle 340 on the road surface 300. Fig. 7A shows an example of a linear visible laser pattern LP included in one piece of low-resolution image data IaLP or multiple pieces of time-series image data IaLP when the unevenness is a depression 330. Fig. 8A shows an example of a linear visible laser pattern LP included in one piece of low-resolution image data IaLP or multiple pieces of time-series image data IaLP when the unevenness is an obstacle 340 on the road surface 300.

[0043] 7A , in the case where the unevenness is a depression 330, the end of the bending line LPa that is relatively far from the vehicle 100 is connected to a straight portion of the linear visible laser pattern LP, and the end that is relatively closer to the vehicle 100 is not connected to the straight portion of the linear visible laser pattern LP and is interrupted. In other words, there is an interruption α between one end of the bending line LPa (the end that is relatively closer to the vehicle 100) and the linear visible laser pattern LP. When the road surface shape estimation unit 22 detects the above-mentioned features (bending line LPa, interruption α) from the linear visible laser pattern LP in one piece of low-resolution image data IaLP or multiple pieces of image data IaLP in time series, it is possible to determine that a depression 330 exists on the road surface (traveling road surface 300) ahead of the vehicle 100.

[0044] 8A , in the case where the unevenness is an obstacle 340 on the traveling road surface 300, the end of the bending line LPb that is relatively closer to the vehicle 100 is connected to a straight portion of the linear visible laser pattern LP, and the end that is relatively farther from the vehicle 100 is not connected to the straight portion of the linear visible laser pattern LP and is interrupted. In other words, there is an interruption γ between one end of the bending line LPb (the end that is relatively farther from the vehicle 100) and the linear visible laser pattern LP. When the road surface shape estimation unit 22 detects the above-mentioned features (bending line LPb, interruption γ) from the linear visible laser pattern LP in one piece of low-resolution image data IaLP or multiple pieces of image data IaLP in time series, it is possible to determine that an obstacle 340 is present on the road surface (traveling road surface 300) ahead of the vehicle 100.

[0045] The road surface shape estimation unit 22 is capable of calculating a group of pixel coordinates in a single low-resolution image data IaLP or multiple time-series image data IaLP of a predetermined region including the detected unevenness. After calculating the group of pixel coordinates, the road surface shape estimation unit 22 is further capable of extracting partial image data (partial image data Ic) of the group of pixel coordinates from a single relatively high-resolution image data IaLP. Alternatively, the road surface shape estimation unit 22 is further capable of extracting partial image data (multiple time-series partial image data Ic) of the group of pixel coordinates from a plurality of relatively high-resolution time-series image data IaLP.

[0046] The road surface shape estimation unit 22 is capable of detecting bending lines LPa and LPb in one extracted piece of partial image data Ic or in multiple extracted pieces of partial image data Ic in time series. When the road surface shape estimation unit 22 detects a bending line LPa in one extracted piece of partial image data Ic or in multiple extracted pieces of partial image data Ic in time series, the road surface shape estimation unit 22 generates a straight line (a virtual straight line LV) on the assumption that no bending is present in the linear visible laser pattern LP, and sets a gap in the vertical direction (Y direction) in the partial image data Ic between the generated virtual straight line LV and the bending line LPa as a specific region β ( FIG. 7B ).

[0047] When the road surface shape estimation unit 22 detects a bending line LPb in one extracted partial image data Ic or multiple extracted partial image data Ic in a time series, it generates a straight line (virtual straight line LV) assuming that there is no bending in the linear visible laser pattern LP, and is able to set the gap in the vertical direction (Y direction) in the partial image data Ic between the generated virtual straight line LV and the bending line LPb as a specific region δ (Figure 8 (B)).

[0048] The road surface shape estimation unit 22 calculates the number of pixels Np1 of the generated specific region β, and is also capable of acquiring position data D1 of the specific region β in one piece of partial image data Ic or multiple pieces of partial image data Ic in time series. The number of pixels Np1 is the number of pixels corresponding to the distance in the Y-axis direction (vertical length Lv1) between the end (end A) of the bending line LPa that is not in contact with the virtual straight line LV and the point (point P1) where the bending line LPa and the virtual straight line LV are connected to each other.

[0049] The road surface shape estimation unit 22 is capable of acquiring, as the position data D1 of the specific region β, for example, coordinates (x1, y1) of a point P1 in the specific region β where the Y coordinate is maximum, and coordinates (x2, y2) of a point P2 in the specific region β where the Y coordinate is minimum. Here, the point P1 is a portion where the virtual straight line Lv and the bent line LPa are connected, and is the starting point of the bent line LPa. The point P2 is a portion where the straight line portion of the linear visible laser pattern LP is interrupted. The road surface shape estimation unit 22 is capable of estimating the position of the depression 330 ahead of the vehicle 100 based on the acquired position data D1 and the forward position table 194.

[0050] The road surface shape estimation unit 22 calculates the number of pixels Np2 of the generated specific region δ, and is also capable of acquiring position data D2 of the specific region δ in one piece of partial image data Ic or in multiple pieces of partial image data Ic in time series. The number of pixels Np2 is the number of pixels corresponding to the distance in the Y-axis direction (vertical length Lv2) between the end (end P3) of the bending line LPb that is not in contact with the virtual straight line LV and the portion (portion P4) where the virtual straight line LV and the bending line LPb are connected to each other.

[0051] The road surface shape estimation unit 22 is capable of acquiring, as the position data D2 of the specific region δ, for example, the coordinates (x3, y3) of a point P3 where the Y coordinate is maximum in the specific region δ and the coordinates (x4, y4) of a point P4 where the Y coordinate is minimum in the specific region δ. Here, the point P3 is a portion of the bending line LPa that is not connected to the imaginary straight line Lv. The point P4 is a portion where the bending line LPa and the imaginary straight line Lv are connected to each other. The road surface shape estimation unit 22 is capable of estimating the position of the obstacle 340 ahead of the vehicle 100 based on the acquired position data D2 and the forward position table 194.

[0052] The road surface shape estimation unit 22 is capable of estimating the depth of the depression 330 based on the number of pixels Np1, the position data of the end A and the location P1, the resolution table 193, and the front position table 194. For example, based on the position data of the end A and the location P1, the road surface shape estimation unit 22 is capable of reading out from the resolution table 193 size data in real space of the pixels of the vertical gap between the end A and the location P1 in the image data IaLP, and estimating the depth of the depression 330 based on the read size data, the number of pixels Np1, and the front position table 194.

[0053] The road surface shape estimation unit 22 is capable of estimating the size of the depression 330 based on the number of pixels Np1, the position data D1, the resolution table 193, and the front position table 194. For example, based on the position data D1, the road surface shape estimation unit 22 is capable of reading out size data in real space of the pixels of the depression 330 defined by the position data D1 from the resolution table 193, and estimating the size of the depression 330 based on the read size data, the number of pixels Np1, and the front position table 194.

[0054] The road surface shape estimation unit 22 is capable of estimating the height and size of the obstacle 340 based on the number of pixels Np2, the position data D2, the resolution table 193, and the front position table 194. For example, the road surface shape estimation unit 22 is capable of estimating the height and size of the obstacle 340 by reading, from the resolution table 193, the size data of the obstacle 340 in real space that is defined by the position data D2, and estimating the height and size of the obstacle 340 based on the read size data, the number of pixels Np2, and the front position table 194.

[0055] The notification control unit 23 is capable of performing notification control in accordance with the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation by the road surface shape estimation unit 22. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the notification control unit 23 is capable of performing control to notify the driver to avoid the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the notification control unit 23 is capable of performing control to notify the driver of the presence of the depression 330 or the obstacle 340.

[0056] In both a manual driving mode (described later) and a cruise control mode (described later), the notification control unit 23 is capable of outputting to the notification unit 170 a video signal for displaying an image in which a marker indicating the position of a dent 330 or an obstacle 340 (e.g., a surrounding marker MK surrounding the dent 330 or the obstacle 340) is superimposed on the image data IaLP, as shown in Fig. 6. In the cruise control mode (described later), when the depth of the dent 330 or the height of the obstacle 340 exceeds a predetermined threshold, the notification control unit 23 is capable of outputting to the notification unit 170 a video signal for displaying an image including a marker indicating the position of the dent 330 or the obstacle 340, and at the same time outputting to the notification unit 170 an audio signal indicating that cruise control will be performed to avoid the dent 330 or the obstacle 340. In a driving control mode described below, when the depth of the depression 330 or the height of the obstacle 340 does not exceed a predetermined threshold, the notification control unit 23 can, for example, output a video signal to the notification unit 170 for displaying an image including a marker indicating the position of the depression 330 or the obstacle 340, and at the same time output an audio signal to the notification unit 170 indicating that driving control will be performed to cause the vehicle 100 to drive along the driving path without avoiding the depression 330 or the obstacle 340. Note that the notification control unit 23 may fill the area of ​​the depression 330 or the obstacle 340 with a specific color as the marker indicating the position of the depression 330 or the obstacle 340.

[0057] The control unit 210 further includes a driving control unit 230, for example, as shown in Fig. 4. The driving control unit 230 is capable of controlling the driving of the vehicle 100. The driving control unit 230 includes a driving environment detection unit 31, a locator calculation unit 32, and an avoidance control unit 33, for example, as shown in Fig. 4.

[0058] The driving control unit 230 controls the vehicle 100 according to, for example, a driving mode. Examples of the driving modes include a manual driving mode and a driving control mode. The manual driving mode is a driving mode that requires the driver to maintain steering, and is a driving mode in which the vehicle 100 is driven according to driving operations such as steering, accelerator, and brake operations by the driver. The driving control mode is a driving mode that supports the driver in driving operations by the driver to increase the safety of pedestrians, vehicles, and the like around the vehicle 100.

[0059] The driving environment detection unit 31 is capable of determining lane markings that divide the road around the vehicle 100 based on the distance image data Ib received from the stereo camera 110. The driving environment detection unit 31 is further capable of determining, for example, the road curvature [1 / m] of the markings that divide the left and right sides of the road (driving lane) on which the vehicle 100 is traveling, and the width between the left and right markings (vehicle width). The driving environment detection unit 31 is also capable of detecting lanes and three-dimensional objects such as structures that exist around the vehicle 100, for example, by performing predetermined pattern matching on the distance image data Ib.

[0060] Here, the detection of a three-dimensional object by the driving environment detection unit 31 includes, for example, detecting the type of the three-dimensional object, the distance to the three-dimensional object, the speed of the three-dimensional object, the relative speed between the three-dimensional object and the vehicle (host vehicle), etc. Examples of three-dimensional objects to be detected include traffic lights, intersections, road signs, stop lines, other vehicles, pedestrians, and various buildings.

[0061] Locator calculation unit 32 estimates the position of vehicle 100 on a road map (host vehicle position) and is capable of estimating the host vehicle position. Vehicle state quantity sensor 130 and GNSS receiver 140 required for estimating the position of vehicle 100 (host vehicle position) are connected to the input side of locator calculation unit 32.

[0062] In the driving control mode, the avoidance control unit 33 is capable of performing driving control in accordance with the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation by the road surface shape estimation unit 22. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 33 is capable of performing driving control to avoid the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the avoidance control unit 33 is capable of performing driving control along the driving path.

[0063] If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 33 is capable of performing driving control to avoid the depression 330 or the obstacle 340 based on, for example, the distance image data Ib obtained from the stereo camera 110, various data obtained from the vehicle state quantity sensor 130, the positioning signal obtained from the GNSS receiver 140, road map information read from the high-precision road map DB 180, and the position of the depression 330 or the obstacle 340 obtained from the road surface shape estimation unit 22.

[0064] When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when the avoidance control unit 33 determines that it is necessary to perform engine control, the avoidance control unit 33 is capable of transmitting an engine control command to the engine control unit 240 as driving control for stopping the vehicle 100 in front of the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when the avoidance control unit 33 determines that it is necessary to perform brake control, the avoidance control unit 33 is capable of transmitting a brake control command to the brake control unit 250 as driving control. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, if the avoidance control unit 33 determines that steering control is necessary, the avoidance control unit 33 can perform, as travel control, steering control to stop the vehicle 100 in front of the depression 330 or the obstacle 340, or to travel while avoiding the depression 330 or the obstacle 340. At this time, the avoidance control unit 33 can, for example, transmit a steering control command to the steering control unit 260 as steering control.

[0065] A throttle actuator 270 is connected to the output side of the engine control unit 240. The throttle actuator 270 opens and closes a throttle valve of an electronically controlled throttle provided in a throttle body of the engine. The engine control unit 240 is able to control the operation of the throttle actuator 270 by outputting a drive signal to the throttle actuator 270. The throttle actuator 270 opens and closes the throttle valve based on the drive signal from the engine control unit 240 to adjust the intake air flow rate, thereby generating a desired engine output.

[0066] A brake actuator 280 is connected to the output side of the brake control unit 250. The brake actuator 280 is capable of adjusting the brake hydraulic pressure supplied to the brake wheel cylinders provided on each wheel. The brake control unit 250 is capable of controlling the operation of the brake actuator 280 by outputting a drive signal to the brake actuator 280. Based on the drive signal from the brake control unit 250, the brake actuator 280 generates a braking force on each wheel using the brake wheel cylinder, thereby forcibly decelerating the vehicle.

[0067] A steering actuator 290 is connected to the output side of the steering control unit 260. The steering actuator 290 is capable of adjusting the steering angle of the steering wheel. The steering control unit 260 is capable of controlling the operation of the steering actuator 290 by outputting a drive signal to the steering actuator 290. The steering actuator 290 is capable of generating a steering torque on the steering wheel based on the drive signal from the steering control unit 260, and forcibly rotating the steering wheel.

[0068] [Operation] Next, the operation of the driving control device 1000 will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining an example of a driving assistance procedure in the driving control device 1000.

[0069] The driving control device 1000 determines whether the control flag 192 is on ("1") (step S101). If the control flag 192 is on ("1") (step S101; Y), the driving control device 1000 outputs a control signal to the visible laser irradiation unit 120 to scan the road surface (traveling road surface 300) ahead of the vehicle 100 with a laser beam L. As a result, the visible laser irradiation unit 120 irradiates the road surface (traveling road surface 300) ahead of the vehicle 100 with the laser beam L in accordance with the input control signal (step S102). As a result, a linear visible laser pattern LP is drawn on the road surface (traveling road surface 300) ahead of the vehicle 100.

[0070] Next, the driving control device 1000 acquires low-resolution image data Ia including the linear visible laser pattern LP, which is captured while the linear visible laser pattern LP is being drawn on the road surface (traveling road surface 300) ahead of the vehicle 100 (step S103). The driving control device 1000 determines whether the acquired low-resolution image data Ia is normal (step S104).

[0071] In step S104, the cruise control device 1000 determines whether the view ahead of the vehicle 100 included in the image data Ia has poor visibility due to, for example, a snowstorm, twilight, or nighttime without streetlights. If the view ahead of the vehicle 100 included in the image data Ia has poor visibility due to a snowstorm or twilight (step S104; N), the cruise control device 1000 terminates the current driving assistance. On the other hand, if the view ahead of the vehicle 100 included in the image data Ia does not have poor visibility due to a snowstorm or twilight (step S104; Y), the cruise control device 1000 detects the presence or absence of unevenness on the road surface (traveling road surface 300) ahead of the vehicle 100 based on a single piece of low-resolution image data IaLP or multiple pieces of time-series image data IaLP obtained by the stereo camera 110 (step S105).

[0072] As a result, if there are no irregularities on the road surface (traveling road surface 300) ahead of the vehicle 100 (step S105; N), the cruise control device 1000 ends the current driving assistance. On the other hand, if there are irregularities on the road surface (traveling road surface 300) ahead of the vehicle 100 (step S105; Y), the cruise control device 1000 acquires high-resolution image data Ia including the linear visible laser pattern LP, which was captured while the linear visible laser pattern LP was being drawn on the road surface (traveling road surface 300) ahead of the vehicle 100 (step S106).

[0073] The cruise control device 1000 calculates a group of pixel coordinates in a single low-resolution image data IaLP or multiple time-series image data IaLP for a predetermined region that includes the detected unevenness. After calculating the group of pixel coordinates, the cruise control device 1000 extracts, for example, partial image data (partial image data Ic) of the group of pixel coordinates from the single high-resolution image data IaLP, or extracts partial image data (multiple time-series partial image data Ic) of the group of pixel coordinates from the multiple high-resolution time-series image data IaLP.

[0074] The cruise control device 1000 detects bent lines LPa and LPb in the extracted single piece of partial image data Ic or multiple pieces of partial image data Ic in a time series. The cruise control device 1000 generates specific regions β and δ based on the detected bent lines LPa and LPb and the generated virtual straight line LV. The cruise control device 1000 acquires position data D1 and D2 for the generated specific regions β and δ in the single piece of partial image data Ic or multiple pieces of partial image data Ic in a time series. The cruise control device 1000 estimates the position of the depression 330 or obstacle 340 ahead of the vehicle 100 based on the acquired position data D1 and D2 and the forward position table 194 (step S107).

[0075] The cruise control device 1000 estimates the depth or height of the depression 330 or the obstacle 340, and also estimates the size of the depression 330 or the obstacle 340 (step S107). The cruise control device 1000 estimates the depth of the depression 330 based on the number of pixels Np1 of the generated specific region β, the position data of the end A and the location P1, the resolution table 193, and the front position table 194. The cruise control device 1000 estimates the size of the depression 330 based on the number of pixels Np1, the position data D1, the resolution table 193, and the front position table 194. The cruise control device 1000 estimates the height and size of the obstacle 340 based on the number of pixels Np2 of the generated specific region δ, the position data D2, the resolution table 193, and the front position table 194.

[0076] If the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation exceeds a predetermined threshold, the driving control device 1000 determines that the depression 330 or the obstacle 340 should be avoided (step S108; Y). If the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation by the road surface shape estimation unit 22 does not exceed a predetermined threshold, the driving control device 1000 determines that there is no need to avoid the depression 330 or the obstacle 340 (step S108; N).

[0077] If the cruise control device 1000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs control to notify the driver of the presence of the depression 330 or the obstacle 340 (step S109). If the cruise control device 1000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs control to guide the vehicle along the travel path (step S110). On the other hand, if the cruise control device 1000 determines that it is necessary to avoid the depression 330 or the obstacle 340, it performs control to notify the driver of the presence and avoidance of the depression 330 or the obstacle 340 (step S111). If the cruise control device 1000 determines that it is necessary to avoid the depression 330 or the obstacle 340, it performs control to avoid the depression 330 or the obstacle 340 (step S112). In this manner, driving assistance is provided by the cruise control device 1000.

[0078] [Effects] Next, the effects of vehicle 100 according to this embodiment will be described.

[0079] In this embodiment, when the linear visible laser pattern LP included in the image data IaLP of the road surface 300 includes bent lines LPa and LPb, the numbers of pixels Np1 and Np2 of the specific regions β and δ formed by the virtual straight line LV and the bent lines LPa and LPb assuming that the bent lines LPa and LPb do not exist in the linear visible laser pattern LP are calculated, and position data D1 and D2 of the specific regions β and δ in the image data Ia are acquired. This allows at least one of the depth, height, and size of the unevenness (pothole 330 or obstacle 340) of the road surface 300 to be estimated based on the numbers of pixels Np1 and Np2 and the position data D1 and D2. As a result, it is possible to perform notification control or driving control in accordance with the estimated unevenness data of the road surface 300. In this way, in this embodiment, image processing that is less demanding and based on linear visible laser patterns and monocular images is used, rather than on complex visible laser patterns and distance images, so that unevenness in the road surface 300 can be detected in real time, and notification control or driving control can be performed according to the detection results.

[0080] Furthermore, in this embodiment, the vertical gaps in the image data IaLP between the virtual straight line LV and the bent lines LPa and LPb are set as specific regions β and δ. This allows for estimation of at least one of the depth, height, and size of the unevenness (pothole 330 or obstacle 340) on the road surface 300 based on the pixel counts Np1 and Np2 of the set specific regions β and δ and the position data D1 and D2. As a result, notification control or driving control can be performed based on the estimated unevenness data of the road surface 300. In this way, this embodiment uses light-load image processing based on a linear visible laser pattern or a monocular image, rather than heavy-load image processing based on a complex-shaped visible laser pattern or a distance image. This allows for real-time detection of unevenness on the road surface 300, and notification control or driving control can be performed based on the detection results.

[0081] Furthermore, in this embodiment, when bent lines LPa, LPb and discontinuities β, δ are detected in the linear visible laser pattern LP, gaps in the image data IaLP between the starting points of the bent lines LPa, LPb and the edges of the discontinuities β, δ in the linear visible laser pattern LP are set as specific regions β, δ. This allows at least one of the depth, height, and size of the unevenness (depression 330 or obstacle 340) on the road surface 300 to be estimated based on the pixel counts Np1, Np2 of the set specific regions β, δ and the position data D1, D2. As a result, notification control or driving control can be performed based on the estimated unevenness data of the road surface 300. In this way, this embodiment uses light-load image processing based on linear visible laser patterns and monocular images, rather than heavy-load image processing based on complex-shaped visible laser patterns or distance images. This allows unevenness on the road surface 300 to be detected in real time, and notification control or driving control can be performed based on the detection results.

[0082] Furthermore, in this embodiment, based on the position data D1 and D2, size data in real space of the pixels of the vertical gap in the image data IaLP between the virtual straight line LV and the bent lines LPa and LPb is read from the resolution table 193. Based on the read size data and the pixel counts Np1 and Np2, the depth or height of the unevenness (pothole 330 or obstacle 340) on the road surface 300 can be estimated. As a result, notification control or driving control can be performed in accordance with the estimated unevenness data of the road surface 300. As such, this embodiment does not use image processing that is heavy-loading based on a complex-shaped visible laser pattern or a distance image, but uses image processing that is light-loading based on a linear visible laser pattern or a monocular image. This makes it possible to detect unevenness on the road surface 300 in real time and perform notification control or driving control in accordance with the detection results.

[0083] Furthermore, in this embodiment, the presence or absence of unevenness (a depression 330 or an obstacle 340) is detected based on a single low-resolution image data IaLP or multiple time-series image data IaLP. When an unevenness (a depression 330 or an obstacle 340) is detected, partial image data of a predetermined region including the position of the detected unevenness (a depression 330 or an obstacle 340) is extracted from the single high-resolution image data IaLP or multiple time-series image data IaLP. Based on the extracted partial image data, pixel counts Np1 and Np2 are calculated, and position data D1 and D2 are acquired. This allows at least one of the depth, height, and size of the unevenness (a depression 330 or an obstacle 340) on the traveling road surface 300 to be estimated based on the pixel counts Np1 and Np2 and the position data D1 and D2. As a result, it is possible to perform notification control or driving control in accordance with the estimated unevenness data of the traveling road surface 300. In this way, in this embodiment, image processing that is less demanding and based on linear visible laser patterns and monocular images is used, rather than on complex visible laser patterns and distance images, so that unevenness in the road surface 300 can be detected in real time, and notification control or driving control can be performed according to the detection results.

[0084] Furthermore, in this embodiment, when the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, a notification control is performed to notify the driver to avoid the depression 330 or the obstacle 340. In this manner, in this embodiment, the notification control is performed using image processing that is light in load based on a linear visible laser pattern or a monocular image, without using image processing that is heavy in load based on a visible laser pattern with a complex shape or a distance image. Therefore, the notification control can be performed in real time.

[0085] Furthermore, in this embodiment, when the depth of the dent 330 or the height of the obstacle 340 exceeds a predetermined threshold, braking control is performed as travel control to stop the vehicle 100 in front of the dent 330 or the obstacle 340. In this way, in this embodiment, braking control is performed using image processing that is light in load based on a linear visible laser pattern or a monocular image, without using image processing that is heavy in load based on a complex-shaped visible laser pattern or a distance image. Therefore, braking control can be performed in real time.

[0086] Furthermore, in this embodiment, when the depth of the dent 330 or the height of the obstacle 340 exceeds a predetermined threshold, steering control is performed as driving control to cause the vehicle 100 to travel while avoiding the dent 330 or the obstacle 340. As described above, in this embodiment, steering control is performed using image processing that is light on the load based on a linear visible laser pattern or a monocular image, rather than using image processing that is heavy on the load based on a complex-shaped visible laser pattern or a distance image. Therefore, steering control can be performed in real time.

[0087] 2. Second Embodiment [Configuration] Next, a vehicle 400 according to a second embodiment of the present disclosure will be described. FIG. 11 illustrates an example of the external appearance of the front portion of the vehicle 400. The vehicle 400 can be driven, for example, by at least one of an engine and a motor. For example, as shown in FIG. 11 , the vehicle 400 corresponds to a vehicle 100 in which a driving control device 2000 is provided instead of the driving control device 1000. For example, as shown in FIGS. 11 and 12 , the driving control device 2000 corresponds to a vehicle in which a stereo camera 410 is provided instead of the stereo camera 110, an infrared laser irradiation unit 420 is provided instead of the visible laser irradiation unit 120, a laser irradiation control unit 24 is provided instead of the laser irradiation control unit 21, and an avoidance control unit 34 is provided instead of the avoidance control unit 33.

[0088] The stereo camera 410 is fixed, for example, to the upper center of the vehicle interior CR and includes, for example, a main camera 411 and a sub-camera 412. The main camera 411 and the sub-camera 412 are autonomous sensors that sense the real space ahead of the vehicle 400. The main camera 411 and the sub-camera 412 are arranged, for example, at symmetrical positions on either side of the central portion in the width direction of the vehicle 400, enabling stereo imaging of the area ahead of the vehicle 400 from different viewpoints. The main camera 411 and the sub-camera 412 are capable of acquiring stereo image data of the area ahead of the vehicle 400 under control of the control unit 210. The stereo camera 410 is further capable of outputting, to the control unit 210, stereo image data of the area ahead of the vehicle 400 obtained by capturing images with the main camera 411 and the sub-camera 412. The stereo image data is composed of near-infrared image data Ia obtained by the main camera 411 and near-infrared image data obtained by the sub-camera 412. The image data obtained by the sub-camera 412 may be the image data Ia. The stereo camera 410 can further generate near-infrared range distance image data Ib calculated from the amount of displacement between the positions of corresponding objects based on the obtained stereo image data, and output the data to the control unit 210.

[0089] The infrared laser irradiator 420 is provided inside the headlight HL, for example, as shown in Fig. 11 . The infrared laser irradiator 420 scans a near-infrared laser beam L on the road surface (traveling road surface 300) ahead of the vehicle 400 under control of the control unit 210, thereby generating a linear near-infrared laser pattern LPi on the traveling road surface 300, for example, as shown in Fig. 13 . Note that while Fig. 13 clearly shows the linear near-infrared laser pattern LPi for convenience, in reality, the driver of the vehicle 400 cannot visually recognize the linear near-infrared laser pattern LPi in the scenery ahead of the vehicle when looking ahead from the driver's seat or on the display screen 170A of the alarm unit 170, for example, as shown in Fig. 14 .

[0090] The infrared laser irradiation unit 420 is capable of scanning a laser beam L in the near-infrared range at a location on the travel road surface 300 where the expected passage area 310 is likely to exist, in accordance with control from the control unit 210. The infrared laser irradiation unit 420 is capable of scanning the laser beam L in the near-infrared range in a direction that obliquely intersects the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 400) in accordance with control from the control unit 210. This makes it possible for the infrared laser irradiation unit 420 to generate a linear near-infrared laser pattern LPi that extends in a direction that obliquely intersects the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 400) at a location on the travel road surface 300 where the expected passage area 310 is likely to exist.

[0091] The infrared laser irradiation unit 420 includes, for example, a laser emission unit capable of emitting near-infrared laser light, an emission control driver capable of controlling the emission of the laser emission unit, an optical system capable of scanning the laser light on the road surface 300, and a scan control driver capable of controlling the scanning of the laser light by the optical system. The emission control driver is capable of controlling the emission of the laser emission unit under control of the control unit 210. The scan control driver is capable of controlling the operation of the optical system under control of the control unit 210. The laser emission unit includes, for example, a semiconductor laser that emits near-infrared laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror reflects the near-infrared laser light emitted from the laser emission unit and is capable of scanning the reflected light of the near-infrared laser light on the road surface 300 via the fθ lens.

[0092] The laser irradiation control unit 24 is capable of controlling the irradiation (drawing) of the laser beam L from the infrared laser irradiation unit 420. The laser irradiation control unit 24 is capable of generating a control signal required for the irradiation (drawing) of the linear near-infrared laser pattern LPi based on, for example, the drawing data 191 in the storage unit 190, and outputting the control signal to the infrared laser irradiation unit 420.

[0093] The laser irradiation control unit 24 is capable of generating, for example, based on the drawing data 191, a control signal required to scan a laser beam L in the near-infrared range on the road surface 300 ahead of the vehicle 400, and outputting the control signal to the infrared laser irradiation unit 420. The laser irradiation control unit 24 is capable of generating, for example, based on the drawing data 191, a control signal required to scan a laser beam L in the near-infrared range at a location on the road surface 300 where the expected passage area 310 is likely to be present. The laser irradiation control unit 24 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the laser beam L in the near-infrared range in a direction obliquely intersecting the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100).

[0094] In the driving control mode, the avoidance control unit 34 is capable of performing driving control in accordance with the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation by the road surface shape estimation unit 22. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 34 is capable of performing driving control to avoid the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the avoidance control unit 34 is capable of performing driving control to follow the driving path.

[0095] If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 34 is capable of performing driving control to avoid the depression 330 or the obstacle 340 based on, for example, the distance image data Ib obtained from the stereo camera 410, various data obtained from the vehicle state quantity sensor 130, the positioning signal obtained from the GNSS receiver 140, road map information read from the high-precision road map DB 180, and the position of the depression 330 or the obstacle 340 obtained from the road surface shape estimation unit 22.

[0096] When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when the avoidance control unit 34 determines that it is necessary to perform engine control, the avoidance control unit 34 is able to transmit, as driving control, an engine control command to the engine control unit 240 for the vehicle 400 to stop in front of the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when the avoidance control unit 34 determines that it is necessary to perform brake control, the avoidance control unit 34 is able to transmit, as driving control, a braking control command to the brake control unit 250 for the vehicle 400 to stop in front of the depression 330 or the obstacle 340. For example, when the avoidance control unit 34 determines that steering control is necessary when the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, it is capable of transmitting a steering control command to the steering control unit 260 as driving control to stop the vehicle 400 in front of the depression 330 or the obstacle 340, or to drive while avoiding the depression 330 or the obstacle 340.

[0097] [Operation] Next, the operation of the driving control device 2000 will be described with reference to Fig. 15. Fig. 15 is a diagram for explaining an example of a driving assistance procedure in the driving control device 2000.

[0098] The driving control device 2000 determines whether the control flag 192 is on ("1") (step S201). If the control flag 192 is on ("1") (step S201; Y), the driving control device 2000 outputs a control signal to the infrared laser irradiation unit 420 to scan a laser beam L in the near-infrared range on the road surface (traveling road surface 300) ahead of the vehicle 400. As a result, the infrared laser irradiation unit 420 irradiates the laser beam L in the near-infrared range onto the road surface (traveling road surface 300) ahead of the vehicle 400 in accordance with the input control signal (step S202). As a result, a linear near-infrared laser pattern LPi is drawn on the road surface (traveling road surface 300) ahead of the vehicle 400.

[0099] Next, the driving control device 2000 acquires low-resolution image data Ia including the linear near-infrared laser pattern LPi, which is captured while the linear near-infrared laser pattern LPi is being drawn on the road surface (traveling road surface 300) ahead of the vehicle 400 (step S203). The driving control device 2000 determines whether the acquired low-resolution image data Ia is normal (step S204).

[0100] In step S204, the driving control device 2000 determines whether the view ahead of the vehicle 400 included in the image data Ia has poor visibility due to, for example, a snowstorm, twilight, or nighttime without street lights. As a result, if the view ahead of the vehicle 400 included in the image data Ia has poor visibility due to a snowstorm or twilight (step S204; N), the driving control device 2000 terminates the current driving assistance. On the other hand, if the view ahead of the vehicle 400 included in the image data Ia does not have poor visibility due to a snowstorm or twilight (step S204; Y), the driving control device 2000 detects the presence or absence of unevenness on the road surface (traveling road surface 300) ahead of the vehicle 400 based on a single piece of low-resolution image data IaLP or multiple pieces of image data IaLP in time series obtained by the stereo camera 410 (step S205).

[0101] As a result, if there are no irregularities on the road surface (traveling road surface 300) ahead of the vehicle 400 (step S205; N), the cruise control device 2000 ends the current driving assistance. On the other hand, if there are irregularities on the road surface (traveling road surface 300) ahead of the vehicle 200 (step S205; Y), the cruise control device 2000 acquires high-resolution image data Ia including the linear near-infrared laser pattern LPi that was captured while the linear visible laser pattern LP was being drawn on the road surface (traveling road surface 300) ahead of the vehicle 400 (step S206).

[0102] The cruise control device 2000 calculates a group of pixel coordinates in a single low-resolution image data IaLP or multiple time-series image data IaLP for a predetermined area that includes the detected unevenness. After calculating the group of pixel coordinates, the cruise control device 2000 extracts, for example, partial image data (partial image data Ic) of the group of pixel coordinates from the single high-resolution image data IaLP, or extracts partial image data (multiple time-series partial image data Ic) of the group of pixel coordinates from the multiple high-resolution time-series image data IaLP.

[0103] The cruise control device 2000 detects bent lines LPa, LPb in the extracted single piece of partial image data Ic or multiple pieces of partial image data Ic in a time series. The cruise control device 2000 generates specific regions β, δ based on the detected bent lines LPa, LPb and the generated virtual straight line LV. The cruise control device 2000 acquires position data D1, D2 for the generated specific regions β, δ in the single piece of partial image data Ic or multiple pieces of partial image data Ic in a time series. The cruise control device 2000 estimates the position of the depression 330 or obstacle 340 ahead of the vehicle 400 based on the acquired position data D1, D2 and the forward position table 194 (step S207).

[0104] The driving control device 2000 estimates the depth or height of the depression 330 or the obstacle 340, and also estimates the size of the depression 330 or the obstacle 340 (step S207). The driving control device 2000 estimates the depth of the depression 330 based on the number of pixels Np1 of the generated specific region β, the position data of the end A and the location P1, the resolution table 193, and the front position table 194. The driving control device 2000 estimates the size of the depression 330 based on the number of pixels Np1, the position data D1, the resolution table 193, and the front position table 194. The driving control device 2000 estimates the height and size of the obstacle 340 based on the number of pixels Np2 of the generated specific region δ, the position data D2, the resolution table 193, and the front position table 194.

[0105] If the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation exceeds a predetermined threshold, the driving control device 2000 determines that the depression 330 or the obstacle 340 should be avoided (step S208; Y). If the depth of the depression 330 or the height of the obstacle 340 obtained as a result of estimation by the road surface shape estimation unit 22 does not exceed a predetermined threshold, the driving control device 2000 determines that there is no need to avoid the depression 330 or the obstacle 340 (step S208; N).

[0106] If the cruise control device 2000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs control to notify the driver of the presence of the depression 330 or the obstacle 340 (step S209). If the cruise control device 2000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs control to keep the vehicle traveling along the travel path (step S210). On the other hand, if it determines that it is necessary to avoid the depression 330 or the obstacle 340, it estimates the water film thickness on the road surface (traveling road surface 300) ahead of the vehicle 400 based on the near-infrared image data Ia (step S211). Next, the cruise control device 2000 performs control to notify the driver of the presence or avoidance of the depression 330 or the obstacle 340 (step S212). If it determines that it is necessary to avoid the depression 330 or the obstacle 340, it determines whether the traveling road surface 300 is wet or dry based on the estimated water film thickness. If the determination result shows that the road surface 300 is wet, the cruise control device 2000 performs wet driving control (braking control or steering control) to avoid the pothole 330 or the obstacle 340 (step S213). If the determination result shows that the road surface 300 is dry, the cruise control device 2000 performs dry driving control (braking control or steering control) to avoid the pothole 330 or the obstacle 340 (step S213). In this way, driving assistance in the cruise control device 2000 is performed.

[0107] [Effects] Next, the effects of vehicle 400 according to this embodiment will be described.

[0108] In this embodiment, as in the first embodiment, when a linear near-infrared laser pattern LPi included in image data IaLP of the road surface 300 includes bent lines LPa and LPb, the number of pixels Np1 and Np2 of specific regions β and δ formed by the virtual line LV and the bent lines LPa and LPb assuming that the linear near-infrared laser pattern LPi does not include the bent lines LPa and LPb is calculated, and position data D1 and D2 of the specific regions β and δ in the image data Ia are acquired. This allows at least one of the depth, height, and size of the unevenness (pothole 330 or obstacle 340) of the road surface 300 to be estimated based on the number of pixels Np1 and Np2 and the position data D1 and D2. As a result, it is possible to perform notification control or driving control in accordance with the estimated unevenness data of the road surface 300. In this way, in this embodiment, rather than using heavy-load image processing based on complex-shaped near-infrared laser patterns or distance images, light-load image processing based on linear near-infrared laser patterns and monocular images is used, making it possible to detect unevenness in the road surface 300 being driven on in real time and perform notification control or driving control according to the detection results.

[0109] Furthermore, in this embodiment, the vertical gaps in the image data IaLP between the virtual straight line LV and the bent lines LPa and LPb are set as specific regions β and δ. This allows for estimation of at least one of the depth, height, and size of the unevenness (pothole 330 or obstacle 340) on the road surface 300 based on the pixel counts Np1 and Np2 of the set specific regions β and δ and the position data D1 and D2. As a result, notification control or driving control can be performed based on the estimated unevenness data of the road surface 300. In this way, this embodiment does not use image processing that requires a heavy load based on a complex-shaped near-infrared laser pattern or a distance image, but instead uses image processing that requires a light load based on a linear near-infrared laser pattern or a monocular image. This allows for real-time detection of unevenness on the road surface 300 and for notification control or driving control to be performed based on the detection results.

[0110] Furthermore, in this embodiment, when bent lines LPa, LPb and discontinuities β, δ are detected in the linear near-infrared laser pattern LPi, gaps in the image data IaLP between the starting points of the bent lines LPa, LPb and the edges of the discontinuities β, δ in the linear near-infrared laser pattern LPi are set as specific regions β, δ. This makes it possible to estimate at least one of the depth, height, and size of the irregularities (potholes 330 or obstacles 340) on the road surface 300 based on the pixel counts Np1, Np2 of the set specific regions β, δ and the position data D1, D2. As a result, it is possible to perform notification control or driving control in accordance with the estimated irregularity data of the road surface 300. In this way, in this embodiment, image processing that is less demanding and is based on linear near-infrared laser patterns and monocular images is used, rather than on complex-shaped near-infrared laser patterns and distance images, so that unevenness in the road surface 300 can be detected in real time, and notification control or driving control can be performed according to the detection results.

[0111] Furthermore, in this embodiment, based on the position data D1 and D2, size data in real space of the pixels of the vertical gap in the image data IaLP between the virtual straight line LV and the bent lines LPa and LPb is read from the resolution table 193. The depth or height of the unevenness (pothole 330 or obstacle 340) on the road surface 300 can be estimated based on the read size data and the pixel counts Np1 and Np2. As a result, notification control or driving control can be performed in accordance with the estimated unevenness data of the road surface 300. As such, this embodiment does not use image processing that requires a heavy load based on a complex-shaped near-infrared laser pattern or a distance image, but instead uses image processing that requires a light load based on a linear near-infrared laser pattern or a monocular image. This allows the unevenness of the road surface 300 to be detected in real time, and notification control or driving control can be performed in accordance with the detection results.

[0112] Furthermore, in this embodiment, the presence or absence of unevenness (a depression 330 or an obstacle 340) is detected based on a single low-resolution image data IaLP or multiple time-series image data IaLP. When an unevenness (a depression 330 or an obstacle 340) is detected, partial image data of a predetermined region including the position of the detected unevenness (a depression 330 or an obstacle 340) is extracted from the single high-resolution image data IaLP or multiple time-series image data IaLP. Based on the extracted partial image data, pixel counts Np1 and Np2 are calculated, and position data D1 and D2 are acquired. This allows at least one of the depth, height, and size of the unevenness (a depression 330 or an obstacle 340) on the traveling road surface 300 to be estimated based on the pixel counts Np1 and Np2 and the position data D1 and D2. As a result, it is possible to perform notification control or driving control in accordance with the estimated unevenness data of the traveling road surface 300. In this way, in this embodiment, image processing that is less demanding and is based on linear near-infrared laser patterns and monocular images is used, rather than on complex-shaped near-infrared laser patterns and distance images, so that unevenness in the road surface 300 can be detected in real time, and notification control or driving control can be performed according to the detection results.

[0113] Furthermore, in this embodiment, when the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, a notification control is performed to notify the driver to avoid the depression 330 or the obstacle 340. In this manner, in this embodiment, notification control is performed using image processing that is light on the load based on a linear near-infrared laser pattern or a monocular image, rather than using image processing that is heavy on the load based on a complex-shaped near-infrared laser pattern or a distance image. Therefore, notification control can be performed in real time.

[0114] Furthermore, in this embodiment, when the depth of dent 330 or the height of obstacle 340 exceeds a predetermined threshold, braking control is performed as travel control to stop vehicle 400 in front of dent 330 or obstacle 340. In this way, in this embodiment, braking control is performed using image processing with a low load based on a linear near-infrared laser pattern or a monocular image, without using image processing with a high load based on a complex-shaped near-infrared laser pattern or a distance image. Therefore, braking control can be performed in real time.

[0115] Furthermore, in this embodiment, when the depth of dent 330 or the height of obstacle 340 exceeds a predetermined threshold, steering control is performed as travel control to cause vehicle 400 to travel while avoiding dent 330 or obstacle 340. As described above, in this embodiment, steering control is performed using image processing that is light on the load based on a linear near-infrared laser pattern or a monocular image, rather than using image processing that is heavy on the load based on a complex-shaped near-infrared laser pattern or a distance image. Therefore, steering control can be performed in real time.

[0116] 3. Modification of First Embodiment In the first embodiment described above, the driving control device 1000 corresponds to, for example, a device in which a stereo camera 510 is provided instead of the stereo camera 110, a visible / infrared laser irradiation unit 520 is provided instead of the visible laser irradiation unit 120, a laser irradiation control unit 25 is provided instead of the laser irradiation control unit 21, and an avoidance control unit 34 is provided instead of the avoidance control unit 33, as shown in FIG.

[0117] The stereo camera 510 is fixed, for example, to the upper center of the vehicle interior CR and includes, for example, a main camera 511 and a sub-camera 512. The main camera 511 and the sub-camera 512 are autonomous sensors that sense the real space ahead of the vehicle 100. The main camera 511 and the sub-camera 512 are, for example, arranged at symmetrical positions on either side of the central portion in the width direction of the vehicle 100, and are capable of capturing stereo images of the area ahead of the vehicle 100 from different viewpoints. The main camera 511 and the sub-camera 512 are capable of acquiring stereo image data of the area ahead of the vehicle 100 under control of the control unit 210. The stereo camera 510 is further capable of outputting, for example, stereo image data of the area ahead of the vehicle 100 obtained by capturing images with the main camera 511 and the sub-camera 512 to the control unit 210. The stereo image data is composed of visible range image data Ia1 and near-infrared range image data Ia2 obtained by main camera 511, and visible range image data and near-infrared range image data obtained by sub-camera 512. Note that the visible range image data obtained by sub-camera 512 may be image data Ia1. Stereo camera 510 is further capable of generating visible range distance image data Ib1 and near-infrared range distance image data Ib2 calculated from the amount of deviation between the positions of corresponding objects based on the obtained stereo image data, and outputting these to control unit 210.

[0118] The visible / infrared laser irradiation unit 520 is provided, for example, inside the headlight HL. The visible / infrared laser irradiation unit 520 scans a laser beam L1 in the visible range and a laser beam L2 in the near-infrared range on the road surface (traveling road surface 300) ahead of the vehicle 100 under the control of the control unit 210, thereby generating a linear visible laser pattern LP and a linear near-infrared laser pattern LPi that are superimposed on each other on the traveling road surface 300.

[0119] The visible / infrared laser irradiation unit 520 is capable of scanning a laser beam L1 in the visible range and a laser beam L2 in the near-infrared range at a location on the road surface 300 where the expected passage area 310 is likely to exist, in accordance with control from the control unit 210. The visible / infrared laser irradiation unit 520 is capable of scanning the laser beam L1 in the visible range and the laser beam L2 in the near-infrared range in a direction that obliquely intersects the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100) in accordance with control from the control unit 210. This allows the visible / infrared laser irradiation unit 520 to generate, in superimposed relation to each other, a linear visible laser pattern LP and a linear near-infrared laser pattern LPi that extend in a direction that obliquely intersects the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100) at a location on the road surface 300 where the expected passage area 310 is likely to exist.

[0120] The visible / infrared laser irradiation unit 520 includes, for example, a laser emission unit capable of emitting visible laser light, an emission control driver capable of controlling the emission of the laser emission unit, an optical system capable of scanning the laser light on the road surface 300, and a scan control driver capable of controlling the scanning of the laser light by the optical system. The emission control driver is capable of controlling the emission of the laser emission unit under control of the control unit 210. The scan control driver is capable of controlling the operation of the optical system under control of the control unit 210. The laser emission unit includes, for example, a semiconductor laser that emits visible laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror reflects the visible laser light emitted from the laser emission unit and is capable of scanning the reflected visible laser light on the road surface 300 via the fθ lens.

[0121] The visible / infrared laser irradiation unit 520 further includes, for example, a laser emission unit capable of emitting near-infrared laser light, an emission control driver capable of controlling the emission of the laser emission unit, an optical system capable of scanning the laser light on the road surface 300, and a scan control driver capable of controlling the scanning of the laser light by the optical system. The emission control driver is capable of controlling the emission of the laser emission unit under control of the control unit 210. The scan control driver is capable of controlling the operation of the optical system under control of the control unit 210. The laser emission unit includes, for example, a semiconductor laser that emits near-infrared laser light (laser beam L). The optical system is configured to include, for example, a polygon mirror and an fθ lens. The polygon mirror reflects the near-infrared laser light emitted from the laser emission unit and scans the reflected light of the near-infrared laser light on the road surface 300 via the fθ lens.

[0122] The laser irradiation control unit 25 is capable of controlling the irradiation (drawing) of the visible laser beam L1 and the near-infrared laser beam L2 from the visible / infrared laser irradiation unit 520. The laser irradiation control unit 25 is capable of generating control signals necessary for the irradiation (drawing) of the linear visible laser pattern LP and the linear near-infrared laser pattern LPi based on, for example, the drawing data 191 in the storage unit 190, and outputting the control signals to the infrared laser irradiation unit 520.

[0123] The laser irradiation control unit 25 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the visible laser beam L1 and the near-infrared laser beam L2 on the road surface 300 ahead of the vehicle 100, and outputting the control signal to the visible / infrared laser irradiation unit 520. The laser irradiation control unit 25 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the visible laser beam L1 and the near-infrared laser beam L2 at a location on the road surface 300 where the expected passage area 310 is likely to be present. The laser irradiation control unit 25 is capable of generating, for example, based on the drawing data 191, a control signal required to scan the visible laser beam L1 and the near-infrared laser beam L2 in a direction obliquely intersecting the longitudinal direction of the expected passage area 310 (i.e., the traveling direction of the vehicle 100).

[0124] Next, the effects of the vehicle 100 according to this modification will be described.

[0125] In this modification, the linear visible laser pattern LP and the linear near-infrared laser pattern LPi are irradiated onto the road surface (traveling road surface 300) ahead of the vehicle 100. This improves the driver's visibility of the unevenness of the road surface (traveling road surface 300) ahead of the vehicle 100, while enabling accurate detection of the unevenness of the traveling road surface 300 in real time and enabling notification control or traveling control to be performed in accordance with the detection results.

[0126] Although the present disclosure has been described above using embodiments, the present disclosure is not limited to these embodiments and various modifications are possible. The effects described in this specification are merely examples, and the effects of the present disclosure are not limited to the effects described in this specification. Therefore, other effects may be obtained with respect to the present disclosure.

[0127] Furthermore, the present disclosure may take the following forms: (1) A driving assistance device comprising: an acquisition unit capable of acquiring image data of a road surface ahead of a vehicle; and a control unit capable of deriving a depth or height of unevenness on the road surface based on the image data, wherein the control unit controls the irradiation of a linear light pattern onto the road surface to cause the acquisition unit to acquire, as the image data, first image data of the road surface irradiated with the linear light pattern; if the linear light pattern included in the first image data contains a bend line, calculates the number of pixels in a specific area formed by the bend line and a straight line assuming that there is no bend in the linear light pattern, and acquires position data of the specific area in the first image data; estimates at least one of the depth or height and size of unevenness on the road surface based on the number of pixels and the position data; and performs notification control or driving control in accordance with the data on unevenness on the road surface obtained as a result of the estimation. (2) The driving assistance device according to (1), wherein the control unit is capable of: setting a vertical gap between the straight line and the bent line in the first image data as the specific area; and estimating a depth or height of unevenness of the road surface based on the number of pixels in the specific area and the position data. (3) The driving assistance device according to (1) or (2), wherein the control unit is capable of: when detecting the presence of the bent line and a discontinuity in the linear light pattern, setting a gap in the first image data between a starting point of the bent line and an edge of the discontinuity as the specific area; and estimating a magnitude of unevenness of the road surface based on the number of pixels in the specific area and the position data.(4) The driving assistance device according to any one of (1) to (3), further comprising: a storage unit that stores a size data table of each pixel of the image data in real space, wherein the control unit is capable of reading size data of the gap pixels in real space from the size data table based on the position data, and estimating the depth or height of unevenness on the road surface based on the read size data and the number of pixels. (5) The driving assistance device according to any one of (1) to (4), further comprising: the acquisition unit that is capable of acquiring second image data of relatively low resolution and third image data of relatively high resolution as the first image data, wherein the control unit is capable of detecting the presence or absence of unevenness based on the second image data, and when the unevenness is detected, extracting partial image data of a predetermined region from the third image data that includes the position of the detected unevenness, and calculating the number of pixels and acquiring the position data based on the extracted partial image data. (6) The driving assistance device according to any one of (1) to (5), wherein the control unit is capable of performing, as the notification control, control to notify the driver to avoid the unevenness on the road surface when the depth or height of the unevenness on the road surface exceeds a predetermined threshold. (7) The driving assistance device according to any one of (1) to (6), wherein the control unit is capable of performing, as the driving control, braking control to stop the vehicle before the unevenness on the road surface when the depth or height of the unevenness on the road surface exceeds a predetermined threshold. (8) The driving assistance device according to any one of (1) to (7), wherein the control unit is capable of performing, as the driving control, steering control to cause the vehicle to travel while avoiding the unevenness on the road surface when the depth or height of the unevenness on the road surface exceeds a predetermined threshold. (9) The driving assistance device according to any one of (1) to (8), wherein the linear light pattern is a linear visible laser pattern. (10) The driving assistance device according to any one of (1) to (8), wherein the linear light pattern is a linear near-infrared light pattern.(11) The driving assistance device described in (10), wherein the control unit determines whether the road surface is wet or dry based on the first image data, and performs braking control or steering control corresponding to the wet road surface if the road surface is wet, and performs braking control or steering control corresponding to the dry road surface if the road surface is dry. (12) A vehicle including a driving assistance device and a controlled device controlled by the driving assistance device, wherein the driving assistance device has an acquisition unit capable of acquiring image data of a road surface ahead of the vehicle, and a control unit capable of deriving the depth or height of unevenness of the road surface based on the image data, and the control unit is capable of: controlling the irradiation of a linear light pattern onto the road surface, thereby causing the acquisition unit to acquire first image data of the road surface irradiated with the linear light pattern as the image data; when the linear light pattern included in the first image data has a bend line, calculating the number of pixels of a specific area formed by the straight line and the bend line when it is assumed that there is no bend in the linear light pattern, and acquiring position data of the specific area in the first image data; estimating at least one of the depth or height and size of unevenness of the road surface based on the number of pixels and the position data; and performing notification control or driving control on the controlled device according to the data of unevenness of the road surface obtained as a result of the estimation.

[0128] The control unit 210 shown in FIGS. 2, 12, and 17 may be implemented by circuitry including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), and / or at least one field-programmable gate array (FPGA). The at least one processor may be configured to perform all or a portion of the various functions of the control unit 210 shown in FIGS. 2, 12, and 17 by reading instructions from at least one non-transitory, tangible computer-readable medium. Such medium may take various forms, including, but not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or nonvolatile memories. Volatile memories may include DRAM and SRAM. Non-volatile memories may include ROM and NVRAM. An ASIC is an integrated circuit (IC) specialized to perform all or a portion of the various functions of the control unit 210 shown in FIGS. 2, 12, and 17. An FPGA is an integrated circuit that is designed to be configurable after manufacture to perform all or part of the various functions of the control unit 210 shown in FIGS.

Claims

1. A driving assistance device comprising: an acquisition unit capable of acquiring image data of a road surface ahead of a vehicle; and a control unit capable of deriving the depth or height of unevenness of the road surface based on the image data, wherein the control unit controls the irradiation of a linear light pattern onto the road surface, thereby causing the acquisition unit to acquire, as the image data, first image data of the road surface irradiated with the linear light pattern; if the linear light pattern included in the first image data contains a bend, calculates the number of pixels in a specific area formed by the bend and the straight line when it is assumed that there is no bend in the linear light pattern, and acquires position data of the specific area in the first image data; estimates at least one of the depth or height and size of unevenness of the road surface based on the number of pixels and the position data; and performs notification control or driving control in accordance with the data on unevenness of the road surface obtained as a result of the estimation.

2. The driving assistance device according to claim 1, wherein the control unit is capable of setting a vertical gap in the first image data between the straight line and the bent line as the specific area, and estimating the depth or height of unevenness in the road surface based on the number of pixels in the specific area and the position data.

3. The driving assistance device of claim 1, wherein the control unit is capable of: when detecting the presence of the bent line and a discontinuity in the linear light pattern, setting a gap in the first image data between the starting point of the bent line and the edge of the discontinuity in the linear light pattern as the specific area; and estimating the magnitude of unevenness of the road surface based on the number of pixels in the specific area and the position data.

4. A driving assistance device as described in claim 1, further comprising a memory unit that stores a size data table of each pixel of the image data in real space, wherein the control unit is capable of reading out size data of the gap pixels in real space from the size data table based on the position data, and estimating the depth or height of unevenness in the road surface based on the read-out size data and the number of pixels.

5. The driving assistance device according to claim 1, wherein the acquisition unit is capable of acquiring second image data of relatively low resolution and third image data of relatively high resolution as the first image data, and the control unit is capable of detecting the presence or absence of unevenness based on the second image data, and when the unevenness is detected, extracting partial image data of a predetermined area from the third image data that includes the position of the detected unevenness, and calculating the number of pixels and acquiring the position data based on the extracted partial image data.

6. The driving assistance device according to claim 1, wherein the control unit is capable of controlling the notification control to notify the driver to avoid the unevenness of the road surface when the depth or height of the unevenness of the road surface exceeds a predetermined threshold.

7. The driving assistance device according to claim 1, wherein the control unit is capable of performing braking control as the driving control to stop the vehicle before the unevenness in the road surface when the depth or height of the unevenness in the road surface exceeds a predetermined threshold.

8. The driving assistance device according to claim 1, wherein the control unit is capable of performing steering control as the driving control so that the vehicle can avoid the unevenness of the road surface when the depth or height of the unevenness of the road surface exceeds a predetermined threshold.

9. The driving assistance device according to claim 1, wherein the control unit is capable of determining whether the road surface is wet or dry based on the first image data, and performing braking control or steering control corresponding to wet road conditions if the road surface is wet, and performing braking control or steering control corresponding to dry road conditions if the road surface is dry.

10. A vehicle comprising a driving assistance device and a controlled device controlled by the driving assistance device, wherein the driving assistance device has an acquisition unit capable of acquiring image data of a road surface ahead of the vehicle, and a control unit capable of deriving the depth or height of unevenness of the road surface based on the image data, and the control unit is capable of: controlling the irradiation of a linear light pattern onto the road surface, thereby causing the acquisition unit to acquire first image data of the road surface irradiated with the linear light pattern as the image data; when the linear light pattern included in the first image data contains a bend line, calculating the number of pixels of a specific area formed by the straight line and the bend line when it is assumed that there is no bend in the linear light pattern, and acquiring position data of the specific area in the first image data; estimating at least one of the depth or height and size of unevenness of the road surface based on the number of pixels and the position data; and performing notification control or driving control on the controlled device in accordance with the data of unevenness of the road surface obtained as a result of the estimation.

Images