Driving assistance device and vehicle

Abstract

The driving assist device of one embodiment of the present disclosure can perform the following three items. (1) In a case where a kink line is present in a straight light pattern included in image data of a travel road surface, the number of pixels of a specific region formed by a straight line when the kink line is assumed not to be present in the straight light pattern and the position data of the specific region in the first image data are calculated; (2) at least one of the depth or height and the size of the concave-convex of the travel road surface is estimated on the basis of the number of pixels and the position data; and (3) notification control or travel control corresponding to data of the concave-convex of the travel road surface obtained from the result of the above estimation is performed.

Description

Technical Field

[0001] This disclosure relates to a driver assistance device mounted on a vehicle, and a vehicle equipped with such a driver assistance device. Background Technology

[0002] There are instances where the road surface has depressions, obstacles, or other irregularities, creating unevenness. In such cases, when a vehicle crosses these depressions or irregularities, depending on their depth and / or height, not only does passenger comfort decrease, but there is also a possibility that the vehicle may become embedded in the depression or collide with the obstacle and veer off the road. Consequently, driving safety may be reduced.

[0003] For example, patent documents 1 and 2 disclose the use of lasers to irradiate the road surface in front of a vehicle in order to detect the shape of the road surface and obstacles on the road.

[0004] Existing technical documents Patent documents Patent Document 1: Japanese Patent No. 6962464 Patent Document 2: Japanese Patent Application Publication No. 2010-18080 Summary of the Invention

[0005] One embodiment of the driving assistance device disclosed herein includes: an acquisition unit capable of acquiring image data of the road surface in front of the vehicle; and a control unit capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing the following four functions.

[0006] (A1) By controlling the illumination of the road surface with a linear light pattern, the acquisition unit acquires first image data of the road surface illuminated with the linear light pattern as image data.

[0007] (A2) In the case that there are bent lines in the straight light pattern included in the first image data, the number of pixels in a specific region formed by the straight lines and bent lines when it is assumed that there are no bends in the straight light pattern is calculated, and the position data of the specific region in the first image data is obtained.

[0008] (A3) Based on the number of pixels and location data, estimate at least one of the depth or height and size of the unevenness of the road surface.

[0009] (A4) Perform notification control or driving control corresponding to the data on the unevenness of the driving surface obtained from the above-estimated results.

[0010] One embodiment of the present disclosure provides a vehicle equipped with a driving assistance device and a controlled device controlled by the driving assistance device. The driving assistance device includes: an acquisition unit capable of acquiring image data of the road surface in front of the vehicle; and a control unit capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing the following four functions.

[0011] (B1) By controlling the illumination of the road surface with a linear light pattern, the acquisition unit acquires first image data of the road surface illuminated with the linear light pattern as image data.

[0012] (B2) In the case that there are bent lines in the straight light pattern included in the first image data, the number of pixels in a specific region formed by the straight lines and bent lines when it is assumed that there are no bends in the straight light pattern is calculated, and the position data of the specific region in the first image data is obtained.

[0013] (B3) Based on the number of pixels and location data, estimate at least one of the depth or height and size of the unevenness of the road surface.

[0014] (B4) Perform notification control or driving control corresponding to the data on the unevenness of the driving surface obtained from the above-mentioned estimated results. Attached Figure Description

[0015] The accompanying drawings are provided to further understand this disclosure and are incorporated in and constitute a part of this specification. The drawings serve to illustrate an embodiment and, together with the description, illustrate the principles of this disclosure.

[0016] Figure 1 This is a diagram showing an example of the appearance of the front of a vehicle according to the first embodiment of this disclosure.

[0017] Figure 2 It means from Figure 1 An example of the view in front of a vehicle from the driver's seat.

[0018] Figure 3 It means from Figure 1 An example of the view of vehicles and road surface from above.

[0019] Figure 4 It means that it is carried on Figure 1 A diagram illustrating the functional block diagram of the vehicle's driving control device.

[0020] Figure 5 It means being able to... Figure 4 A diagram illustrating an example of a network environment in which the driving control device communicates.

[0021] Figure 6 It means from Figure 1 When a vehicle projects a linear visible laser pattern onto the road surface, the image obtained by photographing the front of the vehicle is displayed. Figure 1 The image shows an example of the display screen of a vehicle.

[0022] Figure 7 (A) represents from Figure 1 An example of a linear visible laser pattern included in an image obtained by photographing the front of a vehicle when the vehicle projects a linear visible laser pattern onto the road surface. Figure 7 (B) indicates from Figure 7 An example of an imaginary straight line obtained by including a linear visible laser pattern in image (A).

[0023] Figure 8 (A) represents from Figure 1 An example of a linear visible laser pattern included in an image obtained by photographing the front of a vehicle when the vehicle projects a linear visible laser pattern onto the road surface. Figure 8 (B) indicates from Figure 8 An example of an imaginary straight line obtained by including a linear visible laser pattern in image (A).

[0024] Figure 9 It is used for explanation Figure 4 A diagram illustrating an example of the sequence of driver assistance features in a driving control system.

[0025] Figure 10 It is used to explain what follows. Figure 9 A diagram illustrating an example of the sequence of driver assistance systems.

[0026] Figure 11 This is a diagram showing an example of the appearance of the front of a vehicle according to the second embodiment of this disclosure.

[0027] Figure 12 It means that it is carried on Figure 11 A diagram illustrating the functional block diagram of the vehicle's driving control device.

[0028] Figure 13 It means from Figure 11 An example of the view in front of a vehicle when the driver is looking out from the driver's seat.

[0029] Figure 14 It means from Figure 11 When a vehicle projects a linear visible laser pattern onto the road surface, the image obtained by photographing the front of the vehicle will be displayed on... Figure 11 The image shows an example of the display screen of a vehicle.

[0030] Figure 15It is used for explanation Figure 12 A diagram illustrating an example of the sequence of driver assistance features in a driving control system.

[0031] Figure 16 It is used to explain what follows. Figure 15 A diagram illustrating an example of the sequence of driver assistance systems.

[0032] Figure 17 It means Figure 4 A modified example of the functional block diagram of the driving control device. Detailed Implementation

[0033] There are situations where the road surface has depressions, obstacles, or other irregularities, resulting in unevenness. In such cases, when a vehicle crosses these depressions, depending on their depth and / or height, not only does passenger comfort decrease, but there is also a possibility that the vehicle may become embedded in the depression or collide with the obstacle and veer off the road. Consequently, driving safety may be reduced. It is desirable to provide a driving assistance device capable of detecting road surface irregularities in real time and providing notification control or driving control accordingly, as well as a vehicle equipped with such driving assistance.

[0034] Hereinafter, several exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the following description illustrates a specific example of the present disclosure and should not be construed as limiting the present disclosure. For example, elements including numerical values, shapes, materials, components, the positions of the components, and the connection methods of the components are merely examples and should not be construed as limiting the present disclosure. Furthermore, in the following exemplary embodiments, constituent elements not described in the independent claims based on the highest-level concept of the present disclosure are arbitrary and can be provided as needed. The drawings are schematic and are not intended to be illustrated at their original dimensions. Throughout this specification and the drawings, constituent elements having substantially the same function and substantially the same structure are labeled with the same reference numerals, and repeated descriptions are omitted. Additionally, constituent elements not directly related to an embodiment of the present disclosure are not illustrated in the drawings.

[0035] The description in this disclosure is presented in the following order.

[0036] 1. First implementation method ( Figures 1 to 10 ) An example of illuminating a linear visible laser pattern onto the road surface in front of a vehicle. 2. Second Implementation Method ( Figures 11 to 16 ) An example of illuminating the road surface in front of a vehicle with a linear near-infrared laser pattern. 3. Variations of the first embodiment ( Figure 17 ) Examples of illuminating the road surface in front of a vehicle with linear visible laser patterns and linear near-infrared laser patterns. <1. First Implementation Method> [constitute] Figure 1 This is a diagram showing an example of the appearance of the front of the vehicle 100 according to the first embodiment of this disclosure. The driving method of the vehicle 100 is not particularly limited; the vehicle 100 can, for example, be driven by at least one of an engine and a motor. Figure 1 As shown, vehicle 100, for example, has a pair of left and right headlights HL and a windshield FW at the front. A visual laser illumination unit 120 is disposed inside the headlights HL, and a stereo camera 110, including a main camera 111 and a secondary camera 112, is disposed in the vehicle interior CR, which is visually identifiable from the outside via the windshield FW. The position of the visual laser illumination unit 120 is not particularly limited as long as it is located at the front of vehicle 100; for example, the visual laser illumination unit 120 may be disposed inside the headlight of the pair of left and right headlights HL that is farther from the roadside strip. Figure 1 The example illustrates the situation where vehicle 100 is traveling on road surface 300.

[0037] Figure 2 This is an example of a diagram showing the view in front of a vehicle 100 when the driver looks out from the driver's seat while the vehicle 100 is traveling on road 300. Figure 3 This diagram illustrates an example of the view of vehicle 100 and road surface 300 from above. Figure 2 , Figure 3 For clarity, dashed lines are drawn in the area where the tire TR is expected to pass (expected passage area 310) around 100 km / h of the vehicle. Additionally, in... Figure 2 , Figure 3 The example illustrates the presence of roadside strips 320 on both sides of the driving surface 300.

[0038] like Figure 2 , Figure 3 As shown, there is a dent 330 in front of the vehicle. However, it is sometimes difficult for drivers to visually identify it. For example, in a blizzard, at dusk, or at night without exterior lights, it is extremely difficult for drivers to visually identify dent 330. Therefore, as Figure 2 , Figure 3 As shown, vehicle 100, for example, uses a laser irradiation unit 120 to irradiate a laser beam L onto the road surface 300 in front of the vehicle, thereby generating a linear visible laser pattern LP on the road surface 300. This linear visible laser pattern LP makes the depression 330 appear three-dimensionally raised, thus allowing the driver to easily visually identify the depression 330. It should be noted that in... Figure 2 , Figure 3The example illustrates a scenario where a linear visible laser pattern LP illuminates a portion of the road surface 300 where there is a high probability of a desired passage area 310. When the road surface 300 is smooth and even, the linear visible laser pattern LP is a straight line without the bends LPa and LPb, or the interruptions α and γ (described later). When the road surface 300 is uneven, the linear visible laser pattern LP includes not only the straight line but also the bends LPa and LPb, and the interruptions α and γ (described later).

[0039] From the viewpoint of reducing image processing load, the laser pattern generated by irradiating the road surface 300 with laser light L is straight. In addition, from the viewpoint of easily detecting bends and interruptions in the laser pattern on the image, the laser pattern extends in a direction that intersects obliquely with respect to the long side direction of the intended passage area 310 (i.e., the travel direction of the vehicle 100).

[0040] Figure 4 This is a diagram illustrating an example of the functional block diagram of the driving control device 1000. The vehicle 100 is equipped with the driving control device 1000. The driving control device 1000 corresponds to a specific example of a "driving assistance device" according to an embodiment of this disclosure. Figure 5 This is a diagram illustrating an example of a function block of a control device 2000 set up in a network environment NW, which the driving control device 1000 is connected to via wireless communication.

[0041] The control device 2000 can sequentially integrate and update road map information sent from the driving control devices 1000 of each vehicle, and send the updated road map information to each vehicle. The control device 2000 includes, for example, a road map information integration ECU 270 and a transceiver 280.

[0042] The ECU270 integrates road map boundary information collected from multiple vehicles via transceiver 280, and updates the road map information surrounding the vehicles on the road sequentially. The road map information, for example, consists of dynamic maps and includes static and quasi-static information that primarily constitutes road information, as well as quasi-dynamic and dynamic information that primarily constitutes traffic information.

[0043] Static information constituting road information includes, for example, information such as roads and / or structures on roads, lane information, pavement information, and permanent restrictions, which requires an update frequency of no more than one month. Within "roads," this includes, for example, the road's location and shape, intersections, and road attributes (e.g., national highway, provincial highway, municipal highway, private road, priority road, non-priority road, general road, expressway). Within "structures on roads," this includes, for example, traffic signs, traffic lights, curved mirrors, and overpasses.

[0044] Quasi-static information that constitutes road information includes, for example, traffic restriction information based on road construction and / or events, wide-area meteorological information, and congestion predictions, which require an update frequency of less than one hour.

[0045] Quasi-dynamic information constituting traffic information includes, for example, information such as the actual congestion and / or driving restrictions at the time of observation, falling objects and / or obstacles, temporary driving obstacles, actual accident status, and local weather information, which requires an update frequency of less than one minute.

[0046] The dynamic information constituting traffic information includes, for example, information transmitted / exchanged between moving bodies and / or information about currently displayed signals, pedestrian / bicycle information at intersections, and vehicle information traveling on the road, all of which require an update frequency of one second. Such road map information is maintained / updated on a cycle from the time each vehicle receives the next piece of information, and the updated road map information is appropriately transmitted to each vehicle via transceiver 280.

[0047] like Figure 4 As shown, the driving control device 1000 includes, for example, a stereo camera 110, a visual laser illuminator 120, a vehicle status sensor 130, a GNSS receiver 140, a transceiver 150, a control sign input unit 160, a notification unit 170, and a high-precision road map DB 180. The driving control device 1000 may also further include... Figure 4 Structures other than those shown. In Figure 4 The diagram illustrates a portion of the configuration within the driving control device 1000.

[0048] The stereo camera 110 is fixed, for example, to the upper center of the CR (Center for Rear Access) inside the vehicle interior, and is configured, for example, to include a main camera 111 and a secondary camera 112. The main camera 111 and the secondary camera 112 are autonomous sensors that sense the actual space in front of the vehicle 100. The main camera 111 and the secondary camera 112 are positioned symmetrically to the left and right of the central portion of the vehicle 100 in the width direction, and are capable of capturing stereo images of the front of the vehicle 100 from different viewpoints. The main camera 111 and the secondary camera 112 can acquire stereo image data of the front of the vehicle 100 under the control of the control unit 210 (described later). The stereo camera 110 can also output, for example, stereo image data of the front of the vehicle 100 obtained by capturing images using the main camera 111 and the secondary camera 112 to the control unit 210. The stereo image data consists of image data Ia of the visible area obtained using the main camera 111 and image data of the visible area obtained using the secondary camera 112. It should be noted that the image data obtained using the secondary camera 112 can also be image data Ia. Image data Ia is a specific example of the "first image data" of this disclosure. Stereo camera 110 may also generate distance image data Ib, calculated based on the offset of the position of the corresponding object, based on the obtained stereo image data, and output it to control unit 210.

[0049] like Figure 1 As shown, a visible laser irradiation unit 120 is provided, for example, inside a headlight HL. The visible laser irradiation unit 120, under control from the control unit 210, scans laser light L across a visible area on the road surface (driving road surface 300) in front of the vehicle 100, thereby generating a linear visible laser pattern LP on the driving road surface 300. The visible laser irradiation unit 120, under control from the control unit 210, scans laser light L at locations on the driving road surface 300 where the likelihood of a desired passage area 310 is high. The visible laser irradiation unit 120, under control from the control unit 210, scans laser light L in a direction that intersects obliquely with respect to the long side direction of the desired passage area 310 (i.e., the driving direction of the vehicle 100). Thus, the visible laser irradiation unit 120 can generate a linear visible laser pattern LP extending in a direction that intersects obliquely with respect to the long side direction of the desired passage area 310 (i.e., the driving direction of the vehicle 100) at locations on the driving road surface 300 where the likelihood of a desired passage area 310 is high. The visible laser irradiation unit 120 is capable of emitting a single-wavelength laser beam L, which is included in a wavelength band (e.g., the green band) that is different from the wavelength band of the color normally used on the driving surface 300. Examples of "colors normally used on the driving surface 300" include, for example, the color of paved or unpaved roads, or the color of dividing lines on paved roads.

[0050] The laser irradiation unit 130 includes, for example, a laser emitter capable of emitting visible laser light, a light emission control driver capable of controlling the emission of the laser emitter, an optical system capable of scanning the laser on the road surface 300, and a scan control driver capable of controlling the scanning of the laser by the optical system. The light emission control driver can control the emission of the laser emitter according to control from the control unit 210. The scan control driver can control the operation of the optical system according to control from the control unit 210. The laser emitter includes, for example, a semiconductor laser that emits visible laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror can reflect the visible laser light emitted from the laser emitter, and the reflected light of the visible laser light is scanned on the road surface 300 via the fθ lens.

[0051] The vehicle status sensor 130 is composed of various sensors, such as an acceleration sensor, a vehicle speed sensor, and a gyroscope sensor. The vehicle status sensor 130 can output detection signals obtained from the various sensors to the control unit 210. The GNSS receiver 140 can receive positioning signals transmitted from multiple positioning satellites. The GNSS receiver 140 can output the received positioning signals to the control unit 210.

[0052] The control flag input unit 160 can receive input from the driver as a control flag 192. The control flag input unit 160 may be, for example, a paddle shifter attached to the steering wheel. For example, the control flag input unit 160 can store "1" as a control flag 192 in the storage unit 190 when the driver simultaneously presses and holds both left and right paddle shifters. For example, the control flag input unit 160 can, after previously storing "1" as a control flag 192 in the storage unit 160, store "0" as a control flag 192 again when the driver simultaneously presses and holds both left and right paddle shifters. For example, the control flag input unit 160 can, after previously storing "0" as a control flag 192 in the storage unit 190, store "1" as a control flag 192 again when the driver simultaneously presses and holds both left and right paddle shifters.

[0053] When control flag 192 is "1", control flag 192 indicates, for example, that the system is in laser irradiation mode. When control flag 192 is "0", control flag 192 indicates, for example, that the system is in normal mode where laser irradiation is not performed automatically. It should be noted that the values ​​that can be used for control flag 192 are not limited to the values ​​described above.

[0054] The notification unit 170 may include, for example, a liquid crystal display panel or an organic EL display panel, and a speaker. The notification unit 170 may, for example, display an image on the display screen 170A based on an image signal input from the notification control unit 23 (described later), or output sound based on an audio signal input from the notification control unit 23 (described later).

[0055] The high-precision road map DB180 is stored on a high-capacity storage medium such as an HDD. The high-precision road map DB180 contains high-precision road map information (dynamic map). This high-precision road map information, such as the road map information integrated into ECU270, similarly possesses static and quasi-static information that mainly constitutes road information, as well as quasi-dynamic and dynamic information that mainly constitutes traffic information.

[0056] The storage unit 190 is, for example, composed of non-volatile memory. The storage unit 190 stores, for example, drawing data 191, control flags 192, a resolution table 193, and a forward position table 194. The resolution table 193 corresponds to a specific example of the "size data table" of this disclosure. The drawing data 191 includes data for generating a linear visible laser pattern LP at predetermined locations on the road surface (driving road surface 300) in front of the vehicle 100. The control flags 192 include flags (e.g., "0" or "1") input from the control flag input unit 160.

[0057] Resolution table 193 includes size data for predetermined pixel units (e.g., one pixel) in image data Ia. Resolution table 193 includes, per predetermined pixel unit (e.g., one pixel), the dimensions of the image data Ia relative to the direction of travel of vehicle 100. Figure 3 The direction corresponding to the Y direction (described later) Figure 7 The resolution in the Y direction (the direction of travel of vehicle 100) is also included in image data Ia. Resolution Table 193 also includes the resolution in the image data Ia relative to the direction of travel of vehicle 100 (the direction of travel of vehicle 100). Figure 3 The direction orthogonal to the Y direction ( Figure 3 The direction corresponding to the X direction (described later) Figure 7 The resolution in the X direction (X direction). In resolution table 193, for example, (2mm, 3mm) is stored as the resolution (X direction resolution, Y direction resolution) of a pixel corresponding to a position 20m in front of the vehicle 100. In resolution table 193, for example, (1mm, 2mm) is stored as the resolution (X direction resolution, Y direction resolution) of a pixel corresponding to a position 5m in front of the vehicle 100.

[0058] The forward position table 194 includes position data of the road surface 300 in front of the vehicle 100 for each pixel in the image data Ia. The image data Ia is configured as an m×n pixel array with m pixels in the X direction and n pixels in the Y direction. For example, in the forward position table 194, a position 20m in front of the vehicle 100 at its center in the width direction is defined as the position data of the m / 2th pixel in the X direction and the nth pixel in the Y direction of the image data Ia. Additionally, for example, a position 5m in front of the vehicle 100 at its center in the width direction is defined as the position data of the m / 2th pixel in the X direction and the first pixel in the Y direction of the image data Ia.

[0059] like Figure 4 As shown, the driving control device 1000 further includes, for example, a control unit 210, a throttle actuator 270, a brake actuator 280, and a steering actuator 290. The control unit 210 corresponds to a specific example of the "acquisition unit" or "control unit" of this disclosure. The throttle actuator 270, brake actuator 280, and steering actuator 290 correspond to specific examples of the "controlled device" of this disclosure. The control unit 210 is capable of controlling the entire vehicle 100. The control unit 210 is, for example, a so-called ECU (Electronic Control Unit), and is configured to include, for example, one or more processors and one or more memories. The control unit 210 may include, for example, a CPU (Central Processing Unit). In this case, the control unit 210 can, for example, control the entire vehicle 100 by executing a program stored in the storage unit 190.

[0060] like Figure 4 As shown, the control unit 210 includes, for example, a driver assistance unit 220. The driver assistance unit 220 assists the driver in driving the vehicle 100. Figure 4 As shown, the driving assistance unit 220 includes, for example, a laser illumination control unit 21, a road shape estimation unit 22, and a notification control unit 23.

[0061] The laser irradiation control unit 21 is capable of controlling the irradiation (drawing) of the laser beam L from the visible laser irradiation unit 120. For example, the laser irradiation control unit 21 can generate control signals required for the irradiation (drawing) of a linear visible laser pattern LP based on the drawing data 191 (described later) in the storage unit 190, and output them to the visible laser irradiation unit 120.

[0062] For example, the laser illumination control unit 21 can generate, based on the drawing data 191, the control signal required to scan the laser beam L on the road surface in front of the vehicle 100 in the driving road surface 300, and output it to the variable laser illumination unit 120. The laser illumination control unit 21 can also generate, based on the drawing data 191, the control signal required to scan the laser beam L at locations in the driving road surface 300 where the expected passage area 310 is highly likely to exist. Furthermore, the laser illumination control unit 21 can generate, based on the drawing data 191, the control signal required to scan the laser beam L in a direction that is obliquely intersecting the long side direction (i.e., the travel direction of the vehicle 100) relative to the expected passage area 310.

[0063] When the laser illumination control unit 21 outputs the control signal to the variable laser illumination unit 120, the control unit 210 outputs a control signal to the stereo camera 110, thereby enabling the stereo camera 110 to acquire image data Ia, including a linear visible laser pattern LP. When the laser illumination control unit 21 outputs the control signal to the variable laser illumination unit 120, the control unit 210 outputs a control signal to the stereo camera 110, thereby enabling the road surface shape estimation unit 22 to acquire image data Ia, including a linear visible laser pattern LP, obtained by the stereo camera 110. At this time, the control unit 210 outputs a signal to the stereo camera 110 to set a resolution, thereby enabling the stereo camera 110 to acquire either relatively low-resolution image data Ia or relatively high-resolution image data Ia. The relatively low-resolution image data Ia corresponds to a specific example of the "second image data" of this disclosure. The relatively high-resolution image data Ia corresponds to a specific example of the "third image data" of this disclosure.

[0064] The road surface shape estimation unit 22 can acquire one image data Ia or multiple temporal image data Ia obtained by the stereo camera 110, including a linear visible laser pattern LP. Hereinafter, the one image data Ia obtained by the stereo camera 110, including the linear visible laser pattern LP, will be referred to as one image data IaLP. Conversely, multiple temporal image data Ia obtained by the stereo camera 110, including at least a portion of the linear visible laser pattern LP, will be referred to as multiple temporal image data IaLP. The road surface shape estimation unit 22 can detect the unevenness of the road surface (driving road surface 300) in front of the vehicle 100 based on one image data IaLP or multiple temporal image data IaLP with relatively low resolution obtained by the stereo camera 110. For example, the road surface shape estimation unit 22 can detect whether there is unevenness on the road surface (driving road surface 300) in front of the vehicle 100 based on one image data IaLP or multiple temporal image data IaLP with relatively low resolution obtained by the stereo camera 110.

[0065] The road surface shape estimation unit 22 can detect whether there are unevennesses or bumps on the road surface (driving road surface 300) in front of the vehicle 100 in a single low-resolution image data IaLP or multiple temporal image data IaLP using a predetermined method. For example, the road surface shape estimation unit 22 can determine that there are unevennesses or bumps on the road surface (driving road surface 300) in front of the vehicle 100 in a single low-resolution image data IaLP or multiple temporal image data IaLP when there are bends or breaks in the straight-line visible laser pattern LP included in the single low-resolution image data IaLP or multiple temporal image data IaLP.

[0066] The characteristics of the "bent line" are different when the concavity is a depression 330 and when the concavity is an obstacle 340 on the driving surface 300. Figure 7 (A) represents an example of a linear visible laser pattern LP included in a low-resolution image data IaLP or multiple temporal image data IaLP with a concavity of 330°. Figure 8 (A) represents an example of a linear visible laser pattern LP included in a low-resolution image data IaLP or multiple temporal image data IaLP, where the convexity is an obstacle 340 on the driving road surface 300.

[0067] exist Figure 7 In (A), in the case of a depression 330 in the bend line LPa, the end farther from the vehicle 100 connects to the straight portion of the linear visible laser pattern LP, while the end closer to the vehicle 100 does not connect to the straight portion of the linear visible laser pattern LP and is interrupted. That is, there is an interruption α between the end of the bend line LPa (the end closer to the vehicle 100) and the linear visible laser pattern LP. The road surface shape estimation unit 22 can determine that a depression 330 exists on the road surface (driving road surface 300) in front of the vehicle 100 when the aforementioned features (bend line LPa, interruption α) are detected from the linear visible laser pattern LP in a single low-resolution image data IaLP or multiple temporal image data IaLP.

[0068] exist Figure 8In (A), in the case of an obstacle 340 on the road surface 300 with unevenness, the end of the bend line LPb that is relatively close to the vehicle 100 is connected to the straight portion of the straight visible laser pattern LP, while the end that is relatively far from the vehicle 100 is not connected to the straight portion of the straight visible laser pattern LP and is interrupted. That is, there is an interruption γ between the end of the bend line LPb (the end that is relatively far from the vehicle 100) and the straight visible laser pattern LP. The road surface shape estimation unit 22 can determine that there is an obstacle 340 on the road surface (road surface 300) in front of the vehicle 100 when the aforementioned features (bend line LPb, interruption γ) are detected from the straight visible laser pattern LP in a single low-resolution image data IaLP or multiple temporal image data IaLP.

[0069] The road surface shape estimation unit 22 can calculate a group of pixel coordinates in a low-resolution image data IaLP or multiple temporal image data IaLPs, including detected unevenness or undulation. After calculating the pixel coordinates, the road surface shape estimation unit 22 can also extract partial image data (partial image data Ic) of the pixel coordinates from a relatively high-resolution image data IaLP. Alternatively, the road surface shape estimation unit 22 can also extract partial image data (multiple temporal partial image data Ics) of the pixel coordinates from multiple relatively high-resolution temporal image data IaLPs.

[0070] The road surface shape estimation unit 22 can detect bend lines LPa and LPb in one extracted partial image data Ic or multiple temporal partial image data Ic. When the bend line LPa is detected in one extracted partial image data Ic or multiple temporal partial image data Ic, the road surface shape estimation unit 22 generates a straight line (imaginary straight line LV) that is assumed to be a straight line without bends in the visual laser pattern LP, and sets the longitudinal (Y direction) gap in the partial image data Ic between the generated imaginary straight line LV and the bend line LPa to a specific region β. Figure 7 (B)

[0071] When the road surface shape estimation unit 22 detects a bend line LPb in one extracted partial image data Ic or multiple temporal partial image data Ic, it generates a straight line (imaginary straight line LV) that is assumed to be a straight line without bends in the visual laser pattern LP, and sets the longitudinal (Y direction) gap in the partial image data Ic between the generated imaginary straight line LV and the bend line LPb to a specific region δ. Figure 8 (B)

[0072] The road surface shape estimation unit 22 can calculate the number of pixels Np1 of the generated specific region β, and obtain the position data D1 of the specific region β in one partial image data Ic or multiple temporal partial image data Ic. The number of pixels Np1 is, for example, the number of pixels corresponding to the distance (longitudinal length Lv1) in the Y-axis direction between the end (end A) of the bend line LPa that does not contact the imaginary straight line LV and the part (part P1) where the bend line LPa and the imaginary straight line LV are connected.

[0073] The road surface shape estimation unit 22 can, for example, acquire the coordinates (x1, y1) of the part P1 where the Y-coordinate is the largest in a specific region β and the coordinates (x2, y2) of the part P2 where the Y-coordinate is the smallest in a specific region β as position data D1 for the specific region β. Here, part P1 is the part where the imaginary straight line Lv connects with the bend line LPa, and part P1 is the starting point of the bend line LPa. Part P2 is the part where the straight section of the straight visible laser pattern LP is interrupted. The road surface shape estimation unit 22 can estimate the position of the depression 330 in front of the vehicle 100 based on the acquired position data D1 and the forward position table 194.

[0074] The road surface shape estimation unit 22 can calculate the number of pixels Np2 of the generated specific region δ, and obtain the position data D2 of the specific region δ in one partial image data Ic or multiple temporal partial image data Ic. The number of pixels Np2 is, for example, the number of pixels corresponding to the distance (longitudinal length Lv2) in the Y-axis direction between the end (end P3) of the bend line LPb that does not contact the imaginary straight line LV and the part (part P4) where the imaginary straight line LV and the bend line LPb are connected to each other.

[0075] The road surface shape estimation unit 22 can, for example, acquire the coordinates (x3, y3) of the part P3 in a specific region δ where the Y-coordinate is the largest and the coordinates (x4, y4) of the part P4 in a specific region δ where the Y-coordinate is the smallest, as position data D2 for that specific region δ. Here, part P3 is the portion of the bend line LPa that is not connected to the imaginary straight line Lv. Part P4 is the portion of the bend line LPa that is connected to the imaginary straight line Lv. The road surface shape estimation unit 22 can estimate the position of the obstacle 340 in front of the vehicle 100 based on the acquired position data D2 and the forward position table 194.

[0076] The road surface shape estimation unit 22 can estimate the depth of the depression 330 based on the number of pixels Np1, the position data of end A and part P1, the resolution table 193, and the forward position table 194. For example, the road surface shape estimation unit 22 can read the actual spatial size data of the pixel gap between end A and part P1 in the vertical direction in the image data IaLP from the resolution table 193 based on the position data of end A and part P1, and estimate the depth of the depression 330 based on the read size data, the number of pixels Np1, and the forward position table 194.

[0077] The road surface shape estimation unit 22 can estimate the size of the depression 330 based on the number of pixels Np1, position data D1, resolution table 193, and forward position table 194. For example, the road surface shape estimation unit 22 can read the actual spatial size data of the pixels of the depression 330 specified by the position data D1 from the resolution table 193 based on the position data D1, and estimate the size of the depression 330 based on the read size data, the number of pixels Np1, and the forward position table 194.

[0078] The road surface shape estimation unit 22 can estimate the height and size of the obstacle 340 based on the number of pixels Np2, position data D2, resolution table 193, and forward position table 194. For example, the road surface shape estimation unit 22 can read the actual spatial size data of the obstacle 340 as specified by the position data D2 from the resolution table 193, and estimate the height and size of the obstacle 340 based on the read size data, the number of pixels Np2, and the forward position table 194.

[0079] The notification control unit 23 is capable of providing notification control corresponding to the depth of the depression 330 or the height of the obstacle 340 obtained from the estimation result in the road surface shape estimation unit 22. If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the notification control unit 23 can provide notification to avoid the depression 330 or the obstacle 340. If the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the notification control unit 23 can provide notification of the presence of the depression 330 or the obstacle 340.

[0080] In both the manual driving mode and the driving control mode described later, as well as... Figure 6As shown, the notification control unit 23 can, for example, output an image signal to the notification unit 170 for displaying an image. This image is obtained by overlaying image data IaLP with markers indicating the location of the depression 330 or obstacle 340 (e.g., surrounding markers MK surrounding the depression 330 or obstacle 340). In the driving control mode described later, if the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the notification control unit 23 can, for example, output an image signal to the notification unit 170 for displaying an image including markers indicating the location of the depression 330 or obstacle 340, and simultaneously output an audio signal to the notification unit 170 for driving control aimed at avoiding the depression 330 or obstacle 340. In the driving control mode described later, if the depth of the depression 330 or the height of the obstacle 340 does not exceed a predetermined threshold, the notification control unit 23 can, for example, output an image signal to the notification unit 170 for displaying an image including a mark indicating the location of the depression 330 or obstacle 340, and simultaneously output an audio signal to the notification unit 170 for driving control purposes, namely, to ensure that the vehicle 100 does not avoid the depression 330 or obstacle 340 and travels along the driving road. It should be noted that the notification control unit 23 may also use a mark obtained by filling the area of ​​the depression 330 or obstacle 340 with a specific color as a mark indicating the location of the depression 330 or obstacle 340.

[0081] like Figure 4 As shown, the control unit 210 also includes, for example, a driving control unit 230. The driving control unit 230 is capable of controlling the driving of the vehicle 100. Figure 4 As shown, the driving control unit 230 includes, for example, a driving environment detection unit 31, a positioning calculation unit 32, and an obstacle avoidance control unit 33.

[0082] The driving control unit 230 controls the vehicle 100, for example, according to a driving mode. Examples of driving modes include a manual driving mode and a driving control mode. A manual driving mode is a driving mode that requires the driver to hold the steering wheel; this mode involves driving the vehicle 100 according to driving operations such as steering, accelerator, and braking performed by the driver. A driving control mode is a driving mode that supports the driver in order to improve the safety of pedestrians, vehicles, etc., around the vehicle 100 during driving operations.

[0083] The driving environment detection unit 31 can determine lane markings that divide the road surrounding the vehicle 100 based on distance image data Ib received from the stereo camera 110. The driving environment detection unit 31 can also determine, for example, the road curvature [1 / m] of the lane markings dividing the driving road (driving lane) on the left and right sides, and the width between the left and right markings (vehicle width). The driving environment detection unit 31 can also detect lanes, structures, and other three-dimensional objects existing around the vehicle 100 by performing prescribed pattern matching relative to the distance image data Ib.

[0084] Here, in the detection of three-dimensional objects in the driving environment detection unit 31, for example, the type of three-dimensional object, the distance from the three-dimensional object, the speed of the three-dimensional object, and the relative speed between the three-dimensional object and the vehicle (this vehicle) are detected. Three-dimensional objects that can be detected include, for example, traffic lights, intersections, road signs, stop lines, other vehicles, pedestrians, and various buildings.

[0085] The positioning calculation unit 32 is a unit that estimates the position (vehicle position) of the vehicle 100 on a road map. The vehicle status sensor 130 and GNSS receiver 140 required for estimating the position (vehicle position) of the vehicle 100 are connected to the input side of the positioning calculation unit 32.

[0086] The obstacle avoidance control unit 33 can perform driving control in driving control mode corresponding to the depth of the depression 330 or the height of the obstacle 340 obtained from the estimation result in the road surface shape estimation unit 22. If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the obstacle avoidance control unit 33 can perform driving control to avoid the depression 330 or the obstacle 340. If the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the obstacle avoidance control unit 33 can perform driving control along the driving road.

[0087] If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 33 can, for example, perform driving control to avoid the depression 330 or obstacle 340 based on distance image data Ib obtained from the stereo camera 110, various data obtained from the vehicle state quantity sensor 130, positioning signals obtained from the GNSS receiver 140, road map information read from the high-precision road map DB180, and the position of the depression 330 or obstacle 340 obtained from the road surface shape estimation unit 22.

[0088] When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when it is determined that engine control is required, the avoidance control unit 33 can send an engine control command to the engine control unit 240 as driving control to stop the vehicle 100 in front of the depression 330 or the obstacle 340. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when it is determined that brake control is required, the avoidance control unit 33 can perform braking control to stop the vehicle 100 in front of the depression 330 or the obstacle 340 as driving control. In this case, as braking control, the avoidance control unit 33 can, for example, send a braking control command to the brake control unit 250. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when it is determined that steering control is required, the avoidance control unit 33 can perform steering control as driving control to stop the vehicle 100 in front of the depression 330 or obstacle 340, or to drive while avoiding the depression 330 or obstacle 340. At this time, as steering control, the avoidance control unit 33 can, for example, send a steering control command to the steering control unit 260.

[0089] A throttle actuator 270 is connected to the output side of the engine control unit 240. The throttle actuator 270 causes the throttle valve of the electronically controlled throttle valve installed in the throttle body of the engine to open and close. The engine control unit 240 can control the operation of the throttle actuator 270 by outputting a drive signal to the throttle actuator 270. The throttle actuator 270 can adjust the intake airflow by opening and closing the throttle valve based on the drive signal from the engine control unit 240, thereby producing the desired engine output.

[0090] A brake actuator 280 is connected to the output side of the brake control unit 250. The brake actuator 280 can adjust the brake hydraulic pressure supplied to the brake wheel cylinders provided on each wheel. The brake control unit 250 can control the operation of the brake actuator 280 by outputting a drive signal to the brake actuator 280. Based on the drive signal from the brake control unit 250, the brake actuator 280 can use the brake wheel cylinders to generate braking force on each wheel and forcibly decelerate each wheel.

[0091] A steering actuator 290 is connected to the output side of the steering control unit 260. The steering actuator 290 can adjust the steering angle of the steering wheel. The steering control unit 260 can control the operation of the steering actuator 290 by outputting a drive signal to the steering actuator 290. The steering actuator 290 can generate steering torque on the steering wheel based on the drive signal from the steering control unit 260, forcibly turning the steering wheel.

[0092] [action] Next, refer to Figure 9 The operation of the driving control device 1000 will be explained. Figure 9 This is a diagram illustrating an example of the driving assistance sequence in the driving control device 1000.

[0093] The driving control device 1000 determines whether the control flag 192 is on (“1”) (step S101). If the control flag 192 is on (“1”) (step S101: Yes), the driving control device 1000 outputs a control signal to the visible laser irradiation unit 120 to scan the laser beam L on the road surface (driving road surface 300) in front of the vehicle 100. Consequently, the visible laser irradiation unit 120 irradiates the laser beam L onto the road surface (driving road surface 300) in front of the vehicle 100 according to the input control signal (step S102). As a result, a straight-line visible laser pattern LP is drawn on the road surface (driving road surface 300) in front of the vehicle 100.

[0094] Next, the driving control device 1000 acquires low-resolution image data Ia, including the linear visible laser pattern LP, captured during the process of drawing a linear visible laser pattern LP on the road surface (driving road surface 300) in front of the vehicle 100 (step S103). The driving control device 1000 determines whether the acquired low-resolution image data Ia is normal (step S104).

[0095] In step S104, the driving control device 1000 determines, for example, whether the view in front of the vehicle 100 included in the image data Ia is obstructed due to blizzards, dusk, or nighttime without external lights. As a result, if the view in front of the vehicle 100 included in the image data Ia is obstructed due to blizzards or dusk (step S104: No), the driving control device 1000 terminates the current driving assistance. On the other hand, if the view in front of the vehicle 100 included in the image data Ia is not obstructed due to blizzards or dusk (step S104: Yes), the driving control device 1000 detects whether the road surface (driving road surface 300) in front of the vehicle 100 is uneven based on a single low-resolution image data IaLP or multiple temporal image data IaLP obtained from the stereo camera 110 (step S105).

[0096] As a result, if there are no bumps or depressions on the road surface (driving surface 300) in front of the vehicle 100 (step S105: No), the driving control device 1000 terminates the current driving assistance. On the other hand, if there are bumps or depressions on the road surface (driving surface 300) in front of the vehicle 100 (step S105: Yes), the driving control device 1000 acquires high-resolution image data Ia, including the straight-line visible laser pattern LP, captured during the process of drawing a straight-line visible laser pattern LP on the road surface (driving surface 300) in front of the vehicle 100 (step S106).

[0097] The driving control device 1000 calculates a group of pixel coordinates in a low-resolution image data IaLP or multiple time-series image data IaLPs of a predetermined region, including detected bumps and depressions. After calculating the pixel coordinate group, the driving control device 1000 extracts a portion of the image data (partial image data Ic) of the pixel coordinate group from a high-resolution image data IaLP, or extracts a portion of the image data (multiple time-series partial image data Ics) of the pixel coordinate group from multiple high-resolution time-series image data IaLPs.

[0098] The driving control device 1000 detects bend lines LPa and LPb in one or more extracted partial image data Ic. Based on the detected bend lines LPa and LPb and the generated imaginary straight line LV, the driving control device 1000 generates specific regions β and δ. The driving control device 1000 acquires the position data D1 and D2 of the generated specific regions β and δ in one or more partial image data Ic. Based on the acquired position data D1 and D2 and the forward position table 194, the driving control device 1000 estimates the position of the depression 330 or obstacle 340 in front of the vehicle 100 (step S107).

[0099] The driving control device 1000 estimates the depth or height of the depression 330 or obstacle 340, and also estimates the size of the depression 330 or obstacle 340 (step S107). The driving control device 1000 estimates the depth of the depression 330 based on the number of pixels Np1 of the generated specific region β, the position data of the end A and the part P1, the resolution table 193, and the forward position table 194. The driving control device 1000 estimates the size of the depression 330 based on the number of pixels Np1, the position data D1, the resolution table 193, and the forward position table 194. The driving control device 1000 estimates the height and size of the obstacle 340 based on the number of pixels Np2 of the generated specific region δ, the position data D2, the resolution table 193, and the forward position table 194.

[0100] If the depth of the depression 330 or the height of the obstacle 340, as determined by the estimation, exceeds a predetermined threshold, the driving control device 1000 determines that the depression 330 or obstacle 340 should be avoided (step S108: Yes). If the depth of the depression 330 or the height of the obstacle 340, as determined by the estimation in the road surface shape estimation unit 22, does not exceed the predetermined threshold, the driving control device 1000 determines that it is not necessary to avoid the depression 330 or obstacle 340 (step S108: No).

[0101] When the driving control device 1000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs control to notify the driver of the presence of the depression 330 or the obstacle 340 (step S109). When the driving control device 1000 determines that it is not necessary to avoid the depression 330 or the obstacle 340, it performs driving control along the driving path (step S110). On the other hand, when the driving control device 1000 determines that it is necessary to avoid the depression 330 or the obstacle 340, it performs control to notify the driver of the presence of the depression 330 or the obstacle 340 / avoid it (step S111). When the driving control device 1000 determines that it is necessary to avoid the depression 330 or the obstacle 340, it performs driving control to avoid the depression 330 or the obstacle 340 (step S112). Thus, driving assistance is provided in the driving control device 1000.

[0102] [Effect] Next, the effects of the vehicle 100 according to this embodiment will be explained.

[0103] In this embodiment, when there are bends LPa and LPb in the linear visible laser pattern LP included in the image data IaLP of the driving surface 300, the number of pixels Np1 and Np2 of the specific regions β and δ formed by the imaginary straight line LV (assuming no bends LPa and LPb in the linear visible laser pattern LP) and the bends LPa and LPb are calculated, and the position data D1 and D2 of the specific regions β and δ in the image data Ia are obtained. Therefore, based on the number of pixels Np1 and Np2 and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on visual laser patterns and / or distance images with complex shapes, but uses light image processing based on visual laser patterns and / or monocular images with straight lines, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0104] Furthermore, in this embodiment, the longitudinal gap between the imaginary straight line LV and the bends LPa and LPb in the image data IaLP is set as specific regions β and δ. Therefore, based on the number of pixels Np1 and Np2 in the set specific regions β and δ and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on complex-shaped visual laser patterns and / or distance images, but uses light image processing based on linear visual laser patterns and / or monocular images, the unevenness of the driving surface 300 can be detected in real time, and notification control or driving control corresponding to the detection results can be performed.

[0105] Furthermore, in this embodiment, when bends LPa and LPb and interruptions β and δ are detected in the linear visible laser pattern LP, the gap between the starting point of the bends LPa and LPb and the edge of the interruptions β and δ in the image data IaLP is set as a specific region β and δ. Therefore, based on the number of pixels Np1 and Np2 in the set specific regions β and δ and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on visual laser patterns and / or distance images with complex shapes, but uses light image processing based on visual laser patterns and / or monocular images with straight lines, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0106] Furthermore, in this embodiment, based on position data D1 and D2, the pixel dimensions of the vertical gap between the imaginary straight line LV and the bends LPa and LPb in the image data IaLP in actual space can be read from the resolution table 193. Based on the read-out size data and the number of pixels Np1 and Np2, the depth or height of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on complex-shaped visual laser patterns and / or distance images, but uses light image processing based on linear visual laser patterns and / or monocular images, the unevenness of the driving surface 300 can be detected in real time, and notification control or driving control corresponding to the detection results can be performed.

[0107] Furthermore, in this embodiment, the presence or absence of bumps (depressions 330 or obstacles 340) is detected based on a single low-resolution image data IaLP or multiple temporal image data IaLPs. When a bump (depression 330 or obstacle 340) is detected, a portion of the image data from a specified region, including the location of the detected bump (depression 330 or obstacle 340), is extracted from a single high-resolution image data IaLP or multiple temporal image data IaLPs. Based on the extracted portion of the image data, pixel counts Np1 and Np2 are calculated, and position data D1 and D2 are obtained. Therefore, based on pixel counts Np1 and Np2 and position data D1 and D2, at least one of the depth or height and size of the bump (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated bump data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on visual laser patterns and / or distance images with complex shapes, but uses light image processing based on visual laser patterns and / or monocular images with straight lines, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0108] Furthermore, in this embodiment, if the depth of the recess 330 or the height of the obstacle 340 exceeds a predetermined threshold, a notification to avoid the recess 330 or obstacle 340 is initiated as a notification control. Thus, since this embodiment does not use heavy image processing based on complex-shaped visual laser patterns or distance images, but instead uses light image processing based on linear visual laser patterns or monocular images for notification control, real-time notification control is possible.

[0109] Furthermore, in this embodiment, if the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, braking control is performed as driving control to bring the vehicle 100 to a stop in front of the depression 330 or the obstacle 340. Thus, since this embodiment does not use heavy image processing based on complex-shaped visual laser patterns and / or distance images, but instead uses light image processing based on linear visual laser patterns and / or monocular images for braking control, real-time braking control is possible.

[0110] Furthermore, in this embodiment, if the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, steering control is performed as driving control to allow the vehicle 100 to avoid the depression 330 or the obstacle 340. Thus, since this embodiment does not use heavy image processing based on complex-shaped visual laser patterns and / or distance images, but instead uses light image processing based on linear visual laser patterns and / or monocular images for steering control, real-time steering control is possible.

[0111] <2. Second Implementation Method> [structure] Next, the vehicle 400 of the second embodiment of this disclosure will be described. Figure 11 This is an example diagram showing the appearance of the front of vehicle 400. Vehicle 400 is capable of being driven, for example, by at least one of an engine and a motor. Figure 11 As shown, vehicle 400 is, for example, equivalent to a vehicle 100 in which a driving control device 2000 is installed instead of a driving control device 1000. Figure 11 , Figure 12 As shown, the driving control device 2000 is equivalent to, for example, replacing the stereo camera 110 with a stereo camera 410, replacing the visible laser irradiation unit 120 with an infrared laser irradiation unit 420, replacing the laser irradiation control unit 21 with a laser irradiation control unit 24, and replacing the avoidance control unit 33 with an avoidance control unit 34.

[0112] The stereo camera 410 is fixed, for example, to the upper center of the CR (Center for Rectification) inside the vehicle interior, and is configured, for example, to include a main camera 411 and a secondary camera 412. The main camera 411 and the secondary camera 412 are autonomous sensors that sense the actual space in front of the vehicle 400. The main camera 411 and the secondary camera 412 are positioned symmetrically to the left and right of the central portion of the vehicle 400 in the width direction, and are capable of capturing stereo images of the front of the vehicle 400 from different viewpoints. The main camera 411 and the secondary camera 412 can acquire stereo image data of the front of the vehicle 400 under the control of the control unit 210. The stereo camera 410 can also output, for example, stereo image data of the front of the vehicle 400 obtained by capturing images using the main camera 411 and the secondary camera 412 to the control unit 210. The stereo image data consists of image data Ia in the near-infrared region obtained using the main camera 411 and image data in the near-infrared region obtained using the secondary camera 412. It should be noted that the image data obtained using the secondary camera 412 can also be image data Ia. The stereo camera 410 can, for example, generate distance image data Ib in the near-infrared region based on the obtained stereo image data, which is calculated according to the deviation of the position of the corresponding object, and output it to the control unit 210.

[0113] like Figure 11 As shown, the infrared laser irradiation unit 420 is, for example, installed inside the headlight HL. The infrared laser irradiation unit 420 is capable of scanning the near-infrared region of laser light L on the road surface (driving surface 300) in front of the vehicle 400 under the control of the control unit 210, thereby... Figure 13 As shown, for example, a linear near-infrared laser pattern LPi is generated on the driving surface 300. It should be noted that... Figure 13 For convenience, the linear near-infrared laser pattern LPi is explicitly described in the text. In reality, for example, as shown in the example... Figure 14 As shown, the driver of vehicle 400 cannot visually recognize the linear near-infrared laser pattern LPi in the view in front of the vehicle when looking from the driver's seat, or in the display screen 170A of the notification unit 170.

[0114] The infrared laser irradiation unit 420, under the control of the control unit 210, scans near-infrared laser light L at locations on the road surface 300 where there is a high probability of the expected passage area 310. The infrared laser irradiation unit 420, under the control of the control unit 210, scans near-infrared laser light L in a direction that intersects obliquely with respect to the long side direction of the expected passage area 310 (i.e., the travel direction of the vehicle 400). Thus, the infrared laser irradiation unit 420 generates a linear near-infrared laser pattern LPi extending in a direction that intersects obliquely with respect to the long side direction of the expected passage area 310 (i.e., the travel direction of the vehicle 400) at locations on the road surface 300 where there is a high probability of the expected passage area 310.

[0115] The infrared laser irradiation unit 420 includes, for example, a laser emitter capable of emitting near-infrared laser light, a light emission control driver capable of controlling the emission of the laser emitter, an optical system capable of scanning the laser on the road surface 300, and a scan control driver capable of controlling the scanning of the laser by the optical system. The light emission control driver can control the emission of the laser emitter according to the control unit 210. The scan control driver can control the operation of the optical system according to the control unit 210. The laser emitter includes, for example, a semiconductor laser that emits near-infrared laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror can reflect the near-infrared laser light emitted from the laser emitter, and the reflected near-infrared laser light is scanned on the road surface 300 by the fθ lens.

[0116] The laser irradiation control unit 24 is capable of controlling the irradiation (drawing) of the laser beam L from the infrared laser irradiation unit 420. For example, the laser irradiation control unit 24 can generate control signals required for the irradiation (drawing) of a linear near-infrared laser pattern LPi based on the drawing data 191 in the storage unit 190, and output them to the infrared laser irradiation unit 420.

[0117] The laser illumination control unit 24, for example, can generate, based on the drawing data 191, the control signal required to scan the near-infrared region of the laser beam L on the road surface 300 in front of the vehicle 400, and output it to the infrared laser illumination unit 420. The laser illumination control unit 24, for example, can generate, based on the drawing data 191, the control signal required to scan the near-infrared region of the laser beam L on the road surface 300 at locations where the intended passage area 310 is highly likely. The laser illumination control unit 24, for example, can generate, based on the drawing data 191, the control signal required to scan the near-infrared region of the laser beam L in a direction that is obliquely intersecting the long side direction (i.e., the travel direction of the vehicle 100) of the intended passage area 310.

[0118] The obstacle avoidance control unit 34 can perform driving control in driving control mode corresponding to the depth of the depression 330 or the height of the obstacle 340 obtained from the estimation result in the road surface shape estimation unit 22. If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the obstacle avoidance control unit 34 can perform driving control to avoid the depression 330 or the obstacle 340. If the depth of the depression 330 or the height of the obstacle 340 does not exceed the predetermined threshold, the obstacle avoidance control unit 34 can perform driving control along the driving road.

[0119] If the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, the avoidance control unit 34 can, for example, perform driving control to avoid the depression 330 or obstacle 340 based on distance image data Ib obtained from the stereo camera 410, various data obtained from the vehicle state quantity sensor 130, positioning signals obtained from the GNSS receiver 140, road map information read from the high-precision road map DB180, and the position of the depression 330 or obstacle 340 obtained from the road surface shape estimation unit 22.

[0120] When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when it is determined that engine control is required, the avoidance control unit 34 can send an engine control command to the engine control unit 240 to stop the vehicle 400 in front of the depression 330 or the obstacle 340 as driving control. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example, when it is determined that brake control is required, the avoidance control unit 34 can send a braking control command to the brake control unit 250 to stop the vehicle 400 in front of the depression 330 or the obstacle 340 as driving control. When the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, for example when it is determined that steering control is required, the avoidance control unit 34 can send a steering control command to the steering control unit 260 to stop the vehicle 400 in front of the depression 330 or the obstacle 340, or to drive in a way that avoids the depression 330 or the obstacle 340, as driving control.

[0121] [action] Next, refer to Figure 15 The operation of the driving control device 2000 is explained. Figure 15 This is a diagram illustrating an example of the driving assistance sequence in the driving control device 2000.

[0122] The driving control device 2000 determines whether the control flag 192 is on (“1”) (step S201). If the control flag 192 is on (“1”) (step S201: Yes), the driving control device 2000 outputs a control signal to the infrared laser irradiation unit 420 to scan the near-infrared region laser light L on the road surface (driving road surface 300) in front of the vehicle 400. As a result, the infrared laser irradiation unit 420 irradiates the near-infrared region laser light L onto the road surface (driving road surface 300) in front of the vehicle 400 according to the input control signal (step S202). Consequently, a linear near-infrared laser pattern LPi is drawn on the road surface (driving road surface 300) in front of the vehicle 400.

[0123] Next, the driving control device 2000 acquires low-resolution image data Ia, including the linear near-infrared laser pattern LPi, captured during the process of drawing a linear near-infrared laser pattern LPi on the road surface (driving road surface 300) in front of the vehicle 400 (step S203). The driving control device 2000 determines whether the acquired low-resolution image data Ia is normal (step S204).

[0124] In step S204, the driving control device 2000 determines, for example, whether the view in front of the vehicle 400 included in the image data Ia is obstructed due to blizzards, dusk, or nighttime without outdoor lights. As a result, if the view in front of the vehicle 400 included in the image data Ia is obstructed due to blizzards or dusk (step S204: No), the driving control device 2000 terminates the current driving assistance. On the other hand, if the view in front of the vehicle 400 included in the image data Ia is not obstructed due to blizzards or dusk (step S204: Yes), the driving control device 2000 detects whether the road surface (driving road surface 300) in front of the vehicle 400 is uneven based on a single low-resolution image data IaLP or multiple temporal image data IaLP obtained from the stereo camera 410 (step S205).

[0125] As a result, if there are no bumps or depressions on the road surface (driving surface 300) in front of the vehicle 400 (step S205: No), the driving control device 2000 terminates the current driving assistance. On the other hand, if there are bumps or depressions on the road surface (driving surface 300) in front of the vehicle 200 (step S205: Yes), the driving control device 2000 acquires high-resolution image data Ia, including the linear near-infrared laser pattern LPi, captured during the process of drawing a linear visible laser pattern LP on the road surface (driving surface 300) in front of the vehicle 400 (step S206).

[0126] The driving control device 2000 calculates a group of pixel coordinates in a low-resolution image data IaLP or multiple time-series image data IaLPs of a predetermined region, including detected convexities. After calculating the pixel coordinate group, the driving control device 2000 extracts a portion of the image data (partial image data Ic) of the pixel coordinate group from a high-resolution image data IaLP, or extracts a portion of the image data (multiple time-series partial image data Ics) of the pixel coordinate group from multiple high-resolution time-series image data IaLPs.

[0127] The driving control device 2000 detects bend lines LPa and LPb in one or more extracted partial image data Ic. Based on the detected bend lines LPa and LPb and the generated imaginary straight line LV, the driving control device 2000 generates specific regions β and δ. The driving control device 2000 acquires the position data D1 and D2 of the generated specific regions β and δ in one or more partial image data Ic. Based on the acquired position data D1 and D2 and the forward position table 194, the driving control device 2000 estimates the position of the depression 330 or obstacle 340 in front of the vehicle 400 (step S207).

[0128] The driving control device 2000 estimates the depth or height of the depression 330 or obstacle 340, and also estimates the size of the depression 330 or obstacle 340 (step S207). The driving control device 2000 estimates the depth of the depression 330 based on the number of pixels Np1 of the generated specific region β, the position data of the end A and the part P1, the resolution table 193, and the forward position table 194. The driving control device 2000 estimates the size of the depression 330 based on the number of pixels Np1, the position data D1, the resolution table 193, and the forward position table 194. The driving control device 2000 estimates the height and size of the obstacle 340 based on the number of pixels Np2 of the generated specific region δ, the position data D2, the resolution table 193, and the forward position table 194.

[0129] If the depth of the depression 330 or the height of the obstacle 340, as determined by the estimation, exceeds a predetermined threshold, the driving control device 2000 determines that the depression 330 or obstacle 340 should be avoided (step S208: Yes). If the depth of the depression 330 or the height of the obstacle 340, as determined by the estimation in the road surface shape estimation unit 22, does not exceed the predetermined threshold, the driving control device 2000 determines that it is not necessary to avoid the depression 330 or obstacle 340 (step S208: No).

[0130] When the driving control device 2000 determines that it is not necessary to avoid the depression 330 or obstacle 340, it performs control to notify the presence of the depression 330 or obstacle 340 (step S209). When the driving control device 2000 determines that it is not necessary to avoid the depression 330 or obstacle 340, it performs driving control along the driving road (step S210). On the other hand, when the driving control device 2000 determines that it is necessary to avoid the depression 330 or obstacle 340, it estimates the water film thickness of the road surface (driving road surface 300) in front of the vehicle 400 based on the near-infrared image data Ia (step S211). Next, the driving control device 2000 performs control to notify / avoid the presence of the depression 330 or obstacle 340 (step S212). When the driving control device 2000 determines that it is necessary to avoid the depression 330 or obstacle 340, it determines whether the driving road surface 300 is wet or dry based on the estimated water film thickness. If the driving surface 300 is determined to be wet, the driving control device 2000 performs driving control (braking control or steering control) corresponding to wetness to avoid potholes 330 or obstacles 340 (step S213). If the driving surface 300 is determined to be dry, the driving control device 2000 performs driving control (braking control or steering control) corresponding to dryness to avoid potholes 330 or obstacles 340 (step S213). In this way, driving assistance is provided by the driving control device 2000.

[0131] [Effect] Next, the effects of the vehicle 400 in this embodiment will be explained.

[0132] Similar to the first embodiment described above, in this embodiment, when there are bent lines LPa and LPb in the linear near-infrared laser pattern LPi included in the image data IaLP of the driving surface 300, the number of pixels Np1 and Np2 of the specific regions β and δ formed by the imaginary straight line LV (assuming no bent lines LPa and LPb in the linear near-infrared laser pattern LPi) and the bent lines LPa and LPb are calculated, and the position data D1 and D2 of the specific regions β and δ in the image data Ia are obtained. Therefore, based on the number of pixels Np1 and Np2 and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of near-infrared regions with complex shapes, but uses light image processing based on linear near-infrared laser patterns and / or monocular images, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0133] Furthermore, in this embodiment, the longitudinal gap between the imaginary straight line LV and the bends LPa and LPb in the image data IaLP is set as specific regions β and δ. Therefore, based on the number of pixels Np1 and Np2 in the set specific regions β and δ and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of complex-shaped near-infrared regions, but uses light image processing based on linear near-infrared laser patterns and / or monocular images, the unevenness of the driving surface 300 can be detected in real time, and notification control or driving control corresponding to the detection results can be performed.

[0134] Furthermore, in this embodiment, when bends LPa and LPb and interruptions β and δ are detected in the linear near-infrared laser pattern LPi, the gap between the starting point of the bends LPa and LPb and the edge of the interruptions β and δ in the image data IaLP is set as a specific region β and δ. Therefore, based on the number of pixels Np1 and Np2 in the set specific regions β and δ and the position data D1 and D2, at least one of the depth or height and size of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of near-infrared regions with complex shapes, but uses light image processing based on laser patterns and / or monocular images of near-infrared regions with straight lines, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0135] Furthermore, in this embodiment, based on position data D1 and D2, the pixel dimensions of the vertical gap between the imaginary straight line LV and the bends LPa and LPb in the image data IaLP in actual space can be read from the resolution table 193. Based on the read-out size data and the number of pixels Np1 and Np2, the depth or height of the unevenness (depression 330 or obstacle 340) of the driving surface 300 can be estimated. As a result, notification control or driving control corresponding to the estimated unevenness data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images in near-infrared regions with complex shapes, but uses light image processing based on laser patterns and / or monocular images in linear near-infrared regions, the unevenness of the driving surface 300 can be detected in real time, and notification control or driving control corresponding to the detection results can be performed.

[0136] Furthermore, in this embodiment, the presence or absence of bumps (depressions 330 or obstacles 340) is detected based on a single low-resolution image data IaLP or multiple temporal image data IaLPs. When a bump (depression 330 or obstacle 340) is detected, partial image data of a predetermined region, including the location of the detected bump (depression 330 or obstacle 340), is extracted from a single high-resolution image data IaLP or multiple temporal image data IaLPs. Pixel counts Np1 and Np2 are calculated based on the extracted partial image data, and position data D1 and D2 are obtained. Therefore, at least one of the depth or height and size of the bump (depression 330 or obstacle 340) of the driving surface 300 can be estimated based on the pixel counts Np1 and Np2 and the position data D1 and D2. As a result, notification control or driving control corresponding to the estimated bump data of the driving surface 300 can be performed. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of near-infrared regions with complex shapes, but uses light image processing based on laser patterns and / or monocular images of near-infrared regions with straight lines, it is possible to detect the unevenness of the driving surface 300 in real time and perform notification control or driving control corresponding to the detection results.

[0137] Furthermore, in this embodiment, if the depth of the recess 330 or the height of the obstacle 340 exceeds a predetermined threshold, a notification to avoid the recess 330 or obstacle 340 is initiated as a notification control. Thus, since this embodiment does not use laser patterns in a complex-shaped near-infrared region and / or heavy image processing based on distance images, but instead uses a linear laser pattern in a near-infrared region and / or light image processing based on monocular images for notification control, real-time notification control is possible.

[0138] Furthermore, in this embodiment, if the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, braking control is performed as driving control to bring the vehicle 400 to a stop near the depression 330 or the obstacle 340. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of complex-shaped near-infrared regions, but instead uses light image processing based on laser patterns and / or monocular images of linear near-infrared regions for braking control, real-time braking control is possible.

[0139] Furthermore, in this embodiment, when the depth of the depression 330 or the height of the obstacle 340 exceeds a predetermined threshold, steering control is performed as driving control to allow the vehicle 400 to avoid the depression 330 or the obstacle 340. Thus, since this embodiment does not use heavy image processing based on laser patterns and / or distance images of complex-shaped near-infrared regions, but instead uses light image processing based on laser patterns and / or monocular images of linear near-infrared regions for steering control, real-time steering control is possible.

[0140] <3. Variations of the First Embodiment> In the first embodiment described above, such as Figure 17 As shown, the driving control device 1000 is equivalent to, for example, replacing the stereo camera 110 with a stereo camera 510, replacing the visible laser irradiation unit 120 with a visible / infrared laser irradiation unit 520, replacing the laser irradiation control unit 21 with a laser irradiation control unit 25, and replacing the avoidance control unit 33 with an avoidance control unit 34.

[0141] The stereo camera 510 is fixed, for example, to the upper center of the CR inside the vehicle interior, and is configured, for example, to include a main camera 511 and a secondary camera 512. The main camera 511 and the secondary camera 512 are autonomous sensors that sense the actual space in front of the vehicle 100. The main camera 511 and the secondary camera 512 are, for example, positioned symmetrically to the left and right of the central portion of the vehicle 100 in the width direction, and are capable of capturing stereo images of the front of the vehicle 100 from different viewpoints. The main camera 511 and the secondary camera 512 can acquire stereo image data of the front of the vehicle 100 under control from the control unit 210. The stereo camera 510 can also output, for example, stereo image data of the front of the vehicle 100 obtained by capturing images using the main camera 511 and the secondary camera 512 to the control unit 210. The stereo image data consists of image data Ia1 in the visible area and image data Ia2 in the near-infrared area obtained using the main camera 511, and image data in the visible area and image data in the near-infrared area obtained using the secondary camera 512. It should be noted that the image data in the visible area obtained by the secondary camera 512 can also be image data Ia1. For example, the stereo camera 510 can also generate distance image data Ib1 of the visible area and distance image data Ib2 of the near-infrared area based on the obtained stereo image data, which are calculated according to the offset of the position of the corresponding object, and output them to the control unit 210.

[0142] The visible / infrared laser irradiation unit 520 is installed, for example, inside the headlight HL. Under the control of the control unit 210, the visible / infrared laser irradiation unit 520 scans the visible laser light L1 and the near-infrared laser light L2 on the road surface (driving road surface 300) in front of the vehicle 100, thereby generating a linear visible laser pattern LP and a linear near-infrared laser pattern LPi that overlap each other on the driving road surface 300.

[0143] The visible / infrared laser irradiation unit 520, under the control of the control unit 210, scans the laser beam L1 of the visible area and the laser beam L2 of the near-infrared area at locations on the road surface 300 where the expected passage area 310 is highly likely to exist. The visible / infrared laser irradiation unit 520, under the control of the control unit 210, scans the laser beam L1 of the visible area and the laser beam L2 of the near-infrared area in a direction that intersects obliquely with respect to the long side direction of the expected passage area 310 (i.e., the travel direction of the vehicle 100). Thus, the visible / infrared laser irradiation unit 520 can generate, overlapping each other, linear visible laser patterns LP and linear near-infrared laser patterns LPi at locations on the road surface 300 where the expected passage area 310 is highly likely to exist.

[0144] The visible / infrared laser irradiation unit 520 includes, for example, a laser emitter capable of emitting visible laser light, a light emission control driver capable of controlling the light emission of the laser emitter, an optical system capable of scanning the laser on the road surface 300, and a scan control driver capable of controlling the scanning of the laser by the optical system. The light emission control driver can control the light emission of the laser emitter according to control from the control unit 210. The scan control driver can control the operation of the optical system according to control from the control unit 210. The laser emitter includes, for example, a semiconductor laser that emits visible laser light (laser beam L). The optical system includes, for example, a polygon mirror and an fθ lens. The polygon mirror can reflect the visible laser light emitted from the laser emitter, and the reflected light of the visible laser light is scanned on the road surface 300 via the fθ lens.

[0145] The visible / infrared laser irradiation unit 520, for example, also includes a laser emitter capable of emitting near-infrared laser light, a light emission control driver capable of controlling the light emission of the laser emitter, an optical system capable of scanning the laser on the driving surface 300, and a scan control driver capable of controlling the scanning of the laser by the optical system. The light emission control driver can control the light emission of the laser emitter according to the control unit 210. The scan control driver can control the operation of the optical system according to the control unit 210. The laser emitter, for example, includes a semiconductor laser that emits near-infrared laser light (laser beam L). The optical system, for example, is configured with a polygon mirror and an fθ lens. The polygon mirror can reflect the near-infrared laser light emitted from the laser emitter, and the reflected light of the near-infrared laser light can be scanned on the driving surface 300 via the fθ lens.

[0146] The laser irradiation control unit 25 is capable of controlling the irradiation (drawing) of laser light L1 in the visible area and laser light L2 in the near-infrared area from the visible / infrared laser irradiation unit 520. For example, the laser irradiation control unit 25 can generate control signals required for the irradiation (drawing) of linear visible laser pattern LP and linear near-infrared laser pattern LPi based on the drawing data 191 in the storage unit 190, and output them to the infrared laser irradiation unit 520.

[0147] The laser illumination control unit 25, for example, can generate control signals based on the drawing data 191 to scan the visible area L1 and the near-infrared area L2 on the road surface in front of the vehicle 100 in the driving road surface 300 for scanning the visible area L1 and the near-infrared area L2, and output them to the visible / infrared laser illumination unit 520. The laser illumination control unit 25, for example, can generate control signals based on the drawing data 191 to scan the visible area L1 and the near-infrared area L2 on the driving road surface 300 at locations where the expected passage area 310 is highly likely to exist. The laser illumination control unit 25, for example, can generate control signals based on the drawing data 191 to scan the visible area L1 and the near-infrared area L2 in a direction that intersects obliquely with respect to the long side direction of the expected passage area 310 (i.e., the travel direction of the vehicle 100).

[0148] Next, the effects of the vehicle 100 in this modified example will be explained.

[0149] In this modified example, a linear visible laser pattern LP and a linear near-infrared laser pattern LPi are irradiated onto the road surface (driving surface 300) in front of the vehicle 100. This improves the driver's visual perception of the unevenness of the road surface (driving surface 300) in front of the vehicle 100, while simultaneously detecting the unevenness of the driving surface 300 in real time with high precision, and performing notification control or driving control corresponding to the detection results.

[0150] While embodiments have been listed and the present disclosure described above, the present disclosure is not limited to these embodiments and various modifications are possible. The effects described in this specification are merely illustrative, and the effects of the present disclosure are not limited to those described in this specification. Therefore, other effects can also be obtained with respect to the present disclosure.

[0151] Therefore, this disclosure may take the following forms. (1) A driving assistance device, comprising: The acquisition unit is capable of acquiring image data of the road surface in front of the vehicle; and The control unit is capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing: By controlling the illumination of the driving road surface with a linear light pattern, the acquisition unit acquires first image data of the driving road surface illuminated by the linear light pattern as the image data. If there is a bend in the straight light pattern included in the first image data, calculate the number of pixels in a specific region formed by the straight line and the bend when it is assumed that there is no bend in the straight light pattern, and obtain the position data of the specific region in the first image data. Based on the number of pixels and the location data, at least one of the depth or height and size of the unevenness of the driving surface is estimated; and The notification control or driving control is performed in accordance with the data on the unevenness of the driving surface obtained from the estimated results. (2) According to the driving assistance device described in (1), The control unit is capable of performing: The vertical gap between the straight line and the bend in the first image data is defined as the specific region; and Based on the number of pixels in the specific area and the location data, the depth or height of the unevenness of the driving surface is estimated. (3) According to the driving assistance device described in (1) or (2), The control unit is capable of performing: If a bend and an interruption are detected in the linear light pattern, the gap in the first image data between the starting point of the bend and the end edge of the interruption in the linear light pattern is set as the specific region; and Based on the number of pixels in the specific area and the location data, the size of the unevenness of the driving road surface is estimated. (4) According to any one of (1) to (3), The driving assistance device includes a storage unit that stores a table of the actual spatial dimensions of each pixel in the image data. The control unit can read the actual spatial dimension data of the pixels of the gap from the dimension data table based on the position data, and estimate the depth or height of the unevenness of the driving surface based on the read dimension data and the number of pixels. (5) According to any one of (1) to (4), The acquisition unit can acquire relatively low-resolution second image data and relatively high-resolution third image data as the first image data. The control unit is capable of performing: Based on the second image data, detect whether the unevenness or concavity exists; and If the bump is detected, a portion of the image data of a predetermined region, including the location of the detected bump, is extracted from the third image data. Based on the extracted portion of the image data, the number of pixels is calculated, and the location data is obtained. (6) According to any one of (1) to (5), If the depth or height of the unevenness of the driving surface exceeds a predetermined threshold, the control unit can, as a notification control, notify the driver to avoid the unevenness of the driving surface. (7) According to any one of (1) to (6), If the depth or height of the unevenness of the driving surface exceeds a predetermined threshold, the control unit, as the driving control, is capable of performing braking control to bring the vehicle to a stop near the unevenness of the driving surface. (8) According to any one of (1) to (7), If the depth or height of the unevenness of the road surface exceeds a predetermined threshold, the control unit can perform steering control to make the vehicle avoid the unevenness of the road surface as the driving control. (9) According to any one of (1) to (8), The linear light pattern is a linear visible laser pattern. (10) According to any one of (1) to (8), The linear light pattern is a linear near-infrared light pattern. (11) According to the driving assistance device described in (10), The control unit can determine whether the road surface is wet or dry based on the first image data. If the road surface is wet, it can perform braking or steering control corresponding to wetness, and if the road surface is dry, it can perform braking or steering control corresponding to dryness. (12) A vehicle that has: Driving assistance devices; and The controlled device controlled by the aforementioned driving assistance device, The driving assistance device has: The acquisition unit is capable of acquiring image data of the road surface in front of the vehicle; and The control unit is capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing: By controlling the illumination of the driving road surface with a linear light pattern, the acquisition unit acquires first image data of the driving road surface illuminated by the linear light pattern as the image data. If there is a bend in the straight light pattern included in the first image data, calculate the number of pixels in a specific region formed by the straight line and the bend when it is assumed that there is no bend in the straight light pattern, and obtain the position data of the specific region in the first image data. Based on the number of pixels and the location data, at least one of the depth or height and size of the unevenness of the driving surface is estimated; and The controlled device is subject to notification control or driving control corresponding to the data on the unevenness of the road surface obtained from the estimated result.

[0164] Figure 2 , Figure 12 , Figure 17The control unit 210 shown may be implemented by a circuit including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), and / or at least one field-programmable gate array (FPGA). The at least one processor is configured to execute instructions by reading them from at least one non-transitory and tangible computer-readable medium. Figure 2 , Figure 12 , Figure 17 This refers to all or some of the various functions in the control unit 210 shown. Such media can take various forms, including, but are not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or non-volatile memory. Volatile memory may include DRAM and SRAM. Non-volatile memory may include ROM and NVRAM. ASIC is dedicated to executing... Figure 2 , Figure 12 , Figure 17 The control unit 210 shown contains all or some of the various functions of an integrated circuit (IC). An FPGA is designed to be configured after manufacturing to perform... Figure 2 , Figure 12 , Figure 17 All or some of the functions in the control unit 210 shown.

Claims

1. A driving assistance device, characterized in that, have: The acquisition unit is capable of acquiring image data of the road surface in front of the vehicle; and The control unit is capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing: By controlling the illumination of the driving road surface with a linear light pattern, the acquisition unit acquires first image data of the driving road surface illuminated by the linear light pattern as the image data. If there is a bend in the straight light pattern included in the first image data, calculate the number of pixels in a specific region formed by the straight line and the bend when it is assumed that there is no bend in the straight light pattern, and obtain the position data of the specific region in the first image data. Based on the number of pixels and the location data, at least one of the depth or height and size of the unevenness of the driving surface is estimated; as well as The notification control or driving control is performed in accordance with the data on the unevenness of the driving surface obtained from the estimated results.

2. The driving assistance device according to claim 1, characterized in that, The control unit is capable of performing: The vertical gap between the straight line and the bend in the first image data is defined as the specific region; and Based on the number of pixels in the specific area and the location data, the depth or height of the unevenness of the driving surface is estimated.

3. The driving assistance device according to claim 1, characterized in that, The control unit is capable of performing: If the bend and interruption are detected in the linear light pattern, the gap between the starting point of the bend and the end edge of the interruption in the first image data is set as the specific region. as well as Based on the number of pixels in the specific area and the location data, the size of the unevenness of the driving road surface is estimated.

4. The driving assistance device according to claim 1, characterized in that, The driving assistance device includes a storage unit that stores a table of the actual spatial dimensions of each pixel in the image data. The control unit can read the actual spatial dimension data of the pixels of the gap from the dimension data table based on the position data, and estimate the depth or height of the unevenness of the driving surface based on the read dimension data and the number of pixels.

5. The driving assistance device according to claim 1, characterized in that, The acquisition unit can acquire relatively low-resolution second image data and relatively high-resolution third image data as the first image data. The control unit is capable of performing: The presence or absence of the aforementioned bumps is detected based on the second image data; as well as If the bump is detected, a portion of the image data of a predetermined region, including the location of the detected bump, is extracted from the third image data. Based on the extracted portion of the image data, the number of pixels is calculated, and the location data is obtained.

6. The driving assistance device according to claim 1, characterized in that, If the depth or height of the unevenness of the driving surface exceeds a predetermined threshold, the control unit can, as a notification control, notify the driver to avoid the unevenness of the driving surface.

7. The driving assistance device according to claim 1, characterized in that, If the depth or height of the unevenness of the road surface exceeds a predetermined threshold, the control unit, as the driving control, can perform braking control to stop the vehicle in front of the unevenness of the road surface.

8. The driving assistance device according to claim 1, characterized in that, If the depth or height of the unevenness of the road surface exceeds a predetermined threshold, the control unit can perform steering control to make the vehicle avoid the unevenness of the road surface as the driving control.

9. The driving assistance device according to claim 1, characterized in that, The control unit can determine whether the road surface is wet or dry based on the first image data. If the road surface is wet, it can perform braking or steering control corresponding to wetness, and if the road surface is dry, it can perform braking or steering control corresponding to dryness.

10. A vehicle, characterized in that, have: Driving assistance devices; and The controlled device controlled by the aforementioned driving assistance device, The driving assistance device has: The acquisition unit is capable of acquiring image data of the road surface in front of the vehicle; and The control unit is capable of deriving the depth or height of the road surface's unevenness based on the image data. The control unit is capable of performing: By controlling the illumination of the driving road surface with a linear light pattern, the acquisition unit acquires first image data of the driving road surface illuminated by the linear light pattern as the image data. If there is a bend in the straight light pattern included in the first image data, calculate the number of pixels in a specific region formed by the straight line and the bend when it is assumed that there is no bend in the straight light pattern, and obtain the position data of the specific region in the first image data. Based on the number of pixels and the location data, at least one of the depth or height and size of the unevenness of the driving surface is estimated; as well as The controlled device is subject to notification control or driving control corresponding to the data on the unevenness of the road surface obtained from the estimated result.

Images