Vehicle driving assistance device

Abstract

To provide a travel support apparatus for a vehicle capable of appropriately supporting travel of the vehicle by more accurately detecting a driving disabled state of a driver.SOLUTION: A control device 17 of a travel support apparatus 3 detects a position of the head of a driver based on state information of the driver (ST1) and, in a case where the position of the head moves (ST3: No), limits a control amount based on a driving operation with respect to a travel apparatus (6) (ST10). The control device 17 limits the control amount with a first limit degree (ST10) in a first state where the head moves in a vehicle width direction (ST7: Yes, ST9) and limits the control amount with a second limit degree greater than the first limit degree (ST12) in a second state where the head moves in a vertical direction (ST7: No, ST11).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a vehicle driving assistance device. [Background technology]

[0002] Conventionally, a driver incapacity detection device is known that detects when a driver is unable to drive in order to prevent accidents caused by a driver's inability to drive due to sudden illness or other reasons (Patent Document 1). This driver incapacity detection device sequentially detects the driver's head above the neck based on an image of the driver's seat captured by an imaging device, and detects that the driver is incapacitated when the head is outside a predetermined range of the image while the vehicle is moving. The driver incapacity detection device modifies the predetermined range of the image used to detect the driver incapacity according to the intentions of users, including the driver, or driver information related to the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6372388 Summary of the Invention [Problem to be solved by the invention]

[0004] However, even when the driver is in a normal state where he or she is able to drive (hereinafter simply referred to as the normal state), the driver's head may move as the driver changes posture while driving. However, in the above-mentioned conventional technology, although the driver's state is determined based on whether or not the driver's head is within a predetermined range, no consideration is given to changing the control of the vehicle based on how the head has framed out of the predetermined range, and there is room for improvement in the accuracy of detecting a state in which the driver is unable to drive.

[0005] In addition, while a driver may change the direction of their face due to a driver's inability to drive, such as sudden illness, the driver may also change the direction of their face while driving, even if the driver is able to drive. In these cases, changing vehicle control based on how the driver's face direction has not been considered in the past, and there has been room for improvement in the accuracy of detecting a driver's inability to drive.

[0006] In view of the above background, an object of the present invention is to provide a vehicle driving assistance device that can more accurately detect when a driver is unable to drive and appropriately assist the vehicle in driving. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is a vehicle driving assistance device (3), comprising: a state information acquisition unit (15) that acquires state information of a driver; a driving device (6) for driving the vehicle; and a control device (17) that controls the driving device based on a driving operation of the driver and controls the driving device to assist the driving operation based on the state information of the driver, wherein the control device detects the position of the driver's head or the direction of the driver's face (ST1, ST2) based on the state information, and when the position of the head moves or the direction of the face changes, If the head moves in the vehicle width direction (ST7: Yes) or the face is deflected in the vehicle width direction (ST8: Yes), the control device limits the control amount based on the driving operation of the traveling device (ST10, ST12). If the head moves in the vehicle width direction (ST7: Yes) or the face is deflected in the vehicle width direction (ST8: Yes), the control device limits the control amount with a first limiting degree (ST10). If the head moves in the up / down direction (ST7: No) or the face is deflected in the up / down direction (ST8: No), the control device limits the control amount with a second limiting degree that is greater than the first limiting degree (ST12).

[0008] According to this aspect, by distinguishing between a first state in which the head moves in the vehicle width direction or the face is deflected in the vehicle width direction and a second state in which the head moves in the up and down direction or the face is deflected in the up and down direction, it is possible to more accurately detect the driver's incapacity to drive. Then, when the driver is in the second state in which there is a high possibility that the driver is incapacitated, the control device controls the control amount with a second restriction degree that is greater than the first restriction degree when the driver is in the first state in which there is a low possibility that the driver is incapacitated. limit This makes it possible to prevent the vehicle from being driven by unintended driving operations such as accidentally pressing the accelerator pedal, thereby providing appropriate support for the vehicle's driving.

[0009] In the above aspect, the control device determines, based on the state information, whether the driver's head is in a predetermined area (37) that has been set in advance as an area where the driver's head should be while driving, and limits the control amount if the head is framed out of the predetermined area (ST3: No), and the first state (ST9) includes a state in which the driver's head is framed out of the predetermined area in the vehicle width direction (ST7: Yes), and the second state (ST11) includes a state in which the driver's head is framed out of the predetermined area in the vertical direction (ST7: No).

[0010] According to this aspect, it is possible to appropriately assist the vehicle in driving in accordance with the driver's condition.

[0011] In the above aspect, the control device detects the driver's face based on the state information, and detects multiple feature parts on the face, calculates the degree of deflection of the orientation of the face toward the front based on at least the positions of the feature parts, and limits the control amount when the degree of deflection exceeds a predetermined threshold (ST14: No), and the first state (ST9) includes a state (ST8: Yes) in which the degree of deflection is greater than the threshold because at least one of the driver's face and the feature parts deforms or moves in the vehicle width direction, and the second state (ST11) includes a state (ST8: No) in which the degree of deflection is greater than the threshold because at least one of the driver's face and the feature parts moves in the up and down direction.

[0012] According to this aspect, it is possible to appropriately assist the vehicle in driving in accordance with the driver's condition.

[0013] In the above aspect, the control device sets an upper limit value for the driver's driving operation amount (AP), and limits the control amount based on the driver's driving operation by not using the driving operation amount when the driving operation amount exceeds the upper limit value, and in the second state, the control device sets the upper limit value to a value smaller than in the first state.

[0014] According to this aspect, when the driver is in the second state in which there is a high possibility that he or she is unable to drive, the upper limit value is set to a value smaller than that in the first state, thereby appropriately suppressing the vehicle from traveling due to driving operations unintended by the driver.

[0015] In the above aspect, if the driver starts the driving operation after the driver's head frames out of the specified area (ST32: No), the control device limits the control amount based on the driver's driving operation (ST36, ST37).

[0016] According to this aspect, it is possible to appropriately prevent the vehicle from traveling due to unintended driving operations by the driver.

[0017] In the above aspect, if the driver's driving operation is continued before the driver's head frames out of the specified area (ST32: Yes), the control device does not limit the control amount based on the driver's driving operation (ST33).

[0018] According to this aspect, the driving operation intended by the driver can be appropriately reflected in the running of the vehicle.

[0019] In the above aspect, if the driver's head frames out of the specified area and then frames back into the specified area while the driver is performing the driving operation (ST25: No), the control device continues to limit the control amount based on the driver's driving operation, which has been ongoing since before the head frames back into the specified area (ST29).

[0020] According to this aspect, it is possible to suppress a sudden change in the behavior of the vehicle after the head enters the frame.

[0021] In the above aspect, the vehicle is further provided with an alarm device (16) for notifying the occupants of the vehicle, and when the driver starts the driving operation after the driver's head frames out of the specified area (ST32: No, ST42: No), the control device executes alarm control on the alarm device (ST34, ST44) to notify the occupants of the vehicle.

[0022] According to this aspect, unnecessary alerts by the alert device can be prevented when no driving operations are being performed, while the alert device can alert occupants when there is a possibility that the driver is unable to drive.

[0023] In the above aspect, when the head of the driver frames out of the predetermined area and then frames back into the predetermined area (ST21: Yes), the control device ends the notification control (ST22).

[0024] According to this aspect, when the driver's head is in the frame, it is determined that the driver is ready to drive, and it is possible to prevent the notification device from continuing to provide unnecessary notifications.

[0025] In the above aspect, the control device prohibits the control based on the driving operation of the driver in the second state (ST43, ST46, ST47).

[0026] According to this aspect, it is possible to prevent the vehicle from traveling due to an unintended driving operation by the driver.

[0027] In the above aspect, when the driver starts the driving operation after the driver's head frames out of the specified area in the second state (ST42: No), the control device prohibits the control based on the driver's driving operation (ST46, ST47).

[0028] According to this aspect, it is possible to appropriately prevent the vehicle from traveling due to a driving operation unintentional by the driver.

[0029] In the above aspect, the control device recognizes whether the driver is gripping or not gripping the steering wheel based on the state information of the driver, and when the control device recognizes the driver not gripping the steering wheel in the first state (ST9) or the second state (ST11) (ST35: No, ST45: No), the control device limits the control amount based on the driver's driving operation to a greater degree (ST37, ST47) than when the control device recognizes the driver gripping the steering wheel (ST35: Yes, ST45: Yes).

[0030] According to this aspect, it is possible to appropriately prevent the vehicle from traveling due to unintended driving operations by the driver. [Effects of the Invention]

[0031] According to the above aspect, it is possible to provide a vehicle driving assistance device that can more accurately detect when a driver is unable to drive and appropriately assist the vehicle in driving. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a functional configuration diagram showing a driving assistance device according to an embodiment of the present invention; [Figure 2] Diagram showing images from an in-car camera [Figure 3] Illustration of head position changes in images taken by the in-car camera [Figure 4] Illustration of changes in facial direction in images taken with an in-car camera [Figure 5] Flow diagram of driving assistance control by the control device [Figure 6] Normal control flow diagram in Figure 5 [Figure 7] Flow diagram of control in the first abnormality in Fig. 5 [Figure 8] Flow diagram of the second abnormality control in Figure 5 DETAILED DESCRIPTION OF THE INVENTION

[0033] First, with reference to FIG. 1, the configuration of a driving assistance device 3 of a vehicle 1 according to one embodiment of the present invention will be described.

[0034] The driving assistance device 3 is provided in a vehicle 1. The vehicle 1 is, for example, a four-wheeled automobile. The vehicle 1 includes a drive device 6 (an example of a driving device), a brake device 7, a steering device 8, an HMI 9 (Human Machine Interface), a navigation device 10, a vehicle sensor 11, a driving operator 12, a driving operation sensor 13 (an example of a driving operation information acquisition unit), an external sensor 14, an in-vehicle sensor 15 (an example of a status information acquisition unit), a notification device 16, and a control device 17.

[0035] The drive unit 6 is a device that applies drive force to the vehicle 1. For example, the drive unit 6 includes an internal combustion engine such as a gasoline engine or a diesel engine and / or an electric motor.

[0036] The braking device 7 is a device that applies braking force to the vehicle 1. For example, the braking device 7 includes a brake caliper that presses a pad against a brake rotor, and an electric cylinder that supplies hydraulic pressure to the brake caliper.

[0037] The steering device 8 is a device that changes the steering angle of the wheels. For example, the steering device 8 includes a rack and pinion mechanism that steers the wheels, and an electric motor that drives the rack and pinion mechanism.

[0038] The HMI 9 is a device that displays information to the driver and accepts information input by the driver. The HMI 9 is disposed inside the vehicle 1 (for example, on the dashboard). The HMI 9 includes a touch panel with a display screen.

[0039] The navigation device 10 is a device that provides route guidance to the destination of the vehicle 1. The navigation device 10 stores map information. The navigation device 10 identifies the current position (latitude and longitude) of the vehicle 1 based on GNSS signals received from artificial satellites (positioning satellites). The navigation device 10 sets a route to the destination of the vehicle 1 based on the map information, the current position of the vehicle 1, and the destination of the vehicle 1 input by the driver to the HMI 9.

[0040] The vehicle sensor 11 is a sensor that detects various vehicle conditions. For example, the vehicle sensor 11 may include a vehicle speed sensor that detects the speed of the vehicle 1, an acceleration sensor that detects the acceleration of the vehicle 1, a yaw rate sensor that detects the angular velocity of the vehicle 1 around a vertical axis, and a direction sensor that detects the orientation of the vehicle 1. The vehicle sensor 11 outputs the detection results to the control device 17.

[0041] The driving operation controls 12 are devices that accept the driver's operation to drive the vehicle 1. The driving operation controls 12 include a steering wheel that accepts the driver's steering operation, an accelerator pedal that accepts the driver's acceleration operation, and a brake pedal that accepts the driver's deceleration operation.

[0042] The driving operation sensor 13 is a sensor that detects the amount of driving operation performed on the driving operators 12. In other words, the driving operation sensor 13 is a sensor that acquires information about driving operations performed on the driving operators 12. The driving operation sensor 13 includes a steering angle sensor that detects the rotation angle of the steering wheel, an accelerator sensor that detects the amount of depression of the accelerator pedal, and a brake sensor that detects the amount of depression of the brake pedal. The driving operation sensor 13 outputs the detection results to the control device 17.

[0043] The external sensor 14 is a sensor that detects the state of the outside world of the vehicle 1. For example, the external sensor 14 detects the relative position of a target object present around the vehicle 1 with respect to the vehicle 1. In other words, the external sensor 14 acquires position information of the target object. The target object includes other vehicles such as a vehicle ahead, pedestrians, bicycles, obstacles, etc. The external sensor 14 outputs the detection result to the control device 17.

[0044] The external sensor 14 includes a plurality of external cameras 21, a plurality of radars 22, and a plurality of lidars 23 (LiDAR). The external cameras 21 capture images of targets present around the vehicle 1. The radar 22 emits radio waves such as millimeter waves around the vehicle 1 and captures the reflected waves to detect the relative positions of targets present around the vehicle 1 with respect to the vehicle 1. The lidar 23 irradiates light such as infrared light around the vehicle 1 and captures the reflected light to detect the relative positions of targets present around the vehicle 1 with respect to the vehicle 1.

[0045] The interior sensor 15 is a sensor that detects the interior state of the vehicle 1, specifically the state of the driver. In other words, the interior sensor 15 is a sensor that acquires driver state information. The interior sensor 15 includes an interior camera 24 and a grip sensor 25. The interior camera 24 captures images of the driver's seat and detects the head position and face of the driver sitting in the driver's seat, as well as facial features such as the eyes, nose, and mouth. The interior camera 24 may also detect the driver's arms and hands. The interior camera 24 is located inside the vehicle 1 (for example, in front of the steering wheel). The grip sensor 25 is provided on the steering wheel and detects whether the driver is gripping or not gripping the steering wheel. The interior camera 24 and the grip sensor 25 output the captured images to the control device 17.

[0046] The notification device 16 is a device that notifies the occupants of the vehicle 1, including the driver. The notification device 16 includes a display 26 and a speaker 27. The display 26 is a notification unit that is arranged in a position that is easily visible in front of the driver and occupants, such as a HUD (Head Up Display: an example of a display unit), and that visually transmits information to the occupants. The speaker 27 may be part of an audio device mounted on the vehicle 1, or may be shared with the navigation device 10, etc. Alternatively, the speaker 27 may be provided exclusively for notification. The speaker 27 is a notification unit that audibly transmits information to the occupants.

[0047] The control device 17 is an electronic control unit (ECU) consisting of a computer configured to execute various processes. The control device 17 includes an arithmetic processing unit (a processor such as a CPU or an MPU) and a storage device (memory such as a ROM or a RAM). The arithmetic processing unit reads necessary software from the storage device and executes predetermined arithmetic processing in accordance with the read software. The control device 17 may be configured as a single piece of hardware, or may be configured as a unit consisting of multiple pieces of hardware. The control device 17 is connected to each component of the vehicle 1 via a communication network such as a CAN (Controller Area Network), and controls each component of the vehicle 1.

[0048] The control device 17 includes, as functional units, an external environment recognition unit 31, a driving control unit 32, a driver state recognition unit 33, a grip recognition unit 34, and a notification control unit 35. At least a part of the functional units of the control device 17 may be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0049] The external environment recognition unit 31 recognizes the state of the external environment of the vehicle 1 based on the detection results of the external environment sensor 14. For example, the external environment recognition unit 31 recognizes targets present around the vehicle 1 based on the detection results of the external environment sensor 14, and recognizes the relative position of the target with respect to the vehicle 1, the relative speed of the target with respect to the vehicle 1, the distance from the vehicle 1 to the target, etc.

[0050] The external environment recognition unit 31 acquires the type of target, its position (absolute position), moving speed, past moving direction, and surrounding environment based on the detection results of the external environment sensor 14. Note that the external environment recognition unit 31 may acquire this information not only based on the detection results of the external environment sensor 14 but also based on the detection results of the vehicle sensor 11 and GNSS signals. The type of target includes other vehicles such as a vehicle ahead, pedestrians, bicycles, etc. The surrounding environment of the target includes the shape and width of the road around the target. If the target is a pedestrian, the surrounding environment of the target includes other pedestrians, etc., who are present on the opposite side of the road from the pedestrian. If the target is a vehicle ahead, the surrounding environment of the target includes traffic lights present ahead of the vehicle ahead, the vehicle ahead of the vehicle ahead, etc.

[0051] The external environment recognition unit 31 estimates the future movement direction of the target based on at least one of the type, position, movement speed, past movement direction, and surrounding environment of the target. This allows the external environment recognition unit 31 to accurately estimate the future movement direction of the target. For example, the external environment recognition unit 31 calculates a probability distribution (Gaussian distribution) in which the direction centered on the target is a random variable based on the type, position, movement speed, past movement direction, and surrounding environment of the target. Then, the external environment recognition unit 31 may estimate the direction with the highest probability density in the above probability distribution as the future movement direction of the target.

[0052] The driving control unit 32 controls the drive system 6, the brake system 7, and the steering system 8 required for driving the vehicle 1 based on information related to the driving operation of the driving operators 12 detected by the driving operation sensor 13. For example, the driving control unit 32 controls the control variables (throttle opening, amount of fuel supplied to the internal combustion engine, power supplied to the electric motor, etc.) of the drive system 6 based on the detection result of the accelerator sensor (accelerator depression amount AP). The driving control unit 32 controls the hydraulic pressure of the brake system 7 based on the detection result of the brake sensor (depression amount of the brake pedal). The driving control unit 32 controls the electric motor of the steering system 8 based on the detection result of the steering angle sensor (rotation angle of the steering wheel).

[0053] Furthermore, the driving control unit 32 performs driving assistance control to control driving devices such as the drive unit 6, the brake unit 7, and the steering unit 8 to assist driving operation. For example, the driving control unit 32 performs collision prevention control of the vehicle 1 based on the state of the external environment of the vehicle 1 recognized by the external environment recognition unit 31. For example, the driving control unit 32 performs acceleration / deceleration control and steering control of the vehicle 1 based on the relative position of a target with respect to the vehicle 1 recognized by the external environment recognition unit 31.

[0054] The cruise control unit 32 executes control to follow the vehicle in front, such as adaptive cruise control (ACC), as acceleration / deceleration control for the vehicle 1. In the control to follow the vehicle in front, the cruise control unit 32 controls the drive device 6 and the brake device 7 so that the inter-vehicle distance between the vehicle 1 and the vehicle in front is maintained within a predetermined range.

[0055] The driving control unit 32 executes pedal misapplication prevention control, which limits the amount of control for driving devices such as the drive unit 6 and the brake unit 7 based on the driver's driving operation (detection results of the driving operation sensor 13), as acceleration / deceleration control for the vehicle 1. The driving control unit 32 executes pedal misapplication prevention control based on the recognition results of the driver state recognition unit 33, that is, based on the driver state information acquired by the in-vehicle camera 24 and the grip sensor 25.

[0056] The driving control unit 32 executes lane keeping control such as LKAS (Lane Keeping Assist System) as steering control for the vehicle 1. In the lane keeping control, the driving control unit 32 controls the steering device 8 so that the vehicle 1 travels at a reference position (for example, near the center of the lane in the width direction) within a lane defined by dividing lines. The driving control unit 32 may also limit the steering of the steering device 8 based on the driver's steering operation, based on driver state information acquired by the in-vehicle camera 24 or the grip sensor 25.

[0057] The driver state recognition unit 33 recognizes the position of the driver's head and also recognizes the driver's face and its characteristic features based on the image captured by the in-vehicle camera 24. For example, the driver state recognition unit 33 extracts the positions of the driver's left and right eyes and mouth from the image of the driver's face captured by the in-vehicle camera 24, and recognizes the direction of the driver's face based on the relative positions of these.

[0058] The grip recognition unit 34 recognizes whether the driver is gripping or not gripping the steering wheel based on the driver's state information detected by the in-vehicle sensor 15. For example, the grip recognition unit 34 recognizes whether the driver is gripping or not gripping the steering wheel based on the information of the steering wheel detected by the grip sensor 25. Alternatively, the grip recognition unit 34 may recognize whether the driver is gripping or not gripping the steering wheel based on the image capture results of the in-vehicle camera 24, or may recognize whether the driver is gripping or not gripping the steering wheel based on a combination of these pieces of information.

[0059] The notification control unit 35 controls the notification device 16. More specifically, the notification control unit 35 executes notification control for the notification device 16 based on the recognition result of the external environment recognition unit 31, the execution state of driving support control by the driving control unit 32, the recognition result of the driver state recognition unit 33, etc. For example, when the driver state recognition unit 33 recognizes that the driver is incapable of driving (hereinafter also referred to as an abnormal state), the notification control unit 35 drives the display 26 and the speaker 27 to notify the occupant visually and audibly.

[0060] For the sake of convenience, the functional units of the control device 17 will hereinafter be referred to simply as "control device 17" without distinction.

[0061] Next, the driver's state recognition performed by the control device 17 will be described with reference to FIGS.

[0062] FIG. 2 is a diagram showing an image taken by the in-vehicle camera 24. The control device 17 sets a predetermined area 37 in the image of the driver's seat taken by the in-vehicle camera 24 as an area where the driver's head should be while driving. In this embodiment, the control device 17 sets the predetermined area 37 as a vertically long rectangle. In other embodiments, the predetermined area 37 may be set to a shape other than a rectangle. The position and size of the predetermined area 37 may be set appropriately according to the position and size of the head of the driver seated in the driver's seat. Furthermore, the shape of the predetermined area 37 may be set to match the shape of the driver's head.

[0063] The control device 17 analyzes the image from the in-vehicle camera 24 and sequentially detects the driver's head. Specifically, the control device 17 extracts an edge representing the outline of the driver's face from the image of the driver's seat, and detects a rectangular area enclosed by vertical lines passing through the left and right ends of the extracted edge and horizontal lines passing through the top and bottom ends of the edge as the head (face). The predetermined area 37 is set to a size that includes predetermined margins on the left and right and above and below the head, and when the driver is in a normal state, the driver's face is located approximately in the center of the predetermined area 37. The control device 17 acquires the trajectory of the driver's head from the sequentially detected positions of the driver's head.

[0064] The control device 17 also detects features such as the eyes, nose, and mouth from the driver's face. The control device 17 detects the direction of the driver's face based on the outline of the driver's face and the positions of the features. The direction of the face is calculated as the direction and degree of deflection (angle) of the driver's face relative to the forward direction to which the driver's face should be facing while driving. The control device 17 acquires changes in the direction of the driver's face from the directions of the driver's face that are successively detected.

[0065] Figure 3 is an explanatory diagram of changes in head position in the image captured by the in-vehicle camera 24. If the driver, who is in a normal state as shown in Figure 2, suddenly falls ill, for example, the driver will likely tilt his head significantly forward as shown in Figure 3(A1). In this case, the control device 17 detects that the driver's head has fallen out of the frame downward from the predetermined area 37 as shown in Figure 3(A2).

[0066] On the other hand, as shown in Fig. 3(B1), the driver may move his / her head from side to side while driving, for example, when visually checking a white line through a side window or when visually checking the rear of the vehicle 1. In this case, the control device 17 detects that the driver's head has moved out of the frame in the vehicle width direction from the predetermined area 37, as shown in Fig. 3(B2).

[0067] In this way, when the driver's head goes out of the frame from the predetermined area 37, the control device 17 determines whether the head has gone out of the frame in the vehicle width direction or in the vertical direction.

[0068] FIG. 4 is an explanatory diagram of a change in the face direction in an image captured by the in-vehicle camera 24. A driver in a normal state as shown in FIG. 4(A) may turn his face in the vehicle width direction (left or right) as shown in FIG. 4(B) to visually check, for example, the side mirrors or another vehicle diagonally behind. In this case, the control device 17 detects that the driver's face has turned in the vehicle width direction by deforming so that the driver's face becomes thinner in the vehicle width direction and by deforming or moving the characteristic parts in the vehicle width direction. More specifically, for example, a change in the face direction in the vehicle width direction is detected by detecting that one of the left or right eyes moves in the vehicle width direction, shortening the distance between the left and right eyes.

[0069] On the other hand, if the driver suddenly falls ill, for example, the head often tilts forward significantly from the normal state shown in Fig. 4(A) to the state shown in Fig. 4(C). In this case, the control device 17 detects that the driver's face has become thinner in the vertical direction and that the driver's face has turned upside down due to the deformation or movement of the characteristic parts in the vertical direction. More specifically, for example, the left and right eyes move downward, and the distance between the left and right eyes and the mouth becomes shorter, thereby detecting that the direction of the face has changed downward.

[0070] The control device 17 may also determine the face direction when feature points are no longer detected. For example, when the right eye is no longer detected, the control device 17 detects that the driver's face has turned to the right. When the mouth is no longer detected, the control device 17 detects that the driver's face has turned downward.

[0071] In this way, when the direction of the driver's face changes, the control device 17 determines whether the face has been deflected in the vehicle width direction or in the up-down direction.

[0072] The control device 17 executes the various driving support controls exemplified above based on the driver's state information thus acquired from the image captured by the in-vehicle camera 24.

[0073] Next, the driving support control of the vehicle 1 performed by the control device 17 will be described with reference to Figures 5 to 8. Here, of the driving support controls performed by the control device 17, the pedal misapplication prevention control will be described as an example.

[0074] 5 is a flow diagram of driving support control by control device 17. When driving support control is started, control device 17 detects the position of the driver's head from the image captured by in-vehicle camera 24 (step ST1). Next, control device 17 detects the direction of the driver's face from the image captured by in-vehicle camera 24 (step ST2).

[0075] Next, the control device 17 determines whether the driver's head is within the predetermined area 37 (step ST3). For example, if the entire face of the driver is within the predetermined area 37, the control device 17 determines that the head is within the predetermined area 37. On the other hand, if part of the driver's face protrudes from the predetermined area 37 or the driver's face is not within the predetermined area 37, the control device 17 determines that the head is not within the predetermined area 37.

[0076] If the determination in step ST3 is that the driver's head is within the predetermined area 37 (Yes), the control device 17 determines whether the degree of deflection of the driver's face is equal to or less than a predetermined threshold (step ST4). If the determination in step ST4 is that the degree of deflection of the driver's face is equal to or less than the predetermined threshold (Yes), the control device 17 recognizes that the driver is in a normal state where he or she can drive (step ST5), executes normal state control (step ST6), and repeats the above processing. The normal state control will be described later.

[0077] On the other hand, if the determination in step ST3 is that the driver's head is not within the predetermined area 37 (No), the control device 17 determines whether the driver's head has left or right of the predetermined area 37 (step ST7). If the determination in step ST7 is that the driver's head has left or right of the predetermined area 37 (Yes), the control device 17 proceeds to step ST9. If the determination in step ST7 is that the driver's head has left or right of the predetermined area 37 (No), the control device 17 proceeds to step ST11.

[0078] Furthermore, if the determination in step ST4 indicates that the degree of deflection of the driver's face is greater than a predetermined threshold (No), the control device 17 performs the determination in step ST8. In step ST8, the control device 17 determines whether the driver's face has deflected left or right, that is, whether the degree of deflection has exceeded a threshold due to deformation or movement of the driver's face or characteristic features in the vehicle width direction. If the determination in step ST8 indicates that the driver's face has deflected left or right (Yes), the control device 17 proceeds to the process in step ST9. If the determination in step ST8 indicates that the driver's face has deflected up or down (No), the control device 17 proceeds to the process in step ST11.

[0079] In step ST9, the control device 17 recognizes that the driver is in a first abnormal state, which means that there is a possibility that the driver is in an inoperable state, but that the possibility of the driver being in an inoperable state is low. When the control device 17 recognizes that the driver is in the first abnormal state in step ST9, it executes first abnormality control, which imposes a first restriction degree on the control amount of the drive device 6 based on the driving operation of the driver (step ST10), and repeats the above processing. The first abnormality control will be described later.

[0080] In step ST11, the control device 17 recognizes that the driver may be in an inoperable state and is in a second abnormal state in which the driver is likely to be in an inoperable state. When the control device 17 recognizes that the driver is in the second abnormal state in step ST11, the control device 17 executes second abnormality control that imposes a second restriction degree on the control amount of the drive device 6 based on the driving operation of the driver (step ST12), and repeats the above processing. The second restriction degree is a restriction degree greater than the first restriction degree. The second abnormality control will be described later.

[0081] In this way, the control device 17 distinguishes between a first abnormal state in which the driver's head is out of frame left or right or the driver's face is deflected left or right, and a second abnormal state in which the driver's head is out of frame up or down or the driver's face is deflected up or down. This allows the control device 17 to more accurately detect the driver's incapacity to drive. Then, when the second abnormal state is in which it is highly likely that the driver is incapacitated, the control device 17 adjusts the control amount with a second restriction degree that is greater than the first restriction degree when the first abnormal state is in which it is low likely that the driver is incapacitated. limit This prevents the vehicle 1 from traveling due to unintended driving operations by the driver, such as accidentally stepping on the accelerator pedal, and appropriately supports the traveling of the vehicle 1.

[0082] During normal driving, the driver may move his / her head from side to side to visually check the white line, for example, through the side window. On the other hand, if the driver's head is framed out of the predetermined area 37 in the vertical direction, the driver is more likely to be in a state where he / she is unable to drive, compared to when the driver's head is framed out of the predetermined area 37 in the vehicle width direction. In this embodiment, if the driver's head is framed out of the predetermined area 37 in the vehicle width direction (ST7: Yes), the control device 17 recognizes a first abnormal state in step ST9. Also, if the driver's head is framed out of the predetermined area 37 in the vertical direction (ST7: No), the control device 17 recognizes a second abnormal state in step ST11. In other words, when the driver's head is framed out of the predetermined area 37 in the vertical direction, the control device 17 limits the control amount with a second limiting degree that is greater than the first limiting degree. This allows the vehicle 1 to be appropriately assisted in driving according to the driver's condition.

[0083] During normal driving, the driver may turn his / her face in the vehicle width direction (left or right) to visually check, for example, the side mirrors or other vehicles diagonally behind. On the other hand, when the driver's face is turned up or down, the driver is more likely to be in a state where he / she is unable to drive compared to when the face is turned up or down. In this embodiment, the control device 17 recognizes a first abnormal state in step ST9 when at least one of the driver's face and characteristic features deforms or moves in the vehicle width direction, causing the degree of deflection to be greater than a threshold (ST8: Yes). Furthermore, the control device 17 recognizes a second abnormal state (ST11) in step ST11 when at least one of the driver's face and characteristic features moves up or down, causing the degree of deflection to be greater than a threshold (ST8: No). In other words, when the driver's face is turned up or down, the control device 17 limits the control amount by a second limiting degree that is greater than the first limiting degree. This allows the vehicle 1 to appropriately support its driving in accordance with the driver's condition.

[0084] Fig. 6 is a flow diagram of the normal state control in Fig. 5. In the normal state control, the control device 17 first determines whether or not a notification has been made to the occupant, that is, whether or not the above notification control is being executed (step ST21). If the determination in step ST21 is that a notification has been made (Yes), the control device 17 ends the notification control (step ST22) and proceeds to step ST23. If the determination in step ST21 is that a notification has not been made (No), the control device 17 proceeds directly to step ST23.

[0085] In step ST23, the control device 17 determines whether or not the driver is operating the accelerator (depressing the accelerator pedal). If the determination in step ST23 is that there is no accelerator operation (No), the control device 17 removes the restriction on the control amount of the drive device 6 that is based on the driver's accelerator operation. On the other hand, if the determination in step ST23 is that there is an accelerator operation (Yes), the control device 17 determines whether or not this accelerator operation is an operation that was started in a normal state (step ST25).

[0086] If it is determined in step ST25 that the accelerator operation was started in a normal state (Yes), the control device 17 then determines whether a predetermined restriction release condition is satisfied (step ST26). The predetermined restriction release condition may be, for example, that a restriction release operator provided in the driver's seat has been operated or that the accelerator operation has been repeatedly input. If it is determined in step ST26 that the predetermined restriction release condition is satisfied (Yes), the control device 17 proceeds to step ST24 and releases the restriction.

[0087] If the predetermined restriction release condition is not satisfied in the determination of step ST26 (No) and after step ST24, the control device 17 determines whether the restriction has been released (step ST27). If the restriction has been released in the determination of step ST27 (Yes), the control device 17 controls the drive device 6 in accordance with the accelerator depression amount AP (step ST28) and repeats the above process.

[0088] On the other hand, if the determination in step ST25 is that the accelerator operation was not initiated in a normal state (No), and if the determination in step ST27 is that the restriction has not been released (No), the control device 17 proceeds to step ST29. In step ST29, the control device 17 continues the restriction to control the drive device 6 (step ST29) because the restriction imposed in the first abnormality control in step ST10 or the second abnormality control in step ST12 has not been released, and repeats the above process.

[0089] Fig. 7 is a flow diagram of the first abnormality control in Fig. 5. In the first abnormality control, the control device 17 determines whether or not the driver is operating the accelerator (step ST31). If the determination in step ST31 is that the accelerator is being operated (Yes), the control device 17 determines whether or not this accelerator operation is an operation that has been continued from a normal state (step ST32).

[0090] If the determination in step ST31 is that there is no accelerator operation (No), and if the determination in step ST32 is that the accelerator operation is continuing from a normal state (Yes), the control device 17 proceeds to step ST33. In step ST33, the control device 17 controls the drive device 6 in accordance with the accelerator depression amount AP (step ST33), and repeats the above process.

[0091] If the determination in step ST32 indicates that the accelerator operation is not a continuation of the normal state (No), the control device 17 executes notification control (step ST34). Next, the control device 17 determines whether or not it recognizes that the steering wheel is being gripped (step ST35). If the determination in step ST35 indicates that the steering wheel is being gripped (Yes), the control device 17 controls the drive device 6 with a first restriction degree (step ST36). On the other hand, if the determination in step ST35 indicates that the steering wheel is not being gripped (No), the control device 17 controls the drive device 6 with a restriction degree greater than the first restriction degree (step ST37).

[0092] The first restriction degree may be, for example, 20%. That is, when the throttle opening is restricted by the first restriction degree, the throttle opening is restricted to 20% when the maximum value is 100%, and the upper limit is 80%. Alternatively, when an operation that exceeds the upper limit is performed, the control device 17 may ignore the operation that exceeds the upper limit, thereby restricting the throttle opening. In either case, the control device 17 sets an upper limit value for the throttle opening, and controls the drive device 6 by restricting the throttle opening to 80%, even when the accelerator depression amount AP exceeds the upper limit and becomes 100%.

[0093] The second restriction degree may be, for example, 90%. That is, when the throttle opening is restricted by the second restriction degree, the throttle opening is restricted to 90% and its upper limit is 10%. That is, the control device 17 sets an upper limit value for the throttle opening, and even when the accelerator depression amount AP exceeds the upper limit value and becomes 100%, the control device 17 controls the drive device 6 by restricting the throttle opening to 10%. Alternatively, the second restriction degree may be set to 100%. In this case, the accelerator depression amount AP is ignored. In other words, the driver's driving operation is prohibited.

[0094] In this way, the control device 17 sets an upper limit on the driver's operation amount, and when the operation amount exceeds the upper limit, the control device 17 does not use the operation amount, thereby limiting the control amount based on the driver's operation. Then, when the second abnormal state is present, the control device 17 sets a lower upper limit on the operation amount than when the first abnormal state is present. As a result, when the second abnormal state is present, in which the driver is likely to be unable to drive, the upper limit on the operation amount is set lower than when the first abnormal state is present, and traveling of the vehicle 1 based on unintended driving operations by the driver is appropriately suppressed.

[0095] A driving operation performed after the driver's head has framed out of the predetermined area 37 is likely not an intended driving operation by the driver. In this embodiment, when the driver starts a driving operation after the driver's head has framed out of the predetermined area 37 (ST32: No), the control device 17 limits the amount of control based on the driver's driving operation (ST36, ST37). This appropriately prevents the vehicle 1 from traveling based on a driving operation unintended by the driver.

[0096] A driving operation that has been continuing since before the driver's head has framed out of the predetermined area 37 is likely to be a driving operation intended by the driver. In this embodiment, when the driver's driving operation has been continuing since before the driver's head has framed out of the predetermined area 37 (ST32: Yes), the control device 17 does not limit the control amount based on the driver's driving operation (ST33). This allows the driving operation intended by the driver to be appropriately reflected in the running of the vehicle 1.

[0097] Fig. 8 is a flow diagram of the second abnormality control in Fig. 5. In the second abnormality control, the control device 17 determines whether or not the driver is operating the accelerator (step ST41). If the determination in step ST41 is that the accelerator is being operated (Yes), the control device 17 determines whether or not this accelerator operation is an operation that has been continued from a normal state (step ST42).

[0098] If the determination in step ST41 is that there is no accelerator operation (No), or if the determination in step ST42 is that the accelerator operation is an operation that has been continued from the normal state (Yes), the control device 17 proceeds to step ST43. In step ST43, the control device 17 controls the drive device 6 according to the accelerator depression amount AP, or controls the drive device 6 with the second limiting degree (step ST43), and repeats the above process. The former control in step ST43 is a control in which the accelerator operation that has been continued from the normal state is considered to be correct, even though the accelerator operation has been framed out in the vertical direction. The latter control in step ST43 is a control in which the accelerator operation that has been continued from the normal state is considered to be incorrect because the accelerator operation has been framed out in the vertical direction.

[0099] If it is determined in step ST42 that the accelerator operation is not a continued operation from a normal state (No), the control device 17 executes notification control (step ST44). Next, the control device 17 determines whether or not it recognizes that the steering wheel is being gripped (step ST45). If it is determined in step ST45 that it recognizes that the steering wheel is being gripped (Yes), the control device 17 controls the drive device 6 with the second restriction degree (step ST46) and repeats the above process. On the other hand, if it is determined in step ST45 that it does not recognize that the steering wheel is being gripped (No), the control device 17 controls the drive device 6 with a restriction degree greater than the second restriction degree (step ST47) and repeats the above process.

[0100] In this way, in the second abnormality control performed when the driver is in the second abnormal state, the control device 17 limits the control based on the driver's driving operation to a greater degree than in the first abnormality control (ST46, ST47), thereby suppressing the vehicle 1 from traveling based on driving operations unintended by the driver.

[0101] Here, there is a high possibility that the driving operation after the driver's head has framed out of the predetermined area 37 is not the driving operation intended by the driver. Therefore, in the second abnormality control, if the driver starts a driving operation after the driver's head has framed out of the predetermined area 37 (ST42: No), the control device 17 restricts the control based on the driver's driving operation to a greater degree than in the first abnormality control (ST46, ST47). This appropriately suppresses the vehicle 1 from traveling based on a driving operation unintended by the driver.

[0102] Furthermore, in steps ST43, ST46, and ST47, the control device 17 may prohibit control based on the driving operation of the driver, instead of controlling the drive device 6 with the second restriction degree or a restriction degree greater than the second restriction degree. This appropriately prevents the vehicle 1 from traveling based on a driving operation unintended by the driver.

[0103] However, if the restriction on the control amount is released when the driver's head enters the frame, the behavior of the vehicle 1 changes suddenly. As shown in FIG. 6, in this embodiment, if the driver's head goes out of the frame from the predetermined area 37 and then enters the frame again while the driver is performing a driving operation, the determination in step ST25 is No. In this case, the control device 17 continues to restrict the control amount based on the driver's driving operation that has been ongoing since before the head entered the frame into the predetermined area 37 (ST29). This suppresses a sudden change in the behavior of the vehicle 1 after the head enters the frame.

[0104] The driver may move his / her head when not performing a driving operation. In this embodiment, as shown in FIGS. 7 and 8, if the driver starts a driving operation after the driver's head has framed out of the predetermined area 37, the determinations in steps ST32 and ST42 are both "No." In this case, the control device 17 executes notification control for the notification device 16 (ST34, ST44) to notify the occupants of the vehicle 1. This prevents the notification device 16 from making unnecessary notifications when the driver is not performing a driving operation, and also notifies the occupants when there is a possibility that the driver is unable to drive.

[0105] 6, when the driver's head frame-out from the predetermined area 37 and then frame-in to the predetermined area 37 (ST21: Yes), the control device 17 ends the notification control (ST22). As a result, when the driver's head frame-in, it is determined that the driver is ready to drive, and unnecessary continuation of notification by the notification device 16 is prevented.

[0106] When the driver is not gripping the steering wheel, the driving operation is likely not the driver's intended driving operation. In the first abnormality control of Fig. 6 and the second abnormality control of Fig. 7, when the control device 17 recognizes that the driver is not gripping the wheel (ST35-ST45: No), it limits the control amount based on the driver's driving operation to a greater degree than when it recognizes that the driver is gripping the wheel (ST37, ST47). This appropriately suppresses vehicle 1 from traveling based on driving operations unintended by the driver.

[0107] Although the description of the specific embodiment has been completed above, the present invention is not limited to the above-described embodiment and modifications, and can be implemented in a wide variety of modifications. For example, in the above-described embodiment, the control amount of the drive unit 6 is limited as an example of a traveling device assisted by the traveling assistance device 3, but the control amount of the brake unit 7 or the steering unit 8 may also be limited.

[0108] 7, if the accelerator operation is a continuation of the normal state (Yes), the control device 17 may perform the same determination process as in step ST35, i.e., a process of determining whether or not it is recognized that the steering wheel is being gripped. Similarly, if the accelerator operation is a continuation of the normal state (Yes) in the determination of step ST42 of FIG. 8, the control device 17 may perform the same determination process as in step ST45, i.e., a process of determining whether or not it is recognized that the steering wheel is being gripped.

[0109] On the other hand, not all of the components shown in the above embodiment are necessarily required, and they can be selected appropriately as long as they do not deviate from the spirit of the present invention. [Explanation of symbols]

[0110] 1: Vehicle 3: Driving support device 6: Drive unit (travel unit) 7: Brake device (running device) 8: Steering device (running device) 12: Driving controls 13: Driving operation sensor 15: In-vehicle sensor (status information acquisition unit) 16: Alarm device 17: Control device 24: In-car camera (status information acquisition unit) 25: Grasping sensor (status information acquisition unit) 26: Display (alarm device) 27: Speaker (alarm device) 37: Predetermined area AP: Accelerator pedal pressure (driving operation amount)

Claims

1. A vehicle driving assistance device, a state information acquisition unit that acquires state information of the driver; a traveling device for traveling the vehicle; a control device that controls the traveling device based on a driving operation of the driver and controls the traveling device to assist the driving operation based on the state information of the driver, The control device detects the position of the driver's head or the direction of the driver's face based on the state information, and when the position of the head moves or the direction of the face changes, limits the amount of control based on the driving operation for the traveling device; The control device limits the control amount with a first restriction degree when the head moves in a vehicle width direction or the face is deflected in the vehicle width direction, and limits the control amount with a second restriction degree greater than the first restriction degree when the head moves in a vertical direction or the face is deflected in the vertical direction.

2. The control device determines whether the head of the driver is in a predetermined area that is set in advance as an area where the head should be while driving, based on the state information, and limits the control amount when the head is framed out of the predetermined area; 2. The driving assistance device according to claim 1, wherein the first state includes a state in which the driver's head is framed out of the predetermined area in the vehicle width direction, and the second state includes a state in which the driver's head is framed out of the predetermined area in the up-down direction.

3. the control device detects a face of the driver based on the state information, detects a plurality of characteristic features on the face, calculates a degree of deflection of the orientation of the face relative to a forward direction based on at least positions of the characteristic features, and limits the control amount when the degree of deflection exceeds a predetermined threshold; 3. The driving assistance device according to claim 1, wherein the first state includes a state in which the degree of deflection is greater than the threshold value due to deformation or movement of at least one of the face and the characteristic feature of the driver in the vehicle width direction, and the second state includes a state in which the degree of deflection is greater than the threshold value due to movement of at least one of the face and the characteristic feature of the driver in the up-down direction.

4. the control device sets an upper limit value of a driving operation amount of the driver, and when the driving operation amount exceeds the upper limit value, does not use the driving operation amount, thereby limiting the control amount based on the driving operation of the driver; 4. The driving assistance device according to claim 1, wherein, in the second state, the control device sets the upper limit value to be smaller than in the first state.

5. 3. The driving assistance device according to claim 2, wherein, when the driver starts the driving operation after the head of the driver has moved out of the predetermined area, the control device limits the control amount based on the driving operation of the driver.

6. 6. The driving assistance device according to claim 5, wherein, when the driving operation of the driver is continued before the head of the driver frames out of the predetermined area, the control device does not limit the control amount based on the driving operation of the driver.

7. 3. The driving assistance device according to claim 2, wherein, when the head of the driver frames out of the predetermined area and then frames back into the predetermined area while the driver is performing the driving operation, the control device continues to limit the control amount based on the driving operation of the driver, which has been continuing since before the head frames back into the predetermined area.

8. The vehicle further includes an alarm device for notifying an occupant of the vehicle.

3. The driving assistance device according to claim 2, wherein, when the driver starts the driving operation after the head of the driver has framed out of the predetermined area, the control device executes notification control for the notification device to notify the occupant of the vehicle.

9. The driving assistance device according to claim 8 , wherein the control device terminates the notification control when the head of the driver comes into the predetermined area after being framed out of the predetermined area.

10. 10. The driving assistance device according to claim 1, wherein the control device prohibits the control based on the driving operation of the driver when the vehicle is in the second state.

11. 3. The driving assistance device according to claim 2, wherein, in the second state, when the driving operation of the driver is started after the head of the driver has framed out of the predetermined area, the control device prohibits the control based on the driving operation of the driver.

12. The control device recognizes whether the driver is gripping or not gripping the steering wheel based on the state information of the driver, 12. The driving assistance device according to claim 1, wherein when the control device recognizes the non-grasping in the first state or the second state, the control device limits the control amount based on the driving operation of the driver to a greater degree than when the control device recognizes the gripping.

Images