Traveling assistance device for vehicle

Abstract

The present application provides a travel assistance device for a vehicle, which can more accurately detect an inabiliy of a driver to drive and appropriately assist the travel of the vehicle. A control device (17) of the travel assistance device (3) detects a position of a head of the driver based on state information of the driver (ST1), and restricts a control amount based on a driving operation to a travel device (6) (ST10) in a case where a movement of the position of the head has occurred (ST3: No). The control device (17) restricts the control amount (ST10) by a first restriction degree in a first state where a movement of the head in a vehicle width direction has occurred (ST7: Yes, ST9), and restricts the control amount (ST12) by a second restriction degree that is greater than the first restriction degree in a second state where a movement of the head in a vertical direction has occurred (ST7: No, ST11).

Description

Technical Field

[0001] This invention relates to a vehicle driving assistance device. Background Technology

[0002] Previously, to prevent accidents caused by drivers being unable to drive due to sudden illness or other reasons, there are known devices for detecting a driver's inability to drive (Patent Document 1). This inability-to-drive detection device detects the head above the driver's neck by sequentially scanning an image of the driver's seat captured by a camera. If, during vehicle movement, the head deviates from a predetermined range within the image, the device detects that the driver is in an inability to drive. The inability-to-drive detection device distorts the predetermined range of the image used for detecting inability to drive based on the user's intention, including the driver's, or driver information related to the driver.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 6372388 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, even when the driver is in a normal driving state (hereinafter referred to as the normal state), there are cases where the driver's head moves due to changes in posture while driving. However, in the aforementioned prior art, although the driver's state is determined based on the presence or absence of the driver's head within a specified range, the method of switching vehicle control based on how the head moves out of the specified range is not considered, and there is room for improvement in the detection accuracy of the driver's inability to drive.

[0008] Furthermore, there are instances where drivers change their facial orientation due to sudden illness or other reasons that render them unable to drive. On the other hand, even when drivers are capable of driving, there are instances where they change their facial orientation while driving. Previously, there was no consideration given to switching vehicle controls based on changes in facial orientation in addressing these situations, and there is room for improvement in the accuracy of detecting driver incapacitation.

[0009] In view of the above background, the present invention aims to provide a vehicle driving assistance device that can detect the driver's inability to drive with higher accuracy and appropriately assist the vehicle's driving.

[0010] Methods for solving problems

[0011] To address the aforementioned issues, one aspect of the present invention is a vehicle driving assistance device 3, comprising: a status information acquisition unit 15 that acquires driver status information; a driving device 6 for driving the vehicle; and a control device 17 that controls the driving device based on the driver's driving operations and controls the driving device based on the driver's status information to assist the driving operations. The control device detects the position of the driver's head or the orientation of the driver's face (ST1, ST2) based on the status information, and detects changes in the position of the head or the orientation of the face (ST3) if the head position moves or the face orientation changes. No, ST4: No), the control device limits the control amount based on the driving operation for the driving device (ST10, ST12). In the first state (ST9) when the head moves in the vehicle width direction (ST7: Yes) or the face deflects in the vehicle width direction (ST8: Yes), the control device limits the control amount to a first limiting degree (ST10). In the second state (ST11) when the head moves in the vertical direction (ST7: No) or the face deflects in the vertical direction (ST8: No), the control device limits the control amount to a second limiting degree that is greater than the first limiting degree.

[0012] According to this method, by recognizing a first state where the head moves or the face deflects in the vehicle width direction, and a second state where the head moves or the face deflects in the vertical direction, the driver's inability to drive can be detected with higher precision. Furthermore, when the driver is in the second state, where the probability of being in a driving incapacitated is high, the control device controls the control quantity with a second restriction level that is greater than the first restriction level when the probability of the driver being in a driving incapacitated is low. Therefore, it is possible to suppress vehicle movement caused by unintentional driving operations by the driver, such as accidental pressing of the accelerator pedal, and to appropriately assist vehicle driving.

[0013] In the above-described manner, the control device determines, based on the state information, whether the driver's head is located in a predetermined area 37, which is the area that should be in while driving. If the head moves out of the predetermined area (ST3: No), the control device limits the control amount. The first state (ST9) includes the state in which the driver's head moves out of the predetermined area in the vehicle width direction (ST7: Yes), and the second state (ST11) includes the state in which the driver's head moves out of the predetermined area in the vertical direction (ST7: No).

[0014] This method allows for appropriate assistance in driving the vehicle based on the driver's condition.

[0015] In the above-described manner, the control device detects the driver's face and multiple features in the face based on the state information, and calculates the degree of deflection of the face relative to the front based at least on the position of the features. If the degree of deflection exceeds a predetermined threshold (ST14: No), the control device limits the control amount. The first state (ST9) includes a state where the degree of deflection is greater than the threshold due to deformation or movement of at least one of the driver's face and features in the vehicle width direction (ST8: Yes). The second state (ST11) includes a state where the degree of deflection is greater than the threshold due to movement of at least one of the driver's face and features in the vertical direction (ST8: No).

[0016] This method allows for appropriate assistance in driving the vehicle based on the driver's condition.

[0017] In the above-described manner, the control device sets an upper limit on the amount of driving operation (accelerator pedal depressor) of the driver, and does not use the driving operation if the amount of driving operation exceeds the upper limit, thereby limiting the amount of control based on the driver's driving operation. In the second state, the control device sets the upper limit to be smaller than in the first state.

[0018] According to this method, in the second state where the driver is highly likely to be unable to drive, by setting the upper limit value to be smaller than that in the first state, it is possible to appropriately suppress the movement of the vehicle based on the driver's unintentional driving operation.

[0019] In the above manner, if the driver's driving operation begins after the driver's head has moved out of the designated area (ST32: No), the control device limits the amount of control based on the driver's driving operation (ST36, ST37).

[0020] According to this method, it is possible to appropriately suppress the movement of vehicles based on unintentional driving operations by the driver.

[0021] In the above manner, if the driver's driving operation continues from before the driver's head moves out of the designated area (ST32: Yes), the control device does not limit the control amount based on the driver's driving operation (ST33).

[0022] This method allows the driver's intentional driving actions to be appropriately reflected in the vehicle's operation.

[0023] In the above manner, if the driver's head moves into the designated area after the driver's head has moved out of the designated area while the driver is performing the driving operation (ST25: No), the control device continues to limit the control amount based on the driver's driving operation that has been continuously performed since before the head moved into the designated area (ST29).

[0024] This method can suppress abrupt changes in vehicle behavior after head movement.

[0025] In the above-described manner, a notification device 16 for notifying the passengers of the vehicle is also provided. When the driver begins driving operations after the driver's head has moved out of the designated area (ST32: No, ST42: No), the control device executes notification control (ST34, ST44) for the notification device to notify the passengers of the vehicle.

[0026] According to this method, it is possible to prevent the notification device from making unnecessary notifications when no driving operation is being performed, and to notify passengers through the notification device when the driver may be in a state of inability to drive.

[0027] In the above manner, if the driver's head moves out of the designated area and then back into the designated area (ST21: Yes), the control device terminates the notification control (ST22).

[0028] According to this method, if the driver's head is moved in, the driver is considered to be in a driving state, which can prevent the notification device from continuing to make unnecessary notifications.

[0029] In the above-described manner, in the second state, the control device disables control based on the driver's driving operations (ST43, ST46, ST47).

[0030] This method can prevent vehicles from moving due to unintentional driving operations by the driver.

[0031] In the above-described manner, in the second state, if the driver's driving operation begins after the driver's head has moved out of the designated area (ST42: No), the control device prohibits control based on the driver's driving operation (ST46, ST47).

[0032] According to this method, it is possible to properly prevent the movement of vehicles based on unintentional driving operations by the driver.

[0033] In the above-described manner, the control device identifies whether the driver is holding or not holding the steering wheel based on the driver's state information. If the driver is not holding the steering wheel in the first state (ST9) or the second state (ST11) (ST35: No, ST45: No), the control device limits the amount of control based on the driver's driving operation (ST37, ST47) to a greater degree than if the driver is holding the steering wheel (ST35: Yes, ST45: Yes).

[0034] According to this method, it is possible to appropriately suppress the movement of vehicles based on unintentional driving operations by the driver.

[0035] The effects of the invention

[0036] Based on the above method, a vehicle driving assistance device can be provided, which can detect the driver's inability to drive with higher accuracy and appropriately assist the vehicle's driving. Attached Figure Description

[0037] Figure 1 This is a functional structural diagram illustrating a driving assistance device according to one embodiment of the present invention.

[0038] Figure 2 This is a diagram showing an image from an in-vehicle camera.

[0039] Figure 3 This is an illustration of the changes in head position in images from an in-vehicle camera.

[0040] Figure 4 This is an illustration of the changes in facial orientation in images from an in-car camera.

[0041] Figure 5 This is a flowchart of driving assistance control based on the control device.

[0042] Figure 6 yes Figure 5 The flowchart for normal control.

[0043] Figure 7 yes Figure 5 The flowchart for the first abnormal event control.

[0044] Figure 8 yes Figure 5 The flowchart for the second abnormal event control.

[0045] Label Explanation

[0046] 1: Vehicle

[0047] 3: Driving assistance devices

[0048] 6: Drive unit (travel unit)

[0049] 7: Braking system (driving system)

[0050] 8: Steering system (driving system)

[0051] 12: Driving controls

[0052] 13: Driving Operation Sensors

[0053] 15: In-vehicle sensors (status information acquisition unit)

[0054] 16: Notification device

[0055] 17: Control device

[0056] 24: In-vehicle camera (status information acquisition unit)

[0057] 25: Holding sensor (status information acquisition unit)

[0058] 26: Display (Notification Device)

[0059] 27: Speaker (Notification Device)

[0060] 37: Specified Area

[0061] AP: Accelerator pedal input (driving operation amount) Detailed Implementation

[0062] First, refer to Figure 1 The structure of the driving assistance device 3 of a vehicle 1 according to one embodiment of the present invention will be described.

[0063] A driving assistance device 3 is installed in vehicle 1. Vehicle 1 is, for example, a four-wheeled automobile. Vehicle 1 includes: a drive unit 6 (an example of a driving unit), a braking unit 7, a steering unit 8, an HMI (Human Machine Interface) 9, a navigation device 10, vehicle sensors 11, driving operation components 12, driving operation sensors 13 (an example of a driving operation information acquisition unit), external sensors 14, in-vehicle sensors 15 (an example of a status information acquisition unit), a notification device 16, and a control device 17.

[0064] The drive unit 6 is a device that provides driving force to the vehicle 1. For example, the drive unit 6 includes an internal combustion engine such as a gasoline engine or a diesel engine and / or an electric motor.

[0065] The braking device 7 is a device that applies braking force to the vehicle 1. For example, the braking device 7 includes a brake caliper that presses the brake pads against the brake disc and an electric cylinder that supplies hydraulic pressure to the brake caliper.

[0066] The steering device 8 is a device that changes the steering angle of the wheels. For example, the steering device 8 includes a rack and pinion mechanism that steers the wheels and an electric motor that drives the rack and pinion mechanism.

[0067] HMI9 is ​​a device that displays information to the driver and accepts input from the driver. HMI9 is ​​configured inside the vehicle 1 (e.g., in the dashboard). HMI9 includes a touch panel with a display screen.

[0068] Navigation device 10 is a device for guiding the route to the destination of vehicle 1. Navigation device 10 stores map information. Navigation device 10 determines the current position (latitude and longitude) of vehicle 1 based on GNSS signals received from artificial satellites (positioning satellites). Navigation device 10 sets a route to the destination of vehicle 1 based on the map information, the current position of vehicle 1, and the destination of vehicle 1 input by the driver into HMI 9.

[0069] Vehicle sensor 11 is a sensor that detects various vehicle states. For example, vehicle sensor 11 is preferably a vehicle speed sensor that detects the speed of vehicle 1, an acceleration sensor that detects the acceleration of vehicle 1, a yaw rate sensor that detects the angular velocity of vehicle 1 about its vertical axis, and an orientation sensor that detects the orientation of vehicle 1. Vehicle sensor 11 outputs the detection results to control device 17.

[0070] The driving control unit 12 is a device that receives driving operations from the driver on the vehicle 1. The driving control unit 12 includes: a steering wheel that receives steering operations from the driver, an accelerator pedal that receives acceleration operations from the driver, and a brake pedal that receives deceleration operations from the driver.

[0071] The driving operation sensor 13 is a sensor that detects the amount of driving operation performed on the driving control unit 12. In other words, the driving operation sensor 13 is a sensor that acquires information related to the driving operation performed on the driving control unit 12. The driving operation sensor 13 includes: a steering angle sensor that detects the rotation angle of the steering wheel, an accelerator sensor that detects the amount of accelerator pedal depressed, and a brake sensor that detects the amount of brake pedal depressed. The driving operation sensor 13 outputs the detection results to the control device 17.

[0072] External sensor 14 is a sensor that detects the state of the external environment surrounding vehicle 1. For example, external sensor 14 detects the relative position of objects existing around vehicle 1 with respect to vehicle 1. In other words, external sensor 14 acquires the position information of objects. Objects include other vehicles such as vehicles in front, pedestrians, bicycles, obstacles, etc. External sensor 14 outputs the detection results to control device 17.

[0073] The external sensor 14 includes multiple external cameras 21, multiple radars 22, and multiple lidar sensors 23. The external cameras 21 capture images of objects present around the vehicle 1. The radars 22 emit millimeter-wave or other radio waves around the vehicle 1 and capture the reflected waves, thereby detecting the relative position of objects around the vehicle 1 relative to the vehicle 1. The lidar sensors 23 illuminate infrared or other light around the vehicle 1 and capture the reflected light, thereby detecting the relative position of objects around the vehicle 1 relative to the vehicle 1.

[0074] The in-vehicle sensor 15 is a sensor that detects the state of the interior of the vehicle 1, specifically the state of the driver. In other words, the in-vehicle sensor 15 is a sensor that acquires driver state information. The in-vehicle sensor 15 includes an in-vehicle camera 24 and a grip sensor 25. The in-vehicle camera 24 captures images of the driver's seat, detects the position of the driver's head and face, and detects facial features such as the eyes, nose, and mouth. Additionally, the in-vehicle camera 24 can also detect the driver's arms and hands. The in-vehicle camera 24 is disposed inside the vehicle 1 (e.g., in front of the steering wheel). The grip sensor 25 is disposed on the steering wheel and detects whether the driver is gripping or not gripping the steering wheel. The in-vehicle camera 24 and the grip sensor 25 output the captured results to the control device 17.

[0075] The notification device 16 is a device for notifying passengers of vehicle 1, including the driver. The notification device 16 includes a display 26 and a speaker 27. The display 26 is, for example, a HUD (Head-Up Display) positioned in a location easily accessible to the driver and passengers, visually conveying information to them. The speaker 27 can be an audio device mounted on vehicle 1, or it can be shared with the navigation device 10, etc. Alternatively, the speaker 27 can be configured specifically for notifications. The speaker 27 is a notification device that conveys information to passengers audibly.

[0076] The control device 17 is an electronic control unit (ECU) configured as a computer to perform various processes. The control device 17 includes a processing unit (CPU, MPU, etc.) and a storage unit (ROM, RAM, etc.). The processing unit reads necessary software from the storage unit and executes prescribed processing according to the read software. The control device 17 can be configured as a single hardware unit or as a unit composed of multiple hardware components. The control device 17 connects to various structural elements of the vehicle 1 via a communication network such as CAN (Controller Area Network) to control these structural elements.

[0077] The control device 17 includes an external identification unit 31, a driving control unit 32, a driver status identification unit 33, a grip identification unit 34, and a notification control unit 35 as functional units. At least a portion of each functional unit of the control device 17 can be implemented by hardware such as LSI, ASIC, or FPGA, or by a combination of software and hardware.

[0078] The external recognition unit 31 identifies the external state of the vehicle 1 based on the detection results of the external sensor 14. For example, the external recognition unit 31 identifies objects existing around the vehicle 1 based on the detection results of the external sensor 14, and identifies the relative position of the object with respect to the vehicle 1, the relative speed of the object with respect to the vehicle 1, the distance from the vehicle 1 to the object, etc.

[0079] The external identification unit 31 obtains the object's category, position (absolute position), speed, past direction of movement, and surrounding environment based on the detection results of the external sensor 14. Furthermore, the external identification unit 31 can also obtain this information based on the detection results of the vehicle sensor 11 and GNSS signals, not just the detection results of the external sensor 14. The object's category includes other vehicles such as vehicles in front, pedestrians, bicycles, etc. The object's surrounding environment includes the shape and width of the road around the object. If the object is a pedestrian, the surrounding environment includes other pedestrians on the opposite side of the road. If the object is a vehicle in front, the surrounding environment includes traffic lights in front of the vehicle and vehicles in front of the vehicle.

[0080] The external identification unit 31 estimates the future direction of movement of the object based on at least one of the object's category, position, speed, past direction of movement, and surrounding environment. Therefore, the external identification unit 31 can estimate the future direction of movement of the object with high accuracy. For example, the external identification unit 31 calculates a probability distribution (Gaussian distribution) centered on the object and with direction as the probability variable, based on the object's category, position, speed, past direction of movement, and surrounding environment. Based on this, the external identification unit 31 can estimate the direction with the highest probability density in the above probability distribution as the future direction of movement of the object.

[0081] The driving control unit 32 controls the drive unit 6, braking system 7, and steering system 8 required for the driving of the vehicle 1 based on information related to driving operations for the driving operation components 12 detected by the driving operation sensor 13. For example, the driving control unit 32 controls the control quantities of the drive unit 6 (throttle opening, fuel supply to the internal combustion engine, power supply to the electric motor, etc.) based on the detection results of the throttle sensor (accelerator pedal depressure). The driving control unit 32 controls the hydraulic pressure of the braking system 7 based on the detection results of the brake sensor (brake pedal depressure). The driving control unit 32 controls the electric motor of the steering system 8 based on the detection results of the steering angle sensor (steering wheel rotation angle).

[0082] In addition, the driving control unit 32 performs driving assistance control on driving devices such as the drive unit 6, braking unit 7, and steering unit 8 to assist driving operations. For example, the driving control unit 32 performs collision suppression control on the vehicle 1 based on the external environment state of the vehicle 1 as identified by the external environment recognition unit 31. For example, the driving control unit 32 performs acceleration / deceleration control and steering control on the vehicle 1 based on the relative position of an object to the vehicle 1 as identified by the external environment recognition unit 31.

[0083] As part of the acceleration and deceleration control of vehicle 1, the driving control unit 32 performs follow-the-leader control such as ACC (Adaptive Cruise Control). In follow-the-leader control, the driving control unit 32 controls the drive unit 6 and the braking unit 7 to maintain the distance between vehicle 1 and the preceding vehicle within a specified range.

[0084] As part of the acceleration and deceleration control of vehicle 1, the driving control unit 32 performs accidental depressing suppression control. In this accidental depressing suppression control, the control quantities of the driving devices such as the drive unit 6 and the braking unit 7 based on the driver's driving operation (detection results of the driving operation sensor 13) are limited. The driving control unit 32 performs accidental depressing suppression control based on the recognition results of the driver state recognition unit 33, that is, based on the driver's state information obtained by the in-vehicle camera 24 and the grip sensor 25.

[0085] As a steering control mechanism for vehicle 1, the driving control unit 32 performs lane keeping control such as LKAS (Lane Keeping Assist System). In lane keeping control, the driving control unit 32 controls the steering device 8 to keep vehicle 1 within a reference position (e.g., near the center in the width direction of the lane) within the lane defined by lane markings. Additionally, the driving control unit 32 can also limit the steering of the steering device 8 based on driver status information obtained from the in-vehicle camera 24 and the grip sensor 25, according to the driver's steering input.

[0086] The driver state recognition unit 33 identifies the position of the driver's head and the driver's face and features based on the images captured by the in-vehicle camera 24. For example, the driver state recognition unit 33 extracts the positions of the driver's left and right eyes and mouth from the images of the driver's face captured by the in-vehicle camera 24, and identifies the orientation of the driver's face based on their positional relationship.

[0087] The grip recognition unit 34 identifies whether the driver is gripping or not gripping the steering wheel based on the driver's state information detected by the in-vehicle sensor 15. For example, the grip recognition unit 34 identifies gripping or not gripping based on information about the steering wheel detected by the grip sensor 25. Alternatively, the grip recognition unit 34 may identify gripping or not gripping based on the image captured by the in-vehicle camera 24, or it may combine this information to identify gripping or not gripping.

[0088] The notification control unit 35 controls the notification device 16. More specifically, the notification control unit 35 performs notification control on the notification device 16 based on the recognition results of the external recognition unit 31, the execution status of the driving assistance control of the driving control unit 32, the recognition results of the driver status recognition unit 33, etc. For example, when the driver status recognition unit 33 recognizes that the driver is in an incapacitated state (hereinafter also referred to as an abnormal state), the notification control unit 35 activates the display 26 and the speaker 27 to notify the passengers visually and audibly.

[0089] For ease of explanation, the various functional parts of the control device 17 will not be distinguished and will be referred to simply as "control device 17".

[0090] Next, refer to Figures 2-4 The driver's status identification performed by the control device 17 will be explained.

[0091] Figure 2This diagram shows an image from the in-vehicle camera 24. The control device 17 defines a designated area 37 within the image of the driver's seat captured by the in-vehicle camera 24 as the area where the driver's head should be positioned while driving. In this embodiment, the control device 17 defines the designated area 37 as a long rectangle. In other embodiments, the designated area 37 may be defined in a shape other than a rectangle. It is preferable to appropriately define the position and size of the designated area 37 to match the position and size of the driver's head seated in the driver's seat. Alternatively, the shape of the designated area 37 may be defined to match the shape of the driver's head.

[0092] The control device 17 analyzes the images from the in-vehicle camera 24 and detects the driver's head sequentially. Specifically, the control device 17 extracts the edge of the driver's face outline from the image from the driver's seat, and detects the rectangular area bounded by vertical lines drawn through the left and right ends of the extracted edge and horizontal lines drawn through the top and bottom ends of the edge as the head (face). The specified area 37 is set to a size relative to the head, with a specified margin added horizontally and vertically. When the driver is in a normal state, the driver's face is located approximately in the center of the specified area 37. The control device 17 obtains the trajectory of the driver's head based on the sequentially detected position of the driver's head.

[0093] In addition, the control device 17 detects facial features such as the eyes, nose, and mouth from the driver's face. The control device 17 detects the orientation of the face based on the contour and position of these features. The orientation of the face is calculated as the direction and degree (angle) of deflection relative to the forward direction the driver's face should be facing while driving. The control device 17 obtains the changes in the orientation of the driver's face based on the successively detected orientations of the driver's face.

[0094] Figure 3 This is an illustration of the changes in head position in the images from the in-vehicle camera 24. (The last sentence appears to be incomplete and possibly refers to a different topic.) Figure 2 A driver in a normal state, for example, in the event of a sudden illness, such as Figure 3 As shown in (A1), there are many cases where the head is tilted significantly forward. In this case, as... Figure 3 As shown in (A2), the control device 17 detects that the driver's head has moved downward from the designated area 37.

[0095] On the other hand, such as Figure 3 As shown in (B1), for example, when visually confirming the white line through the side window, or when visually confirming the rear of vehicle 1, the driver sometimes moves their head left or right while driving. In this situation, such as Figure 3 As shown in (B2), the control device 17 detects that the driver's head has moved out of the designated area 37 in the vehicle width direction.

[0096] Thus, when the driver's head moves out of the designated area 37, the control device 17 identifies whether it has moved out in the width direction or in the vertical direction.

[0097] Figure 4 This is an illustration of the changes in facial orientation in the images from the in-vehicle camera 24. (The last phrase appears to be incomplete and possibly refers to a different context.) Figure 4 (A) The driver in a normal state, for example, in order to visually confirm the side mirrors and other vehicles diagonally behind, such as Figure 4 As shown in (B), sometimes the face is turned towards the vehicle width direction (left or right). In this case, the driver's face is deformed in a thinning manner in the vehicle width direction, and the features are deformed or moved in the vehicle width direction, thereby the control device 17 detects that the driver's face is turned towards the vehicle width direction. More specifically, for example, if one of the left or right eyes moves in the vehicle width direction and the distance between the left and right eyes shortens, a change in the orientation of the face in the vehicle width direction is detected.

[0098] On the other hand, when the driver experiences a sudden illness, from Figure 4 The normal state of (A) is as follows Figure 4 As shown in (C), there are many cases where the head is tilted significantly forward. In this case, the driver's face is deformed in a thinning manner in the vertical direction, and the features are deformed or moved in the vertical direction, thereby the control device 17 detects that the driver's face is facing vertically. More specifically, for example, the left and right eyes move downward and the distance between the left and right eyes and the mouth shortens, thereby detecting that the face is facing downward.

[0099] Additionally, the control device 17 can also determine facial orientation based on the absence of detected feature points. For example, if the right eye is no longer detected, the control device 17 detects that the driver's face is facing to the right. Furthermore, if the mouth is no longer detected, the control device 17 detects that the driver's face is facing downwards.

[0100] Thus, when the driver's face changes direction, the control device 17 can identify whether the deflection occurs in the width direction or the vertical direction.

[0101] The control device 17 performs the various driving assistance controls illustrated above based on the driver's state information obtained in this manner from the image of the in-vehicle camera 24.

[0102] Next, refer to Figures 5-8 The driving assistance control of vehicle 1 performed by control device 17 will be explained. Here, the accidental depressor suppression control in the driving assistance control performed by control device 17 will be used as an example.

[0103] Figure 5 This is a flowchart of the driving assistance control performed by the control device 17. When driving assistance control is initiated, the control device 17 detects the position of the driver's head from the image of the in-vehicle camera 24 (step ST1). Next, the control device 17 detects the orientation of the driver's face from the image of the in-vehicle camera 24 (step ST2).

[0104] Next, the control device 17 determines whether the driver's head is within the designated area 37 (step ST3). For example, if the entire driver's face is within the designated area 37, the control device 17 determines that the head is within the designated area 37. On the other hand, if part of the driver's face moves out of the designated area 37 or if the driver's face is not within the designated area 37, the control device 17 determines that the head is not within the designated area 37.

[0105] In step ST3, if the driver's head is within the designated area 37 (yes), the control device 17 determines whether the degree of facial deflection is below a designated threshold (step ST4). If, in step ST4, the degree of facial deflection is below the designated threshold (yes), the control device 17 identifies the driver as being in a normal driving condition (step ST5), executes normal operation control (step ST6), and repeats the above process. Normal operation control will be explained later.

[0106] On the other hand, in step ST3, if the driver's head is not within the designated area 37 (No), the control device 17 determines whether the driver's head has moved out of the designated area 37 to the left or right (Step ST7). If in step ST7 the driver's head has moved out of the designated area 37 to the left or right (Yes), the control device 17 proceeds to step ST9. If in step ST7 the driver's head has moved out of the designated area 37 vertically (No), the control device 17 proceeds to step ST11.

[0107] Furthermore, in step ST4, if the degree of facial deflection of the driver exceeds a predetermined threshold (No), the control device 17 proceeds to step ST8. In step ST8, the control device 17 determines whether the driver's face has deflected to the left or right, that is, whether the degree of deflection exceeds the threshold due to deformation or movement of the driver's face or features in the vehicle width direction. If, in step ST8, the driver's face has deflected to the left or right (Yes), the control device 17 proceeds to step ST9. If, in step ST8, the driver's face has deflected up or down (No), the control device 17 proceeds to step ST11.

[0108] In step ST9, the control device 17 identifies a first abnormal state, in which the driver is likely to be in a state of inability to drive, although the probability of such an inability is low. When the control device 17 identifies the driver as being in a first abnormal state in step ST9, it performs a first abnormality control (step ST10) that restricts the control amount of the drive device 6 based on the driver's driving operation to a first restriction level, and repeats the above process. The first abnormality control will be explained later.

[0109] In step ST11, the control device 17 identifies a second abnormal state, in which the driver is likely to be in a drivable state with a high probability of being unable to drive. When the driver is identified as being in a second abnormal state in step ST11, the control device 17 executes a second abnormality control (step ST12) that imposes a second restriction on the control amount of the drive device 6 based on the driver's driving operation at a second restriction level, and repeats the above process. The second restriction level is a greater restriction level than the first restriction level. The second abnormality control will be explained later.

[0110] Thus, the control device 17 identifies a first abnormal state where the driver's head moves to the left or right, or the driver's face is turned to the left or right, and a second abnormal state where the driver's head moves up or down, or the driver's face is turned up or down. Therefore, the control device 17 can detect the driver's inability to drive with higher precision. Furthermore, when the second abnormal state, where the driver is more likely to be in an incapacitated state, is in the first abnormal state, the control device 17 controls the control quantity with a second restriction level that is greater than the first restriction level when the driver is less likely to be in an incapacitated state. As a result, the movement of vehicle 1 caused by unintentional driving operations by the driver, such as accidental pressing of the accelerator pedal, is suppressed, and the movement of vehicle 1 is appropriately assisted.

[0111] During normal driving, the driver may sometimes move their head left or right, for example, to visually confirm the white line through the side window. However, when the head moves out of the designated area 37 in the vertical direction, the driver is more likely to be in a drivable state compared to when the head moves out of the designated area 37 in the vehicle width direction. In this embodiment, when the driver's head moves out of the designated area 37 in the vehicle width direction (ST7: Yes), the control device 17 identifies it as a first abnormal state in step ST9. Furthermore, when the driver's head moves out of the designated area 37 in the vertical direction (ST7: No), the control device 17 identifies it as a second abnormal state in step ST11. That is, when the driver's head moves out of the designated area 37 in the vertical direction, the control device 17 limits the control amount to a second level of restriction, which is greater than the first restriction level. Thus, the driving of the vehicle 1 is appropriately assisted according to the driver's state.

[0112] During normal driving, drivers sometimes turn their faces towards the vehicle width direction (left or right) to visually confirm the side mirrors or other vehicles diagonally behind. On the other hand, when the driver's face is turned vertically, the driver is more likely to be in a drivable state compared to when the face is turned horizontally. In this embodiment, if the degree of deflection is greater than a threshold due to deformation or movement of at least one of the driver's face or features in the vehicle width direction (ST8: Yes), the control device 17 identifies it as a first abnormal state in step ST9. Furthermore, if the degree of deflection is greater than a threshold due to movement of at least one of the driver's face or features in the vertical direction (ST8: No), the control device 17 identifies it as a second abnormal state (ST11) in step ST11. That is, when the driver's face is turned vertically, the control device 17 limits the control amount to a second limit degree, which is greater than the first limit degree. Thus, the driving of the vehicle 1 is appropriately assisted according to the driver's state.

[0113] Figure 6 yes Figure 5 The flowchart illustrates the normal operation control process. In normal operation control, control device 17 initially determines whether a notification is being sent to the passenger, i.e., whether the aforementioned notification control is being performed (step ST21). If, in step ST21, a notification is being sent (Yes), control device 17 terminates the notification control (step ST22), and the process proceeds to step ST23. If, in step ST21, no notification is being sent (No), control device 17 directly proceeds the process to step ST23.

[0114] In step ST23, the control device 17 determines whether there is a driver's accelerator operation (accelerator pedal depressing operation). If there is no accelerator operation in step ST23 (No), the control device 17 releases the restriction on the control amount of the drive unit 6 based on the driver's accelerator operation. On the other hand, if there is accelerator operation in step ST23 (Yes), the control device 17 determines whether the accelerator operation was started under normal conditions (step ST25).

[0115] In step ST25, if the throttle operation was initiated under normal conditions (yes), the control device 17 then determines whether the prescribed restriction release conditions are met (step ST26). Prescribed restriction release conditions may include, for example, operating the restriction release mechanism located at the driver's seat or repeatedly inputting the throttle operation. In step ST26, if the prescribed restriction release conditions are met (yes), the control device 17 proceeds to step ST24 to release the restriction.

[0116] If the specified restriction release condition is not met in step ST26 (No), and after step ST24, the control device 17 determines whether the restriction has been released (step ST27). If the restriction is released in step ST27 (Yes), the control device 17 controls the drive device 6 according to the throttle input (step ST28) and repeats the above process.

[0117] On the other hand, if the throttle operation is not started under normal conditions in step ST25 (No), and if the restriction is not lifted in step ST27 (No), the control device 17 proceeds to step ST29. In step ST29, since the restriction imposed in the first abnormality control in step ST10 or the second abnormality control in step ST12 is not lifted, the control device 17 continues to restrict and control the drive device 6 (step ST29), and repeats the above process.

[0118] Figure 7 yes Figure 5 The flowchart for the first abnormal situation control is as follows. In the first abnormal situation control, the control device 17 determines whether there is throttle operation by the driver (step ST31). In the determination of step ST31, if there is throttle operation (yes), the control device 17 determines whether the throttle operation is an operation that has been continuously performed since the normal state (step ST32).

[0119] If there is no throttle operation in step ST31 (No), and if the throttle operation in step ST32 is an operation that has been continuously performed since the normal state (Yes), the control device 17 proceeds to step ST33. In step ST33, the control device 17 controls the drive device 6 according to the amount of throttle depressed (step ST33) and repeats the above process.

[0120] In step ST32, if the throttle operation is not a continuous operation that has started from the normal state (No), the control device 17 performs notification control (Step ST34). Next, the control device 17 determines whether steering wheel grip is detected (Step ST35). In step ST35, if grip is detected (Yes), the control device 17 controls the drive unit 6 at a first restriction level (Step ST36). On the other hand, if grip is not detected in step ST35 (No), the control device 17 controls the drive unit 6 at a restriction level greater than the first restriction level (Step ST37).

[0121] The first restriction level could be, for example, 20%. That is, when restricted at the first restriction level, the throttle opening is limited by 20% when the maximum value is set to 100%, with an upper limit of 80%. Alternatively, the throttle opening can be limited by the control device 17 ignoring operations exceeding the upper limit. In either case, the control device 17 sets an upper limit for the throttle opening, and even when the accelerator pedal is depressed beyond the upper limit and reaches 100%, it limits the throttle opening to 80% to control the drive unit 6.

[0122] The second restriction level could be, for example, 90%. That is, when restricted to the second restriction level, the throttle opening is limited to 90%, with an upper limit of 10%. In other words, the control device 17 sets an upper limit for the throttle opening, for example, even if the accelerator pedal is depressed beyond the upper limit and reaches 100%, the throttle opening is limited to 10% to control the drive unit 6. Alternatively, the second restriction level could be set to 100%. In this case, the accelerator pedal depressor is ignored. In other words, the driver's driving operation is prohibited.

[0123] In this way, the control device 17 sets an upper limit for the driver's driving operation amount, and stops using the driving operation amount when it exceeds the upper limit, thereby limiting the control amount based on the driver's driving operation. Furthermore, in the second abnormal state, the control device 17 sets the upper limit for the driving operation amount to be smaller than in the first abnormal state. As a result, in the second abnormal state, where there is a high probability that the driver is in a state of inability to drive, the upper limit for the driving operation amount is set to be smaller than in the first abnormal state, and the movement of the vehicle 1 based on the driver's unintentional driving operation is appropriately suppressed.

[0124] There is a high probability that the driving operation performed after the driver's head has moved out of the designated area 37 is not an intentional driving operation. In this embodiment, if the driver begins driving operation after the driver's head has moved out of the designated area 37 (ST32: No), the control device 17 limits the control amount based on the driver's driving operation (ST36, ST37). As a result, the movement of the vehicle 1 based on unintentional driving operation by the driver is appropriately suppressed.

[0125] Driving operations that continue from before the driver's head leaves the designated area 37 are highly likely to be intentional driving operations by the driver. In this embodiment, if the driver's driving operations continue from before the driver's head leaves the designated area 37 (ST32: Yes), the control device 17 does not limit the control amount based on the driver's driving operations (ST33). Thus, the driver's intentional driving operations are appropriately reflected in the movement of the vehicle 1.

[0126] Figure 8 yes Figure 5 The flowchart for the second abnormal situation control is as follows. In the second abnormal situation control, the control device 17 determines whether there is throttle operation by the driver (step ST41). In the determination of step ST41, if there is throttle operation (yes), the control device 17 determines whether the throttle operation is an operation that has been continuously performed since the normal state (step ST42).

[0127] If there is no throttle operation in step ST41 (No), and if the throttle operation in step ST42 is an operation that has been continuously performed since the normal state (Yes), the control device 17 proceeds to step ST43. In step ST43, the control device 17 controls the drive device 6 according to the amount of throttle depressed, or controls the drive device 6 with a second degree of restriction (step ST43), and repeats the above process. The control in step ST43 for the former is as follows: although the throttle is moved out in the vertical direction, it is considered that the throttle operation that has been continuously performed since the normal state is normal operation. The control in step ST43 for the latter is as follows: since the throttle is moved out in the vertical direction, it is considered that the throttle operation that has been continuously performed since the normal state is abnormal operation.

[0128] In step ST42, if the throttle operation is not a continuous operation that has started from the normal state (No), the control device 17 performs notification control (step ST44). Next, the control device 17 determines whether steering wheel grip is detected (step ST45). In step ST45, if grip is detected (Yes), the control device 17 controls the drive unit 6 at the second restriction level (step ST46), and repeats the above process. On the other hand, if grip is not detected in step ST45 (No), the control device 17 controls the drive unit 6 at a restriction level greater than the second restriction level (step ST47), and repeats the above process.

[0129] Thus, in the second abnormal situation control when the driver is in the second abnormal state, the control device 17 restricts the control based on the driver's driving operation to a greater degree than the first abnormal situation control (ST46, ST47). As a result, the movement of the vehicle 1 based on the driver's unintentional driving operation is suppressed.

[0130] Here, the possibility that the driver's driving operation after the head has moved out of the designated area 37 is not an intentional driving operation is relatively high. Therefore, in the second abnormal situation control, if the driver begins to drive after the head has moved out of the designated area 37 (ST42: No), the control device 17 restricts the control based on the driver's driving operation to a greater extent than in the first abnormal situation control (ST46, ST47). As a result, the movement of vehicle 1 based on unintentional driving operation by the driver is appropriately suppressed.

[0131] Alternatively, the control device 17 can also disable control based on the driver's driving operations in steps ST43, ST46, and ST47, thereby controlling the drive device 6 with a second restriction level or a restriction level greater than the second restriction level. This appropriately prevents the vehicle 1 from moving based on unintentional driving operations by the driver.

[0132] Furthermore, if the control limit is released when the driver's head moves in, the behavior of vehicle 1 will change drastically. For example... Figure 6 As shown, in this embodiment, if the driver's head moves into the designated area 37 after the driver has moved out of the designated area 37 while the driver is performing driving operations, the determination in step ST25 is "No". In this case, the control device 17 continues to limit the control amount based on the driver's driving operations that have been ongoing since the head moved into the designated area 37 (ST29). This suppresses abrupt changes in the behavior of the vehicle 1 after the head moves in.

[0133] Drivers sometimes move their heads when not driving. In this embodiment, such as... Figure 7 and Figure 8 As shown, if the driver begins driving after their head has moved out of the designated area 37, the determinations in steps ST32 and ST42 are both "No". In this case, the control device 17 executes notification control (ST34, ST44) for the notification device 16 to notify the passengers of vehicle 1. This prevents unnecessary notifications from the notification device 16 when driving is not in progress, and allows the notification device 16 to notify the passengers when the driver may be unable to drive.

[0134] like Figure 6 As shown, if the driver's head moves out of the designated area 37 and then back into the designated area 37 (ST21: Yes), the control device 17 ends the notification control (ST22). Thus, when the driver's head moves into the designated area, it is considered that the driver is in a driving state, preventing the notification device 16 from continuously making unnecessary notifications.

[0135] Driving actions when the driver is not holding the steering wheel are more likely not intentional. Figure 6 First abnormality control and Figure 7 In the second abnormal situation control, when non-control is detected (ST35, ST45: No), the control device 17 limits the control amount based on the driver's driving operation to a greater degree than when control is detected (ST37, ST47). As a result, the movement of the vehicle 1 based on the driver's unintentional driving operation is appropriately suppressed.

[0136] The above description of specific embodiments concludes here. However, the present invention is not limited to the above embodiments or modifications, and can be widely modified. For example, in the above embodiments, as an example of a driving device assisted by the driving assistance device 3, the control amount of the drive device 6 is limited, but the control amount of the braking device 7 and the steering device 8 can also be limited.

[0137] In addition, Figure 7 In step ST32, if the throttle operation has been ongoing since the normal state, the control device 17 can also perform the same determination process as in step ST35, that is, it can also determine whether steering wheel grip is detected. Similarly, in Figure 8 In step ST42, if the throttle operation is an operation that has been continuously performed since the normal state, the control device 17 can also perform the same determination process as step ST45, that is, it can also determine whether the steering wheel is being held.

[0138] On the other hand, not all of the structural elements shown in the above embodiments are necessarily necessary. As long as they do not depart from the spirit of the present invention, they can be appropriately selected and omitted.

Claims

1. A driving assistance device, wherein the driving assistance device is a driving assistance device for a vehicle, The driving assistance device is characterized by having: The status information acquisition unit acquires the driver's status information; A driving device for moving the vehicle; as well as A control device that controls the driving device based on the driver's driving operations, and controls the driving device based on the driver's state information to assist the driving operations. The control device detects the position of the driver's head or the orientation of the driver's face based on the state information. If the position of the head moves or the orientation of the face changes, the control device limits the amount of control applied to the driving device based on the driving operation. In the first state, when the head moves in the vehicle width direction or the face deflects in the vehicle width direction, the control device limits the control amount to a first limiting degree. In the second state, when the head moves in the vertical direction or the face deflects in the vertical direction, the control device limits the control amount to a second limiting degree that is greater than the first limiting degree.

2. The driving assistance device according to claim 1, characterized in that, The control device determines, based on the status information, whether the driver's head is located in a pre-defined area that is the area that should be in while driving. If the head moves out of the defined area, the control device limits the control amount. The first state includes the state in which the driver's head has moved out of the designated area in the vehicle width direction, and the second state includes the state in which the driver's head has moved out of the designated area in the vertical direction.

3. The driving assistance device according to claim 1 or 2, wherein, The control device detects the driver's face and multiple features on the face based on the state information, and calculates the degree of facial deflection relative to forward orientation based at least on the position of the features. If the degree of deflection exceeds a predetermined threshold, the control device limits the control amount. The first state includes a state in which the degree of deflection is greater than the threshold due to deformation or movement of at least one of the driver's face and the feature portion in the vehicle width direction. The second state includes a state in which the degree of deflection is greater than the threshold due to movement of at least one of the driver's face and the feature portion in the vertical direction.

4. The driving assistance device according to claim 1 or 2, wherein, The control device sets an upper limit on the amount of driving operations the driver can perform. If the amount of driving operations exceeds the upper limit, the driving operations are not used, thereby limiting the control quantity based on the driver's driving operations. In the second state, the control device sets the upper limit value to be smaller than that in the first state.

5. The driving assistance device according to claim 2, wherein, When the driver begins driving after the driver's head has moved out of the designated area, the control device limits the amount of control based on the driver's driving operation.

6. The driving assistance device according to claim 5, wherein, When the driver's driving operation continues from before the driver's head is removed from the designated area, the control device does not limit the control amount based on the driver's driving operation.

7. The driving assistance device according to claim 2, wherein, After the driver's head moves out of the designated area, if the head moves into the designated area while the driver is performing the driving operation, the control device continues to limit the control amount based on the driver's driving operation that has been ongoing since the head moved into the designated area.

8. The driving assistance device according to claim 2, wherein, The driving assistance device also includes a notification device for notifying passengers of the vehicle. When the driver begins driving after the driver's head has moved out of the designated area, the control device performs notification control to notify the passengers of the vehicle.

9. The driving assistance device according to claim 8, wherein, If the driver's head moves out of the designated area and then back into the designated area, the control device terminates the notification control.

10. The driving assistance device according to claim 1 or 2, wherein, In the second state, the control device disables control based on the driver's driving operations.

11. The driving assistance device according to claim 2, wherein, In the second state, if the driver begins driving after the driver's head has moved out of the designated area, the control device prohibits control based on the driver's driving operation.

12. The driving assistance device according to claim 1 or 2, wherein, The control device identifies whether the driver is holding or not holding the steering wheel based on the driver's state information. When the non-grip is identified in the first state or the second state, the control device limits the amount of control based on the driver's driving operation to a greater degree than when the grip is identified.

Images