Driver state determination method and apparatus thereof

Abstract

To provide a driver state determination method and a device that can determine an abnormal state of a driver early with high accuracy by varying time to determine abnormality between a closed eye state and a non-closed eye state.SOLUTION: A driver abnormal state determination step S19 determines that a driver is in an abnormal state when a closed eye state is not detected in a closed eye determination step S12, if a head posture is an abnormal posture and a situation where a driving operation amount falls below a predetermined threshold corresponding to non-driving operation continues for a period corresponding to a first determination threshold for non-closed eye L1, and determines that the driver is in the abnormal state when the closed eye state is detected in the closed eye determination step S12, if the head posture is an abnormal posture with a smaller degree of abnormality than the above abnormal posture and the situation where the driving operation amount falls below the predetermined threshold corresponding to non-driving operation continues for a period corresponding to a first determination threshold for closed eye L1 shorter than the period corresponding to the first determination threshold for non-closed eye L1.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a driver state determination method and apparatus for detecting an abnormal state of a driver driving a vehicle.

Background Art

[0002] Conventionally, a driver support device is known that accurately grasps the conscious state of a driver, particularly the arousal state of the driver, and supports the driving operation on the vehicle according to this arousal state. In addition, there is also known a driver support device that, when the driver's physical condition suddenly changes and the driver cannot continue driving, i.e., in a so-called abnormal state, the vehicle automatically moves in the direction of the road shoulder and autonomously stops.

[0003] Normally, an abnormal state of a driver including the drowsiness level is determined through head-related information such as the head posture and face state of the driver acquired by an imaging means such as a camera. Among the head-related information, the head posture is detected, for example, through the face orientation and head position (three-dimensional coordinates) of the driver, and the face state is detected, for example, through the expression of the driver and the state of the eye opening degree (eyelid state).

[0004] The drowsiness detection device of Patent Document 1 includes an imaging means for imaging the face of the driver, an arousal degree estimation means for deriving the eye opening degree based on the captured image and estimating the arousal degree whose value is larger as the time when the opening degree is below a specified ratio is shorter, a gaze estimation means for estimating the gaze direction of the driver, and an arousal degree correction means for calculating a corrected arousal degree that cancels out the decrease in arousal degree due to downward gaze in the case of downward gaze where the gaze is directed below the gaze point to the fixation point, and a drowsiness detection means for comparing the corrected arousal degree with a predetermined threshold value to detect the drowsiness of the driver. And when the drowsiness of the driver is detected, an alarm is generated to eliminate the drowsiness of the driver.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The drowsiness detection device of Patent Document 1 can avoid misjudging that the driver is closing his eyes even though he is actually opening his eyes, and can limit unnecessary alarms. However, in the drowsiness detection device of Patent Document 1, even if drowsiness determination considering the driver's downward view can be executed, there is a possibility that the abnormal state of the driver cannot be accurately determined. For example, when the determination threshold for detecting the abnormal state of the driver is set low, in addition to the annoyance caused by frequent alarms, there is a possibility of causing an unexpected situation to other vehicles such as an emergency stop.

[0007] Therefore, in order to avoid misjudgment of the abnormal state, closing the eyes, which is a sign of the driver in an abnormal state, and the collapse of the head posture, which is the physical state of the driver in an abnormal state, are respectively detected, and when both of these two abnormal determination conditions are satisfied, it is conceivable to determine the abnormal state of the driver. However, since the closed-eye determination control and the head posture determination control are independently processed in parallel, when the determination time of the head posture collapse during closed eyes is set to the same time as the determination time of the head posture collapse during non-closed eyes, the determination time during closed eyes becomes unnecessarily long, which may cause a delay in emergency stop or emergency notification. That is, it is not easy to determine the abnormal state of the driver early and with high accuracy.

[0008] An object of the present invention is to provide a driver state determination method and its device etc. capable of determining the abnormal state of a driver early and with high accuracy.

Means for Solving the Problems

[0009] The driver state determination method according to claim 1 is a driver state determination method for detecting an abnormal state of a driver of a vehicle. It includes a head pose acquisition step of acquiring the face orientation and head position of the driver, a closed-eye determination step of detecting that the driver's eyes are in a closed state, a driving operation amount acquisition step of acquiring at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and a driver abnormal state determination step of determining whether the driver is in an abnormal state. The driver abnormal state determination step determines that the driver is in an abnormal state when, when the closed-eye state of the driver is not detected in the closed-eye determination step, the head pose of the driver acquired in the head pose acquisition step is an abnormal pose and the situation where the driving operation amount is below a predetermined threshold continues for a first predetermined time. When the closed-eye state of the driver is detected in the closed-eye determination step, the driver is determined to be in an abnormal state when the head pose of the driver acquired in the head pose acquisition step is an abnormal pose with a lower degree of abnormality than the abnormal pose and the situation where the driving operation amount is below a predetermined threshold continues for a second predetermined time shorter than the first predetermined time.

[0010] In this driver state determination method, since it includes a head pose acquisition step of acquiring the face orientation and head position of the driver, a closed-eye determination step of detecting that the driver's eyes are in a closed state, a driving operation amount acquisition step of acquiring at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and a driver abnormal state determination step of determining whether the driver is in an abnormal state, it is possible to use the head pose collapse, which is the physical state of the driver during an abnormality (hereinafter referred to as the abnormal state), the closed eyes, which are the symptoms of the driver during an abnormality (hereinafter referred to as the abnormal symptoms), and the non-driving operation, which is the action of the driver during an abnormality (hereinafter referred to as the abnormal action), as the determination conditions for the abnormal state of the driver. When the closed-eye state of the driver is not detected in the closed-eye determination step, the driver abnormal state determination step determines that the driver is in an abnormal state when the head pose of the driver acquired in the head pose acquisition step is an abnormal pose and the situation where the driving operation amount is below a predetermined threshold continues for a first predetermined time. Therefore, even in a non-closed-eye state, it is possible to accurately determine the abnormal state of the driver using the abnormal state and the abnormal action as the determination conditions. When the driver abnormal state determination step detects the driver's eyes-closed state in the eyes-closed determination step, if a situation where the driver's head posture obtained in the head posture acquisition step is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is below a predetermined threshold value continues for a second predetermined time shorter than the first predetermined time, it determines that the driver is in an abnormal state. Therefore, in the eyes-closed state, the abnormal state of the driver can be determined in a short time based on the abnormal symptom and the abnormal state.

[0011] The invention according to claim 2 is characterized in that, in the invention according to claim 1, the driver abnormal state determination step determines that the head posture is an abnormal posture based on at least the face orientation angle of the driver. According to this configuration, it is possible to easily determine that the driver's head posture is an abnormal posture.

[0012] The invention according to claim 3 is characterized in that, in the invention according to claim 1 or 2, the eyes-closed determination step is characterized in that it issues an alarm after a predetermined time has elapsed when the driver's eyes-closed state is detected. According to this configuration, the driver can be made aware of closing their eyes.

[0013] The driver state determination device according to claim 4 is a driver state determination device that detects an abnormal state of a driver of a vehicle, and includes a head posture acquisition unit that acquires the face orientation and head position of the driver, an eye closure determination unit that detects that the driver's eyes are in a closed state, a driving operation amount acquisition unit that acquires at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and a driver abnormal state determination unit that determines whether the driver is in an abnormal state. When the eye closure determination unit does not detect the closed state of the driver, the driver abnormal state determination unit determines that the driver is in an abnormal state when the head posture of the driver acquired by the head posture acquisition unit is an abnormal posture and the driving operation amount is below a predetermined threshold value and this situation continues for a first predetermined time. When the eye closure determination unit detects the closed state of the driver, the driver abnormal state determination unit determines that the driver is in an abnormal state when the head posture of the driver acquired by the head posture acquisition unit is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is below a predetermined threshold value and this situation continues for a second predetermined time shorter than the first predetermined time.

[0014] In this driver state determination device, since it has a head posture acquisition unit that acquires the face orientation and head position of the driver, an eye closure determination unit that detects that the driver's eyes are in a closed state, a driving operation amount acquisition unit that acquires at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and a driver abnormal state determination unit that determines whether the driver is in an abnormal state in a driver state determination device that detects an abnormal state of a driver of a vehicle, it is possible to use the head posture collapse in an abnormal state, the eye closure as an abnormal symptom, and the non-driving operation as an abnormal operation as determination conditions for the abnormal state of the driver. When the eye closure determination unit does not detect the closed state of the driver, the driver abnormal state determination unit determines that the driver is in an abnormal state when the head posture of the driver acquired by the head posture acquisition unit is an abnormal posture and the driving operation amount is below a predetermined threshold value and this situation continues for a first predetermined time. Therefore, even in a non-closed eye state, it is possible to accurately determine the abnormal state of the driver using the abnormal state and abnormal operation as determination conditions. When the driver abnormal state determination means detects that the eye-closure determination means has detected the driver's eye-closure state, if the situation where the head posture of the driver obtained by the head posture acquisition means is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is below a predetermined threshold value continues for a second predetermined time shorter than the first predetermined time, it is determined that the driver is in an abnormal state. Therefore, when the driver is in an eye-closed state, the abnormal state of the driver can be determined in a short time based on the abnormal symptom and the abnormal state at the time of abnormality.

Advantages of the Invention

[0015] According to the driver state determination method and its device of the present invention, by making the determination time of the abnormal state different between the eye-closed state and the non-eye-closed state, the abnormal state of the driver can be determined early and with high accuracy.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following description exemplifies the application of the present invention to a driver state determination device for a vehicle, and does not limit the present invention, its application, or its use.

Examples

[0018] Hereinafter, Example 1 of the present invention will be described with reference to FIGS. 1 to 6. The driver state determination device M is a driving support device that can determine the driver's conscious state, particularly an abnormal state in which it is difficult for the driver to perform driving operations on the vehicle (hereinafter also referred to as the host vehicle), and can autonomously perform an emergency stop when the driver's abnormal state is determined. Hereinafter, the abnormal state of the driver will be described as a state in which the driver is not consciously awake and the body posture is collapsed, making it impossible to perform driving operations (including disease onset and dozing). Also, the following description includes an explanation of the driver state determination method.

[0019] As shown in FIG. 1, the driver state determination device M mainly includes a plurality of detection means 1 capable of detecting the driver's operation behavior and the vehicle's running behavior, etc., a driver support device 2 that autonomously provides driving support for the driver from the vehicle side, and an ECU (Electronic Control Unit) 3 that performs various calculations based on the input signal input from the detection means 1 and outputs various command signals to the driver support device 2.

[0020] First, the detection means 1 will be described. As shown in FIG. 1, the detection means 1 includes a camera 11 for imaging the driver, a vehicle speed sensor 12 for detecting the vehicle speed, a steering angle sensor 13 for detecting the steering angle by operating the steering wheel (not shown), a grip sensor 14 for detecting the driver's grip on the steering wheel, an accelerator sensor 15 for detecting the depression amount of the accelerator pedal (not shown), a brake sensor 16 for detecting the depression amount of the brake pedal (not shown), a hazard switch 17 for operating the hazard lamp (not shown), etc. Each of the sensors 12 to 16 corresponds to a driving operation amount acquisition means for acquiring the current driving operation amount at a predetermined timing, that is, immediately before the so-called determination time.

[0021] The camera 11 is constituted by, for example, a CCD (Charge Coupled Device) camera and is mounted at a position facing the driver at the upper center of the instrument panel (not shown). This camera 11 is configured to be able to image the head including the face of the driver sitting in the driver's seat and wearing a seat belt from the front. The captured head image cuts out the image of the driver's face part, and is configured to be able to detect the direction of the face (face direction), identify the driver's expression, magnify and detect the iris of the eyes to detect the line-of-sight direction, magnify and detect the eyelid part to detect the opening degree and opening / closing state of the driver's eyes, and the three-dimensional coordinate position of the driver's head (center of gravity), etc. The camera 11 only needs to be able to acquire necessary captured images, and may be equipped with two cameras, namely a front-facing camera and an upper camera installed directly above the driver's seat, or three or more cameras.

[0022] Next, the driver assistance device 2 will be described. As shown in FIG. 1, the driver assistance device 2 is composed of a monitoring support device 21 that notifies the driver of a warning state, a braking support device 22 that causes the vehicle to perform a braking operation, a parking support device 23 that causes the vehicle to perform an emergency stop operation, etc.

[0023] The monitoring support device 21 is configured to prompt the driver to regain consciousness by notifying the driver when it recognizes the conscious state, that is, the so-called awakening state of the driver and determines that recovery is necessary. When this monitoring support device 21 receives a sidelong glance signal or a drowsiness estimation signal from the ECU 3, it generates an alarm for the driver from one or more speakers (not shown) provided in the vehicle interior. As shown in FIG. 2, the speed meter unit 21a arranged on the instrument panel is provided with a display unit 21b capable of displaying an alarm. When the monitoring support device 21 receives a sidelong glance signal or a drowsiness estimation signal from the ECU 3, it displays an alarm such as "Attention Ahead" on the display unit 21b.

[0024] The braking support device 22 is configured to give an alarm when the own vehicle in motion approaches another vehicle ahead, and to perform an emergency brake when the braking limit distance at which a collision can be avoided is reached. As shown in FIG. 3, when the host vehicle approaches the position of the host vehicle A1 with respect to the other vehicle B that the host vehicle is ahead of, the braking assistance device 22 generates an alarm to the driver from the speaker. The separation distance between the other vehicle B and the host vehicle A1 is a distance corresponding to the time (e.g., 0.8 sec) during which the driver can avoid a collision by his or her own driving operation (manual operation) under normal circumstances. When the host vehicle approaches the position of the host vehicle A2 with respect to the other vehicle B that the host vehicle is ahead of, the braking assistance device 22 automatically performs emergency braking. The separation distance between the other vehicle B and the host vehicle A2 is the limit distance at which a collision can be avoided by emergency braking equivalent to the maximum braking force.

[0025] Also, as shown in FIG. 3, when the braking assistance device 22 receives a side glance signal or a drowsiness estimation signal from the ECU 3, when the host vehicle in motion approaches the position of the host vehicle A3 that is farther away from the other vehicle B that the host vehicle is ahead of than the position of the host vehicle A1, an alarm is generated to the driver from the speaker. Here, the separation distance between the other vehicle B and the host vehicle A3 is a distance corresponding to the time (e.g., 1.2 sec) during which the driver can avoid a collision by his or her own driving operation even when the driver is in a distracted state.

[0026] When the parking assistance device 23 receives a drowsiness estimation signal from the ECU 3, an alarm is generated to the driver from the speaker, and after receiving a confirmed drowsiness signal, the host vehicle is autonomously and emergently stopped. As shown in FIG. 4, when the parking assistance device 23 receives a drowsiness estimation signal corresponding to a closed-eye state at the position of the host vehicle X1, the speaker is activated to generate an alarm, and a confirmed drowsiness signal corresponding to a non-driving operation state or the like is received at the position of the host vehicle X2. Next, at the time of the host vehicle X3, even if it has been a manual driving by the driver until then, autonomous driving is started. The host vehicle that has started autonomous driving moves to a safe area such as the road shoulder and then is autonomously and emergently stopped at the position of the host vehicle X4.

[0027] In addition, when the parking assistance device 23 receives an abnormal signal from the ECU 3, it autonomously performs an emergency stop of the host vehicle. As shown in FIG. 4, the parking assistance device 23 receives an abnormal signal of the driver at the position of the host vehicle X2. Next, when it comes to the host vehicle X3, even if it has been manual driving by the driver until then, autonomous driving is started. After the host vehicle that has started autonomous driving moves to a safe area such as the road shoulder, it is autonomously emergency stopped at the position of the host vehicle X4. After that, the parking assistance device 23 makes an emergency notification to, for example, an emergency center or the like after a predetermined time has elapsed since the start of the parking hold. Hereinafter, these autonomous emergency stop controls are referred to as abnormal-time sequence controls.

[0028] Next, the ECU 3 will be described. The ECU 3 makes respective determinations using the symptoms of the driver at the time of abnormality (hereinafter referred to as abnormal-time symptoms), the physical state of the driver at the time of abnormality (hereinafter referred to as abnormal-time state), and the actions of the driver at the time of abnormality (hereinafter referred to as abnormal-time actions). This ECU 3 is composed of a CPU (Central Processing Unit), a ROM, a RAM, an in-side interface, an out-side interface, and the like. Various programs and data such as brake control and display control are stored in the ROM, and a processing area used when the CPU performs a series of processes is provided in the RAM.

[0029] As shown in FIG. 1, the ECU 3 includes a head posture determination unit 31 that determines the head posture state of the driver, an eye state determination unit 32 (closed-eye determination means) that determines the open / closed state of both the left and right eyes of the driver, a driver state determination unit 33 that determines the driver state that can be an inhibiting factor for vehicle driving, and the like.

[0030] The head posture determination unit 31 determines the head posture state of the driver based on the imaging image information captured by the camera 11. When this head posture determination unit 31 detects that the head posture state of the driver is any one of looking down, falling sideways, slumping forward, and tilting backward, it determines that the head posture has collapsed. When a head posture collapse is determined, 1 is substituted into the count value of the first counter C1. The initial value of the first counter C1 is 0.

[0031] For example, when the absolute value of the face orientation angle in the pitch direction is equal to or greater than a predetermined threshold, the head posture determination unit 31 outputs a face-down determination signal, and when the absolute value of the face orientation angle in the roll direction is equal to or greater than a predetermined threshold, the head posture determination unit 31 outputs a side-down (lying on side) determination signal. Further, for example, when the absolute value of the face orientation angle in the pitch direction is equal to or greater than a predetermined threshold and the current head position is equal to or greater than a value obtained by subtracting a head position threshold from the reference head position, the head posture determination unit 31 outputs a prone determination signal, and when the face orientation angle in the pitch direction is less than the current head position with respect to a value obtained by adding the head position threshold to the reference head position and being equal to or greater than the predetermined threshold, the head posture determination unit 31 outputs a backward tilt determination signal.

[0032] Each threshold related to the angle and head position for determining face-down, side-down, prone, and backward tilt respectively corresponds to a head posture determination threshold for determining an abnormal head posture of the driver. Two types of each threshold are prepared: a head posture determination threshold for closed eyes used when the eyes are closed and a head posture determination threshold for non-closed eyes used when the eyes are not closed. The head posture determination threshold for closed eyes is set to a value smaller than the head posture determination threshold for non-closed eyes aiming at an early determination of an abnormal state.

[0033] The eye state determination unit 32 determines whether the state of the driver's eyes is a closed-eye state. For example, the eye state determination unit 32 detects the upper and lower eyelids of the driver imaged by the camera 11, and detects the eye opening degree of the driver based on the number of pixels between the edges of both eyelids. When the detected eye opening degree is equal to or greater than a predetermined determination threshold, it is determined that the eyes are in an open-eye state, and when the detected eye opening degree is less than the predetermined determination threshold, it is determined that the eyes are in a closed-eye state. When a closed-eye state is determined, 1 is substituted into the count value of the second counter C2. The initial value of the second counter C2 is 0. In this embodiment, a closed-eye determination is made only when both the left and right eyes of the driver are determined to be closed, while in other cases, a closed-eye determination is not made.

[0034] The driver state determination unit 33 includes a side glance determination unit 33a that determines whether the driver is looking in a direction other than the front of the traveling direction, a drowsiness determination unit 33b that determines a drowsiness estimation state in which the driver's drowsiness is estimated and a drowsiness determination state in which the driver's drowsiness is detected with a high probability, and an abnormality determination unit 33c (driver abnormality state determination means) that determines an abnormal state of the driver in which it is difficult to drive the vehicle.

[0035] The side glance determination unit 33a will be described. The side glance determination unit 33a determines whether the driver is making a side glance from the face direction information, line-of-sight direction information, and vehicle information of the driver, and when it is determined that the driver is making a side glance, outputs a side glance signal to the monitoring support device 21 and the braking support device 22. When the count value of the driver's line-of-sight direction in the yaw direction is outside the range of the yaw direction threshold value, it is determined that the driver is making a side glance. The yaw direction threshold value is changed according to the vehicle speed or the radius of curvature when the vehicle is traveling. Also, when the count value of the driver's line-of-sight direction in the pitch direction is outside the range of the pitch direction threshold value, it is determined that the driver is making a side glance. The pitch direction threshold value is changed according to the vehicle speed.

[0036] The count value of the line-of-sight direction in the yaw direction and the count value of the line-of-sight direction in the pitch direction are calculated by a long-time side glance continuous count or a short-time side glance cumulative count. The long-time side glance continuous count is the number of times the determination results for each processing cycle are continuous in time series. Also, the short-time side glance cumulative count is the cumulative count of the determination results for each processing cycle in a state where the conditions match within a predetermined time before that.

[0037] The line-of-sight direction information in the yaw direction and the pitch direction is determined using a reliability threshold value whose reliability is preset. The reliability of each line-of-sight direction information is set based on the sharpness of the imaging image (sharpness of the contours of the face, eyelids, black eyes, etc.) that captures the driver. When the reliability of each line-of-sight direction information is low and each line-of-sight direction is invalid, interpolation is performed using the face direction information in the yaw direction and the pitch direction, respectively. When the reliability of the face direction information in the yaw direction and the pitch direction is low and each face direction is invalid, interpolation is performed using the previous valid line-of-sight direction.

[0038] Next, the drowsiness determination unit 33b will be described. The drowsiness determination unit 33b determines whether the driver is drowsy based on the continuous eye-closure time, vehicle information, etc. When it is estimated that the driver is drowsy, a drowsiness estimation signal is output to the monitoring support device 21, the braking support device 22, and the parking support device 23. When it is determined that the driver is drowsy, a drowsiness confirmation signal is output to the parking support device 23.

[0039] The drowsiness determination unit 33b estimates the drowsy state when the continuous eye-closure time of the driver exceeds a predetermined time threshold, or when the short eye-closure time exceeds the predetermined time threshold repeatedly within a certain time. In addition, when the vehicle is traveling at a certain vehicle speed or higher after a drowsiness alarm, the drowsiness determination unit 33b determines the drowsy state when the continuous eye-closure time of the driver exceeds a predetermined time threshold, or when the continuous eye-closure and non-driving operation state exceed the time threshold.

[0040] The non-driving operation state is determined as an inappropriate driving operation during traveling by comparing a detection signal of the detection means 1 with a determination threshold. Specifically, it is determined based on at least one of the non-steering of the steering wheel, non-gripping of the steering wheel, non-operation of the accelerator, and non-operation of the brake. In addition, for the determination of inappropriate driving operation, an extremely small determination threshold may be set for each operation member to determine inappropriate operation.

[0041] Next, the abnormality determination unit 33c will be described. The abnormality determination unit 33c prepares two types of determination thresholds: a first non-eye-closure determination threshold L1 (first predetermined time) used when the eyes are not closed and a first eye-closure determination threshold L1 (second predetermined time) used when the eyes are closed. The first eye-closure determination threshold L1 is set to a value smaller than the first non-eye-closure determination threshold L1 for the purpose of early determination of an abnormal state.

[0042] When the driver's eyes are open, if the head posture of the driver determined by the head posture determination unit 31 is an abnormal posture and the driving operation amount is less than a predetermined determination threshold corresponding to an inappropriate operation, and this situation continues for a period corresponding to the first determination threshold L1 (first predetermined time) for non-closed eyes, the abnormality determination unit 33c determines that the driver is in an abnormal state. When the driver's eyes are closed, if the head posture of the driver determined by the head posture determination unit 31 is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is less than a predetermined determination threshold corresponding to an inappropriate operation, and this situation continues for a period shorter than the first determination threshold L1 for non-closed eyes, i.e., the first determination threshold L1 (second predetermined time) for closed eyes, the abnormality determination unit 33c determines that the driver is in an abnormal state.

[0043] The inventor has obtained the knowledge that abnormal signs can be detected using eye closure as a parameter, the abnormal state can be detected using the collapse of the head posture as a parameter, and the abnormal behavior can be detected using non-driving operations as a parameter. Based on this knowledge, in this embodiment, when the driver's eyes are open, the abnormality determination unit 33c determines the abnormal state of the driver using the head posture determination threshold for non-closed eyes and the first determination threshold L1 for non-closed eyes. When the driver's eyes are closed, the abnormality determination unit 33c determines the abnormal state of the driver using the head posture determination threshold for closed eyes and the first determination threshold L1 for closed eyes, and outputs an abnormal signal to the driver assistance device 2. Note that non-closed eyes include a state where the closure of both eyes cannot be technically detected and a state where at least one eye is open.

[0044] Next, based on the flowchart of FIG. 5, the eye closure determination control process will be described. Note that Si (i = 1, 2,...) indicates steps for each process.

[0045] First, as shown in FIG. 5, the ECU 3 inputs the output from various detection means 1 such as the camera 11 and various information such as the second counter C2 (S1), and proceeds to S2. In S2, it is determined whether both eyes of the driver are closed. As a result of the determination in S2, if the driver has closed eyes, since the closed eyes, which are signs of an abnormal driver (disease onset or dozing off), are detected, the process proceeds to S3. As a result of the determination in S2, if the driver does not have closed eyes, since at least one eye is open or the reliability of the imaging information is low, the process returns.

[0046] In S3, after adding 1 to the current count value of the second counter C2, the process proceeds to S4. In S4, it is determined whether the count value of the second counter C2 is greater than or equal to the second determination threshold value L2. As a result of the determination in S4, if the count value of the second counter C2 is greater than or equal to the second determination threshold value L2, since it is in a closed-eye state and disease onset or dozing off is estimated, the process proceeds to S5. As a result of the determination in S4, if the count value of the second counter C2 is less than the second determination threshold value L2, since disease onset or dozing off cannot be estimated at present, the process returns. In S5, after generating an alarm, the process returns.

[0047] Next, based on the flowchart of FIG. 6, the head pose determination control process will be described. This head pose determination control process is independently and parallelly controlled from the closed-eye determination control process. Si (i = 11, 12, …) indicates steps for each process.

[0048] First, as shown in FIG. 6, the ECU 3 inputs the output from the detection means 1 such as the camera 11 and the hazard switch 17 (S11), and proceeds to S12. In S12, it is determined whether the count value of the second counter C2 is greater than 0. As a result of the determination in S12, if the count value of the second counter C2 is greater than 0, since the driver has closed eyes, the head pose determination threshold for closed eyes and the first determination threshold value L1 for closed eyes are set (S13), and the process proceeds to S15. As a result of the determination in S12, if the count value of the second counter C2 is 0 or less, since the driver does not have closed eyes, the head pose determination threshold for non-closed eyes, which is greater than the head pose determination threshold for closed eyes, and the first determination threshold value L1 for non-closed eyes, which is greater than the first determination threshold value L1 for closed eyes, are set respectively (S14), and the process proceeds to S15.

[0049] In S15, it is determined whether the driver's head posture has collapsed. Here, the driver's head posture is determined using a head posture determination threshold for eyes closed when the eyes are closed, and a head posture determination threshold for eyes open that is larger than the head posture determination threshold for eyes closed when the eyes are open. Therefore, when the eyes are closed, it is easier to determine that the head posture has collapsed than when the eyes are open. As a result of the determination in S15, if the driver's head posture has collapsed, since it is an abnormal state, the process proceeds to S16. As a result of the determination in S15, if the driver's head posture has not collapsed, since it is not an abnormal state, the process returns.

[0050] In S16, it is determined whether the driving operation by the driver is appropriate. As a result of the determination in S16, if the driving operation is appropriate, since the driver is driving in an intentionally collapsed posture, the process returns. As a result of the determination in S16, if the driving operation is inappropriate (non-driving operation), the process proceeds to S17. The determination of the driving operation is detected by at least one of the steering wheel, accelerator, and brake. In the present embodiment, the operation of the steering wheel is used for the determination.

[0051] In S17, it is determined whether the vehicle is traveling at a speed equal to or higher than a predetermined determination threshold. As a result of the determination in S17, if the vehicle speed is equal to or higher than the determination threshold, since there is no driving operation by the driver despite the vehicle being in motion, after adding 1 to the current count value of the first counter C1 (S18), the process proceeds to S19. As a result of the determination in S17, if the vehicle speed is less than the determination threshold, since the vehicle has already stopped or is traveling slowly, the process returns.

[0052] In S19, it is determined whether the count value of the first counter C1 is equal to or greater than the first determination threshold L1. Here, the driving non-operation, which is an operation during abnormal conditions, is determined using the first closed-eye determination threshold value L1 when the eyes are closed, and using a first non-closed-eye determination threshold value L1 that is greater than the first closed-eye determination threshold value L1 when the eyes are not closed. Therefore, during closed eyes, it is easier to determine that the driving non-operation is an operation during abnormal conditions than during non-closed eyes. As a result of the determination in S19, if the count value of the first counter C1 is greater than or equal to the first determination threshold value L1, an abnormal state is presumed regardless of the state of the driver's eyes, and thus the process proceeds to S20. As a result of the determination in S19, if the count value of the first counter C1 is less than the first determination threshold value L1, since an abnormal state cannot be presumed at present, the process returns.

[0053] In S20, after generating an alarm, the process proceeds to S21. In S21, it is determined whether a cancel signal for stopping the alarm has been generated. As a result of the determination in S21, if a cancel signal has been generated, since the driver has performed an operation to generate the cancel signal, after substituting 0 into the count values of the first and second counters C1 and C2 respectively (S22), the process returns. As a result of the determination in S21, if a cancel signal has not been generated, since the driver cannot cancel it with the hazard switch 17, the process proceeds to S23. In S23, after executing the abnormal sequence control, the process returns. In this S23, 0 is substituted into the count values of the first and second counters C1 and C2 respectively.

[0054] Next, the operation and effects of the above driver state determination method and its device M will be described. According to this driver state determination method, since it has a head posture acquisition step S1, S11 for acquiring the driver's face orientation and head position, a closed-eye determination step S2, S12 for detecting that the driver's eyes are in a closed state, a driving operation amount acquisition step S1, S11 for acquiring at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and a driver abnormal state determination step S19 for determining whether the driver is in an abnormal state, it is possible to use the head posture collapse, which is an abnormal state, the closed eyes, which are abnormal symptoms, and the driving non-operation, which is an operation during abnormal conditions, as the determination conditions for the driver's abnormal state. When the driver abnormal state determination step S19 determines that the driver is in an abnormal state when the closed-eye state of the driver is not detected in the closed-eye determination step S12, and the head posture of the driver obtained in the head posture acquisition step S11 is an abnormal posture and the driving operation amount is lower than a predetermined threshold corresponding to non-driving operation for a period corresponding to the first determination threshold L1 for non-closed eyes. Therefore, even in a non-closed-eye state, the abnormal state of the driver can be accurately determined using the abnormal state and abnormal operation at the time of abnormality as determination conditions. When the closed-eye state of the driver is detected in the closed-eye determination step S12, the head posture of the driver obtained in the head posture acquisition step S11 is an abnormal posture with a lower degree of abnormality than the abnormal posture, and the driving operation amount is lower than a predetermined threshold corresponding to non-driving operation for a period shorter than the period corresponding to the first determination threshold L1 for non-closed eyes and continues for a period corresponding to the first determination threshold L1 for closed eyes, the driver abnormal state determination step S19 determines that the driver is in an abnormal state. Therefore, in the closed-eye state, the abnormal state of the driver can be determined in a short time based on the abnormal symptom and abnormal state at the time of abnormality.

[0055] The driver abnormal state determination step S19 can easily determine that the head posture of the driver is an abnormal posture because it determines that the head posture is an abnormal posture based at least on the face orientation angle of the driver.

[0056] The closed-eye determination steps S4 and S5 can cause the driver to recognize closing of the eyes because an alarm is notified after a period corresponding to the second determination threshold L2 has elapsed since the closed-eye state of the driver was detected.

[0057] According to the driver state determination device M, in the driver state determination device M that detects an abnormal state of the driver of a vehicle, a camera 11 that acquires the driver's face orientation and head position, an eye state determination unit 32 that detects that the driver's eyes are in a closed state, sensors 12 to 16 that acquire at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing, and an abnormal state determination unit 33c that determines whether the driver is in an abnormal state. Therefore, the abnormal head posture during an abnormal state, the closed eyes as an abnormal sign, and the non-driving operation as an abnormal action can be used as the determination conditions for the driver's abnormal state. When the eye state determination unit 32 does not detect the driver's closed-eye state, if the situation where the head posture of the driver acquired by the camera 11 is an abnormal posture and the driving operation amount is below a predetermined threshold corresponding to non-driving operation continues for a period corresponding to the first determination threshold L1 for non-closed eyes, the abnormal state determination unit 33c determines that the driver is in an abnormal state. Therefore, even in a non-closed-eye state, the abnormal state of the driver can be determined with high accuracy using the abnormal state and abnormal action as determination conditions. When the eye state determination unit 32 detects the driver's closed-eye state, if the situation where the head posture of the driver acquired by the camera 11 is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is below a predetermined threshold corresponding to non-driving operation continues for a period shorter than the period corresponding to the first determination threshold L1 for non-closed eyes and corresponding to the first determination threshold L1 for closed eyes, the abnormal state determination unit 33c determines that the driver is in an abnormal state. Therefore, in the closed-eye state, the abnormal state of the driver can be determined in a short time based on the abnormal sign and abnormal state.

[0058] In addition, those skilled in the art can implement the present invention in a form with various modifications added to the above-described embodiments without departing from the spirit of the present invention, and the present invention also includes such modified forms.

Explanation of Reference Numerals

[0059] 1 Detection means 11 Camera 32 Eye state determination unit 33c Abnormal determination unit M Driver state determination device

Claims

1. In a driver state determination method for detecting an abnormal state of a vehicle driver, a head pose acquisition step of acquiring the face orientation and head position of the driver; a closed-eye determination step of detecting that the driver's eyes are in a closed state; a driving operation amount acquisition step of acquiring at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing; a driver abnormal state determination step of determining whether the driver is in an abnormal state, wherein the driver abnormal state determination step when the closed-eye state of the driver is not detected in the closed-eye determination step, if the head pose of the driver acquired in the head pose acquisition step is an abnormal pose and the driving operation amount is below a predetermined threshold for a first predetermined time continuously, it is determined that the driver is in an abnormal state; when the closed-eye state of the driver is detected in the closed-eye determination step, if the head pose of the driver acquired in the head pose acquisition step is an abnormal pose with a lower degree of abnormality than the abnormal pose and the driving operation amount is below a predetermined threshold for a second predetermined time shorter than the first predetermined time continuously, it is determined that the driver is in an abnormal state A driver state determination method characterized by the above.

2. The driver abnormal state determination step determines that the head pose is an abnormal pose based on at least the face orientation angle of the driver. The driver state determination method according to Claim 1, characterized by the above.

3. The closed-eye determination step is characterized in that an alarm is notified after a predetermined time has elapsed since the closed-eye state of the driver is detected. The driver state determination method according to Claim 1 or 2, characterized by the above.

4. In a driver state determination device for detecting an abnormal state of a vehicle driver, a head pose acquisition means for acquiring the face orientation and head position of the driver; a closed-eye determination means for detecting that the driver's eyes are in a closed state; a driving operation amount acquisition means for acquiring at least one of the driving operation amounts of the vehicle by the driver at a predetermined timing; a driver abnormal state determination means for determining whether the driver is in an abnormal state, wherein the driver abnormal state determination means when the closed-eye determination means does not detect the closed-eye state of the driver, if the head pose of the driver acquired by the head pose acquisition means is an abnormal pose and the driving operation amount is below a predetermined threshold for a first predetermined time continuously, it is determined that the driver is in an abnormal state, When the eye-closure determination means detects the closed-eye state of the driver, if a situation where the head posture of the driver acquired by the head posture acquisition means is an abnormal posture with a lower degree of abnormality than the abnormal posture and the driving operation amount is below a predetermined threshold continues for a second predetermined time shorter than the first predetermined time, it is determined that the driver is in an abnormal state. A driver state determination device characterized by this.

Images