Driver state determination method and device

Abstract

To provide a driver condition determination method and system making it possible to upgrade the precision in determining a driver's abnormal condition.SOLUTION: A driver condition determination method includes ambient information acquisition step S1, field-of-view image acquisition step S2, an eye-catching degree distribution generation step S3 of generating a driver's eye-catching degree distribution, sight line behavior acquisition step S4 of acquiring a driver's sight line behavior, sight line determination index arithmetic step S5 of computing plural sight line determination indices of a driver, head posture acquisition step S21 of acquiring the orientation of a driver's face, and driver abnormal condition determination steps S22, S23, and S27 of determining whether a driver is in an abnormal condition. At the driver abnormal condition determination steps S22, S23, and S27, when an abnormal posture is detected but plural sight line determination indices are all normal, it is determined that the driver has intentionally relaxed his / her posture. When the abnormal posture is detected and any of the plural sight line determination indices is abnormal, it is determined that the driver is in the abnormal condition.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a driver condition determination method and device for detecting an abnormal condition of a driver who drives a vehicle. [Background technology]

[0002] BACKGROUND ART Driver support devices have been known that accurately grasp a driver's state of consciousness, particularly the driver's state of alertness, and support the driver in driving a vehicle in accordance with the driver's state of alertness. An abnormal state of a driver, including illness or drowsiness, is determined based on head-related information such as the driver's head posture and facial state acquired by an imaging means such as a camera. Among the head-related information, the head posture is detected based on, for example, the driver's facial direction and head position (three-dimensional coordinates), and the facial state is detected based on, for example, the driver's facial expression and eye opening state (eyelid state).

[0003] The vehicle alarm device of Patent Document 1 has a drowsiness detection unit that detects whether the driver is drowsy, an inattentive detection unit that detects whether the driver is looking away, an abnormal posture detection unit that detects whether the driver is in an abnormal posture, a vehicle speed detection unit that detects the vehicle speed, an alarm unit that issues an alarm, and an alarm control unit that activates the alarm unit based on detection by any of the detection units, and when the vehicle speed is below a predetermined threshold, the alarm control unit issues an alarm when drowsiness is detected and does not issue an alarm when inattentiveness is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-133402 Summary of the Invention [Problem to be solved by the invention]

[0005] The vehicle alarm device of Patent Document 1 issues an alarm in the event of drowsiness or abnormal posture, but can avoid issuing unnecessary alarms in situations where the driver is driving the vehicle at extremely low speed to look for a parking spot, etc., thereby limiting the occurrence of situations that the driver finds annoying. However, although the vehicle warning device of Patent Document 1 can avoid false detection in the inattentive driving judgment, there is a risk that it cannot avoid false detection in the abnormal posture judgment.

[0006] Generally, when a driver develops a disease, an abnormal posture, in other words, an unconscious collapse of posture without the driver's awareness, is recognized by external observation. Therefore, unconscious posture changes are detected by facial orientation and head position. On the other hand, when the driver consciously adopts an unsuitable driving posture, i.e., intentionally loses posture, if an abnormality is determined based on the driver's facial direction and head position, the driver's abnormal state is determined. However, since the driver intentionally loses his / her posture, the warning is issued even though there is no particular problem in terms of consciousness or driving operation, which may cause inconvenience to the driver.

[0007] Some driver assistance devices automatically move the vehicle toward the shoulder of the road and autonomously stop the vehicle when it is determined that the driver is in an abnormal state. Such driver assistance devices automatically stop the vehicle even if it is determined that the driver has intentionally lost posture, which not only increases the inconvenience for the driver but also poses the risk of an unexpected situation in which the vehicle makes an emergency stop in front of other vehicles. That is, although it is essential to improve the accuracy of determining whether a driver is in an abnormal posture, it is not easy.

[0008] An object of the present invention is to provide a driver state determination method and device therefor that can improve the accuracy of determining whether a driver is in an abnormal state. [Means for solving the problem]

[0009] The driver condition determination method of claim 1 is a driver condition determination method for detecting an abnormal state of a driver of a vehicle, comprising: A control process for determining whether the driver's eyes are closed and a control process for determining a head posture of the driver are performed in parallel based on an image of the driver, and the control process for determining whether the driver's eyes are closed determines that the driver is in an abnormal state when the driver does not respond to a warning issued when the driver's eyes remain closed, and the control process for determining whether the driver's head posture is abnormal is performed. a surrounding information acquisition step of acquiring surrounding information relating to a surrounding situation of the vehicle; Based on the surrounding information within the driver's field of vision equivalent a field of view image acquisition step of acquiring a field of view image; Recording an attractiveness distribution generating step of generating an attractiveness distribution of the driver with respect to the field image; Based on the captured image and the field of view image a gaze behavior acquisition step of acquiring a gaze behavior of the driver; and a plurality of gaze determination indices of the driver based on the peripheral information, the field of view image, the eye-attraction distribution, and the gaze behavior. saliency index, saccade amplitude index, and saccade frequency index a gaze determination index calculation step of calculating the gaze determination index; Based on the captured image The vehicle includes a head posture acquisition step of acquiring a face direction and a head position of the driver, and a driver abnormal state determination step of determining whether the driver is in an abnormal state, and the driver abnormal state determination step is performed when the head posture of the driver acquired in the head posture acquisition step is an abnormal posture and the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step are all normal. to When it is determined that the driver has intentionally lost posture, the head posture of the driver acquired in the head posture acquisition step is an abnormal posture, and any one of the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step is abnormal. to The method is characterized in that it is determined that the driver is in an abnormal state.

[0010] In this driver state determination method, The control process for determining whether the driver's eyes are closed and the control process for determining the driver's head posture are performed in parallel based on the captured image of the driver. The control process for determining whether the driver's eyes are closed determines that the driver is in an abnormal state if the driver does not respond to a warning that is issued when the driver's eyes remain closed. The control process for determining whether the driver's head posture is closed is performed in parallel based on the captured image of the driver. The control process for determining whether the driver's eyes are closed determines that the driver is in an abnormal state if the driver does not respond to a warning that is issued when the driver's eyes remain closed. a surrounding information acquisition step of acquiring surrounding information relating to a surrounding situation of the vehicle; equivalent a field of view image acquisition step of acquiring a field of view image; Recording an attractiveness distribution generating step of generating an attractiveness distribution of the driver with respect to the field image; Based on the captured image and the field of view image a gaze behavior acquisition step of acquiring the gaze behavior of the driver; The aforementioned Field of view image and The aforementioned Attractiveness distribution and The aforementioned A plurality of gaze determination indices of the driver based on the gaze behavior saliency index, saccade amplitude index, and saccade frequency index a gaze determination index calculation step of calculating the gaze determination index; Based on the captured imageThe vehicle includes a head posture acquisition step for acquiring the face direction and head position of the driver, and a driver abnormal state determination step for determining whether the driver is in an abnormal state. Related to attention level A plurality of gaze determination indices can be included in the conditions for determining whether the driver is in an abnormal state. The driver abnormal state determination process determines that the driver has intentionally lost posture if the driver's head posture acquired in the head posture acquisition process is abnormal and the driver's multiple gaze determination indices calculated in the gaze determination index calculation process are all normal, thereby making it possible to determine whether a driver has an inappropriate driving posture even if there are no problems with their consciousness or driving operation. The driver abnormal state determination process determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition process is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process is abnormal, so that unconscious posture deviations of the driver can be determined with high accuracy from their posture deviations. Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally.

[0011] The invention of claim 2 is characterized in that in the invention of claim 1, the driver's abnormal state determination step determines that the head posture is abnormal based on at least the angle of the driver's face. According to this configuration, it is possible to easily determine whether the driver's head posture is abnormal.

[0012] The invention of claim 3 is characterized in that, in the invention of claim 1 or 2, the driver abnormal state determination process determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition process is abnormal and any of the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation process is abnormal, for a predetermined period of time. This configuration makes it possible to reliably determine whether the driver is in an abnormal state.

[0013]

[0014] Claim4 The driver condition determination device detects an abnormal state of a vehicle driver, and includes: a surrounding information acquisition means for acquiring surrounding information relating to a surrounding situation of the vehicle; a driving operation amount acquisition means for acquiring a driving operation amount of the driver; within the driver's field of vision equivalent A field of view image acquisition means for acquiring a field of view image; Recording an attractiveness distribution generating means for generating an attractiveness distribution of the driver with respect to the field image; Based on an image captured by an internal camera capturing an image of the driver a gaze behavior acquisition means for acquiring the gaze behavior of the driver; The aforementioned Field of view image and The aforementioned Attractiveness distribution and The aforementioned A plurality of gaze determination indices of the driver based on the gaze behavior saliency index, saccade amplitude index, and saccade frequency index A gaze determination index calculation means for calculating the gaze determination index; Based on the captured image The vehicle includes a head posture acquisition means for acquiring a face direction and a head position of the driver, and a driver abnormal state determination means for determining whether the driver is in an abnormal state, and the driver abnormal state determination means A control process for determining whether the driver's eyes are closed and a control process for determining a head posture of the driver are performed in parallel based on the captured image. In the control process for determining whether the driver's eyes are closed, a warning is issued when the driver continues to have his / her eyes closed. In the control process for determining whether the driver has an abnormal state, if the driver does not respond based on the driving operation amount and the captured image, the control process for determining whether the driver has an abnormal state is performed. In the control process for determining whether the driver's eyes are closed, a warning is issued when the driver continues to have his / her eyes closed. In the control process for determining whether the driver has an abnormal state, When the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and the plurality of gaze determination indices of the driver calculated by the gaze determination index calculation means are all normal to The driver is determined to have intentionally lost his / her posture, and the head posture of the driver acquired by the head posture acquisition means is determined to be abnormal and the line of sight determination index calculation means but When any of the calculated gaze determination indices of the driver is abnormal to The method is characterized in that it is determined that the driver is in an abnormal state.

[0015] In this driver condition determination device, the device detects an abnormal state of a driver of a vehicle, and includes: a surrounding information acquisition means for acquiring surrounding information relating to a surrounding situation of the vehicle; a driving operation amount acquisition means for acquiring a driving operation amount of the driver; within the driver's field of vision equivalent A field of view image acquisition means for acquiring a field of view image; Recording an attractiveness distribution generating means for generating an attractiveness distribution of the driver with respect to the field image; Based on an image captured by an internal camera capturing an image of the driver a gaze behavior acquisition means for acquiring the gaze behavior of the driver; The aforementioned Field of view image and The aforementioned Attractiveness distribution and The aforementionedA plurality of gaze determination indices of the driver based on the gaze behavior saliency index, saccade amplitude index, and saccade frequency index A gaze determination index calculation means for calculating the gaze determination index; Based on the captured image The vehicle includes a head posture acquisition means for acquiring the face direction and head position of the driver, and a driver abnormal state determination means for determining whether the driver is in an abnormal state. Related to attention level A plurality of gaze determination indices can be included in the conditions for determining whether the driver is in an abnormal state. The driver abnormal state determination means determines that the driver has intentionally lost posture if the head posture of the driver acquired by the head posture acquisition means is abnormal and the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means are all normal, and therefore can determine that a driver has an inappropriate driving posture even if there are no problems with their state of consciousness or driving operation. The driver abnormal state determination means determines whether the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and whether the line of sight determination index calculation means but If any of the calculated gaze determination indices of the driver is abnormal, the driver is determined to be in an abnormal state, so that unconscious posture deviation can be determined with high accuracy from the driver's posture deviation. Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally. [Effects of the Invention]

[0016] According to the driver state determination method and device of the present invention, The driver's eyes are closed and the head posture is abnormal. By using a plurality of indicators for determining the driver's line of sight as conditions for determining whether the driver is in an abnormal state, it is possible to improve the accuracy of determining whether the driver is in an abnormal state. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram of a driver state determination device according to a first embodiment. [Figure 2] FIG. 1 is an explanatory diagram of a monitoring support device. [Figure 3] FIG. 2 is an explanatory diagram of a braking assist device. [Figure 4] FIG. 2 is an explanatory diagram of a vehicle stopping assistance device. [Figure 5]10 is a step chart of a line-of-sight determination control process. [Figure 6] 10 is a graph for explaining extraction of a saccade. [Figure 7] 10 is a graph for explaining a noise removal process of a saccade. [Figure 8] FIG. 10 is a diagram showing an example of a time change in saliency related to a driver's gaze point. [Figure 9] FIG. 9 is a diagram showing an example of time-dependent change in saliency related to random points whose coordinates are randomly specified in the same environment as in FIG. 8. [Figure 10] FIG. 1 is an explanatory diagram of a saliency index. [Figure 11] 10 is a flowchart showing a control process for determining whether the eyes are closed. [Figure 12] 10 is a flowchart showing a head posture determination control process. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following description exemplifies the application of the present invention to a driver state determination device for a vehicle, and does not limit the present invention, its applications, or its uses. [Example]

[0019] A first embodiment of the present invention will be described below with reference to FIGS. The driver state determination device M is a driving assistance device that determines the driver's state of consciousness, particularly an abnormal state that makes it difficult for the driver to operate the vehicle (hereinafter also referred to as the driver's own vehicle), and is capable of autonomously making an emergency stop if an abnormal state of the driver is determined. Hereinafter, the abnormal state of the driver will be explained as a state in which the driver is not consciously awake and is physically unable to perform driving operations due to poor posture (including the onset of illness or drowsiness). The following description also includes a description of a driver state determination method.

[0020] As shown in FIG. 1, the driver state determination device M mainly comprises a plurality of detection means 1 capable of detecting the driver's operating behavior and the vehicle's driving behavior, a driver assistance device 2 that autonomously provides vehicle driving assistance to the driver from the vehicle side, and an ECU (Electronic Control Unit) 3 that performs various calculations based on input signals input from the detection means 1 and outputs various command signals to the driver assistance device 2.

[0021] First, the detection means 1 will be described. 1, the detection means 1 includes an external camera 10 that captures an image ahead of the vehicle in the traveling direction, an internal camera 11 that captures an image of the driver, a vehicle speed sensor 12 that detects the speed of the vehicle, a steering angle sensor 13 that detects the steering angle caused by operation of the steering wheel (not shown), a grip sensor 14 that detects the driver's grip on the steering wheel, an accelerator sensor 15 that detects the amount of depression of the accelerator pedal (not shown), a brake sensor 16 that detects the amount of depression of the brake pedal (not shown), and a hazard switch 17 that activates hazard lights (not shown). Each of the sensors 12 to 16 corresponds to a driving operation amount acquisition means that acquires the driving operation amount at a predetermined timing, i.e., the current time immediately before the judgment time.

[0022] The external camera 10 is provided to cover the entire area around the vehicle. The external camera 11 captures an image of the environment surrounding the vehicle (external environment around the vehicle) to obtain an image of the area ahead in the traveling direction, which is part of the external environment of the vehicle. Image information acquired by the external camera 10 is transmitted to the ECU 3. The external camera 10 is a monocular camera with a wide-angle lens, and is configured using a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor).

[0023] The internal camera 11 is configured by, for example, a CCD camera, and is mounted in a position facing the driver at the center of the upper part of an instrument panel (not shown). This internal camera 11 is configured to be able to capture an image of the head, including the face, of a driver seated in the driver's seat and fastening a seat belt from the front. The captured head image is configured to be able to extract an image of the driver's face portion and detect the direction of the face (direction of face orientation), identify the driver's facial expression, enlarge and detect the iris of the eye to determine the direction of the line of sight, enlarge and detect the eyelid portion to determine the degree of eye opening and closing, and the three-dimensional coordinate position of the driver's head (center of gravity), etc. The internal camera 11 is required to be capable of acquiring at least the necessary captured images, and may be equipped with two cameras, a front front camera and an upper camera installed directly above the driver's seat, or three or more cameras.

[0024] Next, the driver assistance device 2 will be described. As shown in FIG. 1, the driver assistance device 2 is composed of a monitoring assistance device 21 that notifies the driver of an alert state, a braking assistance device 22 that causes the vehicle to perform a braking operation, and a stopping assistance device 23 that causes the vehicle to perform an emergency stop operation.

[0025] The monitoring support device 21 is configured to recognize the state of consciousness, that is, the wakefulness state of the driver, and, if it determines that recovery is necessary, to notify the driver and encourage the driver to recover consciousness. When the monitoring support device 21 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, it issues a warning to the driver from one or more speakers (not shown) provided in the vehicle cabin. 2, a speedometer unit 21a disposed on an instrument panel is provided with a display unit 21b capable of displaying a warning. When the monitoring support device 21 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, the monitoring support device 21 displays a warning such as "Watch out for the road ahead" on the display unit 21b.

[0026] The braking assist device 22 is configured to issue a warning when the vehicle approaches another vehicle ahead while traveling, and to apply emergency braking when the braking limit distance at which a collision can be avoided is reached. As shown in FIG. 3, when the host vehicle approaches the leading vehicle B to the position of the host vehicle A1, the braking assist device 22 issues a warning to the driver from the speaker. The separation distance between the other vehicle B and the host vehicle A1 is a distance corresponding to the time (for example, 0.8 seconds) that the driver can normally avoid a collision by his / her own driving operation (manual operation). The braking assist device 22 automatically applies emergency braking when the host vehicle approaches the leading vehicle B up to the position of the host vehicle A2. The separation distance between the leading vehicle B and the host vehicle A2 is the limit distance at which a collision can be avoided by applying emergency braking equivalent to the maximum braking force.

[0027] 3, when the braking assist device 22 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, the braking assist device 22 issues an alarm to the driver from the speaker when the host vehicle approaches a preceding vehicle B to a position of the host vehicle A3 that is farther away from the preceding vehicle B than the position of the host vehicle A1. Here, the separation distance between the other vehicle B and the host vehicle A3 is a distance corresponding to the time (e.g., 1.2 seconds) that the driver can avoid a collision by driving the vehicle himself even if he is distracted.

[0028] When the stopping assistance device 23 receives a drowsiness estimation signal from the ECU 3, it issues an alarm to the driver from a speaker, and after receiving a drowsiness confirmation signal, it autonomously brings the vehicle to an emergency stop. As shown in Figure 4, when the vehicle stopping assistance device 23 receives a drowsiness estimation signal corresponding to an eyes-closed state at the location of the host vehicle X1, it activates the speaker to issue an alarm, and receives a drowsiness confirmation signal corresponding to a state of no driving operation at the location of the host vehicle X2. Next, at the location of the host vehicle X3, automatic driving begins even if the host vehicle has been manually driven by the driver until then. After starting automatic driving, the host vehicle moves to a safety zone such as the shoulder of the road and then makes an autonomous emergency stop at the location of the host vehicle X4.

[0029] Furthermore, when the vehicle stopping assistance device 23 receives an abnormality signal from the ECU 3, it autonomously brings the host vehicle to an emergency stop. As shown in FIG. 4, the vehicle stopping assistance device 23 receives an abnormality signal from the driver at the position of the host vehicle X2. Next, at the host vehicle X3, automatic driving is initiated even if the host vehicle has been manually driven by the driver up until that point. After starting automatic driving, the host vehicle moves to a safety zone such as the shoulder of the road and then makes an autonomous emergency stop at the position of the host vehicle X4. Thereafter, the vehicle stopping assistance device 23 makes an emergency call to an emergency center or the like, for example, after a predetermined vehicle stop maintenance period has elapsed. Hereinafter, this autonomous emergency stop control will be referred to as abnormality sequence control.

[0030] Next, the ECU 3 will be described. The ECU 3 determines whether the driver is in an abnormal state based on the driver's symptoms at the time of an abnormality (hereinafter referred to as "abnormal symptoms"), the driver's physical condition at the time of an abnormality (hereinafter referred to as "abnormal state"), and gaze abnormality, which is one of the driver's gaze characteristics. The ECU 3 is composed of a CPU (Central Processing Unit), ROM, RAM, an internal interface, an external interface, etc. The ROM stores various programs and data for brake control, display control, etc., and the RAM has a processing area used by the CPU when performing a series of processes.

[0031] As shown in FIG. 1, the ECU 3 includes a gaze determination unit 30 (field of view image acquisition means, gaze determination index calculation means) that determines whether the driver has an abnormality in their gaze, a head posture determination unit 31 that determines the head posture state of the driver, an eye state determination unit 32 that determines the open / closed state of both the driver's left and right eyes, and a driver state determination unit 33 that determines the driver state that may be an obstruction to driving the vehicle.

[0032] First, the line-of-sight determination unit 30 will be described. As shown in Fig. 1, the gaze determination unit 30 includes a saliency (attractiveness) determination unit 30a (attractiveness distribution generating means) and a saccade (saccadic eye movement) determination unit 30b (gaze behavior acquiring means). The saliency determination unit 30a determines gaze abnormality based on a saliency index, and the saccade determination unit 30b determines gaze abnormality based on a saccade amplitude index and a saccade frequency index. The gaze determination unit 30 determines the gaze to be normal when all of these gaze determination indices are determined to be normal, and determines the gaze to be abnormal when any one of the three gaze determination indices is determined to be abnormal.

[0033] As shown in the step chart of FIG. 5, the gaze determination control process executed by the gaze determination unit 30 is processed in the order of peripheral information acquisition step S1, field of view image acquisition step S2, attention distribution generation step S3, gaze behavior acquisition step S4, gaze determination index calculation step S5, and gaze index determination step S6. In the peripheral information acquisition step S1, the images captured by the external camera 10 and the images captured by the internal camera 11 are received and processed. This image processing includes distortion correction processing to correct distortion in the images and white balance adjustment processing to adjust the white balance of the images.

[0034] In the field of view image acquisition step S2, a field of view image that appears in the field of view of the driver seated in the driver's seat is created based on inputs from the external camera 11 and the internal camera 12. The field of view image is created by combining an external environment image in front of the vehicle captured by the external camera 10 with vehicle body components such as pillars that are in the driver's field of view while the vehicle is traveling. In addition, the driver's gaze point is calculated from the driver's line of sight direction and the field of view image.

[0035] In the attention distribution generation process S3, saliency is calculated for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, for the parts of the field of view image acquired in the field of view image acquisition process S2 other than the vehicle body components, and a saliency map for each feature is generated by adding up these generated saliency maps for each feature.

[0036] In the gaze behavior acquisition step S4, the amplitude and frequency of saccades are acquired. As shown in Figure 6, the period sandwiched between adjacent fixation periods is a saccade period. The saccade amplitude ds is the distance the gaze moves during the saccade period. As shown in Figure 7, saccade candidates are extracted based on changes in the distance of gaze movement, and a regression curve L10 is derived based on the multiple saccade candidates. A saccade range R10 is set between a first reference curve L11 obtained by shifting the regression curve L10 in the direction of increasing movement speed and a second reference curve L12 obtained by shifting the regression curve L10 in the direction of decreasing movement speed, and saccade candidates included in the saccade range R10 are extracted as saccades from among the multiple saccade candidates.

[0037] In the gaze determination index calculation step S5, a saliency index, a saccade amplitude index, and a saccade frequency index are calculated. The saccade amplitude index is calculated by averaging the amplitude of saccades within a predetermined period, and the saccade frequency index is calculated by dividing the number of saccades within a predetermined period by the duration of the period.

[0038] In calculating the saliency index, the driver's gaze point is detected in the saliency map obtained within the measurement time, and a saliency graph (see Figure 8) is created by extracting the gaze point for each specified time, and a random point is specified in the saliency map obtained within the measurement time, and a saliency graph (see Figure 9) is created by extracting the random point for each specified time. The probability of exceeding the threshold at the fixation point is calculated by dividing the number of fixation point saliencies that exceed the threshold by the total number of fixation point saliencies. Also, the probability of exceeding the threshold at the random point is calculated by dividing the number of random point saliencies that exceed the threshold by the total number of random point saliencies.

[0039] As shown in Figure 10, a receiver operating characteristic (ROC) curve C is derived based on a combination of the probability of exceeding a threshold at the gaze point and the probability of exceeding a threshold at a random point. This ROC curve C changes depending on the strength of the tendency for the driver's gaze to be attracted to high saliency areas. Finally, a saliency index (AUC: Area Under Curve), which is the area below the ROC curve C (hatched area), is calculated.

[0040] In the gaze index determination step S6, if the saliency index exceeds a predetermined standard AUC value, or if the saccade amplitude index is equal to or greater than a determination threshold, or if the saccade frequency index is equal to or greater than a determination threshold, it is determined that the gaze abnormality is causing a decline in attention function. If the saliency index, the saccade amplitude index, and the saccade frequency index are all determined to be normal, the driver's line of sight is determined to be normal.

[0041] Next, the head posture determination unit 31 will be described. Head posture determination unit 31 determines the head posture state of the driver based on image information captured by internal camera 11. This head posture determination unit 31 determines that the head posture state of the driver is out of alignment when it detects that the driver's head posture state is one of the following: head down, lying sideways, leaning face down, or leaning backward. When a head posture error is detected, the first counter C1 is set to 1. The initial value of the first counter C1 is 0.

[0042] For example, the head posture determination unit 31 outputs a head-down determination signal when the absolute value of the face direction angle in the pitch direction is equal to or greater than a predetermined threshold, and outputs a sideways (leaning) determination signal when the absolute value of the face direction angle in the roll direction is equal to or greater than the predetermined threshold. Furthermore, the head posture determination unit 31 outputs a head-down determination signal when the absolute value of the face direction angle in the pitch direction is equal to or greater than the predetermined threshold and the current head position is equal to or greater than a value obtained by subtracting a head position threshold from a reference head position, and outputs a backward tilt determination signal when the face direction angle in the pitch direction is equal to or greater than the predetermined threshold and the current head position is smaller than a value obtained by adding the head position threshold to the reference head position. The thresholds associated with the angles and head positions used to determine head-down, sideways, leaning, and backward tilt correspond to head posture determination thresholds used to determine abnormal head posture of the driver.

[0043] The eye state determination unit 32 determines whether the driver's eyes are closed or not. The eye state determination unit 32 detects the driver's upper and lower eyelids captured by the internal camera 11, for example, and detects the degree of eye opening based on the number of pixels between the edges of the eyelids. If the detected degree of eye opening is equal to or greater than a predetermined determination threshold, it is determined that the driver's eyes are open. If the detected degree of eye opening is less than the predetermined determination threshold, it is determined that the driver's eyes are closed. When the eye state is determined to be closed, 1 is assigned to the second counter C2. The initial value of the second counter C2 is 0. In this embodiment, the eyes are determined to be closed only when both the left and right eyes of the driver are determined to be closed, but in other cases, the eyes are not determined to be closed.

[0044] The driver state determination unit 33 includes an inattentiveness determination unit 33a that determines whether the driver is looking in a direction other than the direction ahead in the direction of travel, a drowsiness determination unit 33b that determines a drowsiness estimation state in which the driver is presumed to be drowsy and a drowsy confirmation state in which the driver is detected to be drowsy with a high probability, and an abnormality determination unit 33c (driver abnormal state determination means) that determines an abnormal state of the driver in which it is difficult to drive the vehicle.

[0045] The inattentive driving determining unit 33a will be described. The inattentiveness determination unit 33a determines whether the driver is looking away from the driver based on the driver's face direction information, gaze direction information, and vehicle information, and if it is determined that the driver is looking away, it outputs an inattentiveness signal to the monitoring support device 21 and the braking support device 22. When the count value of the driver's gaze direction in the yaw direction falls outside the range of the yaw direction threshold, it is determined that the driver is looking aside. The yaw direction threshold is changed according to the vehicle speed or the radius of curvature when the vehicle is traveling. Also, when the count value of the driver's gaze direction in the pitch direction falls outside the range of the pitch direction threshold, it is determined that the driver is looking aside. The pitch direction threshold is changed according to the vehicle speed.

[0046] The count value of the yaw direction gaze direction and the count value of the pitch direction gaze direction are calculated by a long-term continuous inattentive count or a short-term cumulative inattentive count. The long-term continuous inattentive count is the number of times that the determination results for each processing cycle continue in chronological order. The short-term cumulative inattentive count is the cumulative count of the determination results for each processing cycle when the conditions are met within a predetermined period of time prior to that.

[0047] The reliability of the gaze direction information in the yaw and pitch directions is determined using a preset reliability threshold. The reliability of each gaze direction information is set based on the clarity of the captured image of the driver (clarity of the contours of the face, eyelids, pupils, etc.). If the reliability of each gaze direction information is low and each gaze direction is invalid, it is interpolated using face direction information in the yaw and pitch directions.If the reliability of face direction information in the yaw and pitch directions is low and each face direction is invalid, it is interpolated using the previous valid gaze direction.

[0048] Next, the drowsiness determination unit 33b will be described. The drowsiness determination unit 33b determines whether the driver is drowsy or not based on the duration of eye closure and vehicle information, etc., and if it is estimated that the driver is drowsy, it outputs a drowsiness estimation signal to the monitoring support device 21, the braking support device 22, and the stopping support device 23, and if it is confirmed that the driver is drowsy, it outputs a drowsiness confirmation signal to the stopping support device 23.

[0049] The drowsiness determining unit 33b estimates a drowsy state when the driver's continuous eye-closure time exceeds a predetermined time threshold, or when the driver repeatedly closes his / her eyes for short periods exceeding the predetermined time threshold within a certain period of time. In addition, the drowsiness determination unit 33b determines the drowsy state if the driver's continuous eye closure time exceeds a predetermined time threshold, or if the continuous eye closure and no driving operation state exceeds a time threshold, while the vehicle is traveling at a certain speed or above after a drowsiness warning.

[0050] The non-operation state of the driving is determined as an inappropriate driving operation while driving by comparing the detection signal of the detection means 1 with a judgment threshold. Specifically, it is determined by at least one of the following operating states: non-operation of the steering wheel, non-grasping of the steering wheel, and non-operation of the accelerator or brake. Note that an extremely small judgment threshold may be set for each operating member to determine an inappropriate driving operation.

[0051] Next, the abnormality determination unit 33c will be described. The abnormality determination unit 33c determines that the driver has intentionally lost posture if the driver's head posture acquired by the internal camera 11 is abnormal and the three gaze determination indices of the driver calculated by the gaze determination unit 30 are all normal, and determines that the driver is in an abnormal state if the driver's head posture acquired by the internal camera 11 is abnormal and any of the three gaze determination indices of the driver calculated by the gaze determination unit 30 is abnormal.

[0052] The inventor has found that the driver's symptoms of an abnormality can be detected using eye closure as a parameter, the driver's physical state when an abnormality occurs can be detected using head posture as a parameter, and the driver's behavior when an abnormality occurs can be detected using driving inactivity as a parameter. However, head posture abnormalities can be classified into unconscious posture abnormalities caused by illness or the like and intentional posture abnormalities, and if both are determined to be head posture abnormalities, the determination accuracy will decrease. Therefore, in this embodiment, an unconscious posture error and an intentional posture error are distinguished from each other using a gaze determination index.

[0053] Next, the eye closure determination control process will be described with reference to the flowchart of FIG. Incidentally, Si (i=11, 12, . . . ) indicates a step for each process.

[0054] First, as shown in FIG. 11, the ECU 3 receives input of various information such as outputs from various detection means 1 such as the internal camera 11 and the first and second counters C1 and C2 (S11), and proceeds to S12. In S12, it is determined whether or not both eyes of the driver are closed. If the result of the determination in S12 is that the driver has his / her eyes closed, the process proceeds to S13 because closed eyes, which is a sign of an abnormality (disease onset or drowsiness), have been detected. If the result of the determination in S12 is that the driver does not have his / her eyes closed, the process returns because at least one eye is open or the reliability of the imaging information is low.

[0055] In S13, 1 is added to the current count value of the second counter C2, and then the process proceeds to S14. In S14, it is determined whether the count value of the second counter C2 is equal to or greater than the second determination threshold L2. If the result of the determination in S14 is that the count value of the second counter C2 is equal to or greater than the second determination threshold L2, the eyes are still closed, and it is estimated that the patient is experiencing a disease or is dozing, so the process proceeds to S15. If the result of the determination in S14 is that the count value of the second counter C2 is less than the second determination threshold L2, the process returns because it is currently not possible to estimate the onset of a disease or drowsiness. In S15, an alarm is issued, and then the process proceeds to S16.

[0056] In S16, it is determined whether or not the first determination condition is met. The first determination conditions are the following three conditions. (1) The driver's head posture is abnormal. (2) The driver has not operated the vehicle for the first non-operation determination period. (3) The driver's eyes are closed for a certain period of time.

[0057] If the result of the determination in S16 is that the first determination condition is met, the driver will not start driving even if a warning is issued, and therefore the process proceeds to S17. If the result of the determination in S16 is that the first determination condition is not met, the driver has started driving in response to the warning, so the process returns.

[0058] In S17, it is determined whether or not a cancel signal for stopping the alarm has been generated. If the result of the determination in S17 is that a cancel signal has been generated, the driver has performed an operation to generate a cancel signal, so the count values ​​of the first and second counters C1 and C2 are each set to 0 (S18), and then the process returns.If the result of the determination in S17 is that a cancel signal has not been generated, the driver cannot cancel, so the process proceeds to S19. In S19, the abnormality sequence control is executed, and then the process returns. In S6, 0 is substituted into the first and second counters C1 and C2, respectively.

[0059] Next, the head posture determination control process will be described with reference to the flowchart of Fig. 12. This head posture determination control process is controlled independently of and in parallel with the eye closure determination control process. Si (i=21, 22...) indicates the step for each process.

[0060] First, as shown in FIG. 12, the ECU 3 receives an input of an output from the detection means 1 such as the internal camera 11 and the hazard switch 17 (S21), and then proceeds to S22. In S22, it is determined whether the driver's head posture is out of alignment. If the result of the determination in S22 is that the driver's head posture is out of alignment, the process proceeds to S23 to distinguish between unconscious and intentional posture deviations. If the result of the determination in S22 is that the driver's head posture is not out of alignment, the process returns because the state is not abnormal.

[0061] In S23, it is determined whether the driver's line of sight is abnormal. The line-of-sight determination unit 30 determines line-of-sight abnormalities using line-of-sight determination indices (saliency index, saccade amplitude index, saccade frequency index) extracted from the driver's line of sight. If the result of the determination in S23 is that the driver has an abnormal line of sight, the driver's attention function is in an abnormal state of decline, so the process proceeds to S24. If the result of the determination in S23 is that the driver does not have an abnormal line of sight, the driver's attention function is not in a decline, so the process proceeds to S32.

[0062] In S24, it is determined whether the driver is operating the various operating parts (steering wheel, accelerator pedal, brake pedal, etc.) appropriately. If the result of the determination in S24 is that the driver is operating the vehicle appropriately, the process returns because the driver is operating the vehicle appropriately even if the head posture is out of alignment. If the result of the determination in S24 is that the driver is not operating the vehicle appropriately, for example, if the various operating units are not being operated at all, the process proceeds to S25.

[0063] In S25, it is determined whether the vehicle speed is equal to or greater than the determination threshold value. If the result of the determination in S25 is that the vehicle speed is equal to or greater than the determination threshold, Since the driver is unconsciously losing his / her posture and is not operating the vehicle even though the vehicle is moving, the process proceeds to S26. If the result of the determination in S25 is that the vehicle speed is below the determination threshold, the driver may be stopped to wait for recovery, so the process returns. In S26, 1 is added to the current count value of the first counter C1, and the process proceeds to S27.

[0064] In S27, it is determined whether the count value of the first counter C1 is equal to or greater than the first determination threshold L1. If the result of the determination in S27 is that the count value of the first counter C1 is equal to or greater than the first determination threshold L1, the onset of a disease is predicted, and the process proceeds to S28. If the result of the determination in S27 is that the count value of the first counter C1 is less than the first determination threshold L1, the onset of a disease cannot be predicted under the current circumstances, and the process returns. In S28, an alarm is issued, and then the process proceeds to S29.

[0065] In S29, it is determined whether or not a cancel signal for stopping the alarm has been generated. If the result of the determination in S29 is that a cancel signal has been generated, the driver has performed an operation to generate a cancel signal, so the count value of the first counter C1 is set to 0 (S30), and then the process returns. If the result of the determination in S29 is that a cancel signal has not been generated, the driver is not in a state where cancellation is possible, so the process proceeds to S31. In S31, abnormality sequence control is executed, and then the process returns. In this S31, the count values ​​of the first and second counters C1 and C2 are each set to 0.

[0066] In S32, since the driver's posture has been intentionally changed and this does not cause any problems in terms of consciousness or driving operation, an alarm is issued as a warning, and then the process proceeds to S33. In S33, it is determined whether a cancel signal for stopping the alarm has been generated. If the result of the determination in S33 is that a cancel signal has been generated, the driver has noticed the warning and performed an operation to generate a cancel signal, so the warning is stopped (S34), and then the process returns. If the result of the determination in S33 is that the cancel signal is not generated, the driver is not aware of the warning, so the warning continues.

[0067] Next, the operation and effects of the driver state determination method and the device M will be described. According to this driver state determination method, there are provided a peripheral information acquisition step S1 for acquiring peripheral information relating to the situation around the vehicle, a visual field image acquisition step S2 for acquiring a visual field image appearing in the visual field of the driver, an attractiveness distribution generation step S3 for generating an attractiveness distribution of the driver for the visual field image created in the visual field image acquisition step S2, a gaze behavior acquisition step S4 for acquiring the gaze behavior of the driver, and a plurality of gaze determination indices of the driver based on the peripheral information, the visual field image, the attractiveness distribution, and the gaze behavior. saliency index, saccade amplitude index, and saccade frequency index The system includes a gaze determination index calculation step S5 for calculating the gaze direction determination index, a head posture acquisition step S21 for acquiring the driver's face direction and head position, and driver abnormal state determination steps S22, S23, and S27 for determining whether the driver is in an abnormal state. Related to attention levelA plurality of gaze determination indices can be included in the conditions for determining whether the driver is in an abnormal state. The driver abnormal state determination processes S22, S23, and S27 determine that the driver has intentionally lost posture if the driver's head posture acquired in the head posture acquisition process S21 is abnormal and the driver's multiple gaze determination indices calculated in the gaze determination index calculation process S5 are all normal, making it possible to determine whether a driver has an inappropriate driving posture even if there are no problems with their consciousness or driving operation. The driver abnormal state determination processes S22, S23, and S27 determine that the driver is in an abnormal state if the driver's head posture acquired in the head posture acquisition process S21 is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process S5 is abnormal, so that unconscious posture deviations of the driver can be determined with high accuracy from the driver's posture deviations. Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally.

[0068] The driver abnormal state determination step S22 determines whether the head posture is abnormal based on at least the face direction angle of the driver, and therefore can easily determine whether the head posture of the driver is abnormal.

[0069] In the driver abnormal state determination step S27, if the head posture of the driver acquired in the head posture acquisition step S21 is abnormal and any of the gaze determination indices of the driver calculated in the gaze determination index calculation step S5 is abnormal, the driver abnormal state determination step S27 determines that the driver is in an abnormal state. This makes it possible to reliably determine whether the driver is in an abnormal state.

[0070] The gaze determination indices are a saliency index, a saccade amplitude index, and a saccade frequency index, and therefore it is possible to determine the driver's abnormal state based on the driver's level of attention.

[0071] In this driver state determination device M, which detects an abnormal state of a vehicle driver, the device M includes an external camera 10 that acquires peripheral information relating to the surrounding conditions of the vehicle, a gaze determination unit 30 corresponding to a field of view image acquisition means that acquires a field of view image that appears in the field of view of the driver, a saliency determination unit 30a that generates an attractiveness distribution of the driver for the field of view image created by the field of view image acquisition means, a saccade determination unit 30b that acquires the gaze behavior of the driver, and a plurality of gaze determination indices of the driver based on the peripheral information, the field of view image, the attractiveness distribution, and the gaze behavior. saliency index, saccade amplitude index, and saccade frequency index The system includes the line-of-sight determination unit 30, which corresponds to the line-of-sight determination index calculation means for calculating the line-of-sight determination index, the internal camera 11 for acquiring the driver's face direction and head position, and the abnormality determination unit 33c for determining whether the driver is in an abnormal state. Related to attention level A plurality of gaze determination indices can be included in the conditions for determining whether the driver is in an abnormal state. If the driver's head posture acquired by the internal camera 11 is abnormal and the multiple gaze determination indices of the driver calculated by the gaze determination unit 30, which corresponds to the gaze determination index calculation means, are all normal, the abnormality determination unit 33c determines that the driver has intentionally lost posture, and therefore can determine that a driver has an inappropriate driving posture even if there are no problems with their consciousness state or driving operation. The abnormality determination unit 33c determines whether the head posture of the driver acquired by the internal camera 11 is abnormal and whether the eye gaze determination unit 30 corresponds to the eye gaze determination index calculation unit. but If any of the calculated gaze determination indices of the driver is abnormal, the driver is determined to be in an abnormal state, so that unconscious posture deviation can be determined with high accuracy from the driver's posture deviation. Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally.

[0072] In addition, a person skilled in the art can implement the present invention in a form in which various modifications are added to the above-described embodiment without departing from the spirit of the present invention, and the present invention also includes such modifications. [Explanation of symbols]

[0073] 1. Detection Methods 10 External Camera 11 Internal Camera 30 Line of sight determination section 30a Saliency Judgment Section 30b Saccade judgment unit 33c Abnormality judgment section M Driver status determination device

Claims

1. A driver condition determination method for detecting an abnormal condition of a driver of a vehicle, comprising: performing a control process for determining whether the driver has eyes that are closed and a control process for determining a head posture of the driver in parallel based on a captured image of the driver; The eye closure determination control process determines that the driver is in an abnormal state when the driver does not respond to a warning issued when the driver's eyes remain closed, and The head posture determination control process includes: a surrounding information acquisition step of acquiring surrounding information relating to a surrounding situation of the vehicle; a field of view image acquisition step of acquiring a field of view image corresponding to the driver's field of view based on the peripheral information; an attractiveness distribution generating step of generating an attractiveness distribution of the driver with respect to the field of view image; a gaze behavior acquisition step of acquiring a gaze behavior of the driver based on the captured image and the field of view image; a gaze determination index calculation step of calculating a saliency index, a saccade amplitude index, and a saccade frequency index as a plurality of gaze determination indexes of the driver based on the peripheral information, the field of view image, the visual attraction distribution, and the gaze behavior; a head posture acquiring step of acquiring a facial orientation and a head position of the driver based on the captured image; a driver abnormal state determination step of determining whether the driver is in an abnormal state, The driver abnormal state determination step includes: determining that the driver has intentionally lost posture when the head posture of the driver acquired in the head posture acquisition step is abnormal and the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step are all normal; If the head posture of the driver acquired in the head posture acquisition step is an abnormal posture and any one of the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step is abnormal, the driver is determined to be in an abnormal state. A driver state determination method comprising:

2. 2. The method for determining a driver's condition according to claim 1, wherein the step of determining an abnormal driver condition includes determining whether the head posture is abnormal based on at least an angle of the driver's face.

3. The driver's abnormal state determination process determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition process is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process is abnormal for a predetermined period of time.

4. A driver condition determination device for detecting an abnormal condition of a vehicle driver, a surrounding information acquisition means for acquiring surrounding information relating to the surrounding conditions of the vehicle; a driving operation amount acquisition means for acquiring a driving operation amount of the driver; a field of view image acquisition means for acquiring a field of view image corresponding to the driver's field of view based on the peripheral information; an attractiveness distribution generating means for generating an attractiveness distribution of the driver with respect to the field of view image; a gaze behavior acquisition means for acquiring a gaze behavior of the driver based on an image captured by an internal camera that captures an image of the driver; a gaze determination index calculation means for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as a plurality of gaze determination indexes of the driver based on the peripheral information, the field of view image, the attractiveness distribution, and the gaze behavior; a head posture acquisition means for acquiring a facial orientation and a head position of the driver based on the captured image; a driver abnormal state determination means for determining whether the driver is in an abnormal state; the driver abnormal state determination means performs a control process for determining whether the driver has closed eyes and a control process for determining a head posture of the driver in parallel based on the captured image, In the eye closure determination control process, when the driver does not respond to a warning issued when the driver's eyes remain closed based on the driving operation amount and the captured image, it is determined that the driver is in an abnormal state; The head posture determination control process determines that the driver has intentionally lost posture if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means are all normal, and determines that the driver is in an abnormal state if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and any of the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means is abnormal.

Images