Estimation device, estimation method, and program

Abstract

This estimation device (2) comprises: an eye-closing state detection unit (12) which detects an eye-closing time of the eyes of a driver; a drowsiness determination unit (14) which determines the level of drowsiness of the driver; and a micro-sleep estimation unit (16) which estimates the driver to be in a micro-sleep state on the condition that the eye-closing time is detected as equal to or greater than a first time and smaller than a second time by the eye-closing state detection unit (12), and the level of drowsiness is determined as equal to or greater than a first threshold by the drowsiness determination unit (14).

Description

[Technical Field]

[0001] The present disclosure relates to an estimation device, an estimation method, and a program. [Background technology]

[0002] It is known that when a driver feels extremely drowsy while driving a vehicle, they may fall into a momentary sleep state known as "microsleep" before falling completely asleep. From the perspective of risk avoidance and accident prevention, a detection device for detecting such a microsleep state of a driver has been proposed (see, for example, Patent Document 1). This type of detection device detects the driver's microsleep state based on the state of closure of the driver's eyes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-508870 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional detection devices have the problem that they may erroneously detect a driver's microsleep state if the driver intentionally closes their eyes, for example, due to a contact lens slipping out of place.

[0005] Therefore, the present disclosure provides an estimation device, estimation method, and program that can accurately estimate a microsleep state. [Means for solving the problem]

[0006] An estimation device according to one embodiment of the present disclosure is an estimation device for estimating that a driver of a vehicle is in a microsleep state, and includes an eye-closed state detection unit that detects the length of time the driver's eyes are closed, a drowsiness determination unit that determines the driver's drowsiness level, and a microsleep estimation unit that estimates that the driver is in a microsleep state, on the condition that the eye-closed state detection unit detects that the eye-closed time is greater than or equal to a first time and less than a second time, and the drowsiness determination unit determines that the drowsiness level is greater than or equal to a first threshold.

[0007] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM (Compact Disc-Read Only Memory), or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0008] According to an estimation device etc. relating to one aspect of the present disclosure, a microsleep state can be estimated with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an estimation device according to a first embodiment. [Figure 2] 4 is a flowchart showing the flow of operations of the estimation device according to the first embodiment. [Figure 3] FIG. 10 is a block diagram showing the configuration of an estimation device according to a second embodiment. [Figure 4] 10 is a flowchart showing the flow of operations of the estimation device according to the second embodiment. [Figure 5] FIG. 10 is a block diagram showing the configuration of an estimation device according to a third embodiment. [Figure 6] 10 is a flowchart showing the flow of operations of the estimation device according to the third embodiment. [Figure 7]FIG. 10 is a block diagram showing the configuration of an estimation device according to a fourth embodiment. [Figure 8] 10 is a flowchart showing the flow of operations of the estimation device according to the fourth embodiment. [Figure 9] FIG. 10 is a block diagram showing the configuration of an estimation device according to a fifth embodiment. [Figure 10] 10 is a flowchart showing the flow of operations of the estimation device according to the fifth embodiment. [Figure 11] FIG. 13 is a block diagram showing the configuration of an estimation device according to a sixth embodiment. [Figure 12] 13 is a flowchart showing the flow of operations of the estimation device according to the sixth embodiment. [Figure 13] FIG. 13 is a block diagram showing the configuration of an estimation device according to a seventh embodiment. [Figure 14] 13 is a flowchart showing the flow of operations of the estimation device according to the seventh embodiment. [Figure 15] FIG. 13 is a block diagram showing the configuration of an estimation device according to an eighth embodiment. [Figure 16] 13 is a flowchart showing the flow of operations of the estimation device according to the eighth embodiment. [Figure 17] FIG. 20 is a block diagram showing the configuration of an estimation device according to a ninth embodiment. [Figure 18] 13 is a flowchart showing the flow of operations of the estimation device according to the ninth embodiment. [Figure 19] FIG. 22 is a block diagram showing the configuration of an estimation device according to a tenth embodiment. [Figure 20] 20 is a flowchart showing the flow of operations of the estimation device according to the tenth embodiment. [Figure 21] FIG. 22 is a block diagram showing the configuration of an estimation device according to an eleventh embodiment. [Figure 22] 22 is a flowchart showing the flow of operations of the estimation device according to the eleventh embodiment. [Figure 23] FIG. 22 is a block diagram showing the configuration of an estimation device according to a twelfth embodiment. [Figure 24] 22 is a flowchart showing the flow of operations of the estimation device according to the twelfth embodiment. [Figure 25] FIG. 22 is a block diagram showing the configuration of an estimation device according to a thirteenth embodiment. [Figure 26] 23 is a flowchart showing a first operation flow of the estimation device according to the thirteenth embodiment. [Figure 27] 23 is a flowchart showing a second operation flow of the estimation device according to the thirteenth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An estimation device according to a first aspect of the present disclosure is an estimation device for estimating that a driver of a vehicle is in a microsleep state, and includes an eye-closed state detection unit that detects the length of time the driver's eyes are closed, a drowsiness determination unit that determines the driver's drowsiness level, and a microsleep estimation unit that estimates that the driver is in a microsleep state, on the condition that the eye-closed state detection unit detects that the eye-closed time is greater than or equal to a first time and less than a second time, and the drowsiness determination unit determines that the drowsiness level is greater than or equal to a first threshold.

[0011] According to this aspect, the microsleep estimation unit estimates the driver's microsleep state by taking into consideration the result of the eye closure period detected by the eye closure state detection unit and the result of the drowsiness level determination by the drowsiness determination unit. This allows the microsleep state to be estimated with high accuracy.

[0012] Furthermore, in the estimation device according to the second aspect of the present disclosure, in the first aspect, the first time period may be 0.5 seconds, and the second time period may be 3 seconds.

[0013] According to this aspect, when the driver falls into a momentary sleep state lasting 0.5 seconds or more but less than 3 seconds and the drowsiness level is equal to or greater than the first threshold, it can be estimated that the driver is in a microsleep state.

[0014] In addition, in the estimation device related to the third aspect of the present disclosure, in the first or second aspect, the microsleep estimation unit may further be configured to calculate a first reliability, which is an index indicating the likelihood of the estimation result that the driver is in a microsleep state.

[0015] According to this aspect, the accuracy of estimating the microsleep state can be improved.

[0016] In addition, in the estimation device according to the fourth aspect of the present disclosure, in the third aspect, the eye closed state detection unit may further calculate a second reliability which is an index indicating the likelihood of the detection result that the eye closed time is equal to or greater than the first time and less than the second time, the drowsiness determination unit may further calculate a third reliability which is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold, and the microsleep estimation unit may be configured to calculate the first reliability based on the second reliability and the third reliability.

[0017] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0018] In addition, in the estimation device related to the fifth aspect of the present disclosure, in the third aspect, the estimation device further includes an opening / closing state detection unit that detects the opening and closing speed of the driver's eyes, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closing state detection unit detects that the eye closing time is greater than the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is greater than or equal to the first threshold, and the opening / closing state detection unit detects that the opening and closing speed is less than the second threshold.

[0019] According to this aspect, the microsleep estimation unit estimates the driver's microsleep state by taking into consideration the detection result of the eye-closure duration by the eye-closure state detection unit, the determination result of the drowsiness level by the drowsiness determination unit, and also the detection result of the opening and closing speed by the opening and closing state detection unit, thereby enabling more accurate estimation of the driver's microsleep state.

[0020] In addition, in the estimation device according to the sixth aspect of the present disclosure, in the fifth aspect, the eye closed state detection unit may further calculate a second reliability which is an index indicating the likelihood of the detection result that the eye closed time is equal to or greater than the first time and less than the second time, the drowsiness determination unit may further calculate a third reliability which is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold, the open / closed state detection unit may further calculate a fourth reliability which is an index indicating the likelihood of the detection result that the open / close speed is less than the second threshold, and the microsleep estimation unit may be configured to calculate the first reliability based on the second reliability, the third reliability, and the fourth reliability.

[0021] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0022] In addition, in the estimation device according to a seventh aspect of the present disclosure, in the fifth or sixth aspect, the drowsiness determination unit may be configured to include a function as the open / close state detection unit.

[0023] According to this aspect, the configuration of the estimation device can be simplified.

[0024] In addition, in the estimation device according to an eighth aspect of the present disclosure, in the seventh aspect, the drowsiness determination unit may be configured to further include a function as the eye-closure state detection unit.

[0025] According to this aspect, the configuration of the estimation device can be further simplified.

[0026] In addition, in the estimation device related to the ninth aspect of the present disclosure, in the third aspect, the estimation device further includes an open / closed state detection unit that detects the eye closing speed and eye opening speed of the driver, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closing state detection unit detects that the eye closing time is greater than the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is greater than or equal to the first threshold, the open / closed state detection unit detects that the eye closing speed is less than a third threshold, and the open / closed state detection unit detects that the eye opening speed is less than a fourth threshold.

[0027] According to this aspect, the microsleep estimation unit estimates the driver's microsleep state by taking into consideration the results of the eye closing speed and eye opening speed detected by the eye closing state detection unit, in addition to the results of the eye closing duration detection by the eye closing state detection unit and the results of the drowsiness level determination by the drowsiness determination unit, thereby enabling more accurate estimation of the driver's microsleep state.

[0028] In addition, in the estimation device according to the tenth aspect of the present disclosure, in the ninth aspect, the eye closed state detection unit further calculates a second reliability which is an index indicating the likelihood of the detection result that the eye closed time is equal to or greater than the first time and less than the second time, the drowsiness determination unit further calculates a third reliability which is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold, the open / closed state detection unit further calculates a fifth reliability which is an index indicating the likelihood of the detection result that the eye closed speed is less than the third threshold, and a sixth reliability which is an index indicating the likelihood of the detection result that the eye open speed is less than the fourth threshold, and the microsleep estimation unit may be configured to calculate the first reliability based on the second reliability, the third reliability, the fifth reliability, and the sixth reliability.

[0029] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0030] In addition, in the estimation device according to the eleventh aspect of the present disclosure, in the third aspect, the microsleep estimation unit may be configured to calculate the first reliability according to the drowsiness level determined by the drowsiness determination unit.

[0031] According to this aspect, it is possible to improve the accuracy of calculating the first reliability.

[0032] In addition, in the estimation device according to the twelfth aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further include a binocular detection unit that detects both eyes of the driver, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closed state detection unit detects that the eye closed time is greater than or equal to the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is greater than or equal to the first threshold, and both eyes of the driver are detected by the binocular detection unit.

[0033] According to this aspect, the microsleep estimation unit estimates the driver's microsleep state by taking into consideration the detection result of the eye closure period by the eye closure state detection unit, the drowsiness level determination result by the drowsiness determination unit, and also the detection result of the binocular detection unit, thereby enabling more accurate estimation of the microsleep state.

[0034] In addition, in the estimation device according to the thirteenth aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further include a blink count detection unit that detects the number of blinks of the driver, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closure state detection unit detects that the eye closure time is greater than or equal to the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is greater than or equal to the first threshold, and the blink count detection unit detects that the number of blinks per unit time of the driver has increased or decreased by more than a fifth threshold.

[0035] According to this aspect, the driver's microsleep state is estimated by taking into consideration the detection result of the eye closure duration by the eye closure state detection unit, the drowsiness level determination result by the drowsiness determination unit, and the detection result of the blink frequency detection unit, thereby enabling more accurate estimation of the microsleep state.

[0036] In addition, in the estimation device according to the fourteenth aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further include a life log information acquisition unit that acquires life log information related to the driver's life, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closure state detection unit detects that the eye closure time is greater than the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is less than the first threshold, and the life log information acquisition unit acquires the life log information that affects the driver's microsleep state.

[0037] According to this aspect, the driver's microsleep state is estimated by taking into consideration not only the result of eye closure time detection by the eye closure state detection unit and the result of drowsiness level determination by the drowsiness determination unit, but also the life log information acquired by the life log information acquisition unit, thereby enabling more accurate estimation of the driver's microsleep state.

[0038] In addition, in the estimation device according to the 15th aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further include a facial feature information acquisition unit that acquires facial feature information indicating the facial features of the driver, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closure state detection unit detects that the eye closure time is greater than the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is less than the first threshold, and the facial feature information acquisition unit acquires the facial feature information that affects the driver's microsleep state.

[0039] According to this aspect, the driver's microsleep state is estimated by taking into consideration the eye closure duration detection result by the eye closure state detection unit, the drowsiness level determination result by the drowsiness determination unit, and the facial feature information acquired by the facial feature information acquisition unit, thereby enabling more accurate estimation of the microsleep state.

[0040] In addition, in the estimation device according to the 16th aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further include a head movement detection unit that detects head movement of the driver, and the microsleep estimation unit may be configured to estimate that the driver is in a microsleep state under the conditions that the eye closure state detection unit detects that the eye closure time is greater than the first time and less than the second time, the drowsiness determination unit determines that the drowsiness level is less than the first threshold, and the head movement detection unit detects head movement that affects the driver's microsleep state.

[0041] According to this aspect, the driver's microsleep state is estimated by taking into consideration the detection result of the eye closure period by the eye closure state detection unit, the drowsiness level determination result by the drowsiness determination unit, and the detection result of the head movement detection unit, thereby enabling more accurate estimation of the microsleep state.

[0042] In addition, in the estimation device related to the 17th aspect of the present disclosure, in the third aspect, the estimation device further includes an erroneous estimation situation detection unit that detects situations that affect the estimation of the driver's microsleep state by the microsleep estimation unit, and the microsleep estimation unit may be configured to change the first reliability taking into account the detection result of the erroneous estimation situation detection unit.

[0043] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0044] In addition, in the estimation device related to the 18th aspect of the present disclosure, in any one of the first to third aspects, the estimation device further includes an erroneous estimation situation detection unit that detects an erroneous estimation situation, which is a situation that affects the estimation of the driver's microsleep state by the microsleep estimation unit, and the microsleep estimation unit may be configured not to estimate the driver's microsleep state when the erroneous estimation situation detection unit detects the erroneous estimation situation.

[0045] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0046] Furthermore, in the estimation device according to the 19th aspect of the present disclosure, in any one of the first to third aspects, the estimation device may further be configured to include a driving condition detection unit that detects the driving condition of the vehicle by the driver, and an eye closure time change unit that changes the first time and / or the second time used to detect the eye closure time in the eye closure state detection unit based on the driving condition detected by the driving condition detection unit.

[0047] According to this aspect, the accuracy of estimating the microsleep state can be further improved.

[0048] In addition, in the estimation device according to the twentieth aspect of the present disclosure, in any one of the first to eleventh aspects, the eye-closure state detection unit may be configured to detect the eye-closure time based on image information of the driver captured by an imaging unit, and the drowsiness determination unit may be configured to determine the drowsiness level based on biometric information of the driver detected by a biometric sensor.

[0049] According to this aspect, the driver's eye-closure time can be easily detected by using the image information of the driver captured by the imaging unit, and the driver's drowsiness level can be easily determined by using the driver's biological information detected by the biological sensor.

[0050] In addition, in the estimation device according to the 21st aspect of the present disclosure, in any one of the 1st to 11th aspects, the eye-closed state detection unit may be configured to detect the eye-closed time based on image information of the driver captured by an imaging unit, and the drowsiness determination unit may be configured to determine the drowsiness level based on the image information.

[0051] According to this aspect, by using image information of the driver captured by the imaging unit, it is possible to easily detect the time period during which the eyes are closed and to easily determine the drowsiness level.

[0052] An estimation method according to a 22nd aspect of the present disclosure is an estimation method for estimating that a driver of a vehicle is in a microsleep state, and includes the steps of (a) detecting the time the driver's eyes are closed, (b) determining the driver's drowsiness level, and (c) estimating that the driver is in a microsleep state, on condition that (a) it is detected that the time the eyes are closed is greater than or equal to a first time and less than a second time, and (b) it is determined that the drowsiness level is greater than or equal to a first threshold.

[0053] According to this aspect, the driver's microsleep state is estimated taking into consideration the detection result of the eye-closure time and the determination result of the drowsiness level, thereby enabling the microsleep state to be estimated with high accuracy.

[0054] A program according to a 23rd aspect of the present disclosure is a program that causes a computer to execute the estimation method according to the 22nd aspect described above.

[0055] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0056] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0057] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0058] (Embodiment 1) [1-1. Configuration of the estimation device] First, the configuration of an estimation device 2 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an estimation device 2 according to the first embodiment.

[0059] As shown in FIG. 1, an estimation device 2 according to the first embodiment is a device for detecting a microsleep state of a vehicle driver. The vehicle is equipped with the estimation device 2 and a sensor group 3. The vehicle is, for example, a passenger car, a bus, a truck, or other automobile. Note that the vehicle is not limited to an automobile, and may be, for example, a construction machine, an agricultural machine, or the like.

[0060] The sensor group 3 includes, for example, various sensors for detecting information about the vehicle and / or the driver sitting in the driver's seat of the vehicle, etc. Specifically, the sensor group 3 includes, for example, an imaging unit 4, a biometric sensor 6, and a vehicle state sensor 5.

[0061] The imaging unit 4 is a camera for capturing an image of a driver sitting in the driver's seat of the vehicle. For example, a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a camera using a CCD (Charge Coupled Device) image sensor can be used as the imaging unit 4. The imaging unit 4 outputs image information of the driver to the estimation device 2.

[0062] The biological sensor 6 is a sensor for detecting biological information (e.g., blood pressure, body temperature, respiratory rate, heart rate, muscle activity, etc.) of a driver sitting in the driver's seat of a vehicle. The biological sensor 6 outputs the detected biological information to the estimation device 2.

[0063] The vehicle state sensor 5 is a sensor for detecting the speed, acceleration, etc. of the vehicle. The vehicle state sensor 5 outputs vehicle state information indicating the detected speed, acceleration, etc. to the estimation device 2.

[0064] The estimation device 2 includes a sensor information acquisition unit 7, an eye closure state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16. The estimation device 2 may include one or more sensors included in the above-described sensor group 3 as constituent elements.

[0065] The sensor information acquisition unit 7 acquires various types of information output from various sensors included in the sensor group 3, and outputs the acquired various types of information to the eye-closed state detection unit 12 and the drowsiness determination unit 14. In this embodiment, a case will be described in which the sensor information acquisition unit 7 acquires, for example, image information output from the imaging unit 4, and outputs the acquired image information to the eye-closed state detection unit 12. In addition, in this embodiment, a case will be described in which the sensor information acquisition unit 7 acquires, for example, biological information output from the biological sensor 6, and outputs the acquired biological information to the drowsiness determination unit 14.

[0066] The eye-closure state detection unit 12 detects the time period during which the driver's eyes are closed based on various information from the sensor information acquisition unit 7, for example, image information from the sensor information acquisition unit 7. Specifically, the eye-closure state detection unit 12 detects the time period during which the driver's eyes are closed by analyzing an image of the driver's eyes included in the image information. Here, the eye-closure time refers to the time period during which the driver's eyes are closed, more specifically, the time period from when the driver's eyelids start to close until they close and reopen. The eye-closure state detection unit 12 outputs the detection result of the eye-closure time to the microsleep estimation unit 16. Note that in this embodiment, the eye-closure state detection unit 12 detects the time period during which the driver's eyes are closed based on image information from the sensor information acquisition unit 7. However, the present invention is not limited to this, and the time period during which the driver's eyes are closed may also be detected using, for example, deep learning.

[0067] The drowsiness determination unit 14 determines the drowsiness level indicating the degree of drowsiness of the driver based on various information from the sensor information acquisition unit 7, for example, biological information from the sensor information acquisition unit 7. The drowsiness level is expressed, for example, as a five-level numerical value from "1" to "5." The higher the numerical value of the drowsiness level, the higher the degree of drowsiness of the driver. Specifically, drowsiness level "1" is classified as not at all drowsy, drowsiness level "2" as slightly drowsy, drowsiness level "3" as drowsy, drowsiness level "4" as very drowsy, and drowsiness level "5" as very drowsy. The drowsiness determination unit 14 outputs the drowsiness level determination result to the microsleep estimation unit 16.

[0068] In the present embodiment, the drowsiness determination unit 14 determines the drowsiness level of the driver based on biological information from the sensor information acquisition unit 7. However, this is not limiting, and the drowsiness level of the driver may be determined based on image information from the sensor information acquisition unit 7. In this case, the drowsiness determination unit 14 may determine the drowsiness level of the driver by analyzing an image of the driver's eyes included in the image information, for example, based on the eyelid opening degree, which is an index indicating the degree to which the eyelids are open. Alternatively, the drowsiness determination unit 14 may determine the drowsiness level of the driver using, for example, deep learning.

[0069] The microsleep estimation unit 16 has an estimation unit 18. The estimation unit 18 estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds (an example of a first time period) or more and less than 3 seconds (an example of a second time period) and the drowsiness determination unit 14 determines that the drowsiness level is "4" (an example of a first threshold value) or more.

[0070] In this embodiment, the condition for the eye-closure time is set to 0.5 seconds or more and less than 3 seconds, but is not limited to this and may be, for example, 1 second or more and less than 4 seconds, and the first time period and the second time period can be set arbitrarily. Also, in this embodiment, the condition for the drowsiness level is set to "4" or more, but is not limited to this and may be, for example, "5" or more, and the first threshold value can be set arbitrarily.

[0071] The estimation result of the estimation unit 18 is output to, for example, a CAN (Controller Area Network) of the vehicle. As a result, for example, when it is estimated that the driver is in a microsleep state, control is performed such as sounding an alarm to wake the driver or performing a degenerate operation on the vehicle to safely stop the vehicle. Note that the degenerate operation means, for example, controlling the steering to move the vehicle to the edge of the road (shoulder) or controlling the engine or brakes to decelerate the vehicle.

[0072] [1-2. Operation of the estimation device] Next, the operation of the estimation device 2 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of the operation of the estimation device 2 according to the first embodiment.

[0073] 2, the sensor information acquisition unit 7 acquires image information output from the imaging unit 4 (S11), and outputs the acquired image information to the eyes closed state detection unit 12. The sensor information acquisition unit 7 also acquires biological information output from the biological sensor 6 (S11), and outputs the acquired biological information to the drowsiness determination unit 14.

[0074] The eye-closure state detection unit 12 detects the eye-closure time based on the image information from the sensor information acquisition unit 7, and outputs the detection result of the eye-closure time to the microsleep estimation unit 16. In addition, the drowsiness determination unit 14 determines the drowsiness level based on the biological information from the sensor information acquisition unit 7, and outputs the determination result of the drowsiness level to the microsleep estimation unit 16.

[0075] When the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds (S12), and the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (S13), the estimation unit 18 of the microsleep estimation unit 16 estimates that the driver is in a microsleep state (S14).

[0076] [1-3.Effects] In this embodiment, the estimation unit 18 of the microsleep estimation unit 16 estimates that the driver is in a microsleep state by taking into account the time the driver's eyes are closed and the driver's drowsiness level. This allows for accurate estimation of the microsleep state.

[0077] (Embodiment 2) [2-1. Configuration of the estimation device] The configuration of an estimation device 2A according to embodiment 2 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of an estimation device 2A according to embodiment 2. Note that in this embodiment, the same components as those in embodiment 1 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0078] 3, an estimation device 2A according to the second embodiment includes an image information acquisition unit 8 and a biometric information acquisition unit 10 instead of the sensor information acquisition unit 7 described in the first embodiment. Note that in the following embodiments, an example will be described in which a microsleep state is estimated using the image capture unit 4 and biometric sensor 6 of the sensor group 3 shown in FIG. 1 described above.

[0079] The image information acquisition unit 8 acquires the image information output from the imaging unit 4. The image information acquisition unit 8 outputs the acquired image information to the eye closure state detection unit 12.

[0080] The biological information acquiring unit 10 acquires the biological information output from the biological sensor 6. The biological information acquiring unit 10 outputs the acquired biological information to the drowsiness determining unit 14.

[0081] The eye-closure state detection unit 12 detects the time period during which the driver's eyes are closed based on the image information from the image information acquisition unit 8. In the present embodiment, the eye-closure state detection unit 12 detects the time period during which the driver's eyes are closed based on the image information from the image information acquisition unit 8, but is not limited to this. For example, the eye-closure state detection unit 12 may detect the time period during which the driver's eyes are closed using deep learning or the like. Alternatively, the eye-closure state detection unit 12 may detect the time period during which the driver's eyes are closed based on biometric information from the biometric information acquisition unit 10. In this case, the biometric information from the biometric information acquisition unit 10 may be, for example, biometric information output from a biosensor 6 such as an electromyographic sensor.

[0082] The drowsiness determination unit 14 determines the drowsiness level indicating the degree of drowsiness of the driver based on the biological information from the biological information acquisition unit 10.

[0083] In the present embodiment, the drowsiness determination unit 14 determines the drowsiness level of the driver based on the biological information from the biological information acquisition unit 10. However, this is not limiting, and the drowsiness level of the driver may be determined based on image information from the image information acquisition unit 8. In this case, the drowsiness determination unit 14 may determine the drowsiness level of the driver by analyzing an image of the driver's eyes included in the image information, for example, based on the eyelid opening degree, which is an index indicating the degree to which the eyelids are open. Alternatively, the drowsiness determination unit 14 may determine the drowsiness level of the driver using, for example, deep learning.

[0084] Furthermore, in the estimation device 2A according to the second embodiment, the configuration of the microsleep estimation unit 16A is different from that of the above-described first embodiment. Specifically, the microsleep estimation unit 16A has a reliability calculation unit 20 in addition to the estimation unit 18 described in the above-described first embodiment.

[0085] The reliability calculation unit 20 calculates a reliability (an example of a first reliability) that is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state. The reliability is calculated in two stages, for example, "low" and "high." The reliability calculation unit 20 may calculate the reliability according to the drowsiness level determined by the drowsiness determination unit 14, and may calculate the reliability so that, for example, the higher the drowsiness level, the higher the reliability. Note that, although the microsleep estimation unit 16A includes the reliability calculation unit 20 in the present embodiment, this is not limiting, and the reliability calculation unit 20 may be omitted.

[0086] The estimation result of the estimation unit 18 and the calculation result of the reliability calculation unit 20 are output to, for example, the vehicle's CAN. As a result, for example, if the driver is estimated to be in a microsleep state and the reliability is "low," control is performed to sound an alarm to wake the driver. Also, for example, if the driver is estimated to be in a microsleep state and the reliability is "high," control is performed to perform a degenerate operation of the vehicle to safely stop the vehicle.

[0087] [2-2. Operation of the estimation device] Next, the operation of the estimating device 2A according to the second embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the operation of the estimating device 2A according to the second embodiment.

[0088] 4, the image information acquisition unit 8 acquires image information output from the imaging unit 4 (S101), and outputs the acquired image information to the eyes closed state detection unit 12. In addition, the biological information acquisition unit 10 acquires biological information output from the biological sensor 6 (S101), and outputs the acquired biological information to the drowsiness determination unit 14.

[0089] The eye-closure state detection unit 12 detects the eye-closure time based on the image information from the image information acquisition unit 8, and outputs the detection result of the eye-closure time to the microsleep estimation unit 16A. In addition, the drowsiness determination unit 14 determines the drowsiness level based on the biological information from the biological information acquisition unit 10, and outputs the determination result of the drowsiness level to the microsleep estimation unit 16A.

[0090] If the eye closure state detection unit 12 detects that the eye closure time is less than 0.5 seconds or 3 seconds or more (NO in S102), the estimation unit 18 of the microsleep estimation unit 16A estimates that the driver is not in a microsleep state (S103). In this case, the reliability calculation unit 20 of the microsleep estimation unit 16A does not calculate the reliability.

[0091] Returning to step S102, if the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102) and the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S104), the reliability calculation unit 20 of the microsleep estimation unit 16A calculates the reliability to be "high" (S105), and the estimation unit 18 of the microsleep estimation unit 16A estimates that the driver is in a microsleep state (S106).

[0092] Returning to step S104, if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the reliability calculation unit 20 of the microsleep estimation unit 16A calculates the reliability as "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16A estimates that the driver is in a microsleep state (S106).

[0093] The microsleep estimation unit 16A may not have the reliability calculation unit 20, in which case steps S105 and S107 may be omitted. In this case, if the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S104), the process proceeds to step S106, and the estimation unit 18 may estimate that the driver is in a microsleep state. Also, if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the process proceeds to step S103, and the estimation unit 18 may estimate that the driver is not in a microsleep state.

[0094] [2-3. Effects] In this embodiment, the estimation unit 18 of the microsleep estimation unit 16A estimates that the driver is in a microsleep state by taking into account the time the driver's eyes are closed and the driver's drowsiness level. As a result, even if the driver intentionally closes their eyes momentarily, for example, due to a contact lens slipping, if the driver's drowsiness level is low, the reliability of the estimation result that the driver is in a microsleep state can be calculated to be low. As a result, the microsleep state can be estimated with high accuracy.

[0095] (Embodiment 3) [3-1. Configuration of the estimation device] The configuration of an estimation device 2B according to embodiment 3 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of an estimation device 2B according to embodiment 3. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and their description will be omitted.

[0096] As shown in FIG. 5, in an estimation device 2B according to the third embodiment, the configurations of an eye-closed state detection unit 12B, a drowsiness determination unit 14B, and a microsleep estimation unit 16B are different from those in the first embodiment.

[0097] The eye-closure state detection unit 12B includes an eye-closure duration detection unit 22 and a reliability calculation unit 24. The eye-closure duration detection unit 22 detects the duration of time the driver's eyes are closed based on image information from the image information acquisition unit 8. The eye-closure duration detection unit 12B may detect the duration of time the driver's eyes are closed using, for example, deep learning. The eye-closure duration detection unit 22 outputs the detection result of the eye-closure duration to the microsleep estimation unit 16B. The reliability calculation unit 24 calculates a reliability (an example of a second reliability) that is an index indicating the likelihood of the detection result of the eye-closure duration detection unit 22 that the eye-closure duration is 0.5 seconds or more and less than 3 seconds. The reliability calculation unit 24 also calculates a reliability that is an index indicating the likelihood of the detection result of the eye-closure duration detection unit 22 that the eye-closure duration is less than 0.5 seconds or 3 seconds or more. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 24 outputs the calculation result of the reliability to the microsleep estimation unit 16B.

[0098] The drowsiness determination unit 14B includes a drowsiness level determination unit 26 and a reliability calculation unit 28. The drowsiness level determination unit 26 determines the drowsiness level indicating the degree of drowsiness of the driver based on the biological information from the biological information acquisition unit 10. The drowsiness level determination unit 26 may determine the drowsiness level of the driver using, for example, deep learning. The drowsiness level determination unit 26 outputs the drowsiness level determination result to the microsleep estimation unit 16B. The reliability calculation unit 28 calculates a reliability (an example of a third reliability) that is an index indicating the likelihood of the determination result of the drowsiness level determination unit 26 that the drowsiness level is "4" or higher. The reliability calculation unit 28 also calculates a reliability that is an index indicating the likelihood of the determination result of the drowsiness level determination unit 26 that the drowsiness level is less than "4". The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 28 outputs the reliability calculation result to the microsleep estimation unit 16B.

[0099] The reliability calculation unit 20B of the microsleep estimation unit 16B calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye-closed state detection unit 12B and the calculation result by the reliability calculation unit 28 of the drowsiness determination unit 14B. Furthermore, the reliability calculation unit 20B of the microsleep estimation unit 16B calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is not in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye-closed state detection unit 12B. The reliability is calculated as a numerical value between 0% and 100%, for example.

[0100] [3-2. Operation of the estimation device] Next, the operation of the estimating device 2B according to the third embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the operation of the estimating device 2B according to the third embodiment.

[0101] 6, the image information acquisition unit 8 acquires image information output from the imaging unit 4 (S201) and outputs the acquired image information to the eyes closed state detection unit 12B. In addition, the biological information acquisition unit 10 acquires biological information output from the biological sensor 6 (S201) and outputs the acquired biological information to the drowsiness determination unit 14B.

[0102] The eye-closure duration detection unit 22 of the eye-closure state detection unit 12B detects the eye-closure duration based on the image information from the image information acquisition unit 8, and outputs the detection result of the eye-closure duration to the microsleep estimation unit 16B. In addition, the drowsiness level determination unit 26 of the drowsiness determination unit 14B determines the drowsiness level based on the biological information from the biological information acquisition unit 10, and outputs the determination result of the drowsiness level to the microsleep estimation unit 16B.

[0103] If the eye-closure duration detection unit 22 detects that the eye-closure duration is less than 0.5 seconds or greater than 3 seconds (NO in S202), the reliability calculation unit 24 of the eye-closure state detection unit 12B calculates a reliability index indicating the likelihood of the detection result by the eye-closure duration detection unit 22 that the eye-closure duration is less than 0.5 seconds or greater than 3 seconds (S203). The estimation unit 18 of the microsleep estimation unit 16B then estimates that the driver is not in a microsleep state (S204). The reliability calculation unit 20B of the microsleep estimation unit 16B calculates a reliability index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is not in a microsleep state based on the calculation result by the reliability calculation unit 24 of the eye-closure state detection unit 12B (S205).

[0104] Returning to step S202, if the eye-closing time detection unit 22 detects that the eye-closing time is 0.5 seconds or more and less than 3 seconds (YES in S202), the reliability calculation unit 24 of the eye-closing state detection unit 12B calculates the reliability, which is an index showing the likelihood of the detection result by the eye-closing time detection unit 22 that the eye-closing time is 0.5 seconds or more and less than 3 seconds (S206).

[0105] Thereafter, when the drowsiness level determination unit 26 determines that the drowsiness level is "4" or higher (YES in S207), the reliability calculation unit 28 of the drowsiness determination unit 14B calculates a reliability that is an index indicating the likelihood of the determination result of the drowsiness level determination unit 26 that the drowsiness level is "4" or higher (S208). The estimation unit 18 of the microsleep estimation unit 16B estimates that the driver is in a microsleep state (S209), and the reliability calculation unit 20B of the microsleep estimation unit 16B calculates a reliability that is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in a microsleep state based on the calculation result of the reliability calculation unit 24 of the eye-closure state detection unit 12B and the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B (S210).

[0106] Returning to step S207, if the drowsiness level determination unit 26 determines that the drowsiness level is less than "4" (NO in S207), the reliability calculation unit 28 of the drowsiness determination unit 14B calculates the reliability, which is an index showing the likelihood of the determination result of the drowsiness level determination unit 26 that the drowsiness level is less than "4" (S211), and proceeds to the above-mentioned step S209.

[0107] [3-3. Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0108] (Fourth embodiment) [4-1. Configuration of the estimation device] The configuration of an estimation device 2C according to embodiment 4 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of an estimation device 2C according to embodiment 4. Note that in this embodiment, the same components as those in embodiment 3 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0109] As shown in Figure 7, the estimation device 2C of embodiment 4 includes an image information acquisition unit 8, a biometric information acquisition unit 10, an eye closed state detection unit 12B, a drowsiness determination unit 14B, and a microsleep estimation unit 16C, as well as an open / closed state detection unit 30.

[0110] The opening / closing state detection unit 30 includes an opening / closing speed calculation unit 32, an opening / closing speed determination unit 34, and a reliability calculation unit 36. The opening / closing speed calculation unit 32 calculates (detects) the opening / closing speed of the driver's eyes based on image information from the image information acquisition unit 8. The opening / closing speed calculation unit 32 may calculate the opening / closing speed of the driver's eyes using, for example, deep learning. The opening / closing speed determination unit 34 determines whether the calculated opening / closing speed is less than a threshold value (an example of a second threshold value). The reliability calculation unit 36 ​​calculates a reliability (an example of a fourth reliability) that is an index indicating the likelihood of the determination result by the opening / closing speed determination unit 34 that the calculated opening / closing speed is less than the threshold value. The reliability calculation unit 36 ​​also calculates a reliability that is an index indicating the likelihood of the determination result by the opening / closing speed determination unit 34 that the calculated opening / closing speed is equal to or greater than the threshold value. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 36 ​​outputs the reliability calculation result to the microsleep estimation unit 16C.

[0111] The estimation unit 18 of the microsleep estimation unit 16C estimates that the driver is in a microsleep state under the conditions that the eye closure state detection unit 12B detects that the eye closure time is greater than or equal to 0.5 seconds and less than 3 seconds, the drowsiness determination unit 14B determines that the drowsiness level is "4" or higher, and the opening / closing state detection unit 30 detects that the opening / closing speed is less than a threshold value.

[0112] The reliability calculation unit 20C of the microsleep estimation unit 16C calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye closed state detection unit 12B, the calculation result by the reliability calculation unit 28 of the drowsiness determination unit 14B, and the calculation result by the reliability calculation unit 36 ​​of the open / closed state detection unit 30. Furthermore, the reliability calculation unit 20C of the microsleep estimation unit 16C calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is not in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye closed state detection unit 12B. The reliability is calculated as a numerical value between 0% and 100%, for example.

[0113] In the present embodiment, the opening / closing state detection unit 30 includes the opening / closing speed calculation unit 32 and the opening / closing speed determination unit 34. However, the present invention is not limited to this, and the drowsiness determination unit 14B may include the opening / closing speed calculation unit 32 and the opening / closing speed determination unit 34. In other words, the drowsiness determination unit 14B may include the function of the opening / closing state detection unit 30.

[0114] Alternatively, the drowsiness determination unit 14B may have the eye-closing time detection unit 22 in addition to the opening / closing speed calculation unit 32 and the opening / closing speed determination unit 34. That is, the drowsiness determination unit 14B may include the functions of the eye-closing state detection unit 12B and the open / closed state detection unit 30.

[0115] [4-2. Operation of the estimation device] Next, the operation of the estimation device 2C according to the fourth embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the operation of the estimation device 2C according to the fourth embodiment. In the flowchart of Fig. 8, the same processes as those in the flowchart of Fig. 6 described above are assigned the same step numbers, and their description will be omitted.

[0116] 8, steps S201 to S208 and S211 are executed in the same manner as in above-described embodiment 3. After step S208 or step S211, if the open / close state detection unit 30 detects an open / close speed that is less than the threshold (i.e., the open / close speed determination unit 34 of the open / close state detection unit 30 determines that the open / close speed is less than the threshold) (YES in S301), the reliability calculation unit 36 ​​of the open / close state detection unit 30 calculates a reliability that is an index showing the likelihood of the determination result by the open / close speed determination unit 34 that the detected open / close speed is less than the threshold (S302).

[0117] Thereafter, the estimation unit 18 of the microsleep estimation unit 16C estimates that the driver is in a microsleep state (S209), and the reliability calculation unit 20C of the microsleep estimation unit 16C calculates a reliability, which is an index showing the likelihood of the estimation result of the estimation unit 18 that the driver is in a microsleep state, based on the calculation result of the reliability calculation unit 24 of the eye closed state detection unit 12B, the calculation result of the reliability calculation unit 28 of the drowsiness judgment unit 14B, and the calculation result of the reliability calculation unit 36 ​​of the open / closed state detection unit 30 (S210).

[0118] Returning to step S301, if the open / close state detection unit 30 detects an open / close speed equal to or greater than the threshold (i.e., the open / close speed determination unit 34 of the open / close state detection unit 30 determines that the open / close speed is equal to or greater than the threshold) (NO in S301), the reliability calculation unit 36 ​​of the open / close state detection unit 30 calculates a reliability that is an index showing the likelihood of the determination result by the open / close speed determination unit 34 that the detected open / close speed is equal to or greater than the threshold (S303). Then, the process proceeds to step S209 described above.

[0119] The eye-closure state detection unit 12B, the drowsiness determination unit 14B, and the open / close state detection unit 30 may not include the reliability calculation unit 24, the reliability calculation unit 28, and the reliability calculation unit 36, respectively. In this case, steps S203, S206, S208, S211, S302, and S303 may be omitted. In this case, instead of step S210 described above, when the eye-closure state detection unit 12B detects that the eye-closure time is 0.5 seconds or more and less than 3 seconds (YES in S202), the drowsiness determination unit 14B determines that the drowsiness level is "4" or more (YES in S207), and the open / close state detection unit 30 detects that the opening / closing speed is less than the threshold (YES in S301), the reliability calculation unit 20C of the microsleep estimation unit 16C may calculate the reliability to be "high." On the other hand, if the drowsiness determination unit 14B determines that the drowsiness level is less than "4" (NO in S207), or if the opening / closing state detection unit 30 detects that the opening / closing speed is greater than or equal to the threshold (NO in S301), the reliability calculation unit 20C of the microsleep estimation unit 16C may calculate the reliability to be "low".

[0120] [4-3. Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0121] (Embodiment 5) [5-1. Configuration of the estimation device] The configuration of an estimation device 2D according to embodiment 5 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the configuration of an estimation device 2D according to embodiment 5. Note that in this embodiment, the same components as those in embodiment 3 above are denoted by the same reference numerals, and their description will be omitted.

[0122] As shown in Figure 9, the estimation device 2D of embodiment 5 includes an image information acquisition unit 8, a biometric information acquisition unit 10, an eye closed state detection unit 12B, a drowsiness determination unit 14B, and a microsleep estimation unit 16D, as well as an open / closed state detection unit 38.

[0123] The open / closed state detection unit 38 includes an eye-closing speed calculation unit 40, an eye-closing speed determination unit 42, an eye-opening speed calculation unit 44, an eye-opening speed determination unit 46, and a reliability calculation unit 48. The eye-closing speed calculation unit 40 calculates (detects) the eye-closing speed of the driver based on image information from the image information acquisition unit 8. The eye-closing speed calculation unit 40 may calculate the eye-closing speed of the driver using, for example, deep learning. The eye-closing speed determination unit 42 determines whether the calculated eye-closing speed is less than an eye-closing threshold (an example of a third threshold). The eye-opening speed calculation unit 44 calculates (detects) the eye-opening speed of the driver based on image information from the image information acquisition unit 8. The eye-opening speed calculation unit 44 may calculate the eye-opening speed of the driver using, for example, deep learning. The eye-opening speed determination unit 46 determines whether the calculated eye-opening speed is less than an eye-opening threshold (an example of a fourth threshold).

[0124] The reliability calculation unit 48 calculates a reliability (an example of a fifth reliability) that is an index indicating the likelihood of the detection result that the calculated eye closing speed is less than the eye closing threshold. The reliability calculation unit 48 also calculates a reliability (an example of a sixth reliability) that is an index indicating the likelihood of the detection result that the calculated eye opening speed is less than the eye opening threshold. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 48 outputs the reliability calculation result to the microsleep estimation unit 16D.

[0125] The estimation unit 18 of the microsleep estimation unit 16D estimates that the driver is in a microsleep state under the conditions that the eye closure state detection unit 12B detects that the eye closure time is greater than or equal to 0.5 seconds and less than 3 seconds, the drowsiness determination unit 14B determines that the drowsiness level is "4" or higher, the open / close state detection unit 38 detects that the eye closure speed is less than the eye closure threshold, and the open / close state detection unit 38 detects that the eye opening speed is less than the eye opening threshold.

[0126] The reliability calculation unit 20D of the microsleep estimation unit 16D calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye closed state detection unit 12B, the calculation result by the reliability calculation unit 28 of the drowsiness determination unit 14B, and the calculation result by the reliability calculation unit 48 of the open / closed state detection unit 38. Furthermore, the reliability calculation unit 20D of the microsleep estimation unit 16D calculates reliability, which is an index indicating the likelihood of the estimation result by the estimation unit 18 that the driver is not in a microsleep state, based on the calculation result by the reliability calculation unit 24 of the eye closed state detection unit 12B. The reliability is calculated as a numerical value between 0% and 100%, for example.

[0127] [5-2. Operation of the estimation device] Next, the operation of the estimation device 2D according to embodiment 5 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the operation of the estimation device 2D according to embodiment 5. In the flowchart of Fig. 10, the same processes as those in the flowchart of Fig. 6 described above are assigned the same step numbers, and their description will be omitted.

[0128] 10, steps S201 to S208 and S211 are executed in the same manner as in embodiment 3. After step S208 or step S211, if the open / closed state detection unit 38 detects an eye closing speed below the eye closing threshold and an eye opening speed below the eye opening threshold (YES in S401), the reliability calculation unit 48 of the open / closed state detection unit 38 calculates reliability, which is an index showing the likelihood of the detection result that the detected eye closing speed is below the eye closing threshold and the detection result that the detected eye opening speed is below the eye opening threshold (S402).

[0129] Thereafter, the estimation unit 18 of the microsleep estimation unit 16D estimates that the driver is in a microsleep state (S209), and the reliability calculation unit 20D of the microsleep estimation unit 16D calculates a reliability, which is an index showing the likelihood of the estimation result of the estimation unit 18 that the driver is in a microsleep state, based on the calculation result of the reliability calculation unit 24 of the eye closed state detection unit 12B, the calculation result of the reliability calculation unit 28 of the drowsiness judgment unit 14B, and the calculation result of the reliability calculation unit 48 of the open / closed state detection unit 38 (S210).

[0130] Returning to step S401, if the open / closed state detection unit 38 detects an eye closing speed below the eye closing threshold and an eye opening speed equal to or greater than the eye opening threshold (NO in S401, YES in S403), the reliability calculation unit 48 of the open / closed state detection unit 38 calculates a reliability that is an index showing the likelihood of the detection result that the detected eye closing speed is below the eye closing threshold and the detection result that the detected eye opening speed is equal to or greater than the eye opening threshold (S404). Note that the reliability calculated in step S404 is lower than the reliability calculated in step S402. Thereafter, the process proceeds to step S209 described above.

[0131] Returning to step S401, if the open / closed state detection unit 38 detects an eye closing speed equal to or greater than the eye closing threshold and an eye opening speed less than the eye opening threshold (NO in S401, NO in S403, YES in S405), the reliability calculation unit 48 of the open / closed state detection unit 38 calculates a reliability, which is an index indicating the likelihood of the detection result that the detected eye closing speed is equal to or greater than the eye closing threshold and the detection result that the detected eye opening speed is less than the eye opening threshold (S406). Note that the reliability calculated in step S406 is lower than the reliability calculated in step S404. Alternatively, the reliability calculated in step S406 may be the same as the reliability calculated in step S404. Thereafter, the process proceeds to step S209 described above.

[0132] Returning to step S401, if the open / closed state detection unit 38 detects an eye closing speed equal to or greater than the eye closing threshold and an eye opening speed equal to or greater than the eye opening threshold (NO in S401, NO in S403, NO in S405, S407), the reliability calculation unit 48 of the open / closed state detection unit 38 calculates a reliability that is an index showing the likelihood of the detection result that the detected eye closing speed is equal to or greater than the eye closing threshold and the detection result that the detected eye opening speed is equal to or greater than the eye opening threshold (S408). The reliability calculated in step S408 is lower than the reliability calculated in step S406. Then, the process proceeds to step S209 described above.

[0133] [5-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0134] (Sixth embodiment) [6-1. Configuration of the estimation device] The configuration of an estimation device 2E according to embodiment 6 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the configuration of an estimation device 2E according to embodiment 6. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0135] As shown in FIG. 11, in an estimation device 2E according to the sixth embodiment, the configurations of an eye-closed state detection unit 12E and a microsleep estimation unit 16E are different from those in the second embodiment.

[0136] The eye-closure state detection unit 12E has an eye-closure time detection unit 22 and a both-eye detection unit 50. The eye-closure time detection unit 22 is the same as the eye-closure time detection unit 22 described in the third embodiment above. The both-eye detection unit 50 detects both eyes of the driver based on image information from the image information acquisition unit 8. The both-eye detection unit 50 outputs the detection result to the microsleep estimation unit 16E. Note that the both-eye detection unit 50 may also detect both eyes of the driver based on biometric information from the biometric information acquisition unit 10.

[0137] The estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in a microsleep state when the eye closure time detection unit 22 detects that the eye closure time is between 0.5 seconds and 3 seconds, the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher, and the binocular detection unit 50 detects both of the driver's eyes.

[0138] The reliability calculation unit 20E of the microsleep estimation unit 16E calculates the reliability, which is an index showing the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state. The reliability is calculated in three levels, for example, "low," "medium," and "high."

[0139] [6-2. Operation of the estimation device] Next, the operation of the estimation device 2E according to the sixth embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of the operation of the estimation device 2E according to the sixth embodiment. In the flowchart of Fig. 12, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0140] 12, first, step S101 is executed as in the second embodiment, and then, if the binocular detection unit 50 detects both eyes of the driver (YES in S501), steps S102 to S106 are executed as in the second embodiment. That is, the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in a microsleep state (S106) on the condition that the binocular detection unit 50 detects both eyes of the driver (YES in S501), the eye closure time detection unit 22 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102), and the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S104).

[0141] Returning to step S501, if the binocular detection unit 50 detects only one of the driver's eyes, for example, because one of the driver's eyes is hidden by bangs or the driver is wearing an eye patch (NO in S501, YES in S502), the process proceeds to step S503. If the eye-closure time detection unit 22 detects that the eye-closure time is less than 0.5 seconds or 3 seconds or more (NO in S503), the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is not in a microsleep state (S103). In this case, the reliability calculation unit 20E of the microsleep estimation unit 16E does not calculate the reliability.

[0142] Returning to step S503, if the eye closure time detection unit 22 detects that the eye closure time is greater than or equal to 0.5 seconds and less than 3 seconds (YES in S503), and the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S504), the reliability calculation unit 20E of the microsleep estimation unit 16E calculates the reliability to be "medium" (S505), and the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in a microsleep state (S106).

[0143] Returning to step S504, if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S504), the reliability calculation unit 20E of the microsleep estimation unit 16E calculates the reliability to be "low" (S506), and the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in a microsleep state (S106).

[0144] [6-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0145] (Embodiment 7) [7-1. Configuration of the estimation device] The configuration of an estimation device 2F according to embodiment 7 will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the configuration of an estimation device 2F according to embodiment 7. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0146] As shown in FIG. 13 , the estimation device 2F according to the seventh embodiment includes a blink count detection unit 52 in addition to the image information acquisition unit 8, the biological information acquisition unit 10, the eye closure state detection unit 12, the drowsiness determination unit 14, and the microsleep estimation unit 16F. The blink count detection unit 52 detects the number of blinks of the driver per unit time (for example, per minute) based on the image information from the image information acquisition unit 8. The blink count detection unit 52 outputs the detection result to the microsleep estimation unit 16F. Note that the blink count detection unit 52 may also detect the number of blinks of the driver per unit time based on the biological information from the biological information acquisition unit 10.

[0147] The estimation unit 18 of the microsleep estimation unit 16F estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds, the drowsiness determination unit 14 determines that the drowsiness level is 4 or higher, and the blink count detection unit 52 detects that the number of blinks per unit time of the driver has increased or decreased by a predetermined number (e.g., 10 times / minute) (an example of a fifth threshold) or more. This is because when people become drowsy, the number of blinks per unit time often increases or decreases.

[0148] [7-2. Operation of the estimation device] Next, the operation of the estimation device 2F according to the seventh embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of the operation of the estimation device 2F according to the seventh embodiment. In the flowchart of Fig. 14, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0149] 14, first, steps S101 to S104 are executed in the same manner as in the above-described embodiment 2. When the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102) and when the drowsiness determination unit 14 determines that the drowsiness level is "4" or more (YES in S104), the blink count detection unit 52 detects the number of blinks of the driver per unit time.

[0150] If the number of blinks per unit time increases or decreases by more than a predetermined number (YES in S601), the reliability calculation unit 20F of the microsleep estimation unit 16F calculates the reliability as "high" (S105), and the estimation unit 18 of the microsleep estimation unit 16F estimates that the driver is in a microsleep state (S106).

[0151] Returning to step S601, if the number of blinks per unit time has not increased or decreased by more than a predetermined number (NO in S601), the reliability calculation unit 20F of the microsleep estimation unit 16F calculates the reliability as "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16F estimates that the driver is in a microsleep state (S106).

[0152] [7-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0153] (Embodiment 8) [8-1. Configuration of the estimation device] The configuration of an estimation device 2G according to embodiment 8 will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the configuration of an estimation device 2G according to embodiment 8. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0154] As shown in FIG. 15 , the estimation device 2G according to the eighth embodiment includes a life log information acquisition unit 54 in addition to the image information acquisition unit 8, the biological information acquisition unit 10, the eye closure state detection unit 12, the drowsiness determination unit 14, and the microsleep estimation unit 16G. The life log information acquisition unit 54 acquires life log information related to the driver's life based on the image information from the image information acquisition unit 8. The life log information is, for example, information indicating the driver's body type (thin, obese, etc.). The life log information acquisition unit 54 outputs the acquired life log information to the microsleep estimation unit 16G. Note that the life log information acquisition unit 54 may acquire the life log information based on the biological information from the biological information acquisition unit 10.

[0155] The estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds, the drowsiness determination unit 14 determines that the drowsiness level is less than "4," and the life log information acquisition unit 54 acquires life log information that affects the driver's microsleep state. Here, life log information that affects the driver's microsleep state is, for example, information indicating that the driver is obese. This is because obese people are more likely to develop sleep apnea syndrome and fall asleep suddenly.

[0156] The reliability calculation unit 20G of the microsleep estimation unit 16G calculates the reliability, which is an index showing the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state. The reliability is calculated in three levels, for example, "low," "medium," and "high."

[0157] In the present embodiment, the life log information acquiring unit 54 acquires the life log information based on the image information from the image information acquiring unit 8, but this is not limiting. The life log information acquiring unit 54 may acquire the life log information, for example, from a wearable device worn by the driver, or from a cloud server via a network. In this case, the life log information is information indicating, for example, (a) the driver's medical history, (b) the driver's working hours on the previous day, (c) the driver's working style (e.g., night shift), (d) the driver's exercise time, (e) the driver's sleeping time, (f) the driver's sleep habits (e.g., whether the driver is a long sleeper), (g) the driver's medication history, etc.

[0158] [8-2. Operation of the estimation device] Next, the operation of the estimating device 2G according to the eighth embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the flow of the operation of the estimating device 2G according to the eighth embodiment. In the flowchart of Fig. 16, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0159] 16, first, steps S101 to S104 are executed in the same manner as in the above-described embodiment 2. If the eye closure state detection unit 12 detects that the eye closure time is equal to or longer than 0.5 seconds and shorter than 3 seconds (YES in S102), and if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the process proceeds to step S701, and the life log information acquisition unit 54 acquires life log information.

[0160] When the life log information acquisition unit 54 acquires life log information that affects the driver's microsleep state (YES in S701), the reliability calculation unit 20G of the microsleep estimation unit 16G calculates the reliability to be "medium" (S702), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in a microsleep state (S106).

[0161] That is, the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in a microsleep state under the following conditions: the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds (YES in S102); the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104); and the life log information acquisition unit 54 acquires life log information that affects the driver's microsleep state (YES in S701). This is because even if the driver does not feel very drowsy, there is a possibility that the driver may close his or her eyes due to a constitution that makes him or her prone to falling asleep suddenly.

[0162] Returning to step S701, if the life log information acquisition unit 54 does not acquire life log information that affects the driver's microsleep state (NO in S701), the reliability calculation unit 20G of the microsleep estimation unit 16G calculates the reliability to be "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in a microsleep state (S106).

[0163] [8-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0164] (Embodiment 9) [9-1. Configuration of the estimation device] The configuration of an estimation device 2H according to embodiment 9 will be described with reference to Fig. 17. Fig. 17 is a block diagram showing the configuration of an estimation device 2H according to embodiment 9. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0165] As shown in FIG. 17 , the estimation device 2H according to the ninth embodiment includes a facial feature information acquisition unit 56 in addition to the image information acquisition unit 8, the biometric information acquisition unit 10, the eye closure state detection unit 12, the drowsiness determination unit 14, and the microsleep estimation unit 16H. The facial feature information acquisition unit 56 acquires facial feature information indicating the facial features of the driver in time series based on the image information from the image information acquisition unit 8. The facial features refer to facial parts such as the eyes and mouth. The facial feature information acquisition unit 56 outputs the acquired facial feature information to the microsleep estimation unit 16H. Note that the facial feature information acquisition unit 56 may acquire the facial feature information based on the biometric information from the biometric information acquisition unit 10.

[0166] The estimation unit 18 of the microsleep estimation unit 16H estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds, the drowsiness determination unit 14 determines that the drowsiness level is less than "4," and the facial feature information acquisition unit 56 acquires facial feature information that affects the driver's microsleep state. Here, the facial feature information that affects the driver's microsleep state is, for example, information that indicates that the driver's mouth is open due to drowsiness, or that the eyebrows are not moving and the muscles around the eyes are relaxed due to drowsiness.

[0167] The reliability calculation unit 20H of the microsleep estimation unit 16H calculates the reliability, which is an index showing the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state. The reliability is calculated in three levels, for example, "low," "medium," and "high."

[0168] [9-2. Operation of the estimation device] Next, the operation of the estimation device 2H according to the ninth embodiment will be described with reference to Fig. 18. Fig. 18 is a flowchart showing the flow of the operation of the estimation device 2H according to the ninth embodiment. In the flowchart of Fig. 18, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0169] 18, first, steps S101 to S104 are executed in the same manner as in the above-described embodiment 2. If the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102) and if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the process proceeds to step S801, and the facial feature information acquisition unit 56 acquires facial feature information.

[0170] If the facial feature information acquisition unit 56 acquires facial feature information that affects the driver's microsleep state (YES in S801), the reliability calculation unit 20H of the microsleep estimation unit 16H calculates the reliability to be "medium" (S802), and the estimation unit 18 of the microsleep estimation unit 16H estimates that the driver is in a microsleep state (S106).

[0171] That is, the estimation unit 18 of the microsleep estimation unit 16H estimates that the driver is in a microsleep state under the following conditions: the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102); the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104); and the facial feature information acquisition unit 56 acquires facial feature information that affects the driver's microsleep state (YES in S801). This is because the driver may close their eyes due to fatigue or the like even if they do not feel very drowsy.

[0172] Returning to step S801, if the facial feature information acquisition unit 56 does not acquire facial feature information that affects the driver's microsleep state (NO in S801), the reliability calculation unit 20H of the microsleep estimation unit 16H calculates the reliability to be "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in a microsleep state (S106).

[0173] [9-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0174] (Embodiment 10) [10-1. Configuration of the estimation device] The configuration of an estimation device 2J according to embodiment 10 will be described with reference to Fig. 19. Fig. 19 is a block diagram showing the configuration of an estimation device 2J according to embodiment 10. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0175] As shown in FIG. 19 , the estimation device 2J according to the tenth embodiment includes a head movement detection unit 58 in addition to the image information acquisition unit 8, the biological information acquisition unit 10, the eye closure state detection unit 12, the drowsiness determination unit 14, and the microsleep estimation unit 16J. The head movement detection unit 58 detects the driver's head movement in time series based on the image information from the image information acquisition unit 8. The head movement detection unit 58 outputs the detection result to the microsleep estimation unit 16J. Note that the head movement detection unit 58 may also detect head movement based on the biological information from the biological information acquisition unit 10.

[0176] The estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds, the drowsiness determination unit 14 determines that the drowsiness level is less than "4," and the head movement detection unit 58 acquires a head movement that affects the driver's microsleep state. Here, the head movement that affects the driver's microsleep state is, for example, a movement such as the driver nodding, where the head is shaken up and down due to drowsiness.

[0177] The reliability calculation unit 20J of the microsleep estimation unit 16J calculates the reliability, which is an index showing the likelihood of the estimation result that the estimation unit 18 has determined that the driver is in a microsleep state. The reliability is calculated in three levels, for example, "low," "medium," and "high."

[0178] [10-2. Operation of the estimation device] Next, the operation of the estimation device 2J according to the tenth embodiment will be described with reference to Fig. 20. Fig. 20 is a flowchart showing the flow of the operation of the estimation device 2J according to the tenth embodiment. In the flowchart of Fig. 20, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0179] 20, steps S101 to S104 are first executed in the same manner as in the above-described embodiment 2. If the eye closure state detection unit 12 detects that the eye closure time is 0.5 seconds or more and less than 3 seconds (YES in S102) and if the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the process proceeds to step S901, where the head movement detection unit 58 detects the movement of the driver's head.

[0180] If the head movement detection unit 58 detects head movement that affects the driver's microsleep state (YES in S901), the reliability calculation unit 20J of the microsleep estimation unit 16J calculates the reliability to be "medium" (S902), and the estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in a microsleep state (S106).

[0181] That is, the estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in a microsleep state when the eye closure state detection unit 12 detects that the eye closure time is between 0.5 seconds and 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), and the head movement detection unit 58 detects head movement that affects the driver's microsleep state (YES in S901). This is because the driver may close their eyes due to fatigue or the like even if they do not feel very drowsy.

[0182] Returning to step S901, if the head movement detection unit 58 does not acquire any head movement that would affect the driver's microsleep state (NO in S901), the reliability calculation unit 20J of the microsleep estimation unit 16J calculates the reliability to be "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in a microsleep state (S106).

[0183] [10-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0184] (Embodiment 11) [11-1. Configuration of the estimation device] The configuration of an estimation device 2K according to embodiment 11 will be described with reference to Fig. 21. Fig. 21 is a block diagram showing the configuration of an estimation device 2K according to embodiment 11. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0185] As shown in FIG. 21 , the estimation device 2K according to the eleventh embodiment includes an erroneous estimation situation detection unit 60 in addition to the image information acquisition unit 8, the biometric information acquisition unit 10, the eye closure state detection unit 12, the drowsiness determination unit 14, and the microsleep estimation unit 16K. The erroneous estimation situation detection unit 60 detects a situation that affects the estimation of the driver's microsleep state by the estimation unit 18 (hereinafter referred to as an "erroneous estimation situation") based on image information from the image information acquisition unit 8. Examples of erroneous estimation situations include a sunlight situation in which the afternoon sun shines on the driver's face, or a road situation in which a small shadow of a roadside tree or the like is cast on the driver's face. The erroneous estimation situation detection unit 60 outputs the detection result to the microsleep estimation unit 16K. The erroneous estimation situation detection unit 60 may also detect an erroneous estimation situation based on biometric information from the biometric information acquisition unit 10.

[0186] The reliability calculation unit 20K of the microsleep estimation unit 16K calculates the reliability, which is an index showing the likelihood of the estimation result by the estimation unit 18 that the driver is in a microsleep state, taking into account the detection result by the erroneous estimation situation detection unit 60. The reliability is calculated in three levels, for example, "low," "medium," and "high."

[0187] [11-2. Operation of the estimation device] Next, the operation of the estimating device 2K according to the eleventh embodiment will be described with reference to Fig. 22. Fig. 22 is a flowchart showing the flow of the operation of the estimating device 2K according to the eleventh embodiment. In the flowchart of Fig. 22, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0188] 22, steps S101 to S104 are first executed in the same manner as in the above-described embodiment 2. When the eye-closure state detection unit 12 detects that the eye-closure time is equal to or longer than 0.5 seconds and shorter than 3 seconds (YES in S102) and the drowsiness determination unit 14 determines that the drowsiness level is lower than "4" (NO in S104), the process proceeds to step S1001, where the erroneous estimation situation detection unit 60 detects whether or not there is an erroneous estimation situation.

[0189] If an erroneous estimation situation is detected by the erroneous estimation situation detection unit 60 (YES in S1001), the reliability calculation unit 20K of the microsleep estimation unit 16K calculates the reliability as "medium" (S1002), and the estimation unit 18 of the microsleep estimation unit 16K estimates that the driver is in a microsleep state (S106).

[0190] Returning to step S1001, if the incorrect estimation situation detection unit 60 does not detect an incorrect estimation situation (NO in S1001), the reliability calculation unit 20K of the microsleep estimation unit 16K calculates the reliability as "low" (S107), and the estimation unit 18 of the microsleep estimation unit 16K estimates that the driver is in a microsleep state (S106).

[0191] [11-3.Effects] In this embodiment, the microsleep state can be estimated with even greater accuracy.

[0192] (Embodiment 12) [12-1. Configuration of the estimation device] The configuration of an estimation device 2L according to embodiment 12 will be described with reference to Fig. 23. Fig. 23 is a block diagram showing the configuration of an estimation device 2L according to embodiment 12. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0193] As shown in FIG. 23 , the estimation device 2L according to the twelfth embodiment includes an erroneous estimation situation detection unit 62 in addition to an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closure state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16L. The erroneous estimation situation detection unit 62 detects an erroneous estimation situation based on image information from the image information acquisition unit 8 or biological information from the biological information acquisition unit 10. Examples of erroneous estimation situations include a sunlight situation in which the afternoon sun shines on the driver's face, or a road situation in which a small shadow of a roadside tree or the like is cast on the driver's face. The erroneous estimation situation detection unit 62 outputs the detection result to the microsleep estimation unit 16L via the eye-closure state detection unit 12 and the drowsiness determination unit 14.

[0194] When the erroneous estimation state detection unit 62 detects an erroneous estimation state, the estimation unit 18 of the microsleep estimation unit 16L does not estimate the microsleep state of the driver.

[0195] [12-2. Operation of the estimation device] Next, the operation of the estimation device 2L according to embodiment 12 will be described with reference to Fig. 24. Fig. 24 is a flowchart showing the flow of the operation of the estimation device 2L according to embodiment 12. In the flowchart of Fig. 24, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and their description will be omitted.

[0196] 24, first, after step S101 is executed, the erroneous estimation situation detection unit 62 detects whether or not there is an erroneous estimation situation, as in the above-described embodiment 2. If the erroneous estimation situation detection unit 62 does not detect an erroneous estimation situation (NO in S1101), the process proceeds to step S102, and steps S102 to S107 are executed, as in the above-described embodiment 2. On the other hand, if the erroneous estimation situation detection unit 62 detects an erroneous estimation situation (YES in S1101), the estimation unit 18 of the microsleep estimation unit 16L does not estimate the driver's microsleep state (S1102).

[0197] [12-3.Effects] In this embodiment, it is possible to avoid erroneous estimation of the microsleep state.

[0198] Note that erroneous estimation situations are not limited to the above-mentioned situations, and may include, for example, situations where it is okay to close your eyes, such as when the vehicle is stopped, situations where the driver is smiling with their eyes closed, situations where the driver is talking to a passenger, etc. In these situations, it is not necessary to estimate the microsleep state, so by not estimating the microsleep state, erroneous estimation of the microsleep state can be effectively avoided.

[0199] Furthermore, if noise occurs in the image information (image information showing the driver's face) from the image information acquisition unit 8 due to sunlight conditions such as setting sun or road conditions such as small shadows cast on the driver's face by roadside trees, a process for correcting the image information may be performed. This can prevent erroneous estimation of the microsleep state.

[0200] (Embodiment 13) [13-1. Configuration of the estimation device] The configuration of an estimation device 2M according to embodiment 13 will be described with reference to Fig. 25. Fig. 25 is a block diagram showing the configuration of an estimation device 2M according to embodiment 13. Note that in this embodiment, the same components as those in embodiment 2 above are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0201] 25, the estimation device 2M according to the thirteenth embodiment includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye closure state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16A, as well as a driving situation determination unit 64. The driving situation determination unit 64 includes a driving situation detection unit 66 and an eye closure time change unit 68. The biological information from the biological information acquisition unit 10 is output to the drowsiness determination unit 14 via the driving situation determination unit 64.

[0202] The driving condition detection unit 66 detects the driving condition of the vehicle by the driver based on the image information from the image information acquisition unit 8 or the biometric information from the biometric information acquisition unit 10. Here, the driving condition refers to, for example, the driving time period of the vehicle, the location of the vehicle, or the driver's work style on the previous day. The driving condition detection unit 66 outputs the detection result to the eyes closed state detection unit 12.

[0203] The eye-closure time changing unit 68 changes the first time and / or the second time used for detecting the eye-closure time in the eye-closure state detecting unit 12 based on the driving condition detected by the driving condition detecting unit 66.

[0204] [13-2. Operation of the estimation device] Next, a first operation of the estimating device 2M according to the thirteenth embodiment will be described with reference to Fig. 26. Fig. 26 is a flowchart showing the flow of the first operation of the estimating device 2M according to the thirteenth embodiment. In the flowchart of Fig. 26, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and descriptions thereof will be omitted.

[0205] As shown in FIG. 26, first, as in the second embodiment, step S101 is executed, and then the driving condition detection unit 66 detects the driving condition. If the driving condition detection unit 66 does not detect a driving condition that may affect the microsleep state (NO in S1201), the eye-closure time change unit 68 sets the first time and the second time to default values ​​of 0.5 seconds and 3 seconds, respectively (S1202). Driving conditions that may affect the microsleep state include, for example, a situation in which the driver is driving at night or the driver worked the night shift the previous day. Thereafter, as in the second embodiment, the process proceeds to step S102, and steps S102 to S107 are executed as in the second embodiment.

[0206] Returning to step S1201, if the driving condition detection unit 66 detects a driving condition that may affect the microsleep state (YES in S1201), the eye-closure time change unit 68 shortens the second time from the default value of 3 seconds to 2 seconds (S1203). Thereafter, if the eye-closure state detection unit 12 detects that the eye-closure time is less than 0.5 seconds or 2 seconds or more (NO in S1204), the process proceeds to step S103. On the other hand, if the eye-closure state detection unit 12 detects that the eye-closure time is 0.5 seconds or more but less than 2 seconds (YES in S1204), the process proceeds to step S104.

[0207] As described above, in the first operation, when the driver is in a situation where he or she is likely to feel drowsy, for example, when the driver is driving at night or when the driver worked the night shift the previous day, the eye closure time detected by the eye closure state detection unit 12 is shortened to make a stricter estimate of the driver's microsleep state.

[0208] Next, a second operation of the estimating device 2M according to the thirteenth embodiment will be described with reference to Fig. 27. Fig. 27 is a flowchart showing the flow of the second operation of the estimating device 2M according to the thirteenth embodiment. In the flowchart of Fig. 27, the same processes as those in the flowchart of Fig. 4 described above are assigned the same step numbers, and descriptions thereof will be omitted.

[0209] 27, first, as in the second embodiment, step S101 is executed, and then the driving condition detection unit 66 detects the driving condition. If the driving condition detection unit 66 detects a driving condition that may affect the microsleep state (YES in S1301), the eye closure time change unit 68 sets the first time and the second time to default values ​​of 0.5 seconds and 3 seconds, respectively (S1302). Thereafter, as in the second embodiment, the process proceeds to step S102, and as in the second embodiment, steps S102 to S107 are executed.

[0210] Returning to step S1301, if the driving condition detection unit 66 does not detect a driving condition that would affect the microsleep state (NO in S1301), the eye-closure time change unit 68 lengthens the first time from the default value of 0.5 seconds to 1 second, and lengthens the second time from the default value of 3 seconds to 4 seconds (S1303). Thereafter, if the eye-closure state detection unit 12 detects that the eye-closure time is less than 1 second or 4 seconds or more (NO in S1304), the process proceeds to step S103. On the other hand, if the eye-closure state detection unit 12 detects that the eye-closure time is 1 second or more but less than 4 seconds (YES in S1304), the process proceeds to step S104.

[0211] As described above, in the second operation, when the driver is in a situation where he or she is unlikely to feel drowsy, for example, immediately after taking a nap or immediately after consuming caffeine, the eye closure time detected by the eye closure state detection unit 12 is lengthened to loosely estimate the driver's microsleep state.

[0212] [13-3.Effects] In this embodiment, it is possible to avoid erroneous estimation of the microsleep state.

[0213] (Other variations) Although the estimation device according to one or more aspects has been described based on the above-mentioned embodiments, the present disclosure is not limited to the above-mentioned embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the above-mentioned embodiments, or configurations constructed by combining components of different embodiments, may also be included within the scope of one or more aspects.

[0214] In each of the above embodiments, the reliability calculation unit 20 (20B, 20C, 20D, 20E, 20F, 20G, 20H, 20J, 20K, 24, 28, 36, 48) calculated the reliability, but the reliability may also be calculated using, for example, deep learning.

[0215] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0216] Furthermore, some or all of the functions of the estimation device according to each of the above embodiments may be realized by a processor such as a CPU executing a program.

[0217] Some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to each device. The IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned ultra-multifunctional LSI. The IC card or module achieves its functions when the microprocessor operates according to a computer program. The IC card or module may be tamper-resistant.

[0218] The present disclosure may be the above-described methods. It may also be a computer program for implementing these methods on a computer, or a digital signal comprising the computer program. The present disclosure may also be a computer program or a digital signal recorded on a computer-readable non-transitory recording medium, such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray (registered trademark) Disc), semiconductor memory, etc. It may also be a digital signal recorded on such a recording medium. The present disclosure may also be a computer program or a digital signal transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc. The present disclosure may also be a computer system including a microprocessor and a memory, in which the memory stores the computer program, and the microprocessor operates according to the computer program. The present disclosure may also be implemented by another independent computer system by recording the program or the digital signal on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc. [Industrial Applicability]

[0219] The present disclosure is applicable to, for example, an estimation device for estimating a microsleep state of a vehicle driver. [Explanation of symbols]

[0220] 2,2A,2B,2C,2D,2E,2F,2G,2H,2J,2K,2L,2M Estimator 3 Sensor Group 4. Imaging unit 5 Vehicle condition sensor 6. Biometric sensors 7. Sensor information acquisition unit 8 Image information acquisition unit 10 Biometric information acquisition unit 12, 12B, 12E Eye closure detection unit 14,14B Drowsiness determination section 16,16A,16B,16C,16D,16E,16F,16G,16H,16J,16K,16L Microsleep estimation section 18 Estimation part 20, 20B, 20C, 20D, 20E, 20F, 20G, 20H, 20J, 20K, 24, 28, 36, 48 Reliability calculation section 22 Eye closure time detection unit 26 Drowsiness level determination unit 30,38 Open / close state detection unit 32 Opening and closing speed calculation section 34 Opening / closing speed determination section 40 Eye closing speed calculation unit 42 Eye closing speed determination section 44 Eye opening speed calculation unit 46 Eye opening speed determination unit 50 Binocular detection unit 52 Blink detection unit 54 Life log information acquisition unit 56 Facial feature information acquisition unit 58 Head movement detection unit 60,62 Misestimation situation detection unit 64 Driving situation determination unit 66 Driving condition detection unit 68 Eye closing time change unit

Claims

1. An estimation device for estimating that a vehicle driver is in a microsleep state, an eye closure state detection unit that detects a time period during which the driver's eyes are closed; a drowsiness determination unit that determines the drowsiness level of the driver; a microsleep estimation unit that estimates that the driver is in a microsleep state when the eye-closure state detection unit detects that the eye-closure time is equal to or longer than a first time and shorter than a second time, and when the drowsiness determination unit determines that the drowsiness level is equal to or higher than a first threshold. Estimation device.

2. The first time period is 0.5 seconds and the second time period is 3 seconds. The estimation device according to claim 1 .

3. The microsleep estimation unit further calculates a first reliability that is an index indicating the likelihood of the estimation result that the driver is in a microsleep state. The estimation device according to claim 1 .

4. the eye-closure state detection unit further calculates a second reliability that is an index indicating a likelihood of a detection result that the eye-closure time is equal to or longer than the first time and shorter than the second time, The drowsiness determination unit further calculates a third reliability that is an index indicating a likelihood of a determination result that the drowsiness level is equal to or greater than the first threshold; The microsleep estimation unit calculates the first reliability based on the second reliability and the third reliability. The estimation device according to claim 3 .

5. The estimation device further includes an eye opening / closing state detection unit that detects an eye opening / closing speed of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is equal to or longer than the first threshold, and the opening / closing state detection unit detects that the opening / closing speed is shorter than a second threshold. The estimation device according to claim 3 .

6. the eye-closure state detection unit further calculates a second reliability that is an index indicating a likelihood of a detection result that the eye-closure time is equal to or longer than the first time and shorter than the second time, The drowsiness determination unit further calculates a third reliability that is an index indicating a likelihood of a determination result that the drowsiness level is equal to or greater than the first threshold; the opening / closing state detection unit further calculates a fourth reliability that is an index indicating a likelihood of a detection result that the opening / closing speed is less than the second threshold; The microsleep estimation unit calculates the first reliability based on the second reliability, the third reliability, and the fourth reliability. The estimation device according to claim 5 .

7. The drowsiness determination unit includes a function as the open / close state detection unit. The estimation device according to claim 5 .

8. The drowsiness determination unit further includes a function as the eye closure state detection unit. The estimation device according to claim 7 .

9. The estimation device further includes an open / closed state detection unit that detects an eye closing speed and an eye opening speed of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye-closure state detection unit detects that the eye-closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is equal to or longer than the first threshold, the open / close state detection unit detects that the eye-closure speed is shorter than a third threshold, and the open / close state detection unit detects that the eye-open speed is shorter than a fourth threshold. The estimation device according to claim 3 .

10. the eye-closure state detection unit further calculates a second reliability that is an index indicating a likelihood of a detection result that the eye-closure time is equal to or longer than the first time and shorter than the second time, The drowsiness determination unit further calculates a third reliability that is an index indicating a likelihood of a determination result that the drowsiness level is equal to or greater than the first threshold; the open / closed state detection unit further calculates a fifth reliability as an index indicating a likelihood of a detection result that the eye closing speed is less than the third threshold, and a sixth reliability as an index indicating a likelihood of a detection result that the eye opening speed is less than the fourth threshold, The microsleep estimation unit calculates the first reliability based on the second reliability, the third reliability, the fifth reliability, and the sixth reliability. The estimation device according to claim 9 .

11. The microsleep estimation unit calculates the first reliability in accordance with the drowsiness level determined by the drowsiness determination unit. The estimation device according to claim 3 .

12. The estimation device further includes a binocular detection unit that detects both eyes of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state when the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is equal to or higher than the first threshold, and the binocular detection unit detects both eyes of the driver. The estimation device according to claim 1 .

13. The estimation device further includes a blink frequency detection unit that detects the number of blinks of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is equal to or higher than the first threshold, and the blink count detection unit detects that the number of blinks per unit time of the driver has increased or decreased by equal to or greater than a fifth threshold. The estimation device according to claim 1 .

14. The estimation device further includes a life log information acquisition unit that acquires life log information related to the driver's life, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is lower than the first threshold, and the life log information acquisition unit acquires the life log information that affects the driver's microsleep state. The estimation device according to claim 1 .

15. The estimation device further includes a facial feature information acquisition unit that acquires facial feature information indicating facial features of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is lower than the first threshold, and the facial feature information acquisition unit acquires the facial feature information that affects the microsleep state of the driver. The estimation device according to claim 1 .

16. The estimation device further includes a head movement detection unit that detects a head movement of the driver, The microsleep estimation unit estimates that the driver is in a microsleep state on the condition that the eye closure state detection unit detects that the eye closure time is equal to or longer than the first time and shorter than the second time, the drowsiness determination unit determines that the drowsiness level is lower than the first threshold, and the head movement detection unit detects head movement that affects the driver's microsleep state. The estimation device according to claim 1 .

17. The estimation device further includes an erroneous estimation situation detection unit that detects a situation that affects the estimation of the driver's microsleep state by the microsleep estimation unit, The microsleep estimation unit changes the first reliability in consideration of the detection result of the erroneous estimation situation detection unit. The estimation device according to claim 3 .

18. The estimation device further includes an erroneous estimation situation detection unit that detects an erroneous estimation situation that affects the estimation of the driver's microsleep state by the microsleep estimation unit, The microsleep estimation unit does not estimate the microsleep state of the driver when the erroneous estimation state detection unit detects the erroneous estimation state. The estimation device according to claim 1 .

19. The estimation device further a driving condition detection unit that detects a driving condition of the vehicle by the driver; an eye-closure time changing unit that changes the first time period and / or the second time period used for detecting the eye-closure time period in the eye-closure state detecting unit based on the driving situation detected by the driving situation detecting unit. The estimation device according to claim 1 .

20. the eye-closure state detection unit detects the eye-closure time based on image information of the driver captured by an imaging unit, The drowsiness determination unit determines the drowsiness level based on biological information of the driver detected by a biological sensor. The estimation device according to any one of claims 1 to 19.

21. the eye-closure state detection unit detects the eye-closure time based on image information of the driver captured by an imaging unit, The drowsiness determination unit determines the drowsiness level based on the image information. The estimation device according to any one of claims 1 to 19.

22. 1. A method for estimating that a vehicle driver is in a microsleep state, comprising: (a) detecting a time period during which the driver's eyes are closed; (b) determining a drowsiness level of the driver; (c) estimating that the driver is in a microsleep state on condition that the eye-closure time is detected to be equal to or longer than a first time and shorter than a second time in (a) and the drowsiness level is determined to be equal to or higher than a first threshold in (b). Estimation method.

23. A computer is caused to execute the estimation method according to claim 22. program.

Images