Drowsiness determination device and drowsiness determination method

Abstract

A sleepiness assessment device (3) configured to comprise: an image data acquisition unit (11) for acquiring, from a camera (1) for imaging the face of a driver, image data representing a facial image of the driver; an alertness level estimation unit (12) for estimating, on the basis of the image data acquired by the image data acquisition unit (11), an alertness level indicating the degree of alertness of the driver; a waveform information acquisition unit (13) for acquiring, from a sensor (2) for detecting an electrocardiogram waveform of the driver, waveform information indicating the electrocardiogram waveform of the driver; and a heartbeat feature amount calculation unit (14) for calculating, from the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit (13), a feature amount pertaining to the heartbeat of the driver. The sleepiness assessment device (3) also comprises a sleepiness assessment unit (15) for assessing whether the driver is sleepy on the basis of the alertness level estimated by the alertness level estimation unit (12) and the heartbeat-related feature amount calculated by the heartbeat feature amount calculation unit (14).

Description

[Technical Field]

[0001] The present disclosure relates to a drowsiness determination device and a drowsiness determination method. [Background technology]

[0002] There is a drowsiness determination device that determines whether or not a driver is drowsy. As such a drowsiness determination device, for example, Patent Document 1 discloses a device including a drowsiness detection unit. The drowsiness detection unit detects the drowsiness of the driver based on the heartbeat interval obtained from the electrocardiogram waveform of the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-13737 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, the heartbeat interval is longer when the driver is drowsy than when the driver is not drowsy. However, the heartbeat interval changes depending on the activity state of the autonomic nervous system even when the driver is not drowsy. For example, the driver may become more relaxed after a certain period of time has passed since starting to drive a vehicle than when the driver first starts driving, which may change the activity state of the autonomic nervous system and cause the heartbeat interval to become longer even when the driver is not drowsy. In the device disclosed in Patent Document 1, the drowsiness detection unit detects the drowsiness of the driver based only on the heartbeat interval, which poses a problem that the drowsiness detection unit may erroneously detect drowsiness when the driver is not feeling drowsy.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a drowsiness determination device that can reduce false detection of drowsiness more than the device disclosed in Patent Document 1. [Means for solving the problem]

[0006] The drowsiness determination device according to the present disclosure includes an image data acquisition unit that acquires image data showing a face image of the driver from a camera that captures the face of the driver, an alertness estimation unit that estimates an alertness level that indicates the level of alertness of the driver based on the image data acquired by the image data acquisition unit, a waveform information acquisition unit that acquires waveform information showing an electrocardiogram waveform of the driver from a sensor that detects an electrocardiogram waveform of the driver, and a heartbeat feature amount calculation unit that calculates a feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit. The drowsiness determination device also includes a drowsiness presence / absence determination unit that determines whether the driver is drowsy based on the alertness estimated by the alertness estimation unit and the feature amount of the heartbeat calculated by the heartbeat feature amount calculation unit. The waveform information acquisition unit acquires a plurality of pieces of waveform information having different detection times by the sensor, the heartbeat feature amount calculation unit calculates a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information, normalizes the heartbeat feature amount using the normalization coefficient, and outputs the normalized heartbeat feature amount to the drowsiness presence / absence determination unit, and the drowsiness presence / absence determination unit determines whether the driver is drowsy based on the alertness estimated by the alertness estimation unit and the normalized heartbeat feature amount output from the heartbeat feature amount calculation unit. . [Effects of the Invention]

[0007] According to the present disclosure, it is possible to reduce false detection of drowsiness compared to the device disclosed in Patent Document 1. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram showing a drowsiness determination device 3 according to a first embodiment. [Figure 2] 1 is a hardware configuration diagram showing hardware of a drowsiness determination device 3 according to Embodiment 1. FIG. [Figure 3] FIG. 10 is a hardware configuration diagram of a computer when the drowsiness determination device 3 is realized by software, firmware, or the like. [Figure 4] 3 is a flowchart showing a drowsiness determination method, which is a processing procedure of the drowsiness determination device 3. [Figure 5]FIG. 5A is an explanatory diagram showing the distance L1 between the upper eyelid and the lower eyelid when the driver is not feeling drowsy, and FIG. 5B is an explanatory diagram showing the current distance L2 between the upper eyelid and the lower eyelid of the driver. [Figure 6] FIG. 2 is an explanatory diagram showing an example of a driver's heartbeat interval. [Figure 7] FIG. 10 is a configuration diagram showing a drowsiness determination device 3 according to a second embodiment. [Figure 8] FIG. 10 is a hardware configuration diagram showing hardware of a drowsiness determination device 3 according to a second embodiment. [Figure 9] FIG. 10 is a configuration diagram showing a drowsiness determination device 3 according to a third embodiment. [Figure 10] FIG. 10 is a hardware configuration diagram showing hardware of a drowsiness determination device 3 according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1 FIG. 1 is a configuration diagram showing a drowsiness determination device 3 according to the first embodiment. FIG. 2 is a hardware configuration diagram showing the hardware of the drowsiness determination device 3 according to the first embodiment. In FIG. 1, a camera 1 is installed, for example, on the instrument panel of a vehicle, on the windshield of a vehicle, or on the ceiling of a vehicle. The camera 1 is realized by, for example, one or more visible light cameras, one or more infrared cameras, or a video camera. When the camera 1 is realized by an infrared camera, a light source that irradiates an area including the driver's face with infrared light for imaging may be provided. The light source is realized by, for example, an LED (Light Emitting Diode). The camera 1 captures an image of the driver's face and outputs image data showing the face image of the driver to the drowsiness determination device 3.

[0011] The sensor 2 is realized by, for example, an electrocardiogram sensor that detects the electrocardiogram waveform of the driver in a state of contact with the driver, or an electrocardiogram sensor that detects the electrocardiogram waveform of the driver in a state of non-contact with the driver. The sensor 2 outputs waveform information indicating the electrocardiogram waveform of the driver to the drowsiness determination device 3. The sensor 2 is not limited to being realized by an electrocardiogram sensor, but may be realized by an infrared camera. When the sensor 2 is realized by an infrared camera, the sensor 2 detects the electrocardiogram waveform of the driver based on the brightness of the face surface that changes in accordance with the driver's heartbeat.

[0012] The drowsiness determination device 3 includes an image data acquisition unit 11, an arousal level estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature amount calculation unit 14, and a drowsiness presence / absence determination unit 15. The drowsiness determination device 3 determines whether the driver is drowsy based on the image data output from the camera 1 and the waveform information output from the sensor 2.

[0013] The image data acquisition unit 11 is realized by, for example, an image data acquisition circuit 21 shown in FIG. The image data acquisition unit 11 acquires, from the camera 1, image data showing an image of the driver's face. The image data acquisition unit 11 outputs the image data to the awakening level estimation unit 12.

[0014] The awakening level estimation unit 12 is realized by, for example, an awakening level estimation circuit 22 shown in FIG. The arousal level estimation unit 12 includes a face feature amount calculation unit 12a and an arousal level estimation processing unit 12b. The wakefulness level estimation unit 12 acquires image data from the image data acquisition unit 11 . The awakening level estimation unit 12 estimates the awakening level of the driver based on the image data. The awakening level is the driver's apparent awakening level estimated from the face image of the driver. The wakefulness level estimation unit 12 outputs the wakefulness level of the driver to the drowsiness presence / absence determination unit 15.

[0015] The facial feature amount calculation unit 12 a acquires image data from the image data acquisition unit 11 . The facial feature amount calculation unit 12a detects the driver's face from the image data and calculates the facial feature amount. The facial feature amount calculation unit 12a calculates normalization coefficients for facial feature amounts based on the plurality of image data acquired by the image data acquisition unit 11. The multiple image data are multiple image data captured at different times by the camera 1, and include the latest image data output from the image data acquisition unit 11 and previous image data output from the image data acquisition unit 11 before the latest image data. For example, if the number of data items in the multiple image data is N, the multiple image data include the latest image data as well as (N-1) previous image data items. The N image data items are time-series data, and N is an integer equal to or greater than 2. The facial feature amount calculation unit 12a normalizes the facial feature amount using the normalization coefficient. The facial feature amount calculation unit 12a outputs the normalized facial feature amount to the awakening level estimation processing unit 12b.

[0016] The awakening level estimation processing unit 12b acquires the normalized facial feature amount from the facial feature amount calculation unit 12a. The awakening level estimation processing unit 12b estimates the awakening level of the driver based on the normalized facial feature amount. The wakefulness estimation processing unit 12b outputs the wakefulness of the driver to the drowsiness presence / absence determining unit 15.

[0017] The waveform information acquisition unit 13 is realized by, for example, a waveform information acquisition circuit 23 shown in FIG. The waveform information acquisition unit 13 acquires, from the sensor 2, waveform information indicating the electrocardiographic waveform of the driver. The waveform information acquisition unit 13 outputs the waveform information to the heartbeat feature amount calculation unit 14 .

[0018] The pulse feature amount calculation unit 14 is realized by, for example, a pulse feature amount calculation circuit 24 shown in FIG. The pulse feature amount calculation unit 14 includes a pulse interval determination unit 14a and a pulse feature amount calculation processing unit 14b. The pulse feature amount calculation unit 14 acquires waveform information from the waveform information acquisition unit 13 . The heartbeat feature amount calculation unit 14 calculates the feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information. The heartbeat feature amount calculation unit 14 outputs the heartbeat feature amount to the drowsiness presence / absence determination unit 15.

[0019] The heartbeat interval determination unit 14 a acquires waveform information from the waveform information acquisition unit 13 . The heartbeat interval determination unit 14a determines the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specifying unit 14a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 14b.

[0020] The heartbeat feature amount calculation processing unit 14b acquires the heartbeat interval of the driver from the heartbeat interval specification unit 14a. The heartbeat feature amount calculation processing unit 14b calculates the heartbeat feature amount from the heartbeat interval. The heartbeat feature amount calculation processing unit 14b calculates a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information acquired by the waveform information acquisition unit 13. The plurality of pieces of waveform information are pieces of waveform information detected by the sensor 2 at different times, and include the latest waveform information output from the waveform information acquisition unit 13 and past waveform information output from the waveform information acquisition unit 13 before the latest waveform information. For example, if the number of data items in the plurality of pieces of waveform information is M, the plurality of pieces of waveform information includes (M-1) pieces of past waveform information in addition to the latest waveform information. The M pieces of waveform information are time-series data, and M is an integer greater than or equal to 2. The heartbeat feature amount calculation processing unit 14b normalizes the heartbeat feature amount using the normalization coefficient. The heartbeat feature amount calculation processing unit 14b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determining unit 15.

[0021] The drowsiness determination unit 15 is realized by, for example, a drowsiness determination circuit 25 shown in FIG. The drowsiness determination unit 15 determines whether the driver is drowsy based on the level of alertness estimated by the level of alertness estimation unit 12 and the feature amount of the heartbeat calculated by the heartbeat feature amount calculation unit 14. Specifically, the drowsiness presence / absence determining unit 15 acquires the driver's alertness from the alertness estimation processing unit 12b, and acquires the normalized heartbeat feature amount from the heartbeat feature amount calculation processing unit 14b. Then, the drowsiness determination unit 15 determines whether the driver is drowsy based on the driver's level of alertness and the normalized feature amount of the heart rate.

[0022] 1, it is assumed that each of the components of the drowsiness determination device 3, that is, the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature value calculation unit 14, and the drowsiness presence / absence determination unit 15, is realized by dedicated hardware as shown in Fig. 2. That is, it is assumed that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature value calculation circuit 24, and a drowsiness presence / absence determination circuit 25. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the heart rate feature calculation circuit 24, and the drowsiness determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0023] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. The software or firmware is stored as a program in the memory of a computer. A computer refers to hardware that executes the program, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor).

[0024] FIG. 3 is a hardware configuration diagram of a computer when the drowsiness determination device 3 is realized by software, firmware, or the like. When the drowsiness determination device 3 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the image data acquisition unit 11, the arousal level estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature amount calculation unit 14, and the drowsiness presence / absence determination unit 15 is stored in the memory 31. Then, a processor 32 of the computer executes the program stored in the memory 31.

[0025] 2 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, and Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components in the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0026] Next, the operation of the drowsiness determination device 3 shown in FIG. 1 will be described. FIG. 4 is a flowchart showing a drowsiness determination method, which is a processing procedure of the drowsiness determination device 3. For example, the camera 1 starts capturing an image of the driver's face when the driver gets into the vehicle and starts the engine. However, the time when the camera 1 starts capturing an image is not limited to the time when the driver starts the engine, and may be, for example, the time when the driver gets into the vehicle. The camera 1 outputs image data showing a facial image of the driver to the image data acquisition unit 11 of the drowsiness determination device 3. The facial image shown by the image data may be a moving image or a plurality of still images captured intermittently.

[0027] The sensor 2 detects the electrocardiogram waveform of the driver. The sensor 2 outputs waveform information indicating the electrocardiogram waveform of the driver to the waveform information acquisition unit 13 of the drowsiness determination device 3.

[0028] The image data acquisition unit 11 acquires image data showing a face image of the driver from the camera 1 (step ST1 in FIG. 4). The image data acquisition unit 11 outputs the image data to the awakening level estimation unit 12.

[0029] The wakefulness level estimation unit 12 acquires image data from the image data acquisition unit 11 . The awakening level estimation unit 12 estimates the awakening level of the driver based on the image data (step ST2 in FIG. 4). The wakefulness level estimation unit 12 outputs the wakefulness level of the driver to the drowsiness presence / absence determination unit 15.

[0030] The wakefulness level estimation process performed by the wakefulness level estimation unit 12 will be specifically described below. The facial feature amount calculation unit 12 a acquires image data from the image data acquisition unit 11 . The facial feature amount calculation unit 12a detects the face of the driver from the facial image of the driver represented by the image data. Specifically, the facial feature amount calculation unit 12a can detect the driver's face by using, for example, a machine learning algorithm called AdaBoost (Adaptive Boosting) or a Haar-Like detector to which an algorithm called cascade is applied. The Haar-Like detector is a detector that handles Haar-Like features. The Haar-Like features are features obtained from the brightness differences of multiple local regions.

[0031] The facial feature amount calculation unit 12a calculates the feature amount of the detected face. Specifically, the facial feature amount calculation unit 12a detects facial parts from the face of the driver, such as the driver's eyes or the driver's mouth. Next, the facial feature amount calculation unit 12a calculates, for example, the degree to which the driver's eyes are open or the amount to which the driver's mouth is open. The degree of eye opening is calculated, for example, from the vertical distance L1 between the upper and lower eyelids when the driver is not feeling drowsy and the current vertical distance L2 between the upper and lower eyelids of the driver, as shown in the following equation (1): When the driver is seated in the driver's seat and looking ahead of the vehicle, the vertical directions of the upper and lower eyelids are generally the same as the vertical direction.

[0032] As shown in Fig. 5A, the distance L1 is the distance at the position where the distance between the upper eyelid and the lower eyelid is maximum when the driver is not experiencing drowsiness. The distance L1 is calculated, for example, from the driver's eyes, which are facial features, detected within a certain period of time after the driver gets into the vehicle. The distance L1 is not limited to being calculated from the driver's eyes detected within a certain period of time after the driver gets into the vehicle, and may be stored in an internal memory of the facial feature amount calculation unit 12a or provided from outside the drowsiness determination device 3, for example. Distance L2 is calculated from the driver's eyes, which are facial features detected from the face image represented by the latest image data acquired from image data acquisition unit 11, as shown in FIG. 5B. FIG. 5A is an explanatory diagram showing the distance L1 between the upper eyelid and the lower eyelid when the driver is not feeling drowsy. FIG. 5B is an explanatory diagram showing the current distance L2 between the driver's upper eyelid and lower eyelid. Eye opening degree = (L2 / L1) x 100 (1)

[0033] Here, the degree of eye opening is calculated by Equation (1). However, this is merely an example, and the degree of eye opening may be calculated by normalizing the vertical distance L1 between the upper eyelid and the lower eyelid using facial features such as the nose, mouth, or ears. The mouth opening amount is, for example, the vertical distance between the upper lip and the lower lip at the position where the distance between the upper lip and the lower lip is greatest.

[0034] The facial feature amount calculation unit 12a calculates facial feature amounts that are effective for estimating the driver's level of alertness, for example, based on the degree of eye opening or the degree of mouth opening. Such facial features include, for example, the proportion of time that the eyes are closed within a certain period of time, the number of blinks within a certain period of time, or the number of yawns within a certain period of time. The percentage of time during which the eyes are closed within a certain period of time is calculated by dividing the time during which the degree of eye opening is 0% within the certain period of time by the duration of the certain period of time. The number of blinks in a certain period is calculated from the number of times the eye opening degree reaches 0% within the certain period. Yawning is detected based on changes in mouth opening.

[0035] The facial feature amount calculation unit 12a normalizes the facial feature amounts in order to absorb individual differences in the facial feature amounts. Specifically, the facial feature amount calculation unit 12a accumulates facial feature amounts calculated based on N pieces of image data including, for example, the latest image data and past image data, calculates the average value or percentile value of the accumulated feature amounts, and sets the calculated average value or percentile value as the normalization coefficient. The facial feature amount calculation unit 12a normalizes the facial feature amount by subtracting a normalization coefficient from the facial feature amount calculated based on the latest image data, or by dividing the facial feature amount calculated based on the latest image data by the normalization coefficient. The facial feature amount calculation unit 12a outputs the normalized facial feature amount to the awakening level estimation processing unit 12b.

[0036] 1, the facial feature amount calculation unit 12a accumulates facial feature amounts calculated based on each of N pieces of image data including the latest image data and past image data. However, this is merely an example, and the facial feature amount calculation unit 12a may accumulate facial feature amounts calculated based on each of N pieces of image data including only past image data. Furthermore, the facial feature amounts calculated based on each of the N pieces of image data may be facial feature amounts calculated during a certain period of time after the driver gets into the vehicle.

[0037] The awakening level estimation processing unit 12b acquires the normalized facial feature amount from the facial feature amount calculation unit 12a. The awakening level estimation processing unit 12b estimates the awakening level of the driver based on the normalized facial feature amount. Specifically, the alertness estimation processing unit 12b provides normalized facial features to a learning model that uses a common algorithm such as Random Forest or logistic regression, and obtains the driver's alertness from the learning model.

[0038] During learning, the learning model is given normalized facial features and training data indicating the driver's alertness, and learns the driver's alertness. During inference, when the normalized facial feature amount is provided from the arousal level estimation processing unit 12b, the learning model outputs the driver's arousal level to the arousal level estimation processing unit 12b. The driver's arousal level is, for example, data between 0 and 1. The wakefulness estimation processing unit 12b outputs the wakefulness of the driver to the drowsiness presence / absence determining unit 15.

[0039] The waveform information acquisition unit 13 acquires waveform information indicating the electrocardiographic waveform of the driver from the sensor 2 (step ST3 in FIG. 4). The waveform information acquisition unit 13 outputs the waveform information to the heartbeat feature amount calculation unit 14 .

[0040] The pulse feature amount calculation unit 14 acquires waveform information from the waveform information acquisition unit 13 . The heartbeat feature amount calculation unit 14 calculates the feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information (step ST4 in FIG. 4). The heartbeat feature amount calculation unit 14 outputs the heartbeat feature amount to the drowsiness presence / absence determination unit 15.

[0041] Hereinafter, the calculation process of the heartbeat feature amount by the heartbeat feature amount calculation unit 14 will be specifically described. The heartbeat interval determination unit 14 a acquires waveform information from the waveform information acquisition unit 13 . The heartbeat interval determination unit 14a determines the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The beat-to-beat interval is the time between two adjacent heartbeats, as shown in FIG. FIG. 6 is an explanatory diagram showing an example of the heartbeat interval of a driver. The heartbeat interval specifying unit 14a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 14b.

[0042] The heartbeat feature amount calculation processing unit 14b acquires the heartbeat interval of the driver from the heartbeat interval specification unit 14a. The heartbeat feature amount calculation processing unit 14b calculates the heartbeat feature amount from the heartbeat interval. The feature amount of the heartbeat includes, for example, the following feature amounts (1) to (6). (1) The average value of the driver's heart rate over a certain period of time (2) Standard deviation of the driver's heart rate interval within a certain period (3) The average heart rate of the driver over a certain period of time (4) The square root of the average value of the square of the difference between two adjacent heartbeat intervals in the time direction within a certain period (5) The number of times that the difference between two adjacent heartbeat intervals in the time direction exceeds a threshold within a certain period. (6) The rate at which the difference between two adjacent heartbeat intervals in the time direction exceeds a threshold within a certain period.

[0043] The heartbeat feature amount calculation processing unit 14b normalizes the heartbeat feature amount in order to absorb individual differences in the heartbeat feature amount. Specifically, the pulse feature amount calculation processing unit 14b acquires, for example, M pieces of waveform information including the latest waveform information and past waveform information from the waveform information acquisition unit 13 via the heartbeat interval identification unit 14a. The pulse feature amount calculation processing unit 14b accumulates the pulse feature amounts calculated based on each of the M pieces of waveform information, calculates the average value of the accumulated feature amounts, or the percentile value of the accumulated feature amounts, and sets the calculated average value or the calculated percentile value as the normalization coefficient. The heartbeat feature calculation processing unit 14b normalizes the heartbeat feature by subtracting a normalization coefficient from the heartbeat feature calculated based on the latest waveform information, or by dividing the heartbeat feature calculated based on the latest waveform information by the normalization coefficient. The heartbeat feature amount calculation processing unit 14b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determining unit 15.

[0044] 1, the pulse feature amount calculation processing unit 14b accumulates pulse feature amounts calculated based on each of M pieces of waveform information including the latest waveform information and past waveform information. However, this is merely an example, and the pulse feature amount calculation processing unit 14b may accumulate pulse feature amounts calculated based on each of M pieces of waveform information including only past waveform information. Furthermore, the feature amount of the heartbeat calculated based on each of the M pieces of waveform information may be a feature amount of the heartbeat calculated during a certain period of time after the driver gets into the vehicle.

[0045] The drowsiness presence / absence determining unit 15 acquires the driver's level of alertness from the alertness estimation processing unit 12b, and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14b. The drowsiness determination unit 15 determines whether the driver is drowsy based on the driver's level of alertness and the normalized heart rate feature amount (step ST5 in FIG. 4). Specifically, the drowsiness determination unit 15 provides the driver's alertness and normalized heart rate features to a learning model that uses a general algorithm, such as random forest or logistic regression, and obtains a determination result of whether the driver is drowsy or not from the learning model.

[0046] During training, the learning model is given the driver's alertness, normalized heart rate features, and training data indicating whether the driver is drowsy, and learns whether the driver is drowsy. The training data is, for example, "1" if the driver is not drowsy, and "0" if the driver is drowsy. During inference, when the learning model is given the driver's alertness and the normalized heart rate feature values ​​from the drowsiness presence / absence determining unit 15, it outputs the determination result of whether or not drowsiness is present to the drowsiness presence / absence determining unit 15. The drowsiness presence / absence determining unit 15 outputs the determination result of the presence / absence of drowsiness to, for example, a driver monitoring system (not shown).

[0047] In the first embodiment described above, the drowsiness determination device 3 is configured to include an image data acquisition unit 11 that acquires image data representing a facial image of the driver from a camera 1 that captures an image of the driver's face, an alertness estimation unit 12 that estimates an alertness level indicating the level of alertness of the driver based on the image data acquired by the image data acquisition unit 11, a waveform information acquisition unit 13 that acquires waveform information representing an electrocardiogram waveform of the driver from a sensor 2 that detects an electrocardiogram waveform of the driver, and a heartbeat feature amount calculation unit 14 that calculates a feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information acquisition unit 13. The drowsiness determination device 3 also includes a drowsiness presence / absence determination unit 15 that determines whether the driver is drowsy based on the alertness estimated by the alertness estimation unit 12 and the heartbeat feature amount calculated by the heartbeat feature amount calculation unit 14. Therefore, the drowsiness determination device 3 can reduce false detection of drowsiness compared to the device disclosed in Patent Document 1.

[0048] 1, the wakefulness estimation processing unit 12b provides the normalized facial feature amount to a learning model and acquires the wakefulness level of the driver from the learning model. However, this is merely an example, and if the normalized facial feature amount is, for example, the proportion of time the eyes were closed within a certain period of time, the wakefulness estimation processing unit 12b may provide the proportion of time the eyes were closed within a certain period of time to a function that returns a larger wakefulness level the smaller the proportion of time the eyes were closed within a certain period of time, and acquire the wakefulness level of the driver from the function. Furthermore, if the facial feature after normalization is, for example, the number of blinks in a certain period of time, the alertness estimation processing unit 12b may provide the number of blinks in a certain period of time to a function that returns a higher level of alertness the fewer the number of blinks in a certain period of time, and obtain the driver's level of alertness from the function. Furthermore, if the facial feature after normalization is, for example, the number of yawns in a certain period of time, the alertness estimation processing unit 12b may apply the number of yawns in a certain period of time to a function that returns a larger level of alertness the fewer the number of yawns in a certain period of time, and obtain the driver's level of alertness from the function.

[0049] 1, the drowsiness determination unit 15 provides the driver's level of alertness and the normalized heart rate feature amount to a learning model, and acquires a determination result of whether or not the driver is drowsy from the learning model. However, this is merely an example, and the drowsiness determination unit 15 may determine that the driver is drowsy if the driver's level of alertness is lower than a first determination threshold and the normalized heart rate feature amount is higher than a second determination threshold, and may determine that the driver is not drowsy otherwise. The first determination threshold and the second determination threshold may be stored in an internal memory of the drowsiness presence / absence determining unit 15, or may be provided from outside the drowsiness determination device 3. In addition, the drowsiness determination unit 15 may determine whether or not the driver is drowsy by placing more importance on the driver's alertness than on the features of the normalized heart rate, or may determine whether or not the driver is drowsy by placing more importance on the features of the normalized heart rate than on the driver's alertness, by accepting changes to the first determination threshold and the second determination threshold.

[0050] 1, the facial feature amount calculation unit 12a outputs the normalized facial feature amount to the alertness level estimation processing unit 12b. If the driver who gets into the vehicle is always the same person and there is no need to absorb individual differences in the facial feature amount, the facial feature amount calculation unit 12a may output the unnormalized facial feature amount to the alertness level estimation processing unit 12b. 1, the heartbeat feature amount calculation processing unit 14b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determination unit 15. If the driver who rides in the vehicle is always the same person and there is no need to absorb individual differences in the heartbeat feature amount, the heartbeat feature amount calculation processing unit 14b may output the unnormalized heartbeat feature amount to the drowsiness presence / absence determination unit 15.

[0051] Embodiment 2 In embodiment 2, we will describe a drowsiness determination device 3 in which the heart rate feature calculation unit 16 calculates a normalization coefficient for the heart rate feature based on multiple waveform information only when the alertness estimated by the alertness estimation unit 12 is equal to or greater than a threshold.

[0052] Fig. 7 is a configuration diagram showing a drowsiness determination device 3 according to embodiment 2. In Fig. 7, the same reference numerals as in Fig. 1 indicate the same or corresponding parts, and detailed description thereof will be omitted. Fig. 8 is a hardware configuration diagram showing the hardware of the drowsiness determination device 3 according to embodiment 2. In Fig. 8, the same reference numerals as in Fig. 2 indicate the same or corresponding parts, and detailed description thereof will be omitted. The drowsiness determination device 3 shown in FIG. 7 includes an image data acquisition unit 11, an arousal level estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature amount calculation unit 16, and a drowsiness presence / absence determination unit 15.

[0053] The pulse feature amount calculation unit 16 is realized by, for example, a pulse feature amount calculation circuit 26 shown in FIG. The pulse feature amount calculation unit 16 includes a pulse interval determination unit 16a and a pulse feature amount calculation processing unit 16b. The pulse feature amount calculation unit 16 acquires waveform information from the waveform information acquisition unit 13 and acquires the driver's level of alertness from the alertness level estimation unit 12 . The heartbeat feature amount calculation unit 16 calculates the feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information. If the driver's alertness is equal to or higher than a threshold, the heartbeat feature amount calculation unit 16 calculates a normalization coefficient for the heartbeat feature amount based on the plurality of waveform information, and normalizes the heartbeat feature amount using the normalization coefficient. If the driver's alertness is less than the threshold, the heartbeat feature calculation unit 16 does not calculate a normalization coefficient for the heartbeat feature, but normalizes the heartbeat feature using the normalization coefficient when the driver is not drowsy. The heartbeat feature amount calculation unit 16 outputs the normalized heartbeat feature amount to the drowsiness presence / absence determination unit 15.

[0054] The heartbeat interval determination unit 16 a acquires waveform information from the waveform information acquisition unit 13 . The heartbeat interval determination unit 16a determines the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 16b.

[0055] The heartbeat feature amount calculation processing unit 16b acquires the heartbeat interval of the driver from the heartbeat interval specification unit 16a and acquires the wakefulness of the driver from the wakefulness estimation unit 12. The heartbeat feature amount calculation processing unit 16b calculates the heartbeat feature amount from the heartbeat interval. If the driver's alertness is equal to or higher than a threshold, the pulse feature amount calculation processing unit 16b calculates a normalization coefficient for the pulse feature amount based on the plurality of pieces of waveform information acquired by the waveform information acquisition unit 13. The threshold may be stored in an internal memory of the pulse feature amount calculation processing unit 16b, or may be provided from outside the drowsiness determination device 3. The plurality of pieces of waveform information are pieces of waveform information detected by the sensor 2 at mutually different times. The heartbeat feature amount calculation processing unit 16b normalizes the heartbeat feature amount using the normalization coefficient. If the driver's alertness is less than the threshold, the heartbeat feature calculation processing unit 16b does not calculate a normalization coefficient for the heartbeat feature, but normalizes the heartbeat feature using the normalization coefficient when the driver is not drowsy. The heartbeat feature amount calculation processing unit 16b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determining unit 15.

[0056] 7, it is assumed that the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature amount calculation unit 16, and the drowsiness presence / absence determination unit 15, which are components of the drowsiness determination device 3, are each realized by dedicated hardware as shown in Fig. 8. That is, it is assumed that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature amount calculation circuit 26, and a drowsiness presence / absence determination circuit 25. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the heart rate feature calculation circuit 26, and the drowsiness presence / absence determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0057] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. When the drowsiness determination device 3 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the image data acquisition unit 11, the arousal level estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature amount calculation unit 16, and the drowsiness presence / absence determination unit 15 is stored in a memory 31 shown in Fig. 3. Then, a processor 32 shown in Fig. 3 executes the program stored in the memory 31.

[0058] 8 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, while Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components in the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0059] Next, the operation of the drowsiness determination device 3 shown in Fig. 7 will be described. Except for the pulse feature amount calculation unit 16, the drowsiness determination device 3 is the same as that shown in Fig. 1. Therefore, only the operation of the pulse feature amount calculation unit 16 will be described here.

[0060] The heartbeat interval determination unit 16 a acquires waveform information from the waveform information acquisition unit 13 . The heartbeat interval determination unit 16a, like the heartbeat interval determination unit 14a shown in FIG. 1, determines the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 16b.

[0061] The heartbeat feature amount calculation processing unit 16b acquires the heartbeat interval of the driver from the heartbeat interval specification unit 16a and acquires the wakefulness of the driver from the wakefulness estimation unit 12. The pulse feature amount calculation processing unit 16b calculates the pulse feature amount from the pulse interval, similarly to the pulse feature amount calculation processing unit 14b shown in FIG.

[0062] The heartbeat feature amount calculation processing unit 16b acquires, for example, M pieces of waveform information including the latest waveform information and past waveform information from the waveform information acquisition unit 13 via the heartbeat interval identification unit 16a. The heartbeat feature amount calculation processor 16b compares the driver's alertness with a threshold value. If the driver's alertness is equal to or higher than the threshold, the pulse feature amount calculation processing unit 16b calculates a normalization coefficient for the pulse feature amount based on M pieces of waveform information, similar to the pulse feature amount calculation processing unit 14b shown in FIG. The pulse feature amount calculation processing unit 16b normalizes the pulse feature amount using a normalization coefficient, similar to the pulse feature amount calculation processing unit 14b shown in FIG.

[0063] If the driver's alertness is less than the threshold, the pulse feature amount calculation processing unit 16b does not calculate a normalization coefficient for the pulse feature amount, but normalizes the pulse feature amount using a normalization coefficient used when the driver is not drowsy. The normalization coefficient used when the driver is not drowsy may be stored in an internal memory of the pulse feature amount calculation processing unit 16b or may be provided from outside the drowsiness determination device 3. In addition, the normalization coefficient when the driver is not feeling drowsy may be, for example, a normalization coefficient calculated by the pulse feature value calculation processing unit 16b until a certain period of time has elapsed since the driver got into the vehicle. If the normalization coefficient of the heart rate feature is calculated based on waveform information obtained when the driver is feeling drowsy, the normalized heart rate feature will not reflect the level of drowsiness very well. As a result, even if the driver is feeling drowsy, the drowsiness presence / absence determination unit 15 may not obtain a determination result indicating that the driver is feeling drowsy. Therefore, the heart rate feature calculation processing unit 16b normalizes the heart rate feature using the normalization coefficient obtained when the driver is not feeling drowsy. The heartbeat feature amount calculation processing unit 16b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determining unit 15.

[0064] In the second embodiment described above, if the arousal level estimated by the arousal level estimation unit 12 is equal to or higher than a threshold, the pulse feature amount calculation unit 16 calculates a normalization coefficient for the pulse feature amount based on a plurality of pieces of waveform information and normalizes the pulse feature amount using the normalization coefficient. The drowsiness determination device 3 shown in FIG. 7 is configured so that, if the arousal level estimated by the arousal level estimation unit 12 is lower than the threshold, the pulse feature amount calculation unit 16 does not calculate a normalization coefficient for the pulse feature amount, but normalizes the pulse feature amount using a normalization coefficient used when the driver is not drowsy. Therefore, the drowsiness determination device 3 shown in FIG. 7 can detect drowsiness with higher accuracy than the drowsiness determination device 3 shown in FIG. 1.

[0065] Embodiment 3 In the third embodiment, a drowsiness determination device 3 will be described in which, if the level of wakefulness estimated by the wakefulness estimation unit 12 is less than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the level of wakefulness.

[0066] Fig. 9 is a configuration diagram showing a drowsiness determination device 3 according to embodiment 3. In Fig. 9, the same reference numerals as in Fig. 1 and Fig. 7 indicate the same or corresponding parts, and detailed description thereof will be omitted. Fig. 10 is a hardware configuration diagram showing the hardware of a drowsiness determination device 3 according to embodiment 3. In Fig. 10, the same reference numerals as in Fig. 2 and Fig. 8 indicate the same or corresponding parts, and detailed description thereof will be omitted. The drowsiness determination device 3 shown in FIG. 9 includes an image data acquisition unit 11, an arousal level estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature amount calculation unit 14, and a drowsiness presence / absence determination unit 17.

[0067] The drowsiness determination unit 17 is realized by, for example, a drowsiness determination circuit 27 shown in FIG. The drowsiness presence / absence determining unit 17 acquires the driver's level of alertness from the alertness estimation processing unit 12b, and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14b. If the alertness level of the driver is equal to or higher than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the alertness level of the driver and the normalized heart rate feature amount, similar to the drowsiness presence / absence determination unit 15 shown in Fig. 1. The threshold may be stored in an internal memory of the drowsiness presence / absence determination unit 17, or may be provided from outside the drowsiness determination device 3. If the driver's alertness level is less than the threshold, the drowsiness determination unit 17 determines whether the driver is drowsy based on the driver's alertness level.

[0068] 9, it is assumed that the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature value calculation unit 14, and the drowsiness presence / absence determination unit 17, which are components of the drowsiness determination device 3, are each realized by dedicated hardware as shown in Fig. 10. That is, it is assumed that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature value calculation circuit 24, and a drowsiness presence / absence determination circuit 27. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the heart rate feature calculation circuit 24, and the drowsiness determination circuit 27 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0069] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. When the drowsiness determination device 3 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the image data acquisition unit 11, the arousal level estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature amount calculation unit 14, and the drowsiness presence / absence determination unit 17 is stored in a memory 31 shown in Fig. 3. Then, a processor 32 shown in Fig. 3 executes the program stored in the memory 31.

[0070] 10 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, while Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components in the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0071] Next, the operation of the drowsiness determination device 3 shown in Fig. 9 will be described. Except for the drowsiness presence / absence determination unit 17, the drowsiness determination device 3 is the same as that shown in Fig. 1. Therefore, only the operation of the drowsiness presence / absence determination unit 17 will be described here.

[0072] The drowsiness presence / absence determining unit 17 acquires the driver's level of alertness from the alertness estimation processing unit 12b, and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14b. The drowsiness determination unit 17 compares the driver's alertness with a threshold value. If the driver's alertness is equal to or greater than a threshold, the drowsiness determination unit 17 determines whether the driver is drowsy based on the driver's alertness and the normalized heart rate features, similar to the drowsiness determination unit 15 shown in Figure 1.

[0073] If the driver's alertness level is less than the threshold, the drowsiness determination unit 17 determines whether the driver is drowsy based on the driver's alertness level. Specifically, the drowsiness determination unit 17 determines that the driver is drowsy if the driver's alertness is equal to or greater than a set value. If the driver's alertness is less than a set value, the drowsiness determination unit 17 determines that the driver is not drowsy. The set value may be stored in an internal memory of the drowsiness determination unit 17 or may be provided from outside the drowsiness determination device 3. The drowsiness presence / absence determining unit 17 outputs the determination result of the presence / absence of drowsiness to, for example, a driver monitoring system (not shown).

[0074] In the above-described third embodiment, if the alertness estimated by the alertness estimation unit 12 is equal to or greater than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the alertness and the normalized heart rate feature value output from the heart rate feature value calculation unit 14. The drowsiness determination device 3 shown in Fig. 9 is configured so that if the alertness estimated by the alertness estimation unit 12 is less than the threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the alertness. Therefore, the drowsiness determination device 3 shown in Fig. 9 can detect drowsiness with higher accuracy than the drowsiness determination device 3 shown in Fig. 1.

[0075] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments. [Industrial Applicability]

[0076] The present disclosure is suitable for a drowsiness determination device and a drowsiness determination method. [Explanation of symbols]

[0077] 1 camera, 2 sensor, 3 drowsiness determination device, 11 image data acquisition unit, 12 alertness estimation unit, 12a facial feature calculation unit, 12b alertness estimation processing unit, 13 waveform information acquisition unit, 14 heartbeat feature calculation unit, 14a heartbeat interval determination unit, 14b heartbeat feature calculation processing unit, 15 drowsiness presence / absence determination unit, 16 alertness estimation unit, 16a facial feature calculation unit, 16b alertness estimation processing unit, 17 drowsiness presence / absence determination unit, 21 image data acquisition circuit, 22 alertness estimation circuit, 23 waveform information acquisition circuit, 24 heartbeat feature calculation circuit, 25 drowsiness presence / absence determination circuit, 26 heartbeat feature calculation circuit, 27 drowsiness presence / absence determination circuit, 31 memory, 32 processor.

Claims

1. An image data acquisition unit that acquires image data showing a face image of a driver from a camera that photographs the face of the driver; an alertness estimation unit that estimates an alertness level indicating a level of alertness of the driver based on the image data acquired by the image data acquisition unit; a waveform information acquisition unit that acquires waveform information indicating the electrocardiogram waveform of the driver from a sensor that detects the electrocardiogram waveform of the driver; a heartbeat feature amount calculation unit that calculates a feature amount of the driver's heartbeat from an electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit; a drowsiness determination unit that determines whether the driver is drowsy based on the level of arousal estimated by the arousal level estimation unit and the heart rate feature amount calculated by the heart rate feature amount calculation unit, the waveform information acquisition unit acquires a plurality of pieces of waveform information detected by the sensor at different times, The heartbeat feature amount calculation unit calculating a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information, normalizing the heartbeat feature amount using the normalization coefficient, and outputting the normalized heartbeat feature amount to the drowsiness presence / absence determination unit; The drowsiness presence / absence determination unit A drowsiness determination device that determines whether or not the driver is drowsy based on the level of alertness estimated by the alertness estimation unit and the normalized heart rate feature value output from the heart rate feature value calculation unit.

2. The wakefulness level estimation unit The drowsiness determination device according to claim 1, characterized in that it calculates facial features of the driver from the image data acquired by the image data acquisition unit, and estimates the driver's alertness level based on the facial features.

3. the image data acquisition unit acquires a plurality of image data captured by the camera at different times, The wakefulness level estimation unit 3. The drowsiness determination device according to claim 2, further comprising: a normalization coefficient for the facial feature amount calculated based on the plurality of image data; a normalization coefficient for the facial feature amount normalized by the normalization coefficient; and an alertness level of the driver estimated based on the normalized facial feature amount.

4. The heartbeat feature amount calculation unit The drowsiness determination device according to claim 1, characterized in that the driver's heartbeat interval is identified based on the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit, and a characteristic amount of the heartbeat is calculated from the heartbeat interval.

5. The heartbeat feature amount calculation unit 2. The drowsiness determination device according to claim 1, wherein, if the arousal level estimated by the arousal level estimation unit is equal to or higher than a threshold, a normalization coefficient for the heartbeat feature amount is calculated based on the plurality of waveform information, and the heartbeat feature amount is normalized using the normalization coefficient; and, if the arousal level estimated by the arousal level estimation unit is less than the threshold, the normalization coefficient for the heartbeat feature amount is not calculated, but the heartbeat feature amount is normalized using a normalization coefficient used when the driver is not drowsy.

6. The drowsiness presence / absence determination unit 2. The drowsiness determination device according to claim 1, wherein, if the alertness estimated by the alertness estimation unit is equal to or greater than a threshold, it is determined whether the driver is drowsy based on the alertness and the normalized heart rate feature value output from the heart rate feature value calculation unit, and if the alertness is less than the threshold, it is determined whether the driver is drowsy based on the alertness.

7. The drowsiness presence / absence determination unit 2. The drowsiness determination device according to claim 1, wherein during learning, teacher data indicating the driver's alertness, the driver's heart rate features, and whether or not the driver is drowsy is provided to a learning model that learns whether or not the driver is drowsy, and the alertness estimated by the alertness estimation unit and the heart rate features calculated by the heart rate feature calculation unit are provided to the learning model, and a determination result of whether or not the driver is drowsy is obtained from the learning model.

8. The heartbeat feature amount calculation unit The drowsiness determination device according to claim 1, characterized in that, from the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit, the following characteristic quantities of the driver's heart rate are calculated: the average value of the driver's heartbeat intervals within a certain period, the standard deviation of the driver's heartbeat intervals within a certain period, the average value of the driver's heart rate within a certain period, the square root of the average value within a certain period of the square of the difference between two adjacent heartbeat intervals in the time direction, the number of times the difference between two adjacent heartbeat intervals in the time direction exceeds a threshold value within a certain period, or the percentage at which the difference between two adjacent heartbeat intervals in the time direction exceeds a threshold value within a certain period.

9. an image data acquisition unit acquires image data representing a facial image of the driver from a camera that captures an image of the driver's face; an alertness estimation unit estimating an alertness level indicating a level of alertness of the driver based on the image data acquired by the image data acquisition unit; a waveform information acquisition unit acquires waveform information indicating an electrocardiogram waveform of the driver from a sensor that detects an electrocardiogram waveform of the driver; a heartbeat feature amount calculation unit calculates a feature amount of the driver's heartbeat from an electrocardiogram waveform indicated by the waveform information acquisition unit; a drowsiness determination unit that determines whether the driver is drowsy based on the level of alertness estimated by the alertness estimation unit and the heartbeat feature amount calculated by the heartbeat feature amount calculation unit; and the waveform information acquisition unit acquires a plurality of pieces of waveform information detected by the sensor at different times, The heartbeat feature amount calculation unit calculating a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information, normalizing the heartbeat feature amount using the normalization coefficient, and outputting the normalized heartbeat feature amount to the drowsiness presence / absence determination unit; The drowsiness presence / absence determination unit A drowsiness determination method characterized by determining whether or not the driver is drowsy based on the level of alertness estimated by the alertness estimation unit and the normalized heart rate feature output from the heart rate feature calculation unit.

Images