Sleepiness assessment device and sleepiness assessment method

Abstract

A sleepiness assessment device (3) configured to comprise: an image data acquisition unit (11) for acquiring, from a camera (1) for imaging the face of a driver, image data representing a facial image of the driver; an alertness level estimation unit (12) for estimating, on the basis of the image data acquired by the image data acquisition unit (11), an alertness level indicating the degree of alertness of the driver; a waveform information acquisition unit (13) for acquiring, from a sensor (2) for detecting an electrocardiogram waveform of the driver, waveform information indicating the electrocardiogram waveform of the driver; and a heartbeat feature amount calculation unit (14) for calculating, from the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit (13), a feature amount pertaining to the heartbeat of the driver. The sleepiness assessment device (3) also comprises a sleepiness assessment unit (15) for assessing whether the driver is sleepy on the basis of the alertness level estimated by the alertness level estimation unit (12) and the heartbeat-related feature amount calculated by the heartbeat feature amount calculation unit (14).

Description

Drowsiness determination device and drowsiness determination method

[0001] The present disclosure relates to a drowsiness determination device and a drowsiness determination method.

[0002] There is a drowsiness determination device that determines whether a driver is drowsy. For example, Patent Document 1 discloses an example of such a drowsiness determination device that includes a drowsiness detection unit. The drowsiness detection unit detects the drowsiness of the driver based on a heartbeat interval obtained from the driver's electrocardiogram waveform.

[0003] Japanese Patent Application Laid-Open No. 2019-13737

[0004] Generally, the heart rate interval is longer when the driver is drowsy than when the driver is not drowsy. However, the heart rate interval changes depending on the activity state of the autonomic nervous system even when the driver is not drowsy. For example, the driver may become more relaxed after a certain period of time has passed since starting to drive the vehicle than when the driver first starts driving, which may change the activity state of the autonomic nervous system and cause the heart rate interval to become longer even when the driver is not drowsy. In the device disclosed in Patent Document 1, the drowsiness detection unit detects the driver's drowsiness based only on the heart rate interval. Therefore, there is a problem in that the drowsiness detection unit may erroneously detect drowsiness when the driver is not drowsy.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to obtain a drowsiness determination device that can reduce false detection of drowsiness more than the device disclosed in Patent Document 1.

[0006] The drowsiness determination device according to the present disclosure includes an image data acquisition unit that acquires image data representing a face image of the driver from a camera that captures the face of the driver, an alertness estimation unit that estimates an alertness level representing the level of alertness of the driver based on the image data acquired by the image data acquisition unit, a waveform information acquisition unit that acquires waveform information representing an electrocardiogram waveform of the driver from a sensor that detects an electrocardiogram waveform of the driver, and a heartbeat feature amount calculation unit that calculates a feature amount of the driver's heartbeat from the electrocardiogram waveform represented by the waveform information acquired by the waveform information acquisition unit. The drowsiness determination device also includes a drowsiness presence / absence determination unit that determines whether the driver is drowsy based on the alertness estimated by the alertness estimation unit and the heartbeat feature amount calculated by the heartbeat feature amount calculation unit.

[0007] According to the present disclosure, it is possible to reduce false detection of drowsiness compared to the device disclosed in Patent Document 1.

[0008] 5A is an explanatory diagram showing a distance L1 between the upper eyelid and the lower eyelid when the driver is not experiencing drowsiness, and FIG. 5B is an explanatory diagram showing a current distance L2 between the upper eyelid and the lower eyelid of the driver. FIG. 5B is an explanatory diagram showing an example of a heartbeat interval of a driver. FIG. 5A is a hardware configuration diagram showing the hardware of the drowsiness determination device 3 according to Embodiment 1. FIG. 5B is a hardware configuration diagram showing the hardware of the drowsiness determination device 3 according to Embodiment 3.

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1. FIG. 1 is a configuration diagram showing a drowsiness determination device 3 according to embodiment 1. FIG. 2 is a hardware configuration diagram showing hardware of the drowsiness determination device 3 according to embodiment 1. In FIG. 1 , a camera 1 is installed, for example, on the instrument panel of a vehicle, the windshield of the vehicle, or the ceiling of the vehicle. The camera 1 is realized, for example, by one or more visible light cameras, one or more infrared cameras, or a video camera. When the camera 1 is realized by an infrared camera, a light source that irradiates an area including the driver's face with infrared light for imaging may be provided. The light source is realized, for example, by an LED (Light Emitting Diode). The camera 1 captures an image of the driver's face and outputs image data indicating the driver's facial image to the drowsiness determination device 3.

[0011] The sensor 2 is realized, for example, by an electrocardiogram sensor that detects the electrocardiogram waveform of the driver while in contact with the driver, or by an electrocardiogram sensor that detects the electrocardiogram waveform of the driver while not in contact with the driver. The sensor 2 outputs waveform information indicating the electrocardiogram waveform of the driver to the drowsiness determination device 3. The sensor 2 is not limited to being realized by an electrocardiogram sensor, but may also be realized by an infrared camera. When the sensor 2 is realized by an infrared camera, the sensor 2 detects the electrocardiogram waveform of the driver based on the brightness of the face surface that changes in accordance with the driver's heartbeat.

[0012] The drowsiness determination device 3 includes an image data acquisition unit 11, an alertness estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature value calculation unit 14, and a drowsiness presence / absence determination unit 15. The drowsiness determination device 3 determines whether the driver is drowsy based on the image data output from the camera 1 and the waveform information output from the sensor 2.

[0013] The image data acquisition unit 11 is realized by, for example, an image data acquisition circuit 21 shown in Fig. 2. The image data acquisition unit 11 acquires image data representing a facial image of the driver from the camera 1. The image data acquisition unit 11 outputs the image data to the awakening level estimation unit 12.

[0014] The arousal level estimation unit 12 is realized by, for example, the arousal level estimation circuit 22 shown in FIG. 2 . The arousal level estimation unit 12 includes a facial feature amount calculation unit 12a and an arousal level estimation processing unit 12b. The arousal level estimation unit 12 acquires image data from the image data acquisition unit 11. The arousal level estimation unit 12 estimates the arousal level indicating the degree of arousal of the driver based on the image data. The arousal level is the apparent arousal level of the driver estimated from an image of the driver's face. The arousal level estimation unit 12 outputs the arousal level of the driver to the drowsiness presence / absence determination unit 15.

[0015] The facial feature amount calculation unit 12a acquires image data from the image data acquisition unit 11. The facial feature amount calculation unit 12a detects the driver's face from the image data and calculates facial feature amounts. The facial feature amount calculation unit 12a calculates a normalization coefficient for the facial feature amounts based on the multiple image data acquired by the image data acquisition unit 11. The multiple image data are multiple image data captured by the camera 1 at different times, and include the latest image data output from the image data acquisition unit 11 and previous image data output from the image data acquisition unit 11 before the latest image data. For example, if the number of data items in the multiple image data is N, the multiple image data include (N-1) previous image data items in addition to the latest image data. The N image data items are time-series data, and N is an integer greater than or equal to 2. The facial feature amount calculation unit 12a normalizes the facial feature amounts using the normalization coefficient. The facial feature amount calculation unit 12a outputs the normalized facial feature amounts to the arousal level estimation processing unit 12b.

[0016] The wakefulness estimation processing unit 12b acquires the normalized facial feature amounts from the facial feature amount calculation unit 12a. The wakefulness estimation processing unit 12b estimates the wakefulness level of the driver based on the normalized facial feature amounts. The wakefulness estimation processing unit 12b outputs the wakefulness level of the driver to the drowsiness presence determination unit 15.

[0017] The waveform information acquisition unit 13 is realized by, for example, the waveform information acquisition circuit 23 shown in Fig. 2. The waveform information acquisition unit 13 acquires waveform information indicating the electrocardiogram waveform of the driver from the sensor 2. The waveform information acquisition unit 13 outputs the waveform information to the pulse feature amount calculation unit 14.

[0018] The pulse feature amount calculation unit 14 is realized by, for example, the pulse feature amount calculation circuit 24 shown in FIG. 2 . The pulse feature amount calculation unit 14 includes a pulse interval identification unit 14a and a pulse feature amount calculation processing unit 14b. The pulse feature amount calculation unit 14 acquires waveform information from the waveform information acquisition unit 13. The pulse feature amount calculation unit 14 calculates the driver's pulse feature amount from the electrocardiogram waveform indicated by the waveform information. The pulse feature amount calculation unit 14 outputs the pulse feature amount to the drowsiness presence / absence determination unit 15.

[0019] The heartbeat interval specification unit 14a acquires waveform information from the waveform information acquisition unit 13. The heartbeat interval specification unit 14a specifies the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specification unit 14a outputs the heartbeat interval of the driver to the heartbeat feature value calculation processing unit 14b.

[0020] The pulse feature amount calculation processing unit 14b acquires the driver's heartbeat interval from the heartbeat interval identification unit 14a. The pulse feature amount calculation processing unit 14b calculates a heartbeat feature amount from the heartbeat interval. The pulse feature amount calculation processing unit 14b calculates a normalization coefficient for the heartbeat feature amount based on a plurality of pieces of waveform information acquired by the waveform information acquisition unit 13. The plurality of pieces of waveform information are waveform information detected by the sensor 2 at different times, and include the latest waveform information output from the waveform information acquisition unit 13 and previous waveform information output from the waveform information acquisition unit 13 before the latest waveform information. For example, if the number of pieces of waveform information is M, the plurality of pieces of waveform information includes (M-1) pieces of previous waveform information in addition to the latest waveform information. The M pieces of waveform information are time-series data, where M is an integer equal to or greater than 2. The pulse feature amount calculation processing unit 14b normalizes the heartbeat feature amount using the normalization coefficient. The pulse feature amount calculation processing unit 14b outputs the normalized heartbeat feature amount to the drowsiness presence / absence determination unit 15.

[0021] The drowsiness presence / absence determination unit 15 is realized, for example, by the drowsiness presence / absence determination circuit 25 shown in Fig. 2. The drowsiness presence / absence determination unit 15 determines whether the driver is drowsy based on the alertness level estimated by the alertness level estimation unit 12 and the heart rate feature amount calculated by the heart rate feature amount calculation unit 14. Specifically, the drowsiness presence / absence determination unit 15 acquires the driver's alertness level from the alertness level estimation processing unit 12b and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14b. Then, the drowsiness presence / absence determination unit 15 determines whether the driver is drowsy based on the driver's alertness level and the normalized heart rate feature amount.

[0022] 1, it is assumed that each of the components of the drowsiness determination device 3, that is, the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature value calculation unit 14, and the drowsiness presence / absence determination unit 15, is realized by dedicated hardware as shown in Fig. 2. That is, it is assumed that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature value calculation circuit 24, and a drowsiness presence / absence determination circuit 25. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the heartbeat feature calculation circuit 24, and the drowsiness presence / absence determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0023] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. The software or firmware is stored as a program in the memory of a computer. The computer refers to hardware that executes a program, and includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor).

[0024] 3 is a hardware configuration diagram of a computer when the drowsiness determination device 3 is realized by software, firmware, etc. When the drowsiness determination device 3 is realized by software, firmware, etc., a program for causing the computer to execute the respective processing procedures of the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature value calculation unit 14, and the drowsiness presence / absence determination unit 15 is stored in a memory 31. Then, a processor 32 of the computer executes the program stored in the memory 31.

[0025] 2 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, and Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components of the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0026] Next, the operation of the drowsiness determination device 3 shown in Fig. 1 will be described. Fig. 4 is a flowchart showing a drowsiness determination method, which is a processing procedure of the drowsiness determination device 3. For example, the camera 1 starts capturing an image of the driver's face when the driver gets into the vehicle and starts the engine. However, the start time of the camera 1 is not limited to the time when the driver starts the engine, and may be, for example, the time when the driver gets into the vehicle. The camera 1 outputs image data showing an image of the driver's face to the image data acquisition unit 11 of the drowsiness determination device 3. The facial image shown by the image data may be a video or a plurality of still images captured intermittently.

[0027] The sensor 2 detects the electrocardiogram waveform of the driver and outputs waveform information indicating the electrocardiogram waveform of the driver to the waveform information acquisition unit 13 of the drowsiness determination device 3.

[0028] The image data acquisition unit 11 acquires image data representing a face image of the driver from the camera 1 (step ST1 in FIG. 4 ). The image data acquisition unit 11 outputs the image data to the awakening level estimation unit 12.

[0029] The wakefulness estimation unit 12 acquires image data from the image data acquisition unit 11. The wakefulness estimation unit 12 estimates the wakefulness of the driver based on the image data (step ST2 in FIG. 4 ). The wakefulness estimation unit 12 outputs the wakefulness of the driver to the drowsiness presence / absence determination unit 15.

[0030] The wakefulness estimation process performed by the wakefulness estimation unit 12 will be described in detail below. The facial feature amount calculation unit 12a acquires image data from the image data acquisition unit 11. The facial feature amount calculation unit 12a detects the driver's face from the driver's facial image represented by the image data. Specifically, the facial feature amount calculation unit 12a can detect the driver's face by using, for example, a machine learning algorithm called AdaBoost (Adaptive Boosting) or a Haar-Like detector to which an algorithm called cascade is applied. The Haar-Like detector is a detector that handles Haar-Like features. The Haar-Like features are features calculated from the brightness differences of multiple local regions.

[0031] The facial feature amount calculation unit 12a calculates the feature amounts of the detected face. Specifically, the facial feature amount calculation unit 12a detects facial parts from the driver's face. The facial parts are, for example, the driver's eyes or the driver's mouth. Next, the facial feature amount calculation unit 12a calculates, for example, the degree of eye opening of the driver or the degree of mouth opening of the driver. The degree of eye opening is calculated, for example, as shown in the following equation (1), from the vertical distance L1 between the upper eyelid and the lower eyelid when the driver is not feeling drowsy and the current vertical distance L2 between the upper eyelid and the lower eyelid. When the driver is seated in the driver's seat and looking forward of the vehicle, the vertical direction of the upper eyelid and the lower eyelid is generally the same as the vertical direction.

[0032] As shown in FIG. 5A , distance L1 is the distance at the position where the distance between the upper eyelid and the lower eyelid is greatest when the driver is not experiencing drowsiness. Distance L1 is calculated, for example, from the driver's eyes, which are facial features detected within a certain time period after the driver has gotten into the vehicle. Distance L1 is not limited to being calculated from the driver's eyes, which are facial features detected within a certain time period after the driver has gotten into the vehicle. For example, distance L1 may be stored in the internal memory of the facial feature amount calculation unit 12a or provided from outside the drowsiness determination device 3. As shown in FIG. 5B , distance L2 is calculated from the driver's eyes, which are facial features detected from a facial image represented by the latest image data acquired from the image data acquisition unit 11. FIG. 5A is an explanatory diagram showing distance L1 between the upper eyelid and the lower eyelid when the driver is not experiencing drowsiness. FIG. 5B is an explanatory diagram showing distance L2 between the driver's current upper eyelid and the lower eyelid. Eye Opening Degree=(L2 / L1)×100 (1)

[0033] Here, the degree of eye opening is calculated using equation (1). However, this is merely an example, and the degree of eye opening may be calculated by normalizing the vertical distance L1 between the upper eyelid and the lower eyelid using facial features such as the nose, mouth, or ears. The amount of mouth opening is, for example, the vertical distance between the upper lip and the lower lip at the position where the distance between the upper lip and the lower lip is at its maximum.

[0034] The facial feature amount calculation unit 12a calculates facial features that are effective for estimating the driver's alertness based on, for example, the degree of eye opening or the amount of mouth opening. Such facial features include, for example, the proportion of time the eyes are closed within a certain period, the number of blinks within a certain period, or the number of yawns within a certain period. The proportion of time the eyes are closed within a certain period is calculated by dividing the time during which the degree of eye opening reaches 0% within the certain period by the duration of the certain period. The number of blinks within a certain period is calculated from the number of times the degree of eye opening reaches 0% within the certain period. Yawning is detected based on changes in the amount of mouth opening.

[0035] The facial feature amount calculation unit 12a normalizes the facial feature amounts to absorb individual differences in the facial feature amounts. Specifically, the facial feature amount calculation unit 12a accumulates facial feature amounts calculated based on N pieces of image data, including, for example, the latest image data and past image data, and calculates the average value or percentile value of the accumulated feature amounts. Then, the facial feature amount calculation unit 12a sets the calculated average value or percentile value as a normalization coefficient. The facial feature amount calculation unit 12a normalizes the facial feature amounts by subtracting the normalization coefficient from the facial feature amounts calculated based on the latest image data or by dividing the facial feature amounts calculated based on the latest image data by the normalization coefficient. The facial feature amount calculation unit 12a outputs the normalized facial feature amounts to the arousal level estimation processing unit 12b.

[0036] In the drowsiness determination device 3 shown in Fig. 1, the facial feature amount calculation unit 12a accumulates facial feature amounts calculated based on each of N pieces of image data including the latest image data and past image data. However, this is merely an example, and the facial feature amount calculation unit 12a may accumulate facial feature amounts calculated based on each of N pieces of image data including only past image data. Furthermore, the facial feature amounts calculated based on each of the N pieces of image data may be facial feature amounts calculated during a certain period of time after the driver gets into the vehicle.

[0037] The arousal level estimation processing unit 12b acquires the normalized facial feature amounts from the facial feature amount calculation unit 12a. The arousal level estimation processing unit 12b estimates the driver's arousal level based on the normalized facial feature amounts. Specifically, the arousal level estimation processing unit 12b provides the normalized facial feature amounts to a learning model that uses a general algorithm such as Random Forest or logistic regression, and acquires the driver's arousal level from the learning model.

[0038] During learning, the learning model is provided with normalized facial feature amounts and training data indicating the driver's level of alertness, and learns the driver's level of alertness. During inference, when the learning model receives the normalized facial feature amounts from the alertness estimation processing unit 12b, it outputs the driver's level of alertness to the alertness estimation processing unit 12b. The driver's level of alertness is, for example, data between 0 and 1. The alertness estimation processing unit 12b outputs the driver's level of alertness to the drowsiness presence / absence determination unit 15.

[0039] The waveform information acquiring unit 13 acquires waveform information indicating the electrocardiogram waveform of the driver from the sensor 2 (step ST3 in FIG. 4 ), and outputs the waveform information to the pulse feature amount calculating unit 14.

[0040] The pulse feature amount calculation unit 14 acquires waveform information from the waveform information acquisition unit 13. The pulse feature amount calculation unit 14 calculates the feature amount of the driver's heartbeat from the electrocardiogram waveform indicated by the waveform information (step ST4 in FIG. 4 ). The pulse feature amount calculation unit 14 outputs the feature amount of the heartbeat to the drowsiness presence / absence determination unit 15.

[0041] The calculation process of the heartbeat feature amount by the heartbeat feature amount calculation unit 14 will be described in detail below. The heartbeat interval determination unit 14a acquires waveform information from the waveform information acquisition unit 13. The heartbeat interval determination unit 14a determines the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. As shown in FIG. 6, the heartbeat interval is the time between two adjacent heartbeats. FIG. 6 is an explanatory diagram showing an example of the heartbeat interval of the driver. The heartbeat interval determination unit 14a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 14b.

[0042] The heartbeat feature calculation processing unit 14b acquires the heartbeat interval of the driver from the heartbeat interval identification unit 14a. The heartbeat feature calculation processing unit 14b calculates heartbeat feature amounts from the heartbeat intervals. The heartbeat feature amounts include, for example, the following feature amounts (1) to (6): (1) the average value of the driver's heartbeat intervals within a certain period; (2) the standard deviation of the driver's heartbeat intervals within a certain period; (3) the average value of the driver's heart rate within a certain period; (4) the square root of the average value within a certain period of the square of the difference between two heartbeat intervals adjacent in the time direction; (5) the number of times the difference between two heartbeat intervals adjacent in the time direction exceeds a threshold within a certain period; and (6) the rate at which the difference between two heartbeat intervals adjacent in the time direction exceeds a threshold within a certain period.

[0043] The pulse feature quantity calculation processing unit 14b normalizes the heartbeat feature quantities to absorb individual differences in the heartbeat feature quantities. Specifically, the pulse feature quantity calculation processing unit 14b acquires, for example, M pieces of waveform information including latest waveform information and past waveform information from the waveform information acquisition unit 13 via the heartbeat interval identification unit 14a. The pulse feature quantity calculation processing unit 14b accumulates the heartbeat feature quantities calculated based on each of the M pieces of waveform information and calculates the average value or percentile value of the accumulated feature quantities. Then, the pulse feature quantity calculation processing unit 14b sets the calculated average value or the calculated percentile value as the normalization coefficient. The pulse feature quantity calculation processing unit 14b normalizes the heartbeat feature quantities by subtracting the normalization coefficient from the heartbeat feature quantities calculated based on the latest waveform information or by dividing the heartbeat feature quantities calculated based on the latest waveform information by the normalization coefficient. The pulse feature quantity calculation processing unit 14b outputs the normalized heartbeat feature quantities to the drowsiness presence / absence determination unit 15.

[0044] In the drowsiness determination device 3 shown in Fig. 1, the heartbeat feature amount calculation processing unit 14b accumulates heartbeat feature amounts calculated based on each of M pieces of waveform information including the latest waveform information and past waveform information. However, this is merely an example, and the heartbeat feature amount calculation processing unit 14b may accumulate heartbeat feature amounts calculated based on each of M pieces of waveform information including only past waveform information. Furthermore, the heartbeat feature amounts calculated based on each of the M pieces of waveform information may be heartbeat feature amounts calculated during a certain period of time after the driver gets into the vehicle.

[0045] The drowsiness determination unit 15 acquires the driver's alertness from the alertness estimation processing unit 12 b and acquires the normalized heart rate feature from the heart rate feature calculation processing unit 14 b. The drowsiness determination unit 15 determines whether the driver is drowsy based on the driver's alertness and the normalized heart rate feature (step ST5 in FIG. 4 ). Specifically, the drowsiness determination unit 15 provides the driver's alertness and the normalized heart rate feature to a learning model that uses a general algorithm such as random forest or logistic regression, and acquires a determination result of whether the driver is drowsy from the learning model.

[0046] During learning, the learning model is provided with the driver's alertness, normalized heart rate features, and training data indicating whether the driver is drowsy, and learns whether the driver is drowsy. The training data is, for example, data "1" if the driver is not drowsy, and data "0" if the driver is drowsy. During inference, when the learning model is provided with the driver's alertness and normalized heart rate features from the drowsiness presence / absence determination unit 15, it outputs a determination result of whether the driver is drowsy to the drowsiness presence / absence determination unit 15. The drowsiness presence / absence determination unit 15 outputs the determination result of whether the driver is drowsy to, for example, a driver monitoring system (not shown).

[0047] In the first embodiment described above, the drowsiness determination device 3 is configured to include an image data acquisition unit 11 that acquires image data representing a facial image of the driver from a camera 1 that captures an image of the driver's face, an alertness estimation unit 12 that estimates an alertness level indicating the degree of alertness of the driver based on the image data acquired by the image data acquisition unit 11, a waveform information acquisition unit 13 that acquires waveform information representing an electrocardiographic waveform of the driver from a sensor 2 that detects an electrocardiographic waveform of the driver, and a heartbeat feature amount calculation unit 14 that calculates a feature amount of the driver's heartbeat from the electrocardiographic waveform indicated by the waveform information acquisition unit 13. The drowsiness determination device 3 also includes a drowsiness presence / absence determination unit 15 that determines whether the driver is drowsy based on the alertness estimated by the alertness estimation unit 12 and the heartbeat feature amount calculated by the heartbeat feature amount calculation unit 14. Therefore, the drowsiness determination device 3 can reduce false detection of drowsiness compared to the device disclosed in Patent Document 1.

[0048] In the drowsiness determination device 3 shown in FIG. 1 , the wakefulness estimation processing unit 12 b provides the normalized facial feature to a learning model and acquires the driver's wakefulness from the learning model. However, this is merely an example. If the normalized facial feature is, for example, the proportion of time the eyes are closed within a certain period of time, the wakefulness estimation processing unit 12 b may provide the proportion of time the eyes are closed within a certain period of time to a function that returns a higher wakefulness level the smaller the proportion of time the eyes are closed within a certain period of time, and acquire the driver's wakefulness from the function. Furthermore, if the normalized facial feature is, for example, the number of blinks within a certain period of time, the wakefulness estimation processing unit 12 b may provide the number of blinks within a certain period of time to a function that returns a higher wakefulness level the smaller the number of blinks within a certain period of time, and acquire the driver's wakefulness from the function. Furthermore, if the facial feature after normalization is, for example, the number of yawns in a certain period of time, the alertness estimation processing unit 12b may apply the number of yawns in a certain period of time to a function that returns a larger level of alertness the fewer the number of yawns in a certain period of time, and obtain the driver's level of alertness from the function.

[0049] In the drowsiness determination device 3 shown in FIG. 1 , the drowsiness determination unit 15 provides the driver's alertness and the normalized heart rate feature amount to a learning model and obtains a determination result of whether or not the driver is drowsy from the learning model. However, this is merely an example, and the drowsiness determination unit 15 may determine that the driver is drowsy if the driver's alertness is lower than a first determination threshold and the normalized heart rate feature amount is higher than a second determination threshold, and may determine that the driver is not drowsy otherwise. The first determination threshold and the second determination threshold may each be stored in an internal memory of the drowsiness determination unit 15 or may be provided from outside the drowsiness determination device 3. In addition, the drowsiness determination unit 15 may determine whether or not the driver is drowsy by placing more importance on the driver's level of alertness than on the features of the normalized heart rate, by accepting changes to the first determination threshold and the second determination threshold, or may determine whether or not the driver is drowsy by placing more importance on the features of the normalized heart rate than on the driver's level of alertness.

[0050] In the drowsiness determination device 3 shown in Fig. 1, the facial feature amount calculation unit 12a outputs normalized facial feature amounts to the alertness level estimation processing unit 12b. If the same driver always rides in the vehicle and there is no need to absorb individual differences in the facial feature amounts, the facial feature amount calculation unit 12a may output unnormalized facial feature amounts to the alertness level estimation processing unit 12b. Also, in the drowsiness determination device 3 shown in Fig. 1, the heartbeat feature amount calculation processing unit 14b outputs normalized heartbeat feature amounts to the drowsiness presence / absence determination unit 15. If the same driver always rides in the vehicle and there is no need to absorb individual differences in the heartbeat feature amounts, the heartbeat feature amount calculation processing unit 14b may output unnormalized heartbeat feature amounts to the drowsiness presence / absence determination unit 15.

[0051] Embodiment 2 In embodiment 2, a drowsiness determination device 3 will be described in which the heartbeat feature amount calculation unit 16 calculates a normalization coefficient for the heartbeat feature amount based on a plurality of pieces of waveform information only when the arousal level estimated by the arousal level estimation unit 12 is equal to or higher than a threshold level.

[0052] Fig. 7 is a configuration diagram showing a drowsiness determination device 3 according to embodiment 2. In Fig. 7, the same reference numerals as those in Fig. 1 indicate the same or corresponding parts, and therefore detailed description thereof will be omitted. Fig. 8 is a hardware configuration diagram showing hardware of the drowsiness determination device 3 according to embodiment 2. In Fig. 8, the same reference numerals as those in Fig. 2 indicate the same or corresponding parts, and therefore detailed description thereof will be omitted. The drowsiness determination device 3 shown in Fig. 7 includes an image data acquisition unit 11, an alertness level estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature amount calculation unit 16, and a drowsiness presence / absence determination unit 15.

[0053] The pulse feature amount calculation unit 16 is realized by, for example, the pulse feature amount calculation circuit 26 shown in FIG. 8 . The pulse feature amount calculation unit 16 includes a pulse interval identification unit 16a and a pulse feature amount calculation processing unit 16b. The pulse feature amount calculation unit 16 acquires waveform information from the waveform information acquisition unit 13 and acquires the driver's alertness from the alertness level estimation unit 12. The pulse feature amount calculation unit 16 calculates the driver's heartbeat feature amount from the electrocardiogram waveform indicated by the waveform information. If the driver's alertness level is equal to or higher than a threshold, the pulse feature amount calculation unit 16 calculates a normalization coefficient for the heartbeat feature amount based on multiple pieces of waveform information and normalizes the heartbeat feature amount using the normalization coefficient. If the driver's alertness level is lower than the threshold, the pulse feature amount calculation unit 16 does not calculate a normalization coefficient for the heartbeat feature amount, but normalizes the heartbeat feature amount using a normalization coefficient used when the driver is not drowsy. The heartbeat feature amount calculation unit 16 outputs the normalized heartbeat feature amount to the drowsiness presence / absence determination unit 15 .

[0054] The heartbeat interval specifying unit 16a acquires waveform information from the waveform information acquiring unit 13. The heartbeat interval specifying unit 16a specifies the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature value calculation processing unit 16b.

[0055] The pulse feature amount calculation processing unit 16b acquires the driver's heartbeat interval from the heartbeat interval identification unit 16a and acquires the driver's alertness from the alertness level estimation unit 12. The pulse feature amount calculation processing unit 16b calculates a heartbeat feature amount from the heartbeat interval. If the driver's alertness is equal to or greater than a threshold, the pulse feature amount calculation processing unit 16b calculates a normalization coefficient for the heartbeat feature amount based on the plurality of waveform information acquired by the waveform information acquisition unit 13. The threshold may be stored in an internal memory of the pulse feature amount calculation processing unit 16b or may be provided from outside the drowsiness determination device 3. The plurality of waveform information are plurality of waveform information detected by the sensor 2 at different times. The pulse feature amount calculation processing unit 16b normalizes the heartbeat feature amount using the normalization coefficient. If the driver's alertness is less than the threshold, the pulse feature amount calculation processing unit 16b does not calculate a normalization coefficient for the heartbeat feature amount, but normalizes the heartbeat feature amount using a normalization coefficient used when the driver is not drowsy. The heartbeat feature value calculation processing unit 16 b outputs the normalized heartbeat feature value to the drowsiness presence / absence determining unit 15 .

[0056] 7 assumes that the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature amount calculation unit 16, and the drowsiness presence / absence determination unit 15, which are components of the drowsiness determination device 3, are each realized by dedicated hardware such as that shown in Fig. 8. That is, it assumes that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature amount calculation circuit 26, and a drowsiness presence / absence determination circuit 25. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the pulse feature amount calculation circuit 26, and the drowsiness presence / absence determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0057] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. When the drowsiness determination device 3 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature amount calculation unit 16, and the drowsiness presence / absence determination unit 15 is stored in a memory 31 shown in Fig. 3. Then, a processor 32 shown in Fig. 3 executes the program stored in the memory 31.

[0058] 8 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, while Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components of the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0059] Next, the operation of the drowsiness determination device 3 shown in Fig. 7 will be described. Except for the pulse feature amount calculation unit 16, the drowsiness determination device 3 is the same as that shown in Fig. 1. Therefore, only the operation of the pulse feature amount calculation unit 16 will be described here.

[0060] The heartbeat interval specifying unit 16a acquires waveform information from the waveform information acquiring unit 13. Similar to the heartbeat interval specifying unit 14a shown in Fig. 1, the heartbeat interval specifying unit 16a specifies the heartbeat interval of the driver based on the electrocardiogram waveform indicated by the waveform information. The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 16b.

[0061] The pulse feature amount calculation processing unit 16b acquires the heartbeat interval of the driver from the heartbeat interval identification unit 16a and acquires the driver's alertness level from the alertness level estimation unit 12. The pulse feature amount calculation processing unit 16b calculates the heartbeat feature amount from the heartbeat interval, similar to the pulse feature amount calculation processing unit 14b shown in FIG. 1 .

[0062] The pulse feature quantity calculation processing unit 16b acquires, for example, M pieces of waveform information including latest waveform information and past waveform information from the waveform information acquisition unit 13 via the pulse interval identification unit 16a. The pulse feature quantity calculation processing unit 16b compares the driver's alertness with a threshold. If the driver's alertness is equal to or higher than the threshold, the pulse feature quantity calculation processing unit 16b calculates a normalization coefficient for the pulse feature quantity based on the M pieces of waveform information, similar to the pulse feature quantity calculation processing unit 14b shown in FIG. 1. The pulse feature quantity calculation processing unit 16b normalizes the pulse feature quantity using the normalization coefficient, similar to the pulse feature quantity calculation processing unit 14b shown in FIG. 1.

[0063] If the driver's alertness is below the threshold, the heartbeat feature amount calculation processing unit 16b does not calculate a normalization coefficient for the heartbeat feature amount, but normalizes the heartbeat feature amount using a normalization coefficient used when the driver is not drowsy. The normalization coefficient used when the driver is not drowsy may be stored in an internal memory of the heartbeat feature amount calculation processing unit 16b or may be provided from outside the drowsiness determination device 3. Furthermore, the normalization coefficient used when the driver is not drowsy may be, for example, a normalization coefficient calculated by the heartbeat feature amount calculation processing unit 16b within a certain period of time since the driver got into the vehicle. If the normalization coefficient for the heartbeat feature amount is calculated based on waveform information used when the driver is drowsy, the normalized heartbeat feature amount will not reflect the strength of drowsiness very well. As a result, even if the driver is drowsy, the drowsiness presence / absence determination unit 15 may not obtain a determination result indicating drowsiness. Therefore, the heartbeat feature value calculation processing unit 16b normalizes the heartbeat feature value using the normalization coefficient when the driver is not drowsy. The heartbeat feature value calculation processing unit 16b outputs the normalized heartbeat feature value to the drowsiness presence / absence determination unit 15.

[0064] In the second embodiment described above, if the arousal level estimated by the arousal level estimation unit 12 is equal to or higher than a threshold, the pulse feature amount calculation unit 16 calculates a normalization coefficient for the pulse feature amount based on a plurality of pieces of waveform information and normalizes the pulse feature amount using the normalization coefficient. The drowsiness determination device 3 shown in FIG. 7 is configured so that, if the arousal level estimated by the arousal level estimation unit 12 is lower than the threshold, the pulse feature amount calculation unit 16 does not calculate a normalization coefficient for the pulse feature amount, but normalizes the pulse feature amount using a normalization coefficient used when the driver is not drowsy. Therefore, the drowsiness determination device 3 shown in FIG. 7 can detect drowsiness with higher accuracy than the drowsiness determination device 3 shown in FIG. 1.

[0065] Embodiment 3 In embodiment 3, a drowsiness determination device 3 will be described in which, if the wakefulness level estimated by the wakefulness level estimation unit 12 is less than a threshold value, the drowsiness presence / absence determination unit 17 determines whether or not the driver is drowsy based on the wakefulness level.

[0066] Fig. 9 is a configuration diagram showing a drowsiness determination device 3 according to embodiment 3. In Fig. 9, the same reference numerals as those in Figs. 1 and 7 indicate the same or corresponding parts, and therefore detailed description thereof will be omitted. Fig. 10 is a hardware configuration diagram showing hardware of the drowsiness determination device 3 according to embodiment 3. In Fig. 10, the same reference numerals as those in Figs. 2 and 8 indicate the same or corresponding parts, and therefore detailed description thereof will be omitted. The drowsiness determination device 3 shown in Fig. 9 includes an image data acquisition unit 11, an arousal level estimation unit 12, a waveform information acquisition unit 13, a heartbeat feature amount calculation unit 14, and a drowsiness presence / absence determination unit 17.

[0067] The drowsiness presence / absence determination unit 17 is realized by, for example, a drowsiness presence / absence determination circuit 27 shown in FIG. 10 . The drowsiness presence / absence determination unit 17 acquires the driver's alertness from the alertness estimation processing unit 12 b and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14 b. If the driver's alertness is equal to or greater than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the driver's alertness and the normalized heart rate feature amount, similar to the drowsiness presence / absence determination unit 15 shown in FIG. 1 . The threshold may be stored in an internal memory of the drowsiness presence / absence determination unit 17 or may be provided from outside the drowsiness determination device 3. If the driver's alertness is less than the threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the driver's alertness.

[0068] 9 assumes that each of the components of the drowsiness determination device 3, namely, the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the pulse feature amount calculation unit 14, and the drowsiness presence / absence determination unit 17, is realized by dedicated hardware such as that shown in Fig. 10. That is, it assumes that the drowsiness determination device 3 is realized by an image data acquisition circuit 21, an alertness estimation circuit 22, a waveform information acquisition circuit 23, a pulse feature amount calculation circuit 24, and a drowsiness presence / absence determination circuit 27. Each of the image data acquisition circuit 21, the alertness estimation circuit 22, the waveform information acquisition circuit 23, the pulse feature amount calculation circuit 24, and the drowsiness presence / absence determination circuit 27 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0069] The components of the drowsiness determination device 3 are not limited to those realized by dedicated hardware, and the drowsiness determination device 3 may be realized by software, firmware, or a combination of software and firmware. When the drowsiness determination device 3 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the image data acquisition unit 11, the alertness estimation unit 12, the waveform information acquisition unit 13, the heartbeat feature amount calculation unit 14, and the drowsiness presence / absence determination unit 17 is stored in a memory 31 shown in Fig. 3. Then, a processor 32 shown in Fig. 3 executes the program stored in the memory 31.

[0070] 10 shows an example in which each of the components of the drowsiness determination device 3 is realized by dedicated hardware, while Fig. 3 shows an example in which the drowsiness determination device 3 is realized by software, firmware, etc. However, this is merely an example, and some of the components of the drowsiness determination device 3 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0071] Next, the operation of the drowsiness determination device 3 shown in Fig. 9 will be described. The drowsiness determination device 3 is the same as that shown in Fig. 1 except for the drowsiness presence / absence determination unit 17. Therefore, only the operation of the drowsiness presence / absence determination unit 17 will be described here.

[0072] The drowsiness presence / absence determination unit 17 acquires the driver's alertness from the alertness estimation processing unit 12 b and acquires the normalized heart rate feature amount from the heart rate feature amount calculation processing unit 14 b. The drowsiness presence / absence determination unit 17 compares the driver's alertness with a threshold. If the driver's alertness is equal to or greater than the threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the driver's alertness and the normalized heart rate feature amount, similar to the drowsiness presence / absence determination unit 15 shown in FIG. 1 .

[0073] If the driver's alertness is less than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy based on the driver's alertness. Specifically, if the driver's alertness is equal to or greater than a set value, the drowsiness presence / absence determination unit 17 determines that the driver is drowsy. If the driver's alertness is less than the set value, the drowsiness presence / absence determination unit 17 determines that the driver is not drowsy. The set value may be stored in an internal memory of the drowsiness presence / absence determination unit 17, or may be provided from outside the drowsiness determination device 3. The drowsiness presence / absence determination unit 17 outputs the determination result of whether the driver is drowsy to, for example, a driver monitoring system (not shown).

[0074] In the third embodiment described above, if the alertness estimated by the alertness estimation unit 12 is equal to or greater than a threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy, based on the alertness and the normalized heart rate feature value output from the heart rate feature value calculation unit 14. The drowsiness determination device 3 shown in Fig. 9 is configured so that, if the alertness estimated by the alertness estimation unit 12 is less than the threshold, the drowsiness presence / absence determination unit 17 determines whether the driver is drowsy, based on the alertness. Therefore, the drowsiness determination device 3 shown in Fig. 9 can detect drowsiness with higher accuracy than the drowsiness determination device 3 shown in Fig. 1.

[0075] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0076] The present disclosure is suitable for a drowsiness determination device and a drowsiness determination method.

[0077] REFERENCE SIGNS LIST 1 Camera, 2 Sensor, 3 Drowsiness determination device, 11 Image data acquisition unit, 12 Arousal level estimation unit, 12a Facial feature amount calculation unit, 12b Arousal level estimation processing unit, 13 Waveform information acquisition unit, 14 Heartbeat feature amount calculation unit, 14a Heartbeat interval identification unit, 14b Heartbeat feature amount calculation processing unit, 15 Drowsiness presence / absence determination unit, 16 Arousal level estimation unit, 16a Facial feature amount calculation unit, 16b Arousal level estimation processing unit, 17 Drowsiness presence / absence determination unit, 21 Image data acquisition circuit, 22 Arousal level estimation circuit, 23 Waveform information acquisition circuit, 24 Heartbeat feature amount calculation circuit, 25 Drowsiness presence / absence determination circuit, 26 Heartbeat feature amount calculation circuit, 27 Drowsiness presence / absence determination circuit, 31 Memory, 32 Processor.

Claims

1. An image data acquisition unit that acquires image data showing a face image of a driver from a camera that captures the face of the driver; an alertness estimation unit that estimates an alertness level indicating a level of alertness of the driver based on the image data acquired by the image data acquisition unit; a waveform information acquiring unit that acquires waveform information indicating an electrocardiogram waveform of the driver from a sensor that detects an electrocardiogram waveform of the driver; a heartbeat feature amount calculation unit that calculates a feature amount of the driver's heartbeat from an electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit; a drowsiness presence / absence determination unit that determines whether or not the driver is drowsy based on the level of awakening estimated by the awakening level estimation unit and the heart rate feature amount calculated by the heart rate feature amount calculation unit, the waveform information acquisition unit acquires a plurality of pieces of waveform information each having a different detection time by the sensor, The heart rate feature amount calculation unit calculating a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information, normalizing the heartbeat feature amount using the normalization coefficient, and outputting the normalized heartbeat feature amount to the drowsiness presence / absence determination unit; The drowsiness presence / absence determination unit is A drowsiness determination device characterized by determining whether or not the driver is drowsy based on the level of alertness estimated by the alertness estimation unit and the normalized heart rate feature output from the heart rate feature calculation unit.

2. The wakefulness estimation unit, The drowsiness determination device according to claim 1 , further comprising: a facial feature of the driver calculated from the image data acquired by the image data acquisition unit; and an alertness level of the driver estimated based on the facial feature.

3. the image data acquisition unit acquires a plurality of image data captured by the camera at different times, The wakefulness estimation unit, 3. The drowsiness determination device according to claim 2, further comprising: a normalization coefficient for the facial features calculated based on the plurality of image data; a normalization coefficient for the facial features normalized by the normalization coefficient; and an alertness level of the driver estimated based on the normalized facial features.

4. The heart rate feature amount calculation unit The drowsiness determination device according to claim 1, characterized in that the driver's heart beat interval is identified based on an electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit, and a characteristic amount of the heart beat is calculated from the heart beat interval.

5. The heart rate feature amount calculation unit 2. The drowsiness determination device according to claim 1, further comprising: a normalization coefficient for the heartbeat feature amount calculated based on the plurality of waveform information when the alertness estimated by the alertness estimation unit is equal to or greater than a threshold; and a normalization coefficient for when the driver is not drowsy when the alertness estimated by the alertness estimation unit is less than the threshold, the normalization coefficient for the heartbeat feature amount is not calculated, and the heartbeat feature amount is normalized using a normalization coefficient used when the driver is not drowsy.

6. The drowsiness presence / absence determination unit is The drowsiness determination device according to claim 1, characterized in that, if the alertness estimated by the alertness estimation unit is equal to or higher than a threshold, a determination is made as to whether or not the driver is drowsy based on the alertness and the normalized heart rate feature output from the heart rate feature calculation unit, and, if the alertness is less than the threshold, a determination is made as to whether or not the driver is drowsy based on the alertness.

7. The drowsiness presence / absence determination unit is The drowsiness determination device according to claim 1, characterized in that, during learning, a learning model is provided with teacher data indicating the driver's alertness, the driver's heart rate features, and the presence or absence of drowsiness, and the alertness estimated by the alertness estimation unit and the heart rate features calculated by the heart rate feature calculation unit are provided to the learning model which learns the presence or absence of drowsiness, and a determination result of whether the driver is drowsy or not is obtained from the learning model.

8. The heart rate feature amount calculation unit The drowsiness determination device of claim 1, characterized in that, from the electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit, the following characteristic quantities of the driver's heart rate are calculated: the average value of the driver's heart rate intervals within a certain period of time, the standard deviation of the driver's heart rate intervals within a certain period of time, the square root of the average value of the square of the difference between two adjacent heart rate intervals in the time direction within a certain period of time, the number of times that the difference between two adjacent heart rate intervals in the time direction becomes greater than a threshold value within a certain period of time, or the percentage at which the difference between two adjacent heart rate intervals in the time direction becomes greater than a threshold value within a certain period of time.

9. The image data acquisition unit acquires image data representing a facial image of the driver from a camera that captures the face of the driver, an awakening level estimation unit estimating an awakening level indicating a degree to which the driver is awakened based on the image data acquired by the image data acquisition unit; a waveform information acquisition unit acquires waveform information indicating an electrocardiogram waveform of the driver from a sensor that detects an electrocardiogram waveform of the driver; a heartbeat feature amount calculation unit calculates a feature amount of the driver's heartbeat from an electrocardiogram waveform indicated by the waveform information acquired by the waveform information acquisition unit; a drowsiness presence / absence determination unit that determines whether the driver is drowsy based on the level of awakening estimated by the awakening level estimation unit and the heart rate feature amount calculated by the heart rate feature amount calculation unit; the waveform information acquisition unit acquires a plurality of pieces of waveform information each having a different detection time by the sensor, The heart rate feature amount calculation unit calculating a normalization coefficient for the heartbeat feature amount based on the plurality of pieces of waveform information, normalizing the heartbeat feature amount using the normalization coefficient, and outputting the normalized heartbeat feature amount to the drowsiness presence / absence determination unit; The drowsiness presence / absence determination unit is A drowsiness determination method characterized by determining whether or not the driver is drowsy based on the level of alertness estimated by the alertness estimation unit and the normalized heart rate feature output from the heart rate feature calculation unit.