Drowsiness estimation device and drowsiness estimation method

Abstract

This drowsiness estimation device is provided with: a sensing unit (11) that, on the basis of frames of captured images of the face of an occupant of a moving body, acquires drowsiness-related information for each frame; a first noise factor detection unit (12) that detects noise factor activity of the occupant on the basis of the drowsiness-related information; a feature quantity calculation unit (13) that calculates a drowsiness estimation feature quantity on the basis of noise factor-removed drowsiness-related information, which is obtained by removing, from the original drowsiness-related information, the drowsiness-related information on the basis of which the first noise factor detection unit (12) detected the noise factor activity; a drowsiness score calculation unit (14) that calculates a drowsiness score using the drowsiness estimation feature quantity calculated by the feature quantity calculation unit (13); and a drowsiness estimation unit (15) that estimates the occupant's drowsiness on the basis of the drowsiness score calculated by the drowsiness score calculation unit (14).

Description

Drowsiness estimation device and drowsiness estimation method

[0001] The present disclosure relates to a drowsiness estimation device and a drowsiness estimation method.

[0002] Generally, characteristics of a person when drowsiness occurs include a decrease in the degree of eye opening, an increase in the number of blinks, etc. In light of this, a conventional technology for estimating the drowsiness of a vehicle occupant is known that estimates the drowsiness using feature amounts such as the degree of eye opening or the number of blinks extracted from information on the occupant's face (for example, Patent Document 1).

[0003] JP 2011-48531 A

[0004] Some human behaviors, such as looking down, frowning, or smiling, exhibit characteristics similar to those observed when drowsiness occurs. In conventional technologies, when estimating an occupant's drowsiness, if the occupant exhibits behaviors similar to those observed when drowsiness occurs, such as those described above, the accuracy of estimating the occupant's drowsiness may be reduced. The technology disclosed in Patent Document 1, however, does not assess drowsiness if a change in the angle of the face occurs immediately after eyelid movement or if a gaze shift occurs when eyelid movement occurs. Therefore, even a necessary drowsiness assessment may not be performed. This still does not solve the problem of reduced accuracy in estimating the occupant's drowsiness.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a drowsiness estimation device that, when estimating the drowsiness of an occupant of a moving vehicle, prevents the accuracy of estimating the drowsiness of the occupant from decreasing due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0006] The drowsiness estimation device according to the present disclosure includes a sensing unit that acquires drowsiness-related information indicating a state related to the drowsiness of an occupant of a moving body for each frame based on frames of an image capturing the face of the occupant; a first noise factor detection unit that detects noise-causing behavior by the occupant, which is behavior involving eye movement similar to behavior caused by drowsiness, based on the drowsiness-related information acquired by the sensing unit; a feature calculation unit that calculates drowsiness estimation feature amounts for estimating the drowsiness of the occupant based on noise-factor-removed drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit from the drowsiness-related information acquired by the sensing unit; a drowsiness score calculation unit that calculates a drowsiness score using the drowsiness estimation feature amounts calculated by the feature calculation unit; and a drowsiness estimation unit that estimates the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit.

[0007] According to the present disclosure, when estimating the drowsiness of an occupant of a moving body, it is possible to prevent a decrease in the accuracy of estimating the drowsiness of the occupant due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0008] 5A and 5B are diagrams illustrating an example of the configuration of a drowsiness estimation device according to embodiment 1. FIG. 5B is a flowchart illustrating the operation of the drowsiness estimation device according to embodiment 1. FIG. 5C is a flowchart illustrating details of a feature calculation process in a case where, in embodiment 1, the feature calculation unit identifies post-noise-factor-removal drowsiness-related information based on an exclusion target flag assigned by the sensing result selection unit to drowsiness-related information to be excluded, and calculates drowsiness estimation feature amounts based on the identified post-noise-factor-removal drowsiness-related information. FIG. 5D is a flowchart illustrating details of a feature calculation process in a case where, in embodiment 1, the feature calculation unit calculates drowsiness estimation feature amounts based on the post-noise-factor-removal drowsiness-related information output from the sensing result selection unit. 5A and 5B are diagrams illustrating an example of the hardware configuration of a drowsiness estimation device according to embodiment 1. FIG. 5C is a diagram illustrating an example of the configuration of a drowsiness estimation device according to embodiment 2. FIG. 5D is a flowchart illustrating the operation of the drowsiness estimation device according to embodiment 2. FIG. 5E is a diagram illustrating an example of the configuration of a drowsiness estimation device according to embodiment 3. FIG. 5F is a flowchart illustrating the operation of the drowsiness estimation device according to embodiment 3. 1 is a diagram showing an example of the configuration of a drowsiness estimation device that combines the configurations of the drowsiness estimation device according to embodiment 1, the drowsiness estimation device according to embodiment 2, and the drowsiness estimation device according to embodiment 3. FIG. 2 is a flowchart for explaining the operation of a drowsiness estimation device that combines the configurations of the drowsiness estimation device according to embodiment 1, the drowsiness estimation device according to embodiment 2, and the drowsiness estimation device according to embodiment 3.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram illustrating an example configuration of a drowsiness estimation device 1 according to embodiment 1. The drowsiness estimation device 1 according to embodiment 1 is connected to an imaging device 2, and estimates the drowsiness of a person (hereinafter referred to as a "subject") whose drowsiness is to be estimated based on an image captured by the imaging device 2. In embodiment 1, the subject is assumed to be a driver of a vehicle (not shown). It is also assumed that the drowsiness estimation device 1 according to embodiment 1 is mounted on the vehicle. In the following embodiment 1, the driver of the vehicle will also be simply referred to as a "driver."

[0010] The imaging device 2 is mounted on a vehicle. The imaging device 2 is installed in the center of the vehicle's dashboard, on an A-pillar, on an instrument panel, or the like so as to be able to capture an image of at least the driver's face. The imaging device 2 may be shared with a so-called "driver monitoring system (DMS)." The imaging device 2 is a visible light camera or an infrared camera. If the imaging device 2 is an infrared camera, the infrared camera is provided with a light source (not shown) that irradiates an area including the driver's face with infrared light for imaging. This light source is, for example, configured by an LED (Light Emitting Diode). The imaging device 2 outputs the captured image (hereinafter referred to as the "captured image") to the drowsiness estimation device 1.

[0011] The drowsiness estimation device 1 includes a sensing unit 11, a first noise factor detection unit 12, a feature calculation unit 13, a drowsiness score calculation unit 14, and a drowsiness estimation unit 15. The feature calculation unit 13 includes a sensing result selection unit 131.

[0012] The sensing unit 11 acquires information indicating the driver's drowsiness-related state (hereinafter referred to as "drowsiness-related information") based on a captured image of the driver's face. The sensing unit 11 acquires the captured image frame by frame from the imaging device 2. The sensing unit 11 acquires the drowsiness-related information for each frame. In the first embodiment, the process of acquiring drowsiness-related information performed by the sensing unit 11 is referred to as "sensing process."

[0013] The driver's drowsiness-related conditions include the degree to which the driver's eyes are open (in other words, the degree of eyelid opening), the degree to which the driver's mouth is open (in other words, the degree of mouth opening), the positions of the driver's facial feature points, the driver's line of sight, the driver's facial orientation, and the driver's head position. The facial feature points include the outer corners and inner corners of the eyes, points on the outer periphery of the eyes, etc., and the positions of the driver's facial feature points are indicated, for example, by coordinates on a captured image. The driver's facial orientation is expressed, for example, by an angle with 0 degrees as the reference when the driver is facing forward in the vehicle's traveling direction. The driver's head position is expressed by coordinates in real space. Since the installation position and angle of view of the imaging device 2 are known in advance, the sensing unit 11 can calculate the driver's facial orientation and head position from the captured image. In the first embodiment, the conditions that are considered to be conditions related to the driver's drowsiness are determined in advance by an administrator or the like.

[0014] The sensing unit 11 may detect a driver's drowsiness-related state using a known image recognition technology and acquire drowsiness-related information. The sensing unit 11 outputs the acquired drowsiness-related information to the first noise factor detection unit 12. At this time, the sensing unit 11 may, for example, associate the drowsiness-related information with a frame of the captured image from which the drowsiness-related information was acquired and output the associated information to the first noise factor detection unit 12. In the following description, the drowsiness-related information output by the sensing unit 11 to the first noise factor detection unit 12, or information in which the drowsiness-related information is associated with a frame of the captured image from which the drowsiness-related information was acquired, is also referred to as a "sensing result." It is assumed that each frame of the captured image is assigned information indicating the date and time of capture.

[0015] The first noise factor detection unit 12 detects driver behavior accompanied by eye movement (hereinafter referred to as "noise factor behavior") that is similar to behavior caused by drowsiness, based on the drowsiness-related information acquired by the sensing unit 11. In the first embodiment, the process of detecting a noise factor performed by the first noise factor detection unit 12 is referred to as a "first noise factor detection process."

[0016] Examples of noise-causing behaviors include looking downward (so-called downward gaze) or squinting. For example, a driver may look downward when looking at a meter. Also, for example, a driver may make a so-called "frowning face" when the sunlight is dazzling. In the first embodiment, a so-called frowning face is also referred to as a "patient face." People generally narrow their eyes when making a "patient face." Also, for example, people generally narrow their eyes when smiling.

[0017] On the other hand, when people feel sleepy, they cover their eyes with their eyelids to close them, or close their eyes to get rid of the sleepiness. When the eyes are covered with their eyelids or closed, the eyes narrow.

[0018] In this way, the above-mentioned behaviors of looking down, making a distressed face, or smiling can be considered to be noise-causing behaviors similar to behaviors caused by drowsiness, such as covering the eyes with the eyelids or closing the eyes. The first noise factor detection unit 12 detects such noise-causing behaviors. Note that the types of behaviors that are considered to be noise-causing behaviors are determined in advance by an administrator or the like.

[0019] For example, if the noise-causing behavior is squinting, the first noise factor detection unit 12 detects that the driver has performed the noise-causing behavior if the driver's eyelid opening degree is equal to or less than a predetermined threshold (hereinafter referred to as the "eyelid opening degree determination threshold") based on the drowsiness-related information. The first noise factor detection unit 12 may also detect that the driver has performed the noise-causing behavior if, for example, the driver's eyelid opening degree is smaller than the eyelid opening degree based on the immediately preceding drowsiness-related information by a predetermined threshold (hereinafter referred to as the "eyelid opening degree difference determination threshold") or more based on the time-series drowsiness-related information. The first noise factor detection unit 12 stores the sensing results acquired from the sensing unit 11 in chronological order in a memory unit (not shown). The first noise factor detection unit 12 can identify the immediately preceding drowsiness-related information based on the sensing results stored in the memory unit. The memory unit is provided in a location accessible by the drowsiness estimation device 1. The eyelid opening degree determination threshold and the eyelid opening degree difference determination threshold are appropriately set by an administrator or the like and stored in the storage unit. In this case, the drowsiness-related information includes at least the eyelid opening degree of the driver.

[0020] Further, for example, the first noise factor detection unit 12 may detect the occurrence of a noise-causing behavior by the driver using a model (hereinafter referred to as a "machine learning model") that has previously learned a "suffering face." The machine learning model that the first noise factor detection unit 12 uses to detect the noise-causing behavior by the driver is also referred to as a "first machine learning model." The first machine learning model is a machine learning model such as an SVM (Support Vector Machine), a Random Forest, a LightGBM (Light Gradient Boosting Machine), or a convolutional neural network. The first machine learning model is a model that receives information indicating the positions of feature points of a person's face in an image as input and outputs information indicating whether the person is making a stressed face. The first machine learning model is generated in advance and stored in a storage unit. The first noise factor detection unit 12 inputs drowsiness-related information into the first machine learning model to obtain information indicating whether the driver is making a stressed face, thereby detecting that the driver is making a stressed face. In this case, the drowsiness-related information includes at least the position information of the characteristic points of the driver's face on the captured image.

[0021] Furthermore, for example, if the noise-causing behavior is downward gaze, the first noise factor detection unit 12 detects that the driver has performed the noise-causing behavior if the driver's gaze direction is equal to or less than a predetermined threshold (hereinafter referred to as the "gaze direction determination threshold") based on the drowsiness-related information. For example, the first noise factor detection unit 12 may detect that the driver has performed the noise-causing behavior if the driver's gaze direction changes downward by more than a predetermined angle (hereinafter referred to as the "downward gaze determination angle") within a predetermined period (hereinafter referred to as the "downward gaze determination period") based on the time-series drowsiness-related information. The gaze direction determination threshold, the downward gaze determination period, and the downward gaze determination angle are appropriately set by an administrator or the like depending on the installation position and angle of view of the image capture device 2, and are stored in the storage unit. In this case, the drowsiness-related information includes at least the driver's gaze direction.

[0022] When the first noise factor detection unit 12 detects that the driver has engaged in a noise causing behavior, it outputs information about the detected noise causing behavior (hereinafter referred to as "noise causing behavior information") to the feature calculation unit 13 together with the sensing result acquired from the sensing unit 11. The noise causing behavior information includes information that the first noise factor detection unit 12 has detected the noise causing behavior and drowsiness-related information that was the basis for the first noise factor detection unit 12 to detect the noise causing behavior. The noise causing behavior information may further include information that can identify the type of noise causing behavior detected by the first noise factor detection unit 12 (such as looking downward or squinting).

[0023] The feature calculation unit 13 calculates features for estimating the driver's drowsiness (hereinafter referred to as "features for drowsiness estimation") based on drowsiness-related information obtained by the sensing unit 11 after excluding drowsiness-related information that was the basis for detecting noise-causing behaviors by the first noise factor detection unit 12 (hereinafter referred to as "drowsiness-related information after noise factor removal"). In the first embodiment, the process of calculating the features for drowsiness estimation performed by the feature calculation unit 13 is referred to as a "feature calculation process."

[0024] The calculation of the drowsiness estimation feature amount by the feature amount calculation unit 13 will be described in detail.

[0025] First, the sensing result selection unit 131 provided in the feature calculation unit 13 assigns an exclusion flag to drowsiness-related information acquired by the sensing unit 11 that is to be excluded when calculating features for estimating drowsiness, based on noise-causing behavior information regarding noise-causing behavior detected by the first noise factor detection unit 12.

[0026] For example, suppose the first noise factor detection unit 12 detects a noise-causing behavior, which is the behavior of squinting the eyes. In this case, the sensing result selection unit 131 selects, from among the drowsiness-related information included in the sensing results, drowsiness-related information that caused the first noise factor detection unit 12 to detect the noise-causing behavior as drowsiness-related information to be excluded. Then, the sensing result selection unit 131 assigns an exclusion target flag to the drowsiness-related information to be excluded in the sensing results. As described above, the first noise factor detection unit 12 outputs the sensing results acquired from the sensing unit 11, together with the noise-causing behavior information, to the feature calculation unit 13. Thus, the sensing result selection unit 131 can identify the drowsiness-related information to be excluded from the noise-causing behavior information and the sensing results output from the first noise factor detection unit 12. With regard to the sensing results output from the first noise factor detection unit 12, if drowsiness-related information included in the sensing results is drowsiness-related information to be excluded, the sensing result selection unit 131 assigns an exclusion target flag to the drowsiness-related information. Then, the sensing result selection unit 131 assigns an exclusion target flag to the drowsiness-related information to be excluded, and outputs the sensing results to the feature amount calculation unit 13.

[0027] Next, based on the sensing results output from the sensing result selection unit 131, the feature calculation unit 13 excludes, from the drowsiness-related information acquired by the sensing unit 11, drowsiness-related information to which the sensing result selection unit 131 has assigned an exclusion target flag, thereby identifying drowsiness-related information remaining after excluding the drowsiness-related information that was the basis for the first noise factor detection unit 12 detecting the noise-causing behavior from the drowsiness-related information acquired by the sensing unit 11, i.e., drowsiness-related information after noise factors have been removed. Then, the feature calculation unit 13 calculates drowsiness estimation feature amounts based on the identified drowsiness-related information after noise factors have been removed. In other words, the feature calculation unit 13 calculates drowsiness estimation feature amounts based on drowsiness-related information not assigned an exclusion target flag from the drowsiness-related information acquired from the sensing unit 11 via the first noise factor detection unit 12. The drowsiness estimation feature amounts include, for example, the degree of eyelid opening, the degree of opening, the driver's facial orientation, the driver's head position, the driver's gaze direction, the driver's PERCLOS (Percent of the time eyes are closed, i.e., the proportion of time the eyes are closed per unit time), the number of blinks of the driver, or the driver's blink rate. The feature amount calculation unit 13 calculates the above-described drowsiness estimation feature amounts based on the drowsiness-related information after noise factor removal.

[0028] Note that, when the drowsiness estimation feature quantities are feature quantities that need to be determined from the history of the driver's state in the past, such as PERCLOS, the number of blinks, or the blink rate, the feature quantity calculation unit 13, for example, stores the sensing results output from the sensing result selection unit 131, and calculates the drowsiness estimation feature quantities based on the stored sensing results and on a predetermined number of pieces of past drowsiness-related information. The feature quantity calculation unit 13 may also calculate the drowsiness estimation feature quantities based on stored drowsiness-related information after noise factors have been removed and acquired during a predetermined period in the past (hereinafter referred to as a "feature quantity calculation period").

[0029] For example, if the feature calculation period is three minutes, the feature calculation unit 13 calculates the drowsiness estimation feature based on the post-noise-factor-removal drowsiness-related information acquired within the past three minutes. In the post-noise-factor-removal drowsiness-related information, drowsiness-related information acquired based on captured images acquired from the imaging device 2 excludes drowsiness-related information to be excluded. Specifically, drowsiness-related information acquired by the sensing unit 11 based on the imaging device 2, drowsiness-related information that was the basis for the first noise factor detection unit 12 to detect the noise factor behavior, is excluded. That is, the feature calculation unit 13 does not use drowsiness-related information acquired based on frames of captured images in which a driver engaging in noise factor behavior is captured, for calculating the drowsiness estimation feature. More specifically, the feature calculation unit 13 does not use drowsiness-related information acquired based on frames of captured images in which a driver engaging in noise factor behavior is captured, for calculating the drowsiness estimation feature.

[0030] The feature calculation unit 13 outputs information (hereinafter referred to as "feature information") that associates drowsiness-related information, more specifically, drowsiness-related information after noise factor removal, with the calculated feature for estimating drowsiness, to the drowsiness score calculation unit 14.

[0031] As described above, the sensing result selection unit 131 assigns an exclusion flag to the drowsiness-related information acquired by the sensing unit 11, and the feature calculation unit 13 identifies post-noise-factor-removal drowsiness-related information based on the exclusion flag and calculates the drowsiness estimation feature from the identified post-noise-factor-removal drowsiness-related information. However, this is merely an example, and the feature calculation unit 13 may calculate the drowsiness estimation feature using other methods. For example, when the sensing result selection unit 131 identifies drowsiness-related information to be excluded based on noise-causing behavior information related to the noise-causing behavior detected by the first noise factor detection unit 12, the sensing result selection unit 131 excludes the identified drowsiness-related information to be excluded from the drowsiness-related information acquired by the sensing unit 11 that is included in the sensing result, and selects the excluded drowsiness-related information as post-noise-factor-removal drowsiness-related information. When the sensing result selection unit 131 selects the post-noise-factor-removal drowsiness-related information, it outputs the sensing result including the selected post-noise-factor-removal drowsiness-related information (hereinafter referred to as the "post-noise-removal sensing result") to the feature calculation unit 13. The feature calculation unit 13 calculates drowsiness estimation feature amounts based on the post-noise-factor-removal drowsiness-related information output from the sensing result selection unit 131. That is, the feature calculation unit 13 calculates drowsiness estimation feature amounts based on the drowsiness-related information selected by the sensing result selection unit 131 and included in the post-noise-removal sensing result, out of the drowsiness-related information acquired from the sensing unit 11 via the first noise factor detection unit 12. The feature calculation unit 13 may calculate the drowsiness estimation feature amounts in this manner.

[0032] The drowsiness score calculation unit 14 calculates a drowsiness score using the drowsiness estimation feature calculated by the feature calculation unit 13. The drowsiness score calculation unit 14 can identify the drowsiness estimation feature calculated by the feature calculation unit 13 from the feature information output from the feature calculation unit 13. The drowsiness score calculated by the drowsiness score calculation unit 14 is a score indicating the degree of drowsiness of the driver, which is used to estimate the driver's drowsiness. In the first embodiment, as an example, the drowsiness score is expressed from "0" to "100", and a larger drowsiness score indicates a higher degree of drowsiness of the driver. Note that the drowsiness estimation unit 15 estimates the driver's drowsiness using the drowsiness score. In the first embodiment, the process of calculating the drowsiness score performed by the drowsiness score calculation unit 14 is referred to as a "drowsiness score calculation process."

[0033] The drowsiness score calculation unit 14 calculates the drowsiness score using, for example, a machine learning model that has previously learned the drowsiness score. The machine learning model that the drowsiness score calculation unit 14 uses to calculate the drowsiness score is also referred to as a "second machine learning model." The second machine learning model is a machine learning model such as an SVM (Support Vector Machine), a random forest, a LightGBM (Light Gradient Boosting Machine), or a convolutional neural network. The second machine learning model receives, for example, drowsiness estimation feature values ​​as input and outputs a drowsiness score. The second machine learning model is generated in advance and stored in a location that can be referenced by the drowsiness score calculation unit 14, such as a storage unit. The drowsiness score calculation unit 14 calculates the drowsiness score by inputting the drowsiness estimation feature values ​​into the second machine learning model to obtain a drowsiness score.

[0034] The drowsiness score calculation unit 14 may calculate the drowsiness score based on, for example, a preset rule for calculating the drowsiness score (hereinafter referred to as the "sleepiness score calculation rule"). The drowsiness score calculation rule is generated in advance by an administrator or the like and stored in a location that can be referenced by the drowsiness score calculation unit 14, such as a memory unit. The drowsiness score calculation rule sets a calculation rule for the drowsiness score according to the number of blinks in a set period, such as "if the number of blinks in the past three minutes is 20 or more, the drowsiness score is set to 60."

[0035] The drowsiness score calculation unit 14 outputs the calculated drowsiness score of the driver to the drowsiness estimation unit 15 .

[0036] The drowsiness estimation unit 15 estimates the drowsiness of the driver based on the drowsiness score calculated by the drowsiness score calculation unit 14. The drowsiness of the driver estimated by the drowsiness estimation unit 15 may be expressed as a plurality of states such as "awake," "weak drowsiness," and "strong drowsiness," or may be expressed as a binary value of "drowsy" or "not drowsy," or may be a continuous value indicating the degree of drowsiness. In the first embodiment, the process of estimating the drowsiness of the driver performed by the drowsiness estimation unit 15 is referred to as a "drowsiness estimation process."

[0037] The drowsiness estimation unit 15 estimates the drowsiness of the driver using, for example, a machine learning model that has learned drowsiness in advance. The machine learning model used by the drowsiness estimation unit 15 to estimate the drowsiness of the driver is also referred to as a "third machine learning model." The third machine learning model is a machine learning model such as an SVM (Support Vector Machine), a random forest, a LightGBM (Light Gradient Boosting Machine), or a convolutional neural network. The third machine learning model receives, for example, a drowsiness score as input and outputs information indicating drowsiness. The information indicating drowsiness is, for example, information indicating multiple states of the driver's drowsiness, information indicating "drowsy" or "not drowsy," or a continuous value indicating the degree of drowsiness. The third machine learning model is generated in advance and stored in a location that can be referenced by the drowsiness estimation unit 15, such as a storage unit. The drowsiness estimation unit 15 inputs the drowsiness score into the third machine learning model to obtain information indicating drowsiness, thereby estimating the drowsiness of the driver.

[0038] The drowsiness estimation unit 15 may estimate the drowsiness of the driver based on, for example, a preset rule for estimating the drowsiness of the driver (hereinafter referred to as a "drowsiness estimation rule"). The drowsiness estimation rule is generated in advance by an administrator or the like and stored in a location that the drowsiness estimation unit 15 can refer to, such as a storage unit. The drowsiness estimation rule includes, for example, a condition that associates a range of the drowsiness score with information indicating a plurality of states of the driver's drowsiness, such as "if the drowsiness score is "0" to "50", it is determined to be "awake", if the drowsiness score is "50" to "60", it is determined to be "weak drowsiness", and if the drowsiness score is "60" or higher, it is determined to be "strong drowsiness"", a condition that "if the drowsiness score is "60" or higher, it is determined to be "drowsy", and if the drowsiness score is less than "60", it is determined to be "not drowsy", or a calculation formula for the degree of drowsiness based on the drowsiness score".

[0039] The drowsiness estimation unit 15 outputs the driver drowsiness estimation result (hereinafter referred to as the "drowsiness estimation result") to a device (not shown) external to the drowsiness estimation device 1. For example, the drowsiness estimation unit 15 outputs the drowsiness estimation result to an alarm device mounted on the vehicle. The alarm device outputs an alarm when the drowsiness estimation result indicates that the driver is feeling drowsy. For example, the drowsiness estimation unit 15 may store the drowsiness estimation result in a memory unit.

[0040] The operation of the drowsiness estimation device 1 according to embodiment 1 will now be described. Fig. 2 is a flowchart for explaining the operation of the drowsiness estimation device 1 according to embodiment 1. For example, when the power of the vehicle is turned on and a captured image is output from the imaging device 2, the drowsiness estimation device 1 repeats the operation shown in the flowchart of Fig. 2 until the power of the vehicle is turned off.

[0041] The sensing unit 11 acquires an image of the driver's face from the imaging device 2, and performs a sensing process to acquire drowsiness-related information based on the acquired image (step ST10). The sensing unit 11 outputs the sensing result to the first noise factor detection unit 12.

[0042] The first noise factor detection unit 12 performs a first noise factor detection process (step ST20) to detect noise factor behavior by the driver that is similar to behavior caused by drowsiness, based on the drowsiness-related information acquired by the sensing unit 11 in step ST10. When the first noise factor detection unit 12 detects that the driver has performed a noise factor behavior, the first noise factor detection unit 12 outputs noise factor behavior information to the feature calculation unit 13 together with the sensing result acquired from the sensing unit 11. When the first noise factor detection unit 12 does not detect that the driver has performed a noise factor behavior, the first noise factor detection unit 12 may output information to the feature calculation unit 13 indicating that the driver has not performed a noise factor behavior, or may not output anything to the feature calculation unit 13.

[0043] The feature calculation unit 13 performs a feature calculation process to calculate a feature for drowsiness estimation based on post-noise-factor-removal drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit 12 in step ST20 from the drowsiness-related information acquired by the sensing unit 11 in step ST10 (step ST30). More specifically, the feature calculation unit 13 performs a feature calculation process to calculate a feature for drowsiness estimation based on the post-noise-factor-removal drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit 12 from the drowsiness-related information acquired by the sensing unit 11. The feature calculation unit 13 outputs the feature information to the drowsiness score calculation unit 14.

[0044] The drowsiness score calculation unit 14 performs a drowsiness score calculation process (step ST40) to calculate a drowsiness score using the drowsiness estimation feature calculated by the feature calculation unit 13 in step ST30. The drowsiness score calculation unit 14 outputs the calculated drowsiness score of the driver to the drowsiness estimation unit 15.

[0045] The drowsiness estimation unit 15 performs a drowsiness estimation process to estimate the drowsiness of the driver based on the drowsiness score calculated by the drowsiness score calculation unit 14 in step ST40 (step ST50). The drowsiness estimation unit 15 outputs the driver drowsiness estimation result.

[0046] Fig. 3 is a flowchart illustrating an example of the details of the processing of step ST30 in Fig. 2. More specifically, Fig. 3 is a flowchart illustrating the details of the feature calculation process in the case where, in the first embodiment, the feature calculation unit 13 identifies post-noise-factor-removal drowsiness-related information based on the exclusion target flags assigned by the sensing result selection unit 131 to the drowsiness-related information to be excluded, and calculates drowsiness estimation features based on the identified post-noise-factor-removal drowsiness-related information.

[0047] The sensing result selection unit 131 assigns an exclusion flag to drowsiness-related information to be excluded from calculation of drowsiness estimation features among the drowsiness-related information acquired by the sensing unit 11, based on noise-causing behavior information related to the noise-causing behavior detected by the first noise factor detection unit 12 (step ST301). The sensing result selection unit 131 outputs the sensing results after assigning the exclusion flag to the drowsiness-related information to be excluded, to the feature calculation unit 13.

[0048] Based on the sensing results output from the sensing result selection unit 131 in step ST301, the feature calculation unit 13 identifies drowsiness-related information after noise factors have been removed by excluding drowsiness-related information to be excluded, to which the sensing result selection unit 131 has assigned an exclusion target flag, from the drowsiness-related information acquired by the sensing unit 11 (step ST302).

[0049] The feature amount calculation unit 13 then calculates a feature amount for estimating drowsiness based on the drowsiness-related information after noise factor removal identified in step ST302 (step ST303). The feature amount calculation unit 13 outputs the feature amount information to the drowsiness score calculation unit 14.

[0050] Fig. 4 is a flowchart for explaining another example of the details of the process of step ST30 in Fig. 2. More specifically, Fig. 4 is a flowchart for explaining the details of the feature calculation process in the case where the feature calculation unit 13 calculates drowsiness estimation feature amounts based on the drowsiness-related information after noise factors are removed and output from the sensing result selection unit 131 in the first embodiment.

[0051] When the sensing result selection unit 131 identifies drowsiness-related information to be excluded based on the noise-causing behavior information related to the noise-causing behavior detected by the first noise factor detection unit 12, the sensing result selection unit 131 excludes the identified drowsiness-related information to be excluded from the drowsiness-related information included in the sensing result and acquired by the sensing unit 11, and selects the drowsiness-related information after the exclusion as drowsiness-related information after noise factor removal (step ST311). When the sensing result selection unit 131 selects the drowsiness-related information after noise factor removal, it outputs the sensing result after noise removal to the feature calculation unit 13.

[0052] The feature amount calculation unit 13 calculates a feature amount for estimating drowsiness based on the drowsiness-related information after noise factor removal output from the sensing result selection unit 131 in step ST311 (step ST312). The feature amount calculation unit 13 outputs the feature amount information to the drowsiness score calculation unit 14.

[0053] In this way, the drowsiness estimation device 1 acquires drowsiness-related information for each frame of an image capturing the driver's face, and detects noise-causing behaviors by the driver that are similar to behaviors caused by drowsiness based on the acquired drowsiness-related information. The drowsiness estimation device 1 calculates drowsiness estimation feature quantities based on post-noise-removal drowsiness-related information obtained by excluding the drowsiness-related information that was the basis for detecting the noise-causing behaviors from the acquired drowsiness-related information. The drowsiness estimation device 1 then calculates a drowsiness score using the drowsiness estimation feature quantities, and estimates the driver's drowsiness based on the calculated drowsiness score.

[0054] Human behaviors include behaviors that exhibit characteristics similar to those of drowsiness, such as looking down, frowning, or smiling, in other words, noise-causing behaviors. In conventional technology, when estimating a driver's drowsiness, if the driver exhibits the above-mentioned noise-causing behaviors, the accuracy of estimating the driver's drowsiness may be reduced. More specifically, in conventional technology, when estimating a driver's drowsiness, feature amounts calculated based on facial information of the driver exhibiting the noise-causing behavior, i.e., a captured image of the driver's face, become noise, which may reduce the accuracy of estimating the driver's drowsiness.

[0055] In contrast, as described above, the drowsiness estimation device 1 calculates drowsiness estimation features based on the drowsiness-related information after noise factors have been removed, and estimates the driver's drowsiness based on the drowsiness score calculated using the drowsiness estimation features. Therefore, when estimating the driver's drowsiness, the drowsiness estimation device 1 can prevent a decrease in the accuracy of the driver's drowsiness estimation due to the driver's engaging in a noise-causing behavior. More specifically, the drowsiness estimation device 1 can prevent a decrease in the accuracy of the driver's drowsiness estimation due to the use of noise-causing features in estimating the driver's drowsiness, thereby enabling highly accurate drowsiness estimation. Note that, when estimating the driver's drowsiness, the noise-causing features are calculated based on captured images of the face of a driver engaging in a noise-causing behavior, while features calculated based on captured images of the face of a driver not engaging in a noise-causing behavior are necessary for accurate driver drowsiness estimation. Even if the driver is performing a noise-causing behavior such as looking down, the drowsiness estimation device 1 removes only the feature amounts that become noise due to the noise-causing behavior and uses the feature amounts after the noise feature amounts have been removed as feature amounts for drowsiness estimation to estimate the driver's drowsiness. In other words, even if the driver is performing a noise-causing behavior, the drowsiness estimation device 1 does not stop estimating the driver's drowsiness itself, but performs drowsiness estimation using necessary feature amounts. In this way, when estimating the driver's drowsiness, the drowsiness estimation device 1 can prevent a decrease in the accuracy of estimating the driver's drowsiness due to the feature amounts that become noise being used in estimating the driver's drowsiness, and can perform highly accurate drowsiness estimation.

[0056] 5A and 5B are diagrams illustrating an example of the hardware configuration of the drowsiness estimation device 1 according to the first embodiment. In the first embodiment, the functions of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 are realized by a processing circuit 1001. That is, the drowsiness estimation device 1 includes the processing circuit 1001 for performing control to estimate the drowsiness of a driver based on an image acquired from the imaging device 2, using feature amounts for drowsiness estimation that exclude feature amounts calculated due to the driver engaging in noise-causing behavior. The processing circuit 1001 may be dedicated hardware as shown in FIG. 5A or a processor 1004 that executes a program stored in a memory as shown in FIG. 5B.

[0057] If the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0058] When the processing circuit is the processor 1004, the functions of the sensing unit 11, the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005, thereby executing the functions of the sensing unit 11, the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15. In other words, the drowsiness estimation device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST10 to ST50 in FIG. 2 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15. Here, the memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0059] Note that the functions of the sensing unit 11, the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of the sensing unit 11 may be implemented by a processing circuit 1001 as dedicated hardware, and the functions of the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 may be implemented by a processor 1004 reading and executing programs stored in a memory 1005. The drowsiness estimation device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2.

[0060] In the first embodiment described above, the drowsiness estimation device 1 is an in-vehicle device mounted on a vehicle, and the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 are included in the drowsiness estimation device 1. However, this is not limited to this. A drowsiness estimation system may be configured by the in-vehicle device and the server, with some of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 being mounted on the in-vehicle device of the vehicle and the others being included in a server connected to the in-vehicle device via a network. Alternatively, the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, and the drowsiness estimation unit 15 may all be included in the server.

[0061] In the first embodiment described above, the subject is a vehicle driver as an example, but this is merely an example. The subject may be a passenger other than the vehicle driver. The subject may also be a passenger, including the driver, of a moving body other than a vehicle, such as a bus, a train, or an airplane. The drowsiness estimation device 1 according to the first embodiment can be applied as a drowsiness estimation device that estimates the drowsiness of a passenger of a moving body other than a vehicle.

[0062] As described above, according to the first embodiment, the drowsiness estimation device 1 is configured to include a sensing unit 11 that acquires drowsiness-related information indicating a state related to the drowsiness of the occupant for each frame based on frames of an image capturing the face of an occupant of a moving body; a first noise factor detection unit 12 that detects noise-causing behavior by the occupant, which is behavior accompanied by eye movement similar to behavior caused by drowsiness, based on the drowsiness-related information acquired by the sensing unit 11; a feature calculation unit 13 that calculates drowsiness estimation feature amounts for estimating the drowsiness of the occupant based on noise-factor-removed drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit 12 from the drowsiness-related information acquired by the sensing unit 11; a drowsiness score calculation unit 14 that calculates a drowsiness score using the drowsiness estimation feature amounts calculated by the feature calculation unit 13; and a drowsiness estimation unit 15 that estimates the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit 14. Therefore, when estimating the drowsiness of an occupant of a moving body, the drowsiness estimation device 1 can prevent the accuracy of estimating the drowsiness of the occupant from decreasing due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0063] Embodiment 2 Among the driving states of a mobile body, there are driving states in which it is assumed that the occupant is unlikely to become drowsy. If it is estimated that the occupant is drowsy in such driving states, this is an erroneous estimation (overestimation). If an alarm is issued based on the erroneous estimation, the alarm will be an over-alert and may be annoying to the occupant. In embodiment 2, an embodiment will be described in which, taking into account the driving state of the mobile body, when it is estimated from the driving state of the mobile body that the occupant is unlikely to become drowsy, the estimation result of the occupant's drowsiness is corrected so that the occupant's drowsiness is not overestimated. Note that in the following embodiment 2 as well, the subject is assumed to be the driver of the vehicle.

[0064] FIG. 6 is a diagram illustrating a configuration example of a drowsiness estimation device 1a according to a second embodiment. The drowsiness estimation device 1a according to the second embodiment is connected to an imaging device 2 and a vehicle information acquisition device 3. The vehicle information acquisition device 3 outputs vehicle-related information (hereinafter referred to as "vehicle information") to the drowsiness estimation device 1a. The vehicle information includes vehicle speed information, whether or not a turn signal is being used on the vehicle (in other words, information indicating whether or not the turn signal is activated), information indicating the amount of brake depression, steering wheel steering angle information, and information indicating the accelerator opening. The vehicle information acquisition device 3 includes, for example, a vehicle speed sensor that detects the vehicle speed, a turn signal sensor that detects the activation status of the turn signal, a brake sensor that detects the amount of brake depression, a steering angle sensor that detects the steering wheel steering angle, and an accelerator sensor that detects the accelerator opening. Note that FIG. 6 illustrates a single vehicle information acquisition device 3 being connected to the drowsiness estimation device 1a, but this is merely an example, and multiple vehicle information acquisition devices 3 may be connected to the drowsiness estimation device 1a.

[0065] Regarding the configuration of the drowsiness estimation device 1a according to the second embodiment, the same components as those of the drowsiness estimation device 1 described in the first embodiment using Fig. 1 are denoted by the same reference numerals, and redundant description will be omitted. The drowsiness estimation device 1a according to the second embodiment differs from the drowsiness estimation device 1 according to the first embodiment in that it includes a second noise factor detection unit 16 and a score correction unit 17.

[0066] The second noise factor detection unit 16 acquires vehicle information from the vehicle information acquisition device 3, and detects a vehicle driving state that is unlikely to cause drowsiness in the driver (hereinafter referred to as a "noise factor driving state") based on the acquired vehicle information. In the second embodiment, the process of detecting a noise factor driving state performed by the second noise factor detection unit 16 is referred to as a "second noise factor detection process."

[0067] Examples of noise-causing driving conditions include a state in which the vehicle is traveling at a low speed, a state in which the turn signal is used frequently while the vehicle is traveling, a state in which the brakes of the vehicle are used frequently, or a state in which the steering angle of the vehicle changes greatly. Note that the driving conditions of the vehicle that are considered to be noise-causing driving conditions are determined in advance by an administrator or the like.

[0068] When the vehicle is traveling at a low speed and the blinker is frequently used, it is estimated that the vehicle is approaching an intersection or about to change lanes. In such cases, it is estimated that the driver is less likely to become drowsy. When the vehicle's brakes are frequently used, it is estimated that the vehicle is in a traffic jam. When the vehicle's steering angle changes frequently, it is estimated that the vehicle is in a parking lot or passing through an intersection.

[0069] It is assumed that the driver is unlikely to become drowsy in the vehicle driving state described above. The administrator or the like obtains such information in advance by conducting experiments or the like. Then, the administrator or the like determines the vehicle driving state described above, in which the driver is unlikely to become drowsy, as a noise-causing driving state, and stores information about the noise-causing driving state in a location, such as a storage unit, from which the noise-causing driving state can be referenced. The information about the noise-causing driving state is information used by the second noise factor detection unit 16 to determine the noise-causing driving state, and, for example, vehicle information conditions for determining that the vehicle is in the noise-causing driving state are set. The vehicle information conditions for determining that the vehicle is in the noise-causing driving state include, for example, that the vehicle speed is equal to or less than a predetermined threshold, that the frequency of turn signal use in a predetermined period is equal to or greater than a predetermined threshold, that the frequency of brake use in a predetermined period is equal to or greater than a predetermined threshold, and that the amount of change in steering angle in a predetermined period is equal to or greater than a predetermined threshold.

[0070] The second noise factor detection unit 16 stores the vehicle information acquired from the vehicle information acquisition device 3 in a chronological order in a storage unit or the like. Based on the stored vehicle information, the second noise factor detection unit 16 can determine changes in the vehicle's running state, such as changes in the steering angle.

[0071] When the second noise factor detection unit 16 detects that the vehicle's driving state is a noise factor driving state, it outputs information about the detected noise factor driving state (hereinafter referred to as "noise factor driving state information") to the score correction unit 17, together with the vehicle information acquired from the vehicle information acquisition device 3. The noise factor driving state information includes information that the second noise factor detection unit 16 has detected a noise factor driving state. The noise factor driving state information may further include information that can identify the type of noise factor driving state detected by the second noise factor detection unit 16 (low speed driving, frequent turn signal use, frequent brake use, large changes in steering angle, etc.).

[0072] When the second noise factor detection unit 16 detects a noise factor driving state, the score correction unit 17 corrects the drowsiness score calculated by the drowsiness score calculation unit 14. Note that in the second embodiment, the drowsiness score calculation unit 14 outputs the calculated drowsiness score of the driver to the score correction unit 17. The score correction unit 17 corrects the drowsiness score in accordance with a predetermined rule (hereinafter referred to as a "score correction rule"). The score correction rule is set in advance by an administrator or the like, and is stored in a location that can be referenced by the score correction unit 17, such as a memory unit. In the second embodiment, the process of correcting the drowsiness score performed by the score correction unit 17 is referred to as a "drowsiness score correction process."

[0073] A specific example of the correction of the drowsiness score by the score correction unit 17 in the second embodiment will be described. For example, it is assumed that the score correction rule includes a rule that "if a noise factor driving state is detected, the drowsiness score is multiplied by 0.5." It is also assumed that the drowsiness score calculated by the drowsiness score calculation unit 14 is currently "90." It is also assumed that the second noise factor detection unit 16 has detected that the driving state is a noise factor driving state in which the turn signal is frequently used because the frequency of turn signal use in a predetermined period is equal to or greater than a predetermined threshold. Note that a driving state in which the turn signal is frequently used may be, for example, when the vehicle is frequently changing lanes. If such a driving state of the vehicle is detected as a noise factor driving state, it is assumed that the driver is unlikely to become drowsy. Nevertheless, the fact that the drowsiness score calculation unit 14 calculated a high drowsiness score of "90" suggests that the feature calculation unit 13 may have calculated a feature that could result in a high drowsiness score due to some driver behavior, such as the driver checking left and right when frequently changing lanes and this being deemed to be an eye-closing action (i.e., an action that occurs when drowsiness occurs). In other words, the drowsiness score calculated by the drowsiness score calculation unit 14 may be incorrect. In this case, the score correction unit 17 corrects the drowsiness score to "45."

[0074] For example, it is assumed that the drowsiness estimation unit 15 estimates the driver as "drowsy" when the drowsiness score exceeds "50." In this case, if the drowsiness score is "90," the drowsiness estimation unit 15 estimates the driver as "drowsy." Then, for example, an alarm device (not shown) outputs an alarm based on the drowsiness estimation result of the drowsiness estimation unit 15 that the driver is "drowsy." In the above example, if the score correction unit 17 did not correct the drowsiness score, an alarm about drowsiness would be output to the driver even though it was assumed that the driver was not drowsy. By correcting the drowsiness score when the second noise factor detection unit 16 detects a noise-factor driving state, the drowsiness estimation unit 15 can prevent overestimation of the driver's drowsiness and suppress the above-mentioned excessive alarms.

[0075] Note that the above-described specific example is merely an example. The score correction rule may be set to a rule different from the content of the above-described example. Furthermore, for example, the score correction rule may be set so that the degree of correction of the drowsiness score varies depending on the type of noise-causing driving condition. For example, the score correction rule may be set to "multiply the drowsiness score by 0.5 when a noise-causing driving condition in which turn signals are used frequently is detected, and multiply the drowsiness score by 0.7 when a noise-causing driving condition in which turn signals are used frequently is detected."

[0076] The score corrector 17 outputs the corrected drowsiness score (hereinafter referred to as the "corrected drowsiness score") to the drowsiness estimator 15. The drowsiness estimator 15 estimates the drowsiness of the driver based on the corrected drowsiness score.

[0077] The operation of the drowsiness estimation device 1a according to the second embodiment will now be described. FIG. 7 is a flowchart for describing the operation of the drowsiness estimation device 1a according to the second embodiment. For example, when the power of the vehicle is turned on and an image is output from the imaging device 2, the drowsiness estimation device 1a repeats the operation shown in the flowchart of FIG. 7 until the power of the vehicle is turned off. The specific operations in the processing of steps ST10 to ST40 and step ST50 in FIG. 7 are the same as the specific operations in the processing of steps ST10 to ST40 and step ST50 in FIG. 2 that have been described in the first embodiment, and therefore the same step numbers are assigned and redundant description will be omitted.

[0078] The second noise factor detection unit 16 acquires vehicle information from the vehicle information acquisition device 3 and performs a second noise factor detection process to detect a noise factor driving state based on the acquired vehicle information (step ST60). When the second noise factor detection unit 16 detects that the vehicle's driving state is a noise factor driving state, it outputs noise factor driving state information to the score correction unit 17 together with the vehicle information acquired from the vehicle information acquisition device 3. When the second noise factor detection unit 16 does not detect that the vehicle's driving state is a noise factor driving state, it may output information to the score correction unit 17 that it did not detect that the vehicle is in a noise factor driving state, or it may not output anything to the score correction unit 17.

[0079] When the second noise factor detection unit 16 detects a noise factor driving state in step ST60, the score correction unit 17 performs a drowsiness score correction process to correct the drowsiness score calculated by the drowsiness score calculation unit 14 (step ST45). The score correction unit 17 outputs the corrected drowsiness score to the drowsiness estimation unit 15.

[0080] 7, the processes of steps ST10 to ST40 and the process of step ST60 are performed in parallel, but this is merely an example. For example, the drowsiness estimation device 1a may perform the process of step ST60 after the processes of steps ST10 to ST40. It is sufficient that the processes of steps ST10 to ST40 and step ST60 have been performed before the process of step ST45 is performed.

[0081] In this way, the drowsiness estimation device 1a acquires drowsiness-related information for each frame based on frames of captured images of the driver's face, and detects noise-causing behaviors by the driver that are similar to behaviors caused by drowsiness based on the acquired drowsiness-related information. The drowsiness estimation device 1a calculates drowsiness estimation feature quantities based on noise-removed drowsiness-related information obtained by excluding the drowsiness-related information that was the basis for detecting the noise-causing behaviors from the acquired drowsiness-related information. The drowsiness estimation device 1a then calculates a drowsiness score using the drowsiness estimation feature quantities and estimates the driver's drowsiness based on the calculated drowsiness score. Furthermore, if the drowsiness estimation device 1a detects a noise-causing driving state based on the vehicle information, it corrects the calculated drowsiness score. If the drowsiness score has been corrected, the drowsiness estimation device 1a estimates the driver's drowsiness based on the corrected drowsiness score. Therefore, when estimating a driver's drowsiness, the drowsiness estimation device 1a can prevent the accuracy of estimating the driver's drowsiness from decreasing due to the driver engaging in noise-causing behavior, and more specifically, when estimating the driver's drowsiness, can prevent the accuracy of estimating the driver's drowsiness from decreasing due to features that become noise being used to estimate the driver's drowsiness, thereby enabling highly accurate drowsiness estimation and preventing overestimation of the driver's drowsiness.

[0082] The hardware configuration of the drowsiness estimation device 1a according to the second embodiment is the same as the hardware configuration of the drowsiness estimation device 1 described in the first embodiment using Figures 5A and 5B, and therefore is not illustrated. In the second embodiment, the functions of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17 are realized by a processing circuit 1001. That is, the drowsiness estimation device 1a includes the processing circuit 1001 for controlling the estimation of the drowsiness of a driver based on an image acquired from the imaging device 2, using feature amounts excluding feature amounts calculated due to the driver engaging in noise factor behavior as drowsiness estimation feature amounts, and for controlling the correction of the drowsiness score based on vehicle information acquired from the vehicle information acquisition device 3. The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17. That is, the drowsiness estimation device 1a includes the memory 1005 for storing a program that, when executed by the processing circuit 1001, results in the execution of steps ST10 to ST60 in FIG. 7 described above. It can also be said that the program stored in the memory 1005 causes a computer to execute the procedures or methods of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17. The drowsiness estimation device 1a includes devices such as an imaging device 2 and a vehicle information acquisition device 3, an input interface device 1002 that performs wired or wireless communication, and an output interface device 1003.

[0083] In the above-described second embodiment, the drowsiness estimation device 1a is an in-vehicle device mounted on a vehicle, and the drowsiness estimation device 1a is provided with the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17. However, without being limited to this, some of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17 may be mounted on the in-vehicle device of the vehicle and the others may be provided on a server connected to the in-vehicle device via a network, and a drowsiness estimation system may be configured by the in-vehicle device and the server. In addition, the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the second noise factor detection unit 16, and the score correction unit 17 may all be provided on the server.

[0084] In addition, in the above-described second embodiment, the subject is a vehicle driver as an example, but this is merely an example. The subject may be a passenger other than the vehicle driver. Furthermore, the subject may be a passenger, including the driver, of a moving body other than a vehicle, such as a bus, a train, or an airplane. The drowsiness estimation device 1a according to the second embodiment can be applied as a drowsiness estimation device that estimates the drowsiness of a passenger of a moving body other than a vehicle.

[0085] As described above, according to the second embodiment, the drowsiness estimation device 1 a is configured to include the sensing unit 11 that acquires drowsiness-related information indicating a state related to the drowsiness of the occupant for each frame based on frames of an image capturing an image of the face of an occupant of a moving body; the first noise factor detection unit 12 that detects, based on the drowsiness-related information acquired by the sensing unit 11, noise-causing behavior by the occupant, which is behavior accompanied by eye movement that is similar to behavior caused by drowsiness; the feature calculation unit 13 that calculates drowsiness estimation feature amounts for estimating the drowsiness of the occupant based on noise-factor-removed drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit 12 from the drowsiness-related information acquired by the sensing unit 11; the drowsiness score calculation unit 14 that calculates a drowsiness score using the drowsiness estimation feature amounts calculated by the feature calculation unit 13; and the drowsiness estimation unit 15 that estimates the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit 14. Therefore, when estimating the drowsiness of an occupant of a moving body, the drowsiness estimation device 1a can prevent the accuracy of estimating the drowsiness of the occupant from decreasing due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0086] Furthermore, the drowsiness estimation device 1a includes a second noise factor detection unit 16 that detects a noise factor driving state, which is a driving state of the mobile body that is assumed to be less likely to cause drowsiness in the occupant, based on mobile body information related to the mobile body, and a score correction unit 17 that corrects the drowsiness score calculated by the drowsiness score calculation unit 14 when the second noise factor detection unit 16 detects the noise factor driving state. When the score correction unit 17 corrects the drowsiness score calculated by the drowsiness score calculation unit 14, the drowsiness estimation unit 15 is configured to estimate the drowsiness of the occupant based on the corrected drowsiness score after correction by the score correction unit 17. Therefore, the drowsiness estimation device 1a can prevent overestimation of the occupant's drowsiness.

[0087] Embodiment 3. In Embodiment 2, the drowsiness estimation device detects a noise-causing driving condition based on vehicle information and corrects the drowsiness score when the noise-causing driving condition is detected, thereby preventing erroneous (over-estimation) estimation of drowsiness. Meanwhile, a malfunction of the sensing unit may also be considered as an event that can lead to an erroneous estimation of drowsiness. For example, even in a situation where a vehicle driver's eye behavior is normal and the driver is driving with their eyes open, a sensing error by the sensing unit may erroneously detect the driver as having their eyes closed, resulting in erroneous drowsiness-related information being acquired. If the feature calculation unit calculates drowsiness estimation feature amounts based on the erroneous drowsiness-related information, for example, a drowsiness estimation feature amount that may lead to an estimation of "drowsiness present" may be calculated. As a result, the drowsiness estimation unit may over-estimate the driver's drowsiness. In Embodiment 3, an embodiment in which the drowsiness score is corrected taking into account the possibility of such an error in the sensing unit's sensing process will be described. Note that in the following Embodiment 3 as well, the subject is assumed to be the driver of the vehicle.

[0088] Fig. 8 is a diagram showing an example configuration of a drowsiness estimation device 1b according to embodiment 3. In the configuration of the drowsiness estimation device 1b according to embodiment 3, the same components as those of the drowsiness estimation device 1 described in embodiment 1 using Fig. 1 are assigned the same reference numerals, and redundant description will be omitted. The drowsiness estimation device 1b according to embodiment 3 differs from the drowsiness estimation device 1 according to embodiment 1 in that it includes a third noise factor detection unit 18 and a score correction unit 17.

[0089] The third noise factor detection unit 18 detects the occurrence of an event (hereinafter referred to as "noise factor sensing") in which it is presumed that the sensing unit 11 has erroneously detected a state related to occupant drowsiness and thus erroneously acquired drowsiness-related information, based on the drowsiness-related information acquired by the sensing unit 11. Note that in the third embodiment, the sensing unit 11 outputs the sensing result to the first noise factor detection unit 12 and the third noise factor detection unit 18. In the third embodiment, the process of detecting noise factor sensing performed by the third noise factor detection unit 18 is referred to as a "third noise factor detection process."

[0090] An example of noise factor sensing is erroneous acquisition of drowsiness-related information caused by the sensing unit 11 erroneously detecting that the driver's eyes are closed when they are actually open, in other words, that the degree of eyelid opening is low. Such erroneous acquisition of drowsiness-related information may lead to an erroneous estimation of the driver's drowsiness when the drowsiness estimation unit 15 estimates the driver's drowsiness based on the drowsiness-related information. The third noise factor detection unit 18 detects whether the noise factor sensing has occurred based on, for example, whether drowsiness-related information that can be determined to represent an extremely long period of continuous eye closure has been acquired based on the time-series drowsiness-related information acquired by the sensing unit 11. The third noise factor detection unit 18 detects that the noise factor sensing has occurred when drowsiness-related information that can be determined to represent an extremely long period of continuous eye closure has been acquired. For example, if a state determined to be an eye-closure state based on the time-series drowsiness-related information, more specifically, a state in which the eyelid opening degree is equal to or less than a predetermined threshold (hereinafter referred to as the “eyelid opening degree for determining continuous eye closure”), continues for a predetermined period (hereinafter referred to as the “period for determining continuous eye closure”), the third noise factor detection unit 18 detects that drowsiness-related information that may be determined to be an extremely long period of continuous eye closure has been acquired, that is, that noise factor sensing has occurred. Note that the eyelid opening degree for determining continuous eye closure and the period for determining continuous eye closure are determined in advance by an administrator or the like and stored in a location accessible by the third noise factor detection unit 18. Note that the third noise factor detection unit 18 may acquire the time-series drowsiness-related information from the sensing results stored in the storage unit.

[0091] Another example of noise factor sensing is erroneous acquisition of drowsiness-related information caused by the sensing unit 11 repeatedly erroneously detecting that the driver's eyes are closed when they are actually open and erroneously detecting that the driver's eyes are open when they are actually closed. Such erroneous acquisition of drowsiness-related information may lead to an erroneous estimation of the driver's drowsiness when the drowsiness estimation unit 15 estimates the driver's drowsiness based on the drowsiness-related information. The third noise factor detection unit 18 detects whether the noise factor sensing has occurred based on, for example, whether drowsiness-related information indicating an extremely high number of blinks has been acquired based on the time-series drowsiness-related information acquired by the sensing unit 11. The third noise factor detection unit 18 detects that the noise factor sensing has occurred when drowsiness-related information indicating an extremely high number of blinks has been acquired. For example, the third noise factor detection unit 18 detects the driver's blinks based on the time-series drowsiness-related information using a known blink detection method. When the third noise factor detection unit 18 detects blinks equal to or greater than a predetermined number (hereinafter referred to as a "blink count determination threshold") within a predetermined period (hereinafter referred to as a "blink determination period"), it detects that drowsiness-related information that can be determined as an extremely large number of blinks has been acquired, that is, that noise factor sensing has occurred. Note that the blink determination period and the blink count determination threshold are determined in advance by an administrator or the like and stored in a location that can be referenced by the third noise factor detection unit 18.

[0092] Note that the above example is just one example, and the conditions for the third noise factor detection unit 18 to detect that noise factor sensing is occurring based on the drowsiness-related information acquired by the sensing unit 11 are determined in advance as appropriate by an administrator, etc.

[0093] When the third noise factor detection unit 18 detects that noise factor sensing is occurring, it outputs information about the detected noise factor sensing (hereinafter referred to as "noise factor sensing information") together with the sensing result to the score correction unit 17. The noise factor sensing information includes information that the third noise factor detection unit 18 has detected that noise factor sensing is occurring. The noise factor sensing information may further include information that can identify the type of noise factor sensing that the third noise factor detection unit 18 has detected as occurring (extremely continuous eye closure, extremely frequent blinking, etc.).

[0094] When the third noise factor detection unit 18 detects noise factor sensing, the score correction unit 17 corrects the drowsiness score calculated by the drowsiness score calculation unit 14 based on the noise factor sensing. Note that in the third embodiment, the drowsiness score calculation unit 14 outputs the calculated drowsiness score of the driver to the score correction unit 17. The score correction unit 17 corrects the drowsiness score in accordance with predetermined score correction rules. In the third embodiment, the process of correcting the drowsiness score performed by the score correction unit 17 is referred to as a "drowsiness score correction process."

[0095] A specific example of the correction of the drowsiness score by the score correction unit 17 in the third embodiment will be described. For example, it is assumed that the score correction rule includes a rule that "when the occurrence of noise factor sensing is detected, the drowsiness score is multiplied by 0.5." It is also assumed that the drowsiness score calculated by the drowsiness score calculation unit 14 is currently "90." It is also assumed that the third noise factor detection unit 18 has detected, based on the drowsiness-related information acquired from the sensing unit 11, that a state in which the eyelid opening degree is equal to or less than the eyelid opening degree for determining continuous eye closure has continued for a period for determining continuous eye closure (e.g., 10 seconds), and therefore that noise factor sensing has occurred, which is the acquisition of drowsiness-related information that may be considered to be extremely long periods of continuous eye closure. Although continuous eye closure for 10 seconds is abnormal, the fact that the drowsiness score calculation unit 14 calculated a high drowsiness score of "90" despite this abnormal state may mean that the abnormal state of continuous eye closure is considered to be a state in which drowsiness occurs, and that the feature calculation unit 13 has calculated a feature that may result in a high drowsiness score. In other words, there is a possibility that the drowsiness score calculated by the drowsiness score calculation unit 14 is incorrect. In this case, the score correction unit 17 corrects the drowsiness score to "45".

[0096] For example, the drowsiness estimation unit 15 estimates the driver as "drowsy" when the drowsiness score exceeds "50." In this case, if the drowsiness score is "90," the drowsiness estimation unit 15 estimates the driver as "drowsy." Then, for example, an alarm device (not shown) outputs an alarm based on the drowsiness estimation result of the drowsiness estimation unit 15 that the driver is "drowsy." In the above example, if the score correction unit 17 did not correct the drowsiness score, an alarm about drowsiness would be output to the driver even though it was assumed that the driver was not drowsy. By the score correction unit 17 correcting the drowsiness score when the third noise factor detection unit 18 detects that a noise factor has been detected, the drowsiness estimation unit 15 can prevent overestimation of the driver's drowsiness and suppress the above-mentioned excessive alarms.

[0097] Note that the above-described specific example is merely an example. The score correction rule may be set to a rule different from the content of the above-described example. Furthermore, for example, the score correction rule may be set so that the degree of correction of the drowsiness score varies depending on the type of noise factor sensing. For example, the score correction rule may be set to: "When occurrence of noise factor sensing, which is the erroneous acquisition of drowsiness-related information that may be an extremely long continuous eye closure, is detected, the drowsiness score is multiplied by 0.5; and when noise factor sensing, which is the erroneous acquisition of drowsiness-related information that may be an extremely high number of blinks, the drowsiness score is multiplied by 0.7."

[0098] The score corrector 17 outputs the corrected drowsiness score to the drowsiness estimator 15. The drowsiness estimator 15 estimates the drowsiness of the driver based on the corrected drowsiness score.

[0099] The operation of the drowsiness estimation device 1b according to the third embodiment will now be described. FIG. 9 is a flowchart for describing the operation of the drowsiness estimation device 1b according to the third embodiment. For example, when the power of the vehicle is turned on and an image is output from the imaging device 2, the drowsiness estimation device 1b repeats the operation shown in the flowchart of FIG. 9 until the power of the vehicle is turned off. The specific operations in the processing of steps ST10 to ST40 and step ST50 in FIG. 9 are the same as the specific operations in the processing of steps ST10 to ST40 and step ST50 in FIG. 2 that have been described in the first embodiment, and therefore the same step numbers are assigned and redundant description will be omitted.

[0100] The third noise factor detection unit 18 acquires drowsiness-related information from the sensing unit 11 and performs a third noise factor detection process to detect the occurrence of noise factor sensing based on the acquired drowsiness-related information (step ST70). When the third noise factor detection unit 18 detects the occurrence of noise factor sensing, it outputs the noise factor sensing information together with the drowsiness-related information from the sensing unit 11 to the score correction unit 17. When the third noise factor detection unit 18 does not detect the occurrence of noise factor sensing, it may output information to the score correction unit 17 indicating that the occurrence of noise factor sensing was not detected, or it may not output anything to the score correction unit 17.

[0101] When the third noise factor detection unit 18 detects the noise factor sensing in step ST70, the score correction unit 17 performs a drowsiness score correction process to correct the drowsiness score calculated by the drowsiness score calculation unit 14 based on the noise factor sensing (step ST45). The score correction unit 17 outputs the corrected drowsiness score to the drowsiness estimation unit 15.

[0102] 9, the processes of steps ST10 to ST40 and the process of step ST70 are performed in parallel, but this is merely an example. For example, the drowsiness estimation device 1b may perform the process of step ST70 after the processes of steps ST10 to ST40. It is sufficient that the processes of steps ST10 to ST40 and step ST70 have been performed before the process of step ST45 is performed.

[0103] In this way, the drowsiness estimation device 1b acquires drowsiness-related information for each frame based on frames of captured images of the driver's face, and detects noise-causing behaviors by the driver that are similar to behaviors caused by drowsiness based on the acquired drowsiness-related information. The drowsiness estimation device 1b calculates drowsiness estimation feature amounts based on noise-removed drowsiness-related information obtained by excluding the drowsiness-related information that was the basis for detecting the noise-causing behavior from the acquired drowsiness-related information. The drowsiness estimation device 1b then calculates a drowsiness score using the drowsiness estimation feature amounts and estimates the driver's drowsiness based on the calculated drowsiness score. Furthermore, when the drowsiness estimation device 1b detects the occurrence of noise-causing sensing based on the drowsiness-related information, it corrects the drowsiness score. When the drowsiness score has been corrected, the drowsiness estimation device 1b estimates the driver's drowsiness based on the corrected drowsiness score. Therefore, when estimating a driver's drowsiness, the drowsiness estimation device 1b can prevent the accuracy of estimating the driver's drowsiness from decreasing due to the driver engaging in noise-causing behavior, and more specifically, when estimating the driver's drowsiness, can prevent the accuracy of estimating the driver's drowsiness from decreasing due to features that become noise being used to estimate the driver's drowsiness, thereby enabling highly accurate drowsiness estimation and preventing overestimation of the driver's drowsiness.

[0104] The hardware configuration of a drowsiness estimation device 1b according to embodiment 3 is similar to the hardware configuration of the drowsiness estimation device 1 described in embodiment 1 using Figures 5A and 5B , and is therefore not shown in the drawings. In embodiment 3, the functions of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18 are realized by a processing circuit 1001. That is, the drowsiness estimation device 1b includes the processing circuit 1001 for performing control to estimate driver drowsiness based on captured images acquired from the imaging device 2, using feature amounts excluding feature amounts calculated due to the driver engaging in noise-causing behavior as drowsiness estimation feature amounts, and for performing control to correct the drowsiness score based on drowsiness-related information acquired from the captured images. The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18. That is, the drowsiness estimation device 1b includes the memory 1005 for storing a program that, when executed by the processing circuit 1001, results in the execution of steps ST10 to ST50 and step ST70 in FIG. 7 described above. It can also be said that the program stored in the memory 1005 causes a computer to execute the procedures or methods of the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18. The drowsiness estimation device 1b includes a device such as an imaging device 2, an input interface device 1002 that performs wired or wireless communication, and an output interface device 1003.

[0105] In the above-described third embodiment, the drowsiness estimation device 1b is an in-vehicle device mounted on a vehicle, and the drowsiness estimation device 1b is provided with the sensing unit 11, the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18. However, without being limited to this, some of the sensing unit 11, the first noise factor detection unit 12, the feature amount calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, and a drowsiness estimation system may be configured by the in-vehicle device and the server. In addition, the sensing unit 11, the first noise factor detection unit 12, the feature calculation unit 13, the drowsiness score calculation unit 14, the drowsiness estimation unit 15, the score correction unit 17, and the third noise factor detection unit 18 may all be provided on the server.

[0106] In addition, in the above-described third embodiment, the subject is a vehicle driver as an example, but this is merely an example. The subject may be a passenger other than the vehicle driver. Furthermore, the subject may be a passenger including the driver of a moving body other than a vehicle, such as a bus, a train, or an airplane. The drowsiness estimation device 1b according to the second embodiment can be applied as a drowsiness estimation device that estimates the drowsiness of a passenger of a moving body other than a vehicle.

[0107] As described above, according to the third embodiment, the drowsiness estimation device 1b is configured to include the sensing unit 11 that acquires drowsiness-related information indicating a state related to the drowsiness of the occupant for each frame based on frames of an image capturing an image of the face of an occupant of a moving body; the first noise factor detection unit 12 that detects, based on the drowsiness-related information acquired by the sensing unit 11, noise-causing behavior by the occupant that is behavior accompanied by eye movement similar to behavior caused by drowsiness; the feature calculation unit 13 that calculates drowsiness estimation feature amounts for estimating the drowsiness of the occupant based on noise-factor-removed drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise-causing behavior by the first noise factor detection unit 12 from the drowsiness-related information acquired by the sensing unit 11; the drowsiness score calculation unit 14 that calculates a drowsiness score using the drowsiness estimation feature amounts calculated by the feature calculation unit 13; and the drowsiness estimation unit 15 that estimates the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit 14. Therefore, when estimating the drowsiness of an occupant of a moving body, the drowsiness estimation device 1b can prevent the accuracy of estimating the drowsiness of the occupant from decreasing due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0108] Furthermore, the drowsiness estimation device 1b includes a third noise factor detection unit 18 that detects the occurrence of noise factor sensing, which is an event that is estimated to have resulted in the erroneous acquisition of drowsiness-related information by the sensing unit 11 due to the erroneous detection of a state related to the drowsiness of the occupant by the sensing unit 11, based on the drowsiness-related information acquired by the sensing unit 11, and a score correction unit 17 that corrects the drowsiness score calculated by the drowsiness score calculation unit 14 when the third noise factor detection unit 18 detects the occurrence of noise factor sensing. When the score correction unit 17 corrects the drowsiness score calculated by the drowsiness score calculation unit 14, the drowsiness estimation unit 15 is configured to estimate the drowsiness of the occupant based on the corrected drowsiness score corrected by the score correction unit 17. Therefore, the drowsiness estimation device 1b can prevent overestimation of the occupant's drowsiness.

[0109] The configuration of the drowsiness estimation device may be a combination of the configurations of the drowsiness estimation device 1 according to the first embodiment, the drowsiness estimation device 1a according to the second embodiment, and the drowsiness estimation device 1b according to the third embodiment. Fig. 10 is a diagram showing an example configuration of a drowsiness estimation device 1c that combines the configurations of the drowsiness estimation device 1 according to the first embodiment, the drowsiness estimation device 1a according to the second embodiment, and the drowsiness estimation device 1b according to the third embodiment. Fig. 11 is a flowchart for explaining the operation of the drowsiness estimation device 1c that combines the configurations of the drowsiness estimation device 1 according to the first embodiment, the drowsiness estimation device 1a according to the second embodiment, and the drowsiness estimation device 1b according to the third embodiment.

[0110] As described above, the drowsiness estimation device 1c includes a sensing unit 11 that acquires drowsiness-related information indicating a state related to drowsiness of an occupant (e.g., a driver) of a moving body (e.g., a vehicle) for each frame of an image captured of the face of the occupant, and a first noise factor detection unit 12 that detects a noise-causing behavior of the occupant based on the drowsiness-related information acquired by the sensing unit 11. The present invention includes a feature amount calculation unit 13 that calculates a feature amount for estimating drowsiness to estimate the drowsiness of an occupant based on drowsiness-related information obtained by removing drowsiness-related information that was the basis for detecting noise-causing behavior by the first noise factor detection unit 12 from the drowsiness-related information acquired by the sensing unit 11, a drowsiness score calculation unit 14 that calculates a drowsiness score using the feature amount for drowsiness estimation calculated by the feature amount calculation unit 13, a drowsiness estimation unit 15 that estimates the drowsiness of an occupant based on the drowsiness score calculated by the drowsiness score calculation unit 14, a second noise factor detection unit 16 that detects a noise factor driving state that is assumed to be less likely to cause drowsiness in an occupant (e.g., a driver) based on mobile body information (vehicle information) related to the mobile body (e.g., a vehicle), and a sensor. The vehicle is equipped with a third noise factor detection unit 18 that detects the occurrence of noise factor sensing, which is an event in which the sensing unit 11 is presumed to have erroneously acquired drowsiness-related information due to an erroneous detection of a state related to the occupant's drowsiness, based on the drowsiness-related information acquired by the sensing unit 11, and a score correction unit 17 that corrects the drowsiness score calculated by the drowsiness score calculation unit 14 based on the noise factor driving state or the occurrence of noise factor sensing when the second noise factor detection unit 16 detects a noise factor driving state or when the third noise factor detection unit 18 detects the occurrence of noise factor sensing, and the drowsiness estimation unit 15 can be configured to estimate the drowsiness of the occupant based on the corrected drowsiness score corrected by the score correction unit 17. As a result, when estimating the drowsiness of an occupant, the drowsiness estimation device 1c can prevent the accuracy of the estimation of the occupant's drowsiness from decreasing due to the occupant engaging in noise-causing behavior, and more specifically, when estimating the occupant's drowsiness, can prevent the accuracy of the estimation of the occupant's drowsiness from decreasing due to the use of features that become noise in estimating the occupant's drowsiness, thereby enabling highly accurate drowsiness estimation and preventing overestimation of the occupant's drowsiness.

[0111] In the drowsiness estimation device 1c, the score correction rule used by the score correction unit 17 when correcting the drowsiness score is set to a rule that takes into account both the noise factor driving state and the occurrence of noise factor sensing. For example, the score correction rule may be set to: "When a noise factor driving state is detected, the drowsiness score is multiplied by 0.5; when the occurrence of noise factor sensing is detected, the drowsiness score is multiplied by 0.5; when both a noise factor driving state and the occurrence of noise factor sensing are detected, the drowsiness score is corrected by a value obtained by multiplying the multiplier corresponding to the noise factor driving state (i.e., 0.5) by the multiplier corresponding to the occurrence of noise factor sensing (i.e., 0.5)." In addition, for example, the score correction rule may be set to: "When a noise factor driving state is detected, the drowsiness score is multiplied by 0.5; when the occurrence of noise factor sensing is detected, the drowsiness score is multiplied by 0.5." A rule may be set that states, "When both a noise factor driving condition and the occurrence of noise factor sensing are detected, the drowsiness score is corrected according to the weights set for the noise factor driving condition and the noise factor sensing." The weights to be assigned to the respective noise factor driving conditions and noise factor sensing are determined in advance by an administrator or the like. This allows the drowsiness estimation device 1c to further enhance the effect of correcting the drowsiness score.

[0112] Furthermore, the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0113] When estimating the drowsiness of an occupant of a moving vehicle, the drowsiness estimation device of the present disclosure can prevent a decrease in the accuracy of estimating the drowsiness of the occupant due to the occupant engaging in behavior that exhibits characteristics similar to those observed when drowsiness occurs.

[0114] 1, 1a, 1b, 1c Drowsiness estimation device, 2 Imaging device, 3 Vehicle information acquisition device, 11 Sensing unit, 12 First noise factor detection unit, 13 Feature calculation unit, 131 Sensing result selection unit, 14 Drowsiness score calculation unit, 15 Drowsiness estimation unit, 16 Second noise factor detection unit, 17 Score correction unit, 18 Third noise factor detection unit, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. A drowsiness estimation device comprising: a sensing unit that acquires drowsiness-related information indicating a state related to drowsiness of an occupant of a moving body for each frame based on frames of an image capturing an image of the face of the occupant of the moving body; a first noise factor detection unit that detects a noise factor behavior by the occupant that is a behavior involving eye movement similar to a behavior caused by drowsiness based on the drowsiness-related information acquired by the sensing unit; a feature calculation unit that calculates drowsiness estimation feature amounts for estimating the drowsiness of the occupant based on drowsiness-related information after noise factor removal is removed from the drowsiness-related information acquired by the sensing unit, the drowsiness-related information that was the basis for the detection of the noise factor behavior by the first noise factor detection unit; a drowsiness score calculation unit that calculates a drowsiness score using the drowsiness estimation feature amounts calculated by the feature calculation unit; and a drowsiness estimation unit that estimates the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit.

2. The feature calculation unit is equipped with a sensing result selection unit that, based on noise factor behavior information regarding the noise factor behavior detected by the first noise factor detection unit, assigns an exclusion flag to the drowsiness-related information among the drowsiness-related information acquired by the sensing unit, which is the source of the detection of the noise factor behavior by the first noise factor detection unit, as the drowsiness-related information to be excluded when calculating the feature for drowsiness estimation, and identifies the drowsiness-related information after noise factors have been removed by excluding the drowsiness-related information to which the sensing result selection unit has assigned the exclusion flag from the drowsiness-related information acquired by the sensing unit, and calculates the feature for drowsiness estimation based on the identified drowsiness-related information after noise factors have been removed.

3. The drowsiness estimation device described in claim 1, characterized in that the feature calculation unit is provided with a sensing result selection unit that selects, based on noise factor behavior information regarding the noise factor behavior detected by the first noise factor detection unit, the drowsiness-related information obtained by the sensing unit after excluding the drowsiness-related information that was the basis for the first noise factor detection unit to detect the noise factor behavior, as the drowsiness-related information after noise factor removal, and calculates the feature for drowsiness estimation based on the drowsiness-related information after noise factor removal selected by the sensing result selection unit.

4. The drowsiness estimation device as described in claim 1, further comprising: a second noise factor detection unit that detects a noise factor driving state, which is a driving state of the moving body that is assumed to be less likely to cause drowsiness in the occupant, based on moving body information related to the moving body; and a score correction unit that corrects the drowsiness score calculated by the drowsiness score calculation unit when the second noise factor detection unit detects the noise factor driving state, and when the score correction unit corrects the drowsiness score calculated by the drowsiness score calculation unit, the drowsiness estimation unit estimates the drowsiness of the occupant based on the corrected drowsiness score after correction by the score correction unit.

5. The drowsiness estimation device according to claim 4, characterized in that the noise-causing driving conditions include a state in which the moving body is traveling at a low speed, a state in which the turn signal is used frequently while the moving body is traveling, a state in which the brakes of the moving body are used frequently, or a state in which there is a large change in the steering angle of the moving body.

6. The drowsiness estimation device as described in claim 1, further comprising: a third noise factor detection unit that detects the occurrence of noise factor sensing, which is an event in which the sensing unit is presumed to have erroneously acquired the drowsiness-related information due to an erroneous detection of a state related to the drowsiness of the occupant based on the drowsiness-related information acquired by the sensing unit; and a score correction unit that corrects the drowsiness score calculated by the drowsiness score calculation unit when the third noise factor detection unit detects the occurrence of the noise factor sensing, and when the score correction unit corrects the drowsiness score calculated by the drowsiness score calculation unit, the drowsiness estimation unit estimates the drowsiness of the occupant based on the corrected drowsiness score after correction by the score correction unit.

7. The drowsiness estimation device of claim 6, characterized in that the third noise factor detection unit detects that the noise factor sensing has occurred when the sensing unit acquires drowsiness-related information that can be determined to indicate a long continuous state of eye closure, or when the sensing unit acquires drowsiness-related information that can be determined to indicate a large number of blinks.

8. The drowsiness estimation device as described in claim 1, further comprising: a second noise factor detection unit that detects a noise factor driving state, which is a driving state of the moving body that is assumed to be unlikely to cause drowsiness in the occupant, based on moving body information related to the moving body; a third noise factor detection unit that detects the occurrence of a noise factor sensing, which is an event in which it is assumed that the sensing unit has erroneously detected the drowsiness-related information of the occupant due to an erroneous detection of the drowsiness-related information of the occupant by the sensing unit, based on the drowsiness-related information acquired by the sensing unit; and a score correction unit that corrects the drowsiness score calculated by the drowsiness score calculation unit based on the noise factor driving state or the occurrence of the noise factor sensing when the second noise factor detection unit detects the noise factor driving state or when the third noise factor detection unit detects the occurrence of the noise factor sensing, and the drowsiness estimation unit estimates the drowsiness of the occupant based on the corrected drowsiness score after correction by the score correction unit.

9. The drowsiness estimation device as described in claim 8, characterized in that when the second noise factor detection unit detects the noise factor driving condition and the third noise factor detection unit detects the occurrence of the noise factor sensing, the score correction unit corrects the drowsiness score in accordance with weights set for the noise factor driving condition and the noise factor sensing.

10. The drowsiness estimation device according to any one of claims 1 to 9, characterized in that the occupant of the moving body is a driver of the vehicle.

11. A drowsiness estimation method comprising: a step of a sensing unit acquiring drowsiness-related information indicating a state related to drowsiness of an occupant of a moving body for each frame based on frames of an image capturing an image of the face of the occupant of the moving body; a step of a first noise factor detection unit detecting a noise factor behavior by the occupant which is a behavior involving eye movement similar to a behavior caused by drowsiness based on the drowsiness-related information acquired by the sensing unit; a step of a feature calculation unit calculating a drowsiness estimation feature for estimating the drowsiness of the occupant based on post-noise-factor-removal drowsiness-related information obtained after excluding the drowsiness-related information that was the basis for the detection of the noise factor behavior by the first noise factor detection unit from the drowsiness-related information acquired by the sensing unit; a step of a drowsiness score calculation unit calculating a drowsiness score using the drowsiness estimation feature calculated by the feature calculation unit; and a step of a drowsiness estimation unit estimating the drowsiness of the occupant based on the drowsiness score calculated by the drowsiness score calculation unit.