Wakefulness degree detection device, driving assistance system, and wakefulness degree detection method

Abstract

A wakefulness degree detection device (2) comprises: a wakefulness degree detection unit (21) that detects the wakefulness degree of a driver; a stimulus generation unit (22) that generates a stimulus for the driver in accordance with the wakefulness degree detected by the wakefulness degree detection unit (21); a reaction detection unit (23) that detects the reaction of the driver with respect to the stimulus generated by the stimulus generation unit (22); and a correction unit (24) that corrects the wakefulness degree of the driver on the basis of the reaction of the driver detected by the reaction detection unit (23).

Description

Arousal level detection device, driving assistance system, and arousal level detection method

[0001] The present disclosure relates to a wakefulness detection device, a driving assistance system, and a wakefulness detection method.

[0002] There are known technologies for determining the state of a vehicle driver and preventing traffic accidents caused by a decrease in the driver's level of alertness. For example, Patent Document 1 describes a driving assistance system that sequentially calculates the degree of eye opening of the driver, outputs an alarm to the driver when the degree of eye opening of the driver is equal to or less than a threshold value for a predetermined period of time, and promptly stops the vehicle being driven by the driver when it is not determined that the degree of eye opening has exceeded the threshold value in response to the alarm sound.

[0003] JP 2014-002561 A

[0004] The conventional technology described in Patent Document 1 has a problem in that it is not possible to objectively evaluate the degree of wakefulness of a driver when a stimulus is given to the driver (hereinafter referred to as the wakefulness level). Although the degree of eye opening of a driver affects driving ability, for example, even if the eyes are half-open, the driver may be able to maintain attention, and it cannot be said that the state is unsuitable for driving. For this reason, it is necessary to objectively evaluate the wakefulness of the driver and determine the content of driving assistance.

[0005] The present disclosure is intended to solve the above-mentioned problem, and aims to provide an alertness detection device that can objectively evaluate the alertness of an occupant of a moving vehicle when a stimulus is given to the occupant based on their reaction to the stimulus.

[0006] The alertness detection device of the present disclosure is an alertness detection device that detects the alertness of an occupant of a moving body, and includes an alertness detection unit that detects the alertness of the occupant, a stimulus generation unit that generates a stimulus for the occupant in accordance with the alertness detected by the alertness detection unit, a response detection unit that detects the occupant's response to the stimulus generated by the stimulus generation unit, and a correction unit that corrects the occupant's alertness based on the occupant's response detected by the response detection unit.

[0007] According to the present disclosure, the alertness detection device detects the alertness of an occupant, generates a stimulus for the occupant according to the detected alertness, detects the occupant's response to the stimulus, and corrects the alertness of the occupant based on the detected response of the occupant. By referring to the alertness corrected based on the response of the occupant of the mobile body, the alertness detection device according to the present disclosure can objectively evaluate the alertness of the occupant of the mobile body in response to the stimulus applied to the occupant.

[0008] Fig. 1 is a block diagram showing an example of the configuration of a driving assistance system according to embodiment 1. Fig. 2 is a flowchart showing an alertness detection method according to embodiment 1. Fig. 3 is a diagram showing the relationship between alertness and alertness level over time. Fig. 4 is a block diagram showing an example of the hardware configuration for realizing the functions of an alertness detection device according to embodiment 1. Fig. 5 is a block diagram showing an example of the configuration of a driving assistance system according to embodiment 2. Fig. 6 is a flowchart showing an alertness detection method according to embodiment 2.

[0009] Embodiment 1. FIG. 1 is a block diagram showing an example of the configuration of a driving assistance system 1 according to embodiment 1. In FIG. 1, the driving assistance system 1 is a system connected to an alertness detection device 2, a stimulation device 3, and an occupant state detection device 4 via wired or wireless signal lines. The driving assistance system 1 provides driving assistance to an occupant of a moving body, for example, a driver of an automobile. However, FIG. 1 omits the illustration of components that provide driving assistance to the driver. The moving body is not limited to an automobile, but may also be a railroad vehicle, an aircraft, a ship, or the like. The automobile may be an autonomous vehicle equipped with an autonomous driving function.

[0010] (Overview of Driving Assistance System) The driving assistance system 1 calculates the driver's level of alertness based on detected information indicating the driver's state, and if the calculated level of alertness is equal to or lower than a determination level, provides a stimulus to the driver and corrects the driver's level of alertness based on the driver's response to the stimulus. The driving assistance system 1 outputs the corrected level of alertness (hereinafter referred to as the corrected level of alertness). For example, the driving assistance system 1 displays the driver's corrected level of alertness on a display device (not shown in FIG. 1 ) installed in the vehicle cabin. By referring to this corrected level of alertness, the driver's level of alertness in response to the stimulus provided to the driver can be objectively evaluated. Furthermore, driving assistance by the driving assistance system 1 can also be applied to passengers other than the driver. Below, a case where the target for detecting the level of alertness is the driver of a vehicle will be described.

[0011] (Outline of the awakening detection device) The awakening detection device 2 detects the awakening level of the driver. The awakening detection device 2 also calculates the awakening level of the driver based on information indicating the state of the driver detected by the occupant state detection device 4, and if the calculated awakening level is below a threshold, controls the stimulation device 3 to provide a stimulus to the driver and corrects the awakening level of the driver based on the driver's response to the provided stimulus.

[0012] (Level of Alertness) Level of alertness is a numerical representation of a state in which a vehicle occupant is awake, alert, and able to respond appropriately. For example, level of alertness may be a numerical representation of the driver's eye condition, head movement, biometric information, or a combination thereof. Levels of alertness determined by eye condition include the degree of eye opening or blinking pattern. When the degree of eye opening becomes narrower or when blinking patterns with long eye closure periods become more frequent, the driver's level of alertness is reduced. Level of alertness may also be a numerical representation of head movement. When alertness is reduced due to drowsiness or fatigue, head movement tends to slow down. Level of alertness may also be a numerical representation of biometric information such as heart rate or electrodermal activity (EDA). When the heart rate is stable or EDA is within a normal range, the driver's level of alertness is considered high. For example, level of alertness may be expressed numerically as a value between 0 and 1, with 1.0 being the maximum value and 0 being the minimum value.

[0013] (Stimulation Device) The stimulation device 3 provides stimuli to the driver of the automobile under the control of the alertness detection device 2. The stimuli include auditory stimuli, visual stimuli, and tactile stimuli. Examples of auditory stimuli include alarm sounds and voice messages. Examples of alarm sounds include beeps at specific frequencies, melody sounds, or sounds set as pulse sounds or combinations of different patterns. Examples of voice messages include messages that alert the driver. The stimulation device 3 that provides auditory stimuli is, for example, a voice synthesis system installed inside the automobile or on the dashboard, such as a voice output device provided in a navigation system.

[0014] Examples of visual stimuli include warning lamps or warning displays, additional visual information displayed to the driver, etc. Examples of visual information include a message display such as "Look ahead," a marker display such as a caution color, a warning display using virtual reality (VR) or augmented reality (AR) technology, etc. Examples of the stimulating device 3 that provides visual stimuli include an interior lamp (LED), a head-up display (HUD), a navigation system display, etc., or a combination of these.

[0015] Examples of tactile stimuli include seat vibration, steering wheel vibration, and pedal vibration. Examples of the stimulating device 3 that provides tactile stimuli include a vibration device provided in the seat, a vibration device provided in the steering wheel, or a vibration device provided in the accelerator pedal or brake pedal. Seat vibration is vibration generated in the seat by a vibration device provided in the seat. Steering wheel vibration is vibration generated in the steering wheel by a vibration device provided in the steering wheel. Pedal vibration is vibration generated in the pedal by a vibration device provided in the accelerator pedal or brake pedal.

[0016] (Occupant State Detection Device) The occupant state detection device 4 detects the state of the driver. The driver's state may include, for example, the driver's blinking, eye opening, gaze direction, or body tilt. The occupant state detection device 4 detects the state of the occupant by analyzing sensor information detected by, for example, an in-vehicle camera, a steering wheel sensor, a biometric sensor, a vibration sensor, or a combination of these. For example, the occupant state detection device 4 analyzes images or videos captured by the in-vehicle camera to detect the driver's facial movement, eye opening / closing, gaze direction, body tilt, or the driver's alertness itself as information indicating the driver's state.

[0017] The occupant condition detection device 4 also analyzes sensor information detected by a sensor incorporated in the steering wheel to detect the driver's hand movements, grip strength, steering wheel operation pattern, etc., or the driver's level of alertness itself as information indicating the driver's state. Furthermore, the occupant condition detection device 4 analyzes biometric information detected by a biometric sensor, such as the driver's heart rate, electrodermal activity (EDA), and brain waves, to detect the driver's physical state as information indicating the driver's state. Furthermore, the occupant condition detection device 4 analyzes the driver's body movements or subtle vibrations detected by a vibration sensor incorporated in the seat or steering wheel to detect the driver's sleep state or level of alertness itself as information indicating the driver's state.

[0018] (Details of the arousal detection device) The arousal detection device 2 will be described in detail below. The arousal detection device 2 includes an arousal detection unit 21, a stimulus generation unit 22, a response detection unit 23, and a correction unit 24. For example, the arousal detection device 2 is realized by a computer. A memory included in the computer stores a program constituting an information processing application for realizing each function of the arousal detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24. A processor included in the computer reads the information processing application from the memory and executes the information processing application, thereby realizing each function of the arousal detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24.

[0019] (Arousal Level Detection Unit) The awakening level detection unit 21 detects the awakening level of the occupant. For example, the awakening level detection unit 21 detects the awakening level of the driver based on the state of the driver detected by the occupant state detection device 4. For example, the awakening level detection unit 21 detects the awakening level of the driver based on the awakening level detection indices (1) to (7) below. The awakening level detection unit 21 may also detect the awakening level comprehensively by arbitrarily combining these indices.

[0020] (1) Index f1(t) Based on the Driver's Eye Opening Degree The alertness detection unit 21 calculates an alertness index that decreases as the driver's current eye opening degree decreases relative to the eye opening degree when the driver is awake. For example, the eye opening degree may be the average eye opening degree over one minute. If the driver's eyes remain completely closed, the alertness detection unit 21 determines that the driver has fallen asleep. For example, the alertness detection unit 21 acquires captured images of the driver's face or eyes from the occupant state detection device 4 and detects the open / closed state of the driver's eyes based on the results of image processing performed on the acquired captured images. The alertness detection unit 21 may, for example, numerically represent the alertness level so that a completely closed eye state is 0 and a completely open eye state is 1, and calculate the alertness level based on the detected open / closed state of the driver's eyes. Alternatively, the alertness detection unit 21 may use a machine learning algorithm to identify the eye state and estimate the eye opening degree. The alertness detection unit 21 acquires a captured image of the driver's face or eyes from the occupant state detection device 4, and detects the position of the face, the position of the eyes, or the pattern of eye opening and closing based on the result of image processing performed on the acquired captured image. For example, when data indicating the result of image processing performed on the captured image is input, the alertness detection unit 21 may detect the alertness of the driver using a machine learning model that outputs the level of alertness according to the state of eye opening.

[0021] (2) Index f2(t) Based on Proportion of Blinks with Long Eye Closure Durations The wakefulness detection unit 21 may calculate the proportion of blinks with long eye-closure durations as the wakefulness level. For example, if the number of blinks with long eye-closure durations (0.5 seconds or longer) among the number of blinks Ntotal over a five-minute period is Nlong, the wakefulness detection unit 21 may calculate the value of Nlong / Ntotal and calculate the wakefulness level value such that the wakefulness level decreases as Nlong / Ntotal increases. Furthermore, the wakefulness detection unit 21 may detect the wakefulness level of the driver using a machine learning model that outputs the wakefulness level based on Nlong / Ntotal when the total number of blinks Ntotal and the number of blinks with long eye-closure durations Nlong, which are obtained by performing image processing on captured images of the driver's face, etc., are input.

[0022] (3) Index f3(t) Based on Gaze Pattern The alertness detection unit 21 may calculate the alertness as a numerical representation of the driver's gaze or gaze variability while driving. This alertness becomes lower as the driver's gaze or gaze variability disappears. For example, the alertness detection unit 21 tracks the position of the driver's eyes based on video information of the driver acquired from the occupant state detection device 4. Here, a tracking algorithm is used to detect the position of the driver's pupils and estimate the gaze direction. The alertness detection unit 21 can quantify the gaze or gaze variability by determining the direction the driver is looking, and the alertness may be calculated using these. Alternatively, the detection of gaze or gaze variability may be performed using head tracking, which tracks the movement of the driver's head and estimates the gaze direction. Furthermore, when an image of the driver's face or eye position detected by the occupant state detection device 4 is input, the alertness detection unit 21 may calculate the driver's alertness using a machine learning algorithm that outputs the alertness based on eye gaze or eye gaze variability.

[0023] (4) Index f4(t) Based on Driver's Facial Expression The alertness detection unit 21 may detect the alertness level from the driver's sleepy facial expression. Sleepy facial expressions include blinking, frowning, or a blank, unchanging facial expression. For example, the alertness detection unit 21 may calculate the alertness level using a score (e.g., a numerical value between 0 and 1) associated with each type of facial expression. Note that, when an image capturing the driver's face or eye position detected by the occupant state detection device 4 is input, the alertness detection unit 21 may calculate the alertness level of the driver using a machine learning model that calculates the alertness level using the index f4(t) based on the driver's facial expression. For example, the machine learning model may use Haar Cascade or OpenCV or Dlib, which are deep learning-based face detectors.

[0024] (5) Index f5(t) Based on Yawning or Arousal Level-Increasing Behavior The arousal level detection unit 21 may calculate the driver's arousal level from the number of yawns or the number of times the driver stimulates himself or herself to wake up. For example, the arousal level detection unit 21 may store information on the number of ranges related to the number of yawns or the number of times the driver stimulates himself or herself to wake up for each predetermined monitoring period, with each range being associated with a score. The arousal level detection unit 21 may perform image analysis on the video of the driver acquired from the occupant state detection device 4 to detect the number of yawns or the number of times the driver stimulates himself or herself to wake up for each predetermined monitoring period, and calculate the driver's arousal level based on which of the ranges associated with the scores the detected number of times belongs to. Note that the driver's self-stimulating behavior to wake up is, for example, the driver slapping his or her cheek. The number of times may be the number of times the driver slaps his or her cheek.

[0025] (6) Index f6(t) Based on Face Direction or Gaze Direction The alertness detection unit 21 may calculate the driver's alertness based on the frequency of the driver's behavior of nodding down and then looking up again when the driver's face turns downward, when the driver's gaze remains downward, or when the driver's gaze remains downward. For example, the alertness detection unit 21 may store information relating to a plurality of ranges relating to the number of times the driver turns downward, the duration of the face-down state, or the number of times the driver nods down and then looks up for each predetermined monitoring cycle, with each range being associated with a score. The alertness detection unit 21 may perform image analysis on the video of the driver acquired from the occupant state detection device 4 to detect the number of times the driver turns downward, the duration of the face-down state, or the number of times the driver nods down and then looks up for each predetermined monitoring cycle, and calculate the driver's alertness based on which of the ranges associated with the score the detected result belongs to.

[0026] (7) Index f7(t) Based on Posture Disorder The alertness detection unit 21 may calculate the driver's alertness based on the degree to which the body sways back and forth and side to side, or the degree to which the driver has lost normal driving posture by lying face down. For example, the alertness detection unit 21 may be configured with information on the number or duration of the body swaying back and forth and side to side, the number or duration of the driver lying face down and losing normal driving posture, or a combination thereof, with a score associated with each range. The alertness detection unit 21 may perform image analysis of the video of the driver acquired from the occupant state detection device 4 to detect the number or duration of the body swaying back and forth and side to side, the number or duration of the driver lying face down and losing normal driving posture, or a combination thereof, and calculate the driver's alertness based on which of the ranges associated with the score the detected result belongs to.

[0027] The above-described alertness detection indices (1) to (7) are merely examples, and other indices may be used as long as they are information relating to the driver's alertness. Furthermore, the alertness detection indices may include physiological information detected from the driver using a physiological measurement technique. For example, the alertness detection unit 21 may acquire information indicating the driver's pulse or brain waves detected by the occupant state detection device 4, and digitize the acquired information to calculate the alertness.

[0028] (Stimulus Generator) The stimulus generator 22 generates a stimulus to the occupant according to the level of alertness detected by the alertness detector 21. For example, when the level of alertness of the driver detected by the alertness detector 21 is below a threshold, the stimulus generator 22 controls the stimulus device 3 to generate a stimulus to the driver. For example, the stimulus generator 22 may control the stimulus device 3 to cause the driver to receive any of the following stimuli (1) to (3). Furthermore, the stimulus generator 22 may provide the driver with a comprehensive stimulus that is an arbitrary combination of these stimuli.

[0029] (1) Auditory Stimulus The stimulus generator 22 may provide an auditory stimulus to the driver by controlling, for example, an audio output device provided in a navigation system as the stimulus device 3 that provides the auditory stimulus. The auditory stimulus may be, for example, an alarm sound or a voice message, and may have a directionality such that it can be heard from a sound image in the vehicle cabin.

[0030] (2) Visual Stimulus The stimulus generator 22 may provide a visual stimulus to the driver by controlling, for example, an interior lamp (LED), a HUD, a navigation system display, or a combination thereof as the stimulus device 3 that provides a visual stimulus. Examples of visual stimuli include a warning display on the HUD, a warning stimulus using external illumination or interior lighting, and lighting up an LED installed on the dashboard.

[0031] (3) Tactile Stimulation The stimulus generator 22 may provide the driver with a tactile stimulus by controlling, for example, a vibration device provided on the seat, a vibration device provided on the steering wheel, a vibration device provided on the accelerator pedal or the brake pedal, an air controller in the vehicle, or a combination of these as the stimulus device 3 that provides the tactile stimulus. Examples of the tactile stimulus include seat vibration, steering wheel vibration, pedal vibration, or blowing cool air toward the driver.

[0032] (Response Detection Unit) The response detection unit 23 detects the response of the occupant to the stimulus generated by the stimulus generation unit 22. For example, the response detection unit 23 may detect the response of the occupant to the stimulus based on the state of the driver detected by the occupant state detection device 4. Furthermore, the response detection unit 23 may detect the following responses (1) to (4) based on the state of the driver detected by the occupant state detection device 4. Furthermore, the response detection unit 23 may provide the driver with a comprehensive stimulus that is an arbitrary combination of these stimuli.

[0033] (1) Physical Movement The response detection unit 23 may detect the physical behavior of the driver when a stimulus is generated. For example, the response detection unit 23 detects the physical behavior of the driver as the driver's response to the stimulus by performing image analysis on the video of the driver acquired from the occupant state detection device 4. The driver's response to the stimulus often includes not only the state of eye opening but also the driver's physical behavior. By detecting the physical behavior of the driver, the response detection unit 23 can correct the driver's alertness based on the driver's physical behavior. Furthermore, the response detection unit 23 may detect the following behaviors (A), (B), and (C) as the driver's physical behavior when a stimulus is generated. Here, response (A) is a response when the driver's alertness is high, response (B) is a response when the driver's alertness is lower than the state when response (A) was generated, and response (C) is a response when the driver's alertness is significantly lower than the state when response (B) was generated.

[0034] (A) Examples of physical behaviors that indicate a high level of arousal of the driver include, for example, gestures that indicate annoyance with a stimulus, hand gestures, or shaking of the head. These behaviors can be identified by using image analysis or machine learning algorithms of video footage of the driver. (B) Examples of physical behaviors that indicate a low level of arousal of the driver before the stimulus was applied include, for example, a startled reaction to the stimulus, a slightly downward-facing face turning upward, or a startled shoulder movement. These behaviors can also be identified by using image analysis or machine learning algorithms of video footage of the driver. (C) Examples of physical behaviors that indicate a significantly lower level of arousal of the driver than state (B) include, for example, a vigorous head or body shake by the driver in an attempt to dispel a decrease in arousal. In other words, the reaction detection unit 23 may detect, as a reaction, behaviors in which the driver tries to stimulate himself or herself to increase his or her arousal level. Note that these behaviors can also be identified by using image analysis or machine learning algorithms of video footage of the driver. The response detection unit 23 detects the driver's behavior of stimulating himself to increase his level of alertness as a response, and can correct the level of alertness according to the driver's response that is expected to restore his level of alertness.

[0035] (2) Facial Expression Change The response detection unit 23 may detect a change in the driver's facial expression in response to a stimulus. For example, the response detection unit 23 detects a change in the driver's facial expression as the driver's response to a stimulus by performing image analysis on the video of the driver's face acquired from the occupant state detection device 4. The driver's response to a stimulus is often not only the state of eye opening but also a change in the driver's facial expression. By detecting a change in the driver's facial expression by the response detection unit 23, the alertness detection device 2 can correct the alertness level based on the type of the driver's facial expression. Furthermore, the response detection unit 23 may detect the following behaviors (A1), (B1), and (C1) as changes in the driver's facial expression when a stimulus is generated. The response (A1) is a response when the driver's alertness level is high, the response (B1) is a response when the driver's alertness level is lower than the state when the response (A1) was generated, and the response (C1) is a response when the driver's alertness level is significantly lower than the state when the response (B1) was generated.

[0036] (A1) Examples of facial expressions that indicate a high level of arousal of the driver include, for example, facial expressions that indicate discomfort in response to a stimulus or facial expressions that indicate annoyance in response to a stimulus. These facial expressions can be identified by using image analysis or machine learning algorithms of video footage of the driver. (B1) Examples of facial expressions that indicate a low level of arousal of the driver until the stimulus is applied include, for example, a surprised expression upon noticing the stimulus, or an expression in which the driver opens their eyes wide or blinks in surprise after a predetermined time (e.g., one second) has passed. These facial expressions can also be identified by using image analysis or machine learning algorithms of video footage of the driver. (C1) Examples of facial expressions that indicate a significantly lower level of arousal of the driver than the state described in (B1) include, for example, facial expressions that show little change even when exposed to a stimulus. This facial expression can be said to be unresponsive to a stimulus. This facial expression can also be identified by using image analysis or machine learning algorithms of video footage of the driver. In this way, the reaction detection unit 23 detects the driver's facial expression as one of a surprised expression, a frown expression, or an expression that changes little even when stimulated, and the alertness detection device 2 can correct the driver's alertness based on the surprised expression, the frown expression, or an expression that changes little even when stimulated.

[0037] (3) Blink Reflex The response detection unit 23 may detect the driver's blink reflex in response to a stimulus. For example, the response detection unit 23 detects the driver's blink reflex as the driver's response to the stimulus by performing image analysis on the video of the driver's face acquired from the occupant state detection device 4. The driver's response to the stimulus may be the driver's blink reflex. This allows the wakefulness detection device 2 to correct the wakefulness level based on the driver's blink reflex pattern. Furthermore, the response detection unit 23 may detect the following behaviors (A2), (B2), and (C2) as the driver's blink reflex when a stimulus is generated. The response (A2) is a response when the driver's wakefulness level is high, the response (B2) is a response when the driver's wakefulness level is lower than the state when the response (A2) was generated, and the response (C2) is a response when the driver's wakefulness level is significantly lower than the state when the response (B2) was generated.

[0038] (A2) A blinking pattern that is estimated to indicate a high level of arousal of the driver is, for example, a series of actions in which the driver blinks his / her eyes approximately 0.3 seconds after the stimulus is applied. This behavior can be identified by using image analysis of a video of the driver captured or a machine learning algorithm. (B2) A blinking pattern that is estimated to indicate a low level of arousal of the driver until the stimulus is applied is, for example, a series of actions in which the driver blinks his / her eyes approximately 1.0 seconds after the stimulus is applied. This behavior can be identified by using image analysis of a video of the driver captured or a machine learning algorithm. (C2) A blinking pattern that is estimated to indicate a significantly lower level of arousal of the driver than the state in (B2) is, for example, a blinking pattern that shows little change even when the stimulus is applied. This blinking pattern can be said to indicate a lack of reaction to the stimulus. This behavior can also be identified by using image analysis of a video of the driver captured or a machine learning algorithm.

[0039] (4) Gaze Movement The response detection unit 23 may detect a change in the driver's gaze when receiving a stimulus. For example, the response detection unit 23 detects a change in the driver's gaze as the driver's response to the stimulus by performing image analysis of the driver's eye position acquired from the occupant state detection device 4. The driver's response to the stimulus may be a change in the driver's gaze. This allows the alertness detection device 2 to correct the alertness level based on the change in the driver's gaze. Furthermore, when the stimulus generation unit 22 generates a directional stimulus through hearing or vision, the response detection unit 23 may detect the driver's behavior of directing his / her gaze in the direction of the stimulus as the driver's response to the stimulus. This allows the alertness detection device 2 to correct the alertness level based on the driver's response to the direction of the stimulus. The response detection unit 23 may detect the following behaviors (A3), (B3), and (C3) as the driver's behavior of directing his / her gaze in the direction of the stimulus when a stimulus is generated. The reaction (A3) is the reaction when the driver's level of alertness is high, the reaction (B3) is the reaction when the driver's level of alertness is lower than when the reaction (A3) was taken, and the reaction (C3) is the reaction when the driver's level of alertness is significantly lower than when the reaction (B3) was taken.

[0040] (A3) An example of a behavior that indicates a high level of arousal of the driver is, for example, the driver turning their gaze toward the stimulus approximately 0.3 seconds after the stimulus is applied. This behavior can be identified by using image analysis or a machine learning algorithm of the video of the driver captured. (B3) The longer it takes for the driver to turn their gaze toward the stimulus after the stimulus is applied, the lower the level of arousal of the driver is estimated to have been before the stimulus was applied. This behavior can be identified by using image analysis or a machine learning algorithm of the video of the driver captured. (C3) An example of a behavior that indicates a significantly lower level of arousal of the driver than state (B3) is the driver not showing any reflex movement even when stimulated. This behavior can be considered unresponsive to the stimulus. This facial expression can also be identified by using image analysis or a machine learning algorithm of the video of the driver captured. Note that even if the stimulus is non-directional, if the driver is awake, they are likely to immediately move their gaze when the stimulus is applied.

[0041] (Correction Unit) The correction unit 24 corrects the occupant's alertness based on the occupant's response detected by the response detection unit 23. For example, the correction unit 24 corrects the driver's alertness based on the driver's response. If the time it takes the driver to respond to a stimulus is equal to or longer than a threshold, the driver's alertness may be lower than the previously detected alertness. Therefore, the correction unit 24 corrects the driver's alertness in a direction that decreases the driver's alertness based on the length of the response time. If the driver's time to respond to a stimulus is shorter than the reaction time corresponding to the detected alertness by more than a predetermined difference value, the alertness is corrected in a direction that increases the driver's alertness. This allows the alertness detection device 2 to dynamically correct the driver's alertness.

[0042] Furthermore, the correction unit 24 may correct the level of alertness of the driver based on the details of the physical behavior of the driver when a stimulus is generated and detected by the response detection unit 23. For example, the correction unit 24 refers to correction information in which a correction value of alertness is linked to each detail of physical behavior, and compares the correction information with the details of the physical behavior detected by the response detection unit 23 to determine a correction value, and corrects the level of alertness with the determined correction value.

[0043] Furthermore, the correction unit 24 may correct the driver's level of alertness based on the driver's behavior in which the stimulated driver stimulates himself or herself to increase his or her level of alertness (hereinafter referred to as an alertness-increasing behavior). For example, the correction unit 24 determines a correction value by referring to correction information in which a correction value for alertness is linked to each of various alertness-increasing behaviors, and compares the correction value with the alertness-increasing behavior detected by the response detection unit 23, and corrects the alertness with the determined correction value. An example of an alertness-increasing behavior is a behavior in which the driver slaps or pinches his or her cheek.

[0044] The correction unit 24 may correct the level of alertness of the driver based on the type of facial expression corresponding to the change in the driver's facial expression in response to the stimulus, detected by the response detection unit 23. For example, the correction unit 24 determines a correction value by referring to correction information in which a correction value for alertness is linked to each type of facial expression, and comparing the correction value with the type of facial expression corresponding to the change in facial expression detected by the response detection unit 23, and corrects the level of alertness with the determined correction value.

[0045] The correction unit 24 may correct the alertness of the driver based on a blink reflex pattern corresponding to the blink reflex of the driver in response to a stimulus, detected by the response detection unit 23. For example, the correction unit 24 refers to correction information in which a correction value of the alertness is linked to each blink reflex pattern, and compares the correction value with the blink reflex pattern corresponding to the blink reflex detected by the response detection unit 23, thereby determining the correction value and correcting the alertness with the determined correction value.

[0046] The correction unit 24 may correct the level of alertness of the driver based on a change in the driver's line of sight when the driver receives a stimulus. For example, the correction unit 24 determines a correction value by referring to correction information in which a correction value for alertness is linked for each time range from the time the stimulus is given to the time when the line of sight starts to change, and compares the correction information with the time range corresponding to the line of sight change detected by the response detection unit 23, and corrects the level of alertness with the determined correction value.

[0047] The correction unit 24 may correct the level of alertness of the driver based on the state of movement of the driver's line of sight in the direction of the auditory or visual stimulus given directionality by the stimulus generation unit 22. For example, the correction unit 24 refers to correction information in which a correction value of the level of alertness is linked for each time range from the time the stimulus is given to the time the movement of the line of sight in the direction of the stimulus starts, and compares it with the time range corresponding to the movement of the line of sight in the direction of the stimulus detected by the response detection unit 23, thereby determining a correction value and correcting the level of alertness with the determined correction value.

[0048] Next, the awakening level detection method according to the first embodiment will be described. Fig. 2 is a flowchart showing the awakening level detection method according to the first embodiment. The awakening level detection unit 21 detects the driver's awakening level D(t) based on detection information indicating the driver's state detected by the occupant state detection device 4 (step ST1). Here, t is the measurement time of the driver's state by the occupant state detection device 4. Note that the awakening level is highest when D(t) = 1 and lowest when D(t) = 0.

[0049] 3 is a diagram showing the relationship between the degree of alertness D(t) and the alertness level DL over time, illustrating the image of the change in the level of alertness D(t) until the driver falls asleep at the wheel and the alertness level DL corresponding to each level of alertness D(t). Note that the changes in the driver's state before falling asleep vary from person to person, and generally, there is a very large individual difference in the tendency for the level of alertness to decrease in the initial stage. As shown in FIG. 3, the level of alertness D(t) gradually decreases over time t, fluctuating depending on the influences of the surroundings of the vehicle being driven by the driver, the interior of the vehicle (cabin), and the driver's own psychology, physiology, or physical condition. The level of alertness D(t) then progresses from a light drowsiness to a full drowsiness, with frequent occurrences of poor posture or closed eyes, and the occurrence of microsleep.

[0050] When the alertness level DL is DL5, the driver is fully alert. The driver is capable of rapid blinking, has wide eye opening, and maintains a stable posture. When the alertness level DL is DL4, the driver's alertness is slightly reduced. The driver begins to blink with long eye closure periods and slightly narrows the eye opening. When the alertness level DL is DL3, the driver's alertness is moderately reduced. The driver's long eye closure periods become more pronounced, the eye opening becomes even narrower, and, depending on the individual, the driver may yawn or exhibit a change in facial expression. Driving operations also become somewhat sluggish. When the alertness level DL is DL2, the driver's alertness is severely reduced. The driver may repeatedly lose posture and recover, frequently blink with long eye closure periods, frequently close their eyes for long periods (e.g., for 3 seconds), or experience microsleep, which may impair driving. A driver in this state needs to have their alertness restored. When the alertness level DL is DL1, the driver is in a mild state of drowsiness. At this time, the driver is likely to have lost their posture (posture change) and their eyes are completely closed. The driver can respond to warnings, but is in a state where they are unable to drive. When the alertness level DL is DL0, the driver is in a state where they are completely asleep. At this time, the driver's posture continues to be lost (posture change) and their eyes are completely closed. The driver cannot even respond to warnings. Normally, a decrease in alertness to this extent does not occur while driving, but it may occur due to severe fatigue or the effects of alcohol. A driver in this state is in a state where they are unable to drive.

[0051] The alertness detection indices, index f1(t) based on the driver's eye opening degree and index f2(t) based on the blink rate with a long eye closing time, are effective for detecting alertness across the entire alertness level DL shown in FIG. 3 . That is, the alertness calculated based on indices f1(t) and f2(t) can be used across the entire alertness level DL. On the other hand, index f3(t) based on the gaze pattern, index f4(t) based on the driver's facial expression, and index f5(t) based on yawning or alertness-enhancing behavior are effective for detecting further alertness reduction, including alertness level DL3 (moderate alertness reduction). Index f6(t) based on face direction or gaze direction and index f7(t) based on poor posture are effective for detecting further alertness reduction, including alertness level DL2 (severe alertness reduction). However, alertness levels vary from person to person, and typically, some drivers fall asleep with their eyes half-open. In this case, drowsiness may not be detected based on eye opening degree alone. The classification of arousal levels shown in FIG. 3 is an example, and the classification may be broader or more detailed than this.

[0052] The stimulus generator 22 determines whether the wakefulness D(t) detected by the wakefulness detector 21 is less than a threshold Dth1 (step ST2). If the wakefulness D(t) is equal to or greater than the threshold Dth1 (step ST2; NO), no wakefulness correction is required, and the process proceeds to step ST6. On the other hand, if the wakefulness D(t) is less than the threshold Dth1 (step ST2; YES), the process proceeds to step ST3. The threshold Dth1 is, for example, the lower limit of the wakefulness at the wakefulness level DL5, and is set to Dth=0.8 in FIG. 3 .

[0053] The stimulus generator 22 controls the stimulus device 3 to generate a stimulus for the driver (step ST3). The stimulus provided to the driver does not interfere with driving. For example, the stimulus provided to the driver may be any of the above-mentioned auditory, visual, or tactile stimuli, or any combination thereof.

[0054] The response detection unit 23 detects the driver's response to the stimulus generated by the stimulus generation unit 22 (step ST4). For example, the response detection unit 23 may detect any one of the above-mentioned body movement, facial expression change, blink reflex, and eye movement, or any combination thereof.

[0055] The correction unit 24 corrects the driver's alertness D(t) based on the driver's response detected by the response detection unit 23 to calculate a corrected alertness Dc(t) (step ST5). For example, the correction unit 24 calculates the corrected alertness Dc(t) according to the following formula (1): Dc(t) = D(t) + correction value (1). Note that if the response corresponds to the alertness D(t) detected by the alertness detection unit 21, the correction unit 24 does not correct the alertness D(t). For example, if the correction value associated with the driver's response detected by the response detection unit 23 is 0, the correction unit 24 determines that the response corresponds to the alertness D(t) and does not perform correction. On the other hand, if the response does not correspond to the alertness D(t) detected by the alertness detection unit 21, the correction unit 24 corrects the alertness D(t) to calculate the corrected alertness Dc(t). For example, if the correction value associated with the driver's reaction detected by the reaction detection unit 23 is greater than 0, the correction unit 24 determines that the reaction does not correspond to the wakefulness D(t) and makes a correction to increase the wakefulness. If the correction value is less than 0, the correction unit 24 makes a correction to decrease the wakefulness. For example, if the blink reflex time to the stimulus is (A2) above, i.e., less than 0.3 seconds, no correction is made if the wakefulness D(t) is 0.8 or greater. If the wakefulness D(t) is less than 0.8 but greater than or equal to 0.6, the upper limit of the corrected wakefulness Dc(t) is set to 0.8, and the correction value is +0.1. If the wakefulness D(t) is less than 0.6, the correction value is set to +0.2. When the blink reflex time to the stimulus is (B2) above, i.e., between 0.3 seconds and less than 1.0 seconds, if the arousal level D(t) is 0.6 or greater, the correction value is −0.1. If the arousal level D(t) is less than 0.6 and greater than 0.5, no correction is made. If the arousal level D(t) is less than 0.5 and greater than 0.3, the correction value is +0.1. When the blink reflex time to the stimulus is (B3) above, i.e., there is no reaction, if the arousal level D(t) is 0.6 or greater, the correction value is −0.2. If the arousal level D(t) is less than 0.6 and greater than 0.5, the correction value is −0.1. If the arousal level D(t) is less than 0.5, no correction is made. In this way, correction values ​​according to the arousal level can be determined in a correction table. Note that this correction table may also include facial expressions, gaze movement, etc. The correction value is expressed as a function of f (arousal level D, arousal level estimated from the stimulus response).The correction may also be performed above or below the alertness level DLn (n=0, 1, 2, 3, 4, 5).

[0056] The stimulus generator 22 or the corrector 24 checks whether the driving of the vehicle has ended (step ST6). For example, the awakening detection device 2 acquires vehicle information indicating the vehicle state and determines whether the driving of the vehicle has ended based on the acquired vehicle information. If the driving has ended (step ST6; YES), all the processes shown in FIG. 2 are terminated. If the driving has not ended (step ST6; NO), the process returns to step ST1.

[0057] (Variation 1) Up to now, the alertness level has been corrected when the alertness level D(t) is less than the threshold value Dth1. However, the timing of the correction determination is not limited to this, and the timing of the correction may be determined based on various factors. For example, the correction unit 24 may periodically correct the alertness level every 30 minutes after the start of driving, or may correct the alertness level when it is detected that the vehicle's driving operation has become sluggish. Furthermore, the correction unit 24 may correct the alertness level according to the vehicle's driving conditions, such as when it detects meandering.

[0058] (Variant 2) The correction unit 24 determines the reliability of the alertness detected by the alertness detection unit 21, and if the reliability of the determination result is low, for example, if the alertness changes suddenly in about one minute, or if the results of multiple alertness detection indices are different from each other, or if the results obtained from the image analysis are unclear and there is a large amount of invalid time, the correction unit 24 may determine that the reliability of the alertness detection result is low and perform a correction of the alertness detected by the alertness detection unit 21.

[0059] (Variation 3) Note that, once the wakefulness level has been corrected, the correction unit 24 may not correct the wakefulness level as long as there is no change in the wakefulness level D(t), or may correct the wakefulness level again after a predetermined time, for example, 15 minutes, has elapsed since the wakefulness level correction was performed. Furthermore, the correction unit 24 may correct the wakefulness level based on the results of multiple reactions, rather than on the reaction to a single stimulus. This makes it possible to correct the wakefulness level under various conditions.

[0060] (Variation 4) The wakefulness detection unit 21 may detect the wakefulness of the driver using detection logic based on one of a plurality of wakefulness detection indices linked to each of a plurality of types of responses, and may dynamically change the detection logic to one based on the wakefulness detection indices linked to the driver's response detected by the response detection unit 23. For example, the wakefulness detection unit 21 changes the wakefulness detection logic by increasing the weight of the wakefulness detection indices that match the response results to the stimulus. This allows the wakefulness detection device 2 to change the wakefulness detection logic to one that corresponds to individual characteristics, thereby improving the accuracy of wakefulness detection.

[0061] The stimulus generator 22 may also learn stimuli that are likely to elicit a driver's reaction. For example, the stimulus generator 22 may learn a machine learning model that uses information indicating a stimulus given to the driver and detection information of the driver's reaction to the stimulus as learning data and outputs information indicating stimuli that are likely to elicit a driver's reaction. By using this machine learning model, the alertness detection device 2 can select stimuli that are likely to elicit a driver's reaction for each individual driver.

[0062] (Variation 5) The stimulus generator 22 may generate a stronger stimulus as the level of arousal detected by the arousal level detector 21 decreases. A stronger stimulus would be a loud sound pressure or a varied melody in the case of audio. The order of strength of the stimulus is visual stimulus < audio stimulus < tactile stimulus. A stronger stimulus is a stimulus method that uses multiple stimulus means. By employing a stronger stimulus as the level of arousal decreases, the arousal level detector 2 can check the driver's response to the stimulus and correct the level of arousal, even when the level of arousal is low.

[0063] Next, a description will be given of a hardware configuration that realizes the functions of the wakefulness detection device 2. The functions of the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24 included in the wakefulness detection device 2 are realized by processing circuits. That is, the wakefulness detection device 2 includes a processing circuit for executing the processes from step ST1 to step ST6 shown in Fig. 2. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0064] Fig. 4A is a block diagram showing a hardware configuration for realizing the functions of the wakefulness detection device 2. Fig. 4B is a block diagram showing a hardware configuration for executing software for realizing the functions of the wakefulness detection device 2. In Fig. 4A and Fig. 4B, the wakefulness detection device 2 acquires detection information indicating the state of an occupant from the occupant state detection device 4 via an input interface 100. In addition, the wakefulness detection device 2 outputs control information to the stimulation device 3 via an output interface 101.

[0065] 4A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24 of the wakefulness detection device 2 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.

[0066] 4B , the functions of the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24 included in the wakefulness detection device 2 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0067] The processor 103 reads and executes programs stored in the memory 104 to realize the functions of the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24 of the wakefulness detection device 2. For example, the wakefulness detection device 2 includes a memory 104 for storing programs that, when executed by the processor 103, result in the processing of steps ST1 to ST6 shown in FIG. 2 being executed. These programs cause a computer to execute the procedures or methods of the processing performed by the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24. The memory 104 may be a computer-readable storage medium that stores programs for causing a computer to function as the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24.

[0068] The memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically-EPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0069] Some of the functions of the wakefulness detection unit 21, the stimulus generation unit 22, the response detection unit 23, and the correction unit 24 included in the wakefulness detection device 2 may be realized by dedicated hardware, and the other functions may be realized by software or firmware. For example, the functions of the wakefulness detection unit 21 and the stimulus generation unit 22 may be realized by a processing circuit 102 that is dedicated hardware, and the functions of the response detection unit 23 and the correction unit 24 may be realized by the processor 103 reading and executing a program stored in the memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0070] 4B, the wakefulness detection device 2 is configured using a single processor 103, but is not limited to this. For example, the wakefulness detection device 2 may be realized using a plurality of processors that operate in cooperation with each other.

[0071] As described above, the wakefulness detection device 2 according to the first embodiment detects the wakefulness of a vehicle driver, and includes the wakefulness detection unit 21 that detects the wakefulness of the driver, the stimulus generation unit 22 that generates a stimulus for the driver in accordance with the wakefulness detected by the wakefulness detection unit 21, the response detection unit 23 that detects the driver's response to the stimulus generated by the stimulus generation unit 22, and the correction unit 24 that corrects the wakefulness of the driver based on the driver's response detected by the response detection unit 23. By referring to the wakefulness corrected based on the response of the occupant of the mobile body, the wakefulness detection device 2 can objectively evaluate the wakefulness of the occupant of the mobile body in response to the stimulus applied to the occupant. By referring to the corrected wakefulness, it is possible to grasp the wakefulness more accurately. In particular, the corrected wakefulness improves the accuracy of detecting an initial decrease in wakefulness and improves the accuracy of correcting the wakefulness to account for individual differences.

[0072] In the wakefulness detection device 2 according to the first embodiment, the correction unit 24 corrects the wakefulness of the driver based on the driver's reaction. This allows the wakefulness detection device 2 to dynamically correct the wakefulness of the driver. Furthermore, depending on the design concept of the wakefulness detection device 2, it may be possible to take a harsh view and perform correction only in the direction that decreases the wakefulness.

[0073] In the wakefulness detection device 2 according to the first embodiment, the response detection unit 23 detects the physical behavior of the driver when a stimulus is generated. The correction unit 24 corrects the wakefulness of the driver based on the details of the physical behavior of the driver. This allows the wakefulness detection device 2 to correct the wakefulness based on the physical behavior of the driver.

[0074] In the wakefulness detection device 2 according to the first embodiment, the correction unit 24 corrects the wakefulness of the driver based on the behavior of the stimulated driver in an attempt to stimulate himself or herself to increase the wakefulness of the driver. This allows the wakefulness detection device 2 to correct the wakefulness based on the driver's response that is expected to restore the wakefulness of the driver.

[0075] In the wakefulness detection device 2 according to the first embodiment, the response detection unit 23 detects a change in the driver's facial expression in response to a stimulus. The correction unit 24 corrects the wakefulness of the driver based on the type of the driver's facial expression. This enables the wakefulness detection device 2 to correct the wakefulness based on the type of the driver's facial expression.

[0076] In the wakefulness detection device 2 according to the first embodiment, the type of facial expression is one of a surprised facial expression, a frown facial expression, and a facial expression that does not change much even when stimulated. This allows the wakefulness detection device 2 to correct the wakefulness of the driver based on the surprised facial expression, the frown facial expression, or a facial expression that does not change much even when stimulated.

[0077] In the wakefulness detection device 2 according to the first embodiment, the response detection unit 23 detects the blink reflex of the driver in response to a stimulus. The correction unit 24 corrects the wakefulness of the driver based on the blink reflex pattern of the driver. This allows the wakefulness detection device 2 to correct the wakefulness based on the blink reflex pattern of the driver.

[0078] In the wakefulness detection device 2 according to the first embodiment, the correction unit 24 corrects the wakefulness of the driver based on a change in the line of sight of the driver when the driver receives a stimulus. This enables the wakefulness detection device 2 to correct the wakefulness based on a change in the line of sight of the driver.

[0079] In the wakefulness detection device 2 according to the first embodiment, the stimulus generator 22 generates a directional stimulus by hearing or seeing. The corrector 24 corrects the wakefulness of the driver based on the state of gaze movement relative to the direction of the stimulus. This allows the wakefulness detection device 2 to correct the wakefulness based on the driver's response to the direction of the stimulus.

[0080] In the wakefulness detection device 2 according to the first embodiment, the wakefulness detection unit 21 detects the wakefulness of the driver using detection logic based on one of a plurality of wakefulness detection indices linked to each of a plurality of types of reactions. The detection logic is dynamically changed to one based on the wakefulness detection indices linked to the driver's reaction detected by the reaction detection unit 23. This allows the wakefulness detection device 2 to change to a wakefulness detection logic that corresponds to the individual characteristics, thereby improving the accuracy of wakefulness detection.

[0081] In the awakening level detection device 2 according to the first embodiment, the stimulus generation unit 22 generates a stronger stimulus as the awakening level detected by the awakening level detection unit 21 decreases. This allows the awakening level detection device 2 to check the driver's response to the stimulus and correct the awakening level even when the driver's awakening level is low.

[0082] In the awakening detection device 2 according to the first embodiment, the stimulus generator 22 learns stimuli that are likely to elicit a reaction from the driver, thereby enabling the awakening detection device 2 to select stimuli that are likely to elicit a reaction from each individual driver.

[0083] A driving assistance system 1 according to a first embodiment includes an alertness detection device 2 that detects the alertness of a driver, a stimulating device 3 that provides a stimulus to the driver, and an occupant state detection device 4 that detects the state of the driver. The alertness detection device 2 includes an alertness detection unit 21 that detects the alertness of the driver based on the state of the driver detected by the occupant state detection device 4, a stimulus generation unit 22 that controls the stimulating device 3 in accordance with the alertness detected by the alertness detection unit 21 to generate a stimulus for the driver using the stimulating device 3, a response detection unit 23 that detects a response of the driver to the stimulus based on the state of the driver that has been provided with the stimulus by the stimulating device 3, detected by the occupant state detection device 4, and a correction unit 24 that corrects the alertness of the driver based on the response of the driver detected by the response detection unit 23. This enables the driving assistance system 1 to objectively evaluate the alertness of an occupant of a mobile vehicle with respect to a stimulus provided to the occupant.

[0084] The wakefulness detection method according to the first embodiment includes the steps of: a step (ST1) in which the wakefulness detection unit 21 detects the wakefulness of the driver; a step (ST2 and ST3) in which the stimulus generation unit 22 generates a stimulus for the driver according to the wakefulness detected by the wakefulness detection unit 21; a step (ST4) in which the response detection unit 23 detects the response of the driver to the stimulus generated by the stimulus generation unit 22; and a step (ST5) in which the correction unit 24 corrects the wakefulness of the driver based on the response of the driver detected by the response detection unit 23. The wakefulness detection device 2 executes the wakefulness detection method according to the first embodiment, thereby making it possible to objectively evaluate the wakefulness of an occupant of a mobile body in response to a stimulus applied to the occupant.

[0085] Embodiment 2. Fig. 5 is a block diagram showing an example of the configuration of a driving assistance system 1A according to embodiment 2. In Fig. 5, the driving assistance system 1A is a system connected to an alertness detection device 2A, a stimulation device 3, an occupant state detection device 4, a vehicle state detection device 5, an automatic driving control device 6, and an alertness enhancement device 7 via wired or wireless signal lines. The driving assistance system 1A provides driving assistance to an occupant of a moving body, for example, a driver of a vehicle. The moving body is not limited to an automobile, but may also be a railroad vehicle, an aircraft, a ship, or the like. The automobile may be an autonomous vehicle equipped with an automatic driving function.

[0086] (Overview of Driving Assistance System) The driving assistance system 1A calculates the driver's level of alertness based on detected information indicating the driver's state, and if the calculated level of alertness is equal to or lower than a determination level, provides a stimulus to the driver and corrects the driver's level of alertness based on the driver's response to the stimulus. The driving assistance system 1A provides driving assistance to the driver based on the corrected level of alertness. By using the corrected level of alertness, which is an objective evaluation of the driver's level of alertness in response to the stimulus, it is possible to grasp the driver's accurate level of alertness and provide driving assistance according to this level of alertness. Furthermore, driving assistance by the driving assistance system 1A can also be applied to passengers other than the driver. Below, a case where the target of alertness detection is the driver of a vehicle will be described.

[0087] (Overview of the Arousal Level Detection Device) The arousal level detection device 2A detects the arousal level of the driver. The arousal level detection device 2A calculates the arousal level of the driver based on information indicating the driver's state detected by the occupant state detection device 4, controls the stimulation device 3 to provide a stimulus to the driver based on the result of comparing the calculated arousal level with multiple thresholds, and corrects the arousal level of the driver based on the driver's response to the provided stimulus. The arousal level detection device 2A performs driving assistance by controlling the automatic driving control device 6 or the alertness improvement device 7 based on the corrected arousal level.

[0088] (Vehicle State Detection Device) The vehicle state detection device 5 is a device that detects the vehicle state of an automobile. For example, the vehicle state detection device 5 detects the vehicle state based on sensor information detected by various sensors mounted on the automobile. The vehicle state includes, for example, the automobile's speed, acceleration, running position, driving operation, window open / close status, object detection results in front of the vehicle, or object detection results behind the vehicle. The information detected by the vehicle state detection device 5 is stored in the memory unit 26 as appropriate.

[0089] (Autonomous driving control device) The automatic driving control device 6 is a device that controls the automatic driving of the automobile based on sensor information detected by various sensors mounted on the automobile. For example, the automatic driving control device 6 can change the automatic driving level in response to an instruction from the alertness detection device 2A.

[0090] (Arousal Level Enhancement Device) The alertness enhancement device 7 is a device that enhances the alertness of the driver's vehicle. For example, the alertness enhancement device 7 controls the stimulation device 3 in response to an instruction from the alertness detection device 2 to provide the driver's vehicle with a stimulus that enhances alertness. Examples of the stimulus that enhances alertness include an audio alarm, a tactile stimulus on the seat, airflow, a vibration stimulus on the steering wheel, or opening and closing a window. It is sufficient to have at least one of the automatic driving control device 6 and the alertness enhancement device 7.

[0091] (Details of the Arousal Level Detection Device) The following describes the details of the arousal level detection device 2A. The arousal level detection device 2A includes an arousal level detection unit 21, a stimulus generation unit 22, a response detection unit 23, a correction unit 24, a driving assistance unit 25, and a storage unit 26. For example, the arousal level detection device 2A is realized by a computer. A memory included in the computer stores a program constituting an information processing application for realizing each of the functions of the arousal level detection unit 21, the stimulus generation unit 22, the response detection unit 23, the correction unit 24, the driving assistance unit 25, and the storage unit 26. A processor included in the computer reads the information processing application from the memory and executes it, thereby realizing each of the functions of the arousal level detection unit 21, the stimulus generation unit 22, the response detection unit 23, the correction unit 24, the driving assistance unit 25, and the storage unit 26.

[0092] (Driving Assistance Unit) The driving assistance unit 25 provides driving assistance for the vehicle based on the alertness detected by the alertness detection unit 21 or the corrected alertness corrected by the correction unit 24. For example, the driving assistance unit 25 controls at least one of the automatic driving control device 6 or the alertness enhancement device 7 to provide driving assistance for the vehicle.

[0093] (Memory Unit) The memory unit 26 stores at least one of the alertness detected by the alertness detection unit 21, the type of stimulus generated by the stimulus generation unit 22, the type of driver's response before and after the stimulus generation unit 22 generates the stimulus, and the corrected alertness corrected by the correction unit 24. Based on the contents stored in the memory unit 26, it is possible to alert driving personnel and to understand the state of the vehicle and the driver. In addition, by connecting to the memory unit 26 via communication and using the stored contents as learning data, it is possible to learn the alertness detection logic, etc. For example, it is possible to learn the alertness detection logic on the server side communicatively connected to the memory unit 26, and it is also possible to learn the alertness detection logic on the local terminal side communicatively connected to the memory unit 26.

[0094] Next, a method for detecting a level of alertness according to the second embodiment will be described. FIG. 6 is a flowchart showing the method for detecting a level of alertness according to the second embodiment. The alertness detection unit 21 detects the level of alertness D(t) of the driver based on detection information indicating the state of the driver detected by the occupant state detection device 4 (step ST1A). Here, t is the time period during which the state of the driver is measured by the occupant state detection device 4. Note that the level of alertness is highest when D(t) = 1 and lowest when D(t) = 0. It is also assumed that the first threshold Dth1 ≥ the second threshold Dth2 > the third threshold Dth3.

[0095] The stimulus generator 22 compares the alertness D(t) detected by the alertness detector 21 with a first threshold Dth1, a second threshold Dth2, and a third threshold Dth3 (step ST2A). If the alertness D(t) is equal to or greater than the first threshold Dth1 (i.e., the alertness is at a level at which correction is not required), the stimulus generator 22 determines that the alertness is high enough to cause no problems, and proceeds to step ST9A. If the first threshold Dth1 > the alertness D(t) ≥ the third threshold Dth3 (i.e., the alertness is at a level at which driving assistance is not required but correction is required), the stimulus generator 22 determines that the alertness has not yet decreased to the level at which emergency driving assistance is required, and proceeds to step ST3A. If the alertness D(t) is less than the third threshold Dth3 (i.e., the alertness is at a level at which driving assistance is required), the stimulus generator 22 determines that emergency driving assistance is required, and proceeds to step ST7A.

[0096] Through the processes of steps ST3A to ST5A, the correction unit 24 calculates the corrected alertness Dc(t). The driving assistance unit 25 compares the corrected alertness Dc(t) calculated by the correction unit 24 with the second threshold value Dth2 and the third threshold value Dth3 (step ST6A). If the corrected alertness Dc(t) is equal to or greater than the second threshold value Dth2 (the corrected alertness is at a level where driving assistance is not required), the driving assistance unit 25 determines that the alertness is high enough to be acceptable, and proceeds to the process of step ST9A. If the second threshold value Dth2 > the corrected alertness Dc(t) is equal to or greater than the third threshold value Dth3 (the corrected alertness is at a level where the driver's alertness needs to be improved but driving assistance is not required), the driving assistance unit 25 determines that the alertness has not decreased to the emergency driving assistance level, and proceeds to the process of step ST8A. If the third threshold Dth3 > the corrected alertness Dc(t) (the corrected alertness has dropped to a level where driving assistance is required), the driving assistance unit 25 determines that emergency driving assistance is required and proceeds to processing of step STST7A.

[0097] The driving assistance unit 25 instructs the automatic driving control device 6 to execute automatic driving control of the vehicle (step ST7A). For example, the driving assistance unit 25 instructs the automatic driving control device 6 to execute fully automatic driving of the vehicle. However, driving assistance according to the alertness level may be performed, such as turning on the LKAS and ACC when the corrected alertness Dc(t) is the second threshold value Dth2 (strongly decreased alertness), and performing fully automatic driving when the corrected alertness Dc(t) is the alertness level DL1 (weak drowsiness) or DL0 (strong drowsiness). This allows the alertness detection device 2A to provide various driving assistance based on the corrected alertness.

[0098] The driving assistance unit 25 controls the alertness enhancement device 7, which enhances the alertness of the driver, based on the corrected alertness Dc(t) corrected by the correction unit 24, thereby restoring the alertness of the driver (step ST8A). For example, the alertness enhancement device 7 may be used to provide an audio warning, provide tactile stimulation to the seat, blow air, provide vibration stimulation to the steering wheel, or open and close a window. This allows the alertness detection device 2A to enhance the alertness of the driver.

[0099] The driving assistance unit 25 may perform an operation to increase the driver's alertness with a stronger stimulus as the driver's alertness level decreases. For example, when using an audio alarm, the sound may be increased in volume, the sound image may be changed, or the melody may be changed as the driver's alertness level decreases. This provides an operation to increase the driver's alertness with a stronger stimulus, which is expected to increase the driver's alertness. Furthermore, when the driver's alertness level is low, the driving assistance unit 25 may use a tactile stimulus rather than an audio stimulus, or may use multiple stimuli.

[0100] The driving assistance unit 25 checks whether driving of the vehicle has ended (step ST9A). For example, the awakening detection device 2A acquires vehicle information indicating the vehicle state and determines whether driving of the vehicle has ended based on the acquired vehicle information. If driving has ended (step ST9A; YES), all of the processes shown in FIG. 6 are terminated. If driving has not ended (step ST9A; NO), the process returns to step ST1A.

[0101] (Variation 6) The driving assistance unit 25 instructs the automatic driving control device 6 to control the automatic driving function of the vehicle, thereby increasing the automatic driving level of the vehicle when the stimulus generation unit 22 generates a stimulus to the driver, and decreasing the automatic driving level of the vehicle when the stimulus generation unit 22 has finished generating a stimulus to the driver or a certain time after the end of the generation of the stimulus to the driver. Note that "lowering the automatic driving level" may also be interpreted as returning to the original automatic driving level. This allows the alertness detection device 2A to reduce the impact on the driving task even if the generation of the stimulus takes up part of the driver's attention. Furthermore, the stimulus generation unit 22 may also provide a stimulus when there are few surrounding vehicles. Furthermore, the stimulus generation unit 22 may also provide a stimulus when the automatic driving control device 6 has increased the distance between the vehicle and other vehicles. Furthermore, the stimulus generation unit 22 may request the driving assistance unit 25 to control automatic driving.

[0102] As described above, the alertness detection device 2A according to the second embodiment includes a driving assistance unit 25 that controls the automatic driving function by instructing the automatic driving control device 6 of the automatic driving vehicle. The driving assistance unit 25 instructs the automatic driving control device 6 to control the automatic driving function of the automatic driving vehicle, thereby increasing the automatic driving level of the automatic driving vehicle when the stimulus generation unit 22 generates a stimulus for the driver, and decreasing the automatic driving level of the automatic driving vehicle when the generation of the stimulus for the driver by the stimulus generation unit 22 has ended. In this way, the alertness detection device 2A can reduce the impact on the driving task even if the driver's attention is partially taken up by the generation of a stimulus.

[0103] The alertness detection device 2A according to the second embodiment includes a driving assistance unit 25 that provides driving assistance for the vehicle based on the alertness corrected by the correction unit 24. This enables the alertness detection device 2A to provide various driving assistance based on the corrected alertness.

[0104] In the awakening level detection device 2A according to the second embodiment, the driving assistance unit 25 restores the driver's awakening level by controlling the awakening level enhancement device 7 that enhances the driver's awakening level, based on the awakening level corrected by the correction unit 24. In this way, the awakening level detection device 2A can enhance the driver's awakening level.

[0105] The alertness detection device 2A according to the second embodiment includes a memory unit 26 that stores at least one of the alertness detected by the alertness detection unit 21, the type of stimulus generated by the stimulus generation unit 22, the type of driver's response before and after the stimulus generation unit 22 generates the stimulus, and the alertness corrected by the correction unit 24. Based on the contents stored in the memory unit 26, it is possible to alert people involved in driving and it can also be used to understand the state of the vehicle and the driver.

[0106] The driving assistance system 1A according to the second embodiment includes an alertness improvement device 7 that improves the alertness of the driver. The alertness detection device 2A includes a driving assistance unit 25 that restores the alertness of the driver by controlling the alertness improvement device 7 that improves the alertness of the driver based on the alertness corrected by the correction unit 24. This enables the driving assistance system 1A to improve the alertness of the driver.

[0107] A driving assistance system 1A according to the second embodiment includes an automatic driving control device 6 for an automatic driving vehicle. The driving assistance unit 25 instructs the automatic driving control device 6 to control the automatic driving function of the automatic driving vehicle, thereby increasing the automatic driving level of the automatic driving vehicle when the stimulus generation unit 22 generates a stimulus to the driver, and decreasing the automatic driving level of the automatic driving vehicle when the stimulus generation unit 22 has finished generating a stimulus to the driver. This enables the driving assistance system 1A to provide various driving assistance based on the corrected alertness.

[0108] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.

[0109] The wakefulness detection device according to the present disclosure can be used, for example, for driving assistance in automobiles.

[0110] 1, 1A Driving assistance system, 2, 2A Arousal detection device, 3 Stimulation device, 4 Occupant state detection device, 5 Vehicle state detection device, 6 Automatic driving control device, 7 Arousal improvement device, 21 Arousal detection unit, 22 Stimulation generation unit, 23 Response detection unit, 24 Correction unit, 25 Driving assistance unit, 26 Storage unit, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory.

Claims

1. A consciousness level detection device for detecting the consciousness level of the occupants of a mobile vehicle, A wakefulness detection unit for detecting the wakefulness level of the crew member, A stimulus generating unit that generates a stimulus to the occupant according to the level of alertness detected by the alertness detection unit, A reaction detection unit for detecting the occupant's reaction to the stimulus generated by the stimulus generating unit, A correction unit corrects the level of alertness of the occupant detected by the alertness detection unit based on the content of the occupant's response after the stimulation is applied by the stimulation generation unit, as detected by the response detection unit. An alertness level detection device equipped with the following features.

2. The correction unit corrects the occupant's level of alertness in a direction that decreases based on the occupant's response. The alertness detection device according to claim 1.

3. The reaction detection unit detects the physical behavior of the occupant when a stimulus occurs, The correction unit corrects the occupant's level of alertness based on the content of the occupant's physical behavior. The alertness detection device according to claim 1.

4. The correction unit corrects the arousal level of the occupant based on the behavior of the occupant who has been stimulated and attempts to stimulate himself to increase his arousal level. The alertness detection device according to claim 3.

5. The reaction detection unit detects the change in the occupant's facial expression in response to a stimulus, The correction unit corrects the crew member's level of alertness based on the type of the crew member's facial expression. The alertness detection device according to claim 1.

6. The aforementioned types are one of the following: a surprised expression, a grimacing expression, or an expression that does not change much even when stimulated. The alertness detection device according to claim 5.

7. The reaction detection unit detects the occupant's blink reflex in response to a stimulus, The correction unit corrects the occupant's level of alertness based on the occupant's blink reflex pattern. The alertness detection device according to claim 1.

8. The correction unit corrects the occupant's level of alertness based on the change in the occupant's gaze when stimulated. The alertness detection device according to claim 1.

9. The aforementioned stimulus generating unit generates stimuli that are directed by auditory or visual means. The correction unit corrects the occupant's level of alertness based on the movement of the gaze relative to the direction of the stimulus. The alertness detection device according to claim 8.

10. The aforementioned alertness detection unit is The crew member's level of alertness is detected using a detection logic based on one of several alertness detection indices associated with each of several types of responses. The detection logic is dynamically changed based on the alertness detection index linked to the occupant's reaction detected by the reaction detection unit. The alertness detection device according to claim 1.

11. The stimulus generating unit generates a stronger stimulus the lower the level of arousal detected by the arousal level detection unit. The alertness detection device according to claim 1.

12. The stimulus generating unit learns stimuli that are likely to elicit a response from the occupant. The alertness detection device according to claim 1.

13. The system includes a driving support unit that instructs the automatic driving control device of the moving body to control the automatic driving function, The driving support unit instructs the automatic driving control device to control the automatic driving function of the mobile vehicle, thereby increasing the automatic driving level of the mobile vehicle when the stimulus generating unit generates a stimulus to the occupant, and decreasing the automatic driving level of the mobile vehicle when the generation of the stimulus to the occupant by the stimulus generating unit ends. The alertness detection device according to claim 1.

14. The vehicle is equipped with a driver assistance unit that provides driver assistance for the mobile vehicle based on the level of alertness corrected by the correction unit. The alertness detection device according to claim 1.

15. The driver assistance unit restores the occupant's level of alertness by controlling an alertness enhancement device that improves the occupant's level of alertness based on the alertness level corrected by the correction unit. The alertness level detection device according to claim 14.

16. The system includes a storage unit that stores at least one of the following: the level of alertness detected by the alertness detection unit, the type of stimulus generated by the stimulus generation unit, the type of response of the occupant before and after the stimulus generation unit generates the stimulus, and the level of alertness corrected by the correction unit. The alertness detection device according to claim 1.

17. A occupant status detection device that detects the status of the occupants of a mobile vehicle, A stimulator that provides stimulation to the aforementioned occupant, The system includes an alertness detection device for detecting the alertness level of the crew member, The aforementioned alertness detection device is Based on the state of the occupant detected by the occupant state detection device, an alertness detection unit detects the occupant's alertness level, A stimulation generating unit that generates stimulation to the occupant by the stimulation device, by controlling the stimulation device according to the level of arousal detected by the arousal level detection unit, A response detection unit detects the occupant's response to a stimulus based on the occupant's state detected by the occupant state detection device and the occupant's state to which the stimulus device has stimulated, The system includes a correction unit that corrects the level of alertness of the occupant detected by the alertness detection unit based on the content of the occupant's response after stimulation is applied by the stimulation generation unit, as detected by the response detection unit. A driver assistance system characterized by the following features.

18. The aforementioned occupant is equipped with an alertness-enhancing device that improves the occupant's level of alertness, The aforementioned alertness detection device includes a driving support unit that restores the occupant's alertness by controlling the alertness enhancement device, which improves the occupant's alertness, based on the alertness level corrected by the correction unit. The driver assistance system according to feature 17.

19. The mobile body is equipped with an automatic driving control device, The driving support unit instructs the automatic driving control device to control the automatic driving function of the mobile vehicle, thereby increasing the automatic driving level of the mobile vehicle when the stimulus generating unit generates a stimulus to the occupant, and decreasing the automatic driving level of the mobile vehicle when the generation of the stimulus to the occupant by the stimulus generating unit ends. The driver assistance system according to feature 18.

20. A method for detecting the level of alertness of an occupant of a mobile vehicle, performed by an alertness detection device, The alertness detection unit performs the step of detecting the alertness level of the occupant, The stimulation generating unit generates a stimulus to the occupant according to the level of arousal detected by the arousal level detection unit, The reaction detection unit detects the occupant's reaction to the stimulus generated by the stimulus generation unit, The correction unit corrects the arousal level of the occupant detected by the arousal level detection unit based on the content of the occupant's response after the stimulus is applied by the stimulus generation unit, as detected by the response detection unit. A method for detecting the level of alertness, equipped with the following features.