Estimation system, control system, estimation method, and control method

Abstract

To estimate an arousal direction of a driver of a vehicle.SOLUTION: A first derivation unit 41 derives a first index value corresponding to a driver's visual stimulation based on a front image of a vehicle obtained by an imaging unit. A second derivation unit 42 derives a second index value corresponding to a relatively strong visual stimulation among a driver's visual stimuli based on a result of comparing the first index value with a predetermined stimulation threshold. An arousal direction estimation unit 43 estimates an arousal direction, which is a direction of change in the driver's arousal level, based on a correlation between a temporal change in the first index value and a temporal change in the second index value.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The technology disclosed herein relates to a technology for estimating the wakefulness state of a vehicle driver. [Background technology]

[0002] Patent Document 1 discloses an arousal level estimation device that estimates the arousal level of a vehicle occupant based on a plurality of feature quantities that indicate the state of the vehicle occupant. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-169100 Summary of the Invention [Problem to be solved by the invention]

[0004] For the purpose of improving the accuracy of estimating the driver's level of alertness, it is effective to consider the direction of change in the driver's level of alertness (whether the level of alertness is decreasing or recovering). However, Patent Document 1 does not disclose or suggest anything about the direction of alertness of a vehicle driver.

[0005] The technology disclosed herein has been made in consideration of the above points, and its purpose is to estimate the direction of awakening of a vehicle driver. [Means for solving the problem]

[0006] The technology disclosed herein relates to an estimation system for estimating the state of alertness of a vehicle driver, and the estimation system includes an imaging unit that obtains an image of the area in front of the vehicle by capturing an image of the area in front of the vehicle, a first derivation unit that derives a first index value corresponding to a visual stimulus of the driver based on the image of the area in front of the vehicle obtained by the imaging unit, a second derivation unit that derives a second index value corresponding to a relatively strong visual stimulus among the visual stimuli of the driver based on a result of comparing the first index value with a predetermined stimulus threshold, and an alertness direction estimation unit that estimates the alertness direction, which is the direction of change in the driver's alertness, based on a correlation between the temporal change in the first index value and the temporal change in the second index value.

[0007] As a result of intensive research, the inventors of the present application have found that "visual stimuli," which are stimuli given to the driver's vision by scenery within the driver's field of view, include "arousing stimuli," which are visual stimuli that affect changes in the driver's level of arousal, and that these "arousing stimuli" depend on "the correlation between temporal changes in visual stimuli and temporal changes in perceptual stimuli (comparatively strong visual stimuli among visual stimuli)." The inventors have also found that there is a correlation between the "arousing stimuli" and the "direction of driver arousal," and that the "direction of driver arousal" can be estimated based on the "arousing stimuli."

[0008] In the above configuration, the first index value is a value corresponding to a visual stimulus, the second index value is a value corresponding to a perceptual stimulus (a relatively strong visual stimulus), and the correlation between the temporal change in the first index value and the temporal change in the second index value is a value corresponding to the arousal stimulus. Then, the arousal direction of the vehicle driver can be estimated based on the correlation between the temporal change in the first index value and the temporal change in the second index value (the value corresponding to the arousal stimulus).

[0009] In the estimation system, the awakening direction estimation unit may estimate that the driver's awakening direction is a downward direction when a correlation coefficient indicating the correlation between the temporal change in the first index value and the temporal change in the second index value is below a predetermined correlation threshold, and may estimate that the driver's awakening direction is a recovery direction when the correlation coefficient is above the correlation threshold.

[0010] With the above configuration, it is possible to estimate whether the driver's level of wakefulness of the vehicle is decreasing or recovering.

[0011] The estimation system may include an information acquisition unit that acquires information used to estimate the driver's level of alertness, and an alertness estimation unit that estimates the driver's level of alertness based on the information acquired by the information acquisition unit and the driver's direction of alertness estimated by the alertness direction estimation unit.

[0012] In the above configuration, the driver's level of alertness can be estimated taking into consideration the driver's direction of alertness, thereby improving the accuracy of estimating the driver's level of alertness.

[0013] The technology disclosed herein relates to a control system, which includes the estimation system and a response control unit that performs response control in accordance with the driver's alertness state estimated by the estimation system, and the response control includes at least one of alertness control for restoring the driver's alertness and evacuation control for controlling the vehicle's driving so that the vehicle evacuates to a safe area.

[0014] In the above configuration, the response control can be performed taking into consideration the degree to which the driver is awake, so that the response control can be performed appropriately.

[0015] The technology disclosed herein relates to an estimation method for estimating the state of alertness of a vehicle driver, which includes a first derivation step of deriving a first index value corresponding to a visual stimulus of the driver based on an image of the area in front of the vehicle obtained by an imaging unit that images the area in front of the vehicle, a second derivation step of deriving a second index value corresponding to a relatively strong visual stimulus among the visual stimuli of the driver based on the result of comparing the first index value with a predetermined stimulus threshold, and an alertness direction estimation step of estimating the alertness direction, which is the direction of change in the driver's alertness, based on the correlation between the temporal change in the first index value and the temporal change in the second index value.

[0016] In the above method, the first index value is a value corresponding to a visual stimulus, the second index value is a value corresponding to a perceptual stimulus (a relatively strong visual stimulus), and the correlation between the temporal change in the first index value and the temporal change in the second index value is a value corresponding to an arousal stimulus. Then, the arousal direction of the vehicle driver can be estimated based on the correlation between the temporal change in the first index value and the temporal change in the second index value (the value corresponding to the arousal stimulus).

[0017] The technology disclosed herein relates to a control method, which includes the estimation method and a response control step of performing response control according to the driver's state of alertness estimated by the estimation method, and the response control includes at least one of alertness control for restoring the driver's level of alertness and evacuation control for controlling the vehicle's driving so that the vehicle evacuates to a safety area.

[0018] In the above-described method, the countermeasure control can be performed in consideration of the driver's awakening direction, and therefore the countermeasure control can be performed appropriately. [Effects of the Invention]

[0019] According to the technology disclosed herein, it is possible to estimate the direction of awakening of a vehicle driver. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram illustrating a configuration of an in-vehicle system according to an embodiment; [Figure 2] 10 is a graph illustrating an example of a change over time in a first index value (visual stimulation). [Figure 3] 10 is a graph illustrating an example of a change over time in a second index value (sensory stimulation). [Figure 4] 10 is a graph illustrating an example of a change in correlation coefficient (arousal stimulus) over time. [Figure 5] 10 is a graph illustrating an example of a change in wakefulness over time. [Figure 6] 10 is a graph illustrating an example of a change in a direction coefficient (arousal direction) over time. [Figure 7]10 is a graph illustrating the relationship between a direction coefficient and a correlation coefficient. [Figure 8] 10 is a graph illustrating a distribution of correlation coefficients in which the directional coefficient is zero. [Figure 9] 10 is a flowchart illustrating a flow of processing by a control device. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, the embodiments will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals and their description will not be repeated.

[0022] (Terminology explanation) First, the terms used in the following description will be explained. In the following, the terms driver, alertness, alertness direction, visual stimulus, sensory stimulus, and alert stimulus will be used.

[0023] A driver is a person who gets into a vehicle and drives the vehicle. In the following, an example is given in which the vehicle is a four-wheeled motor vehicle.

[0024] Arousal level refers to the degree to which a driver is awake. The higher the level of arousal, the clearer the driver's consciousness. If the driver becomes drowsy due to drowsiness, the level of arousal decreases. In the following, arousal level is expressed as a numerical value (positive number).

[0025] The direction of arousal refers to the direction (tendency) of change in the driver's level of arousal. There are three directions of arousal: a "decreasing direction" in which the level of arousal gradually decreases, a "recovery direction" in which the level of arousal gradually recovers (increases), and a "maintenance direction" in which the level of arousal remains unchanged. Hereinafter, the direction of arousal is expressed by a numerical value (negative number, zero, positive number). If the direction of arousal is a "decreasing direction," the sign of the numerical value indicating the direction of arousal will be "negative," and if the direction of arousal is a "recovery direction," the sign of the numerical value indicating the direction of arousal will be "positive." The greater the degree of change in the driver's level of arousal, the greater the absolute value of the numerical value indicating the direction of arousal. Note that if the direction of arousal is a "maintenance direction," the numerical value indicating the direction of arousal will be zero.

[0026] A visual stimulus is a stimulus given to the driver's vision by a scene within the driver's field of vision. The more monotonous the scene within the driver's field of vision (for example, a highway scene), the smaller the visual stimulus tends to be, and the more complex the scene within the driver's field of vision (for example, an urban scene), the larger the visual stimulus tends to be. This is a new finding obtained by the inventor of the present application.

[0027] A perceptual stimulus is a visual stimulus that is considered to be perceptible by the driver (that reaches the driver's brain). It is presumed that not all visual stimuli are perceived by the driver, but rather that relatively strong visual stimuli (visual stimuli that exceed a certain threshold) are perceived by the driver. This is a new finding obtained by the inventor of the present application. In the following explanation, a relatively strong visual stimuli among visual stimuli is referred to as a "perceptual stimulus."

[0028] An arousal stimulus is a visual stimulus that affects changes in the driver's level of alertness. The stronger the correlation between temporal changes in visual stimuli and temporal changes in sensory stimuli (the closer the correlation coefficient is to 1), the stronger the arousal stimulus, and as a result, the easier it tends to be for the driver's level of alertness to recover (increase). Conversely, the weaker the correlation between temporal changes in visual stimuli and temporal changes in sensory stimuli (the closer the correlation coefficient is to zero), the weaker the arousal stimulus, and as a result, the easier it tends to be for the driver's level of alertness to decrease. This is a new finding obtained by the inventor of the present application. The new finding obtained by the inventor of the present application will be explained in detail later.

[0029] (In-vehicle system) 1 illustrates the configuration of an in-vehicle system 10 according to an embodiment. The in-vehicle system 10 is mounted on a vehicle (not shown) and controls various parts of the vehicle. In this example, the in-vehicle system 10 includes an exterior camera 11, an interior camera 12, an information acquisition device 15, an actuator 20, a speaker 21, a display 22, an air conditioning device 23, a vibration device 24, a storage device 30, and a control device 40. In this example, the vehicle is switchable between manual driving, in which the vehicle travels in response to driver operation (e.g., accelerator operation), assisted driving, in which the vehicle travels with assistance from the driver, and automatic driving, in which the vehicle travels without driver operation.

[0030] [External camera (imaging unit)] The exterior camera 11 captures an image of the area ahead of the vehicle (image data showing the image). Specifically, the exterior camera 11 repeatedly captures images at a predetermined frame rate to obtain multiple images (images of the area ahead of the vehicle) arranged in chronological order. The images of the area ahead of the vehicle can be said to be images showing the scenery within the field of view of the driver of the vehicle. The exterior camera 11 is an example of an imaging unit that captures an image of the area ahead of the vehicle.

[0031] [In-car camera (information acquisition unit)] The in-vehicle camera 12 is disposed in front of the driver inside the vehicle and captures an image of the driver (particularly the face) to obtain an image including the driver (particularly the face). Specifically, the in-vehicle camera 12 repeatedly captures images at a predetermined frame rate to obtain a plurality of images (images including the driver) arranged in chronological order. The in-vehicle camera 12 is an example of an information acquisition unit that acquires information used to estimate the driver's alertness.

[0032] [Information acquisition equipment] The information acquisition device 15 acquires various types of information used for vehicle driving control (specifically, control of the actuator 20). The various types of information acquired by the information acquisition device 15 are transmitted to the control device 40. The information acquisition device 15 includes multiple cameras, multiple radars, a communication unit, a vehicle state sensor, a driving operation sensor, a driver state sensor, etc.

[0033] The multiple cameras capture images of the environment (external environment) surrounding the vehicle to obtain images (image data) showing the external environment of the vehicle. The multiple radars transmit search waves toward the external environment of the vehicle and receive reflected waves from the external environment of the vehicle to obtain information showing the external environment of the vehicle (direction and distance to objects). Note that some of the multiple cameras may also be used as "exterior vehicle cameras 11."

[0034] The communication unit receives various types of information via external networks such as the Internet. Examples of information received by the communication unit include GPS information (information indicating the vehicle's location) from the Global Positioning System, communication information from other vehicles located around the vehicle, car navigation data from a navigation system, traffic information, and high-precision map information such as dynamic maps.

[0035] The vehicle state sensor detects the state of the vehicle. Examples of the vehicle state include the vehicle speed, acceleration, and yaw rate. For example, the vehicle state sensor includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The driving operation sensor detects driving operations input to the vehicle. Examples of driving operations include the accelerator operation amount, steering angle, and brake operation amount. For example, the driving operation sensor includes an accelerator opening sensor, a steering angle sensor, and a brake oil pressure sensor.

[0036] The driver status sensor detects the status of the driver of the vehicle. Examples of the driver status include physical behavior and biological information. For example, the driver status sensor includes a driver camera that captures an image (image data) including the driver by capturing an image of the driver, and a biological information sensor that acquires the driver's biological information. Note that the driver camera may also be used as the "in-vehicle camera 12."

[0037] [Actuator] The actuators 20 include drive system actuators, steering system actuators, braking system actuators, etc. Examples of drive system actuators include an engine, a transmission, a motor, etc. Examples of steering system actuators include a steering wheel, etc. Examples of braking system actuators include a brake, etc.

[0038] [Speakers, displays, air conditioning devices, and vibration devices (awakening devices)] The speaker 21 outputs audio to the driver. The display 22 displays images to the driver. The air conditioning device 23 conditions the air inside the vehicle and blows temperature-controlled air toward the driver. The vibration device 24 applies vibrations to the driver. For example, the vibration device 24 vibrates a seat belt (not shown) worn by the driver. The speaker 21, the display 22, the air conditioning device 23, and the vibration device 24 are an example of an awakening device that operates to restore the driver's level of awakening. Hereinafter, these will be collectively referred to as "awakening device 25."

[0039] [Storage device] The storage device 30 stores various types of information and data. Specifically, the storage device 30 stores information and data (e.g., setting values ​​such as thresholds) used for control or information processing in the in-vehicle system 10, information and data (e.g., image data and calculation results) obtained by each unit of the in-vehicle system 10, and the like. For example, the storage device 30 is configured by a non-volatile memory or the like.

[0040] [Control device] The control device 40 is connected to each part of the in-vehicle system 10 (in this example, the exterior camera 11, the interior camera 12, the alertness device 25, and the storage device 30) via signal lines, and controls each part of the in-vehicle system 10. The control device 40 also transmits and receives various information and data to and from each part of the in-vehicle system 10. The control device 40 then performs various processes based on instructions input to the control device 40, various information and data obtained in each part of the in-vehicle system 10, various information and data stored in the storage device 30, etc.

[0041] For example, the control device 40 is configured by a computer having one or more processors and one or more memories that store various programs for operating the one or more processors. The processors execute the various programs to realize various processes (functions) of the control device 40. The memory may store information and data (e.g., setting values ​​such as thresholds) used for processing by the control device 40. The control device 40 may include a microcontroller that performs predetermined control (calculation), may be realized by hardware such as a dedicated circuit, or may be realized by a combination of hardware and software such as a program.

[0042] In this example, the control device 40 estimates the wakefulness state of the vehicle driver and performs control in accordance with the wakefulness state of the driver. The control device 40 is an example of an estimation system that estimates the wakefulness state of the vehicle driver. The control device 40 is also an example of a control system that performs control in accordance with the wakefulness state of the driver estimated by the estimation system. Among the processes performed by the control device 40, the process related to "estimating the wakefulness state of the driver" is an example of an estimation method for estimating the wakefulness state of the vehicle driver. The process performed by the control device 40 is also an example of a control method that performs control in accordance with the wakefulness state of the driver estimated by the estimation method.

[0043] [Functional configuration of the control device] The control device 40 has a first derivation unit 41, a second derivation unit 42, an arousal direction estimation unit 43, an arousal level estimation unit 44, an arousal control unit 45, and a driving control unit 46. Specifically, the control device 40 functions as the first derivation unit 41, the second derivation unit 42, the arousal direction estimation unit 43, the arousal level estimation unit 44, the arousal control unit 45, and the driving control unit 46 by the processor thereof executing various programs.

[0044] The first derivation unit 41 derives a first index value corresponding to a visual stimulus of the driver based on an image of the area ahead of the vehicle obtained by the exterior camera 11. The second derivation unit 42 derives a second index value corresponding to a relatively strong visual stimulus (perceptual stimulus) among the visual stimuli of the driver based on a result of comparing the first index value with a predetermined stimulus threshold.

[0045] The arousal direction estimation unit 43 estimates the arousal direction, which is the direction of change in the driver's arousal level, based on the correlation between the temporal change in the first index value derived by the first derivation unit 41 and the temporal change in the second index value derived by the second derivation unit 42. Specifically, the arousal direction estimation unit 43 derives a correlation coefficient indicating the correlation between the temporal change in the first index value and the temporal change in the second index value, and estimates that "the driver's arousal direction is decreasing" when the correlation coefficient is below a predetermined correlation threshold, and estimates that "the driver's arousal direction is recovering" when the correlation coefficient is above the correlation threshold.

[0046] The awakening level estimation unit 44 estimates the awakening level of the driver based on the image acquired by the in-vehicle camera 12 and the awakening direction of the driver estimated by the awakening direction estimation unit 43. The image acquired by the in-vehicle camera 12 is an example of information acquired by the information acquisition unit.

[0047] The awakening control unit 45 performs an awakening process to restore the awakening level of the driver in accordance with the awakening level (an example of an awakening state) of the driver estimated by the awakening level estimation unit 44. Specifically, the awakening control unit 45 controls one or more awakening devices 25 so that the one or more awakening devices 25 perform an operation to restore the awakening level of the driver. The awakening control will be described in detail later.

[0048] The driving control unit 46 controls the driving of the vehicle. In this example, the driving control unit 46 controls the actuator 20 so that the vehicle drives along a target route in assisted driving or automatic driving. Specifically, the driving control unit 46 determines a target route, which is the route the vehicle should drive, based on various information obtained by the information acquisition device 15, determines a target motion, which is the vehicle motion required to drive along the target route, and controls the actuator 20 so that the vehicle motion becomes the target motion. Note that well-known automatic driving technology can be adopted for the driving control by the driving control unit 46.

[0049] The driving control unit 46 also performs evacuation control to control the driving of the vehicle so that the vehicle evacuates to a safe area, depending on the level of wakefulness of the driver (an example of an awakening state) estimated by the wakefulness level estimation unit 44. The evacuation control will be described in detail later.

[0050] The level of wakefulness of the driver estimated by the wakefulness estimation unit 44 is an example of the wakefulness state of the driver estimated by the estimation system. The wakefulness processing and the evacuation processing are examples of response control according to the wakefulness state of the driver estimated by the estimation system. The wakefulness control unit 45 and the driving control unit 46 are examples of response control units that perform response control. Hereinafter, the wakefulness control unit 45 and the driving control unit 46 will be collectively referred to as the "response control unit 47."

[0051] [Findings Obtained by the Inventors of the Present Application] As a result of extensive research, the inventors of the present application have found that "visual stimuli," which are stimuli given to the driver's vision by scenery within the driver's field of view, include "arousing stimuli," which are visual stimuli that affect changes in the driver's level of arousal, and that these "arousing stimuli" depend on "the correlation between temporal changes in visual stimuli and temporal changes in perceptual stimuli."The inventors have also found that there is a correlation between the "arousing stimuli" and the "direction of driver arousal," and that the "direction of driver arousal" can be estimated based on the "arousing stimuli."

[0052] The following describes experiments carried out by the inventors of the present invention.

[0053] 1. Image preparation First, a vehicle equipped with an exterior camera 11 was actually driven, and multiple images (multiple images arranged in chronological order) showing the scenery ahead of the vehicle were obtained by continuously capturing images of the area ahead of the vehicle using the exterior camera 11. In the following, the subject to whom the multiple images were shown will be referred to as the "driver."

[0054] 2. Quantification of visual stimuli Next, we attempted to quantify visual stimuli. Visual stimuli fluctuate according to changes in the scenery ahead of the vehicle (the scenery within the driver's field of vision). Specifically, the smaller the change in the scenery ahead of the vehicle (the more monotonous the scenery), the smaller the visual stimuli tend to be. Conversely, the greater the change in the scenery ahead of the vehicle (the more complex the scenery), the greater the visual stimuli tend to be.

[0055] In order to quantify the visual stimulus, a first index value corresponding to the visual stimulus was derived based on the image acquired by the above-described outside-vehicle camera 11. In this experiment, the first index value is a value indicating the temporal change in the average brightness value (hereinafter referred to as "average brightness value") of each of multiple pixels included in the image acquired by the outside-vehicle camera 11. Specifically, the first index value at the time when the k+1th image (k is an integer equal to or greater than 1) in chronological order was acquired is a difference value (difference between frames) obtained by subtracting the "average brightness value of the k+1th image" from the "average brightness value of the kth image."

[0056] Figure 2 shows the first index value in response to visual stimuli. The first index value fluctuated in response to changes in the scenery ahead of the vehicle shown in multiple images arranged in chronological order. Specifically, in periods when the scenery ahead of the vehicle changed relatively little (periods when the scenery was relatively monotonous), the first index value in response to visual stimuli tended to be relatively small. Conversely, in periods when the scenery ahead of the vehicle changed relatively much (periods when the scenery was relatively complex), the first index value in response to visual stimuli tended to be relatively large.

[0057] 3. Quantification of Perceptual Stimuli Next, we attempted to quantify the perceptual stimuli. It is assumed that not all visual stimuli are perceived by the driver, but rather that relatively strong visual stimuli (visual stimuli that exceed a certain threshold) are perceived by the driver. In other words, it is assumed that relatively strong visual stimuli become "perceptual stimuli."

[0058] To quantify the sensory stimulus, the first index value derived in "2. Quantification of visual stimulus" above was compared with a predetermined stimulus threshold, and a second index value corresponding to the sensory stimulus was derived based on the result of the comparison. The second index value is a value indicating the presence or absence of a sensory stimulus. Specifically, the second index value at the time when the kth image in chronological order was acquired (hereinafter referred to as the "kth time") is "1" if the first index value at the kth time exceeds the stimulus threshold, and is "0" if the first index value at the kth time does not exceed the stimulus threshold.

[0059] The stimulation threshold can be said to be the minimum value of the first index value corresponding to the visual stimulation that can be considered to be perceptible by the driver. Such a stimulation threshold can be derived as an appropriate value through experiments, etc. The stimulation threshold may also be adjusted so that the distribution of the correlation coefficient (described later) becomes an appropriate distribution (a distribution suitable for estimating the direction of arousal).

[0060] Figure 3 shows the temporal change in the second index value in response to sensory stimulation. The second index value was "1" when the first index value exceeded the stimulation threshold (a time when it can be considered that a sensory stimulation was present), and was "0" when the first index value did not exceed the stimulation threshold (a time when it can be considered that no sensory stimulation was present).

[0061] 4. Quantification of arousal stimuli Next, we attempted to quantify the arousal stimulus. It is assumed that the greater the amount of sensory stimulus within a given period (the proportion of the period during which the sensory stimulus is applied), the more likely the driver's level of arousal will increase (recover). Conversely, it is assumed that the smaller the amount of sensory stimulus within a given period, the more likely the driver's level of arousal will decrease. In other words, it is assumed that the amount of sensory stimulus within a given period is the "strength of the arousal stimulus."

[0062] The strength of the arousal stimulus depends on the correlation between the temporal changes in visual stimuli and sensory stimuli within a given period of time. The stronger the correlation between the temporal changes in visual stimuli and sensory stimuli within a given period of time (the closer the correlation coefficient is to 1), the greater the amount of sensory stimulus within the given period of time, resulting in a stronger arousal stimulus. Conversely, the weaker the correlation between the temporal changes in visual stimuli and sensory stimuli within a given period of time (the closer the correlation coefficient is to zero), the less the amount of sensory stimulus within the given period of time, resulting in a weaker arousal stimulus.

[0063] In order to quantify the arousal stimulus, a correlation coefficient was derived that indicates the correlation between the temporal change in the first index value derived in the above "2. Quantification of visual stimulus" and the temporal change in the second index value derived in the above "3. Quantification of sensory stimulus." Specifically, for each predetermined period, the correlation coefficient for the predetermined period was derived by substituting n (n is an integer of 2 or more) first index values ​​(x) and n second index values ​​(y) obtained within that predetermined period into the following formula. In this experiment, the length of the predetermined period was 30 seconds.

[0064]

number

[0065] In the above formula, “x i " is the i-th (i is an integer between 1 and n) first index value within a predetermined period. The overlined "x" is the average value of n first index values. "y i" is the i-th second index value within a predetermined period. The overlined "y" is the average value of n second index values.

[0066] Figure 4 shows the temporal change (distribution) of the correlation coefficient in response to arousal stimuli. During periods when the scenery ahead of the vehicle was relatively monotonous, the amount of sensory stimuli within a given period (the proportion of the period during which sensory stimuli were provided) was relatively small, and as a result, the correlation coefficient tended to be relatively small. Conversely, during periods when the scenery ahead of the vehicle was relatively complex, the amount of sensory stimuli within a given period was relatively large, and as a result, the correlation coefficient tended to be relatively large.

[0067] 5. Deriving the level of arousal In addition, multiple images (images showing the scenery ahead of the vehicle) obtained in "1. Image Preparation" above were displayed on a display (not shown) of the driving simulator, and the driver (subject) was shown the images displayed on the display to derive the driver's level of alertness. A well-known technique (technology for determining alertness) can be used to derive the driver's level of alertness. In this experiment, a camera (a camera equivalent to the in-vehicle camera 12) placed in front of the driver captured an image of the driver's face, and the driver's level of eye opening was derived based on the image (image including the driver's face) obtained by the camera, and the driver's level of alertness was derived based on the driver's level of eye opening.

[0068] Figure 5 shows the change in the driver's alertness over time. The driver's alertness fluctuated according to changes in the scenery ahead of the vehicle (the scenery within the driver's field of vision) shown in multiple images arranged in chronological order. Specifically, during periods when the scenery ahead of the vehicle changed relatively little (periods when the scenery was relatively monotonous), the driver's alertness tended to gradually decrease. Conversely, during periods when the scenery ahead of the vehicle changed relatively much (periods when the scenery was relatively complex), the driver's alertness tended to gradually increase.

[0069] 6. Quantification of the direction of arousal Next, we attempted to quantify the direction of a driver's arousal. If the driver's level of arousal decreases over time, the direction of the driver's arousal is considered to be "decreasing," and if the driver's level of arousal increases over time, the direction of the driver's arousal is considered to be "recovering." Furthermore, the greater the degree of change in the driver's level of arousal over time, the greater the magnitude of the driver's arousal direction (the magnitude of the numerical value indicating the direction of arousal).

[0070] In order to quantify the direction of the driver's arousal, a directional coefficient corresponding to the driver's arousal direction was derived based on the driver's arousal level derived in "5. Derivation of arousal level" above. The directional coefficient is a value that indicates the direction and magnitude of the driver's arousal direction. The sign of the directional coefficient is "negative" when the driver's arousal direction is "decreasing" and "positive" when the driver's arousal direction is "recovering." The greater the degree of change in the driver's arousal level, the greater the absolute value of the directional coefficient.

[0071] In this experiment, the directional coefficient was a value indicating the temporal change in the average value of the driver's alertness for each predetermined time period. Specifically, for each predetermined period, the average value of the driver's alertness for that predetermined period (hereinafter referred to as the "average alertness value") was derived, and the "directional coefficient for the j+1th predetermined period" was derived by subtracting the "average alertness value for the j+1th predetermined period" from the "average alertness value for the jth (j is an integer greater than or equal to 1) predetermined period." Note that the predetermined period related to the derivation of the directional coefficient is the same as the predetermined period related to the derivation of the correlation coefficient. For example, the directional coefficient for the j+1th predetermined period in chronological order corresponds to the correlation coefficient for the j+1th predetermined period.

[0072] Figure 6 shows the temporal change in the direction coefficient according to the driver's arousal direction. During periods when the scenery ahead of the vehicle was relatively "monotonous," the sign of the direction coefficient tended to be "negative." Conversely, during periods when the scenery ahead of the vehicle was relatively "complex," the sign of the direction coefficient tended to be "positive." The absolute value of the direction coefficient indicates the degree of change in the driver's arousal level.

[0073] 7. Quantifying the relationship between arousal stimuli and arousal direction As described above, during a period when the scenery ahead of the vehicle is relatively monotonous, the correlation coefficient tends to be relatively small and the sign of the directional coefficient tends to be "negative." Conversely, during a period when the scenery ahead of the vehicle is relatively complex, the correlation coefficient tends to be relatively large and the sign of the directional coefficient tends to be "positive."

[0074] Next, to quantify the relationship between arousal stimuli and arousal direction, we plotted the relationship between the correlation coefficient and the directional coefficient based on the temporal change in the correlation coefficient derived in the above "4. Quantification of arousal stimuli" (see Figure 4) and the temporal change in the directional coefficient derived in the above "6. Quantification of arousal direction" (see Figure 6).

[0075] FIG. 7 shows the relationship between the "correlation coefficient according to the arousal stimulus" and the "directional coefficient according to the driver's arousal direction." The plotted points in FIG. 7 represent combinations of correlation coefficients and directional coefficients within the same predetermined period. For example, if the correlation coefficient within the j+1th predetermined period is "0.45" and the directional coefficient is "-2," the plotted point corresponding to the j+1th predetermined period has a coordinate value of "0.45" on the horizontal axis indicating the correlation coefficient and a coordinate value of "-2" on the vertical axis indicating the directional coefficient. As shown in FIG. 7, a positive correlation was observed between the correlation coefficient and the directional coefficient.

[0076] From the above, it has been found that it is possible to estimate whether the direction coefficient corresponding to the driver's arousal direction is positive or negative based on the magnitude of the correlation coefficient corresponding to the arousal stimulus (whether the correlation coefficient is closer to zero or 1), and as a result, it is possible to estimate the driver's arousal direction (whether the arousal direction is decreasing or increasing). In other words, it has been found that it is possible to estimate how the scenery ahead of the vehicle at a given time point (for example, the current time point) is affecting the driver's arousal level (whether it is acting in a direction that decreases arousal level or in a direction that increases arousal level) based on the value corresponding to the arousal stimulus at a given time point.

[0077] 8. Derivation of correlation threshold as a criterion for determining the direction of arousal From the above "7. Relationship between arousal stimulus and arousal direction," it is conceivable that the correlation coefficient when the direction coefficient is zero can be used as a "correlation threshold that serves as a criterion for determining the driver's arousal direction (a criterion for determining whether the arousal direction is a downward direction or a recovery direction)."

[0078] To find the correlation coefficient at which the directional coefficient is zero, an approximate line (a line sloping upward to the right in Figure 7) is derived using the least squares method for the set of plot points (data values ​​indicating combinations of correlation coefficients and directional coefficients within the same predetermined period) shown in Figure 7, and the "horizontal axis coordinate value (correlation coefficient)" corresponding to the intersection of the approximate line and the "line (a line parallel to the horizontal axis) at which the vertical axis coordinate value (directional coefficient) is zero" is derived as the "correlation coefficient at which the directional coefficient is zero." In the example of Figure 7, the correlation coefficient at which the directional coefficient is zero was a value within the range of "0.3" to "0.4" (specifically, approximately 0.35).

[0079] The correlation coefficient at which the directional coefficient is zero varies depending on the driver (subject). Therefore, in order to find a highly versatile correlation threshold, the above processes (calculations) "5. Deriving the level of arousal," "6. Quantifying the direction of arousal," and "7. Quantifying the relationship between arousal stimuli and the direction of arousal" were performed for each of the 61 drivers, and the "correlation coefficient at which the directional coefficient is zero" was derived for each of the 61 drivers.

[0080] Figure 8 shows the distribution of the "correlation coefficients for which the directional coefficient is zero" for 61 drivers. As shown in Figure 8, the correlation coefficients for which the directional coefficient is zero were mostly distributed within the range of "0.5" to "0.6." From these results, it was found that by setting the correlation threshold, which is the criterion for determining the driver's direction of arousal, in the "range of 0.5 to 0.6," it is possible to appropriately determine the driver's direction of arousal over a wide range.

[0081] [Processing by the control device] Next, the processing by the control device 40 will be described with reference to Fig. 9. The control device 40 repeatedly performs the following processing at predetermined time intervals. For example, the following processing is performed every time the number of images acquired by the outside camera 11 reaches "the number at which a correlation coefficient can be derived" (every time the above-mentioned predetermined period has elapsed).

[0082] <Step S11: First Derivation Step> The first derivation unit 41 derives a first index value corresponding to the visual stimulus of the driver based on the image (image of the area ahead of the vehicle) obtained by the outside-vehicle camera 11. In this example, the procedure for deriving the first index value by the first derivation unit 41 is the same as the procedure for deriving the first index value in "2. Quantification of visual stimulus" in the above experiment. The first derivation unit 41 derives the average luminance value of the image obtained by the outside-vehicle camera 11 (the average luminance value of each of the multiple pixels included in the image) as the first index value.

[0083] Step S12: Second Derivation Step The second derivation unit 42 derives a second index value corresponding to a relatively strong visual stimulus among the visual stimuli of the driver based on the result of comparing the first index value derived in step S11 with a predetermined stimulus threshold. In this example, the procedure for deriving the second index value by the second derivation unit 42 is the same as the procedure for deriving the second index value in "3. Quantification of Perceptual Stimuli" in the above experiment. The second derivation unit 42 derives a second index value indicating "1" when the first index value exceeds the stimulus threshold, and derives a second index value indicating "0" when the first index value does not exceed the stimulus threshold.

[0084] <Step S13: Correlation Coefficient Deriving Step> The arousal direction estimation unit 43 derives a correlation coefficient indicating the correlation between the temporal change in the first index value derived in step S11 and the temporal change in the second index value derived in step S12. In this example, the procedure for deriving the correlation coefficient by the arousal direction estimation unit 43 is the same as the procedure for deriving the correlation coefficient in "4. Quantification of arousal stimulus" in the above experiment. The arousal direction estimation unit 43 derives the correlation coefficient within a predetermined period by substituting n first index values ​​(x) and n second index values ​​(y) obtained within the predetermined period into the following formula.

[0085]

number

[0086] <Step S14: Awakening Direction Estimation Step> Next, the awakening direction estimation unit 43 determines whether the correlation coefficient derived in step S13 is equal to a predetermined correlation threshold value. If the correlation coefficient is equal to the correlation threshold value, the process proceeds to step S15. If not, the process proceeds to step S16.

[0087] In this example, the correlation threshold is set to a value that allows the driver's wakefulness direction to be considered to be a maintaining direction. The correlation threshold may be indicated by a single value or may be indicated by a range defined by an upper limit value and a lower limit value. For example, the correlation threshold is "a range from 0.5 to 0.6." When the correlation threshold is a "range," the correlation coefficient is determined to be equal to the correlation threshold when it is included in the "range indicating the correlation threshold."

[0088] <Step S15: Awakening Direction Estimation Step> If the correlation coefficient is equal to the correlation threshold, the arousal direction estimation unit 43 estimates that the driver's arousal direction is the "maintenance direction." In this example, the arousal direction estimation unit 43 sets the numerical value indicating the driver's arousal direction to "zero." In this example, setting the numerical value indicating the driver's arousal direction to "zero" is synonymous with estimating that the driver's arousal direction is the "maintenance direction." Next, the processing of step S20 is performed.

[0089] <Step S16: Awakening Direction Estimation Step> If the correlation coefficient is not equal to the correlation threshold, the awakening direction estimation unit 43 determines whether the correlation coefficient is below the correlation threshold. If the correlation coefficient is below the correlation threshold, the process of step S17 is performed; if not, the process of step S18 is performed.

[0090] <Step S17: Awakening Direction Estimation Step> If the correlation coefficient is lower than the correlation threshold, the arousal direction estimation unit 43 estimates that the direction of the driver's arousal is "decreasing". In this example, the arousal direction estimation unit 43 sets the sign of the number indicating the driver's arousal direction to "negative" and sets the absolute value of the number to "a value corresponding to the difference between the correlation coefficient and the correlation threshold". The larger the difference between the correlation coefficient and the correlation threshold, the larger the absolute value of the number indicating the driver's arousal direction. In this example, setting the sign of the number indicating the driver's arousal direction to "negative" is synonymous with estimating that the direction of the driver's arousal is "decreasing". Next, the processing of step S20 is performed.

[0091] <Step S18: Awakening Direction Estimation Step> If the correlation coefficient exceeds the correlation threshold, the arousal direction estimation unit 43 estimates that the driver's arousal direction is "direction of recovery." In this example, the arousal direction estimation unit 43 sets the sign of the number indicating the driver's arousal direction to "positive," and sets the absolute value of the number to "a value corresponding to the difference between the correlation coefficient and the correlation threshold." The larger the difference between the correlation coefficient and the correlation threshold, the larger the absolute value of the number indicating the driver's arousal direction. In this example, setting the sign of the number indicating the driver's arousal direction to "positive" is synonymous with estimating that the driver's arousal direction is "direction of recovery." Next, the process of step S20 is performed.

[0092] <Step S20: Arousal Level Estimation Step> Next, the alertness estimation unit 44 estimates the driver's alertness based on the image acquired by the in-vehicle camera 12 (an example of information used to estimate the driver's alertness) and the driver's alertness direction estimated in one of steps S15, S17, or S18.

[0093] In this example, the arousal level estimation unit 44 derives an arousal index value corresponding to the arousal level of the driver based on the image obtained by the in-vehicle camera 12, and performs a arousal level estimation process to estimate the arousal level of the driver based on the result of comparing the arousal index value with a predetermined arousal threshold. Furthermore, the arousal level estimation unit 44 corrects at least one of the "arousal index value" derived in the arousal level estimation process, the "arousal threshold" used in the arousal level estimation process, and the "driver's arousal level" estimated by the arousal level estimation process, based on the driver's arousal direction estimated by the arousal direction estimation unit 43.

[0094] In the following, the arousal index value derived in the arousal level estimation process will be referred to as the "arousal index value in the arousal level estimation process," the arousal threshold used in the arousal level estimation process will be referred to as the "arousal threshold in the arousal level estimation process," and the driver's arousal level estimated by the arousal level estimation process will be referred to as the "driver's arousal level in each level estimation process."

[0095] Specifically, when the direction of the driver's wakefulness estimated by the wakefulness direction estimation unit 43 is a "decreasing direction," the wakefulness estimation unit 44 corrects at least one of the "wakefulness index value," the "wakefulness threshold," and the "driver's wakefulness" in the wakefulness estimation process so that the "driver's wakefulness" in the wakefulness estimation process is more likely to decrease. Also, when the direction of the driver's wakefulness estimated by the wakefulness direction estimation unit 43 is a "recovery direction," the wakefulness estimation unit 44 corrects at least one of the "wakefulness index value," the "wakefulness threshold," and the "driver's wakefulness" in the wakefulness estimation process so that the "driver's wakefulness" in the wakefulness estimation process is more likely to recover (increase).

[0096] The above-mentioned alertness estimation process includes an "alertness estimation process in which the estimated driver's alertness increases as the alertness index value increases relative to the alertness threshold (hereinafter referred to as "first alertness estimation process")" and an "alertness estimation process in which the estimated driver's alertness increases as the alertness index value decreases relative to the alertness threshold (hereinafter referred to as "second alertness estimation process")."

[0097] When the "first awakening level estimation process" is performed in the awakening level estimation unit 44, the awakening level estimation unit 44 may be configured as follows: The following predetermined amount (correction amount) may be set to increase according to an increase in the magnitude of the driver's awakening direction (specifically, the absolute value of the numerical value indicating the driver's awakening direction).

[0098] The alertness estimation unit 44 may be configured to perform at least one of the following processes when the driver's alertness direction estimated by the alertness direction estimation unit 43 is a "decreasing direction" so that the "driver's alertness" in the first alertness estimation process is more likely to decrease: a process of decreasing the "alertness index value" in the first alertness estimation process by a predetermined amount, a process of increasing the "alertness threshold" in the first alertness estimation process by a predetermined amount, and a process of decreasing the "driver's alertness" in the first alertness estimation process by a predetermined amount.

[0099] In addition, when the driver's wakefulness direction estimated by the wakefulness direction estimation unit 43 is a "recovery direction," the wakefulness estimation unit 44 may be configured to perform at least one of the following processes: increasing the "wakefulness index value" in the first wakefulness estimation process by a predetermined amount; decreasing the "wakefulness threshold" in the first wakefulness estimation process by a predetermined amount; and increasing the "driver's wakefulness" in the first wakefulness estimation process by a predetermined amount, so that the "driver's wakefulness" in the first wakefulness estimation process is more likely to recover (increase).

[0100] Furthermore, when the "second awakening level estimation process" is performed in the awakening level estimation unit 44, the awakening level estimation unit 44 may be configured as follows: The following predetermined amount (correction amount) may be set to increase according to an increase in the magnitude of the driver's awakening direction (specifically, the absolute value of the numerical value indicating the driver's awakening direction).

[0101] The alertness estimation unit 44 may be configured to perform at least one of the following processes when the driver's alertness direction estimated by the alertness direction estimation unit 43 is a "decreasing direction" so that the "driver's alertness" in the second alertness estimation process is more likely to decrease: increasing the "alertness index value" in the second alertness estimation process by a predetermined amount; decreasing the "alertness threshold" in the second alertness estimation process by a predetermined amount; and decreasing the "driver's alertness" in the second alertness estimation process by a predetermined amount.

[0102] In addition, when the driver's wakefulness direction estimated by the wakefulness direction estimation unit 43 is a "recovery direction," the wakefulness estimation unit 44 may be configured to perform at least one of the following processes: a process of decreasing the "wakefulness index value" in the second wakefulness estimation process by a predetermined amount; a process of increasing the "wakefulness threshold" in the second wakefulness estimation process by a predetermined amount; and a process of increasing the "driver's wakefulness" in the second wakefulness estimation process by a predetermined amount, so that the "driver's wakefulness" in the second wakefulness estimation process is more likely to recover (increase).

[0103] The arousal index value and arousal threshold value can be index values ​​and threshold values ​​in well-known techniques (techniques for estimating the driver's arousal level). In this example, the arousal index value is the "driver's eye openness level" derived based on an image (an image including the driver's eyes) obtained by the in-vehicle camera 12, and the arousal threshold value is a threshold value related to the driver's eye openness level (for example, the minimum value of the driver's eye openness level at which the driver is able to drive the vehicle normally). The greater the driver's eye openness level relative to the threshold value, the higher the driver's arousal level estimated in the arousal level estimation process.

[0104] <Step S21> Next, the response control unit 47 determines whether or not it is necessary to execute response control based on the driver's wakefulness estimated in step S20. If it is necessary to execute response control, the process of step S22 is performed; if not, the process ends.

[0105] In this example, the response control unit 47 determines that response control needs to be performed when the driver's alertness estimated in step S20 is below a predetermined execution threshold. For example, the execution threshold is set to a lower limit of the driver's alertness at which the driver is deemed to be able to drive the vehicle normally.

[0106] <Step S22: Response Control Step> When it is necessary to execute countermeasure control, countermeasure control unit 47 executes the countermeasure control. In this example, the countermeasure control by countermeasure control unit 47 includes awakening control by awakening control unit 45 and evacuation control by traveling control unit 46. When it is necessary to execute countermeasure control, awakening control unit 45 and traveling control unit 46 perform the following controls.

[0107] In the awakening control, the awakening control unit 45 performs at least one of the following processes: outputting an awakening sound to the speaker 21 to awaken the driver; displaying an awakening image to awaken the driver on the display 22; causing the air conditioner 23 to blow cool air to awaken the driver; and causing the vibration device 24 to generate vibrations to awaken the driver. Examples of the awakening sound include a warning sound notifying the driver that the level of awakening is decreasing, and a resting sound urging the driver to take a break. Examples of the awakening image include a warning image notifying the driver that the level of awakening is decreasing, and a resting image urging the driver to take a break.

[0108] In the evacuation control, the traveling control unit 46 controls the traveling of the vehicle so that the vehicle evacuates to a safety area. The safety area is an area where the vehicle can be stopped safely, such as a road shoulder. Specifically, the traveling control unit 46 identifies the safety area based on various information obtained by the information acquisition device 15, and determines an evacuation route, which is a traveling route that sets the safety area as a target stopping position, as a target route. Then, the traveling control unit 46 controls the actuator 20 so that the vehicle travels along the evacuation route, which is the target route, and stops in the safety area.

[0109] In the wakefulness control, the wakefulness control unit 45 may control one or more wakefulness devices 25 according to the wakefulness level of the driver so that the degree to which the wakefulness level of the driver is restored (increased) increases as the wakefulness level of the driver decreases. For example, the wakefulness control unit 45 may control multiple wakefulness devices 25 according to the wakefulness level of the driver so that the number of wakefulness devices 25 that perform an operation to restore the wakefulness level of the driver increases as the wakefulness level of the driver decreases.

[0110] Furthermore, in the wakefulness control, the wakefulness control unit 45 may control one or more wakefulness devices 25 according to the wakefulness level of the driver so that the intensity of the action of the wakefulness device 25 (the effect of restoring the wakefulness level of the driver) increases as the wakefulness level of the driver decreases. For example, the speaker 21 may be controlled so that the wakefulness sound output from the speaker 21 increases as the wakefulness level of the driver decreases. The display 22 may be controlled so that the wakefulness image displayed on the display 22 becomes more noticeable as the wakefulness level of the driver decreases. The air conditioner 23 may be controlled so that the temperature of the cool air blown out from the air conditioner 23 decreases (or the volume of the cool air increases) as the wakefulness level of the driver decreases. The vibration device 24 may be controlled so that the vibration applied to the driver by the vibration device 24 increases as the wakefulness level of the driver decreases.

[0111] Furthermore, alertness control and evacuation control may be selectively performed depending on the alertness of the driver. For example, the response control unit 47 (specifically, the alertness control unit 45 and the driving control unit 46) may be configured to perform alertness control when the alertness of the driver falls below the execution threshold but not below a safety threshold (a value lower than the execution threshold), and to perform evacuation control when the alertness of the driver falls below the safety threshold. For example, the safety threshold is set to a lower limit of the alertness of the driver that is deemed to be capable of restoring the alertness of the driver and returning the driver to "a state in which the driver can normally drive the vehicle."

[0112] [Effects of the embodiment] As described above, in the in-vehicle system 10 of this embodiment, the first derivation unit 41 derives a first index value corresponding to the driver's visual stimulus based on an image of the area ahead of the vehicle acquired by the exterior camera 11 (image capture unit). The second derivation unit 42 derives a second index value corresponding to a relatively strong visual stimulus among the driver's visual stimuli based on a comparison between the first index value and a predetermined stimulus threshold. The arousal direction estimation unit 43 estimates the arousal direction, which is the direction of change in the driver's arousal level, based on the correlation between the temporal change in the first index value and the temporal change in the second index value.

[0113] In the above configuration, the first index value is a value corresponding to the visual stimulus, the second index value is a value corresponding to the perceptual stimulus (a relatively strong visual stimulus among the visual stimuli), and the correlation between the temporal change in the first index value and the temporal change in the second index value is a value corresponding to the arousal stimulus. Then, the arousal direction of the vehicle driver can be estimated based on the correlation between the temporal change in the first index value and the temporal change in the second index value (the value corresponding to the arousal stimulus).

[0114] In addition, in the embodiment of the in-vehicle system 10, the awakening direction estimation unit 43 estimates that the driver's awakening direction is in a downward direction when the correlation coefficient indicating the correlation between the temporal change in the first index value and the temporal change in the second index value is below a predetermined correlation threshold, and estimates that the driver's awakening direction is in a recovery direction when the correlation coefficient is above the correlation threshold.

[0115] With the above configuration, it is possible to estimate whether the driver's alertness is decreasing or recovering.

[0116] In addition, in the embodiment of the in-vehicle system 10, the alertness estimation unit 44 estimates the driver's alertness based on the image obtained by the in-vehicle camera 12 (information used to estimate the driver's alertness acquired by the information acquisition unit) and the driver's alertness direction estimated by the alertness direction estimation unit 43.

[0117] In the above configuration, the driver's wakefulness level can be estimated taking into consideration the driver's wakefulness direction, thereby improving the accuracy of estimating the driver's wakefulness level.

[0118] Furthermore, in the in-vehicle system 10 of the embodiment, the response control unit 47 performs response control in accordance with the driver's alertness state estimated by the control device 40 (estimation system). The response control includes at least one of alertness control for restoring the driver's alertness and evacuation control for controlling the vehicle's driving so that the vehicle evacuates to a safe area. In this example, the alertness control is performed by the alertness control unit 45, and the evacuation control is performed by the driving control unit 46.

[0119] In the above configuration, the response control can be performed taking into consideration the degree to which the driver is awake, so that the response control can be performed appropriately.

[0120] (Other embodiments) In the above description, the following configuration or processing may be performed.

[0121] An example has been given in which the direction and magnitude of the driver's arousal direction estimated by the control device 40 (estimation system) are expressed by "the sign and absolute value of a number indicating the driver's arousal direction," but this is not limited to this.

[0122] For example, instead of both the direction and magnitude of the driver's arousal direction, only the direction of the driver's arousal direction may be represented by a flag. Specifically, the arousal direction estimation unit 43 may be configured to set a flag indicating that the driver's arousal direction is a "maintenance direction" in the process of step S15 (a process performed when the correlation coefficient is equal to the correlation threshold), to set a flag indicating that the driver's arousal direction is a "decline direction" in the process of step S17 (a process performed when the correlation coefficient exceeds the correlation threshold), and to set a flag indicating that the driver's arousal direction is a "recovery direction" in the process of step S18 (a process performed when the correlation coefficient does not exceed the correlation threshold). In this example, setting a flag is synonymous with estimating the driver's arousal direction.

[0123] Although the example has been given in which response control is performed in accordance with the driver's level of alertness estimated by the control device 40 (estimation system), the present invention is not limited to this. For example, response control may be performed in accordance with the driver's direction of alertness estimated by the control device 40 (estimation system). The "driver's state of alertness" in the "response control in accordance with the driver's state of alertness estimated by the control device 40 (estimation system)" may be the driver's level of alertness estimated in consideration of the driver's direction of alertness, or may be the driver's direction of alertness.

[0124] The awakening device 25 that performs an operation to awaken the driver is not limited to the speaker 21, the display 22, the air conditioning device 23, and the vibration device 24. For example, the awakening device 25 may include a lamp that emits light to awaken the driver, a buzzer that outputs sound to awaken the driver, or the like.

[0125] The information used to estimate the alertness level of the vehicle driver is not limited to images acquired by the in-vehicle camera 12. The information acquisition unit that acquires the information used to estimate the alertness level of the vehicle driver is not limited to the in-vehicle camera 12. For example, the information used to estimate the alertness level of the vehicle driver may include the driver's heart rate. The information acquisition unit may include a biosensor that detects the driver's heart rate.

[0126] The arousal index value corresponding to the driver's arousal level is not limited to the degree of eye opening of the driver. For example, the arousal index value may include the distance between predetermined parts of the driver's face, the orientation and position of the driver's face, the dominance of the driver's parasympathetic nervous system, and other well-known parameters (various parameters used to estimate the driver's arousal level). Furthermore, multiple arousal index values ​​may be used in the arousal level estimation process.

[0127] The components constituting the estimation system (specifically, the first derivation unit 41, the second derivation unit 42, the arousal direction estimation unit 43, and the arousal level estimation unit 44) may be integrated into one control device 40 (information processing device) or may be distributed among multiple control devices. The same applies to the components constituting the control system.

[0128] The above embodiments may be combined as appropriate. The above embodiments are essentially preferred examples and are not intended to limit the scope of the technology disclosed herein, its applications, or its uses. [Industrial Applicability]

[0129] As described above, the technology disclosed herein is useful as a technology for estimating the wakefulness state of a vehicle driver. [Explanation of symbols]

[0130] 10 In-Vehicle Systems 11. Outside camera (imaging unit) 12 In-car camera (information acquisition unit) 15 Information acquisition equipment 20 Actuator 21 Speaker 22 Display 23 Air conditioner 24 Vibration device 25 Awakening Device 30 Storage device 40 Control device (estimation system, control system) 41 First derivation part 42 Second derivation part 43 Awakening direction estimation unit 44 Arousal level estimation unit 45 Awakening Control Unit 46 Travel control unit 47 Response control section

Claims

1. An estimation system for estimating the wakefulness state of a vehicle driver, comprising: an imaging unit that captures an image of the area in front of the vehicle to obtain an image of the area in front of the vehicle; a first derivation unit that derives a first index value corresponding to a visual stimulus of the driver based on an image of a front of the vehicle obtained by the imaging unit; a second derivation unit that derives a second index value corresponding to a relatively strong visual stimulus among the visual stimuli of the driver based on a result of comparing the first index value with a predetermined stimulus threshold; an arousal direction estimation unit that estimates an arousal direction, which is a direction of change in the driver's arousal level, based on a correlation between a temporal change in the first index value and a temporal change in the second index value. Estimation system.

2. 2. The estimation system of claim 1, The awakening direction estimation unit estimates that the driver's awakening direction is a downward direction when a correlation coefficient indicating a correlation between the temporal change of the first index value and the temporal change of the second index value is below a predetermined correlation threshold, and estimates that the driver's awakening direction is a recovery direction when the correlation coefficient is above the correlation threshold. Estimation system.

3. 2. The estimation system of claim 1, an information acquisition unit that acquires information used to estimate the driver's wakefulness; an arousal level estimation unit that estimates the arousal level of the driver based on the information acquired by the information acquisition unit and the arousal direction of the driver estimated by the arousal direction estimation unit; Estimation system.

4. The estimation system according to any one of claims 1 to 3; a response control unit that performs response control in accordance with the wakefulness state of the driver estimated by the estimation system, The countermeasure control includes at least one of an awakening control for restoring the awakening level of the driver, and an escape control for controlling the traveling of the vehicle so that the vehicle escapes to a safety area. Control system.

5. A method for estimating a wakefulness state of a vehicle driver, comprising: a first derivation step of deriving a first index value corresponding to a visual stimulus of the driver based on an image of a front of the vehicle obtained by an imaging unit that images a front of the vehicle; a second derivation step of deriving a second index value corresponding to a relatively strong visual stimulus among the visual stimuli of the driver based on a result of comparison between the first index value and a predetermined stimulus threshold; and estimating an arousal direction, which is a direction of change in the driver's arousal level, based on a correlation between a temporal change in the first index value and a temporal change in the second index value. Estimation method.

6. The estimation method of claim 5; a countermeasure control step of performing a countermeasure control in accordance with the wakefulness state of the driver estimated by the estimation method, The countermeasure control includes at least one of an awakening control for restoring the awakening level of the driver, and an escape control for controlling the traveling of the vehicle so that the vehicle escapes to a safety area. Control method.

Images