Alertness detection device, driver-assistance system, and alertness detection method

Abstract

An alertness detection device according to the present invention includes an alertness acquisition unit (131) that acquires alertness of a driver calculated from a facial image including a face of the driver of a vehicle, a viewing direction acquisition unit (132) that acquires a viewing direction including a line-of-sight direction or a facial direction of the driver calculated from the facial image, a vicinity information acquisition unit (133) that acquires vicinity information indicating a situation in a vicinity from a vicinity detection device that detects the situation in the vicinity of the vehicle, an attention level estimation unit (134) that estimates an attention level of the driver in the situation in the vicinity on the basis of the viewing direction that is acquired and the vicinity information that is acquired, and a correction unit (135) that calculates a corrected alertness by correcting the alertness that is acquired using the attention level that is estimated, and outputs the corrected alertness that is calculated.

Description

Arousal level detection device, driving assistance system, and arousal level detection method

[0001] The present disclosure relates to wakefulness detection technology.

[0002] Patent Document 1 discloses a technology that measures the eye opening degree and the distance between the eyebrows and the eyes from a facial image of a vehicle driver, thereby detecting the driver's level of drowsiness in stages (paragraphs 0045 to 0047, Figure 3, etc. of Patent Document 1).

[0003] JP 2008-220424 A

[0004] However, the technology described in Patent Document 1 detects the alertness level from direct information obtained by directly observing the driver's state, and therefore may erroneously estimate the alertness level. For example, at a stage where the alertness level is significantly reduced, characteristics such as long-term eye closure or body (face) tilt are commonly observed in many drivers, so the alertness level can be determined quite accurately by direct observation. However, at an early stage of reduced alertness before the stage where the alertness level is significantly reduced, there are large individual differences in how characteristics such as eye blinking or the degree to which the eyes are opened and closed, making it difficult to accurately detect the alertness level by direct observation.

[0005] The present disclosure was made in response to the recognition of such a problem, and aims to provide a wakefulness detection technique that can detect the wakefulness level more accurately.

[0006] One aspect of an alertness detection device according to an embodiment of the present disclosure includes an alertness acquisition unit that acquires an alertness of a driver of a vehicle calculated from a facial image including the face of the driver; a gaze direction acquisition unit that acquires a gaze direction including a gaze direction or a facial orientation of the driver calculated from the facial image; a peripheral information acquisition unit that acquires peripheral information indicating the peripheral situation from a peripheral detection device that detects the peripheral situation of the vehicle; an attention level estimation unit that estimates an attention level of the driver in the peripheral situation based on the acquired gaze direction and the acquired peripheral information; and a correction unit that corrects the acquired alertness using the estimated attention level to calculate a corrected alertness and outputs the calculated corrected alertness.

[0007] According to the wakefulness detection device according to the embodiment of the present disclosure, the wakefulness can be detected more accurately.

[0008] Fig. 1 is a block diagram showing an example of the configuration of an alertness detection device and a driving assistance system according to embodiment 1. Fig. 2 is a diagram showing the change in alertness over time until falling asleep. Fig. 3 is a diagram showing an example of the configuration of the hardware of the alertness detection device. Fig. 4 is a diagram showing an example of the configuration of the hardware of the alertness detection device. Fig. 5 is a flowchart of a method for detecting alertness according to embodiment 1. Fig. 6 is a block diagram showing an example of the configuration of an alertness detection device and a driving assistance system according to embodiment 2. Fig. 7 is a flowchart of a method for detecting alertness according to embodiment 2. Fig. 8 is a diagram showing the configuration of a modified example.

[0009] Various embodiments of the present disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are designated by identical or similar reference numerals, and redundant explanations of such parts will be omitted. In addition, in this disclosure, the term "or" is used to mean an inclusive logical OR unless otherwise specified.

[0010] Embodiment 1. <Configuration> Referring to FIG. 1 , an alertness detection device 13 and a driving assistance system 10A according to embodiment 1 of the present disclosure will be described. The driving assistance system 10A according to embodiment 1 is a system that is mounted on a vehicle (not shown) and assists the driver of the vehicle in driving. To assist the driver in driving, the driving assistance system 10A detects the driver's alertness more accurately than conventional technology. In particular, the driving assistance system 10A more accurately detects the alertness of the driver in the early stage of a process in which the driver's alertness decreases, or a decrease in alertness in the early stage.

[0011] To achieve this objective, the driving assistance system 10A is connected to an imaging device 20 as shown in FIG. 1. The imaging device 20 captures images of an area including the driver's face (hereinafter simply referred to as "face images") at a frame rate of, for example, 30 fps (frames per second). The driving assistance system 10A receives time-series data of the face images captured by the imaging device 20. As shown in FIG. 1, the driving assistance system 10A also includes a periphery detection device 11, a driver state estimation device 12, and an alertness detection device 13. The driving assistance system 10A is configured as a system including the driver state estimation device 12 and the alertness detection device 13, and the periphery detection device 11 may be provided as an external device to the driving assistance system 10A.

[0012] (Periphery Detection Device) The periphery detection device 11 is a device that detects the situation around the vehicle (hereinafter simply referred to as "vehicle V") in which the driving assistance system 10A is installed. Examples of the periphery detection device 11 include an image sensor, a Lidar (Light Detection and Ranging), or a millimeter-wave sensor. For example, the periphery detection device 11 detects objects present around the vehicle V, such as moving objects, obstacles, traffic signs, traffic lights, fallen objects on the road, and the shape of the lane ahead (road shape). Examples of moving objects include nearby vehicles, bicycles, and pedestrians. Examples of obstacles include curbs and guardrails. The periphery detection device 11 may detect not only the position of an object but also information such as the object's speed, relative speed, degree of change, type, or shape. The periphery detection device 11 supplies information indicating the detected surrounding situation to the awakening level detection device 13 as peripheral information.

[0013] (Driver State Estimation Device) The driver state estimation device 12 is a device that estimates the driver's alertness level, which indicates the driver's level of alertness, and the driver's gaze direction or facial orientation, by performing image processing on the driver's facial image received from the imaging device 20. Since the technology for detecting the driver's gaze direction or facial orientation is known, detailed description thereof will be omitted. In the present disclosure, the gaze direction or facial orientation will be collectively referred to as the viewing direction.

[0014] The driver state estimating device 12 estimates the level of alertness from factors such as the direction of the driver's gaze, blinking, facial expression, poor posture, etc. This will be explained in more detail below.

[0015] The driver state estimation device 12 estimates the level of alertness based on, for example, the following indices. Furthermore, these indices may be arbitrarily combined to comprehensively estimate the level of alertness. For example, the indices to be combined may be linked using an AND condition to estimate the level of alertness. Furthermore, these indices are merely examples, and other indices or contact-type physiological measurement methods may also be used.

[0016] Index 1) Driver's eye opening degree f １ (t) The smaller the current eye opening degree is compared to the eye opening degree when the driver was awake, the lower the level of wakefulness is calculated. For example, the average eye opening degree over one minute may be used as the eye opening degree. The eye opening degree when the driver was awake is stored in a memory (not shown) and compared with the current eye opening degree.

[0017] Index 2) Percentage of blinks with long eye closure times (f) ２ (t) For example, if the number of blinks with a long eye-closure time (e.g., blinks with an eye-closure time of 0.5 seconds or more) among the number of blinks Ntotal over a 5-minute period is Nlong, then the larger the value of Nlong / Ntotal, the lower the level of alertness is determined to be.

[0018] Indicator 3) Gaze pattern f ３ (t) The less gaze attention or variation in gaze while driving, the lower the level of wakefulness is determined to be. In the present disclosure, gaze attention is expressed by a time series of gaze and refers to a gaze transition pattern. When determining the viewing direction based on facial direction instead of gaze direction, a facial direction transition pattern expressed by a time series of facial direction may be used instead of gaze attention. The same applies below. In the present disclosure, the term "gaze direction pattern" is used as a general term to refer to the gaze or facial direction transition pattern.

[0019] Indicator 4) Driver's facial expression ４ (t) The driver's level of wakefulness is estimated from the sleepy facial expression of the driver. A sleepy facial expression is determined by the occurrence of blinking, grimacing, or a blank expression without any change in facial expression.

[0020] Indicator 5) Yawning or increased alertness ５ (t) Determined by frequent yawning or self-stimulating behavior to increase alertness.

[0021] Indicator 6) Facial or gaze direction (f) ６ (t) The level of arousal is estimated from the fact that the face turns downward, the gaze continues to turn downward, or the face nods downward and then turns upward again frequently.

[0022] Indicator 7) Poor posture ７ (t) The level of alertness is estimated based on the degree to which the body sways back and forth and side to side, or the driver is lying face down and not in a normal driving position.

[0023] FIG. 2 shows an example of how the level of alertness D(t) changes until the driver falls asleep at the wheel. However, the changes in the driver's state before falling asleep vary from person to person, and the level of decreased alertness in the initial stage varies greatly from person to person. The initial stage refers to a stage belonging to alertness level DL4 or DL3 in the following description. In other words, the initial stage refers to a stage where the level of alertness is between half the maximum level and below the lower limit of the fully alert state, or a stage before DL2 where a steep decrease in alertness is observed.

[0024] The level of alertness fluctuates over time and gradually decreases due to the influence of various factors, such as the surrounding conditions of the vehicle V, the state of the interior of the vehicle V, or the driver's own psychology, physiology, or physical condition. The level of alertness continues to decrease, and the driver goes through states such as poor posture, frequent eye closure, and occurrence of microsleep, before falling into a weak drowsiness and progressing to complete drowsiness. Figure 2 shows typical states or behaviors that appear in a driver, with alertness levels divided into six levels. In this embodiment, the level of alertness will be described based on the case where it is divided into six levels as follows, but the level of alertness may be divided into multiple levels other than six levels.

[0025] ・Alertness level DL5 (fully awake): Rapid blinking, wide eye opening, no loss of posture. ・Alertness level DL4 (weakly decreased alertness): Noticeable blinking with long eye closure, slightly narrowed eye opening. ・Alertness level DL3 (moderately decreased alertness): Occurrence of long eye closure, eye opening becomes even narrower, and in some individuals, yawning and changes in facial expression occur. Driving operations become slightly slower. ・Alertness level DL2 (strongly decreased alertness): Repeated loss of posture and recovery, frequent long eye closure, frequent long-term eye closure (e.g., 3 seconds), occurrence of microsleep. Alertness level DL2 is a level at which driving is impaired, and recovery of alertness is necessary. ・Alertness level DL1 (weakly dozing): Loss of posture (change in posture), completely closed eyes, able to respond to alarms but unable to drive. ・Alertness level DL0 (completely dozing): Continued loss of posture, completely closed eyes, unable to respond to alarms, unable to drive. Normally, the level of alertness during driving does not drop to the alertness level DL0, but there are cases where the level of alertness drops to DL0 due to the influence of severe fatigue or alcohol consumption.

[0026] Among the examples of the indices for estimating the level of alertness described above, index 1 or index 2 is effective for estimating the overall level of alertness, index 3, index 4, or index 5 is effective for estimating a further decrease in alertness, including alertness level DL3, and index 6 or index 7 is effective for estimating a further decrease in alertness, including alertness level DL2. However, the relationship between the indices and the level of alertness varies from person to person. For example, some drivers fall asleep with their eyes half-open. For such drivers, the technology disclosed in Patent Document 1, which measures the degree of eye opening and the distance between the eyebrows and the eyes to determine whether they are asleep, may not be able to detect their drowsiness (see FIG. 3 of Patent Document 1).

[0027] The driver state estimation device 12 supplies the estimated alertness and viewing direction to the alertness detection device 13 .

[0028] (Arousal Level Detection Device) The alertness detection device 13 is a device that corrects the alertness level based on the alertness level and gaze direction received from the driver state estimation device 12 and peripheral information received from the periphery detection device 11. To perform such correction, the alertness detection device 13 includes, as shown in FIG. 1 , an alertness level acquisition unit 131, a viewing direction acquisition unit 132, a peripheral information acquisition unit 133, an attention level estimation unit 134, and a correction unit 135.

[0029] (Arousal Level Acquisition Unit) The awakening level acquisition unit 131 is a functional unit that acquires the awakening level from the driver state estimation device 12 and supplies the acquired awakening level to the correction unit 135.

[0030] (Viewing Direction Acquisition Unit) The viewing direction acquisition unit 132 is a functional unit that acquires the viewing direction from the driver state estimation device 12 and supplies the acquired viewing direction to the attention level estimation unit 134.

[0031] (Peripheral Information Acquisition Unit) The peripheral information acquisition unit 133 is a functional unit that acquires peripheral information from the periphery detection device 11 and supplies the acquired peripheral information to the attention level estimation unit 134 .

[0032] (Attention Level Estimation Unit) The attention level estimation unit 134 is a functional unit that estimates the degree of attention of the driver with respect to the situation outside the vehicle V, based on the viewing direction supplied from the viewing direction acquisition unit 132 and the peripheral information supplied from the peripheral information acquisition unit 133. The attention level estimation unit 134 supplies the estimated degree of attention to the correction unit 135.

[0033] The attention level estimation unit 134 calculates the level of attention using, for example, the following index. Here, it is assumed that an attention level AT(t) = 1 represents the highest level of attention, and AT(t) = 0 represents the lowest level of attention. The attention level AT(t) may be expressed as a binary value indicating whether or not the user is paying attention, or as a multi-value of three or more values, or as a continuous value. Note that the following indexes are examples, and other indexes may also be used.

[0034] Attention Level Index 1) Steady-State Gaze Attention The attention level estimation unit 134 estimates the attention level AT(t) by determining whether an appropriate gaze distribution is being made with respect to the situation around the vehicle V. For example, the attention level estimation unit 134 estimates the attention level AT(t)=1 if the gaze distribution for three seconds is a balanced visual recognition of all of the conditions of the road ahead, the vehicle ahead, surrounding vehicles, and the sidewalk.

[0035] The attention level estimation unit 134 reduces the value of the attention level AT(t) depending on the frequency with which features or moving objects that should be visible while driving were not visible. The lower limit of the attention level AT(t)=0 corresponds to a case where the line of sight does not move, such as when the line of sight changes by no more than 2 degrees.

[0036] Attention Level Indicator 2) Eye Attention When a Caution-Required Event Occurs When a change in the state around the vehicle V causes a predetermined caution-required event to occur, the attention level estimation unit 134 determines whether the gaze is appropriately directed toward the occurred caution-required event and estimates the attention level AT(t). For example, when a change in the state of a nearby moving object occurs, such as the appearance of a cutting-in vehicle or the braking of a preceding vehicle, the attention level estimation unit 134 determines whether the gaze is directed toward the moving object that has changed. The attention level estimation unit 134 estimates the attention level AT(t) as follows: if the gaze immediately recognizes the caution-required event through a saccade, AT(t) = 1; if the gaze recognizes the attention-required event with a time delay, AT(t) = 0.1 to 0.9 depending on the degree of the time delay (for example, AT(t) = 0.5 if there is a time delay of 1 second); and if the gaze does not recognize the event at all, AT(t) = 0.

[0037] In addition to the above, the level of caution AT(t) when a moving object such as a pedestrian or a bicycle suddenly appears near the roadway may also be estimated using a similar method.

[0038] In this way, the attention level estimation unit 134 may detect changes in moving objects present around the vehicle V and estimate the attention level AT(t) based on the reaction speed of the viewing direction to the detected change.

[0039] Attention Level Index 3) Gaze Attention to Changes in Objects Such as Signs, Indicator Lights, or Traffic Lights The attention level estimation unit 134 estimates the attention level AT(t) based on the speed of gaze reaction to changes in signs or indicator lights around the vehicle V. Examples of indicator lights include the on / off operation of turn signals or brake lights of a vehicle ahead. For example, when the color of an indicator light changes, the attention level estimation unit 134 estimates the attention level AT(t) based on the speed of gaze reaction from the time the color changes to the time the gaze is directed toward the indicator light after the change. The method described above in accordance with attention level index 2 may be used.

[0040] In this way, when the surrounding information includes a display change indicating an on / off change of the brake lights or turn signal lights of a vehicle ahead, or a change in traffic light signage, the attention level estimation unit 134 may estimate the attention level AT(t) based on the reaction speed of the viewing direction to the display change.

[0041] Attention Level Indicator 4) Gaze at Objects such as Signs, Billboards, or Traffic Lights Even if an object such as a sign, billboard, or traffic light does not change, the object such as a sign should be visually recognized. Therefore, the attention level estimation unit 134 estimates the value of the attention level AT(t) to be lower if a predetermined object such as a sign, billboard, or traffic light is not visually recognized.

[0042] Attention Level Indicator 5) Confirmation During Lane Changes: When the current visual pattern lacks visual elements included in a normal visual pattern when changing lanes, the attention level estimation unit 134 estimates a lower value for the level of attention AT(t). For example, if the normal visual pattern when changing lanes includes four visual elements—a rearview mirror, door mirrors, direct rearward visibility, and electronic mirror visibility—to confirm a vehicle behind when changing lanes, the attention level estimation unit 134 may estimate a lower value for the driver's attention level AT(t) when the current visual pattern does not include, for example, a direct rearward visibility. The normal visual pattern may be determined by an arbitrary functional unit of the alertness detection device 13 analyzing the visual elements of the driver when the alertness output from the driver state estimation device 12 is equal to or greater than a predetermined threshold. The determined normal visual pattern is stored in a memory (not shown), and the attention level estimation unit 134 acquires the normal visual pattern by referring to the memory.

[0043] In this way, the attention level estimation unit 134 may detect the driver's rearward checking visual pattern when the vehicle V changes lanes, and estimate the attention level AT(t) by comparing the detected visual pattern with the normal rearward checking visual pattern when changing lanes.

[0044] Attention Level Indicator 6) Checking Rear Vehicles When the current visual pattern lacks a visual element included in a normal visual pattern when checking a rear vehicle, the attention level estimation unit 134 estimates a lower value for the level of attention AT(t). For example, if the normal visual pattern for checking a rear vehicle includes four visual elements, namely, a rearview mirror, door mirrors, direct rearward visual confirmation, and electronic mirror visual confirmation, and the current visual pattern does not include, for example, a direct rearward visual confirmation element, the attention level estimation unit 134 may estimate a lower value for the driver's level of attention AT(t). Note that the normal visual pattern can be determined in a manner similar to that described with reference to attention level indicator 5.

[0045] Attention Level Indicator 7) Abnormal Location The attention level estimation unit 134 downwardly adjusts the value of the attention level AT(t) when a location that should be visually recognized to ensure safe driving, such as a road abnormality such as a depression in the road, a fallen object on the road, or an abnormality in the road wall, is not visually recognized. The location that should be visually recognized can be set, for example, by generating a saliency map from surrounding information.

[0046] Attention Level Indicator 8) Road Shape The attention level estimation unit 134 estimates the attention level AT(t) based on the degree to which the driver is visually recognizing the position in the driving lane of vehicle V, which vehicle V will arrive at in 2 to 4 seconds. When there are no other vehicles, 50% or more of the objects to be visually recognized should be located in the lane ahead. In particular, when the road shape ahead is curved, the attention level estimation unit 134 estimates the attention level AT(t) based on the degree to which the driver is visually recognizing the road ahead according to the road shape. Furthermore, a unique point ahead, such as a branch point or a merging point, may also be a position to be visually recognized.

[0047] For example, within a specified time, for example, 2 seconds, if the driver spends 50% or more of their time looking at the position in the driving lane of vehicle V, which vehicle V will arrive at in 2 to 4 seconds, then the attention level AT(t) = 1; if the looking time is less than 20%, then the attention level AT(t) = 0; if the looking time is between less than 50% and 20% or more, then the attention level AT(t) is a value greater than 0 and less than 1; the longer the looking time, the closer the attention level AT(t) is to 1, and the shorter the looking time, the closer the attention level AT(t) is to 0.

[0048] Furthermore, if there is another vehicle in the vicinity that requires attention, if the driver visually confirms the position of vehicle V in the driving lane that vehicle V will arrive in 2 to 4 seconds from now for 25% or more, attention level AT(t) = 1, if the visual confirmation time is less than 10%, attention level AT(t) = 0, and if the visual confirmation time is between less than 25% and 10% or more, attention level AT(t) is a value greater than 0 and less than 1, and the longer the visual confirmation time, the closer to 1 the attention level AT(t) is, and the shorter the visual confirmation time, the closer to 0 the attention level AT(t) is. The more complex the surrounding conditions, the smaller the proportion of the road surface ahead that the driver should visually confirm.

[0049] In this way, the attention level estimation unit 134 may estimate the attention level AT(t) according to the degree to which the acquired viewing direction views a position according to the road shape.

[0050] (Correction Unit) The correction unit 135 is a functional unit that corrects the level of alertness D(t) supplied from the alertness acquisition unit 131 using the level of attention AT(t) supplied from the attention level estimation unit 134, and outputs the corrected level of alertness as a corrected level of alertness DC(t). The function performed by the correction unit 135 can be expressed as a mathematical formula as shown in the following formula (1): DC(t)=g(D(t), AT(t)) (1)

[0051] Generally, it is appropriate that the greater the level of alertness D(t), the greater the level of attention AT(t). Therefore, in this embodiment, the level of alertness AT(t) is normalized to a value that matches the level of alertness D(t), and the level of alertness is corrected according to the difference between the level of alertness AT(t) and the level of alertness D(t). For example, the correction is made as shown in the following equation (2). The following equation (2) corrects the difference between the level of alertness AT(t) and the level of alertness D(t) by weighting it by 1 / 2. Corrected level of alertness DC(t) = level of alertness D(t) + {level of alertness AT(t) - level of alertness D(t)} / 2 (2)

[0052] The specific correction method is not limited to formula (2). For example, for the detection result of the alertness level DL4 or DL3 in Fig. 2, if the attention level AT(t) is appropriate for the detected alertness level, the alertness level D(t) may not be corrected, but if the attention level AT(t) is greater than the alertness level D(t), the alertness level may be corrected upward by one, and if the attention level AT(t) is smaller than the alertness level D(t), the alertness level may be corrected downward by one.

[0053] The levels of attention AT normalized in correspondence with the alertness level DL are, for example, as follows: Alertness level DL5: AT5; complete gaze; Alertness level DL4: AT4; somewhat incomplete gaze. At this level, there is a slight delay in visual recognition of signs or when a cautionary event occurs. Alertness level DL3: AT3; slow or sloppy gaze. At this level, there is a time delay in visual recognition, and less gaze attention to the surroundings. Alertness level DL2: AT2; frequent failure to move the gaze in response to changes in the surroundings. At this level, rearward checks, such as when changing lanes, are insufficient. Alertness level DL1 or DL0: AT1 or AT0. At this level, the driver is equivalent to falling asleep at the wheel, with their eyes closed.

[0054] Next, an example of the hardware configuration of the wakefulness detection device 13 will be described with reference to Figures 3A and 3B. Each function of the wakefulness detection device 13 is realized by a processing circuitry. The processing circuitry may be a dedicated processing circuit 100a as shown in Figure 3A, or a processor 100b as a computer that executes a program stored in a memory 100c as shown in Figure 3B.

[0055] When the processing circuitry is a dedicated processing circuit 100a, the dedicated processing circuit 100a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. The functions of the wakefulness detection device 13 may be realized by a plurality of separate processing circuits, or the functions of the wakefulness detection device 13 may be realized together by a single processing circuit.

[0056] When the processing circuitry is the processor 100b, the functions of the wakefulness detection device 13 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 100c. The processor 100b realizes the functions of the wakefulness detection device 13 by reading and executing the programs stored in the memory 100c. Here, examples of the memory 100c include non-volatile or volatile semiconductor memories such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

[0057] Note that some of the functions of the wakefulness detection device 13 may be realized by dedicated hardware, and other functions may be realized by software or firmware. In this way, the processing circuit can realize the functions of the wakefulness detection device 13 by hardware, software, firmware, or a combination of these.

[0058] <Operation> Next, the operation of the awakening level detection device 13 will be described with reference to Fig. 4. The flow in Fig. 4 is executed on the condition that driving of the vehicle V has started. Note that the operations of steps ST10 to ST12 may be executed in parallel.

[0059] (Step ST10) In step ST10, the alertness obtaining unit 131 obtains the driver's alertness D(t) from the driver state estimating device 12.

[0060] (Step ST11) In step ST11, the surrounding information acquisition unit 133 acquires surrounding information from the surrounding detection device 11.

[0061] (Step ST12) In step ST12, the viewing direction acquisition unit 132 acquires the viewing direction from the driver state estimation device 12.

[0062] (Step ST13) In step ST13, the attention level estimation unit 134 estimates the driver's attention level to the situation outside the vehicle V based on the viewing direction acquired by the viewing direction acquisition unit 132 and the surrounding information acquired by the surrounding information acquisition unit 133.

[0063] (Step ST14) In step ST14, the correction unit 135 determines whether the wakefulness D(t) acquired by the wakefulness acquisition unit 131 is equal to or greater than a predetermined threshold Dth1. The threshold Dth1 may be, for example, the lower limit of the highest wakefulness level. In the example of FIG. 2 , the threshold Dth1 may be 0.8, which is the lower limit of the wakefulness level DL5. If the wakefulness D(t) is equal to or greater than the threshold Dth1, the correction unit 135 does not correct the wakefulness D(t) and proceeds to step ST16. If the wakefulness D(t) is not equal to or greater than the threshold Dth1, the correction unit 135 proceeds to step ST15.

[0064] (Step ST15) In step ST15, the correction unit 135 corrects the alertness D(t) acquired by the alertness acquisition unit 131 based on the attention AT(t) acquired by the attention estimation unit 134, and outputs the corrected alertness as the corrected alertness DC(t).

[0065] 4 when the driving is completed, and repeats the processing of steps ST10 to ST16 while the driving is not completed. Note that the determination of whether the driving is completed may be made by a control unit (not shown) of the driving detection device 13.

[0066] As described above, the wakefulness detection device 13 corrects the obtained wakefulness level according to the driver's level of attention to the situation outside the vehicle. Therefore, it is possible to calculate the wakefulness level more accurately than conventional techniques that calculate the wakefulness level without considering the level of attention to the situation outside the vehicle. In particular, by detecting the wakefulness level more accurately in the early stage of changes in the wakefulness level, it is possible to improve the accuracy of detecting a decrease in wakefulness and improve the accuracy of correcting the wakefulness detection result to account for individual differences.

[0067] <Modification 1-1: Correction Timing 1> In the first embodiment, the alertness D(t) is corrected when the alertness D(t) is lower than a predetermined threshold Dth1, but the timing for determining whether to perform the correction is not limited to this and may be set based on various factors. For example, the correction in step ST15 may be performed periodically, for example, every 30 minutes, after the start of driving.

[0068] Furthermore, the correction of step ST15 may be performed depending on the driving conditions of the vehicle V, such as when the driving operation becomes slow or when meandering is detected. When performing such an operation, the alertness detection device 13 further includes a vehicle information acquisition unit (not shown) that acquires the driving conditions of the vehicle V, and the correction unit 135 accepts the driving conditions acquired from the vehicle information acquisition unit and performs the correction.

[0069] <Modification 1-2: Correction Timing 2> The correction unit 135 may perform correction when it determines that the reliability of the detection result of the wakefulness D(t) is low. For example, the correction unit 135 may correct the wakefulness D(t) when it detects that the wakefulness has changed suddenly in about one minute, when the results of multiple wakefulness detection indices are different, or when there is a long period of time during which the image processing results are invalid.

[0070] <Variant 1-3: Frequency of Correction Processing> In the first embodiment, the alertness D(t) is corrected when the alertness D(t) is lower than a predetermined threshold Dth1. However, once the alertness D(t) has been corrected, the alertness correction processing may not be performed as long as there is no change in the alertness D(t), or the correction processing may be performed again after a predetermined time, for example, 15 minutes.

[0071] Furthermore, the level of arousal may be corrected based on the results of multiple reactions rather than on the reaction to a single stimulus.

[0072] <Modification 1-4: Learning Individual Characteristics> The characteristics of eye gaze distribution while driving vary from person to person. Therefore, the eye gaze (viewing direction pattern) at a high alertness level DL5 may be machine-learned as an eye gaze with a high level of attention, and the learned eye gaze and the current eye gaze may be used to calculate the attention level AT(t).

[0073] During learning, the attention level estimation unit 134 acquires the alertness level D(t) from the alertness level acquisition unit 131, the viewing direction from the viewing direction acquisition unit 132, peripheral information from the peripheral information acquisition unit 133, and the attention level AT(t) from the attention level estimation unit 134, and extracts the gaze attitude, peripheral information, and attention level AT(t) at the time of alertness level DL5, thereby performing machine learning on the extracted gaze attitude as a gaze attitude with a high level of attention. The attention level estimation unit 134 stores the learning model generated by machine learning in a storage device (not shown). For example, the learning model can be constructed by combining a neural network that receives gaze direction data as input with a neural network that receives peripheral information as input, and a known machine learning method can be used.

[0074] During inference, the attention level estimation unit 134 acquires the viewing direction from the viewing direction acquisition unit 132 and the peripheral information from the peripheral information acquisition unit 133, inputs the acquired gaze direction and peripheral information into a learning model, and acquires the attention level AT(t) from the learning model.

[0075] In this way, when the driver's level of alertness is high, the attention level estimation unit 134 performs machine learning to generate a learning model by treating the driver's viewing direction pattern with respect to the surrounding situation as a viewing direction pattern with a high level of attention, and can estimate the driver's current level of attention from the current line of sight by referring to the generated learning model.

[0076] <Modification 1-5: Driver Monitoring> The alertness acquisition unit 131 may acquire a facial image of the driver from the imaging device 20 and calculate the alertness by performing image processing on the acquired facial image, instead of acquiring the alertness from the driver state estimation device 12. The method for calculating the alertness can be the method described above in accordance with the driver state estimation device 12.

[0077] <Modification 1-6: Autonomous Driving Level> In the first embodiment, no reference is made to the relationship between eye contact during autonomous driving and the level of attention. However, correction may be made according to the level of autonomous driving. Specifically, in the case of autonomous driving level 1 or higher, the vehicle performs part of the driving, reducing the driver's driving burden and allowing the driver to look with a somewhat relaxed attitude. Therefore, even if the detected level of attention is low, there is no need to excessively correct the level of alertness downward. Therefore, the degree of correction of the level of alertness based on the level of attention may be adjusted according to the level of autonomous driving. In Modification 6, the corrected level of alertness DC(t) is generally expressed by the following equation (2). The value of the autonomous driving level n is obtained by the correction unit 135 of the alertness detection device 13 from an autonomous driving control device (not shown). DC(t) = f(D(t), AT(t), autonomous driving level n) (2)

[0078] At autonomous driving level 4 and above, the driver is completely freed from driving operations, so the calculation of the corrected alertness DC(t) may be performed by dividing the calculation into the following cases: - For autonomous driving level 0 (n = 0): DC(t) = g(D(t), AT(t)) (1) - For autonomous driving levels 1 to 3 (n = 1, 2, or 3): DC(t) = D(t) + {α ｎ AT(t)-D(t)} / 2 (3) For example, α １ = 1.1, α ２ = 1.2, α ３ = 1.3, where α ｎ If AT(t)≧1, then α ｎCorrect to AT(t) = 1. In the case of autonomous driving level 4 or 5 (n = 4 or 5): DC(t) = D(t) (4) In other words, if the autonomous driving level is 4 or 5, no correction is made.

[0079] In this way, the correction unit 135 may calculate the corrected alertness DC(t) according to the autonomous driving level n of the vehicle V.

[0080] Furthermore, the gaze and attention level of an individual may be learned according to the autonomous driving level. As in Modification 4, the gaze at the time of the alertness level DL5, which has a high level of alertness, may be learned as the gaze at a high level of attention, and the attention level AT(t) may be calculated using the learned gaze.

[0081] Second Embodiment <Configuration> Next, a driving assistance system 10B according to a second embodiment of the present disclosure will be described with reference to Fig. 5. The driving assistance system 10B is a system that specifically performs driving assistance using the corrected alertness DC output by the alertness detection device 13. Of the elements included in the driving assistance system 10B, duplicated descriptions of elements that are similar to those of the driving assistance system 10A will be omitted.

[0082] 5, the driving assistance system 10B includes a driving assistance unit 14 as an additional element to the driving assistance system 10A. The driving assistance system 10B also includes an automatic driving control device 15 or an alertness enhancement device 16.

[0083] (Driving Support Unit) The driving support unit 14 is a functional unit that controls the automatic driving control device 15 or the alertness enhancing device 16 based on the corrected alertness DC output by the alertness detection device 13 to provide driving support.

[0084] (Automatic Driving Control Device) The automatic driving control device 15 is a device that executes automatic driving control of level 1 or higher. The automatic driving control includes adaptive cruise control (ACC), autonomous emergency braking (AEB), lane keeping assist system (LKAS), or fully automatic driving.

[0085] (Awakening Level Enhancement Device) The alertness enhancement device 16 is a device for enhancing the driver's alertness. The alertness enhancement device 16 enhances the driver's alertness by, for example, an audio warning, a tactile stimulation on the seat, airflow, a vibration stimulation on the steering wheel, opening and closing of a window, or an olfactory stimulation.

[0086] The driving assistance unit 14, the automatic driving control device 15, and the alertness enhancement device 16 can all be realized by the hardware configuration of FIG. 3A or FIG. 3B.

[0087] <Operation> Next, the operation of the driving assistance system 10B will be described with reference to Fig. 6. The flowchart in Fig. 6 additionally includes step ST20 compared to the flowchart in Fig. 4. Steps ST10 to ST15 have already been described, so only step ST20 will be described here.

[0088] (Step ST20) In step ST20, the driving assistance unit 14 performs driving assistance according to the corrected alertness DC. Driving assistance is performed, for example, as follows: Corrected alertness = DL5: No assistance Corrected alertness = DL4: The alertness enhancement device 16 is used to enhance the driver's alertness. Corrected alertness = DL3: The alertness enhancement device 16 is used to enhance the driver's alertness, and the automatic driving control device 15 activates ACC, AEB, or LKAS. Corrected alertness = DL2: A warning is issued to the driver, and driving assistance of automatic driving level 3 or higher is performed. Corrected alertness = DL1: Fully automatic driving control is performed, and the vehicle is parked or stopped in an appropriate location. Corrected alertness = DL0: Fully automatic driving control is performed, and the vehicle is parked or stopped in an appropriate location.

[0089] According to the driving assistance system 10B of the second embodiment, it is possible to provide appropriate driving assistance in accordance with the corrected alertness level.

[0090] <Modification 2-1: Recording Device> The driving assistance system 10B according to the second embodiment may be modified to become a driving assistance system 10C as shown in Fig. 7. As shown in Fig. 7, the awakening level detection device 13C included in the driving assistance system 10C additionally includes a position information acquisition unit 136 and a recording control unit 137.

[0091] The location information acquisition unit 136 acquires location information of the vehicle V from an ECU (not shown) of the vehicle V, and supplies the acquired location information to the recording control unit 137 .

[0092] The recording control unit 137 acquires a facial image from the imaging device 20, acquires the level of alertness from the driver state estimation device 12, acquires surrounding information including footage outside the vehicle from the surrounding detection device 11, acquires the level of attention from the attention level estimation unit 134, and acquires the corrected level of alertness from the correction unit 135.

[0093] By configuring the alertness detection device 13C in this manner, the recording control unit 137 may record the surrounding situation at the time of decreased alertness, the alertness level, the corrected alertness level, the attention level, or video of the inside and outside of the vehicle in association with the location information of the vehicle V in the recording device 17. In Fig. 7, the recording device 17 is illustrated as a device provided in the driving assistance system 10C, but the recording device 17 may also be provided outside the vehicle.

[0094] This recorded information can be used to alert or understand the situation of those involved in driving. The people involved in driving can be, for example, the occupants including the driver of the vehicle V, predetermined people such as the driver's family, or companies that provide services using the vehicle V, such as taxi companies.

[0095] It can also be used to train alertness detection logic or attention detection logic on the server side or locally. For example, the alertness or attention level when appropriate eye contact is made according to the surrounding situation may be annotated as correct data.

[0096] The awakening level detection device 13 according to the first embodiment may also be modified in the same manner as the driving assistance system 10C. That is, the awakening level detection device 13 according to the first embodiment may be modified to include the position information acquisition unit 136 and the recording control unit 137.

[0097] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate.

[0098] The awakening level detection device of the present disclosure can be used as a device constituting a system for assisting a driver of a vehicle.

[0099] 10 (10A to 10C) driving assistance system, 11 surrounding detection device, 12 driver state estimation device, 13 (13C) alertness detection device, 14 driving assistance unit, 15 automatic driving control device, 16 alertness improvement device, 17 recording device, 20 imaging device, 100a processing circuit, 100b processor, 100c memory, 131 alertness acquisition unit, 132 viewing direction acquisition unit, 133 surrounding information acquisition unit, 134 attention level estimation unit, 135 correction unit, 136 position information acquisition unit, 137 recording control unit.

Claims

1. An alertness detection device comprising: an alertness acquisition unit that acquires the alertness of a vehicle driver calculated from a facial image including the face of the driver; a viewing direction acquisition unit that acquires the viewing direction including the line of sight or facial orientation of the driver calculated from the facial image; a surrounding information acquisition unit that acquires surrounding information indicating the surrounding situation from a surrounding detection device that detects the surrounding situation of the vehicle; an attention level estimation unit that estimates the driver's attention level in the surrounding situation based on the acquired viewing direction and the acquired surrounding information; and a correction unit that corrects the acquired alertness using the estimated attention level to calculate a corrected alertness and outputs the calculated corrected alertness.

2. The alertness detection device according to claim 1, wherein the attention level estimation unit detects changes in moving objects present around the vehicle and estimates the level of attention based on the speed of reaction of the direction of vision to the detected changes.

3. The alertness detection device of claim 1, wherein the attention level estimation unit estimates the level of attention based on the reaction speed of the viewing direction to a display change when the surrounding information includes a display change indicating an on / off change of the brake lights or blinker lights of a forward vehicle or a change in traffic light sign.

4. The alertness detection device of claim 1, wherein the attention level estimation unit detects the driver's rearward checking visual pattern when the vehicle changes lanes, and estimates the level of attention by comparing the detected visual pattern with the normal rearward checking visual pattern when changing lanes.

5. The alertness detection device of claim 1, wherein the surrounding information includes the shape of the road on which the vehicle is traveling, and the attention level estimation unit estimates the attention level based on the degree to which the acquired viewing direction views a position corresponding to the road shape.

6. The alertness detection device according to claim 1, wherein the correction unit calculates the corrected alertness according to an autonomous driving level of the vehicle.

7. The alertness detection device of claim 1, wherein, when the driver's alertness is high, the attention level estimation unit performs machine learning to generate a learning model by treating the driver's viewing direction pattern with respect to the surrounding situation as a viewing direction pattern with a high level of attention, and estimates the driver's current level of attention from the current viewing direction pattern by referring to the generated learning model.

8. The alertness detection device according to claim 1, further comprising: a location information acquisition unit that acquires location information of the vehicle; and a recording control unit that records the situation when correcting the acquired alertness in a recording device in association with the acquired location information.

9. A driving assistance system comprising: an alertness detection device according to any one of claims 1 to 8; and a driving assistance unit that controls an automatic driving control device or an alertness enhancement device based on the corrected alertness output from the alertness detection device.

10. A driving assistance system comprising: an alertness detection device according to any one of claims 1 to 8; and a driver state estimation device that detects the alertness of the driver and the direction of the driver's gaze.

11. A method for detecting a level of alertness performed by an alertness detection device having an alertness acquisition unit, a viewing direction acquisition unit, a peripheral information acquisition unit, an attention level estimation unit, and a correction unit, comprising: a step in which the alertness acquisition unit acquires the alertness of a driver of a vehicle calculated from a facial image including the face of the driver; a step in which the viewing direction acquisition unit acquires the viewing direction including the line of sight or facial orientation of the driver calculated from the facial image; a step in which the peripheral information acquisition unit acquires peripheral information indicating the peripheral situation from a peripheral detection device that detects the peripheral situation of the vehicle; a step in which the attention level estimation unit estimates the level of attention of the driver in the peripheral situation based on the acquired viewing direction and the acquired peripheral information; and a step in which the correction unit corrects the acquired alertness using the estimated attention level to calculate a corrected alertness, and outputs the calculated corrected alertness.

Images