Driving concentration level estimation device, driving concentration level estimation method, and program

Abstract

The purpose of the present invention is to estimate the driving concentration level of a driver, which changes moment to moment, by using blink frequency. The driving concentration level estimation device according to the disclosed technology comprises a blink frequency update unit and an estimation unit. With regions obtained by dividing a one- to three-dimensional acceleration space having been set as acceleration regions and the average and variance of the blinking frequency of a driver having powers of concentration in each acceleration region having been set as estimation reference values, the blink frequency update unit measures the blink frequency of the driver in association with the acceleration regions. The estimation unit estimates the driving concentration level of the driver using the estimation reference values and the measured blink frequency.

Description

Driving concentration level estimation device, driving concentration level estimation method, and program

[0001] The disclosed technology relates to a device that estimates the state of a person driving a vehicle or the like, and in particular to a device that estimates the degree of concentration on driving.

[0002] To estimate the state of a vehicle driver, indicators related to blinking, such as the number of blinks or blink clusters, the interval between blinks, the speed of eyelid movement, etc., are used. In these methods, blink indicators are sometimes evaluated on their own, and sometimes they are evaluated by roughly taking into account the driver's situation.

[0003] JM Cori et al., "Narrative review: Do spontaneous eye blink parameters provide a useful assessment of state drowsiness?", Sleep Medicine Reviews, 45(2019), 95-104.

[0004] According to Non-Patent Document 1, blink frequency, duration, eyelid movement speed, and eye-closure time ratio, as well as a combination of these, are used to estimate the driver's state. However, because blinking is a phenomenon that occurs only about 20 times per minute, the accumulation of information is slow, making it difficult to capture state changes in a short period of time when attempting to estimate the driver's internal state, which changes from moment to moment.

[0005] To solve the above problems, a driving concentration level estimation device according to the disclosed technology includes a blink frequency update unit and an estimation unit. Regions obtained by dividing one- to three-dimensional acceleration space are defined as acceleration regions, and the average and variance of the blink frequency at which a driver with concentration blinks in each acceleration region are used as estimation reference values. The blink frequency update unit measures the driver's blink frequency in association with the acceleration region. The estimation unit estimates the driver's level of concentration on driving using the estimation reference value and the measured blink frequency.

[0006] As described above, the disclosed technology simultaneously records the vehicle state and the blinking state, and the estimation unit estimates the driver's concentration on driving. The absence of blinking is also treated as information in combination with the vehicle state. As a result, compared to conventional technologies, information can be accumulated more efficiently, and the accuracy and resolution of driver state estimation can be improved.

[0007] 1 is a diagram for explaining the correlation between acceleration and blinking when driving a vehicle at high speed. A diagram for explaining the difference in the correlation between acceleration and blinking between fast driving and slow driving. A diagram for explaining the correlation between acceleration and blinking when driving on a wet road surface. A functional block diagram showing an example configuration of a driving concentration level estimation device and peripheral devices according to the first embodiment. A flowchart mainly explaining the operation of the blink frequency update unit 412. A flowchart explaining the operation of the estimation unit 414. A diagram for explaining regions of an acceleration diagram. A functional block diagram showing an example configuration of a driving concentration level estimation device and peripheral devices according to the second embodiment. A flowchart explaining the operation of an optimization unit 803. A diagram showing an example functional configuration of a computer.

[0008] Hereinafter, embodiments of the disclosed technology will be described in detail. Note that components having the same functions are assigned the same numbers, and duplicated descriptions will be omitted.

[0009] As a result of extensive research, the inventors have discovered a correlation between the driver's eye blinking and the vehicle's acceleration state. Below, we will first explain the correlation we discovered, and then explain an apparatus / method for estimating the driver's concentration level using the correlation.

[0010] [Correlation between Eye Blinks and Acceleration States] One method for analyzing the driving of vehicles traveling on land is to analyze the acceleration of the vehicle. A standard method involves attaching accelerometers in two directions, one in the direction of travel and the other in the left-right direction perpendicular to the direction of travel in a horizontal plane, and analyzing the acceleration when turning a corner. In this case, the driver observes the acceleration that decelerates in the direction opposite to the direction of travel as they step on the brakes to slow down to a speed that allows them to safely turn the corner, and then observes the acceleration in the left-right direction as they turn the steering wheel. When the direction of travel of the vehicle matches the direction it should be heading, they step on the accelerator and accelerate again to cruising speed.

[0011] The researchers behind the disclosed technology found that humans blink very little while decelerating just before or at the beginning of a corner, or while steering inside a corner (accelerating left or right), but blink more while accelerating in a straight line or during constant-velocity linear motion. They also found that this tendency is stronger in situations requiring greater concentration.

[0012] First, Figure 1 shows the relationship between acceleration and blinking when driving a vehicle at high speed (when concentration is thought to be high). (a-1) and (a-2) are plots of the vehicle's acceleration dwell time. The horizontal axis represents lateral acceleration, and the vertical axis represents acceleration in the direction of travel. The darker the color, the longer the vehicle was in that acceleration state. (a-1) is an acceleration dwell time diagram for the entire driving time, while (a-2) is an acceleration dwell time diagram extracted from (a-1) for simultaneously recorded blinking periods. Compared to (a-1), (a-2) shows a significant reduction in the number of darker areas in regions 1 and 2. This indicates that blinking is suppressed in region 1 (deceleration in the direction of travel) and region 2 (lateral acceleration, i.e., during a curve).

[0013] (b) was created from the data in (a-2), and is a histogram comparing the blink frequency (number of blinks / stay time) in area 3, which is set to 1.0, with the blink frequency in areas 1 and 2. It was confirmed that the blink frequency was significantly reduced in areas 1 and 2. This is also statistically significant.

[0014] Next, Figure 2 compares the acceleration dwell time graphs for fast and slow driving. (a-1) and (a-2) are for "fast driving" and are the same as Figure 1. (b-1) and (b-2) are for "slow driving." Comparing (a-2) and (b-2), in (b-2) the acceleration dwell time region (during blinking) extends to regions 1 and 2. This can also be seen from histograms (c-1) and (c-2), which compare the blink frequency in each region, as before. As such, when driving relatively slowly, concentration is thought to be lower than when driving "fast," and blinking tends to be more frequent even during deceleration and accelerating left and right (while cornering).

[0015] As mentioned above, drivers who are concentrating on driving significantly reduce blinking when decelerating before entering a corner or accelerating laterally while cornering. This tendency weakens when the driver is less focused. In other words, blinking during deceleration or cornering, when observing lateral acceleration, can be considered to indicate a weakening of concentration. Alternatively, if blinking is suppressed as expected, it can be assumed that the driver is maintaining an appropriate level of concentration.

[0016] <Influence of road surface conditions> The acceleration range associated with blinking becomes extremely small when it rains. This is shown in Figure 3. This is thought to be due to the fact that wet roads are slippery, and drivers unconsciously sense that high acceleration increases the risk of danger. For this reason, when investigating the relationship between acceleration and blinking, it is necessary to take into account whether the road surface is wet.

[0017] [First embodiment] Fig. 4 is a functional block diagram showing an example of the configuration of a driving concentration level estimation device and peripheral devices according to the first embodiment. A driver 401 is assumed to be driving a vehicle 403. The vehicle 403 is equipped with a camera 402, a wiper control unit 404, an acceleration sensor 405, and a driving concentration level estimation device 406. It is assumed that the image captured by the camera 402 and the measurement by the acceleration sensor 405 are synchronized. The acceleration sensor 405 performs measurements in at least two axes: the "direction of travel" of the vehicle and the "left-right direction" perpendicular to the direction of travel in a horizontal plane. It is assumed that wiper operation information can be acquired from the wiper control unit 404.

[0018] The concentration on driving estimation device 406 includes a blink detection unit 411, a blink frequency update unit 412, a wiper information processing unit 413, an estimation unit 414, and an output unit 415. Fig. 5 is a flowchart illustrating an example of the operation of the blink frequency update unit 412 in the concentration on driving estimation device 406. Fig. 6 is a flowchart illustrating an example of the operation of the estimation unit 414 in the concentration on driving estimation device 406. The concentration on driving estimation device according to the first embodiment will be described in detail below with reference to Figs. 4, 5, and 6.

[0019] <Updating Blink Frequency> The driving concentration level estimation device 406 acquires facial images of the driver 401 periodically captured by the camera 402 (step S501 in FIG. 5 ). The blink detection unit 411 detects blinks based on the facial images acquired by the camera 402. If a blink is detected (Yes in step S502), the blink frequency update unit 412 acquires the acceleration of the vehicle 403 at the same time as the blink detection from the acceleration sensor 405 (step S503).

[0020] The blink frequency update unit 412 determines the region to which the acceleration acquired from the acceleration sensor 405 belongs (step S504). ｘ [m / s ２ ], and the acceleration in the forward direction is a ｙ [m / s ２ ] and can be defined as follows: Area 1 a ｙ <-0.5 Area 2 (a ｙ ≧-0.5) and (|a ｘ |≧1.25) Area 3 (a ｙ ≧-0.5) and (|a ｘ |<1.25) Figure 7 illustrates the above acceleration regions. As mentioned above, region 1 indicates that the vehicle is decelerating, region 2 indicates that the vehicle is traveling around a curve, and region 3 indicates that the vehicle is accelerating in the direction of travel. As a result of the above, a set of data (blink data) is obtained for each blink, which is the time of the blink and the acceleration state of the vehicle.

[0021] The blink frequency update unit 412 holds the blink frequency (e.g., the number of blinks per minute) for a predetermined period (e.g., the past five minutes) for each of the acceleration regions 1, 2, and 3. The blink frequency update unit 412 updates the blink frequency for each acceleration region based on newly obtained blink data and data indicating that no blinks were obtained (No in step S502) (step S505).

[0022] The concentration-on-driving estimation device 406 detects the end of driving the vehicle by, for example, stopping the engine. If the end of driving is not detected (No in step S506), the concentration-on-driving estimation device 406 acquires the next facial image from the camera 402. If the end of driving is detected (Yes in step S506), the concentration-on-driving estimation device 406 ends the process.

[0023] <Estimation of Road Surface Condition> The wiper information processing unit 413 periodically acquires wiper operation information from the wiper control unit 404. Based on the acquired wiper operation information, the wiper information processing unit estimates whether the vehicle is traveling on a wet road or a dry road, and stores the latest result.

[0024] <Estimation of Concentration on Driving> The estimation unit 414 receives the blink frequencies f １ , f ２ , f ３ The estimation unit 414 also acquires road surface information from the wiper information processing unit 413. The estimation unit 414 estimates the average blink frequency μ Ｄ１ , μ Ｄ２ , μ Ｄ３ and standard deviation σ Ｄ１ , σ Ｄ２ , σ Ｄ３ In addition, when the driver is concentrating and the road surface is wet, the average blink frequency μ Ｗ１ , μ Ｗ２ , μ Ｗ３ and standard deviation σ Ｗ１ , σ Ｗ２ , σ Ｗ３ Hereafter, μ Ｄｉ , σ Ｄｉ , μ Ｗｉ , σ Ｗｉ (i=1, 2, 3) are referred to as “estimated reference values.” Note that the estimated reference values ​​in the first embodiment are values ​​calculated from blink frequencies measured for many people.

[0025] When the acquired road surface information is “dry,” the estimation unit 414 calculates the driving concentration level C ｉ Calculate. When the acquired road surface information is “wet,” the estimation unit 414 calculates the driving concentration level C ｉ Calculate. According to the findings in "Correlation between blinking and acceleration state," if concentration is maintained, μ Ｄ１ , μ Ｄ２ , μ Ｗ１ , μ Ｗ２ The value of is close to zero. Therefore, C １ or C ２ If a large value is obtained for μ, it is estimated that the concentration level is decreasing. On the other hand, if the concentration level is maintained, μ Ｄ３ , μ Ｗ３ has a given blink frequency value. Therefore, C ３ If a small value is obtained for , it is estimated that concentration is being maintained.

[0026] The output unit 415 of the device for estimating a level of concentration on driving 406 outputs the calculated level of concentration on driving (step S604). This concludes the description of the device for estimating a level of concentration on driving according to the first embodiment.

[0027] [First Modification] In the first embodiment, the degree of concentration on driving was calculated from the average blinking frequency when the driver was concentrating and the observed blinking frequency of the driver. Alternatively, the observed blinking frequency of the driver may be evaluated using both the average blinking frequency when the driver was concentrating and the average blinking frequency when the driver was not concentrating. This will be described below as a first modification. Note that the only difference between the first modification and the first embodiment is the content of step 603 in FIG. 6 , and therefore FIG. 6 will be used as is for the description.

[0028] The estimation unit 414 receives the blink frequency vector f = (f １ , f ２ , f ３ ) is acquired (step S601). Road surface information is also acquired from the wiper information processing unit 413 (step S602).

[0029] The estimation unit 414 calculates the average value vector μ of the blink frequency in the speed ranges 1, 2, and 3 when the road surface is dry and the driver is concentrating.Ｄ = (μ Ｄ１ , μ Ｄ２ , μ Ｄ３ ), and the average value vector μ' of the blink frequencies in speed regions 1, 2, and 3 when the road surface is dry and the driver is not concentrated Ｄ = (μ' Ｄ１ , μ' Ｄ２ , μ' Ｄ３ ). The estimation unit 414 holds the average value vector μ of the blink frequencies in speed regions 1, 2, and 3 when the road surface is wet and the driver is concentrated Ｗ = (μ Ｗ１ , μ Ｗ２ , μ Ｗ３ ), and the average value vector μ' of the blink frequencies in speed regions 1, 2, and 3 when the road surface is wet and the driver is not concentrated Ｗ = (μ' Ｗ１ , μ' Ｗ２ , μ' Ｗ３ ).

[0030] When the acquired road surface information is "dry", the estimation unit 414 calculates, for example, Equation (3). If C < C' (if the vector f approximates to "concentrated"), the driver is determined to be in a concentrated state, and if C > C' (if the vector f approximates to "not concentrated"), the driver is determined to be in a distracted state. When the acquired road surface information is "wet", the estimation unit 414 calculates, for example, Equation (4). If C < C' (if the vector f approximates to "concentrated"), the driver is determined to be in a concentrated state, and if C > C' (if the vector f approximates to "not concentrated"), the driver is determined to be in a distracted state.

[0031] The above is the description of the first modification example.

[0032] [Second Embodiment] <Optimization for Individuals> It is known that blinking behavior is unique to each individual, and ideally, a model for calculating the concentration level should be created to suit each individual. The estimation unit that calculates the concentration level from the observed blink frequency can use a model obtained from the summary amounts of many people as in the first embodiment, but since there are significant individual differences in blinking behavior, it is possible to use a model based on data from many people as a default when a driver gets in the car for the first time, and gradually switch to a model using personal data.

[0033] Fig. 8 is a functional block diagram showing an example of the configuration of a concentration-on-driving estimation device 801 and peripheral devices according to the second embodiment. The difference from Fig. 4 is that the concentration-on-driving estimation device 801 includes an individual identification unit 802 and an optimization unit 803. Fig. 9 is a flowchart illustrating an example of the operation of the optimization unit 803 of the concentration-on-driving estimation device 801. The optimization unit 803 optimizes the reference blink frequency F obtained by measuring many people. Ｓ The optimization unit 803 corrects the estimation reference value stored in the determination unit 414 for each driver. Since the operation of the concentration on driving estimation device 801 after correction is the same as in the first embodiment, the operation of the optimization unit 803 will mainly be described in detail.

[0034] Individual characteristics are strongly reflected in overall blink frequency. While the frequency remains in the order of deceleration (area 1) < lateral acceleration (area 2) < straight-line acceleration (area 3), overall blink frequency varies greatly from person to person, ranging from about 10 to 50 blinks per minute. Therefore, the blink frequency specific to each driver is measured, and the estimated reference value is corrected for each driver using the measured specific blink frequency.

[0035] First, the optimization unit 801 performs facial recognition on the image of the driver taken by the camera 402 using the individual identification unit 802, and assigns an ID to the facial feature amount (step S901). That is, an ID is assigned to the driver. This ID is assumed to be k.

[0036] The optimization unit 801 checks the blink frequency of the driver over a predetermined period without taking into account the vehicle state (acceleration state) and calculates the intrinsic blink frequency F of the driver k. ｋＵ is calculated (step S902).

[0037] The optimization unit 801 optimizes the estimated reference value μ Ｄｉ , σ Ｄｉ , μ Ｗｉ , σ Ｗｉ (i = 1, 2, 3), the unique reference value μ of driver k ｋ Ｄｉ , σ ｋ Ｄｉ , μ ｋ Ｗｉ , σ ｋ Ｗｉ is calculated as follows (step S903):

[0038] The optimization unit 803 calculates the unique reference value μ ｋ Ｄｉ , σ ｋ Ｄｉ , μ ｋ Ｗｉ , σ ｋ Ｗｉ is registered in the estimation unit (step S904).

[0039] Thereafter, the driving concentration level estimation device 801 recognizes the face of the driver at startup to determine the ID, and estimates the driver's driving concentration level using a unique reference value corresponding to the ID.

[0040] [Supplementary Note] In the first embodiment, blinks are detected using a camera image and a blink detection algorithm, but a wearable eye tracker may also be used for blink detection. ｉ but C ｉ The system may determine whether the acceleration is a threshold value and issue a "distraction" warning to the driver or passengers. Furthermore, in the above-described embodiments and modifications, the acceleration in the forward and left-right directions of a vehicle traveling on the ground is measured, mapped into a two-dimensional acceleration space, and analyzed for correlation with eye blinks. However, for a moving object traveling on a one-dimensional trajectory or in a three-dimensional space, it is also possible to measure acceleration in one or three axes, map it into a one-dimensional acceleration space or a three-dimensional acceleration space, analyze the correlation with eye blinks, and use the analysis results to estimate the level of concentration on driving.

[0041] [Program, Recording Medium] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in FIG. 10 and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0042] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0043] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0044] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0045] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0046] 402 Camera 403 Vehicle 404 Wiper control unit 405 Acceleration sensor 406, 801 Driving concentration level estimation device 411 Blink detection unit 412 Blink frequency update unit 413 Wiper information processing unit 414 Estimation unit 415 Output unit 802 Individual identification unit 803 Optimization unit 2000 Computer 2010 Control unit 2020 Recording unit 2030 Input unit 2040 Output unit 2050 Display unit

Claims

1. A device for estimating the level of concentration of a driver driving a moving vehicle, comprising: an acceleration area defined by dividing one or three dimensional acceleration space; an estimation unit for estimating the level of concentration on driving, the device defining the acceleration area as an average and variance of the frequency at which a driver with concentration blinks in each of said acceleration areas as estimation reference values; a blink frequency update unit for measuring the blink frequency of the driver in association with said acceleration areas; and an estimation unit for estimating the level of concentration on driving of the driver using said estimation reference value and the measured blink frequency.

2. A driving concentration estimation device as described in claim 1, wherein the acceleration space is a two-dimensional space consisting of the direction of travel of the moving body and a left-right direction perpendicular to the direction of travel within a horizontal plane, and the acceleration region consists of a first region that is negative in the direction of travel, a second region that is positive in the direction of travel and has a predetermined value or more in the left-right direction, and a third region that is positive in the direction of travel and has a predetermined value or less in the left-right direction.

3. A driving concentration estimation device as described in claim 2, wherein the moving body is a vehicle traveling on a road, and the device is equipped with a wiper information processing unit that estimates whether the road is dry or wet, and the estimation unit estimates the driver's level of concentration on driving using an estimated reference value measured when driving on a dry road or an estimated reference value measured when driving on a wet road depending on the output of the wiper information processing unit.

4. A device for estimating a level of concentration on driving as described in claim 2, comprising: an optimization unit that measures a specific blink frequency specific to the driver, sets the blink frequency across the entire acceleration space of the estimated reference value as a reference blink frequency, measures a specific blink frequency specific to the driver, corrects the estimated reference value using the reference blink frequency and the specific blink frequency, and calculates a specific reference value, and the estimation unit calculates the driver's level of concentration on driving using the specific reference value.

5. A device for estimating a level of concentration on driving as described in claim 4, further comprising an individual identification unit that identifies the driver and assigns an ID to the driver, and the optimization unit links the ID with the unique reference value and registers them in the estimation unit.

6. A method for estimating a level of concentration of a driver driving a moving vehicle, comprising: defining regions obtained by dividing one or three dimensional acceleration space as acceleration regions; defining the average and variance of the frequency at which a driver with concentration blinks in each of said acceleration regions as estimation reference values; a blink frequency update unit measuring the blink frequency of the driver in association with said acceleration regions; and a step by an estimation unit estimating the driver's level of concentration on driving using said estimation reference value and the measured blink frequency.

7. A program for causing a computer to function as the device for estimating concentration on driving according to any one of claims 1 to 5.

Images