Driver state estimation device

Abstract

Provided is a driver state estimation device capable of estimating that a driver is in a distraction state different from other abnormal states such as diseases. A driver state estimation device (100) is provided with a controller (10) configured so as to estimate whether a driver is in a distraction state on the basis of travel environment information and the line of sight of the driver, and the controller acquires a feature quantity xi on the basis of the travel environment information and the line of sight of the driver. The feature quantity xi includes a frequency and an amplitude of instantaneous eye movement of the driver and a top-down attention score indicating a degree of deviation from an appropriate line-of-sight distribution to an object to be noticed in the vicinity of the vehicle, and the feature quantity xi, a weight coefficient ai set in advance for each feature quantity xi, and a constant a0 set in advance are used to obtain a weight coefficient ai set in advance for each feature quantity xi, and the weight coefficient ai and the constant a0 set in advance for each object to be noticed in the vicinity of the vehicle. A distraction probability p indicating a probability that the driver is in a distraction state is calculated using a Sigmoid function, and when a state in which the calculated distraction probability p is equal to or greater than a predetermined value continues for a predetermined time or greater, it is estimated that the driver is in a distraction state.

Description

Technical Field

[0001] The present invention relates to a driver state estimating device for estimating the state of a driver driving a vehicle. Background Art

[0002] One of the main causes of traffic accidents is a driver's lack of concentration on driving, known as distraction. Previously, technologies for detecting distraction have been proposed (see, for example, Patent Document 1) based on the understanding that the speed and duration of each oscillation of the driver's eye, known as eye blinks, occur when the driver's gaze shifts, differ between intentionally looking to the side and normally looking forward.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-224066

[0006] Technical problem to be solved by the invention

[0007] However, in the above-mentioned prior art, even when the driver is not distracted, it is possible to infer that the driver is distracted if the frequency or amount of change in the driver's gaze changes due to illness, aging, etc. In other words, it is difficult to accurately infer that the driver is distracted, distinguishing from other abnormal conditions such as illness, using prior art. Summary of the Invention

[0008] The present invention has been made to solve such a problem, and an object of the present invention is to provide a driver state estimating device capable of estimating that a driver is in a distracted state, distinguishing it from other abnormal states such as illness.

[0009] Technical means for solving technical problems

[0010] In order to solve the above-mentioned technical problems, the present invention is a driver state estimation device that estimates the state of a driver driving a vehicle, comprising: a driving environment information acquisition device that acquires driving environment information of the vehicle; a sight line detection device that detects the driver's sight line; and a controller that is configured to estimate whether the driver is distracted based on the driving environment information and the driver's sight line. The controller is configured to obtain, based on the driving environment information and the driver's sight line, each characteristic quantity x for a plurality of indicators of exploratory actions according to changes in the driver's state. i , using the acquired feature x i , for each feature quantity x i The weight coefficients a are set in advance i, and a preset constant a0, the distraction probability p, which represents the probability that the driver is distracted, is calculated using the following formula:

[0011]

[0012] Where i = 1, ..., n,

[0013] If the calculated distraction probability p is greater than or equal to a predetermined value and continues for a predetermined time or longer, it is estimated that the driver is distracted, and the characteristic value x i It includes the frequency and amplitude of the driver's eye blinks obtained based on the driver's line of sight, and a top-down attention score obtained based on driving environment information and the driver's line of sight, which represents the degree of deviation from the appropriate line of sight allocation to the attention objects around the vehicle.

[0014] According to the present invention thus constituted, the controller acquires the characteristic value x for a plurality of indices of the exploration action according to the change in the driver's state. i , the feature quantity x i The acquired feature quantity x is used to calculate the frequency and amplitude of the driver's eye blinks and the top-down attention score indicating the degree of deviation from the appropriate distribution of the driver's sight to the objects of attention around the vehicle. i , for each feature quantity x i The weight coefficients a are set in advance i , and a pre-set constant a0, and uses the Sigmoid function to calculate the distraction probability p. Therefore, rather than focusing solely on a single indicator of the driver's exploratory behavior, the method comprehensively captures the frequency and amplitude of each characteristic quantity of eye movements and the unique changes in the top-down attention score when the driver is distracted, and quantitatively evaluates the probability of the driver being distracted. This makes it possible to infer the distraction state, distinguishing it from changes in each characteristic quantity caused by the driver's illness, aging, etc.

[0015] In the present invention, it is preferable that the controller is configured to correct each acquired feature value x based on the driving environment information. i .

[0016] According to the present invention thus constituted, the controller corrects each acquired feature value x based on the driving environment information. i , so the feature quantity x can be corrected i By eliminating the influence of the vehicle's driving environment and more accurately calculating the distraction probability p, it is possible to prevent the driver's state from being misjudged due to the driving environment.

[0017] In the present invention, the controller is preferably configured to obtain the slope of the road on which the vehicle is traveling based on the driving environment information, and to correct the characteristic value x in a direction in which the greater the slope, the less likely it is to estimate that the driver is distracted.i .

[0018] According to the present invention thus constituted, the controller corrects the characteristic value x in the direction that the greater the slope of the road on which the vehicle is traveling, the more difficult it is to estimate that the driver is distracted. i Therefore, when the driver's line of sight tends to be focused on a narrow area due to a large road slope and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of the road slope, the distraction probability p can be calculated more accurately.

[0019] In the present invention, the controller is preferably configured to obtain the curvature of the road on which the vehicle is traveling based on the driving environment information, and to correct the characteristic value x in a direction in which the greater the curvature, the less likely it is to estimate that the driver is distracted. i .

[0020] According to the present invention thus constituted, the controller corrects the characteristic value x in the direction that the greater the curvature of the road on which the vehicle is traveling, the more difficult it is to estimate that the driver is distracted. i Therefore, when the curvature of the road is large and the driver's line of sight tends to be focused on a narrow range and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of road curvature, the distraction probability p can be calculated more accurately.

[0021] In the present invention, the controller is preferably configured to obtain the illuminance outside the vehicle based on the driving environment information, and to correct the characteristic value x in a direction in which it is more difficult to estimate that the driver is distracted as the illuminance is lower. i .

[0022] According to the present invention thus constituted, the lower the illumination intensity toward the outside of the vehicle, the more difficult it is for the controller to estimate that the driver is distracted. i Therefore, when the driver's line of sight tends to be focused on a narrow range due to low illumination outside the vehicle and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of illumination, the probability of distraction p can be calculated more accurately.

[0023] In the present invention, the controller is preferably configured to obtain the speed of the vehicle based on the driving environment information and to correct the characteristic value x so that the higher the speed, the less likely it is to estimate that the driver is distracted. i .

[0024] According to the present invention thus constituted, the controller corrects the characteristic value x in the direction that it becomes more difficult to estimate that the driver is distracted as the vehicle speed increases. i Therefore, when the vehicle speed is high and the driver's line of sight tends to be focused on a narrow area and is easily estimated to be in a distracted state, the feature value x can be corrected.i By eliminating the influence of vehicle speed, the probability of distraction p can be calculated more accurately.

[0025] Effects of the Invention

[0026] According to the driver state estimating device of the present invention, it is possible to estimate that the driver is in a distracted state, distinguishing it from other abnormal states such as illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is an explanatory diagram of a vehicle equipped with a driver state estimation device according to an embodiment of the present invention.

[0028] Figure 2 This is a block diagram of a driver state estimation device according to an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of the personal learning process according to the embodiment of the present invention.

[0030] Figure 4 This is a flowchart of the driver state estimation process according to the embodiment of the present invention.

[0031] Figure 5 This is a diagram illustrating a correction coefficient map according to an embodiment of the present invention.

[0032] Figure 6 This is a time series diagram illustrating temporal changes in the characteristic amount of the driver's exploratory action index and the probability of distraction according to the embodiment of the present invention.

[0033] Explanation of symbols

[0034] 1 vehicle

[0035] 10 controllers

[0036] 100 Driver status estimation device

[0037] 21Exterior camera device

[0038] 22 radar

[0039] 23 Navigation System

[0040] 24 positioning system

[0041] 25 vehicle speed sensor

[0042] 26 accelerometers

[0043] 27 yaw rate sensor

[0044] 28 steering angle sensor

[0045] 29 Steering torque sensor

[0046] 30 accelerometer sensors

[0047] 31 brake sensor

[0048] 32In-car camera

[0049] 36 displays

[0050] 37 speakers. DETAILED DESCRIPTION

[0051] Hereinafter, a driver state estimating device according to an embodiment of the present invention will be described with reference to the drawings.

[0052] [System Structure]

[0053] First, refer to Figure 1 and Figure 2 , the structure of the driver state estimation device of this embodiment is described. Figure 1 is an explanatory diagram of a vehicle equipped with a driver state estimation device. Figure 2 This is a block diagram of a driver state estimation device.

[0054] The vehicle 1 of this embodiment includes: a driving force source 2 such as an engine or an electric motor that outputs driving force; a transmission 3 that transmits the driving force output from the driving force source 2 to the drive wheels; a brake 4 that applies braking force to the vehicle 1; and a steering device 5 for steering the vehicle 1.

[0055] The driver state estimation device 100 is configured to estimate the state of the driver of the vehicle 1 and perform control of the vehicle 1 and driving support control as needed. Figure 2 As shown, the driver state estimation device 100 includes a controller 10, a plurality of sensor devices, a plurality of control systems, and a plurality of information output devices.

[0056] Specifically, the multiple sensor devices include an exterior camera 21 and a radar 22 for acquiring information about the driving environment of vehicle 1; a navigation system 23 and a positioning system 24 for detecting the position of vehicle 1. Furthermore, the multiple sensor devices include a vehicle speed sensor 25, an acceleration sensor 26, a yaw rate sensor 27, a steering angle sensor 28, a steering torque sensor 29, an accelerator sensor 30, and a brake sensor 31 for detecting the movement of vehicle 1 or the driver's driving operations. Furthermore, the multiple sensor devices include an interior camera 32 for detecting the driver's line of sight. The multiple control systems include a powertrain control module (PCM) 33 for controlling drive source 2 and transmission 3; a dynamic stability control system (DSC) 34 for controlling drive source 2 and brakes 4; and an electric power steering system (EPS) 35 for controlling steering system 5. The multiple information output devices include a display 36 for outputting image information and a speaker 37 for outputting audio information.

[0057] In addition, other sensor equipment may also include peripheral sonar for measuring the distance and position of surrounding structures relative to the vehicle 1, corner radar for measuring the approach of surrounding structures at the corners of the four parts of the vehicle 1, and various sensors for detecting the driver's status (such as heart rate sensors, electrocardiogram sensors, steering wheel grip force sensors, etc.).

[0058] The controller 10 performs various calculations based on signals received from a plurality of sensor devices, sends control signals to the PCM 33, DSC 34, and EPS 35 to properly operate the drive source 2, transmission 3, brakes 4, and steering system 5, and sends control signals to the display 36 and speaker 37 to output desired information. The controller 10 is configured as a computer having one or more processors 10a (typically CPUs), memory 10b (ROM, RAM, etc.) that stores various programs and data, and input / output devices.

[0059] The off-vehicle camera 21 captures the surroundings of the vehicle 1 and outputs image data. The controller 10 identifies objects (e.g., preceding vehicles, parked vehicles, pedestrians, the road, dividing lines (lane boundaries, white lines, yellow lines), traffic signals, traffic signs, stop lines, intersections, obstacles, etc.) based on the image data received from the off-vehicle camera 21. Furthermore, the controller 10 can determine the curvature of the road on which the vehicle 1 is traveling and the external illumination of the vehicle 1 based on the image data received from the off-vehicle camera 21. Furthermore, the off-vehicle camera 21 is an example of a "driving environment information acquisition device" in the present invention.

[0060] The radar 22 measures the position and speed of an object (especially a preceding vehicle, a parked vehicle, a pedestrian, a fallen object on the road, etc.). As the radar 22, for example, a millimeter wave radar can be used. The radar 22 transmits radio waves in the direction of travel of the vehicle 1, and receives reflected waves generated when the transmitted waves are reflected by the object. Then, the radar 22 measures the distance between the vehicle 1 and the object (for example, the distance between the vehicles), and the relative speed of the object relative to the vehicle 1 based on the transmitted waves and the received waves. In addition, in this embodiment, a laser radar, an ultrasonic sensor, etc. can be used instead of the radar 22 to measure the distance and relative speed to the object. In addition, a plurality of sensor devices can also be used to form a position and speed measuring device. In addition, the radar 22 is equivalent to an example of a "driving environment information acquisition device" in the present invention.

[0061] The navigation system 23 stores map information internally and can provide the map information to the controller 10. Based on the map information and the current vehicle position information, the controller 10 determines the roads, intersections, traffic signals, buildings, etc. that exist around the vehicle 1 (especially the direction of travel). In addition, the controller 10 can determine the curvature and slope of the road on which the vehicle 1 is traveling based on the map information and the current vehicle position information. The map information can also be stored in the controller 10. The positioning system 24 is a GPS system and / or a gyroscope system that detects the position of the vehicle 1 (current vehicle position information). In addition, the navigation system 23 and the positioning system 24 are also equivalent to an example of the "driving environment information acquisition device" in the present invention.

[0062] The vehicle speed sensor 25 detects the speed of the vehicle 1 based on, for example, the rotational speed of the wheels and the drive shaft. The acceleration sensor 26 detects the acceleration of the vehicle 1. The acceleration includes the acceleration in the front-rear direction and the lateral acceleration (i.e., lateral acceleration) of the vehicle 1. In addition, the controller 10 can determine the slope of the road on which the vehicle 1 is traveling based on the speed and acceleration of the vehicle 1. In addition, in this specification, the acceleration includes not only the rate of change of speed in the direction in which the speed increases, but also the rate of change of speed in the direction in which the speed decreases (i.e., deceleration). In addition, the vehicle speed sensor 25 and the acceleration sensor 26 are also equivalent to an example of the "driving environment information acquisition device" in the present invention.

[0063] The yaw rate sensor 27 detects the yaw rate of the vehicle 1. The steering angle sensor 28 detects the rotation angle (steering angle) of the steering wheel of the steering system 5. The steering torque sensor 29 detects the torque (steering torque) applied to the steering shaft via the steering wheel. The accelerator sensor 30 detects the amount of accelerator pedal depression. The brake sensor 31 detects the amount of brake pedal depression. Furthermore, the yaw rate sensor 27, steering angle sensor 28, steering torque sensor 29, accelerator sensor 30, and brake sensor 31 also constitute an example of the "driving environment information acquisition device" in the present invention.

[0064] The in-vehicle camera 32 captures the driver's image and outputs image data. The controller 10 detects the driver's line of sight based on the image data received from the in-vehicle camera 32. The in-vehicle camera 32 is an example of a "line of sight detection device" in the present invention.

[0065] The PCM 33 controls the driving force source 2 of the vehicle 1 to adjust the driving force of the vehicle 1. For example, the PCM 33 controls the engine's spark plugs, fuel injection valves, throttle valve, variable valve mechanism, transmission 3, and the inverter that supplies power to the electric motor. When the vehicle 1 needs to be accelerated or decelerated, the controller 10 sends a control signal to the PCM 33 to adjust the driving force.

[0066] The DSC 34 controls the driving force source 2 and brakes 4 of the vehicle 1, performing deceleration control and posture control of the vehicle 1. For example, the DSC 34 controls the hydraulic pump and valve unit of the brakes 4, controlling the driving force source 2 via the PCM 33. When deceleration control or posture control of the vehicle 1 is required, the controller 10 sends a control signal to the DSC 34 to adjust the driving force or generate braking force.

[0067] The EPS 35 controls the steering system 5 of the vehicle 1. For example, the EPS 35 controls a motor that applies torque to the steering shaft of the steering system 5. When the traveling direction of the vehicle 1 needs to be changed, the controller 10 sends a control signal to the EPS 35 to change the steering direction.

[0068] The display 36 is provided in front of the driver in the vehicle cabin and displays image information to the driver. For example, a liquid crystal display or a head-up display is used as the display 36. The speaker 37 is provided in the vehicle cabin and outputs various audio information.

[0069] [Driver status estimation]

[0070] Next, refer to Figures 3 to 5 , the driver state estimation by the driver state estimation device 100 according to this embodiment will be described. Figure 3 This is a flowchart of a personal learning process for performing personal learning to standardize the feature quantities of exploration action indicators. Figure 4 This is a flowchart of a driver state estimation process for estimating whether the driver is distracted or normal. Figure 5 : is a diagram illustrating a correction coefficient map for correcting the feature amount of the exploration action indicator.

[0071] First, an overview of driver state estimation in this embodiment will be described. To investigate how a driver's visual activity (hereinafter referred to as "exploratory activity") changes between a normal driver state and a distracted driver state, the inventors conducted driving experiments using a driving simulator with over 100 test subjects. Specifically, the researchers measured the driver's gaze movements while driving, simulating a distracted state by having the driver perform mental arithmetic and lose focus on driving, and simulating a normal state by having the driver perform normal driving without mental arithmetic. The researchers conducted driving experiments in various driving environments (city streets, highways, mountain roads, daytime, nighttime, etc.).

[0072] As a result, it was found that the characteristic quantities of multiple indicators associated with the driver's exploratory behavior (for example, including the frequency and amplitude of eye blinks) change according to the unique trends of each indicator when the driver is normal and distracted. Therefore, the inventors believe that based on the data of eye gaze movement obtained from the above-mentioned driving experiments and the data of the driving environment simulated by the driving simulator, the binary objective variable is used, which of the cases corresponds to the simulated distracted state of the driver or the simulated normal state. The standardized values of the characteristic quantities of the multiple exploratory behavior indicators are used as explanatory variables, and a logistic regression analysis is performed to obtain the regression coefficients in advance. This allows the probability of the driver being distracted to be calculated from the characteristic quantities during actual driving.

[0073] Specifically, the driver state estimation device 100 acquires various feature quantities for multiple indicators of the driver's exploratory behavior based on information about the vehicle 1's driving environment and the driver's line of sight. For example, the driver state estimation device 100 acquires the amplitude and frequency of the driver's eye blinks, a top-down attention score indicating the degree of deviation from the appropriate distribution of attention to objects surrounding the vehicle 1, and a bottom-up attention score indicating the degree to which the gaze is directed toward highly salient locations. These feature quantities are then corrected based on driving conditions such as road slope and curvature. The corrected feature quantities are then normalized using the mean and variance of each feature quantity, obtained through pre-trained individual learning for each driver, when the driver is normal. Finally, the normalized feature quantities are applied to a sigmoid function containing the regression coefficients obtained through logistic regression analysis based on the aforementioned driving experiment to calculate the probability that the driver is distracted. If the calculated probability exceeds a predetermined threshold and persists for a predetermined period of time, the driver state estimation device 100 estimates that the driver is distracted. In this way, instead of focusing on just one indicator of the driver's exploratory behavior, the unique changes in the characteristic quantities of multiple indicators when the driver is distracted are comprehensively captured, and the probability of the driver being distracted is quantitatively evaluated. In this way, the distracted state can be estimated in a way that is different from the changes in the characteristic quantities caused by the driver's illness, aging, etc.

[0074] [Personal learning process]

[0075] Next, refer to Figure 3 The individual learning process is described below. During the individual learning process, the driver state estimation device 100 calculates the mean and variance of each feature quantity for each driver, used when normalizing the feature quantities of multiple indicators of exploration behavior during the driver state estimation process. Specifically, individual learning of the mean and variance of each feature quantity is performed when the driver is in a normal state. The individual learning process begins, for example, when the vehicle 1 begins its first trip of the day.

[0076] When the individual learning process is started, first, the controller 10 recognizes the current driver based on information received from the in-vehicle camera 32 , for example (step S1 ).

[0077] Next, the controller 10 obtains driving environment information based on the signals received from the sensor equipment including the external vehicle camera 21, radar 22, navigation system 23, positioning system 24, vehicle speed sensor 25, acceleration sensor 26, yaw rate sensor 27, steering angle sensor 28, steering torque sensor 29, accelerator sensor 30 and brake sensor 31 (step S2).

[0078] Next, the controller 10 determines whether the conditions for executing individual learning (learning conditions) are met based on the driving environment information obtained in step S1 (step S3). Individual learning needs to be performed when the driving environment has a relatively small impact on the driver's exploratory behavior and when the driver is in a normal state. For example, when the current position of the vehicle 1 is on a city street, the vehicle speed is within a specified range (for example, above 20 km / h and below 60 km / h), the road being driven is flat (for example, the slope is less than 3%), the road being driven is a straight line (for example, the radius of curvature is more than 2000m), and it is daytime, it is considered that the driving environment has a relatively small impact on the driver's exploratory behavior. In addition, in the absence of sharp driving operations or collisions, it is considered that dangerous avoidance operations or collisions caused by the driver's distraction will not occur, that is, the driver is in a normal state. Therefore, when the current position of the vehicle 1 is on a city street, the vehicle speed is within the specified range (for example, above 20 km / h and below 60 km / h), the road on which the vehicle is traveling is flat (for example, the slope is less than 3%), the road on which the vehicle is traveling is a straight line (for example, the radius of curvature is above 2000 m), it is daytime, and there are no sudden driving operations or collisions, the controller 10 determines that the conditions for executing personal learning are met.

[0079] As a result, if the conditions for executing individual learning are not satisfied (step S3: No), the process returns to step S2, and steps S2 and S3 are repeated until the conditions for executing individual learning are satisfied.

[0080] On the other hand, when the conditions for executing individual learning are satisfied (step S3 : Yes), the controller 10 detects the driver's line of sight based on the signal received from the in-vehicle camera 32 (step S4 ).

[0081] Next, the controller 10 calculates the frequency x1 and amplitude x2 of eye blinks based on the detected driver's line of sight (step S5). Eye blinks are one of the indicators associated with the driver's exploratory behavior. Eye blinks refer to jumping eye movements in order to capture a visual target at the fovea of the retina. They are eye movements that move the line of sight from a point of fixation where the line of sight has been stationary for a specified time to the next point of fixation. As characteristic quantities of eye blinks, the amplitude and frequency of eye blinks are used in this embodiment. The amplitude of an eye blink refers to the amount of movement when the driver's line of sight moves from one point of fixation to the next, and the frequency of an eye blink refers to the number of times the line of sight moves from one point of fixation to the next within a specified time. For example, based on the number of eye blinks in a specified time (e.g., 30 seconds), the controller 10 calculates the number of eye blinks per unit time as the eye blink frequency x1. In addition, the controller 10 calculates the average value of the eye blink amplitudes in the most recent specified time (e.g., 30 seconds) as the eye blink amplitude x2.

[0082] Next, the controller 10 obtains objects (attention objects) that the driver should pay attention to in the direction of travel of the vehicle 1 based on the driving environment information obtained in step S2 (step S6). Examples of attention objects include other vehicles, obstacles, pedestrians, traffic lights, road signs, etc.

[0083] Next, the controller 10 calculates the top-down attention score x3 based on the driver's line of sight detected in step S4 and the attention object obtained in step S6 (step S7). Top-down attention is one of the indicators associated with the driver's exploratory behavior, and refers to an attention mechanism that actively moves the line of sight to the part that the person is concerned about. For example, if the driver recognizes in advance that other vehicles are attention objects, the driver can take precedence over other parts and actively turn his or her line of sight toward other vehicles. As a characteristic quantity of top-down attention, a top-down attention score is used in this embodiment. The top-down attention score refers to a numerical value that represents the degree of deviation from the appropriate line of sight allocation to the attention objects around the vehicle 1.

[0084] For example, the controller 10 calculates the appropriate number of times and duration of attention that the driver should focus on each attention object in front of the vehicle 1 within a specified time period (e.g., 10 seconds) based on a pre-created top-down attention model and driving environment information. The top-down attention model is a mathematical formula with coefficients set to calculate the appropriate number of times and duration of attention for each attention object by substituting vehicle speed, the time to collision margin (TTC) of the attention object, and the time the attention object is visible in front of the vehicle 1. The top-down attention model is pre-created, for example, by conducting driving experiments on multiple test subjects in a normal state using a driving simulator and learning from the results of the driving experiments, and is stored in the memory 10b.

[0085] The controller 10 then obtains the number of times and duration of the driver's gaze at each attention object in front of the vehicle 1 within a recent predetermined period (e.g., 10 seconds) from the driving environment information and the driver's line of sight. For each attention object, the controller 10 calculates the difference between the aforementioned number of gazes and duration and the appropriate number of gazes and duration calculated using the top-down attention model. The top-down attention score x3 is then calculated by multiplying the average of the calculated differences in the number of gazes for each attention object by the average of the differences in the gaze durations.

[0086] Next, the controller 10 obtains the distribution of prominence in the most recent specified time (e.g., 30 seconds) in the direction of travel of the vehicle 1 based on the driving environment information obtained in step S2 (step S8). Prominence refers to a characteristic that indicates how easily a person's attention is attracted. That is, in the driver's field of view, an area with high prominence is an area that easily attracts the driver's attention due to, for example, a large color difference or brightness difference relative to the surrounding area or a large movement. The controller 10 can obtain the distribution of prominence by processing the temporal and spatial configuration of color, brightness, contrast, movement, etc. in the image obtained from the off-vehicle camera 21 using known image processing techniques.

[0087] Next, the controller 10 calculates a bottom-up attention score x4 based on the driver's line of sight detected in step S4 and the salience distribution obtained in step S8 (step S9). Bottom-up attention is one of the indicators associated with the driver's exploratory behavior and is an attention mechanism in which the line of sight passively moves toward a highly salient area. In this embodiment, a bottom-up attention score is used as a characteristic quantity of bottom-up attention. The bottom-up attention score is a numerical value that indicates the degree of deviation from the appropriate line of sight distribution for the attention objects around the vehicle 1.

[0088] For example, the controller 10 generates a receiver operating characteristic (ROC) curve. This ROC curve plots the probability of the salience of a random point in front of the vehicle 1 exceeding a predetermined threshold value and the probability of the salience exceeding a predetermined threshold value in the direction of the driver's gaze within a recent predetermined period (e.g., 30 seconds) based on driving environment information and the driver's line of sight, while varying the predetermined threshold value. The bottom-up attention score x4 is calculated by multiplying the AUC (Area Under the Curve) of this ROC curve by a predetermined coefficient. In this case, the stronger the driver's tendency to move toward a highly salient object, the closer the AUC approaches the maximum value of 1, and the higher the bottom-up attention score x4.

[0089] Next, the controller 10 converts each feature value x calculated in steps S5, S7, and S9 into i (i=1, 2, 3, 4) is stored in the learning database (step S10). The learning database is stored in the memory 10b.

[0090] Next, the controller 10 determines whether each feature value x i The total time accumulated in the learning database, that is, the time that satisfies the learning conditions after the start of the individual learning process, reaches a predetermined time (e.g., 20 minutes) (step S11). i If the total time of x does not reach the specified time (step S11: No), return to step S2 and repeat the process from step S2 to S11 until each feature value x is accumulated. i Until the total time reaches the specified time.

[0091] On the other hand, when accumulating each feature value x i When the total time reaches the predetermined time (step S11: Yes), the controller 10 calculates each feature value x i The average value of each μ i and variance σ i (Step S12).

[0092] Next, the controller 10 stores the average value μi and the variance σi calculated in step S12 in the memory 10b in association with the driver identified in step S1 (step S13 ). Thereafter, the controller 10 ends the individual learning process.

[0093] [Driver status estimation process]

[0094] Next, refer to Figure 4 The driver state estimation process is described. The driver state estimation process is started when the vehicle 1 is powered on and is repeatedly executed by the controller 10 at a predetermined cycle (for example, every 0.05 to 0.2 seconds). The driver state estimation process can be compared with the reference Figure 3 The individual learning processes described are executed in parallel.

[0095] When the driver state estimation process starts, first, the controller 10 obtains driving environment information based on the signals received from the sensor equipment including the external vehicle camera 21, radar 22, navigation system 23, positioning system 24, vehicle speed sensor 25, acceleration sensor 26, yaw angular velocity sensor 27, steering angle sensor 28, steering torque sensor 29, accelerator sensor 30 and brake sensor 31 (step S21).

[0096] Next, the controller 10 determines whether the distraction determination condition for determining the driver's distraction state, that is, the condition for estimating the driver's state, is met based on the driving environment information obtained in step S21 (step S22). For example, when the vehicle 1 is driving in a tunnel or on an overpass, or when the vehicle 1 is changing lanes, there are driving scenarios that are easily misjudged as being in a distracted state because the driver's line of sight is focused on a narrow range. Therefore, driving scenarios that are easily misjudged as being in a distracted state and driving scenarios in which it is difficult for the driver to be distracted are pre-defined as driving scenarios outside the object of driver state estimation. Then, when the driving scenario determined based on the driving environment information obtained in step S21 does not meet the driving scenario outside the object of driver state estimation, the controller 10 determines that the distraction determination condition is met.

[0097] As a result, if the distraction determination condition is not satisfied (step S22 : NO), the controller 10 ends the driver state estimation process.

[0098] On the other hand, when the distraction determination condition is satisfied (step S22 : YES), the controller 10 detects the driver's line of sight based on the signal received from the in-vehicle camera 32 (step S23 ).

[0099] Next, the controller 10 calculates the frequency x1 and amplitude x2 of the eye blink based on the detected line of sight of the driver (step S24). The method of calculating the frequency x1 and amplitude x2 of the eye blink is the same as step S5 of the individual learning process.

[0100] Next, the controller 10 acquires an object (a caution object) that the driver should pay attention to ahead of the vehicle 1 in the traveling direction based on the traveling environment information acquired in step S21 (step S25 ).

[0101] Next, the controller 10 calculates the top-down attention score x3 based on the driver's line of sight detected in step S23 and the attention object acquired in step S25 (step S26). The calculation method of the top-down attention score x3 is the same as step S7 of the personal learning process.

[0102] Next, the controller 10 acquires the distribution of prominence in the most recent predetermined time (for example, 30 seconds) ahead in the traveling direction of the vehicle 1 based on the traveling environment information acquired in step S21 (step S27 ).

[0103] Next, the controller 10 calculates the bottom-up attention score x4 based on the driver's line of sight detected in step S23 and the salience distribution acquired in step S27 (step S28). The calculation method of the bottom-up attention score x4 is the same as step S9 of the personal learning process.

[0104] Next, the controller 10 corrects each feature value x calculated in steps S24, S26, and S28 based on the driving scene. i (i=1, 2, 3, 4) (step S29). Specifically, for the road slope, road curvature, illumination and vehicle speed, the characteristic values x are determined. i The correction coefficient map of the correction coefficient of the correction coefficient is stored in the memory 10b. Based on the driving environment information obtained in step S21, the controller 10 obtains the slope and curvature of the road on which the vehicle 1 is traveling, the illuminance outside the vehicle 1, and the vehicle speed, and refers to the correction coefficient map stored in the memory 10b to obtain the correction coefficients corresponding to the obtained road slope, road curvature, illuminance, and vehicle speed. Then, the controller 10 multiplies the obtained correction coefficients by each feature quantity x. i to make corrections.

[0105] Figure 5 (a) is an example of a mapping that determines a correction coefficient for the eye blink frequency x1 corresponding to road gradient. When the gradient of an uphill or downhill road increases, the driver carefully checks the road conditions in the direction of travel, and therefore tends to focus their gaze on a narrow area. As a result, the eye blink frequency x1 decreases, sometimes reaching the same value as when the driver is distracted. Therefore, by increasing the correction coefficient for the eye blink frequency x1 to greater than 1 as the uphill or downhill road gradient increases, in other words, by setting the correction coefficient in a direction that makes it less likely that the driver is distracted, the eye blink frequency x1 can be corrected to eliminate the influence of the road gradient.

[0106] Figure 5 (b) shows an example of a mapping that determines the correction coefficient for the eye blink frequency x1 corresponding to road curvature. When the road curvature increases (i.e., when the turn becomes sharper), the driver carefully checks the road conditions in the direction of travel, and therefore tends to focus their gaze on a narrow area. As a result, the eye blink frequency x1 decreases, sometimes reaching the same value as when the driver is distracted. Therefore, by setting the correction coefficient for the eye blink frequency x1 so that it is greater than 1 as the road curvature increases, i.e., in a direction that makes it less likely that the driver is distracted, the eye blink frequency x1 can be corrected to eliminate the influence of road curvature.

[0107] Figure 5(c) is an example of a mapping that determines the correction coefficient for the eye blink frequency x1 corresponding to the illuminance outside the vehicle. When the illuminance decreases (i.e., it gets darker outside the vehicle), the driver carefully checks the road conditions in the direction of travel, and therefore tends to focus their vision on a narrow area. As a result, the eye blink frequency x1 decreases, sometimes reaching the same value as the eye blink frequency during a distracted state. Therefore, by setting the correction coefficient for the eye blink frequency x1 so that it is greater than 1 as the illuminance decreases, that is, by setting the correction coefficient in a direction that makes it difficult to infer that the driver is distracted, the eye blink frequency x1 can be corrected to eliminate the influence of illuminance.

[0108] Figure 5 (d) is an example of a mapping that determines the correction coefficient for the eye blink frequency x1 according to vehicle speed. As vehicle speed increases, the driver's field of vision narrows, leading to a tendency for their line of sight to be concentrated within a narrow range. As a result, the eye blink frequency x1 decreases, sometimes reaching the same value as when the driver is distracted. Therefore, by setting the correction coefficient for the eye blink frequency x1 so that it is greater than 1 as vehicle speed increases, i.e., by making it less likely that the driver is distracted, the eye blink frequency x1 can be corrected to eliminate the influence of vehicle speed.

[0109] Figure 5 The map for determining the correction coefficient for the eye blink frequency x1 is shown as an example. However, correction coefficient maps are similarly set for the eye blink amplitude x2, the top-down attention score x3, and the bottom-up attention score x4, and are stored in the memory 10b.

[0110] Next, the controller 10 uses each feature value x stored in the memory 10b in the individual learning process. i The average value of each μ i and variance σ i , the feature values x corrected in step S29 are i Normalization (step S30). In this way, the characteristic value x of the driver in the normal state can be normalized. i = 0 as a reference value to evaluate each feature value x that reflects the current driver state i .

[0111] Next, the controller 10 determines whether the road on which the vehicle 1 is traveling corresponds to a general road or a highway based on the driving environment information acquired in step S21, and acquires the corresponding characteristic value x for each road. i Pre-set weight coefficient a i(Step S31). As described above, based on the data on the gaze movement obtained from the driving experiment using the driving simulator and the data on the driving environment simulated by the driving simulator, the binary objective variable is used to determine whether the driver's distracted state or the normal state is simulated. Each feature value x is set as i The standardized values are used as explanatory variables, and the regression coefficients calculated in advance by performing logistic regression analysis are used as the regression coefficients for each feature value x. i The weight coefficient a i And stored in the memory 10b.

[0112] Here, the driver's exploration behavior differs on ordinary roads with low vehicle speeds but many pedestrians, intersections, and other objects of attention, and on highways with high vehicle speeds but no pedestrians, intersections, and other objects of attention. Therefore, in this embodiment, driving experiments simulating ordinary roads and driving experiments simulating highways were conducted using a driving simulator. The above-mentioned logistic regression analysis was performed on the experimental results of each experiment, and the weight coefficient a was calculated for driving on ordinary roads and driving on highways respectively. i , and the weight coefficient a i Stored in memory 10b.

[0113] Next, the controller 10 uses each feature value x normalized in step S30 i and the weight coefficient a obtained in step S31 i The following Sigmoid function is used to calculate the distraction probability p indicating the probability that the driver is distracted, and the distraction probability p is stored in the memory 10b (step S32).

[0114]

[0115] Next, the controller 10 obtains the mind-wandering probability p stored in the memory 10b and determines that the mind-wandering probability p is equal to the threshold value p at the current time point. th The state of being above (for example, 80%) continues for a predetermined time (for example, 16 seconds) or longer (step S33).

[0116] As a result, the probability of distraction p up to the current time point is the threshold p th When the above state does not continue for a predetermined time or longer (step S33 : NO), the controller 10 estimates that the driver's state is normal (step S34 ), and ends the driver state estimation process.

[0117] On the other hand, the probability of distraction p up to the current time point is the threshold p th When the above state continues for a predetermined time or longer (step S33 : YES), the controller 10 estimates that the driver is distracted (step S35 ).

[0118] Next, the controller 10 sends control signals to the display 36 and speaker 37, causing the display 36 and speaker 37 to output an alert notifying the driver that the driver is distracted (step S36). At this time, the display 36 and speaker 37 may also output image information and audio information (sight guidance information) to guide the driver's line of sight to an object that the driver has not visually recognized. After step S36, the controller 10 terminates the driver state estimation process.

[0119] Figure 6 The characteristic quantities x of the exploration action index are shown in the case of a driving experiment simulating a city street using a driving simulator. i And the time series diagram of the time change of the probability of mind wandering p. Figure 6 In (a), (b), and (c), the horizontal axis represents time. Figure 6 The vertical axis of (a) represents each feature value x after correction and standardization. i The value of Figure 6 The vertical axis of (b) represents the corrected and normalized feature values x i Multiply by the weight coefficient a i After a i x i , Figure 6 The vertical axis of (c) represents the probability of distraction p. Figure 6 In (a) and (b), the dotted line represents the eye blink frequency x1, the single-dot chain line represents the eye blink amplitude x2, the double-dot chain line represents the top-down attention score x3, and the dotted line represents the bottom-up attention score x4. Figure 6 In (b), the solid line represents a i x i The comprehensive Figure 6 In (c), the solid line represents the probability of distraction p.

[0120] In the driver state estimation process, each feature quantity x i Correction and standardization, e.g. Figure 6 As shown in (a), the influence of the driving scene on the exploration behavior can be eliminated, and 0 is used as a universal standard value to evaluate each feature value x i .exist Figure 6 In the example of (a), it can be seen that the eye blink frequency x1 (dashed line), the eye blink amplitude x2 (single-dot chain line), and the bottom-up attention score x4 (dotted line) take values that are relatively far from 0.

[0121] Furthermore, by multiplying the weight coefficient a obtained by logistic regression analysis i , it is also possible to consider each feature quantity x iEvaluation of the degree of influence on the presumption of whether or not the mind is wandering. Figure 6 In the example (a), the eye blink frequency x1 (dashed line), eye blink amplitude x2 (single-dot chain line), and bottom-up attention score x4 (dotted line) take values far away from 0, but according to Figure 6 (b) shows that compared with the eye blink amplitude x2 (single-dotted line) and the bottom-up attention score x4 (dotted line), the eye blink frequency x1 (dashed line) takes a value that is relatively far away from 0 (especially between time t1 and t2).

[0122] When using Figure 6 Each feature quantity x shown in (b) i and weight coefficient a i The product of a i x i When calculating the probability of distraction p, as Figure 6 As shown in (c), between time t1 and t2, the probability of distraction p is the threshold p th Therefore, if the time from time t1 to t2 is longer than a predetermined time (eg, 18 seconds), it is estimated that the driver is in a distracted state.

[0123] [Modification]

[0124] In the above embodiment, eye blink frequency x1 and amplitude x2, top-down attention score x3, and bottom-up attention score x4 are used as characteristic quantities of multiple indicators of the driver's exploratory behavior. However, some of them may be used in combination, and characteristic quantities of other indicators may be combined.

[0125] In the above embodiment, the controller 10 multiplies each feature value x by a correction coefficient. i To make corrections, it is also possible to make corrections by i Add or subtract the correction value to make the correction.

[0126] [Function, effect]

[0127] Next, the effects of the driver state estimation device 100 according to the above-described embodiment will be described.

[0128] The controller 10 obtains characteristic values x for a plurality of indices of the exploration action according to the change in the driver's state. i , the feature quantity x i The acquired feature quantity x is used to calculate the frequency and amplitude of the driver's eye blinks and the top-down attention score indicating the degree of deviation from the appropriate distribution of the sight lines to the attention objects around the vehicle 1. i And for each feature x i The weight coefficients a are set in advancei and a pre-set constant a0, and uses the sigmoid function to calculate the distraction probability p. Therefore, rather than focusing solely on a single indicator of the driver's exploratory behavior, the system comprehensively captures the frequency and amplitude of eye blinks and the unique changes in the top-down attention score when the driver is distracted, quantitatively evaluating the probability of the driver being distracted. This allows the inference of distraction to be distinguished from changes in various characteristic quantities caused by the driver's illness, aging, etc.

[0129] In addition, the controller 10 corrects each acquired feature quantity x based on the driving environment information. i , so the feature value x can be corrected i By eliminating the influence of the driving environment of the vehicle 1 and more accurately calculating the distraction probability p, it is possible to prevent erroneous estimation of the driver's state due to the driving environment.

[0130] Furthermore, the controller 10 corrects the characteristic value x in the direction in which the vehicle 1 is traveling, which indicates that it is more difficult to estimate that the driver is distracted. i Therefore, when the driver's line of sight tends to be focused on a narrow area due to a large road slope and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of the road slope, the distraction probability p can be calculated more accurately.

[0131] Furthermore, the controller 10 corrects the characteristic value x in the direction in which the vehicle 1 is traveling, as the curvature of the road becomes greater and it becomes more difficult to estimate that the driver is distracted. i Therefore, when the curvature of the road is large and the driver's line of sight tends to be focused on a narrow range and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of road curvature, the distraction probability p can be calculated more accurately.

[0132] Furthermore, the controller 10 makes it more difficult to estimate the direction correction feature value x that the driver is distracted as the illumination outside the vehicle 1 decreases. i Therefore, when the driver's line of sight tends to be focused on a narrow range due to low illumination outside the vehicle and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of illumination, the probability of distraction p can be calculated more accurately.

[0133] Furthermore, the controller 10 corrects the characteristic value x in the direction that it is difficult to estimate that the driver is distracted as the speed of the vehicle 1 increases. i Therefore, when the vehicle speed is high and the driver's line of sight tends to be focused on a narrow area and is easily estimated to be in a distracted state, the feature value x can be corrected. i By eliminating the influence of vehicle speed, the probability of distraction p can be calculated more accurately.

Claims

1. A driver state estimation device for estimating the state of a driver driving a vehicle, characterized in that: have: A driving environment information acquisition device, the driving environment information acquisition device acquires driving environment information of the vehicle; a sight line detection device, the sight line detection device detecting the sight line of the driver; as well as a controller configured to estimate whether the driver is distracted based on the driving environment information and the driver's line of sight, The controller is configured as follows: Based on the driving environment information and the driver's sight line, each feature value x is obtained for a plurality of indices of the exploration action according to the change of the driver's state. i , Using the acquired feature x i , for each of the feature quantities x i The weight coefficients a are set in advance i , and a preset constant a0, the distraction probability p representing the probability that the driver is distracted is calculated using the following formula: Where i = 1, ..., n, If the calculated distraction probability p is greater than or equal to a predetermined value and continues for a predetermined time or longer, it is estimated that the driver is in a distracted state. The characteristic value x i It includes the frequency and amplitude of the driver's eye blinks obtained based on the driver's line of sight, and a top-down attention score obtained based on the driving environment information and the driver's line of sight, wherein the top-down attention score represents the degree of deviation from the appropriate line of sight allocation to the attention objects around the vehicle.

2. The driver state estimation device according to claim 1, characterized in that: The controller is configured to correct each of the acquired feature quantities x based on the driving environment information. i .

3. The driver state estimation device according to claim 2, characterized in that: The controller is configured to obtain the slope of the road on which the vehicle is traveling based on the driving environment information and to correct the characteristic value x in a direction in which the greater the slope, the more difficult it is to estimate that the driver is distracted. i .

4. The driver state estimation device according to claim 2, characterized in that: The controller is configured to obtain the curvature of the road on which the vehicle is traveling based on the driving environment information, and to correct the feature value x in a direction in which it is difficult to estimate that the driver is distracted as the curvature increases. i .

5. The driver state estimation device according to claim 2, characterized in that: The controller is configured to obtain the illuminance outside the vehicle based on the driving environment information and to correct the feature value x in a direction in which it is difficult to estimate that the driver is distracted as the illuminance decreases. i .

6. The driver state estimation device according to claim 2, characterized in that: The controller is configured to obtain the speed of the vehicle based on the driving environment information and to correct the feature value x so that the higher the speed, the less likely it is to estimate that the driver is distracted. i .

Images