Driver abnormality sign detection apparatus

Abstract

To provide a driver abnormality sign detection apparatus capable of highly accurately detecting an abnormality sign state of a driver, even with limited calculation resources, taking into consideration an influence of a saliency distribution on motion of a visual line of the driver.SOLUTION: A driver abnormality sign detection apparatus 100 includes a controller 10 for detecting an abnormality sign state of a driver on the basis of visual line information acquired from a visual line detection device and image data acquired from an out-vehicle camera. The controller acquires a predetermined reference value of a saccade of the driver, calculates dispersion of a part including a saliency peak in a visual field of the driver on the basis of the image data, corrects the reference value with a correction value acquired on the basis of the dispersion of the part including the saliency peak, thereby calculating a prediction value of the saccade, and detects the abnormality sign state of the driver on the basis of a visual line abnormality degree representing a degree of deviation of an actual value of the saccade from the prediction value.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a driver abnormality sign detection device for detecting an abnormality sign state of a driver while driving a vehicle. [Background technology]

[0002] In recent years, development of driver abnormality response systems has been progressing, which detect abnormalities when the driver falls into a state where he or she cannot drive safely and automatically stop the vehicle. For example, when a driver abnormality is detected by detecting a change in the driver's posture, it is assumed that the system will gradually decelerate the vehicle while maintaining the lane, and if possible, move the vehicle to the shoulder of the road and automatically stop it.

[0003] In order to safely stop the vehicle while avoiding lane departure and contact with obstacles when a driver abnormality occurs, it is preferable to shorten the time from when the driver abnormality occurs to when it is detected as much as possible while preventing erroneous detection. Therefore, a vehicle control device aimed at improving the accuracy of driver abnormality determination has been proposed (for example, see Patent Document 1). The device described in Patent Document 1 calculates the saliency of the direction in which the driver's gaze is directed, and determines the driver's state based on the tendency that the driver's gaze is attracted to an area with high saliency. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-77140 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology described above, in order to accurately identify the degree of coincidence between the gaze direction and the high saliency area, it is necessary to obtain high-resolution saliency distribution data. However, since the calculation load for calculating the saliency distribution from image data is high, it is difficult to calculate the saliency distribution with a resolution sufficient to identify the degree of coincidence between the gaze direction and the high saliency area with the calculation resources of the in-vehicle computer.

[0006] The present invention has been made to solve such problems, and aims to provide a driver abnormality precursor detection device that is capable of detecting a driver's abnormality precursor state with high accuracy, even with limited computing resources, by taking into account the effect that saliency distribution has on the driver's line of sight movement. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention provides a driver abnormality sign detection device that detects an abnormality sign state of a driver driving a vehicle, comprising: a gaze detection device that detects the driver's gaze; an exterior camera that photographs the surroundings of the vehicle and outputs image data; and a controller configured to detect the driver's abnormality sign state based on gaze information acquired from the gaze detection device and image data acquired from the exterior camera, wherein the controller is configured to acquire a predetermined reference value of the frequency and / or amplitude of the driver's saccades, calculate a variation in a location including a saliency peak in the distribution of saliency in the driver's visual field based on the image data, acquire a correction value for correcting the reference value based on the variation in the location including the saliency peak, calculate a predicted value of the frequency and / or amplitude of the saccades by correcting the reference value with the correction value, calculate a gaze abnormality degree that indicates the degree to which the actual value of the saccade frequency and / or amplitude acquired based on the gaze information deviates from the predicted value, and determine that the driver's abnormality sign state has been detected based on the gaze abnormality degree.

[0008] According to the present invention configured as described above, the controller obtains a predetermined reference value of the frequency and / or amplitude of the driver's saccade, calculates the variation of the saliency peak in the distribution of saliency in the driver's field of view based on the image data obtained from the outside camera, and calculates the predicted value of the frequency and / or amplitude of the saccade by correcting the reference value with the correction value obtained based on the variation of the saliency peak. Therefore, it is possible to obtain a predicted value of the gaze movement that takes into account the influence of the saliency distribution on the gaze movement of the driver, even with limited calculation resources, on the degree to which the actual measured value of the gaze movement deviates from the predicted value of the gaze movement of a healthy driver, and it is possible to detect the abnormality prediction state of the driver with higher accuracy based on the state of the gaze movement that has been grasped.

[0009] In the present invention, the correction value is preferably set so as to correct the reference value in a direction in which the reference value increases as the variation in a portion including a saliency peak increases.

[0010] According to the present invention configured as described above, it is possible to obtain a predicted value of the eye movement that takes into account the tendency of the driver to make more saccades when the saliency variation is large. This makes it possible to detect the driver's abnormal sign state with high accuracy.

[0011] In the present invention, the correction value is preferably set to correct the reference value in a direction in which the reference value of the saccade frequency increases as the variation in the point including the saliency peak increases, and the controller is configured to obtain the reference value of the saccade frequency and calculate a predicted value of the saccade frequency by correcting the reference value with the correction value.

[0012] According to the present invention thus configured, the correction value is set so as to correct the reference value in the direction in which the reference value of the frequency of saccades increases as the variation in the portion including the peak of saliency increases, and the controller calculates the predicted value of the frequency of saccades by correcting the reference value of the frequency of saccades with the correction value, so that it is possible to obtain a predicted value of the movement of the line of sight that takes into account the tendency of the driver's saccade frequency to increase when the variation in saliency is large. This makes it possible to detect the driver's abnormal sign state with high accuracy.

[0013] In the present invention, the controller is preferably configured to obtain predetermined reference values ​​for the frequency and amplitude of the driver's saccades, calculate predicted values ​​for the frequency and amplitude of the saccades by correcting the reference values ​​with a correction value, accumulate two-dimensional data in which the difference between the actual and predicted values ​​of the saccade frequency is set as a first variable and the difference between the actual and predicted values ​​of the saccade amplitude is set as a second variable, and calculate the degree of gaze abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the center of gravity of the set of accumulated two-dimensional data.

[0014] According to the present invention thus configured, the controller accumulates two-dimensional data in which the difference between the actual value and the predicted value of the saccade frequency is set as a first variable, and the difference between the actual value and the predicted value of the saccade amplitude is set as a second variable, and calculates the degree of gaze abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the center of gravity of the accumulated two-dimensional data set, so that the degree of deviation of the gaze movement from a normal state can be comprehensively expressed with one index. Therefore, even at a sufficiently early stage before the driver reaches an abnormal state where it is difficult to drive, where the deterioration of the driver's driving function cannot be detected only by either the saccade frequency or the amplitude, the abnormality precursor state can be detected early and with high accuracy. Effect of the Invention

[0015] According to the driver abnormality prediction device of the present invention, even with limited computational resources, it is possible to detect the driver's abnormality prediction state with high accuracy by taking into account the influence that the distribution of saliency has on the driver's line of sight movement. [Brief description of the drawings]

[0016] [Figure 1] 1 is an explanatory diagram of a vehicle equipped with a driver abnormality sign detection device according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram of a driver abnormality sign detection device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a control block diagram of abnormality sign detection according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram of an eye gaze model according to an embodiment of the present invention. [Diagram 5] 1 is a diagram showing an example of a distribution of saliency obtained based on a signal output from an exterior camera according to an embodiment of the present invention; FIG. [Figure 6] 13 is a diagram showing a correction map used to calculate a predicted value of saccade frequency in abnormality sign detection according to an embodiment of the present invention. FIG. [Figure 7] 13 is a diagram showing a correction map used to calculate a predicted value of saccade amplitude in abnormality sign detection according to an embodiment of the present invention. FIG. [Figure 8] 11 is a time chart showing a comparison between actual measured values ​​of saccade frequency and amplitude according to an embodiment of the present invention and predicted values ​​according to a gaze model. [Figure 9] 1 is a diagram showing a total abnormality degree in an embodiment of the present invention, expressed in a three-dimensional orthogonal coordinate system with the gaze abnormality degree, the driving operation abnormality degree, and the driving risk as the respective coordinate axes. [Figure 10] 10 is a flowchart of a gaze abnormality degree calculation process according to an embodiment of the present invention. [Figure 11] 4 is a flowchart of a driving operation abnormality degree calculation process according to an embodiment of the present invention. [Figure 12] 4 is a flowchart of a driving risk calculation process according to an embodiment of the present invention. [Figure 13]4 is a flowchart of an abnormality sign detection process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A driver abnormality sign detection device according to an embodiment of the present invention will now be described with reference to the accompanying drawings.

[0018] [System Configuration] First, the configuration of a driver abnormality sign detection device according to the present embodiment will be described with reference to Figures 1 and 2. Figure 1 is an explanatory diagram of a vehicle equipped with a driver abnormality sign detection device, and Figure 2 is a block diagram of the driver abnormality sign detection device.

[0019] The vehicle 1 according to this embodiment is equipped with a driving force source 2 such as an engine or an electric motor that outputs driving force, a transmission 3 that transmits the driving force output from the driving force source 2 to the driving wheels, a brake 4 that applies a braking force to the vehicle 1, and a steering device 5 for steering the vehicle 1.

[0020] The driver abnormality sign detection device 100 is configured to detect an abnormality sign state of the driver of the vehicle 1 and, if necessary, control the vehicle 1 or drive assistance control. As shown in Fig. 2, the driver abnormality sign detection device 100 has a controller 10, a plurality of sensors, a plurality of control systems, and a plurality of information output devices.

[0021] Specifically, the sensors include an outside camera 21 for acquiring driving environment information of the vehicle 1, a radar 22, a navigation system (navi system) 23 for detecting the position of the vehicle 1, and a positioning system 24. The sensors also include a vehicle speed sensor 25, an acceleration sensor 26, a yaw rate sensor 27, a steering angle sensor 28, a steering torque sensor 29, an accelerator sensor 30, and a brake sensor 31 for detecting the behavior of the vehicle 1 and the driving operation by the driver. The sensors also include an in-vehicle camera 32 for detecting the line of sight of the driver. The control systems include a powertrain control module (PCM) 33 for controlling the driving force source 2 and the transmission 3, a dynamic stability control system (DSC) 34 for controlling the driving force source 2 and the brake 4, and an electric power steering system (EPS) 35 for controlling the steering device 5. The information output devices include a display 36 for outputting image information, and a speaker 37 for outputting audio information.

[0022] Other sensors may also include a surrounding sonar that measures the distance and position of surrounding structures relative to vehicle 1, a corner radar that measures the approach of surrounding structures at the four corners of vehicle 1, and various sensors that detect the driver's condition (e.g., a heart rate sensor, an electrocardiogram sensor, a steering wheel grip force sensor, etc.).

[0023] The controller 10 executes various calculations based on signals received from a plurality of sensors, transmits control signals to the PCM 33, DSC 34, and EPS 35 for appropriately operating the driving force source 2, transmission 3, brake 4, and steering device 5, and transmits control signals to the display 36 and speaker 37 for outputting desired information. The controller 10 is composed of a computer equipped with one or more processors 10a (typically a CPU), memory 10b (ROM, RAM, etc.) for storing various programs and data, input / output devices, etc.

[0024] The exterior camera 21 captures images of the surroundings of the vehicle 1 and outputs image data. The controller 10 identifies the positions and speeds of objects (e.g., preceding vehicles, parked vehicles, pedestrians, roads, dividing lines (lane boundaries, white lines, yellow lines), traffic signals, traffic signs, stop lines, intersections, obstacles, etc.) based on the image data received from the exterior camera 21. The exterior camera 21 corresponds to an example of the "gaze parameter information acquisition device" of the present invention.

[0025] The radar 22 measures the position and speed of an object (particularly, a preceding vehicle, a parked vehicle, a pedestrian, an object fallen on the road, etc.). For example, a millimeter wave radar can be used as the radar 22. The radar 22 transmits radio waves in the traveling direction of the vehicle 1, and receives a reflected wave generated when the transmitted wave is reflected by the object. Then, the radar 22 measures the distance between the vehicle 1 and the object (for example, the inter-vehicle distance) and the relative speed of the object with respect to the vehicle 1 based on the transmitted wave and the received wave. Note that in this embodiment, instead of the radar 22, a laser radar, an ultrasonic sensor, etc. may be used to measure the distance to the object and the relative speed. Also, a position and speed measuring device may be configured using a plurality of sensors.

[0026] The navigation system 23 stores map information internally and can provide the map information to the controller 10. The controller 10 identifies roads, intersections, traffic signals, buildings, etc. that exist around the vehicle 1 (particularly in the traveling direction) based on the map information and current vehicle position information. The map information may be stored in the controller 10. The positioning system 24 is a GPS system and / or a gyro system, and detects the position of the vehicle 1 (current vehicle position information).

[0027] The vehicle speed sensor 25 detects the speed of the vehicle 1 based on, for example, the rotation speed of the wheels or the drive shaft. The acceleration sensor 26 detects the acceleration of the vehicle 1. This acceleration includes the acceleration in the longitudinal direction of the vehicle 1 and the acceleration in the lateral direction (i.e., lateral acceleration). In this specification, the acceleration includes not only the rate of change of the speed in the direction in which the speed increases, but also the rate of change of the speed in the direction in which the speed decreases (i.e., deceleration). The yaw rate sensor 27 detects the yaw rate of the vehicle 1. The vehicle speed sensor 25 corresponds to an example of the "gaze parameter information acquisition device" in the present invention.

[0028] The steering angle sensor 28 detects the rotation angle (steering angle) of the steering wheel of the steering device 5. The steering torque sensor 29 detects the torque (steering torque) applied to the steering shaft via the steering wheel. The accelerator sensor 30 detects the amount of depression of the accelerator pedal. The brake sensor 31 detects the amount of depression of the brake pedal. In other words, the steering angle sensor 28, the steering torque sensor 29, the accelerator sensor 30 and the brake sensor 31 detect the driving operation of the driver, and the steering angle sensor 28 corresponds to an example of the "gaze parameter information acquisition device" in the present invention.

[0029] The in-vehicle camera 32 captures an image of the driver and outputs image data. The controller 10 detects the behavior of the driver's head and the direction of his / her gaze based on the image data received from the in-vehicle camera 32. The in-vehicle camera 32 corresponds to an example of the "gaze detection device" and the "gaze parameter information acquisition device" of the present invention.

[0030] The PCM 33 controls the driving force source 2 of the vehicle 1 to adjust the driving force of the vehicle 1. For example, the PCM 33 controls the engine's spark plugs, fuel injection valves, throttle valves, variable valve mechanisms, the transmission 3, and an inverter that supplies power to the electric motor. When it is necessary to accelerate or decelerate the vehicle 1, the controller 10 transmits a control signal to the PCM 33 to adjust the driving force.

[0031] The DSC 34 controls the driving force source 2 and the brakes 4 of the vehicle 1 to perform deceleration control and attitude control of the vehicle 1. For example, the DSC 34 controls the hydraulic pump and valve unit of the brakes 4, and controls the driving force source 2 via the PCM 33. When it is necessary to perform deceleration control or attitude control of the vehicle 1, the controller 10 transmits a control signal to the DSC 34 to adjust the driving force or generate a braking force.

[0032] The EPS 35 controls the steering device 5 of the vehicle 1. For example, the EPS 35 controls an electric motor that applies torque to a steering shaft of the steering device 5. When it is necessary to change the traveling direction of the vehicle 1, the controller 10 transmits a control signal to the EPS 35 to change the steering direction.

[0033] The display 36 is provided in front of the driver in the vehicle cabin and displays image information to the driver. For example, a liquid crystal display or a head-up display is used as the display 36. The speaker 37 is provided in the vehicle cabin and outputs various types of audio information.

[0034] [Driver Abnormality Prediction Detection Overview] Next, a basic concept of driver abnormality sign detection executed by the above-described controller 10 in this embodiment will be described with reference to Fig. 3. Fig. 3 is a control block diagram of abnormality sign detection according to this embodiment.

[0035] In the abnormality prediction state before the driver reaches an abnormal state where driving is difficult, it is considered that the driver's driving function is at least temporarily deteriorating due to a minor illness, aging, etc. When the driver's driving function is deteriorating, the driver's driving behavior, for example, the confirmation behavior of checking the driving environment and the operation behavior of driving the vehicle, correspondingly changes. Furthermore, as a result of the change in driving behavior, it is considered that the risk (driving risk) of lane departure, approaching an obstacle, etc. increases. Therefore, the present inventors thought that it would be possible to detect the abnormality prediction state early and with high accuracy without detecting a clear deterioration of each driving function by comprehensively determining these changes in driving behavior and the increase in driving risk.

[0036] Here, the driving function of the driver includes a perception function for perceiving objects present in the driving environment, and attention functions such as a function for simultaneously viewing multiple objects in the driving environment (distributed attention function), a function for selecting and viewing one of multiple objects (selective attention function), a function for switching objects (switchable attention function), and a function for continuing to view an object (sustained attention function). The driving function of the driver also includes motor functions necessary for operating the steering wheel, accelerator pedal, brake pedal, etc. of the vehicle.

[0037] As a result of research by the present inventors, it was found that when the perceptual function, attention function, or motor function is deteriorated, the driver's confirmation behavior changes as a result of changes in the movement of the driver's gaze. For example, when the perceptual function is deteriorated due to visual field loss or the like, the range of gaze movement may become narrower. Also, when the attention function is deteriorated, the frequency of gaze movement decreases, and the distance of gaze movement becomes shorter. Furthermore, when the motor function is deteriorated due to paralysis of the limbs or the like, the driver may be concerned about the condition of the paralyzed limbs and may turn his / her gaze in the direction of the paralyzed limbs. Therefore, by determining the degree of abnormality in the driver's gaze movement based on the gaze movement according to the driving environment and vehicle condition in a state without a disease (healthy state), it is possible to grasp the change in the confirmation behavior due to the deterioration of the driver's perceptual function, attention function, or motor function.

[0038] In addition, it has been found that when the perceptual function, attention function, or motor function is deteriorated, the operation behavior of the steering wheel, accelerator pedal, and brake pedal is changed, and the operation of the steering wheel, accelerator pedal, and brake pedal is changed. For example, when the motor function is deteriorated, the operation of the steering wheel and the pedal may be delayed compared to when the driver is in a healthy state. In addition, when the perceptual function or attention function is deteriorated, it affects the driver's recognition of the driving environment, which is the basis of the driving operation, and therefore the operation of the steering wheel and the pedal may become unstable compared to when the driver is in a healthy state, for example, due to the driver being unable to grasp the position of the vehicle in the lane well, or the delay or overlooking of the object. Therefore, by calculating the abnormality degree of the driver's driving operation based on the driving operation according to the driving environment and the vehicle state in the healthy state, it is possible to grasp the change in the operating behavior due to the deterioration of the driver's perceptual function, attention function, or motor function.

[0039] In addition, when the driver's perception function, attention function, or motor function is deteriorated, the driving risk such as lane departure and approach to obstacles increases due to delay or oversight of the object and unstable driving operation. According to the research by the present inventors, in a healthy driver, even if the driving risk increases temporarily due to inattention or disturbance of operation, the driving risk is immediately reduced because an appropriate corrective operation is performed after that. Therefore, the average value of the driving risk remains at a low level. In contrast, in a driver with a deteriorated driving function, it becomes gradually difficult to perform driving operations to reduce the driving risk, so the driving risk gradually increases. As a result, the average value of the driving risk becomes higher than that of a healthy person, and the trend of increasing the driving risk continues. Therefore, by determining the average value and increasing trend of the driving risk, the influence of the change in driving behavior due to the deterioration of the driver's motor function, perception function, or attention function can be grasped.

[0040] Therefore, the controller 10 of this embodiment calculates the degree of abnormality of the driver's gaze (gaze abnormality degree), the degree of abnormality of the driver's driving operation (driving operation abnormality degree), and the driving risk based on the driver's gaze, driving operation, the driving environment of the vehicle 1, etc. Then, the controller 10 calculates the overall abnormality degree (overall abnormality degree) of the driver's state based on the gaze abnormality degree, driving operation abnormality degree, and driving risk, and is configured to detect the driver's abnormality precursor state based on the overall abnormality degree.

[0041] 3, the controller 10 acquires gaze information based on a signal received from the in-vehicle camera 32, acquires driving operation information based on signals received from the steering angle sensor 28, the steering torque sensor 29, the accelerator sensor 30, and the brake sensor 31, and acquires driving environment information and vehicle state information based on signals received from sensors including the outside camera 21, the radar 22, the navigation system 23, the positioning system 24, the vehicle speed sensor 25, the acceleration sensor 26, and the yaw rate sensor 27. The controller 10 also calculates gaze parameters based on signals received from the outside camera 21, the vehicle speed sensor 25, the steering angle sensor 28, and the in-vehicle camera 32.

[0042] The controller 10 inputs a predetermined reference value of the feature amount representing the gaze movement and the calculated gaze parameters to the gaze model, and calculates a predicted value (predicted gaze value) of the gaze movement of a healthy driver. For example, the amplitude and frequency of saccades can be used as the feature amount representing the gaze movement.

[0043] Here, a saccade is a saccadic eye movement for capturing a visual target on the fovea of ​​the retina, and refers to an eye movement in which the line of sight moves from a fixation point where the line of sight remains for a predetermined period of time to the next fixation point. The amplitude of a saccade refers to the amount of movement when the driver's line of sight moves from a fixation point to the next fixation point, and the frequency of a saccade refers to the number of times the line of sight moves from a fixation point to the next fixation point within a predetermined period of time.

[0044] The movement of the line of sight is affected by the driver's steering operation, the vehicle state (e.g., vehicle speed, etc.), the driving environment (e.g., variations in illuminance and saliency, etc.), the driver's head behavior, etc. Here, saliency is a characteristic that indicates how easily an object attracts human attention, and refers to a visual feature determined by the temporal and spatial arrangement of color, brightness, movement, etc. In other words, an area with high saliency in the driver's visual field is an area that is likely to attract the driver's attention, for example, because it has a large color difference or brightness difference from the surrounding areas, or because it is moving significantly.

[0045] For example, when a driver is performing steering operation at an intersection or in a situation where there are successive curves, the driver focuses his / her gaze in the direction in which the vehicle is turning, so the variance in the gaze tends to decrease. In addition, when the vehicle speed is relatively high on a highway, the driver's gaze tends to focus in the direction of travel. In addition, when the illumination of the driving environment is relatively low, such as at night, the driver's gaze also tends to focus in the direction of travel. In addition, when the variance in saliency in the driver's field of vision is small (i.e., when areas with high saliency are concentrated in a part of the driver's field of vision), the driver's gaze tends to focus in areas with high saliency. In addition, the driver's gaze tends to be biased in the direction in which the driver's face is facing.

[0046] Therefore, the gaze model is configured to output a gaze prediction value for a healthy driver when parameters (gaze parameters) that affect gaze movement, such as steering operations, vehicle conditions (vehicle speed), driving environment (saliency variation, illuminance), and driver head behavior (face direction), are input, taking into account the influence of these parameters.

[0047] Fig. 4 is a block diagram of the gaze model according to this embodiment. As shown in Fig. 4, the gaze model has reference values ​​for the saccade frequency and amplitude in a normal state. These reference values ​​can be set in advance, for example, and machine learning can be performed each time the vehicle 1 is driven to reflect individual differences between drivers in the reference values.

[0048] The gaze model also includes a vehicle state influence model for correcting the reference value based on the influence of steering operation and vehicle state on gaze movement, a driving environment model for correcting the reference value based on the influence of the driving environment, and a head behavior influence model for correcting the reference value based on the influence of the driver's head behavior.

[0049] The vehicle state affecting model is configured to output correction values ​​of the saccade frequency and amplitude corresponding to the standard deviation of the steering angle and the average vehicle speed value when the standard deviation of the steering wheel operation amount (steer angle) and the average vehicle speed value during a predetermined time (e.g., 30 seconds) are input as parameters representing the steering operation and the vehicle state. Specifically, the vehicle state affecting model is configured with a correction map that defines the relationship between the standard deviation of the steering angle and the average vehicle speed value and the correction values ​​of the saccade frequency and amplitude. Then, when the standard deviation of the steering angle and the average vehicle speed are input to the vehicle state affecting model, the vehicle state affecting model outputs correction values ​​of the saccade frequency and amplitude corresponding to the input standard deviation of the steering angle and the average vehicle speed value based on the correction map.

[0050] The driving environment influence model is configured to output correction values ​​of the saccade frequency and amplitude corresponding to the saliency variation and the average illuminance value of the driving environment when the saliency variation and the average illuminance value of the driving environment are input as parameters representing the driving environment. Specifically, the driving environment influence model is configured with a correction map that defines the relationship between the saliency variation and the average illuminance value of the driving environment and the correction values ​​of the saccade frequency and amplitude. When the saliency variation and the average illuminance value of the driving environment are input to the driving environment influence model, the driving environment influence model outputs correction values ​​of the saccade frequency and amplitude corresponding to the input saliency variation and the average illuminance value of the driving environment based on the correction map.

[0051] Fig. 5 shows an example of the distribution of saliency acquired based on the signal output from the outside camera 21, where Fig. 5(a) shows a state where the saliency variation is large using high-resolution saliency distribution data, Fig. 5(b) shows a state where the saliency variation is large using low-resolution saliency distribution data, Fig. 5(c) shows a state where the saliency variation is small using high-resolution saliency distribution data, and Fig. 5(d) shows a state where the saliency variation is small using low-resolution saliency distribution data. In Fig. 5, the closer the color is to white, the higher the saliency, and the closer the color is to black, the lower the saliency.

[0052] In order to identify the tendency of the driver's gaze to be attracted to areas with high saliency as in the conventional technology, it is necessary to acquire high-resolution (40×30=1200 pixels) saliency distribution data as shown in Fig. 5(a) and Fig. 5(c) in order to accurately identify the degree of agreement between the gaze direction and the high saliency area. However, since the calculation load for calculating the saliency distribution from image data is high, it is difficult to calculate a saliency distribution with a resolution sufficient to identify the degree of agreement between the gaze direction and the high saliency area with the calculation resources of the on-board computer.

[0053] Therefore, in this embodiment, instead of specifying the degree of agreement between the direction of the line of sight and the area with high saliency, the variation of the high saliency area in the driver's field of view is obtained in order to calculate the predicted value of the movement of the line of sight taking into account the influence of the distribution of saliency on the movement of the driver's line of sight. In this case, it is not necessary to calculate high-resolution saliency distribution data as in the case of specifying the degree of agreement between the direction of the line of sight and the area with high saliency. For example, even low-resolution (10×6=60 pixels) saliency distribution data as shown in FIG. 5(b) or FIG. 5(d) has sufficient resolution to obtain the variation of the high saliency area.

[0054] Specifically, the controller 10 calculates saliency distribution data in the driver's field of view based on image data acquired from the outside-vehicle camera 21. Then, the controller 10 calculates the variance of the coordinates of the points including the saliency peak in the saliency distribution, for example, the points where the height of the saliency is equal to or higher than the 75th percentile value (points surrounded by white circles in Fig. 5(b) and Fig. 5(d)), as the saliency variation. For example, in Fig. 5(b), there are multiple points including the saliency peak, but in Fig. 5(d), there is only one point including the saliency peak, so the saliency variation is larger in Fig. 5(b) than in Fig. 5(d). When the saliency variation calculated in this way is input to the driving environment influence model, the correction values ​​of the saccade frequency and amplitude corresponding to the input saliency variation are output based on the correction map.

[0055] The head behavior influence model is configured to output correction values ​​of the saccade frequency and amplitude according to the standard deviation of the face direction of the driver during a predetermined time (e.g., 30 seconds) when the standard deviation of the face direction of the driver is input as a parameter representing the driver's head behavior. Specifically, the head behavior influence model is configured with a correction map that defines the relationship between the standard deviation of the face direction and the correction values ​​of the saccade frequency and amplitude. When the standard deviation of the face direction is input to the head behavior influence model, the head behavior influence model outputs correction values ​​of the saccade frequency and amplitude corresponding to the input standard deviation of the steering angle and vehicle speed based on the correction map.

[0056] Here, the correction maps set in the vehicle state influence model, the driving environment influence model, and the head behavior influence model will be described with reference to FIG. 6 and FIG. 7. FIG. 6 is a diagram showing an example of the correction map used to calculate the predicted value of the saccade frequency. FIG. 7 is a diagram showing an example of the correction map used to calculate the predicted value of the saccade amplitude. The inventors had more than 100 healthy subjects drive vehicles, and measured the time changes of the average vehicle speed, the standard deviation of the steering angle, the variance of the saliency, the average illuminance, and the standard deviation of the face direction, and the time changes of the frequency and amplitude of the saccades. Then, by machine learning, the influence of each of the average vehicle speed, the standard deviation of the steering angle, the variance of the saliency, the average illuminance, and the standard deviation of the face direction on the frequency and amplitude of the saccades was extracted. Based on the results, the correction map of the saccade frequency shown in FIG. 6 and the correction map of the saccade amplitude shown in FIG. 7 were created.

[0057] In the saccade frequency correction map shown in Fig. 6, the horizontal axis indicates the magnitude of the average vehicle speed, the standard deviation of the steering angle, the saliency variation, the average illuminance, and the standard deviation of the face direction, and the vertical axis indicates the correction value to be added to the reference value of the saccade frequency. That is, when the correction value is larger than 0, the reference value of the saccade frequency is corrected in the increasing direction, and when the correction value is smaller than 0, the reference value of the saccade frequency is corrected in the decreasing direction.

[0058] Similarly, in the saccade frequency correction map shown in Fig. 7, the horizontal axis indicates the magnitude of the average vehicle speed, the standard deviation of the steering angle, the saliency variation, the average illuminance, and the standard deviation of the face direction, and the vertical axis indicates the correction value to be added to the reference value of the saccade amplitude. That is, when the correction value is larger than 0, the reference value of the saccade amplitude is corrected in the increasing direction, and when the correction value is smaller than 0, the reference value of the saccade amplitude is corrected in the decreasing direction.

[0059] Of the correction maps shown in Figures 6 and 7, the correction maps for the average vehicle speed and the standard deviation of the steering angle are set in the vehicle state influence model, the correction maps for the saliency variation and the average illuminance value are set in the driving environment influence model, and the correction map for the standard deviation of the face direction is set in the head behavior influence model.

[0060] For example, as shown in Fig. 6, the correction value is set so that the saccade frequency decreases as the average vehicle speed increases. On the other hand, as shown in Fig. 7, the correction value is set so that the saccade amplitude decreases as the average vehicle speed increases, but the effect on the saccade amplitude decreases when the average vehicle speed increases to a certain degree. This is thought to indicate that the driver's gaze tends to be concentrated in the traveling direction as the vehicle speed increases, but the range of gaze movement is maintained to a certain degree.

[0061] As for the influence of steering operation, the influence of the standard deviation of the steering angle on the frequency of saccades is not large, as shown in Fig. 6. On the other hand, as shown in Fig. 7, the correction value is set so that the saccade amplitude becomes smaller as the standard deviation of the steering angle increases, but the saccade amplitude gradually increases when the standard deviation of the steering angle reaches a certain level. This is thought to indicate that although the driver turns his / her gaze to various objects regardless of the magnitude of the steering operation, the range of gaze itself tends to be concentrated in the direction in which the vehicle turns due to the steering operation.

[0062] As for the effect of saliency variation, as shown in Fig. 6, the correction value is set so that the saccade frequency gradually increases as the saliency variation increases. On the other hand, as shown in Fig. 7, the effect of saliency variation on saccade amplitude is not large. This is thought to indicate that when high saliency points are widely distributed, the number of targets to which the driver looks increases, so the frequency of moving the eyes increases, but since there are still targets to look at regardless of the level of saliency (for example, vehicles ahead or obstacles), the range of gaze itself tends not to change significantly.

[0063] As for the influence of the average illuminance, a correction value is set so that the saccade frequency increases as the average illuminance increases, as shown in Fig. 6. On the other hand, as shown in Fig. 7, the influence of the average illuminance on the saccade amplitude is not large. This is thought to indicate that during the daytime, when the average illuminance is high, there are more objects to which the driver's gaze is directed than at night, when the average illuminance is low, so the frequency of moving the driver's gaze increases, but since there are still objects to which the driver's gaze is directed (e.g., vehicles ahead and obstacles) around the vehicle even at night, the range to which the driver's gaze is directed tends not to change significantly.

[0064] As for the influence of the standard deviation of the face direction, the correction value is set so that the frequency and amplitude of the saccades increase as the standard deviation of the face direction increases, as shown in Figures 6 and 7. This is considered to indicate that the driver's line of sight is biased in the direction in which the driver's face is facing, so that the more frequently the face moves, the more frequently the line of sight moves and the wider the range in which the line of sight is directed.

[0065] In the gaze model, the reference values ​​of the saccade frequency and amplitude in a healthy state are corrected by the correction values ​​of the saccade frequency and amplitude output from each of the vehicle state influence model, the driving environment model, and the head behavior influence model configured as described above, thereby calculating the predicted values ​​of the saccade frequency and amplitude (saccade frequency / amplitude prediction). As a result, the predicted values ​​of the saccade frequency and amplitude that reflect the influence of the steering operation, the vehicle state, the driving environment, and the driver's head behavior are output.

[0066] The inventors had a healthy driver actually drive a vehicle and drove it through an urban area, a highway, and a mountain road. The measured values ​​of the saccade frequency and amplitude were compared with the predicted values ​​of the saccade frequency and amplitude output from the gaze model configured as described above. Figure 8 is a time chart showing a comparison between the measured values ​​of the saccade frequency and amplitude and the predicted values ​​by the gaze model, where Figure 8(a) shows a comparison of the saccade frequency and Figure 8(b) shows a comparison of the saccade amplitude. In Figure 8, the solid line shows the measured values ​​of the saccade frequency and amplitude, and the dashed line shows the predicted values ​​by the gaze model.

[0067] As shown in Figure 8, for both saccade frequency and amplitude, the predicted values ​​by the gaze model can predict with high accuracy the change in trend of the measured values ​​when the driving area changes from urban areas to expressways to mountain roads. In addition, the small peaks in the saccade frequency and amplitude, which are thought to be mainly due to the driver's head behavior, are also accurately reproduced by the predicted values ​​by the gaze model.

[0068] Returning to FIG. 3 , the controller 10 calculates a gaze abnormality degree indicating the degree to which the gaze actual measurement value deviates from the gaze predicted value (gaze abnormality degree calculation) based on the actual measurement value of the driver's gaze movement specified from the acquired gaze information and the gaze predicted value (saccade frequency / amplitude predicted value) output from the gaze model configured as described above.

[0069] For example, the controller 10 calculates the number of saccades per unit time as the actual value of the saccade frequency based on the number of saccades in a predetermined time (e.g., 30 seconds). The controller 10 also calculates the average value of the saccade amplitude in a predetermined time (e.g., 30 seconds) as the actual value of the saccade amplitude. Then, the controller 10 accumulates two-dimensional data in which the difference between the calculated actual value of the saccade frequency and the predicted value output from the gaze model is set as a first variable, and the difference between the calculated actual value of the saccade amplitude and the predicted value output from the gaze model is set as a second variable. The controller 10 calculates the Mahalanobis distance between the latest data point of this two-dimensional data and the center of gravity (average) of the accumulated data set. There is a correlation between the difference between the actual value and the predicted value of the saccade frequency and the difference between the actual value and the predicted value of the saccade amplitude, so by using the Mahalanobis distance in this way, the degree of deviation of the gaze movement from a normal state can be expressed by one index. Furthermore, the controller 10 calculates the gaze abnormality degree by dividing the calculated Mahalanobis distance by a preset representative value and normalizing it. As this representative value, the Mahalanobis distance when the driver is in an abnormality prediction state can be used. That is, when the gaze abnormality degree is 1, the driver is in an abnormality prediction state.

[0070] Then, the controller 10 accumulates the calculated gaze anomalous degree in a buffer (gaze anomalous degree buffer), and acquires the maximum value of the gaze anomalous degree in the most recent predetermined time period (for example, 60 seconds) from the gaze anomalous degree buffer (acquire maximum value).

[0071] The controller 10 also inputs the acquired driving environment information and vehicle state information into a driving operation prediction model to calculate a predicted value of the driving operation of a healthy driver (driving operation predicted value). As a feature quantity representing the driving operation, for example, the operation amount of the steering wheel (steer angle) and the operation amounts of the accelerator pedal and the brake pedal (accelerator pedal depression amount and brake pedal depression amount) can be used.

[0072] The driving operation prediction model is configured to output a driving operation prediction value for a healthy driver when the driving environment and vehicle state required for driving operation are input as parameters. Specifically, when the driving environment such as the positions of lane markings and obstacles in the traveling direction of the vehicle 1, the vehicle speed limit, the position and speed of the preceding vehicle, and the vehicle state such as the vehicle speed and acceleration are input as parameters, the driving operation prediction model is configured to output, as a driving operation prediction value, the operation amount of the steering wheel required for driving in the center of the driving lane, and the operation amount of the accelerator pedal and the brake pedal required for following the preceding vehicle or driving the vehicle 1 at the vehicle speed limit.

[0073] The controller 10 calculates a driving operation abnormality degree indicating the degree to which the actual driving operation value deviates from the predicted driving operation value based on the actual driving operation value (actual driving operation value) of the driver identified from the acquired driving operation information and the predicted driving operation value output from the driving operation prediction model (calculation of driving operation abnormality degree).

[0074] For example, the controller 10 accumulates two-dimensional data in which the difference between the actual value of the steering wheel operation amount and the predicted value output from the driving operation prediction model is set as a first variable, and the difference between the actual value of the accelerator pedal or brake pedal operation amount and the predicted value output from the driving operation prediction model is set as a second variable. The controller 10 calculates the Mahalanobis distance between the latest data point of this two-dimensional data and the center of gravity (average) of the accumulated data set. Since there is a correlation between the difference between the actual value and the predicted value of the steering wheel operation amount and the difference between the actual value and the predicted value of the pedal operation amount, by using the Mahalanobis distance in this way, the degree of deviation of the driving operation from a normal state can be expressed by one index. Furthermore, the controller 10 calculates the driving operation abnormality degree by dividing the calculated Mahalanobis distance by a preset representative value and normalizing it. As this representative value, the Mahalanobis distance when the driver is in an abnormality prediction state can be used. That is, when the driving operation abnormality degree is 1, the driver is in an abnormality prediction state.

[0075] The controller 10 then accumulates the calculated driving performance abnormality degree in a buffer (driving performance abnormality degree buffer), and obtains the maximum value of the driving performance abnormality degree in the most recent predetermined time period from the driving performance abnormality degree buffer (obtain maximum value).

[0076] The controller 10 also calculates a driving risk based on the acquired driving environment information and vehicle state information. The driving risk is a numerical representation of the possibility of lane departure, approach to an obstacle, and the like. For example, the driving risk is set to 0 when it is predicted that the vehicle 1 will be located in the center of the lane and the distance to the obstacle will be a safe distance (e.g., 1 m) or more after a predetermined time (e.g., 2 seconds), and the driving risk approaches 1 as the predicted values ​​of the distance to lane departure and the distance to the obstacle after a predetermined time become smaller, and the driving risk is set to 1 when it is predicted that lane departure will occur or the distance to the obstacle will be less than a limit distance (e.g., 0.3 m) after a predetermined time. That is, the controller 10 specifies the dividing line of the road in the traveling direction of the vehicle 1, the position of the vehicle 1 on the road, and the position and speed of the obstacle from the acquired driving environment information. Also, the controller 10 specifies the current speed and acceleration of the vehicle 1 from the acquired driving environment information or vehicle state information, and predicts the position of the vehicle 1 after a predetermined time. Then, based on the predicted position of vehicle 1 and the positions and speed of marking lines and obstacles, predicted values ​​of the distance until vehicle 1 deviates from the lane after a specified time and the distance to the obstacle are calculated, and the driving risk is calculated based on the calculated predicted values.

[0077] Then, the controller 10 accumulates the calculated driving risk in a buffer (risk buffer), and acquires an average value of the driving risk in the most recent predetermined time from the risk buffer (acquire average value).

[0078] The controller 10 calculates a total abnormality degree from the acquired maximum gaze abnormality degree, maximum driving operation abnormality degree, and average value of the driving risk (total abnormality degree calculation). For example, in a three-dimensional orthogonal coordinate system with the maximum gaze abnormality degree, maximum driving operation abnormality degree, and average value of the driving risk as axes, the controller 10 sets a three-dimensional vector having the acquired maximum gaze abnormality degree, maximum driving operation abnormality degree, and average value of the driving risk as components as a total abnormality degree vector, and calculates the magnitude of the total abnormality degree vector as the total abnormality degree.

[0079] FIG. 9 is a diagram showing the overall abnormality degree in a three-dimensional orthogonal coordinate system with the gaze abnormality degree, driving operation abnormality degree, and driving risk as the coordinate axes. As shown in FIG. 9, if the maximum acquired gaze abnormality degree is Gab_max, the maximum acquired driving operation abnormality degree is Oab_max, and the average driving risk is R_ave, the magnitude of the overall abnormality degree vector Tab is (Gab_max 2 +Oab_max 2 +R_ave 2 ) 1 / 2 9, the maximum value of the gaze abnormality degree Gab_max1, the maximum value of the driving operation abnormality degree Oab_max1, and the average value of the driving risk R_ave1 are all less than 1, but even in this case, the magnitude of the overall abnormality degree vector Tab is greater than 1. Furthermore, if any one of the acquired maximum value of the gaze abnormality degree, the maximum value of the driving operation abnormality degree, and the average value of the driving risk is greater than 1, the overall abnormality degree is greater than 1.

[0080] The controller 10 judges whether the calculated total abnormality degree is equal to or greater than a predetermined threshold value (threshold value judgment). When the magnitude of the total abnormality degree vector obtained as described above is calculated as the total abnormality degree, the threshold value is, for example, 1, and in the example of Fig. 9, a sphere with a radius of 1 corresponds to the threshold value.

[0081] In addition, when the controller 10 determines, based on the calculated driving risk, that the driving risk is increasing and that the rate of increase will reach 1 within a specified time (for example, within 5 seconds), if that rate is maintained, it detects an increase in driving risk (detection of increased driving risk).

[0082] Then, when the total abnormality degree is equal to or greater than the threshold value and an increase in driving risk is detected, the controller 10 determines that the driver is in an abnormality premonition state, that is, detects the driver's abnormality premonition state (abnormality premonition detection).

[0083] When an abnormality predictor state is detected, the controller 10 transmits control signals to the PCM 33, the DSC 34, and the EPS 35 for appropriately operating the driving force source 2, the transmission 3, the brake 4, and the steering device 5, and transmits control signals to the display 36 and the speaker 37 for outputting desired information. For example, the controller 10 causes the EPS 35 to vibrate the steering wheel at a predetermined frequency, and determines that the driver is in an abnormal state based on the driver's response. The controller 10 also causes the display 36 to output a warning display, and determines that the driver is in an abnormal state based on the driver's response to the warning display.

[0084] [Anomaly detection process] Next, a flow of detection processing of an abnormality sign state by the driver abnormality sign detection device 100 of this embodiment will be described with reference to Fig. 10 to Fig. 13. Fig. 10 is a flowchart of gaze abnormality degree calculation processing, Fig. 11 is a flowchart of driving operation abnormality degree calculation processing, Fig. 12 is a flowchart of driving risk calculation processing, and Fig. 13 is a flowchart of abnormality sign detection processing. The gaze abnormality degree calculation processing, driving operation abnormality degree calculation processing, driving risk calculation processing, and abnormality sign detection processing are each started when the power supply of the vehicle 1 is turned ON, and are repeatedly executed in parallel by the controller 10 at a predetermined cycle (for example, every 0.05 to 0.2 seconds).

[0085] 10 is started, the controller 10 detects the driver's gaze based on a signal received from the in-vehicle camera 32 (step S1). Next, the controller 10 calculates actual measurement values ​​of the frequency and amplitude of saccades based on the detected driver's gaze (step S2).

[0086] The controller 10 also acquires reference values ​​of the saccade frequency and amplitude in the driver's healthy state (step S3). These reference values ​​are stored in advance in the memory 10b, for example, and individual differences of the driver are reflected in the reference values ​​by executing machine learning every time the vehicle 1 travels.

[0087] Next, the controller 10 calculates the average vehicle speed based on the signal received from the vehicle speed sensor 25, calculates the standard deviation of the steering angle based on the signal received from the steering angle sensor 28, calculates the saliency variation and the average illuminance of the driving environment based on the signal received from the outside camera 21, and calculates the standard deviation of the driver's face direction based on the signal received from the inside camera 32 (step S4). That is, the controller 10 calculates the gaze parameters based on the information acquired from the gaze parameter information acquisition device.

[0088] Next, the controller 10 inputs the line of sight parameters calculated in step S4 to a vehicle state influence model, a driving environment influence model, and a head behavior influence model, and obtains correction values ​​for correcting the reference values ​​(step S5).

[0089] Next, the controller 10 calculates predicted values ​​of the frequency and amplitude of saccades by correcting the reference values ​​using the correction values ​​calculated in step S5 (step S6).

[0090] Next, the controller 10 calculates the gaze abnormality degree based on the actual measurement values ​​of the saccade frequency and amplitude calculated in step S2 and the predicted values ​​of the saccade frequency and amplitude calculated in step S6, and stores the degree in a buffer (step S7). As described above, for example, the controller 10 calculates the Mahalanobis distance between the latest data point of the two-dimensional data, in which the difference between the actual measurement value and the predicted value of the saccade frequency is set as a first variable and the difference between the actual measurement value and the predicted value of the saccade amplitude is set as a second variable, and the center of gravity (average) of the data set accumulated so far. Furthermore, the controller 10 calculates the gaze abnormality degree by dividing the calculated Mahalanobis distance by a preset representative value and normalizing it. After step S7, the controller 10 ends the gaze abnormality degree calculation process.

[0091] When the driving operation abnormality calculation process of FIG. 11 is started, the controller 10 detects the driver's driving operation, specifically, the amount of operation of the steering wheel, the amount of operation of the accelerator pedal, and the amount of operation of the brake pedal, based on signals received from the steering angle sensor 28, the steering torque sensor 29, the accelerator sensor 30, and the brake sensor 31 (step S11).

[0092] The controller 10 also acquires driving environment information and vehicle state information based on signals received from sensors including the outside camera 21, the radar 22, the navigation system 23, the positioning system 24, the vehicle speed sensor 25, the acceleration sensor 26, and the yaw rate sensor 27 (step S12).The controller 10 then inputs the information acquired in step S12 into a driving performance prediction model and calculates a predicted value of the driving performance (step S13).

[0093] Next, the controller 10 calculates a driving operation abnormality degree based on the actual measurement value of the driver's driving operation detected in step S11 and the predicted value of the driving operation calculated in step S13, and stores the degree in a buffer (step S14). As described above, for example, the controller 10 calculates the Mahalanobis distance between the latest data point of the two-dimensional data, in which the difference between the actual measurement value and the predicted value of the operation amount of the steering wheel is set as a first variable and the difference between the actual measurement value and the predicted value of the operation amount of the accelerator pedal or the brake pedal is set as a second variable, and the center of gravity (average) of the data set accumulated so far. Furthermore, the controller 10 calculates the driving operation abnormality degree by dividing the calculated Mahalanobis distance by a preset representative value and normalizing it. After step S14, the controller 10 ends the driving operation abnormality degree calculation process.

[0094] When the driving risk calculation process of Figure 12 is started, the controller 10 acquires driving environment information and vehicle state information based on signals received from sensors including the exterior camera 21, radar 22, navigation system 23, positioning system 24, vehicle speed sensor 25, acceleration sensor 26, and yaw rate sensor 27 (step S21).

[0095] Next, the controller 10 calculates a driving risk based on the driving environment information and vehicle state information acquired in step S21, and stores the calculated driving risk in a buffer (step S22). As described above, for example, the controller 10 identifies the lane markings on the road in the traveling direction of the vehicle 1, the position of the vehicle 1 on the road, and the positions and speed of obstacles from the acquired driving environment information. In addition, the controller 10 identifies the current speed and acceleration of the vehicle 1 from the acquired driving environment information or vehicle state information, and predicts the position of the vehicle 1 after a predetermined time. Then, based on the predicted position of the vehicle 1 and the positions and speed of the lane markings and obstacles, it calculates predicted values ​​of the distance to the lane departure of the vehicle 1 after a predetermined time and the distance to the obstacles, and calculates the driving risk based on the calculated predicted values.

[0096] Next, the controller 10 judges whether the driving risk is increasing based on the driving risk calculated in step S22 (step S23). As described above, for example, when the controller 10 judges that the driving risk is increasing and that the increasing rate will reach 1 within a predetermined time if it is maintained, the controller 10 judges that the driving risk is increasing.

[0097] As a result, if it is determined that the driving risk is increasing (step S23: Yes), that is, if the controller 10 detects an increase in the driving risk, the controller 10 sets the driving risk increase flag to TRUE (step S24). On the other hand, if it is not determined that the driving risk is increasing (step S23: No), the controller 10 sets the driving risk increase flag to FALSE (step S25). After the processing of step S24 or S25, the controller 10 ends the driving risk calculation processing.

[0098] When the abnormality sign detection process of FIG. 13 is started, the controller 10 acquires the maximum gaze abnormality degree in the most recent predetermined time period (for example, 60 seconds) from the gaze abnormality degree buffer (step S31).

[0099] Moreover, the controller 10 acquires the maximum value of the driving performance abnormality degree in the most recent predetermined time from the gaze abnormality degree buffer (step S32).

[0100] Furthermore, the controller 10 acquires the average value of the driving risk in the most recent predetermined time from the risk buffer (step S33).

[0101] Next, the controller 10 calculates an overall abnormality degree from the acquired maximum gaze abnormality degree, maximum driving operation abnormality degree, and average value of the driving risk (step S34). As described above, for example, in a three-dimensional orthogonal coordinate system with the maximum gaze abnormality degree, maximum driving operation abnormality degree, and average value of the driving risk as axes, the controller 10 calculates an overall abnormality degree vector and its magnitude as the overall abnormality degree.

[0102] Next, the controller 10 judges whether or not the total abnormality degree calculated in step S34 is equal to or greater than a predetermined threshold value (step S35).

[0103] When it is determined that the total abnormality degree is equal to or greater than the predetermined threshold value (step S35: Yes), the controller 10 determines whether or not the driving risk increase flag is TRUE (step S36).

[0104] When it is determined that the driving risk increase flag is TRUE (step S36: Yes), the controller 10 determines that the driver is in an abnormality premonition state, that is, detects the abnormality premonition state of the driver (step S37).

[0105] On the other hand, if the total abnormality degree is not determined to be equal to or greater than the predetermined threshold in step S35 (i.e., if the total abnormality degree is less than the threshold) (step S35: No), or if the driving risk increase flag is not determined to be TRUE in step S36 (i.e., if the driving risk increase flag is FALSE) (step S36: No), the controller 10 determines that the driver is not in an abnormality premonition state. In other words, it does not detect the driver's abnormality premonition state (step S38). After processing step S37 or S38, the controller 10 terminates the abnormality premonition detection processing.

[0106] In the above-described embodiment, in addition to the distribution of saliency in the driver's field of view, the amount of steering wheel operation of the vehicle 1, the vehicle speed, the illuminance around the vehicle 1, and the direction of the driver's face are used as gaze parameters, but any one or more of these gaze parameters may be used.

[0107] In the above embodiment, the gaze abnormality degree and the driving operation abnormality degree are calculated using the Mahalanobis distance, but the gaze abnormality degree and the driving operation abnormality degree may be calculated using other calculation methods. For example, the gaze abnormality degree may be a value obtained by normalizing the difference between the actual measured value and the predicted value of each of the saccade frequency and amplitude. Also, the driving operation abnormality degree may be a value obtained by normalizing the difference between the actual measured value and the predicted value of each of the steering wheel operation amount and the pedal operation amount.

[0108] Furthermore, in the above-described embodiment, the amplitude and frequency of saccades are used as the feature amount representing the movement of the line of sight. However, either the amplitude or the frequency of saccades may be used.

[0109] In addition, in the above-described embodiment, the operation amounts of the steering wheel, accelerator pedal, and brake pedal are used as feature quantities representing driving operations. However, any one or two of the operation amounts of the steering wheel, accelerator pedal, and brake pedal may also be used.

[0110] In the above embodiment, a three-dimensional vector having the maximum gaze abnormality degree, the maximum driving operation abnormality degree, and the average value of the driving risk as components is used as the overall abnormality degree vector, and the magnitude of the overall abnormality degree vector is calculated as the overall abnormality degree, but the overall abnormality degree may be calculated by other calculation methods. For example, the overall abnormality degree may be calculated as the sum or average value of the maximum gaze abnormality degree, the maximum driving operation abnormality degree, and the average value of the driving risk.

[0111] [Actions and Effects] Next, the effects of the driver abnormality sign detection device 100 of the above-described embodiment will be described.

[0112] The controller 10 obtains a predetermined reference value of the frequency and / or amplitude of the driver's saccade, calculates the variation of the saliency peak in the distribution of saliency in the driver's field of view based on the image data obtained from the outside camera 21, and corrects the reference value with the correction value obtained based on the variation of the saliency peak, thereby calculating the predicted value of the frequency and / or amplitude of the saccade. Therefore, it is possible to obtain a predicted value of the gaze movement that takes into account the influence of the saliency distribution on the gaze movement of the driver, even with limited calculation resources, on the degree to which the actual measured value of the gaze movement deviates from the predicted value of the gaze movement of a healthy driver, and it is possible to detect the abnormality prediction state of the driver with higher accuracy based on the state of the gaze movement that has been obtained.

[0113] In addition, the correction value is set so that the reference value is corrected in the direction in which the reference value increases as the variation in the portion including the saliency peak increases, so that it is possible to obtain a predicted value of the eye movement that takes into account the tendency of the driver's saccades to increase when the variation in saliency is large. This makes it possible to detect the driver's abnormal sign state with high accuracy.

[0114] In addition, the correction value is set so as to correct the reference value in the direction in which the reference value of the saccade frequency increases as the variation in the portion including the saliency peak increases, and the controller 10 calculates the predicted value of the saccade frequency by correcting the reference value of the saccade frequency with the correction value, so that it is possible to obtain a predicted value of the eye movement that takes into account the tendency of the driver's saccade frequency to increase when the variation in the saliency is large. This makes it possible to detect the driver's abnormal sign state with high accuracy.

[0115] In addition, the controller 10 accumulates two-dimensional data in which the difference between the actual value and the predicted value of the saccade frequency is set as a first variable, and the difference between the actual value and the predicted value of the saccade amplitude is set as a second variable, and calculates the degree of gaze abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the center of gravity of the accumulated two-dimensional data set, so that the degree of deviation of the gaze movement from a normal state can be comprehensively expressed with one index. Therefore, even at a sufficiently early stage before the driver reaches an abnormal state where it is difficult to drive, where the deterioration of the driver's driving function cannot be detected only by either the saccade frequency or the amplitude, the abnormality precursor state can be detected early and with high accuracy. [Explanation of symbols]

[0116] 1 vehicle 10 Controller 10a processor 10b Memory 100 Driver Abnormality Prediction Device 21 Exterior camera 22 Radar 23 Navigation System 24 Positioning System 25 Vehicle speed sensor 26 Acceleration Sensor 27 Yaw rate sensor 28 Steering angle sensor 29 Steering torque sensor 30 Accelerator Sensor 31 Brake sensor 32 In-car camera 33 PCM 34 DSC 35 EPS 36 Display 37 Speaker

Claims

1. A driver abnormality sign detection device that detects an abnormality sign state of a driver who drives a vehicle, comprising: A gaze detection device for detecting the gaze of the driver; an exterior camera that captures images of the surroundings of the vehicle and outputs image data; and a controller configured to detect an abnormality prediction state of the driver based on gaze information acquired from the gaze detection device and image data acquired from the exterior camera, The controller: Obtaining a predetermined reference value for the frequency and / or amplitude of the driver's saccades; Calculating the variation of a location including a saliency peak in the distribution of saliency in the driver's field of view based on the image data; obtaining a correction value for correcting the reference value based on the variation of a portion including the saliency peak; Calculating a predicted value of the frequency and / or amplitude of the saccade by correcting the reference value with the correction value; Calculating a gaze abnormality degree that indicates a degree of deviation of an actual measurement value of the frequency and / or amplitude of the saccade acquired based on the gaze information from the predicted value; The system is configured to determine that an abnormality prediction state of the driver has been detected based on the gaze abnormality degree. Driver abnormality prediction device.

2. The correction value is set so as to correct the reference value in a direction in which the reference value increases as the variation in a portion including the saliency peak increases. The driver abnormality sign detection device according to claim 1 .

3. The correction value is set so as to correct the reference value in a direction in which the reference value of the frequency of saccades increases as the variation in a portion including a peak of the saliency increases, The controller: Obtaining a reference value for the frequency of the saccades; The method is configured to calculate a predicted value of the frequency of the saccade by correcting the reference value with the correction value. The driver abnormality sign detection device according to claim 2 .

4. The controller: Obtaining predetermined baseline values ​​for the frequency and amplitude of saccades of the driver; Calculating predicted values ​​of the frequency and amplitude of the saccade by correcting the reference value with the correction value; Accumulating two-dimensional data in which a difference between an actual value and a predicted value of the frequency of the saccade is set as a first variable and a difference between an actual value and a predicted value of the amplitude of the saccade is set as a second variable; and calculating the gaze abnormality degree based on a Mahalanobis distance between a latest data point of the two-dimensional data and a center of gravity of a set of the accumulated two-dimensional data. The driver abnormality sign detection device according to any one of claims 1 to 3.

Images