Driver abnormality omen detection device

Abstract

Provided is a driver abnormality indication detection device capable of detecting an abnormality indication state of a driver with high accuracy by taking into account the influence of a distribution of significance on the movement of the line of sight of the driver, even with limited computing resources. A driver abnormality indication detection device (100) is provided with a controller (10) configured so as to detect an abnormality indication state of a driver on the basis of line-of-sight information acquired from a line-of-sight detection device and image data acquired from an exterior camera, and the controller acquires a predetermined reference value for the glancing of the driver. A prediction value of glancing is calculated by calculating a deviation of a region including a peak value of saliency in the field of view of the driver on the basis of the image data and correcting the reference value on the basis of a correction value acquired on the basis of the deviation of the region including the peak value of saliency. An abnormality indication state of the driver is detected on the basis of a line-of-sight abnormality indicating the degree to which the measured value of the glancing deviates from the predicted value.

Description

Technical Field

[0001] The invention relates to a driver abnormal sign detection device for detecting an abnormal sign state of a driver during vehicle driving. Background Art

[0002] In recent years, the development of driver abnormality response systems that detect abnormalities and automatically stop the vehicle when the driver is in a state where he cannot drive safely has been promoted. For example, if the driver's abnormality is detected by detecting the driver's posture distortion, the vehicle is gradually decelerated while maintaining the driving route, and the vehicle is automatically stopped when possible by approaching the shoulder of the road.

[0003] In order to avoid deviation from the driving route, contact with obstacles, etc. and to stop the vehicle safely when the driver's abnormality occurs, it is preferred to prevent false detection and shorten the time from the occurrence of the driver's abnormality to the detection as much as possible. Therefore, a vehicle control device for the purpose of improving the accuracy of the driver's abnormality determination is proposed (for example, refer to patent document 1). In the device described in patent document 1, the prominence of the target to which the driver's line of sight is directed is calculated, and the driver's state is determined based on the tendency of the driver's line of sight to be induced to an area with high prominence.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-77140

[0007] Technical problem to be solved by the invention

[0008] In the above-mentioned prior art, in order to accurately determine the degree of coincidence between the direction of the sight line and the high-saliency region, it is necessary to obtain high-resolution saliency distribution data. However, since the calculation load for calculating the saliency distribution from the image data is high, it is difficult to calculate the saliency distribution with a resolution sufficient to determine the degree of coincidence between the direction of the sight line and the high-saliency region using the calculation resources of the on-board computer. Summary of the invention

[0009] The present invention is made to solve such problems, and its purpose is to provide a driver abnormal sign detection device that can detect the driver's abnormal sign state with high accuracy by considering the influence of the distribution of saliency on the movement of the driver's line of sight even with limited computing resources.

[0010] Technical means for solving technical problems

[0011] In order to solve the above technical problems, the driver abnormal sign detection device of the present invention detects the abnormal sign state of the driver driving the vehicle, and comprises: a line of sight detection device, which detects the line of sight of the driver; an outside camera, which shoots the surroundings of the vehicle and outputs image data; and a controller, which is configured to detect the abnormal sign state of the driver based on the line of sight information obtained from the line of sight detection device and the image data obtained from the outside camera, and the controller is configured to obtain a predetermined reference value of the driver's scanning frequency and / or amplitude under the driver's health state, calculate the deviation of the part containing the peak of the significance in the distribution of the significance in the driver's field of vision based on the image data, obtain a correction value for correcting the reference value based on the deviation of the part containing the peak of the significance, correct the reference value by the correction value, thereby calculating the predicted value of the frequency and / or amplitude of the scanning, calculate the line of sight abnormality, the line of sight abnormality indicates the degree to which the measured value of the frequency and / or amplitude of the scanning obtained based on the line of sight information deviates from the predicted value, and determine whether the abnormal sign state of the driver is detected based on the line of sight abnormality.

[0012] According to the present invention constructed as described above, the controller obtains a predetermined reference value of the frequency and / or amplitude of the driver's glance, calculates the deviation of the portion containing the peak of the significance in the distribution of the significance in the driver's field of vision based on the image data obtained from the camera outside the vehicle, and corrects the reference value by the correction value obtained based on the deviation of the portion containing the peak of the significance, thereby calculating the predicted value of the frequency and / or amplitude of the glance, so that even if the high-resolution significance distribution data is not calculated as in the case of determining the consistency between the direction of the sight line and the area with high significance, the predicted value of the movement of the sight line that takes into account the influence of the distribution of the significance on the movement of the driver's sight line can be obtained. Thus, even with limited computing resources, it is possible to correctly grasp the degree to which the measured value of the movement of the sight line deviates from the predicted value of the movement of the sight line of a healthy driver, and the abnormal sign state of the driver can be detected more accurately based on the grasped state of the movement of the sight line.

[0013] In the present invention, it is preferable that the correction value is set so that the reference value is corrected in a direction in which the reference value increases as the deviation of the portion including the peak of significance increases.

[0014] According to the present invention thus constituted, it is possible to obtain a predicted value of the movement of the line of sight that takes into account the tendency of the driver's glance to increase when the variance in saliency is large. This makes it possible to detect the driver's abnormal sign state with high accuracy.

[0015] In the present invention, it is preferred that the correction value is set so that the greater the deviation of the portion containing the peak value of significance, the more the reference value is corrected in the direction of increasing the reference value of the frequency of sweeping. The controller is configured to obtain the reference value of the frequency of sweeping, correct the reference value by the correction value, and thereby calculate the predicted value of the frequency of sweeping.

[0016] According to the present invention constructed as above, the correction value is set such that the larger the deviation of the position including the peak value of the significance is, the more the reference value is corrected in the direction of increasing the reference value of the frequency of the glance, and the controller corrects the reference value of the frequency of the glance by the correction value, thereby calculating the predicted value of the frequency of the glance, and thus the predicted value of the movement of the line of sight that takes into account the tendency of the driver's glance frequency to increase when the deviation of the significance is large can be obtained. Thus, the abnormal omen state of the driver can be detected with high accuracy.

[0017] In the present invention, preferably, the controller is configured to obtain specified reference values ​​of the frequency and amplitude of the driver's glance, correct the reference values ​​by using correction values ​​to calculate predicted values ​​of the frequency and amplitude of the glance, accumulate two-dimensional data with the difference between the actual value and the predicted value of the glance frequency as the first variable and the difference between the actual value and the predicted value of the glance amplitude as the second variable, and calculate the degree of line of sight abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the centroid of the set of accumulated two-dimensional data.

[0018] According to the present invention constructed as described above, the controller accumulates two-dimensional data using the difference between the measured value and the predicted value of the frequency of the glance as the first variable and the difference between the measured value and the predicted value of the amplitude of the glance as the second variable, and calculates the degree of sight abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the center of gravity of the set of the accumulated two-dimensional data, so that the degree of deviation of the movement of the sight line from the healthy state can be comprehensively represented by one index. Therefore, even at a sufficiently early stage before the driver reaches an abnormal state of driving difficulty, such as a reduction in the driver's driving function cannot be detected by only either the frequency or the amplitude of the glance, the abnormal omen state can be detected early and with high accuracy.

[0019] Effects of the Invention

[0020] According to the driver abnormality sign detection device of the present invention, even with limited computing resources, it is possible to detect the driver's abnormality sign state with high accuracy by taking into account the influence of the saliency distribution on the movement of the driver's line of sight. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an explanatory diagram of a vehicle equipped with the driver abnormality sign detection device according to the embodiment of the present invention.

[0022] Figure 2This is a block diagram of a driver abnormality sign detection device according to an embodiment of the present invention.

[0023] Figure 3 This is a control block diagram of abnormality sign detection according to the embodiment of the present invention.

[0024] Figure 4 It is a block diagram of a sight line model according to an embodiment of the present invention.

[0025] Figure 5 This is a diagram showing an example of the distribution of saliency acquired based on a signal output from an exterior camera according to an embodiment of the present invention.

[0026] Figure 6 This is a diagram showing a correction map used for calculating a predicted value of a scanning frequency in abnormality sign detection according to the embodiment of the present invention.

[0027] Figure 7 This is a diagram showing a correction map used for calculating a predicted value of a saccade amplitude in abnormality sign detection according to an embodiment of the present invention.

[0028] Figure 8 It is a timing chart showing a comparison between the measured values ​​of the scanning frequency and amplitude and the predicted values ​​based on the line of sight model according to the embodiment of the present invention.

[0029] Fig. 9 This is a diagram showing the comprehensive abnormality degree in the embodiment of the present invention using a three-dimensional orthogonal coordinate system having the sight line abnormality degree, the driving operation abnormality degree, and the travel risk as coordinate axes, respectively.

[0030] Fig.10 This is a flowchart of the visual line abnormality degree calculation process according to the embodiment of the present invention.

[0031] Fig.11 This is a flowchart of a driving operation abnormality degree calculation process according to the embodiment of the present invention.

[0032] Fig.12 This is a flowchart of a travel risk calculation process according to the embodiment of the present invention.

[0033] Fig.13 This is a flowchart of abnormality sign detection processing according to the embodiment of the present invention.

[0034] Explanation of symbols

[0035] 1 Vehicle

[0036] 10 Controller

[0037] 10a processor

[0038] 10b memory

[0039] 100 Driver abnormality warning detection device

[0040] 21 External cameras

[0041] 22 Radar

[0042] 23Navigation system

[0043] 24 Positioning System

[0044] 25 Vehicle speed sensor

[0045] 26 Acceleration sensor

[0046] 27 Yaw rate sensor

[0047] 28 Steering angle sensor

[0048] 29 Steering torque sensor

[0049] 30 Acceleration sensor

[0050] 31 Brake sensor

[0051] 32 In-car cameras

[0052] 33PCM

[0053] 34DSC

[0054] 35EPS

[0055] 36 Displays

[0056] 37 speakers. DETAILED DESCRIPTION

[0057] Hereinafter, a driver abnormality sign detection device according to an embodiment of the present invention will be described with reference to the drawings.

[0058] [System Structure]

[0059] First, refer to Figure 1 and Figure 2 , the structure of the driver abnormality sign detection device based on this embodiment is described. Figure 1 This is an illustration of a vehicle equipped with a driver abnormality sign detection device. Figure 2 This is a block diagram of a driver abnormality sign detection device.

[0060] The vehicle 1 according to this embodiment includes: a driving force source 2 such as an engine or an electric motor that outputs driving force, a transmission 3 that transmits the driving force output from the driving force source 2 to driving wheels, a brake 4 that applies braking force to the vehicle 1, and a steering device 5 for steering the vehicle 1.

[0061] The driver abnormality sign detection device 100 is configured to detect the abnormality sign state of the driver of the vehicle 1 and perform control of the vehicle 1 and driving assistance control as needed. Figure 2 As shown, the driver abnormality sign detection device 100 includes a controller 10, a plurality of sensors, a plurality of control systems, and a plurality of information output devices.

[0062] Specifically, the plurality of sensors include an exterior camera 21 for acquiring information about the driving environment of the vehicle 1, a radar 22, a navigation system 23 for detecting the position of the vehicle 1, and a positioning system 24. In addition, the plurality of sensors include a vehicle speed sensor 25 for detecting the movement of the vehicle 1 based on the driving operation of the driver, an acceleration sensor 26, a yaw rate sensor 27, a steering angle sensor 28, a steering torque sensor 29, an acceleration sensor 30, and a brake sensor 31. In addition, the plurality of sensors include an interior camera 32 for detecting the driver's line of sight. The plurality of control systems include a powertrain control module (PCM) 33 for controlling the driving force source 2 and the transmission 3, a dynamic stability control system (DSC) 34 for controlling the driving force source 2 and the brake 4, and an electric power steering system (EPS) 35 for controlling the steering device 5. The plurality of information output devices include a display 36 for outputting image information and a speaker 37 for outputting sound information.

[0063] In addition, other sensor types may also include: peripheral sonar for measuring the distance and position of surrounding structures relative to the vehicle 1, corner radar for measuring the approach of surrounding structures at the four corners of the vehicle 1, and various sensors for detecting the driver's status (for example, heart rate sensors, electrocardiogram sensors, steering wheel grip force sensors, etc.).

[0064] The controller 10 performs various operations based on the signals received from the plurality of sensors, sends control signals for properly operating the driving force source 2, the transmission 3, the brake 4, and the steering device 5 to the PCM 33, the DSC 34, and the EPS 35, and sends control signals for outputting desired information to the display 36 and the speaker 37. The controller 10 is composed of a computer having one or more processors 10a (typically a CPU); a memory 10b (ROM, RAM, etc.) storing various programs and data; and an input / output device.

[0065] The vehicle exterior camera 21 captures the surroundings of the vehicle 1 and outputs image data. The controller 10 determines the position and speed of an object (e.g., a preceding vehicle, a parked vehicle, a pedestrian, a driving road, a line (driving route boundary line, white line, yellow line), a traffic signal, a traffic sign, a stop line, an intersection, an obstacle, etc.) based on the image data received from the vehicle exterior camera 21. In addition, the vehicle exterior camera 21 is equivalent to an example of the "sight parameter information acquisition device" in the present invention.

[0066] The radar 22 measures the position and speed of an object (especially, a preceding vehicle, a parked vehicle, a pedestrian, a fallen object on the road, etc.). As the radar 22, for example, a millimeter wave radar can be used. The radar 22 transmits radio waves in the direction of travel of the vehicle 1, and receives reflected waves generated when the transmitted waves are reflected by the object. Then, the radar 22 measures the distance between the vehicle 1 and the object (for example, the distance between the vehicles), and the relative speed of the object relative to the vehicle 1 based on the transmitted waves and the received waves. In addition, in the present embodiment, a laser radar, an ultrasonic sensor, etc. may be used instead of the radar 22 to measure the distance to the object and the relative speed. In addition, a plurality of sensor types may be used to form a position and speed measuring device.

[0067] The navigation system 23 stores map information internally and can provide the map information to the controller 10. The controller 10 determines the roads, intersections, traffic signals, buildings, etc. existing around the vehicle 1 (especially in the direction of travel) based on the map information and the current vehicle position information. The map information can also be stored in the controller 10. The positioning system 24 is a GPS system and / or a gyroscope system, and detects the position of the vehicle 1 (current vehicle position information).

[0068] The vehicle speed sensor 25 detects the speed of the vehicle 1 based on, for example, the rotation speed of the wheels and the drive shaft. The acceleration sensor 26 detects the acceleration of the vehicle 1. The acceleration includes the acceleration in the front-rear direction and the acceleration in the lateral direction (i.e., lateral acceleration) of the vehicle 1. In addition, in this specification, the acceleration includes the rate of change of the speed in the direction of increasing speed (i.e., deceleration). The yaw rate sensor 27 detects the yaw rate of the vehicle 1. In addition, the vehicle speed sensor 25 is equivalent to an example of the "sight parameter information acquisition device" in the present invention.

[0069] The steering angle sensor 28 detects the rotation angle (steering angle) of the steering wheel of the steering device 5. The steering torque sensor 29 detects the torque (steering torque) applied to the steering shaft via the steering wheel. The acceleration sensor 30 detects the amount of depression of the accelerator pedal. The brake sensor 31 detects the amount of depression of the brake pedal. That is, the steering angle sensor 28, the steering torque sensor 29, the acceleration sensor 30, and the brake sensor 31 detect the driving operation of the driver, and the steering angle sensor 28 is equivalent to an example of the "sight parameter information acquisition device" in the present invention.

[0070] The in-vehicle camera 32 takes a picture of the driver and outputs image data. The controller 10 detects the driver's head movement and line of sight direction based on the image data received from the in-vehicle camera 32. In addition, the in-vehicle camera 32 is equivalent to an example of the "line of sight detection device" and the "line of sight parameter information acquisition device" in the present invention.

[0071] The PCM 33 controls the driving force source 2 of the vehicle 1 to adjust the driving force of the vehicle 1. For example, the PCM 33 controls the spark plugs, fuel injection valves, throttle valves, variable valve mechanisms, transmission 3, inverters that supply power to the motor, etc. When the vehicle 1 needs to be accelerated or decelerated, the controller 10 sends a control signal to the PCM 33 to adjust the driving force.

[0072] The DSC 34 controls the driving force source 2 and the brake 4 of the vehicle 1 to perform deceleration control and posture control of the vehicle 1. For example, the DSC 34 controls the hydraulic pump and valve unit of the brake 4, and controls the driving force source 2 via the PCM 33. When deceleration control and posture control of the vehicle 1 are required, the controller 10 sends a control signal to the DSC 34 to adjust the driving force or generate a braking force.

[0073] The EPS 35 controls the steering device 5 of the vehicle 1. For example, the EPS 35 controls a motor that applies torque to a steering shaft of the steering device 5. When the traveling direction of the vehicle 1 needs to be changed, the controller 10 sends a control signal to the EPS 35 to change the steering direction.

[0074] The display 36 is provided in front of the driver in the vehicle interior, and displays image information to the driver. For example, a liquid crystal display or a head-up display can be used as the display 36. The speaker 37 is provided in the vehicle interior, and outputs various sound information.

[0075] [Overview of driver abnormality sign detection]

[0076] Next, refer to Figure 3 , the basic concept of the driver abnormality sign detection executed by the above-mentioned controller 10 in this embodiment is explained. Figure 3 This is a control block diagram of abnormality sign detection according to the present embodiment.

[0077] In the abnormal omen state before the driver reaches the abnormal state of driving difficulty, it can be considered that the driver's driving function begins to decrease at least temporarily due to mild illness, aging, etc. If the driver's driving function decreases, the driver's driving actions, such as confirmation actions for confirming the driving environment and operating actions for driving the vehicle, will change accordingly. Moreover, it can also be considered that as a result of the change in driving actions, the risk of deviating from the driving route, approaching obstacles, etc. (driving risks) increases. Therefore, the inventors of the present application believe that by comprehensively determining the changes in these driving actions and the increase in driving risks, the abnormal omen state can be detected early and with high accuracy without detecting a significant decrease in individual driving functions.

[0078] Here, the driver's driving function includes: a perception function of perceiving objects in the driving environment, a function of simultaneously observing multiple objects in the driving environment (distributed attention function), a function of selecting objects to observe from multiple objects (selective attention function), a function of switching objects (conversion attention function), and a function of continuously observing objects (continuous attention function). In addition, the driver's driving function also includes motor functions required to operate the vehicle's steering wheel, accelerator pedal, brake pedal, etc.

[0079] According to the results of the research of the inventors of this application, the following insights are obtained: if the perception function, attention function, or motor function is reduced, the movement of the line of sight will change as a change in the driver's confirmation action. For example, if the perception function is reduced due to visual field defects, the range of moving the line of sight will become narrower. In addition, if the attention function is reduced, the frequency of moving the line of sight will decrease, or the moving distance of the line of sight will become shorter. Moreover, if the motor function is reduced due to paralysis of the hands and feet, the driver will pay attention to the condition of the paralyzed hands and feet, and therefore will turn his line of sight in the direction of the paralyzed hands and feet. Therefore, by taking the movement of the line of sight corresponding to the driving environment and the vehicle state in a state without disease (healthy state) as a reference, the abnormality of the movement of the driver's line of sight is obtained, and the change of the confirmation action caused by the reduction of the driver's perception function, attention function, or motor function can be grasped.

[0080] In addition, the following insights have been obtained: if the perception function, attention function, or motor function is reduced, the operation of the steering wheel, accelerator pedal, and brake pedal will change as the operation action of operating the steering wheel, accelerator pedal, and brake pedal changes. For example, if the motor function is reduced, the operation of the steering wheel and the operation of the pedal will be delayed compared to the healthy state. In addition, if the perception function and attention function are reduced, the driver's recognition of the driving environment, which is the basis of the driving operation, will be affected. Therefore, for example, the position of the own vehicle within the driving route cannot be well grasped or the delay or omission of the discovery of the object occurs, resulting in the operation of the steering wheel and the operation of the pedal becoming unstable compared to the healthy state. Therefore, by taking the driving operation corresponding to the driving environment and the vehicle state in the healthy state as a reference, the abnormality of the driver's driving operation can be obtained, and the change of the operation action caused by the reduction of the driver's perception function, attention function, or motor function can be grasped.

[0081] In addition, when the driver's perception function, attention function, or motor function is reduced, the risk of driving such as leaving the driving route and approaching obstacles increases due to delays / missing the discovery of the object and unstable driving operations. According to the research of the inventors of this application, in a healthy driver, even if the temporary driving risk increases due to inattention and disorder of operation, the driving risk will be reduced immediately because appropriate correction operations will be performed later. Therefore, the average value of the driving risk remains at a low level. In contrast, if the driver has reduced driving function, the driving risk gradually increases because it becomes difficult to perform driving operations for reducing the driving risk. As a result, the average value of the driving risk becomes higher than that of healthy people, and the rising trend of the driving risk continues. Therefore, by obtaining the average value and rising trend of the driving risk, the influence of the change in driving behavior caused by the reduction of the driver's motor function, perception function, or attention function can be grasped.

[0082] Therefore, the controller 10 of the present embodiment calculates the abnormality of the driver's line of sight (line of sight abnormality), the abnormality of the driver's driving operation (driving operation abnormality), and the driving risk based on the driver's line of sight, driving operation, and the driving environment of the vehicle 1. Furthermore, the controller 10 is configured to calculate the comprehensive abnormality of the driver's state (comprehensive abnormality) based on these line of sight abnormality, driving operation abnormality, and driving risk, and detect that the driver is in an abnormal sign state based on the comprehensive abnormality.

[0083] Specifically, if Figure 3As shown, the controller 10 obtains sight line information based on the signal received from the in-vehicle camera 32, obtains driving operation information based on the signals received from the steering angle sensor 28, the steering torque sensor 29, the acceleration sensor 30, and the brake sensor 31, and obtains driving environment information and vehicle state information based on the signals received from the sensors including the out-vehicle camera 21, the radar 22, the navigation system 23, the positioning system 24, the vehicle speed sensor 25, the acceleration sensor 26, and the yaw rate sensor 27. In addition, the controller 10 calculates sight line parameters based on the signals received from the out-vehicle camera 21, the vehicle speed sensor 25, the steering angle sensor 28, and the in-vehicle camera 32.

[0084] The controller 10 inputs the predetermined reference value of the characteristic amount representing the movement of the line of sight and the calculated line of sight parameter into the line of sight model, thereby calculating the predicted value (line of sight predicted value) of the movement of the line of sight of the driver in a normal state. As the characteristic amount representing the movement of the line of sight, for example, the amplitude and frequency of the glance can be used.

[0085] Here, saccade refers to the jumping eye movement used to capture the visual target in the fovea of ​​the retina, and refers to the eye movement that moves the line of sight from the fixation point where it stays for a specified time to the next fixation point. The amplitude of the saccade refers to the amount of movement of the driver's line of sight from the fixation point to the next fixation point, and the frequency of the saccade refers to the number of times the line of sight moves from the fixation point to the next fixation point within a specified time.

[0086] The movement of sight is affected by the driver's steering operation, vehicle status (such as vehicle speed), driving environment (such as illumination, saliency (Japanese: サリエンシー) deviation, etc.), driver's head movement, etc. Here, saliency refers to the characteristics that indicate the ease of attracting people's attention, and is a visual feature determined by the temporal and spatial configuration of color, brightness, movement, etc. That is, the area with high saliency in the driver's field of vision is an area that easily attracts the driver's attention due to, for example, large color difference or brightness difference relative to the surrounding area or large movement.

[0087] For example, when the driver performs a steering operation at an intersection or in a continuous curve, the driver focuses his sight on the direction in which the vehicle turns, so the deviation of the sight tends to decrease. In addition, when the vehicle speed is high, such as on a highway, the driver's sight tends to be concentrated in the direction of travel. In addition, when the illumination of the driving environment is low, such as at night, the driver's sight also tends to be concentrated in the direction of travel. In addition, when the deviation of the saliency in the driver's field of view is small (that is, when the area with high saliency is concentrated in a part of the driver's field of view), the driver's sight tends to be concentrated in the area with high saliency. Moreover, the driver's sight tends to be biased in the direction in which the driver's face is facing.

[0088] Therefore, the line of sight model is constructed so that when parameters (line of sight parameters) such as steering operation, vehicle status (vehicle speed), driving environment (deviation of saliency, illumination), driver's head movement (face orientation) that may affect the movement of line of sight are input, the line of sight model outputs a predicted value of the driver's line of sight in a healthy state taking these influences into account.

[0089] Figure 4 is a block diagram of a sight line model based on this embodiment. Figure 4 As shown, the sight line model sets reference values ​​of the scanning frequency and amplitude in a healthy state. The reference values ​​can be set in advance, for example, and machine learning can be performed each time the vehicle 1 travels so that individual differences of the driver are reflected in the reference values.

[0090] In addition, the line of sight model also includes: a vehicle state influence model for correcting the baseline value based on the influence of steering operation and vehicle state on the movement of line of sight, a driving environment model for correcting the baseline value based on the influence of the driving environment, and a head movement influence model for correcting the baseline value based on the influence of the driver's head movement.

[0091] The vehicle state influence model is constructed such that when the standard deviation of the steering wheel operation amount (steering angle) within a specified time (for example, 30 seconds) and the average value of the vehicle speed are input as parameters representing the steering operation and the vehicle state, the correction values ​​of the scanning frequency and amplitude corresponding to the standard deviation of the steering angle and the average value of the vehicle speed are output. Specifically, a correction map is set in the vehicle state influence model, and the correction map determines the relationship between the standard deviation of the steering angle and the average value of the vehicle speed and the correction values ​​of the scanning frequency and amplitude. In addition, the vehicle state influence model is: when the standard deviation of the steering angle and the average value of the vehicle speed are input to the vehicle state influence model, based on the correction map, the correction values ​​of the scanning frequency and amplitude corresponding to the input standard deviation of the steering angle and the average value of the vehicle speed are output.

[0092] In addition, the driving environment impact model is configured to output correction values ​​of the scanning frequency and amplitude corresponding to the deviation of the saliency and the average value of the illumination of the driving environment within a specified time (for example, 30 seconds) as parameters representing the driving environment. Specifically, a correction map is set in the driving environment impact model, and the correction map determines the relationship between the deviation of the saliency and the average value of the illumination of the driving environment and the correction values ​​of the scanning frequency and amplitude. And, the driving environment impact model is: when the deviation of the saliency and the average value of the illumination of the driving environment are input to the driving environment impact model, based on the correction map, the correction values ​​of the scanning frequency and amplitude corresponding to the input deviation of the saliency and the average value of the illumination of the driving environment are output.

[0093] Figure 5 is a diagram showing an example of the distribution of saliency obtained based on the signal output from the vehicle exterior camera 21. Figure 5 (a) is a diagram showing a state where the deviation of significance is large using high-resolution significance distribution data. Figure 5 (b) is a diagram showing a state where the deviation of saliency is large using low-resolution saliency distribution data. Figure 5 (c) is a diagram showing a state where the deviation of significance is small using high-resolution significance distribution data. Figure 5 (d) is a diagram showing a state where the deviation of significance is small using low-resolution significance distribution data. Figure 5 In the figure, the closer the color is to white, the higher the saliency is, and the closer the color is to black, the lower the saliency is.

[0094] In order to determine the tendency of the driver's line of sight to be guided to the high-saliency area as in the prior art, it is necessary to obtain the information in order to accurately determine the degree of consistency between the line of sight and the high-saliency area. Figure 5 (a) Figure 5 However, the computational load for calculating the saliency distribution from the image data is high, so it is difficult to calculate the saliency distribution with a resolution sufficient to determine the degree of coincidence between the line of sight direction and the high saliency area within the computational resources of the on-board computer.

[0095] Therefore, in this embodiment, the consistency between the direction of the sight line and the area with high saliency is not determined, but the predicted value of the movement of the sight line is calculated in order to consider the influence of the distribution of saliency on the movement of the driver's sight line, and the deviation of the high saliency part in the driver's field of vision is obtained. In this case, it is not necessary to calculate high-resolution saliency distribution data as in the case of determining the consistency between the direction of the sight line and the area with high saliency. For example, even if Figure 5 (b) Figure 5 The saliency distribution data with a low resolution (10×6=60 pixels) as shown in (d) also has a resolution sufficient to find the deviation of the portion with high saliency.

[0096] Specifically, the controller 10 calculates the saliency distribution data in the driver's field of view based on the image data acquired from the vehicle exterior camera 21. In addition, the controller 10 calculates the location containing the saliency peak in the saliency distribution, for example, the location with a saliency height of 75 percentile or more (in Figure 5 (b) and Figure 5 The variance of the coordinates of the area surrounded by white circles in (d) is used as the deviation of the significance. Figure 5 In (b), there are multiple locations containing the peak of significance, but Figure 5 In (d), there is only one location with a significant peak, so Figure 5 Compared with (d), Figure 5 When the calculated saliency deviation is input to the driving environment influence model, correction values ​​of the scanning frequency and amplitude corresponding to the input saliency deviation are output based on the correction map.

[0097] In addition, the head movement influence model is configured to output respective correction values ​​of the scanning frequency and amplitude corresponding to the standard deviation of the face orientation when the standard deviation of the driver's face orientation within a specified time (e.g., 30 seconds) is input as a parameter representing the driver's head movement. Specifically, a correction map is set in the head movement influence model, and the correction map determines the relationship between the standard deviation of the face orientation and the correction values ​​of the scanning frequency and amplitude. And, the head movement influence model is: when the head movement influence model is input with the standard deviation of the face orientation, based on the correction map, the correction values ​​of the scanning frequency and amplitude corresponding to the standard deviation of the input steering angle and the vehicle speed are output.

[0098] Here, refer to Figure 6 and Figure 7 , the correction mapping set in the vehicle state influence model, the driving environment influence model and the head movement influence model is explained. Figure 6 This is a diagram showing an example of a correction map used for calculating a predicted value of a scanning frequency. Figure 7 This is a diagram showing an example of a correction mapping for calculating the predicted value of the glance amplitude. The inventors of the present application had more than 100 healthy subjects drive a vehicle and measured the time variations of the average vehicle speed, the standard deviation of the steering angle, the deviation of the saliency, the average illuminance, and the standard deviation of the face orientation, as well as the time variations of the frequency and amplitude of the glances. Furthermore, machine learning was used to extract the effects of the average vehicle speed, the standard deviation of the steering angle, the deviation of the saliency, the average illuminance, and the standard deviation of the face orientation on the frequency and amplitude of the glances. Based on the results, a correction mapping was created. Figure 6 The correction map of the scanning frequency is shown and Figure 7 Corrected maps of saccade amplitudes are shown.

[0099] exist Figure 6 In the correction map of the scan frequency shown, the horizontal axis represents the average vehicle speed, the standard deviation of the steering angle, the deviation of the saliency, the average illumination value and the standard deviation of the face direction, and the vertical axis represents the correction value added to the reference value of the scan frequency. That is, it is shown that when the correction value is greater than 0, the reference value of the scan frequency is corrected in the increasing direction, and when the correction value is less than 0, the reference value of the scan frequency is corrected in the decreasing direction.

[0100] Similarly, in Figure 7In the correction map of the sweep frequency shown, the horizontal axis represents the average value of vehicle speed, the standard deviation of steering angle, the deviation of saliency, the average value of illumination and the standard deviation of face orientation, and the vertical axis represents the correction value added to the reference value of the sweep amplitude. That is, it is shown that when the correction value is greater than 0, the reference value of the sweep amplitude is corrected in the increasing direction, and when the correction value is less than 0, the reference value of the sweep amplitude is corrected in the decreasing direction.

[0101] Figure 6 and Figure 7 In the correction maps shown, the correction maps for the average vehicle speed and the standard deviation of the steering angle are set in the vehicle state influence model, the correction maps for the deviation of the saliency and the average illumination value are set in the driving environment influence model, and the correction map for the standard deviation of the face orientation is set in the head movement influence model.

[0102] For example, regarding the impact of the average vehicle speed, Figure 6 As shown in FIG. 1 , the correction value is set so that the scanning frequency becomes smaller as the average vehicle speed becomes larger. Figure 7 As shown in FIG. 1 , the correction value is set such that, although the scanning amplitude increases as the average vehicle speed increases, the influence on the scanning amplitude decreases when the average vehicle speed increases to a certain extent. This is to indicate that, although the driver's line of sight tends to focus on the direction of travel when the vehicle speed increases, the amplitude of the movement of the line of sight is maintained at a certain level.

[0103] In addition, regarding the influence of steering operation, such as Figure 6 As shown in , the standard deviation of the steering angle has little effect on the scanning frequency. Figure 7 As shown in FIG. 1 , the correction value is set such that when the steering angle standard deviation increases, the glance amplitude decreases, but when the steering angle standard deviation increases to a certain extent, the glance amplitude gradually increases. This is to indicate that although the driver directs his / her sight toward various objects regardless of the magnitude of the steering operation, the range of sight direction itself tends to be concentrated in the direction in which the vehicle rotates due to the steering operation.

[0104] In addition, regarding the impact of deviations in saliency, Figure 6 As shown in FIG. 1 , the correction value is set so that the scanning frequency gradually increases as the deviation of the saliency increases. Figure 7 As shown in FIG. 1 , the deviation of saliency has little effect on the sweep amplitude. This is to indicate that although when the parts with high saliency are widely distributed, the frequency of moving the line of sight increases because the number of objects the driver directs his or her line of sight toward increases, but since the objects the driver directs his or her line of sight toward (e.g., the vehicle in front, an obstacle, etc.) regardless of the saliency still exist, the range of the line of sight itself tends not to change significantly.

[0105] In addition, regarding the influence of the average illumination value, such as Figure 6 As shown in FIG. 1 , the correction value is set so that when the average illumination value increases, the scanning frequency increases. On the other hand, Figure 7 As shown in FIG. 1 , the average illumination value has little effect on the sweep amplitude. This is to indicate that although the driver directs his or her sight toward more objects during the day when the average illumination value is high, compared with the night when the average illumination value is low, and thus the frequency of moving the sight increases, even at night, since the objects directed toward (e.g., the vehicle ahead, obstacles, etc.) still exist around the vehicle, the range of the sight itself tends not to change significantly.

[0106] In addition, regarding the influence of the standard deviation of face orientation, e.g. Figure 6 and Figure 7 As shown in FIG. 1 , the correction value is set so that when the standard deviation of the face direction increases, the scanning frequency and amplitude increase. This is to indicate that since the driver's line of sight is biased toward the direction in which the driver's face is facing, the higher the frequency of moving the face, the higher the frequency of moving the line of sight, and the range of the line of sight also tends to become larger.

[0107] In the sight line model, the reference values ​​of the scanning frequency and amplitude in the healthy state are corrected by the correction values ​​of the scanning frequency and amplitude outputted from the vehicle state influence model, the driving environment model and the head movement influence model respectively constructed as described above, thereby calculating the predicted values ​​of the scanning frequency and amplitude (scanning frequency / amplitude prediction). Thus, the predicted values ​​of the scanning frequency and amplitude reflecting the influence of the steering operation, the vehicle state, the driving environment and the driver's head movement are outputted.

[0108] The inventors of the present application compared the measured values ​​of the sweep frequency and amplitude when a healthy driver actually drove a vehicle in urban areas, on highways, and on mountain roads with the predicted values ​​of the sweep frequency and amplitude output from the line of sight model constructed as described above. Figure 8 is a timing diagram showing the comparison between the measured values ​​of scanning frequency and amplitude and the predicted values ​​based on the line of sight model. Figure 8 (a) shows the comparison of scanning frequency, Figure 8 (b) shows the comparison of saccade amplitude. Figure 8 In the figure, the solid lines represent the measured values ​​of the scanning frequency and amplitude, and the dashed lines represent the predicted values ​​based on the gaze model.

[0109] like Figure 8 As shown, it can be seen that for either the frequency or amplitude of the glance, the change in the trend of the measured value when the driving area shifts from the urban area to the mountain road via the highway can be predicted with high accuracy by the predicted value based on the sight line model. In addition, the subtle peaks of the frequency and amplitude of the glance, which are believed to be mainly based on the driver's head movement, can also be reproduced with high accuracy by the predicted value based on the sight line model.

[0110] return Figure 3 The controller 10 calculates a line of sight abnormality degree (line of sight abnormality degree calculation) indicating the degree to which the line of sight measured value deviates from the line of sight predicted value based on the measured value of the line of sight movement of the driver determined according to the acquired line of sight information (line of sight measured value) and the line of sight predicted value (scanning frequency / amplitude predicted value) output from the line of sight model constructed as described above.

[0111] For example, controller 10 calculates the number of times of the sweep per unit time as the measured value of the sweep frequency based on the number of times of the sweep in the prescribed time (for example 30 seconds). In addition, controller 10 calculates the mean value of the sweep amplitude in the prescribed time (for example 30 seconds) as the measured value of the sweep amplitude. And, accumulate two-dimensional data, this two-dimensional data is the difference of the calculated measured value of the sweep frequency and the predicted value output from the line of sight model as the first variable and the difference of the calculated measured value of the sweep amplitude and the predicted value output from the line of sight model as the second variable. Controller 10 calculates the Mahalanobis distance of the center of gravity (average) of the latest data point of this two-dimensional data and the accumulated data set. There is a correlation between the measured value of the sweep frequency and the difference of the predicted value and the measured value of the sweep amplitude and the predicted value, so by using the Mahalanobis distance like this, the deviation degree of the movement of the line of sight with the healthy state can be represented by an index. And, controller 10 is standardized by dividing the representative value set in advance with the calculated Mahalanobis distance, thereby calculates the line of sight abnormality. As the representative value, the Mahalanobis distance when the driver is in a state of abnormality prediction can be used. That is, when the degree of abnormality of the sight line is 1, the driver is in a state of abnormality prediction.

[0112] Then, the controller 10 accumulates the calculated sight line abnormality in a buffer (sight line abnormality buffer), and acquires the maximum value of the sight line abnormality within a recent predetermined time (for example, 60 seconds) from the sight line abnormality buffer (maximum value acquisition).

[0113] In addition, the controller 10 inputs the acquired driving environment information and vehicle state information into the driving operation prediction model, thereby calculating the predicted value of the driving operation of the driver in a healthy state (driving operation prediction value). As the feature quantity representing the driving operation, for example, the operation amount of the steering wheel (steering angle), the operation amount of the accelerator pedal and the brake pedal (accelerator pedal depression amount and brake pedal depression amount) can be used.

[0114] The driving operation prediction model is configured to output a driving operation prediction value of a healthy driver when the driving environment and vehicle state required for driving operation are input as parameters. Specifically, the driving operation prediction model is configured to output the steering wheel operation amount required to drive in the center of the driving route, the accelerator pedal and brake pedal operation amounts required to follow the vehicle in front or make the vehicle 1 drive at the limited speed as driving operation prediction values, such as the driving environment such as the lines in the direction of travel of the vehicle 1, the position of obstacles, the speed limit, the position and speed of the preceding vehicle, and the vehicle state such as the speed and acceleration, when the parameters are input.

[0115] The controller 10 calculates a driving operation abnormality degree (driving operation abnormality degree calculation) indicating the degree to which the driving operation measured value deviates from the driving operation predicted value based on the measured value of the driver's driving operation determined according to the acquired driving operation information (driving operation measured value) and the driving operation predicted value output from the driving operation prediction model.

[0116] For example, the controller 10 accumulates two-dimensional data, which uses the difference between the measured value of the steering wheel operation amount and the predicted value output from the driving operation prediction model as the first variable, and uses the difference between the measured value of the operation amount of the accelerator pedal or the brake pedal and the predicted value output from the driving operation prediction model as the second variable. The controller 10 calculates the Mahalanobis distance between the latest data point of the two-dimensional data and the centroid (average) of the accumulated data set. There is a correlation between the difference between the measured value and the predicted value of the steering wheel operation amount and the difference between the measured value and the predicted value of the pedal operation amount, so by using the Mahalanobis distance like this, the degree of deviation from the healthy driving operation can be represented by one indicator. In addition, the controller 10 calculates the driving operation abnormality by dividing the calculated Mahalanobis distance by a preset representative value for standardization. As the representative value, the Mahalanobis distance when the driver is in an abnormal sign state can be used. That is, when the driving operation abnormality is 1, the driver is in an abnormal sign state.

[0117] Then, the controller 10 accumulates the calculated driving operation abnormality in a buffer (driving operation abnormality buffer), and acquires the maximum value of the driving operation abnormality within the most recent predetermined time from the driving operation abnormality buffer (maximum value acquisition).

[0118] In addition, the controller 10 calculates the driving risk based on the acquired driving environment information and vehicle state information. The driving risk is expressed by a numerical value to indicate the possibility of deviating from the driving route, approaching an obstacle, etc. For example, it is set as follows: when it is predicted that the vehicle 1 is located in the center of the driving route and the distance to the obstacle is a safe distance (for example, 1m) or more after a predetermined time (for example, 2 seconds), the driving risk = 0. Similarly, the smaller the predicted value of the distance to the distance to the obstacle after the predetermined time, the closer the driving risk is to 1. When it is predicted that the vehicle 1 deviates from the driving route after the predetermined time and the distance to the obstacle is less than the restricted distance (for example, 0.3m), the driving risk = 1. That is, the controller 10 determines the line of the driving road in the direction of travel of the vehicle 1, the position of the vehicle 1 on the driving road, the position and speed of the obstacle based on the acquired driving environment information. In addition, the current speed and acceleration of the vehicle 1 are determined based on the acquired driving environment information or vehicle state information, and the position of the vehicle 1 after the predetermined time is predicted. Then, based on the predicted position of the vehicle 1 and the line, and the position and speed of the obstacle, the predicted values ​​of the distance until the vehicle 1 deviates from the driving route and the distance to the obstacle after a predetermined time are calculated, and the driving risk is calculated based on the calculated predicted values.

[0119] Then, the controller 10 accumulates the calculated travel risks in a buffer area (risk buffer area), and acquires the average value of the travel risks within the most recent predetermined time from the risk buffer area (average value acquisition).

[0120] The controller 10 calculates the comprehensive abnormality (comprehensive abnormality calculation) based on the maximum value of the abnormality of the line of sight, the maximum value of the abnormality of the driving operation, and the average value of the driving risk. For example, the controller 10 uses the maximum value of the abnormality of the line of sight, the maximum value of the abnormality of the driving operation, and the average value of the driving risk as axes in a three-dimensional orthogonal coordinate system, and uses the three-dimensional vector having the maximum value of the abnormality of the line of sight, the maximum value of the abnormality of the driving operation, and the average value of the driving risk as components as a comprehensive abnormality vector, and calculates the magnitude of the comprehensive abnormality vector as the comprehensive abnormality.

[0121] Fig. 9 It is a graph showing the comprehensive abnormality in a three-dimensional orthogonal coordinate system with the visual abnormality, driving operation abnormality and driving risk as coordinate axes. Fig. 9 As shown in FIG. 1 , when the maximum value of the acquired sight abnormality is set to Gab_max, the maximum value of the driving operation abnormality is set to Oab_max, and the average value of the driving risk is set to R_ave, the size of the comprehensive abnormality vector Tab can be expressed as (Gab_max 2 +Oab_max 2 +R_ave 2 )1 / 2 For example, in Fig. 9 In the example, any of the maximum value of the sight abnormality Gab_max1, the maximum value of the driving operation abnormality Oab_max1, and the average value of the driving risk R_ave1 is less than 1, but even in this case, the magnitude of the comprehensive abnormality vector Tab is greater than 1. In addition, as long as any of the obtained maximum value of the sight abnormality, the maximum value of the driving operation abnormality, and the average value of the driving risk is greater than 1, the comprehensive abnormality is greater than 1.

[0122] The controller 10 determines whether the calculated comprehensive abnormality is greater than a predetermined threshold value (threshold value determination). When the magnitude of the comprehensive abnormality vector obtained as described above is calculated as the comprehensive abnormality, the threshold value is, for example, 1. Fig. 9 In the example, the sphere with radius 1 corresponds to the threshold.

[0123] Furthermore, when the controller 10 determines that the driving risk is increasing based on the calculated driving risk and the increasing rate is maintained, if the rate reaches 1 within a predetermined time (eg, within 5 seconds), the controller 10 detects an increase in the driving risk (driving risk increase detection).

[0124] Then, when the comprehensive abnormality degree is equal to or greater than the threshold value and an increase in the driving risk is detected, the controller 10 determines that the driver is in an abnormality sign state. In other words, the controller 10 detects that the driver is in an abnormality sign state (abnormality sign detection).

[0125] When an abnormal sign state is detected, the controller 10 sends a control signal to the PCM 33, DSC 34, and EPS 35 for properly operating the driving force source 2, the transmission 3, the brake 4, and the steering device 5, and sends a control signal to the display 36 and the speaker 37 for outputting desired information. For example, the controller 10 vibrates the steering wheel at a prescribed frequency through the EPS 35, and determines that the driver is in an abnormal state based on the driver's response. In addition, the controller 10 outputs a warning display through the display 36, and determines that the driver is in an abnormal state based on the driver's response to the warning display.

[0126] [Abnormal sign detection and processing]

[0127] Next, refer to Figure 10 to Figure 13 , the flow of the detection process of the abnormality sign state by the driver abnormality sign detection device 100 according to the present embodiment will be described. Fig.10 This is a flowchart of the calculation process of the sight line abnormality. Fig.11 is a flowchart of the driving operation abnormality calculation process, Fig.12 is a flow chart of the driving risk calculation process, Fig.13The process of calculating the degree of abnormality in sight, the process of calculating the degree of abnormality in driving operation, the process of calculating the risk of driving, and the process of detecting the abnormality are respectively started when the power of the vehicle 1 is turned on, and are repeatedly executed in parallel by the controller 10 at a predetermined period (for example, every 0.05 to 0.2 seconds).

[0128] when Fig.10 When the sight line abnormality calculation process starts, the controller 10 detects the sight line of the driver based on the signal received from the in-vehicle camera 32 (step S1). Next, the controller 10 calculates the measured values ​​of the frequency and amplitude of the glance based on the detected sight line of the driver (step S2).

[0129] The controller 10 also acquires reference values ​​of scanning frequency and amplitude in the health state of the driver (step S3). The reference values ​​are stored in advance in the memory 10b, for example, and the individual differences of the driver are reflected in the reference values ​​by executing machine learning every time the vehicle 1 travels.

[0130] Next, the controller 10 calculates the average vehicle speed based on the signal received from the vehicle speed sensor 25, calculates the standard deviation of the steering angle based on the signal received from the steering angle sensor 28, calculates the deviation of the saliency and the average illumination of the driving environment based on the signal received from the vehicle exterior camera 21, and calculates the standard deviation of the driver's face orientation based on the signal received from the vehicle interior camera 32 (step S4). That is, the controller 10 calculates the sight line parameter based on the information obtained from the sight line parameter information obtaining device.

[0131] Next, the controller 10 inputs the sight line parameter calculated in step S4 into the vehicle state influence model, the driving environment influence model, and the head movement influence model, thereby acquiring a correction value for correcting the reference value (step S5 ).

[0132] Next, the controller 10 calculates predicted values ​​of the frequency and amplitude of saccades by correcting the reference value using the correction value calculated in step S5 (step S6 ).

[0133] Then, controller 10 calculates sight line abnormality based on the measured value of the frequency of the pan calculated in step S2 and the predicted value of the pan calculated in step S6 and the amplitude, and is stored in buffer (step S7).As mentioned above, for example, controller 10 calculates the Mahalanobis distance of the latest data point of the two-dimensional data of the measured value of the pan frequency and the predicted value as the first variable and the measured value of the pan amplitude and the predicted value as the second variable and the center of gravity (average) of the data set accumulated so far.And then, controller 10 standardizes the sight line abnormality with the calculated Mahalanobis distance divided by the pre-set representative value, calculates the sight line abnormality thus.After step S7, controller 10 ends the sight line abnormality calculation process.

[0134] when Fig.11 When the calculation process of the driving operation abnormality starts, the controller 10 detects the driver's driving operation based on the signals received from the steering angle sensor 28, the steering torque sensor 29, the acceleration sensor 30 and the brake sensor 31, specifically, detects the operation amount of the steering wheel, the accelerator pedal and the brake pedal (step S11).

[0135] In addition, the controller 10 obtains driving environment information and vehicle state information based on signals received from sensors including the exterior camera 21, the radar 22, the navigation system 23, the positioning system 24, the vehicle speed sensor 25, the acceleration sensor 26, and the yaw rate sensor 27 (step S12). Then, the controller 10 inputs the information obtained in step S12 into the driving operation prediction model to calculate the predicted value of the driving operation (step S13).

[0136] Next, the controller 10 calculates the driving operation abnormality based on the measured value of the driver's driving operation detected in step S11 and the predicted value of the driving operation calculated in step S13, and stores it in the buffer (step S14). As described above, for example, the controller 10 calculates the Mahalanobis distance between the latest data point of the two-dimensional data and the centroid (average) of the data set accumulated so far, using the difference between the measured value and the predicted value of the steering wheel operation amount as the first variable and the difference between the measured value and the predicted value of the accelerator pedal or the brake pedal operation amount as the second variable. Furthermore, the controller 10 calculates the driving operation abnormality by dividing the calculated Mahalanobis distance by a pre-set representative value for standardization. After step S14, the controller 10 ends the driving operation abnormality calculation process.

[0137] when Fig.12 When the driving risk calculation process starts, the controller 10 obtains the driving environment information and the vehicle status information based on the signals received from the sensors including the external camera 21, the radar 22, the navigation system 23, the positioning system 24, the vehicle speed sensor 25, the acceleration sensor 26, and the yaw rate sensor 27 (step S21).

[0138] Next, the controller 10 calculates the driving risk based on the driving environment information and vehicle status information acquired in step S21, and stores it in the buffer (step S22). As described above, for example, the controller 10 determines the lines of the driving road in the direction of travel of the vehicle 1, the position of the vehicle 1 on the driving road, and the position and speed of obstacles based on the acquired driving environment information. In addition, the current speed and acceleration of the vehicle 1 are determined based on the acquired driving environment information or vehicle status information, and the position of the vehicle 1 after a specified time is predicted. Then, based on the predicted position of the vehicle 1 and the position and speed of the lines and obstacles, the predicted values ​​of the distance of the vehicle 1 until it leaves the driving route and the distance to the obstacle after a specified time are calculated, and the driving risk is calculated based on the calculated predicted values.

[0139] Next, the controller 10 determines whether the driving risk is increasing based on the driving risk calculated in step S22 (step S23). As described above, for example, the controller 10 determines that the driving risk is increasing when it is determined that the driving risk is increasing and reaches 1 within a specified time while the increasing rate is maintained.

[0140] As a result, if it is determined that the driving risk is increasing (step S23: Yes), that is, if the controller 10 detects an increase in the driving risk, the controller 10 sets the driving risk increase flag to TRUE (step S24). On the other hand, if it is not determined that the driving risk is increasing (step S23: No), the controller 10 sets the driving risk increase flag to FALSE (step S25). After the processing of step S24 or S25, the controller 10 ends the driving risk calculation processing.

[0141] when Fig.13 When the abnormal sign detection process starts, the controller 10 obtains the maximum value of the line of sight abnormality degree within the most recent specified time (for example, 60 seconds) from the line of sight abnormality degree buffer (step S31).

[0142] Furthermore, the controller 10 acquires the maximum value of the driving operation abnormality degree within the most recent predetermined time from the sight line abnormality degree buffer (step S32 ).

[0143] Furthermore, the controller 10 acquires the average value of the travel risk within the most recent predetermined time from the risk buffer (step S33 ).

[0144] Next, the controller 10 calculates the comprehensive abnormality according to the maximum value of the abnormality of sight line, the maximum value of the abnormality of driving operation, and the average value of the driving risk (step S34). As described above, for example, the controller 10 calculates the magnitude of the comprehensive abnormality vector as the comprehensive abnormality in a three-dimensional orthogonal coordinate system with the maximum value of the abnormality of sight line, the maximum value of the abnormality of driving operation, and the average value of the driving risk as axes, respectively.

[0145] Next, the controller 10 determines whether the comprehensive abnormality degree calculated in step S34 is equal to or greater than a predetermined threshold value (step S35 ).

[0146] When it is determined that the comprehensive abnormality is equal to or greater than the predetermined threshold value (step S35 : Yes), the controller 10 determines whether the travel risk increase flag is TRUE (step S36 ).

[0147] When it is determined that the travel risk increase flag is TRUE (step S36 : YES), the controller 10 determines that the driver is in an abnormality sign state. In other words, the controller 10 detects that the driver is in an abnormality sign state (step S37 ).

[0148] On the other hand, if it is not determined in step S35 that the comprehensive abnormality is above the predetermined threshold value (i.e., if the comprehensive abnormality is less than the threshold value) (step S35: No), or if it is not determined in step S36 that the driving risk increase flag is TRUE (i.e., if the driving risk increase flag is FALSE) (step S36: No), the controller 10 determines that the driver is not in an abnormal sign state. That is, the abnormal sign state of the driver is not detected (step S38). After the processing of step S37 or S38, the controller 10 ends the abnormal sign detection processing.

[0149] In addition, in the above-mentioned embodiment, in addition to the distribution of saliency within the driver's field of view, the operation amount of the steering wheel of the vehicle 1, the vehicle speed, the illumination around the vehicle 1 and the direction of the driver's face are also used as line of sight parameters, but any one or more of these line of sight parameters may also be used.

[0150] In addition, in the above-mentioned embodiment, the use of Mahalanobis distance to calculate the sight line abnormality and the driving operation abnormality is described, but the sight line abnormality and the driving operation abnormality can also be obtained by other calculation methods. For example, the difference between the actual measured value and the predicted value of the scanning frequency and amplitude can also be standardized and synthesized to obtain the value as the sight line abnormality. In addition, the difference between the actual measured value and the predicted value of the operation amount of the steering wheel and the operation amount of the pedal can also be standardized and synthesized to obtain the value as the driving operation abnormality.

[0151] Furthermore, in the above-described embodiment, the amplitude and frequency of saccade are described as feature quantities indicating the movement of the line of sight, but either the amplitude or the frequency of saccade may be used.

[0152] In the above-described embodiment, the operation amounts of the steering wheel, accelerator pedal, and brake pedal are used as feature amounts representing driving operation. However, any one or two of the operation amounts of the steering wheel, accelerator pedal, and brake pedal may be used.

[0153] In addition, in the above-mentioned embodiment, a three-dimensional vector having the maximum value of the sight abnormality, the maximum value of the driving operation abnormality, and the average value of the driving risk as components is used as a comprehensive abnormality vector, and the magnitude of the comprehensive abnormality vector is calculated as the comprehensive abnormality. However, the comprehensive abnormality can also be obtained by other calculation methods. For example, the sum of the maximum value of the sight abnormality, the maximum value of the driving operation abnormality, and the average value of the driving risk can also be calculated as the comprehensive abnormality.

[0154] [Function / Effect]

[0155] Next, the effects of the driver abnormality sign detection device 100 according to the above-mentioned embodiment will be described.

[0156] The controller 10 obtains a predetermined reference value of the frequency and / or amplitude of the driver's glance, calculates the deviation of the portion containing the peak of significance in the distribution of significance in the driver's field of vision based on the image data obtained from the vehicle-external camera 21, and corrects the reference value by a correction value obtained based on the deviation of the portion containing the peak of significance, thereby calculating a predicted value of the frequency and / or amplitude of the glance. Therefore, even if high-resolution significance distribution data is not calculated as in the case of determining the consistency between the direction of the sight line and the area with high significance, a predicted value of the movement of the sight line that takes into account the influence of the significance distribution on the movement of the driver's sight line can be obtained. Thus, even with limited computing resources, it is possible to correctly grasp the degree to which the measured value of the movement of the sight line deviates from the predicted value of the movement of the sight line of a healthy driver, and the abnormal sign state of the driver can be detected more accurately based on the grasped state of the movement of the sight line.

[0157] In addition, the correction value is set so that the larger the deviation of the portion containing the peak value of the significance is, the more the reference value is corrected in the direction of increasing the reference value, so that a predicted value of the movement of the line of sight can be obtained that takes into account the tendency of the driver's glance to increase when the deviation of the significance is large. Thus, the abnormal omen state of the driver can be detected with high accuracy.

[0158] In addition, the correction value is set such that the larger the deviation of the position including the peak value of the significance is, the more the reference value is corrected in the direction of increasing the reference value of the frequency of the glance, and the controller 10 corrects the reference value of the frequency of the glance by the correction value, thereby calculating the predicted value of the frequency of the glance, and thus can obtain the predicted value of the movement of the line of sight that takes into account the tendency of the driver's glance frequency to increase when the deviation of the significance is large. Thus, the abnormal omen state of the driver can be detected with high accuracy.

[0159] In addition, the controller 10 accumulates two-dimensional data using the difference between the measured value and the predicted value of the frequency of the glance as the first variable and the difference between the measured value and the predicted value of the amplitude of the glance as the second variable, and calculates the degree of sight abnormality based on the Mahalanobis distance between the latest data point of the two-dimensional data and the center of gravity of the set of the accumulated two-dimensional data, so that the degree of deviation of the movement of the sight line from the healthy state can be comprehensively represented by one index. Therefore, even at a sufficiently early stage before the driver reaches an abnormal state of driving difficulty, such as a reduction in the driver's driving function cannot be detected by only either the frequency and amplitude of the glance, it is possible to detect the abnormal omen state early and with high accuracy.

Claims

1. A driver abnormal sign detection device for detecting an abnormal sign state of a driver driving a vehicle, characterized in that: have: A sight line detection device, the sight line detection device detects the sight line of the driver; An external camera for photographing the surroundings of the vehicle and outputting image data; as well as a controller configured to detect an abnormal sign state of the driver based on the sight line information obtained from the sight line detection device and the image data obtained from the external camera, The controller is configured as follows: obtaining a predetermined reference value of the frequency and / or amplitude of the driver's glance under the driver's health condition, calculating, based on the image data, a deviation of a portion including a saliency peak in a saliency distribution within the driver's field of vision, acquiring a correction value for correcting the reference value based on a deviation of a portion including a peak of the significance, Correcting the reference value by the correction value, thereby calculating a predicted value of the frequency and / or amplitude of the saccade, Calculating a degree of sight abnormality, where the degree of deviation of a measured value of the frequency and / or amplitude of the glance obtained based on the sight information from the predicted value, It is determined that the abnormality sign state of the driver is detected based on the degree of abnormality of the sight line.

2. The driver abnormality sign detection device according to claim 1, characterized in that: The correction value is set so that the reference value is corrected in a direction in which the reference value increases as the deviation of the portion including the peak of the significance increases.

3. The driver abnormality sign detection device according to claim 2, characterized in that: The correction value is set so that the reference value is corrected in a direction in which the reference value of the saccade frequency increases as the deviation of the portion including the peak value of the saliency increases. The controller is configured as follows: obtaining a reference value of the frequency of the scan, The reference value is corrected by the correction value, thereby calculating a predicted value of the frequency of the saccade.

4. The driver abnormality sign detection device according to any one of claims 1 to 3, characterized in that: The controller is configured as follows: obtaining a predetermined reference value of the frequency and amplitude of the driver's glance, The reference value is corrected by the correction value, thereby calculating the predicted value of the frequency and amplitude of the saccade, accumulating two-dimensional data with the difference between the measured value and the predicted value of the frequency of the sweep as the first variable and the difference between the measured value and the predicted value of the amplitude of the sweep as the second variable, The degree of visual line abnormality is calculated based on the Mahalanobis distance between the latest data point of the two-dimensional data and the centroid of a set of accumulated two-dimensional data.

Images