Driver abnormality prediction device

Abstract

To provide a driver abnormality sign detector capable of detecting a decline in the physical functions of a driver at a sufficiently early stage before the driver becomes incapable of driving, and of determining the abnormality status at an early stage.SOLUTION: A driver abnormality sign detector 100 includes: an external camera 21 that acquires driving environment information of a vehicle; a radar 22; a navigation system 23; a positioning system 24; an in-vehicle camera 32 that detects the driver's line of sight; and a controller 10 configured to detect an abnormality sign of the driver based on the driving environment information and the driver's line of sight. The controller identifies points needed to be viewed that the driver should view based on the driving environment information, determines whether or not the driver has viewed the visible points based on the driver's line of sight, and detects the abnormality sign of the driver based on the fact that the driver has not viewed the points needed to be viewed.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a driver abnormality prediction detection device for detecting an abnormal prediction state of a driver during vehicle operation.

Background Art

[0002] Conventionally, a driver state detection device for detecting an abnormality of a vehicle driver has been proposed (see, for example, Patent Document 1). In the device described in Patent Document 1, the amplitude and frequency of the saccade (jumping eye movement in which the driver intentionally moves the line of sight) of the vehicle driver are detected, and the degree of attention that increases as the number of attention points that the driver should check during driving increases in the external environment of the vehicle is detected. Based on the high degree of attention and the amplitude and frequency of the driver's saccade, the abnormality of the driver is detected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in a state where the driver's attention function is slightly reduced due to a mild disease or aging but the driver can still drive, that is, in an abnormal prediction state before reaching an abnormal state where driving is difficult, there is no obvious difference in the amplitude and frequency of the driver's saccade compared to the state without abnormality. Therefore, the conventional technology as described above cannot detect the abnormal prediction state.

[0005] The present invention has been made to solve such problems, and an object thereof is to provide a driver abnormality prediction detection device capable of detecting an abnormal prediction state of a driver at a stage sufficiently earlier than when the driver enters an abnormal state where driving is difficult.

Means for Solving the Problems

[0006] To solve the above-mentioned problems, the present invention provides a driver abnormality prediction detection device for detecting abnormal signs in a driver operating a vehicle, comprising: a driving environment information acquisition device for acquiring information on the vehicle's driving environment; a gaze detection device for detecting the driver's gaze; and a controller configured to detect abnormal signs in the driver based on the driving environment information and the driver's gaze, wherein the controller is configured to identify a plurality of points of interest that the driver should look at in the vehicle's driving environment based on the acquired driving environment information, to determine whether the driver has looked at the points of interest based on the identified plurality of points of interest and the driver's gaze, and to detect abnormal signs in the driver based on the fact that the driver did not look at the points of interest. The controller is configured to estimate the width of the effective field of view visible to the driver based on the angle formed by the driver's line of sight and the direction from the driver's head towards the area to be seen, and the result of determining whether or not the driver has seen the area to be seen. If the estimated width of the effective field of view is less than a predetermined value, it is configured to detect an abnormal condition in the driver. It is.

[0007] According to the present invention configured in this way, the controller identifies areas that the driver should visually inspect based on driving environment information, determines whether the driver has visually inspected these areas based on the driver's line of sight, and detects a driver abnormality based on the fact that the driver did not visually inspect the areas. Therefore, it is possible to detect a driver abnormality by utilizing the fact that the driver was unable to visually inspect the areas due to a decrease in distributed attention function or a reduction in the effective field of view in the early stages of a decline in the driver's attention function. This makes it possible to detect a driver abnormality at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult, for example, when there is no clear difference in the amplitude or frequency of the driver's saccades. Furthermore, by estimating the degree to which the effective field of view has narrowed and the driver begins to miss important visual cues in the early stages of declining attention function, it is possible to detect warning signs of an abnormal condition. This allows for the appropriate detection of warning signs of an abnormal condition in the driver at a sufficiently early stage before the driver reaches a state where driving becomes difficult.

[0008] In the present invention, preferably, the controller is configured to determine that the driver has viewed a designated area when the driver's line of sight is directed within a predetermined range including the designated area, and to detect an abnormal condition in the driver based on the number of designated areas that the driver did not view.

[0009] According to the present invention configured in this way, the driver's abnormal state is detected based on the number of points of interest that the driver did not direct their gaze towards. Therefore, in the initial stages of a decline in attentional function, when the driver becomes unable to simultaneously view multiple points of interest due to a decline in distributed attention function, and points of interest begin to be overlooked, the abnormal state can be detected based on the extent to which points of interest are being overlooked. This makes it possible to appropriately detect the driver's abnormal state at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult.

[0010] In the present invention, preferably, the controller is configured to detect a driver's abnormal condition when the ratio of the number of locations that the driver did not see to the total number of locations that need to be seen is greater than or equal to a predetermined value.

[0011] According to the present invention configured in this way, in the initial stage of a decline in attention function, when a decline in distributed attention function prevents the driver from simultaneously viewing multiple points of interest, and the rate of overlooking points of interest exceeds a predetermined value, an abnormal warning state can be detected. This makes it possible to appropriately detect an abnormal warning state in the driver at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult. In the present invention, preferably, the driver abnormality prediction detection device includes a sensor for detecting driving operations performed by the driver, and a memory for storing driving operations corresponding to each of the locations requiring visual inspection. The controller detects driving operations performed by the driver based on signals received from the sensor, and determines that the driver has visually inspected the location requiring visual inspection if the driver's gaze is directed within a predetermined range including the location requiring visual inspection, and a driving operation corresponding to the location requiring visual inspection is subsequently performed.

[0014] In the present invention, preferably, the driver abnormality prediction detection device further includes an information output device that outputs information to the driver, and the controller is configured to output information via the information output device that guides the driver's gaze to a point requiring visual inspection when an abnormality prediction state of the driver is detected.

[0015] According to the present invention configured in this way, when an abnormal condition in the driver is detected, the driver's gaze is guided to the area requiring visual inspection. Therefore, appropriate driving assistance can be provided to drivers who have begun to overlook areas requiring visual inspection in the early stages of a decline in attentional function. [Effects of the Invention]

[0016] According to the driver abnormality prediction detection device of the present invention, it is possible to detect the abnormal prediction state of the driver at a stage sufficiently earlier than when the driver enters an abnormal state where driving becomes difficult.

Brief Description of the Drawings

[0017] [Figure 1] It is an explanatory diagram of a vehicle equipped with a driver abnormality prediction detection device according to an embodiment of the present invention. [Figure 2] It is a block diagram of a driver abnormality prediction detection device according to an embodiment of the present invention. [Figure 3] It is a control block diagram of abnormality prediction detection according to an embodiment of the present invention. [Figure 4] It is a table exemplifying information stored in a visually recognizable point database according to an embodiment of the present invention. [Figure 5] It is a flowchart of a detection process for an abnormal prediction state according to an embodiment of the present invention.

Modes for Carrying Out the Invention

[0018] Hereinafter, a driver abnormality prediction detection device according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0019] [System Configuration] First, referring to FIGS. 1 and 2, the configuration of the driver abnormality prediction detection device according to the present embodiment will be described. FIG. 1 is an explanatory diagram of a vehicle equipped with the driver abnormality prediction detection device, and FIG. 2 is a block diagram of the driver abnormality prediction detection device.

[0020] A vehicle 1 according to the present embodiment includes a driving force source 2 such as an engine or an electric motor that outputs a driving force, a transmission 3 that transmits the driving force output from the driving force source 2 to the driving wheels, a brake 4 that applies a braking force to the vehicle 1, and a steering device 5 for steering the vehicle 1.

[0021] The driver abnormality prediction detection device 100 is configured to detect the abnormal prediction state of the driver of the vehicle 1 and perform control of the vehicle 1 and driving support control as necessary. As shown in FIG. 2, the driver abnormality prediction detection device 100 includes a controller 10, a plurality of sensors, a plurality of control systems, and a plurality of information output devices.

[0022] Specifically, the plurality of sensors include an external camera 21 that acquires running environment information of the vehicle 1, a radar 22, a navigation system 23 for detecting the position of the vehicle 1, and a positioning system 24. Further, the plurality of sensors include a vehicle speed sensor 25, an acceleration sensor 26, a yaw rate sensor 27, a steering angle sensor 28, a steering torque sensor 29, an accelerator sensor 30, and a brake sensor 31 for detecting the behavior of the vehicle 1 and driving operations by the driver. Further, the plurality of sensors include an in-vehicle camera 32 for detecting the driver's line of sight. The plurality of control systems include a power train control module (PCM) 33 that controls the drive power source 2 and the transmission 3, a dynamic stability control system (DSC) 34 that controls the drive power source 2 and the brake 4, and an electric power steering system (EPS) 35 that controls the steering device 5. The plurality of information output devices include a display 36 that outputs image information and a speaker 37 that outputs voice information.

[0023] Also, as other sensors, a peripheral sonar that measures the distance and position of peripheral structures with respect to the vehicle 1, a corner radar that measures the approach of peripheral structures at four corners of the vehicle 1, and various sensors that detect the driver's state (for example, a heartbeat sensor, an electrocardiogram sensor, a grip force sensor of the steering wheel, etc.) may be included.

[0024] The controller 10 performs various calculations based on signals received from multiple sensors and sends control signals to the PCM 33, DSC 34, and EPS 35 to appropriately operate the drive source 2, transmission 3, brakes 4, and steering system 5, and sends control signals to the display 36 and speaker 37 to output desired information. The controller 10 is composed of a computer equipped with one or more processors 10a (typically a CPU), memory 10b (ROM, RAM, etc.) for storing various programs and data, input / output devices, and so on.

[0025] The external camera 21 captures images of the area around the vehicle 1 and outputs image data. The controller 10 identifies objects (for example, preceding vehicles, parked vehicles, pedestrians, roads, lane markings (lane boundaries, white lines, yellow lines), traffic signals, traffic signs, stop lines, intersections, obstacles, etc.) based on the image data received from the external camera 21. The external camera 21 is an example of a "driving environment information acquisition device" in this invention.

[0026] Radar 22 measures the position and speed of objects (especially preceding vehicles, parked vehicles, pedestrians, objects on the road, etc.). For example, millimeter-wave radar can be used as radar 22. Radar 22 transmits radio waves in the direction of travel of vehicle 1 and receives reflected waves generated when the transmitted waves are reflected by the object. Based on the transmitted and received waves, radar 22 measures the distance between vehicle 1 and the object (e.g., inter-vehicle distance) and the relative speed of the object with respect to vehicle 1. In this embodiment, instead of radar 22, a laser radar or ultrasonic sensor may be used to measure the distance to the object and the relative speed. Alternatively, a position and speed measuring device may be configured using multiple sensors. Note that radar 22 corresponds to an example of a "driving environment information acquisition device" in the present invention.

[0027] The navigation system 23 stores map information internally and can provide this map information to the controller 10. Based on the map information and current vehicle position information, the controller 10 identifies roads, intersections, traffic signals, buildings, etc., that exist around the vehicle 1 (especially in the direction of travel). The map information may also be stored within the controller 10. The positioning system 24 is a GPS system and / or a gyro system and detects the position of the vehicle 1 (current vehicle position information). The navigation system 23 and the positioning system 24 also correspond to examples of "driving environment information acquisition devices" in the present invention.

[0028] The vehicle speed sensor 25 detects the speed of the vehicle 1 based on, for example, the rotational speed of the wheels or drive shaft. The acceleration sensor 26 detects the acceleration of the vehicle 1. This acceleration includes the acceleration in the longitudinal direction of the vehicle 1 and the acceleration in the lateral direction (i.e., lateral acceleration). In this specification, acceleration includes not only the rate of change of speed in the direction in which the speed increases, but also the rate of change of speed in the direction in which the speed decreases (i.e., deceleration).

[0029] The yaw rate sensor 27 detects the yaw rate of the vehicle 1. The steering angle sensor 28 detects the rotation angle (steering angle) of the steering wheel of the steering device 5. The steering torque sensor 29 detects the torque (steering torque) applied to the steering shaft via the steering wheel. The accelerator sensor 30 detects the amount the accelerator pedal is depressed. The brake sensor 31 detects the amount the brake pedal is depressed.

[0030] The in-vehicle camera 32 photographs the driver and outputs image data. The controller 10 detects the driver's gaze direction based on the image data received from the in-vehicle camera 32. The in-vehicle camera 32 is an example of a "gaze detection device" in this invention.

[0031] The PCM33 controls the power source 2 of the vehicle 1 to adjust the driving force of the vehicle 1. For example, the PCM33 controls the engine's spark plugs, fuel injectors, throttle valves, variable valve timing mechanism, transmission 3, and inverter that supplies power to the electric motor. When the controller 10 needs to accelerate or decelerate the vehicle 1, it sends a control signal to the PCM33 to adjust the driving force.

[0032] The DSC34 controls the vehicle 1's drive source 2 and brake 4 to perform deceleration control and attitude control of the vehicle 1. For example, the DSC34 controls the hydraulic pump and valve unit of the brake 4 and controls the drive source 2 via the PCM33. When the controller 10 needs to perform deceleration control or attitude control of the vehicle 1, it sends control signals to the DSC34 to adjust the drive force or generate braking force.

[0033] The EPS35 controls the steering system 5 of the vehicle 1. For example, the EPS35 controls an electric motor that applies torque to the steering shaft of the steering system 5. When the controller 10 needs to change the direction of travel of the vehicle 1, it sends a control signal to the EPS35 to change the steering direction.

[0034] The display 36 is located in front of the driver inside the vehicle and displays image information to the driver. For example, an LCD display or a head-up display can be used as the display 36. The speaker 37 is installed inside the vehicle and outputs various audio information.

[0035] [Overview of Driver Abnormality Prediction Detection] Next, with reference to Figure 3, the basic concept of driver abnormality prediction detection performed by the controller 10 described above in this embodiment will be explained. Figure 3 is a control block diagram of abnormality prediction detection according to this embodiment.

[0036] In the premonitory state of an abnormal situation that precedes a driver reaching a state where driving becomes difficult, it is thought that the driver's attention function is beginning to decline due to a minor illness or aging. Therefore, the inventors of this invention believe that it is possible to detect this premonitory state of an abnormal situation by detecting this decline in the driver's attention function.

[0037] A driver's attention function primarily includes the ability to view multiple objects simultaneously (distributed attention), the ability to select and view multiple objects (selective attention), the ability to switch between objects (shifting attention), and the ability to continuously view an object (sustained attention). Research by the inventors has shown that when attention function declines due to certain diseases (e.g., heart disease, brain disease, hypoglycemia, etc.) or aging, the decline in attention function occurs in the order of distributed attention → selective attention → shifting attention → sustained attention. In other words, in the initial stages of attention function decline, the decline in distributed attention prevents the driver from simultaneously viewing multiple points that should be visually confirmed (recognized by sight) while driving, leading to oversights of these points. Therefore, an abnormal warning state can be detected based on the extent to which oversights of points that should be visually confirmed are occurring.

[0038] Furthermore, research conducted by the inventors revealed that in patients with impaired attention, the effective field of view (the area within the entire field of view from which information can be effectively obtained) is narrowed, and even when the gaze is directed close to an object, the object cannot be seen. In other words, in the early stages of impaired attention, the narrowing of the effective field of view makes it difficult to see the area that needs to be seen, leading to the overlooking of the area that needs to be seen. Therefore, by estimating the degree to which the effective field of view has narrowed from the distance between the line of sight and the area that needs to be seen when the area that needs to be seen is not seen, it is possible to detect an abnormal premonitory state.

[0039] Therefore, the controller 10 of this embodiment is configured to detect an abnormal condition in the driver by identifying areas requiring visibility based on driving environment information, determining whether the driver has seen these areas based on the driver's line of sight, and, based on whether the driver has seen these areas, identifying the degree to which the areas requiring visibility were overlooked and the width of the effective field of view.

[0040] Specifically, as shown in Figure 3, the controller 10 acquires driving environment information (acquisition of driving environment information) based on signals received from sensors including an external camera 21, radar 22, navigation system 23, and positioning system 24. The driving environment information is information about the environment in which the vehicle 1 is driving, and includes, for example, information about the type and location of objects present around the vehicle 1, such as preceding vehicles, parked vehicles, pedestrians, roads, lane markings (lane boundaries, white lines, yellow lines), traffic signals, traffic signs, stop lines, intersections, and obstacles, as well as information about the type and shape of the road on which the vehicle 1 is driving, surrounding buildings, and the current location of the vehicle 1.

[0041] Based on the acquired driving environment information, the controller 10 refers to a database of locations requiring visual inspection, which stores the correspondence between the driving environment and locations requiring visual inspection, and identifies locations that the driver should visually inspect (identification of locations requiring visual inspection). The database of locations requiring visual inspection is stored in memory 10b or the like beforehand.

[0042] Figure 4 is a table illustrating the information stored in the visibility-required location database according to this embodiment. Each column in the visibility-required location database illustrated in Figure 4 represents the driving environment in which Vehicle 1 is traveling (for example, "straight ahead" and "curve" in "straight ahead"). Each row in the visibility-required location database illustrated in Figure 4 represents the visibility-required location that the driver should check (for example, "direction of travel" and "rearview mirror"). In each driving environment, the visibility-required location that the driver should check is represented by "○". Based on the acquired driving environment information, the controller 10 identifies the driving environment of Vehicle 1 and retrieves the visibility-required locations in the identified driving environment from the visibility-required location database. For example, if the driving environment is "intersection" and "left turn", in the example in Figure 4, the visibility-required locations are identified as "direction of travel", "left door mirror", "left side", "preceding vehicle", "traffic light", "crossing point", "pedestrian", and "two-wheeled vehicle".

[0043] Furthermore, the controller 10 detects the driver's gaze based on the signal received from the in-vehicle camera 32 (gaze detection). The controller 10 then determines whether the driver has viewed the identified area requiring visibility based on the detected driver's gaze (viewing determination). Specifically, the controller 10 determines that the driver has viewed the area requiring visibility if the driver's gaze is directed within a predetermined range that includes the area requiring visibility (for example, if the angle between the direction from the driver's head towards the area requiring visibility and the driver's gaze is within 5 degrees). This predetermined range is set in advance and stored in memory 10b, etc.

[0044] Furthermore, the controller 10 detects driving operations performed by the driver based on signals received from sensors including a vehicle speed sensor 25, an acceleration sensor 26, a yaw rate sensor 27, a steering angle sensor 28, a steering torque sensor 29, an accelerator sensor 30, and a brake sensor 31 (driving operation detection). Driving operations include, for example, acceleration, deceleration, left turn, and right turn.

[0045] The controller 10 then determines whether the driver has seen the identified area requiring visibility, based on the detected driver's gaze and driving operations (visibility determination). Specifically, the controller 10 determines that the driver has seen the area requiring visibility if the driver's gaze is directed within a predetermined range including the area requiring visibility (for example, if the angle between the direction from the driver's head towards the area requiring visibility and the driver's gaze is within 30 degrees), and if a driving operation corresponding to that area is subsequently performed. This predetermined range is set in advance and stored in memory 10b, etc. Furthermore, "driving operation corresponding to the area requiring visibility" refers to the driving operation that the driver should perform when the area requiring visibility is seen (for example, if the area requiring visibility is a red light, the corresponding driving operation is deceleration), and this is set in advance for each area requiring visibility and stored in memory 10b, etc.

[0046] Furthermore, the controller 10 calculates the ratio of the number of areas that the driver did not see to the total number of areas that the driver did see (oversight rate calculation) based on the results of a visibility determination based on whether or not the driver's gaze is directed within a predetermined range including the areas that need to be seen (oversight rate calculation). That is, the oversight rate Fr can be calculated by the following formula. The rate of oversight, Fr, is calculated as follows: (Number of locations that the driver did not see) / (Total number of locations that the driver did not see).

[0047] Then, if the oversight rate Fr is above a predetermined threshold A (for example, 40% or more), it is determined that the driver is in an abnormal condition. In other words, an abnormal condition of the driver is detected (abnormal condition detection). This threshold A is set in advance and stored in memory 10b, etc.

[0048] Furthermore, the controller 10 estimates the width of the effective field of view (effective field of view estimation) based on the results of a visibility determination, which is based on whether or not a driving operation corresponding to a visibility point was performed when the driver's gaze was directed within a predetermined range including the visibility point. Specifically, the controller 10 obtains the minimum value (minimum angle outside the effective field of view) from the angle between the direction from the driver's head towards the visibility point and the driver's line of sight for each visibility point that the driver determined not to have seen. The controller 10 also obtains the maximum value (maximum angle within the effective field of view) from the angle between the direction from the driver's head towards the visibility point and the driver's line of sight for each visibility point that the driver determined to have seen. The controller 10 then takes the midpoint between the minimum angle outside the effective field of view and the maximum angle within the effective field of view as the width of the effective field of view (effective field of view angle) Fv.

[0049] Then, if the effective field of view Fv is less than a predetermined threshold B (for example, less than 5 degrees), the system determines that the driver is in an abnormal condition. In other words, it detects the driver's abnormal condition (abnormal condition detection). This threshold B is set in advance and stored in memory 10b, etc.

[0050] When an abnormal condition is detected, the controller 10 sends control signals to the PCM 33, DSC 34, and EPS 35 to appropriately activate the power source 2, transmission 3, brakes 4, and steering system 5, and sends control signals to the display 36 and speaker 37 to output desired information. For example, the controller 10 displays image information on the display 36 to guide the driver's gaze to areas that the driver has not seen, or outputs audio information from the speaker 37.

[0051] [Anomaly detection process] Next, with reference to Figure 5, the flow of the abnormality prediction state detection process by the driver abnormality prediction detection device 100 of this embodiment will be described. Figure 5 is a flowchart of the abnormality prediction detection process.

[0052] The abnormality prediction detection process shown in Figure 5 is started when the power to vehicle 1 is turned ON and is repeatedly executed by the controller 10 at a predetermined interval (for example, every 0.05 to 0.2 seconds).

[0053] When the abnormality prediction detection process is started, the controller 10 first acquires driving environment information based on signals received from sensors including the external camera 21, radar 22, navigation system 23, and positioning system 24 (step S1).

[0054] Furthermore, the controller 10 detects the driver's line of sight based on the signal received from the in-vehicle camera 32 (step S2). Then, based on the driving environment information acquired in step S1, the controller 10 identifies areas that the driver should check, and determines whether the driver has checked the areas based on the identified areas and the driver's line of sight detected in step S2 (step S3). As described above, the controller 10 determines that the driver has checked the areas if the driver's line of sight is directed within a predetermined range that includes the areas that need to be checked. In addition, the controller 10 determines that the driver has checked the areas if, when the driver's line of sight is directed within a predetermined range that includes the areas that need to be checked, a driving operation corresponding to the areas that need to be checked is subsequently performed.

[0055] Next, the controller 10 determines whether the driver is in an abnormal state based on signals received from sensors including the external camera 21, radar 22, navigation system 23, positioning system 24, vehicle speed sensor 25, acceleration sensor 26, yaw rate sensor 27, steering angle sensor 28, steering torque sensor 29, accelerator sensor 30, brake sensor 31, and the internal camera 32 (step S4).

[0056] For example, the controller 10 detects the amplitude and frequency of the driver's saccades, as described in Japanese Patent Publication No. 2021-077136, and also detects the level of attention, which increases as the number of points of attention that the driver needs to check in the external environment of the vehicle increases while driving. Based on the level of attention and the amplitude and frequency of the driver's saccades, the controller 10 determines whether the driver is in an abnormal state. Alternatively, the controller can detect the driver's gaze direction, posture (position of the upper body or head), degree of eyelid opening, and the driver's gripping force on the steering wheel, and determine whether the driver is in an abnormal state based on these detection results. For example, the controller 10 can determine that the driver is in an abnormal state if the stability of the gaze direction is below a predetermined value, the stability of the posture is below a predetermined value, the eyelids are closed for a predetermined time or longer, or the gripping force on the steering wheel is less than a predetermined value. Furthermore, for example, the controller 10 can estimate that the driver is in an abnormal state if the stability of the vehicle 1's position relative to the center line on the road, the stability of the steering angle, etc., are below a predetermined value.

[0057] If the system determines that the driver is in an abnormal state (Step S4: Yes), the controller 10 sends control signals to the display 36 and speaker 37 to output an alarm, and also sends control signals to the PCM 33, DSC 34, and EPS 35 to appropriately activate the drive source 2, transmission 3, brakes 4, and steering system 5, thereby controlling the behavior of the vehicle 1, for example, to safely stop the vehicle 1 on the shoulder of the road (Step S5). After Step S5, the controller 10 terminates the abnormality prediction detection process.

[0058] On the other hand, if it is determined that the driver is not in an abnormal state (Step S4: No), the controller 10 calculates the oversight rate Fr from the results of the visibility determination based on whether or not the driver's gaze is directed within a predetermined range including the area that needs to be seen (Step S6).

[0059] Next, the controller 10 determines whether the oversight rate Fr calculated in step S6 is equal to or greater than a predetermined threshold A (step S7). If the oversight rate Fr is equal to or greater than the predetermined threshold A (step S7: Yes), the controller 10 determines that the driver is in an abnormal warning state. In other words, it detects the driver's abnormal warning state (step S8).

[0060] Next, the controller 10 sends control signals to the display 36 and speaker 37, causing the display 36 and speaker 37 to output image information and audio information (eye-tracking information) to guide the driver's gaze to areas that the driver did not see (step S9). After step S9, the controller 10 terminates the abnormality prediction process.

[0061] Furthermore, in step S7, if the oversight rate Fr is not equal to or greater than a predetermined threshold A (i.e., less than threshold A) (step S7: No), the controller 10 estimates the effective field of view Fv from the results of the visibility determination in step S3, which is based on whether or not a driving operation corresponding to the area to be seen was performed when the driver's gaze was directed within a predetermined range including the area to be seen (step S10).

[0062] Next, the controller 10 determines whether the effective field of view Fv estimated in step S10 is less than a predetermined threshold B (step S11). If the effective field of view Fv is less than the predetermined threshold B (step S11: Yes), the controller 10 determines that the driver is in an abnormal condition. That is, it detects the driver's abnormal condition (step S8). In this case, the controller 10 outputs eye-tracking guidance information from the display 36 and speaker 37 (step S9), and terminates the abnormal condition detection process.

[0063] On the other hand, in step S11, if the effective field of view Fv is not less than a predetermined threshold B (threshold B is abnormal) (step S11: No), the controller 10 determines that the driver is not in an abnormal condition. That is, it does not detect an abnormal condition in the driver (step S12). After step S12, the controller 10 terminates the abnormal condition detection process.

[0064] In this embodiment, the abnormality prediction process detects the driver's abnormality state based on whether the oversight rate Fr is greater than or equal to a predetermined threshold A, and whether the effective field of view Fv is less than a predetermined threshold B. However, the abnormality state may be detected based on only one of either the oversight rate Fr or the effective field of view Fv.

[0065] Furthermore, in the abnormality prediction detection process of this embodiment, the controller 10 detects the driver's abnormality prediction state when it is determined in step S4 that the driver is not in an abnormal state (step S4: No). However, the determination of whether or not the driver is in an abnormal state may be omitted, or the determination of whether or not the driver is in an abnormal state may be made after detecting the abnormality prediction state.

[0066] Furthermore, in this embodiment, the effective field of view Fv is estimated from the results of a visibility determination based on whether or not a driving operation corresponding to a point requiring visibility was performed when the driver's gaze was directed to a predetermined range including that point requiring visibility. However, the effective field of view may be estimated by a different method. For example, the effective field of view can also be estimated based on the angle between the direction from the driver's head towards an object and the driver's gaze when the driver's gaze is directed to an object with high saliency (for example, an object with a large color difference or brightness difference compared to the surrounding area, or an object that is moving significantly relative to its surroundings).

[0067] [Effects / Effects] Next, the operation and effects of the driver abnormality prediction device 100 of this embodiment described above will be explained.

[0068] The controller 10 identifies areas that the driver should visually check based on driving environment information, determines whether the driver has visually checked these areas based on the driver's line of sight, and detects a premonitory state of driver abnormality based on whether the driver has not visually checked these areas. This allows for the detection of a premonitory state of abnormality by utilizing the fact that the driver was unable to visually check these areas due to a decrease in distributed attention function or a reduction in the effective field of view in the early stages of a decline in the driver's attention function. As a result, a premonitory state of driver abnormality can be detected at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult, for example, when there are no clear differences in the amplitude or frequency of the driver's saccades.

[0069] Furthermore, the controller 10 determines that the driver has seen a designated area if the driver's gaze is directed within a predetermined range that includes the area requiring visibility, and detects a warning sign of the driver's abnormal behavior based on the number of areas that the driver did not see. Therefore, in the initial stages of a decline in attentional function, when a decline in distributed attention prevents the driver from simultaneously seeing multiple areas requiring visibility and causes areas to be overlooked, the controller 10 can detect a warning sign of abnormal behavior based on the extent to which areas are being overlooked. This allows for the appropriate detection of a warning sign of abnormal behavior at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult.

[0070] Furthermore, the controller 10 is configured to detect a driver's abnormal state when the rate of overlooking areas requiring visual inspection exceeds a predetermined value. Therefore, in the initial stages of a decline in attention function, if the driver becomes unable to simultaneously view multiple areas requiring visual inspection due to a decline in distributed attention function, and the rate of overlooking areas requiring visual inspection exceeds a predetermined value, the abnormal state can be detected. This allows for the appropriate detection of a driver's abnormal state at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult.

[0071] Furthermore, the controller 10 estimates the width of the effective field of view visible to the driver based on the distance between the driver's line of sight and the area that the driver failed to see. If the estimated width of the effective field of view is less than a predetermined value, the controller detects a warning sign of the driver's abnormal condition. Therefore, when the effective field of view narrows and areas that require visibility begin to be overlooked in the early stages of a decline in attentional function, the controller can estimate how much the effective field of view has narrowed, thereby detecting a warning sign of the driver's abnormal condition at a sufficiently early stage before the driver reaches an abnormal state that makes driving difficult.

[0072] Furthermore, the driver abnormality prediction device 100 is equipped with a display 36 and a speaker 37 that output information to the driver. When the controller 10 detects a driver abnormality, it outputs information via the display 36 and speaker 37 that guides the driver's gaze to areas requiring visual attention. Therefore, when a driver abnormality is detected, the driver's gaze can be guided to areas requiring visual attention, providing appropriate driving assistance to drivers who have begun to overlook areas requiring visual attention in the early stages of a decline in attentional function. [Explanation of Symbols]

[0073] 1 vehicle 10 Controllers 100 Driver abnormality prediction device 21. Exterior car camera 22 Radar 23 Navigation System 24 Positioning Systems 25. Vehicle speed sensor 26 Accelerometer 27 Yaw rate sensor 28 Steering Angle Sensor 29 Steering Torque Sensor 30 Accelerator sensor 31 Brake Sensor 32 In-car cameras 36 displays 37 speakers

Claims

1. A driver abnormality prediction device that detects abnormal signs in the driver of a vehicle, A driving environment information acquisition device that acquires driving environment information of the aforementioned vehicle, A gaze detection device for detecting the driver's line of sight, The system includes a controller configured to detect abnormal signs in the driver based on the aforementioned driving environment information and the driver's gaze, The aforementioned controller, Based on the acquired driving environment information, multiple points of interest that the driver should visually check in the driving environment of the vehicle are identified. Based on the identified multiple locations requiring visual inspection and the driver's line of sight, it is determined whether or not the driver has visually inspected the locations requiring visual inspection. The system is configured to detect an abnormal condition in the driver based on the fact that the driver failed to visually inspect the area requiring visual inspection. The aforementioned controller, Based on the angle formed by the driver's line of sight and the direction from the driver's head toward the area to be seen, and the result of determining whether or not the driver has seen the area to be seen, the width of the effective field of view visible to the driver is estimated. The system is configured to detect an abnormal condition in the driver if the estimated effective field of view is less than a predetermined value. Driver abnormality warning device.

2. The aforementioned controller, If the driver's line of sight is directed within a predetermined range including the area requiring visibility, it is determined that the driver has seen the area requiring visibility. The system is configured to detect the driver's abnormal behavior based on the number of locations that the driver failed to visually confirm. The driver abnormality prediction device according to claim 1.

3. The controller is configured to detect an abnormal condition in the driver when the ratio of the number of locations that the driver did not see to the total number of locations that the driver did see exceeds a predetermined value. The driver abnormality prediction device according to claim 2.

4. A sensor for detecting the driving operations performed by the aforementioned driver, Each of the aforementioned locations requiring visual inspection is provided with a memory for storing the driving operations corresponding to that location. The aforementioned controller, Based on the signals received from the aforementioned sensors, the driving operations performed by the driver are detected. When the driver's gaze is directed within a predetermined range including the area requiring visual inspection, and a driving operation corresponding to that area requiring visual inspection is subsequently performed, it is determined that the driver has visually inspected that area requiring visual inspection. The driver abnormality prediction device according to claim 1.

5. Furthermore, it is equipped with an information output device that outputs information to the driver, The controller is configured to output information via the information output device that guides the driver's gaze to the area requiring visual inspection when it detects an abnormal condition in the driver. A driver abnormality prediction device according to any one of claims 1 to 4.

Images