Driver state determination method and determination system

Abstract

To provide a driver state determination method and a driver state determination system which can secure accuracy in state determination regardless of a learning period of time for a visual line of a driver.SOLUTION: A driver state determination system S that determines a state of a driver in a vehicle comprises: a learning information generating part 26a that generates a second correction coefficient β having a physical state of the vehicle, characteristics of the driver and visual line behavior of the driver associated with one another; a learning visual line distribution generating part 26 that generates a second corrected visual line distribution Y using a reference visual line distribution A and the second correction coefficient β; a learning progress-degree calculating part 27a that calculates a learning progress degree St on the basis of the number C2 of images used in generating the second corrected visual line distribution Y; and an abnormality estimating part 27 that determines that the driver is abnormal when the difference between the visual line behavior distribution B and the second corrected visual line distribution Y is equal to or more than a determination threshold, and increases a weighing ratio of elicited characteristics with respect to latent characteristics as the learning progress-degree St becomes larger.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driver state determination method and a determination system for determining the state of a driver of a vehicle. [Background technology]

[0002] Recently, the development of automated driving systems has been promoted nationwide. The applicant believes that there are currently two main directions for automated driving systems.

[0003] The first direction is a system in which the vehicle takes the lead in transporting passengers to their destination without the need for driver operation, known as fully automated driving of automobiles. The second direction is an autonomous driving system that is based on the premise that humans will be driving, with the aim of "providing an environment in which driving a car is enjoyable."

[0004] In the second direction of automated driving systems, it is expected that the vehicle will automatically take over driving in place of the occupants in the event that, for example, the driver develops an illness or other condition that makes normal driving difficult. For this reason, it is extremely important from the perspective of improving the driver's survival rate and ensuring the safety of those around them to detect as early and accurately as possible any abnormalities in the driver, particularly any functional impairment or illness that the driver has.

[0005] As a method for determining abnormalities in a driver, for example, Patent Document 1 discloses a computational framework system that uses a convolutional neural network based on a Bayesian framework and incorporating task-dependent top-down and bottom-up factors. Salient regions in driving scenes are adjusted using weights estimated based on priors learned from driving data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2020-509466 Summary of the Invention [Problem to be solved by the invention]

[0007] The technique of Patent Document 1 incorporates task-dependent top-down and bottom-up factors into the estimation to determine important and salient areas to which the driver will pay attention. However, the technology of Patent Document 1 requires learning a prior distribution for estimating the weight for each factor from the driver's actual driving data, and there is a risk that judgment accuracy cannot be ensured until learning of the prior distribution is completed.

[0008] Furthermore, depending on the type of information processing, there are some information processing methods that are difficult to reflect personal characteristics, and some that are easy to reflect personal characteristics. Therefore, in a status determination system that employs information processing methods that are difficult to reflect personal characteristics of gaze behavior, there is a concern that when collecting learning information about personal characteristics, the learning period for collecting the learning information will be extended. That is, it is not easy to ensure state determination accuracy regardless of the driver's line of sight learning period.

[0009] An object of the present invention is to provide a driver state determination method and a determination system therefor that can ensure state determination accuracy regardless of the driver's gaze learning period. [Means for solving the problem]

[0010] The invention of claim 1 is a driver state determination method for determining the state of a driver of a vehicle, comprising: a surrounding information acquisition step for acquiring information on a surrounding situation of the vehicle; a vehicle state acquisition step for acquiring a physical state of the vehicle; a driver characteristic acquisition step for acquiring characteristics of the driver; currenta reference gaze distribution generation step of generating a reference gaze distribution based on a standard field of view image for a latent characteristic reflecting the low-order processing of the brain and an explicit characteristic reflecting the high-order processing of the brain; a gaze behavior distribution generation step of generating a gaze behavior distribution based on the gaze behavior of the driver for the latent characteristic reflecting the low-order processing of the brain and the explicit characteristic reflecting the high-order processing of the brain; a learning information generation step of generating learning information relating a physical state of the vehicle, a characteristic of the driver, and the gaze behavior of the driver; a driver learning gaze distribution generation step of generating a learning gaze distribution using the reference gaze distribution and the learning information; a learning progress calculation step of calculating a learning progress based on the number of images used to generate the learning gaze distribution; and a learning progress calculation step of calculating a learning progress based on the number of images used to generate the learning gaze distribution when a difference between the gaze behavior distribution and the learning gaze distribution is equal to or greater than a judgment threshold. but abnormality That is and a driver abnormality inference step of determining whether the driver is in a state where ...

[0011] This driver state determination method includes a learning information generation step of generating learning information that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver, and a driver learning gaze distribution generation step of generating a learning gaze distribution using the reference gaze distribution and the learning information, so that the learning gaze distribution can be generated using the learning information that associates the gaze behavior of the driver.A learning progress calculation step of calculating a learning progress degree based on the number of images used to generate the learning gaze distribution, and a step of calculating a learning progress degree based on the number of images used to generate the learning gaze distribution when the difference between the gaze behavior distribution and the learning gaze distribution is equal to or greater than a determination threshold. but abnormality That isand a driver abnormality inference step of determining whether or not a driver is abnormal, and increasing the weighting ratio of the manifest trait to the latent trait as the learning progress increases, so that when the driver's gaze learning period is insufficient, the weight of the latent trait, which is less affected by individual differences, can be made greater than the weight of the manifest trait, which is more affected by individual differences, and the determination can be made mainly on the bottom-up function, which is a low-level processing of the brain.Also, when the driver's gaze learning period is sufficient, the weight of the latent trait, which is less affected by individual differences, can be made less than the weight of the manifest trait, which is more affected by individual differences, and the determination can be made mainly on the top-down function, which is a high-level processing of the brain.

[0012] The invention of claim 2 is characterized in that, in the invention of claim 1, the driver abnormality estimation process determines the difference between the gaze behavior distribution and the learned gaze distribution as the sum of the difference between the probability distribution corresponding to the latent characteristics of the gaze behavior distribution and the probability distribution corresponding to the latent characteristics of the learned gaze distribution and the difference between the probability distribution corresponding to the manifest characteristics of the gaze behavior distribution and the probability distribution corresponding to the manifest characteristics of the learned gaze distribution. According to this configuration, the driver's gaze behavior can be evaluated from both the bottom-up function and the top-down function perspectives.

[0013] The invention of claim 3 is a driver state determination system for determining the state of a driver of a vehicle, comprising: a surrounding information acquisition means for acquiring information on a surrounding situation of the vehicle; a vehicle state acquisition means for acquiring a physical state of the vehicle; a driver characteristic acquisition means for acquiring characteristics of the driver; current a reference gaze distribution generating means for generating a reference gaze distribution based on a standard field of view image for a latent characteristic reflecting the brain's low-level processing and an explicit characteristic reflecting the brain's high-level processing; a gaze behavior distribution generating means for generating a gaze behavior distribution based on the gaze behavior of the driver for the latent characteristic reflecting the brain's low-level processing and the explicit characteristic reflecting the brain's high-level processing; a learning information generating means for generating learning information relating the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver; a learning gaze distribution generating means for generating a learning gaze distribution using the reference gaze distribution and the learning information; Driver Learning gaze distribution generation means a learning progress calculation means for calculating a learning progress degree based on the number of images used in the learning; and a learning progress calculation means for calculating a learning progress degree based on the number of images used in the learning; and a learning progress calculation means for calculating a learning progress degree based on the number of images used in the learning; but abnormality That is and a driver abnormality estimation means for determining whether the driver is in a state where ...

[0014] This driver state determination system has a learning information generation means for generating learning information that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver, and a driver learning gaze distribution generation means for generating a learning gaze distribution using the reference gaze distribution and the learning information, so that a learning gaze distribution can be generated using the learning information. a learning progress calculation means for calculating a learning progress degree based on the number of images used to generate the learning gaze distribution; and a learning progress calculation means for calculating a learning progress degree based on the number of images used to generate the learning gaze distribution when a difference between the gaze behavior distribution and the learning gaze distribution is equal to or greater than a determination threshold. but abnormality That is and a driver abnormality estimation means for estimating the degree of learning progress and for increasing the weighting ratio of the manifest trait to the latent trait as the learning progress increases, so that when the driver's gaze learning period is insufficient, the weight of the latent trait, which is less affected by individual differences, can be made greater than the weight of the manifest trait, which is more affected by individual differences, and the judgment can be made mainly on the bottom-up function, which is a low-level processing of the brain.Also, when the driver's gaze learning period is sufficient, the weight of the latent trait, which is less affected by individual differences, can be made less than the weight of the manifest trait, which is more affected by individual differences, and the judgment can be made mainly on the top-down function, which is a high-level processing of the brain. [Effects of the Invention]

[0015] According to the driver state determination method and the determination system of the present invention, the weight of the latent characteristic and the weight of the explicit characteristic are adjusted according to the learning progress. fruit The weight of Add By changing the time t, it is possible to ensure the accuracy of state determination regardless of the driver's line of sight learning period. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating a configuration of a driver state determination system according to a first embodiment. [Figure 2] FIG. 4 is a diagram illustrating processing by a control unit. [Figure 3] The gaze behavior distributions show a normal distribution corresponding to saliency, a normal distribution corresponding to saccade amplitude, and a normal distribution corresponding to saccade frequency. [Figure 4] FIG. 10 is a diagram showing an example of a time change in saliency related to a driver's gaze point. [Figure 5] FIG. 4 is a diagram showing an example of time-dependent changes in saliency related to random points whose coordinates are randomly specified in the same environment as FIG. 3. [Figure 6] 10 is a graph comparing the probability of exceeding a threshold at a random point with the probability of exceeding a threshold at a driver's gaze point. [Figure 7] FIG. 1 is an explanatory diagram of a saliency index. [Figure 8] FIG. 10 is a diagram showing the time series distribution of saliency indices. [Figure 9] FIG. 10 is a diagram showing a time series distribution of risk potentials. [Figure 10] FIG. 10 is a diagram showing a combined time series distribution of a saliency index and a risk potential. [Figure 11] FIG. 10 is a diagram showing the probability distribution of saliency indices for each divided area of ​​the risk potential. [Figure 12] 10 is a graph for explaining extraction of a saccade. [Figure 13] 10 is a graph for explaining a noise removal process of a saccade. [Figure 14] 10 is a flowchart of a pre-learning process. [Figure 15] 10 is a flowchart of a state determination process. [Figure 16] 10 is a flowchart of a total evaluation value calculation process. [Figure 17]10 is a flowchart of a learning phase process. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following description exemplifies the application of the present invention to a driver state determination system for a vehicle, and does not limit the present invention, its applications, or its uses. The following description also includes a description of a driver state determination method. [Example]

[0018] A first embodiment of the present invention will be described below with reference to FIGS. This driver state determination system S is a system that estimates the state of a driver by detecting signs of an abnormal state of the driver based on the gaze behavior of the driver operating the vehicle. The vehicle is, for example, a right-hand drive four-wheeled automatic vehicle that can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels according to the driver's operation (for example, accelerator operation, etc.). Assisted driving is driving in which the vehicle assists the driver's operation. Automatic driving is driving in which the vehicle travels without the driver's operation.

[0019] 1, the driver state determination system S controls the behavior of the vehicle in manual driving and assisted driving. Specifically, the driver state determination system S controls the operation of the vehicle (particularly the driving operation) by controlling an actuator 3 provided in the vehicle. The driver condition determination system S mainly comprises an information acquisition unit 1, a control unit 2, an actuator 3, and a notification unit 4. When an abnormality in the driver is determined, the actuator 3 causes the vehicle to retreat to a predetermined safety area, and the notification unit 4 notifies the management center of the abnormality in the driver.

[0020] First, the information acquisition unit 1 will be described. The information acquisition unit 1 acquires various types of information used for vehicle control, mainly for driving control, various types of information related to the vehicle surroundings, and various types of information used for determining the driver's state. As shown in FIG. 1, the information acquisition unit 1 includes a plurality of external cameras 11, an internal camera 12, a plurality of radars 13, a position sensor 14, an external input unit 15, a vehicle state sensor 16, a driving operation sensor 17, a biometric information sensor 18, and the like. The external camera 11, the multiple radars 13, the position sensor 14, and the external input unit 15 correspond to the surrounding information acquisition means, the internal camera 12 and the biometric information sensor 18 correspond to the driver characteristic acquisition means, and the vehicle state sensor 16 and the driving operation sensor 17 correspond to the vehicle state acquisition means.

[0021] The multiple external cameras 11 are provided to cover the entire area around the vehicle and have the same configuration. Each external camera 11 captures an image of a part of the environment surrounding the vehicle (external environment around the vehicle) to obtain image data showing a part of the external environment of the vehicle. The image data acquired by the external camera 11 is transmitted to the control unit 2. The external camera 11 is a monocular camera with a wide-angle lens, and is configured using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor).

[0022] The interior camera 12 is provided inside the vehicle. This internal camera 12 captures an image of a predetermined area including the driver's eyeballs to obtain image data including the driver's eyes. The internal camera 12 is disposed in front of the driver, and its imaging range is set so that the driver's eyeballs are within the imaging range. The vehicle has a drowsiness determination device (not shown) for determining the driver's drowsiness, and the internal camera 12 also serves as the imaging means of the drowsiness determination device. The image data obtained by the internal camera 12 is transmitted to the control unit 2.

[0023] The multiple radars 13 are provided to cover the entire area around the vehicle and have similar configurations. The radars 13 detect parts of the external environment of the vehicle (e.g., obstacles such as other vehicles or buildings) in order to determine the direction, relative speed, and distance to those parts. The radar 13 detects a part of the external environment of the vehicle by transmitting radio waves toward the part of the external environment and receiving waves reflected from the part of the external environment. The detection result of the radar 13 is transmitted to the control unit 2.

[0024] The position sensor 14 detects the position (for example, latitude and longitude) of the vehicle. The sensor 14 receives GPS information from a global positioning system and detects the position of the vehicle based on the GPS information. The vehicle position information obtained by the position sensor 14 is transmitted to the control unit 2.

[0025] The external input unit 15 inputs information via an external network (for example, the Internet) provided outside the vehicle. The external input unit 15 receives communication information (vehicle-to-vehicle communication information) from other vehicles located around the vehicle, car navigation data from a navigation system, traffic information, high-precision map information, etc. The vehicle position information obtained by the external input unit 15 is transmitted to the control unit 2.

[0026] The vehicle state sensor 16 detects the vehicle state (for example, speed, acceleration, yaw rate, etc.). The vehicle state sensor 16 includes a vehicle speed sensor that detects the traveling speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, etc. The vehicle state information obtained by the vehicle state sensor 16 is transmitted to the control unit 2.

[0027] The driving operation sensor 17 detects driving operations applied to the vehicle and includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of accelerator pedal operation. The steering angle sensor detects the amount of steering wheel operation. The brake oil pressure sensor detects the amount of brake pedal operation. Vehicle driving operation information obtained by driving operation sensor 17 is sent to control unit 2.

[0028] The biological information sensor 18 detects biological information of the driver (for example, sweating, heart rate, etc.). The information obtained by the biological information sensor 18 is transmitted to the control unit 2.

[0029] Next, the control unit 2 will be described. As shown in Figure 2, the control unit 2 determines the driver's abnormal state based on an evaluation value (difference), which is the distance between the gaze behavior distribution Ba, Bb, Bc based on the driver's most recent gaze behavior and the second corrected gaze distribution Ya, Yb, Yc (learned gaze distribution) that reflects external disturbance factors. The second corrected gaze distributions Ya, Yb, Yc are obtained by correcting the reference gaze distributions Aa, Ab, Ac based on a standard field of view image using a first correction coefficient α (described later) to obtain the first corrected gaze distributions Xa, Xb, Xc, and then correcting the first corrected gaze distributions Xa, Xb, Xc using a second correction coefficient β (learning information) (described later).

[0030] The distributions Aa, Ba, Xa, and Ya are normal distributions for saliency (attractiveness), the distributions Ab, Bb, Xb, and Yb are normal distributions for saccade (saccadic eye movement) amplitude, and the distributions Ac, Bc, Xc, and Yc are normal distributions for saccade frequency. Saliency and saccade frequency are indicators of the brain's low-level processing, or the driver's latent characteristics, while saccade amplitude is an indicator of the brain's high-level processing, or the driver's explicit characteristics. Unless otherwise specified, the reference gaze distributions Aa, Ab, Ac will be collectively referred to as the reference gaze distribution A, the gaze behavior distributions Ba, Bb, Bc as the gaze behavior distribution B, the first corrected gaze distributions Xa, Xb, Xc as the first corrected gaze distribution X, and the second corrected gaze distribution Ya, Yb, Yc as the second corrected gaze distribution Y.

[0031] As shown in FIG. 1, the control unit 2 mainly includes an image processing unit 21, an attention level detection unit 22, a field of view image creation unit 23 (field of view image creation means), a gaze behavior distribution generation unit 24 (gaze behavior distribution generation means), a correction unit 25, a learning gaze distribution generation unit 26 (learning gaze distribution generation means), and an abnormality estimation unit 27 (driver abnormality estimation means).

[0032] First, the image processing unit 21 will be described. The image processing unit 21 receives and processes images captured by the multiple external cameras 11 and the internal camera 12. These captured images include images used to recognize the external environment, images used to generate a saliency map, and images used to detect the driver's line of sight. The image processing unit 21 performs distortion correction processing to correct distortion in the image, white balance adjustment processing to adjust the white balance of the image, and the like.

[0033] Next, the caution level detection unit 22 will be described. The caution level detection unit 22 recognizes environmental factors around the vehicle based on information output from the radar 13, the position sensor 14, and the external input unit 15. Specifically, it detects the road environment and obstacles, etc. The road environment factors mainly include the shape of intersections and T-junctions, the topography such as road gradient, the pavement condition, etc. The obstacle factors mainly include the presence or absence of obstacles, the type of obstacle (other vehicles, pedestrians, buildings, etc.), the relative relationship with the obstacle (separation distance, relative speed, positional relationship, etc.), etc. The caution level detection unit 22 recognizes environmental factors and detects whether the level of caution of the external environment in which the vehicle is traveling is high, that is, higher than a predetermined threshold value.

[0034] Next, the field of view image creation unit 23 will be described. The field of view image creation unit 23 creates a field of view image of the driver seated in the driver's seat based on input from the external camera 11 and the internal camera 12. current It creates a visual image that can be seen. The field of view image creation unit 23 calculates the driver's line of sight direction from the image of the driver's eyeball captured by the internal camera 12. For example, the field of view image creation unit 23 calculates the driver's line of sight direction by detecting a change in the pupil based on the state in which the driver looks into the lens of the internal camera 12. This field of view image creation unit 23 creates a line of sight image by combining an external environment image in front of the vehicle captured by the external camera 11 with vehicle body components that are in the field of view of the driver while the vehicle is traveling. The visual field image creating unit 23 calculates the driver's gaze point from the driver's line of sight direction and the line of sight image.

[0035] Next, the gaze behavior distribution generating unit 24 will be described. The gaze behavior distribution generation unit 24 generates gaze behavior distributions Ba, Bb, and Bc based on the most recent gaze behavior of the driver. As shown in Fig. 3, the gaze behavior distribution B is composed of three normal distributions: a normal distribution Ba related to saliency (attractiveness), a normal distribution Bb related to the amplitude of saccades (saccadic eye movements), and a normal distribution Bc related to the frequency of saccades. As shown in FIG. 1, the gaze behavior distribution generating unit 24 includes a saliency map generating unit 24a, an amplitude characteristic generating unit 24b, and a frequency characteristic generating unit 24c.

[0036] The saliency map creating unit 24a creates a normal distribution Ba consisting of a saliency index (AUC: Area Under Curve) and a risk potential (RP). This saliency map creation unit 24a calculates saliency for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, for parts other than the vehicle body components in the line-of-sight image created by the field-of-view image creation unit 23, and generates a saliency map for each feature.The final saliency map is created by adding together these generated saliency maps for each feature.

[0037] Figure 4 plots the time course of fixation saliency for a normal third-person subject. This graph fits the fixation change to a saliency map for a given environment and plots the height of fixation saliency for each saccade. On the other hand, Figure 5 is generated by randomly specifying coordinates (hereinafter referred to as random coordinates) from the same saliency map as the saliency map used in Figure 4, and calculating the saliency of the random coordinates each time a saccade occurs.

[0038] Next, a receiver operating characteristic (ROC) curve is calculated for the probability of exceeding the threshold at a random point and the probability of exceeding the threshold at a third party's gaze point. As shown in Figure 6, the horizontal axis represents the probability of exceeding the threshold at a random point (first probability), and the vertical axis represents the probability of exceeding the threshold at a third-party gaze point (second probability), representing the gaze probability relative to the random probability at the same threshold. When the curve is convex upward, as in curve C1, the probability that the third party's gaze point exceeds the threshold is higher than that of a random point, and the curve is influenced by the high saliency region.When the curve is convex downward, as in curve C2, the probability that the third party's gaze point exceeds the threshold is lower than that of a random point, and the curve is not influenced by the high saliency region.

[0039] The saliency index (AUC) is an index whose numerical value increases as the degree of attention to high saliency areas increases. As shown in Figure 7, the saliency index is the integral value of the lower right part of the ROC curve. By using the saliency index compared with random points, it is possible to reduce the dependency on the driving scene, such as the number and spread of high saliency areas.

[0040] The risk potential (RP) is an artificially set field that appropriately reflects the degree of danger in the driving environment of the vehicle (the sense of danger felt by the driver), and for example, an appropriate function is set that has a shape that is maximum at the center position of each surrounding vehicle with respect to the surrounding vehicles and spreads out around each surrounding vehicle. Methods for determining the risk potential may also use the inter-vehicle distance to the surrounding vehicles, TTC (Time To Collision), road alignment, distance to a line change point, etc.

[0041] Then, the time series distribution of the saliency index in a given environment is obtained (see FIG. 8). In addition, the time series distribution of the risk potential in the same environment as the time series distribution of the saliency index is obtained (see Figure 9). As shown in Figure 10, the time series distribution of the saliency index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined. The combined time series distribution of the saliency index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saliency index in each divided region (hereinafter referred to as a bin) is calculated.

[0042] As shown in Figure 11, if the number of divided areas is, for example, 10, the probability distribution of the saliency index in bin 1 to the probability distribution of the saliency index in bin 10 are all added together to calculate the final normal distribution Ba (the driver's most recent normal distribution Ba).

[0043] The amplitude characteristic generating unit 24b generates a normal distribution Bb consisting of amplitude, which is one of the saccade indices, and risk potential (RP). A saccade is a saccadic eye movement in which a driver involuntarily and reflexly moves their line of sight, and is an eye movement in which the line of sight moves from a fixation point where the line of sight remains stationary for a predetermined period of time to the next fixation point. 12, a period sandwiched between adjacent fixation periods is a saccade period. The amplitude ds of the saccade is the distance the line of sight moves during the saccade period.

[0044] A period (e.g., 0.1 seconds) during which the gaze movement speed is less than a predetermined speed threshold (e.g., 2 deg / s) and continues for a predetermined stagnation time is extracted as a gaze period, and gaze movements during the period sandwiched between adjacent gaze periods, in which the movement speed is equal to or greater than the speed threshold (e.g., 40 deg / s) and the movement distance is equal to or greater than the distance threshold (e.g., 3 deg), are extracted as saccade candidates.

[0045] 13, a regression curve L10 is derived using a plurality of saccade candidates by, for example, the least squares method. Next, a first reference curve L11 is derived by shifting the regression curve L10 in a direction in which the movement speed decreases, and a second reference curve L12 is derived by shifting the regression curve L10 in a direction in which the movement speed increases. Then, the area between the first reference curve L11 and the second reference curve L12 is defined as a saccade range R10, and saccade candidates included in this saccade range R10 are extracted as selected saccades.

[0046] The amplitude characteristic creation unit 24b calculates a time series distribution (not shown) of the amplitude of the saccade index in a predetermined environment, as in the case of the normal distribution Ba. Also, it calculates a time series distribution (not shown) of the risk potential in the same environment as the time series distribution of the saccade index. The time series distribution of the saccade index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined. The combined time series distribution of the saccade index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saccade index for each divided region is calculated. If the number of divided regions is, for example, 10, the probability distributions of the saccade index from bin 1 to bin 10 are all combined to calculate the normal distribution Bb for saccades.

[0047] The frequency characteristic creation unit 24c creates a normal distribution Bc consisting of a frequency, which is one of the saccade indices, and a risk potential (RP). The frequency, which is one of the saccade indices, is a value obtained by dividing the number of saccades in a predetermined time (e.g., 30 seconds) by the predetermined time. As with the normal distribution Ba, the frequency characteristic creation unit 24c calculates the normal distribution Bc for saccades using a time series distribution (not shown) relating to the frequency of saccade indicators in a predetermined environment and a time series distribution (not shown) of the risk potential.

[0048] Next, the correction unit 25 will be described. The correction unit 25 calculates the first corrected gaze distribution X by correcting the reference gaze distribution A using the degree of influence of the first disturbance factor caused by the environment and vehicle characteristics. The basic reference gaze distribution A is created in advance before the vehicle is shipped from the factory, specifically, by using the gaze behavior of a normal third party through a preliminary experiment, etc. Furthermore, a reference gaze distribution generating means for generating the reference gaze distribution A is provided in the testing site where the preliminary experiment is conducted. The data (reference line of sight distribution A, first correction coefficient α) created in advance is stored in the vehicle in advance.

[0049] Human gaze behavior has the characteristic of changing under the influence of N factors, mainly environmental factors, human factors, and vehicle factors among the disturbance factors. The accuracy of estimating the driver's state changes due to the influence of these N factors. The first disturbance factors are environmental factors and vehicle factors, which are less influenced by individual differences among the N factors. Examples of environmental factors are as follows: Roads (road type, shape, number of lanes, number) Road boundaries (road: type, relative position, TTC) Local transport (type: number, relative location, TTC) Pedestrians (type: number, relative position, TTC) Structures (blind spots, predicted collision distance) Weather (sunny, rainy) Lighting (day and night)

[0050] The vehicle factors are, for example, as follows: Vehicle behavior (vehicle speed, acceleration, deceleration, yaw rate) Vehicle characteristics (response to operation / road input) Window frame structure (shape / decoration, instrument panel depth) Eye point (height) HMI (HUD and other layouts) For the environmental factors and vehicle factors, one or more factors are set from the above items, and the following processing is performed for the set factors.

[0051] The correction unit 25 takes into consideration the levels of the environmental factors and vehicle factors and calculates the first correction coefficients α (αa, αb, αc) according to the environmental factors and vehicle factors using a known statistical method. After calculating the first correction coefficient α, the correction unit 25 obtains the first corrected gaze distributions Xa, Xb, and Xc by the following equation. X = α × A … (1) Since the first correction coefficient α can be pre-calculated by arbitrarily changing the environmental factors and vehicle factors, in this embodiment, the reference gaze distribution A, the first correction coefficient α, and the first corrected gaze distribution X are stored in the vehicle before it is shipped from the factory.

[0052] Next, the learning gaze distribution generating unit 26 will be described. The learning gaze distribution generation unit 26 generates the second corrected gaze distribution Y using the second correction coefficient β related to the human factors of the driver. As shown in Fig. 1, the learning gaze distribution generation unit 26 includes a learning information generation unit 26a. This learning information generation unit 26a generates the second correction coefficient β that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver. Among N factors, human factors have the characteristic of being highly influenced by individual differences.

[0053] Human factors are, for example: Physical condition (illness, fatigue, alertness, drunkenness) Psychological (pleasure / discomfort; anxiety / tension / excitement, activity, expectation) Experience (previous short-term, long-term) Driving skills (mental and physical functions, metacognition) Driving style (values, personality, preferences) Driving awareness (safety awareness, driving motivation, self-awareness) Intention (driving purpose, presence of other passengers) culture Age (young / old, values) sex The human factor is set to one or more factors from the above items, and the following processing is performed for the set factors.

[0054] The learning information generating unit 26a calculates the second correction coefficient β (βa, βb, βc) according to the human factor by taking into consideration the level of the human factor and using a known statistical method. After calculating the second correction coefficient β, the learning gaze distribution generating unit 26 obtains second corrected gaze distributions Ya, Yb, Yc by the following equation. Y = β × X … (2)

[0055] Next, the abnormality estimation unit 27 will be described. The abnormality estimation unit 27 but abnormality or not The abnormality estimation unit 27 includes a learning progress calculation unit 27a. The learning progress calculation unit 27a calculates the progress of learning about the individual characteristics of the driver, that is, the learning progress St. This learning progress level St is calculated using the following formula, where F1 is the number of frames of image data that are necessary and sufficient to identify the driver's personal characteristics, and F2 is the number of image frames related to the driver that were used to generate the second corrected gaze distribution Y. St = F2 / F1 …(3)

[0056] The abnormality estimation unit 27 but abnormality or not To determine the abnormality, the system has three determination functions: a first abnormality determination function that uses the deviation of the probability distribution, a second abnormality determination function that uses the saliency index (see Figure 7), and a third abnormality determination function that uses the degree of caution.

[0057] The first abnormality determination function determines the abnormal state of the driver when the evaluation value, which is the difference between the recent (e.g., within a 30 - second period) line - of - sight behavior distribution B of the driver and the second corrected line - of - sight distribution Y, is greater than or equal to the determination threshold. In this embodiment, as shown in FIG. 2, the abnormality estimation unit 26 calculates, using the Kullback - Leibler information amount as the evaluation value, the degree of divergence between the line - of - sight behavior distributions Ba, Bb, Bc and the second corrected line - of - sight distributions Ya, Yb, Yc respectively.

[0058] The evaluation values Da of the distributions Ba, Ya, the evaluation value Db of the distributions Bb, Yb, and the evaluation value Dc of the distributions Bc, Yc are obtained respectively, and the total evaluation value D is calculated by substituting them into the following formula. D = W×(Da + Dc)+(1 - W)×Db …(4) Here, W(0 < W < 1) is a weighting coefficient. The total evaluation value D is set such that when the learning progress degree St is small (the learning period is short), the weights of the evaluation values Da and Dc are larger than the weight of the evaluation value Db, and when the learning progress degree St is large (the learning period is long), the weights of the evaluation values Da and Dc are smaller than the weight of the evaluation value Db. When the total evaluation value D is greater than or equal to the determination threshold (e.g., 0.6), the state abnormality of the driver is determined.

[0059] The weighting coefficient W can be obtained by the following formula when P, Q, and R are coefficients. W = 1 / (1 + e -P×(St×Q-R) ) …(5) This weighting coefficient W is a monotonically increasing function that is point - symmetric like the sigmoid function, which is one of the activation functions, and the coefficients P, Q, and R can be set arbitrarily.

[0060] The second abnormality determination function estimates that the driver's attention is decreased when the saliency index (the integral value of the lower - right part of the ROC curve) is greater than or equal to a predetermined determination threshold. The third abnormality determination function estimates that the driver's attention is decreased when high attention is detected by the attention detection unit 22 and the amplitude ds of the saccade is significantly decreased, and also estimates that the driver's attention is decreased when low attention is detected by the attention detection unit 22 and the frequency of the saccade is significantly decreased.

[0061] An example of a driver's state determination process will be described with reference to the flowcharts of Figures 14 to 17. In the figures, Si (i=1, 2, . . . ) indicates each step.

[0062] An example of the advance learning process is shown in the flowchart of Fig. 14. The advance learning process is basically performed before the vehicle is shipped from the factory, for example, at a testing site. First, various information such as information from the information acquisition unit 1 and various experimental results is read (S1), and then the process proceeds to S2. In S2, the standard saliency index and standard saccade index (amplitude, frequency) for a normal third person are calculated, and then the process proceeds to S3.

[0063] In S3, it is determined whether sufficient data regarding standard saliency indices and standard saccade indices for normal third parties has been obtained. If the result of the determination in S3 is that sufficient data is secured, the reference gaze distributions Aa, Ab, and Ac are calculated (S4), and the process moves to S5. If the result of the determination in S3 is that sufficient data is not secured, the process returns to S1.

[0064] In S5, the first corrected gaze distribution X is obtained, and the first correction coefficient α is calculated using equation (1), and then the process proceeds to S6. Since the first corrected gaze distribution X corresponds to an index of a normal state, the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index correspond to the first corrected gaze distribution X. Therefore, the first correction coefficient α is calculated by substituting the reference gaze distribution A and the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index, which correspond to the first corrected gaze distribution X, into equation (1). In S6, the reference gaze distribution A and the first correction coefficient α are stored in the control unit 2, and then the process ends.

[0065] Next, an example of the driver's state determination process will be shown with reference to the flowchart of FIG. The state determination process is performed when the driver operates the vehicle. First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S11), and the process proceeds to S12. In S12, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S13.

[0066] In S13, it is determined whether a highly reliable second correction coefficient β is held. If the result of the determination in S13 is that the second correction coefficient β is held, the second corrected gaze distribution Y can be calculated, and the process proceeds to S14. In S14, the reference gaze distribution A and the first correction coefficient α are substituted into equation (1) to calculate the first corrected gaze distribution X, and then the first corrected gaze distribution X and the second correction coefficient β are substituted into equation (2) to determine the second corrected gaze distribution Y.

[0067] In S15, the difference between the gaze behavior distribution B and the second corrected gaze distribution Y is calculated, and the process proceeds to S16. The difference between the gaze behavior distribution B and the second corrected gaze distribution Y is calculated as a deviation (separation distance) using Kullback-Leibler divergence. Each deviation is represented by an evaluation value Da for distributions Ba and Ya, an evaluation value Db for distributions Bb and Yb, and an evaluation value Dc for distributions Bc and Yc.

[0068] In S16, the final total evaluation value D is calculated using the evaluation values ​​Da, Db, and Dc obtained in S15, and the process proceeds to S17. In S17, it is determined whether the total evaluation value D is equal to or greater than the determination threshold value. If the result of the judgment in S17 is that the total evaluation value D is equal to or greater than the judgment threshold, the gaze behavior distribution B representing the driver's current state is deviated from the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, so an abnormality judgment is made that the driver's state is abnormal (S18), and the process ends. If the result of the judgment in S17 is that the total evaluation value D is less than the judgment threshold, the gaze behavior distribution B representing the driver's current state is close to the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, so a normal judgment is made that the driver's state is normal (S19), and the process ends.

[0069] If the result of the determination in S13 is that the second correction coefficient β is not held at the time of the determination, the second corrected gaze distribution Y cannot be calculated, and the process proceeds to S20. In S20, since the second correction coefficient β is not held, the driver's condition is estimated using biometric indicators other than saliency and saccades related to gaze behavior, such as heart rate and blood pressure, and then the process proceeds to S21.

[0070] In S21, it is determined whether the driver's condition is normal or not using biological information indicators other than saliency and saccades related to gaze behavior. If the result of the determination in S21 is that the driver is in a normal state, the process proceeds to S22 to acquire the second correction coefficient β. In S22, the gaze behavior distribution B is accumulated and learned until a highly accurate second correction coefficient β can be acquired, and then the process ends. If the result of the determination in S21 is that the driver is not in a normal state, the second correction coefficient β cannot be acquired, and the process ends.

[0071] Next, an example of the total evaluation value calculation process (S16) will be shown with reference to the flowchart of FIG. The total evaluation value calculation process is performed when calculating the total evaluation value D from the gaze behavior distribution B and the second corrected gaze distribution Y. First, the number of image frames F1 necessary and sufficient to identify the driver's personal characteristics and the number of image frames F2 used to generate the second corrected gaze distribution Y are substituted into equation (3) to calculate the learning progress level St (S31), and the process proceeds to S32.

[0072] In S32, it is determined whether the learning progress level St is equal to or greater than a predetermined determination threshold value. If the result of the determination in S32 is that the learning progress level St is equal to or greater than a preset determination threshold, the process proceeds to S33 because image data necessary and sufficient to identify the driver's personal characteristics has been acquired and highly accurate learning has already been performed. If the result of the determination in S32 is that the learning progress level St is less than the preset determination threshold, the process returns to S31 because image data necessary and sufficient to identify the driver's personal characteristics has not been acquired and there is insufficient information on human factors related to the driver.

[0073] In S33, a weighting factor W is calculated using the learning progress level St and equation (5), and the process proceeds to S34. In S34, a total evaluation value D is calculated using the weighting factor W, evaluation values ​​Da, Db, and Dc, and equation (4), and the process ends.

[0074] Next, an example of the learning phase process (S22) will be shown with reference to the flowchart of FIG. The learning phase process is performed when learning the gaze behavior distribution B in order to acquire the second correction coefficient β. First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S41), and the process proceeds to S42. In S42, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S43.

[0075] In S43, it is determined whether the driver's condition is normal. If the result of the determination in S43 is that the driver's condition is normal, imaging data relating to gaze behavior distribution B is accumulated in order to learn gaze behavior distribution B, and the process proceeds to S44. If the result of the determination in S43 is that the driver's condition is not normal, data regarding the gaze behavior distribution B cannot be accumulated, and the process returns to S41.

[0076] In S44, it is determined whether or not a necessary and sufficient amount of data regarding the gaze behavior distribution B has been secured in order to calculate the second correction coefficient β. If the result of the determination in S44 is that the amount of data regarding gaze behavior distribution B has been secured, the process proceeds to S45. If the result of the determination in S44 is that the amount of data regarding gaze behavior distribution B has not been secured, the process returns to S41 to secure more data.

[0077] In S45, the second correction coefficient β is calculated, and after saving the second correction coefficient β (S46), the process ends. Since the second corrected gaze distribution Y corresponds to an index of the normal state, the gaze behavior distribution B in the normal state corresponds to the most recent second corrected gaze distribution Y. Therefore, the second correction coefficient β is calculated by substituting the first corrected gaze distribution X (reference gaze distribution A, first correction coefficient α) and the gaze behavior distribution B corresponding to the most recent second corrected gaze distribution Y into equation (2).

[0078] Next, the operation and effects of the driver state determination system S will be described. This driver state determination method includes a learning information generation step (S45) of generating a second correction coefficient β that correlates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver, and a driver learning gaze distribution generation step (S14) of generating a second corrected gaze distribution Y using the reference gaze distribution A and the second correction coefficient β, so that the second corrected gaze distribution Y can be generated using the second correction coefficient β that correlates with the gaze behavior of the driver.A learning progress calculation step (S31) of calculating a learning progress St based on the number of images C2 used to generate the second corrected gaze distribution Y, and when the difference between the gaze behavior distribution B and the second corrected gaze distribution Y is equal to or greater than a determination threshold, but abnormality That is Since the method includes a driver abnormality estimation step (S17) of determining whether or not a driver is abnormal, and increasing the weighting ratio of the manifest trait to the latent trait as the learning progress level St increases, when the driver's gaze learning period is insufficient, the weight of the latent trait, which is less affected by individual differences, can be made greater than the weight of the manifest trait, which is more affected by individual differences, and the determination can be made mainly based on the bottom-up function, which is a low-level processing of the brain. Also, when the driver's gaze learning period is sufficient, the weight of the latent trait, which is less affected by individual differences, can be made less than the weight of the manifest trait, which is more affected by individual differences, and the determination can be made mainly based on the top-down function, which is a high-level processing of the brain.

[0079] The driver abnormality estimation process (S17) determines the difference between the gaze behavior distribution B and the second corrected gaze distribution Y as the sum of the difference between the probability distributions Ba, Bc corresponding to the latent characteristics of the gaze behavior distribution B and the probability distributions Ya, Yc corresponding to the latent characteristics of the second corrected gaze distribution Y, and the difference between the probability distribution Bb corresponding to the manifest characteristics of the gaze behavior distribution B and the probability distribution Yb corresponding to the manifest characteristics of the second corrected gaze distribution Y, thereby making it possible to evaluate the driver's gaze behavior from both the bottom-up and top-down perspectives.

[0080] This driver state determination system S has a learning information generation unit 26a that generates a second correction coefficient β that associates the physical state of the vehicle, the characteristics of the driver, and the driver's gaze behavior, and a learning gaze distribution generation unit 26 that generates a second corrected gaze distribution Y using the reference gaze distribution A and the second correction coefficient β, so that the second corrected gaze distribution Y can be generated using the second correction coefficient β. a learning progress calculation unit (27a) that calculates a learning progress degree (St) based on the number of images (C2) used to generate the second corrected gaze distribution (Y); and a driver's gaze behavior distribution (B) that calculates a learning progress degree (St) based on the number of images (C2) used to generate the second corrected gaze distribution (Y). but abnormality That is Since the system has a driver abnormality estimation unit 27 that determines whether or not a driver is abnormal, and that increases the weighting ratio of the manifest trait to the latent trait as the learning progress level St increases, when the driver's gaze learning period is insufficient, the weight of the latent trait, which is less affected by individual differences, can be made greater than the weight of the manifest trait, which is more affected by individual differences, and the system can make a determination mainly based on the bottom-up function, which is a low-level processing of the brain. Also, when the driver's gaze learning period is sufficient, the weight of the latent trait, which is less affected by individual differences, can be made less than the weight of the manifest trait, which is more affected by individual differences, and the system can make a determination mainly based on the top-down function, which is a high-level processing of the brain.

[0081] Next, a modified example in which the above embodiment is partially modified will be described. 1) In the above embodiment, an example was described in which the reference gaze distribution A was corrected using the first correction coefficient α due to the environment and vehicle characteristics. However, the reference gaze distribution A may also be corrected using the first correction coefficient α due to only the environment, or the reference gaze distribution A may also be corrected using the first correction coefficient α due to only the vehicle characteristics.

[0082] 2) In the above embodiment, an example has been described in which the reference gaze distribution generating means is provided in the testing site where the preliminary experiment is conducted, but the reference gaze distribution generating means may also be provided in the control unit 2 of the vehicle.

[0083] 3) In the above embodiment, an example was described in which the learning progress level St was calculated using the number of image frames and equation (3), but it is sufficient to be able to detect at least the learning period, and the learning progress level St may also be calculated using the travel distance or driving time as a parameter.

[0084] 4) In addition, a person skilled in the art may implement the present invention in a form in which various modifications are added to the above-described embodiment without departing from the spirit of the present invention, and the present invention also includes such modifications. [Explanation of symbols]

[0085] 11 External Camera 12 Internal Camera 13 Radar 14 Position Sensor 15 External input section 16 Vehicle condition sensor 17 Driving operation sensor 18 Biometric information sensor 23 Visual field image creation unit 24 Gaze behavior distribution generation unit 26 Learning gaze distribution generation unit 26a Learning information generation unit 27 Anomaly estimation part 27a Learning progress calculation unit S Driver status determination system

Claims

1. A driver state determination method for determining a driver state of a vehicle, comprising: a surrounding information acquisition step of acquiring information about a surrounding situation of the vehicle; a vehicle state acquisition step of acquiring a physical state of the vehicle; a driver characteristic acquisition step of acquiring characteristics of the driver; a visual field image creation step of creating a visual field image that appears in the driver's field of view; a reference gaze distribution generating step of generating in advance a reference gaze distribution based on a standard visual field image for latent characteristics reflecting low-level processing of the brain and explicit characteristics reflecting high-level processing of the brain; a gaze behavior distribution generating step of generating a gaze behavior distribution based on the gaze behavior of the driver for a latent characteristic reflecting a low-level processing of the brain and an explicit characteristic reflecting a high-level processing of the brain; a learning information generating step of generating learning information that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver; a driver learning gaze distribution generating step of generating a learning gaze distribution using the reference gaze distribution and the learning information; a learning progress calculation step of calculating a learning progress level based on the number of images used to generate the learning gaze distribution; a driver abnormality inference step of determining that the driver is abnormal when a difference between the gaze behavior distribution and the learned gaze distribution is equal to or greater than a judgment threshold, and increasing a weighting ratio of the apparent characteristic to the latent characteristic as the learning progress degree increases; A driver state determination method comprising:

2. The driver's state determination method of claim 1, characterized in that the driver abnormality estimation step determines the difference between the gaze behavior distribution and the learned gaze distribution by adding the difference between the probability distribution corresponding to the latent characteristics of the gaze behavior distribution and the probability distribution corresponding to the latent characteristics of the learned gaze distribution and the difference between the probability distribution corresponding to the manifest characteristics of the gaze behavior distribution and the probability distribution corresponding to the manifest characteristics of the learned gaze distribution.

3. A driver state determination system for determining the state of a vehicle driver, a surrounding information acquisition means for acquiring information about the surrounding conditions of the vehicle; a vehicle state acquisition means for acquiring a physical state of the vehicle; a driver characteristic acquisition means for acquiring the characteristics of the driver; a field of view image creating means for creating a field of view image that appears in the driver's field of view; a reference gaze distribution generating means for generating a reference gaze distribution based on a standard visual field image in advance for latent characteristics reflecting low-level processing of the brain and manifest characteristics reflecting high-level processing of the brain; a gaze behavior distribution generating means for generating a gaze behavior distribution based on the gaze behavior of the driver for latent characteristics reflecting low-level processing of the brain and explicit characteristics reflecting high-level processing of the brain; a learning information generating means for generating learning information that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver; a driver learning gaze distribution generating means for generating a learning gaze distribution using the reference gaze distribution and the learning information; a learning progress calculation means for calculating a learning progress degree based on the number of images used in the driver learning gaze distribution generation means; a driver abnormality estimation means for judging that the driver is abnormal when a difference between the gaze behavior distribution and the learned gaze distribution is equal to or greater than a judgment threshold, and for increasing a weighting ratio of the apparent characteristic to the latent characteristic as the learning progress degree increases; A driver state determination system comprising:

Images