Driver state determination method and determination system

Abstract

To provide a driver state determination method and its determination system capable of securing state determination accuracy while shortening a learning period.SOLUTION: A periphery information acquiring unit, a vehicle state acquiring unit, a visual field image creating unit 23 that creates a visual field image appearing in a field of view of a driver, a reference line-of-sight distribution generating unit 2 that generates in advance a reference line-of-sight distribution A based on a standard visual field image, an abnormality estimating unit 28 that compares a line-of-sight behavior distribution B with a reference line-of-sight distribution A corrected by using a second correction coefficient β that is learning information, a reliability setting unit 27a that sets a reliability N as learning data of the visual field image for generating the second correction coefficient β, and an abnormality determining unit 28 that determines an abnormality of the driver by comparing the reliability N as the learning data with a determination threshold value or more. And a learning restriction part 27 for executing learning based on the visual field image, and for restricting learning based on the visual field image when the reliability N as learning data is less than a determination threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driver state determination method and a determination system for determining the state of a driver of a vehicle. [Background technology]

[0002] Recently, the development of automated driving systems has been promoted nationwide. The applicant believes that there are currently two main directions for automated driving systems.

[0003] The first direction is a system in which the vehicle takes the lead in transporting passengers to their destination without the need for driver operation, known as fully automated driving of automobiles. The second direction is an autonomous driving system that is based on the premise that humans will be driving, with the aim of "providing an environment in which driving a car is enjoyable."

[0004] In the second direction of automated driving systems, it is expected that the vehicle will automatically take over driving in place of the occupants in the event that, for example, the driver develops an illness or other condition that makes normal driving difficult. For this reason, it is extremely important from the perspective of improving the driver's survival rate and ensuring the safety of those around them to detect as early and accurately as possible any abnormalities in the driver, particularly any functional impairment or illness that the driver has.

[0005] As a method for determining the driver's state, for example, the visual target determination device of Patent Document 1 discloses a configuration that includes a gaze detection unit that detects the driver's gaze behavior such as the direction of gaze, a vehicle information acquisition unit that acquires vehicle information related to the driving state, an image acquisition unit that acquires captured images of the vehicle's surroundings, a gaze area extraction unit that extracts a gaze area based on the driver's gaze behavior within the captured images, a candidate detection unit that detects objects with saliency in a top-down saliency map as candidates for the visual target, and determines the visual target based on the results of the gaze area extraction and the results of the detection of candidate visual targets. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-071528 Summary of the Invention [Problem to be solved by the invention]

[0007] The top-down saliency map in Patent Document 1 is a saliency map that is uniquely determined from gaze behavior, taking into account individual differences such as the driver's viewing order, physique, and habits, as well as ambiguity in the scene from which the captured image was obtained, and fluctuations in gaze direction detected by the gaze detection unit. Normally, the individual characteristics (individual differences) of the driver affect the driver's gaze behavior and are reflected as learning information in the state determination control of the abnormality determination system. It is believed that the more data collected, the more reliable such learning information becomes, and the accuracy of abnormality determination also increases in proportion to the amount of collected data.

[0008] As a result of extensive research, the inventors have found that in learning information related to human factors, there are environmental conditions and vehicle conditions in which personal characteristics are likely to appear, and that in conditions in which these personal characteristics are likely to appear, the personal characteristics are emphasized, resulting in a relative decrease in the reliability of the learning information. Therefore, even if information that overemphasizes individual characteristics is collected and applied to learning control, the accuracy of judgment cannot be improved as much as expected, despite the consumption of data collection time. That is, it is not easy to shorten the learning period while ensuring the accuracy of state determination.

[0009] An object of the present invention is to provide a driver state determination method and a determination system therefor that can ensure state determination accuracy while shortening the learning period. [Means for solving the problem]

[0010] The invention of claim 1 is a driver state determination method for determining the state of a vehicle driver, comprising a peripheral information acquisition step for acquiring information about the vehicle's peripheral conditions, a vehicle state acquisition step for acquiring the vehicle's physical conditions including speed, acceleration, and yaw rate, a field of view image creation step for creating a field of view image that appears in the driver's field of view, a reference gaze distribution generation step for pre-generating a reference gaze distribution based on a standard field of view image, a gaze behavior distribution generation step for generating a gaze behavior distribution based on the driver's gaze behavior, a learning information generation step for generating learning information that associates the vehicle's peripheral conditions, the vehicle's physical conditions, and the driver's gaze behavior, and The system is characterized by having a driver abnormality estimation process for comparing the gaze distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal or not; a reliability setting process for setting the reliability of a field of view image included in the driver's field of view and used to generate the learning information as learning data based on the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle state obtained from the physical state of the vehicle; and a learning restriction process for executing learning based on the field of view image when the reliability as learning data is equal to or greater than a judgment threshold, and restricting learning based on the field of view image when the reliability as learning data is less than the judgment threshold.

[0011] This driver condition determination method includes a driver abnormality estimation step for comparing the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal, and therefore it is possible to estimate whether the driver is in an abnormal state based on the gaze behavior. Since the system has a reliability setting process for setting the reliability of the field of view image included in the driver's field of view and used to generate the learning information as learning data based on the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle conditions obtained from the physical conditions of the vehicle, it is possible to set the reliability of the learning data representing the driver's personal characteristics for the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle conditions obtained from the physical conditions of the vehicle. When the reliability of the learning data is equal to or greater than the judgment threshold, learning based on the field of view image is executed, and when the reliability of the learning data is less than the judgment threshold, a learning restriction process is included which restricts learning based on the field of view image. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

[0012] The invention of claim 2 is a driver state determination system for determining the state of a vehicle driver, comprising: a surrounding information acquisition means for acquiring information on the surrounding conditions of the vehicle; a vehicle state acquisition means for acquiring the physical conditions of the vehicle including the speed, acceleration, and yaw rate; a field of view image creation means for creating a field of view image that appears in the field of view of the driver; a reference gaze distribution generation means for generating in advance a reference gaze distribution based on a standard field of view image; a gaze behavior distribution generation means for generating a gaze behavior distribution based on the gaze behavior of the driver; learning information generation means for generating learning information that associates the surrounding conditions of the vehicle, the physical conditions of the vehicle, and the gaze behavior of the driver; The system is characterized by having a driver abnormality estimation means for comparing the dynamic distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal or not; a reliability setting means for setting the reliability of a field of view image included in the driver's field of view and used to generate the learning information as learning data based on the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle state obtained from the physical state of the vehicle; and a learning restriction means for executing learning based on the field of view image when the reliability as learning data is equal to or greater than a judgment threshold, and restricting learning based on the field of view image when the reliability as learning data is less than the judgment threshold.

[0013] This driver condition determination system has a driver abnormality estimation means that compares the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is in an abnormal state, and therefore can estimate whether the driver is in an abnormal state based on the gaze behavior. Since the system has a reliability setting means for setting the reliability of the field of view image included in the driver's field of view and used to generate the learning information as learning data based on the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle state obtained from the physical state of the vehicle, it is possible to set the reliability of the learning data representing the driver's personal characteristics for the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle state obtained from the physical state of the vehicle. The system has a learning restriction means that executes learning based on the field of view image when the reliability of the learning data is equal to or greater than the judgment threshold, and restricts learning based on the field of view image when the reliability of the learning data is less than the judgment threshold.This makes it possible to restrict learning in situations where personal characteristics become more pronounced, thereby ensuring the accuracy of state judgment while shortening the learning period. [Effects of the Invention]

[0014] According to the driver state determination method and determination system of the present invention, by limiting learning in accordance with the tendency of personal characteristics for each scene, it is possible to shorten the learning period while ensuring state determination accuracy. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram illustrating a configuration of a driver state determination system according to a first embodiment. [Figure 2] FIG. 4 is a diagram illustrating processing by a control unit. [Figure 3] The gaze behavior distributions show a normal distribution corresponding to saliency, a normal distribution corresponding to saccade amplitude, and a normal distribution corresponding to saccade frequency. [Figure 4] FIG. 10 is a diagram showing an example of a time change in saliency related to a driver's gaze point. [Figure 5] FIG. 4 is a diagram showing an example of time-dependent changes in saliency related to random points whose coordinates are randomly specified in the same environment as FIG. 3. [Figure 6] 10 is a graph comparing the probability of exceeding a threshold at a random point with the probability of exceeding a threshold at a driver's gaze point. [Figure 7] FIG. 1 is an explanatory diagram of a saliency index. [Figure 8] FIG. 10 is a diagram showing the time series distribution of saliency indices. [Figure 9] FIG. 10 is a diagram showing a time series distribution of risk potentials. [Figure 10] FIG. 10 is a diagram showing a combined time series distribution of a saliency index and a risk potential. [Figure 11] FIG. 10 is a diagram showing the probability distribution of saliency indices for each divided region of the risk potential.

[0016] [Figure 12] 10 is a graph for explaining extraction of a saccade. [Figure 13] 10 is a graph for explaining a noise removal process of a saccade. [Figure 14] 10 is a reliability setting table. [Figure 15] 10 is a flowchart of a pre-learning process. [Figure 16] 10 is a flowchart of a state determination process. [Figure 17] 10 is a flowchart of a total evaluation value calculation process. [Figure 18] 10 is a flowchart of a learning phase process. [Figure 19] 10 is a flowchart of a reliability determination process. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following description exemplifies the application of the present invention to a driver state determination system for a vehicle, and does not limit the present invention, its applications, or its uses. The following description also includes a description of a driver state determination method. [Example]

[0018] A first embodiment of the present invention will be described below with reference to FIGS. This driver state determination system S is a system that estimates the state of a driver by detecting signs of an abnormal state of the driver based on the gaze behavior of the driver operating the vehicle. The vehicle is, for example, a right-hand drive four-wheeled automatic vehicle that can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels according to the driver's operation (for example, accelerator operation, etc.). Assisted driving is driving in which the vehicle assists the driver's operation. Automatic driving is driving in which the vehicle travels without the driver's operation.

[0019] 1, the driver state determination system S controls the behavior of the vehicle in manual driving and assisted driving. Specifically, the driver state determination system S controls the operation of the vehicle (particularly the driving operation) by controlling an actuator 3 provided in the vehicle. The driver condition determination system S mainly comprises an information acquisition unit 1, a control unit 2, an actuator 3, and a notification unit 4. When an abnormality in the driver is determined, the actuator 3 causes the vehicle to retreat to a predetermined safety area, and the notification unit 4 notifies the management center of the abnormality in the driver.

[0020] First, the information acquisition unit 1 will be described. The information acquisition unit 1 acquires various types of information used for vehicle control, mainly for driving control, various types of information related to the vehicle surroundings, and various types of information used for determining the driver's state. As shown in FIG. 1, the information acquisition unit 1 includes a plurality of external cameras 11, an internal camera 12, a plurality of radars 13, a position sensor 14, an external input unit 15, a vehicle state sensor 16, a driving operation sensor 17, a biometric information sensor 18, and the like. The external camera 11, the multiple radars 13, the position sensor 14, and the external input unit 15 correspond to the surrounding information acquisition means, the internal camera 12 and the biometric information sensor 18 correspond to the driver characteristic acquisition means, and the vehicle state sensor 16 and the driving operation sensor 17 correspond to the vehicle state acquisition means.

[0021] The multiple external cameras 11 are provided to cover the entire area around the vehicle and have the same configuration. Each external camera 11 captures an image of a part of the environment surrounding the vehicle (external environment around the vehicle) to obtain image data showing a part of the external environment of the vehicle. The image data acquired by the external camera 11 is transmitted to the control unit 2. The external camera 11 is a monocular camera with a wide-angle lens, and is configured using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor).

[0022] The interior camera 12 is provided inside the vehicle. This internal camera 12 captures an image of a predetermined area including the driver's eyeballs to obtain image data including the driver's eyes. The internal camera 12 is disposed in front of the driver, and its imaging range is set so that the driver's eyeballs are within the imaging range. The vehicle has a drowsiness determination device (not shown) for determining the driver's drowsiness, and the internal camera 12 also serves as the imaging means of the drowsiness determination device. The image data obtained by the internal camera 12 is transmitted to the control unit 2.

[0023] The multiple radars 13 are provided to cover the entire area around the vehicle and have similar configurations. The radars 13 detect parts of the external environment of the vehicle (e.g., obstacles such as other vehicles or buildings) in order to determine the direction, relative speed, and distance to those parts. The radar 13 detects a part of the external environment of the vehicle by transmitting radio waves toward the part of the external environment and receiving waves reflected from the part of the external environment. The detection result of the radar 13 is transmitted to the control unit 2.

[0024] The position sensor 14 detects the position (for example, latitude and longitude) of the vehicle. The sensor 14 receives GPS information from a global positioning system and detects the position of the vehicle based on the GPS information. The vehicle position information obtained by the position sensor 14 is transmitted to the control unit 2.

[0025] The external input unit 15 inputs information via an external network (for example, the Internet) provided outside the vehicle. The external input unit 15 receives communication information (vehicle-to-vehicle communication information) from other vehicles located around the vehicle, car navigation data from a navigation system, traffic information, high-precision map information, etc. The vehicle position information obtained by the external input unit 15 is transmitted to the control unit 2.

[0026] The vehicle state sensor 16 detects the vehicle state (for example, speed, acceleration, yaw rate, etc.). The vehicle state sensor 16 includes a vehicle speed sensor that detects the traveling speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, etc. The vehicle state information obtained by the vehicle state sensor 16 is transmitted to the control unit 2.

[0027] The driving operation sensor 17 detects driving operations applied to the vehicle, and includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of accelerator pedal operation. The steering angle sensor detects the amount of steering wheel operation. The brake oil pressure sensor detects the amount of brake pedal operation. Vehicle driving operation information obtained by driving operation sensor 17 is sent to control unit 2.

[0028] The biological information sensor 18 detects biological information of the driver (for example, sweating, heart rate, etc.). The information obtained by the biological information sensor 18 is transmitted to the control unit 2.

[0029] Next, the control unit 2 will be described. As shown in Figure 2, the control unit 2 determines the driver's abnormal state based on an evaluation value (difference) which is the distance between the gaze behavior distribution Ba, Bb, Bc based on the driver's most recent gaze behavior and the second corrected gaze distribution Ya, Yb, Yc which reflects external disturbance factors. The second corrected gaze distributions Ya, Yb, Yc are obtained by correcting the reference gaze distributions Aa, Ab, Ac based on a standard field of view image using a first correction coefficient α (described later) to obtain the first corrected gaze distributions Xa, Xb, Xc, and then correcting the first corrected gaze distributions Xa, Xb, Xc using a second correction coefficient β (learning information) (described later).

[0030] The distributions Aa, Ba, Xa, and Ya are normal distributions for saliency (attractiveness), the distributions Ab, Bb, Xb, and Yb are normal distributions for saccade (saccadic eye movement) amplitude, and the distributions Ac, Bc, Xc, and Yc are normal distributions for saccade frequency. Saliency and saccade frequency are indicators of the brain's low-level processing, or the driver's latent characteristics, while saccade amplitude is an indicator of the brain's high-level processing, or the driver's explicit characteristics. Unless otherwise specified, the reference gaze distributions Aa, Ab, Ac will be collectively referred to as the reference gaze distribution A, the gaze behavior distributions Ba, Bb, Bc as the gaze behavior distribution B, the first corrected gaze distributions Xa, Xb, Xc as the first corrected gaze distribution X, and the second corrected gaze distribution Ya, Yb, Yc as the second corrected gaze distribution Y.

[0031] As shown in FIG. 1, the control unit 2 mainly includes an image processing unit 21, an attention level detection unit 22, a field of view image creation unit 23 (field of view image creation means), a gaze behavior distribution generation unit 24 (gaze behavior distribution generation means), a correction unit 25, a learning gaze distribution generation unit 26, a learning restriction unit 27 (learning restriction means), and an abnormality estimation unit 28 (driver abnormality estimation means).

[0032] First, the image processing unit 21 will be described. The image processing unit 21 receives and processes images captured by the multiple external cameras 11 and the internal camera 12. These captured images include images used to recognize the external environment, images used to generate a saliency map, and images used to detect the driver's line of sight. The image processing unit 21 performs distortion correction processing to correct distortion in the image, white balance adjustment processing to adjust the white balance of the image, and the like.

[0033] Next, the caution level detection unit 22 will be described. The caution level detection unit 22 recognizes environmental factors around the vehicle based on information output from the radar 13, the position sensor 14, and the external input unit 15. Specifically, it detects the road environment and obstacles, etc. The road environment factors mainly include the shape of intersections and T-junctions, the topography such as road gradient, the pavement condition, etc. The obstacle factors mainly include the presence or absence of obstacles, the type of obstacle (other vehicles, pedestrians, buildings, etc.), the relative relationship with the obstacle (separation distance, relative speed, positional relationship, etc.), etc. The caution level detection unit 22 recognizes environmental factors and detects whether the level of caution of the external environment in which the vehicle is traveling is high, that is, higher than a predetermined threshold value.

[0034] Next, the field of view image creation unit 23 will be described. The field of view image creating unit 23 creates a field of view image that appears in the field of view of the driver seated in the driver's seat, based on input from the external camera 11 and the internal camera 12. The field of view image creation unit 23 calculates the driver's line of sight direction from the image of the driver's eyeball captured by the internal camera 12. For example, the field of view image creation unit 23 calculates the driver's line of sight direction by detecting a change in the pupil based on the state in which the driver looks into the lens of the internal camera 12. This field of view image creation unit 23 creates a line of sight image by combining an external environment image in front of the vehicle captured by the external camera 11 with vehicle body components that are in the field of view of the driver while the vehicle is traveling. The visual field image creating unit 23 calculates the driver's gaze point from the driver's line of sight direction and the line of sight image.

[0035] Next, the gaze behavior distribution generating unit 24 will be described. The gaze behavior distribution generation unit 24 generates gaze behavior distributions Ba, Bb, and Bc based on the most recent gaze behavior of the driver. As shown in Fig. 3, the gaze behavior distribution B is composed of three normal distributions: a normal distribution Ba related to saliency (attractiveness), a normal distribution Bb related to the amplitude of saccades (saccadic eye movements), and a normal distribution Bc related to the frequency of saccades. As shown in FIG. 1, the gaze behavior distribution generating unit 24 includes a saliency map generating unit 24a, an amplitude characteristic generating unit 24b, and a frequency characteristic generating unit 24c.

[0036] The saliency map creating unit 24a creates a normal distribution Ba consisting of a saliency index (AUC: Area Under Curve) and a risk potential (RP). This saliency map creation unit 24a calculates saliency for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, for parts other than the vehicle body components in the line-of-sight image created by the field-of-view image creation unit 23, and generates a saliency map for each feature.The final saliency map is created by adding together these generated saliency maps for each feature.

[0037] Figure 4 plots the time course of fixation saliency for a normal third-person subject. This graph plots the height of fixation saliency for each saccade, applying the fixation change to a saliency map for a given environment. On the other hand, Figure 5 is generated by randomly specifying coordinates (hereinafter referred to as random coordinates) from the same saliency map as the one used in Figure 4, and calculating the saliency of the random coordinates each time a saccade occurs.

[0038] Next, a receiver operating characteristic (ROC) curve is calculated for the probability of exceeding the threshold at a random point and the probability of exceeding the threshold at a third party's gaze point. As shown in Figure 6, the horizontal axis represents the probability of exceeding the threshold at a random point (first probability), and the vertical axis represents the probability of exceeding the threshold at a third-party gaze point (second probability), representing the gaze probability relative to the random probability at the same threshold. When the curve is convex upward, as in curve C1, the probability that the third party's gaze point exceeds the threshold is higher than that of a random point, and the curve is influenced by the high saliency region.When the curve is convex downward, as in curve C2, the probability that the third party's gaze point exceeds the threshold is lower than that of a random point, and the curve is not influenced by the high saliency region.

[0039] The saliency index (AUC) is an index whose numerical value increases as the degree of attention to high saliency areas increases. As shown in Figure 7, the saliency index is the integral value of the lower right part of the ROC curve. By using the saliency index compared with random points, it is possible to reduce the dependency on the driving scene, such as the number and spread of high saliency areas.

[0040] The risk potential (RP) is an artificially set field that appropriately reflects the degree of danger in the driving environment of the vehicle (the sense of danger felt by the driver), and for example, an appropriate function is set that has a shape that is maximum at the center position of each surrounding vehicle with respect to the surrounding vehicles and spreads out around each surrounding vehicle. Methods for determining the risk potential may also use the inter-vehicle distance to the surrounding vehicles, TTC (Time To Collision), road alignment, distance to a line change point, etc.

[0041] Then, the time series distribution of the saliency index in a given environment is obtained (see FIG. 8). In addition, the time series distribution of the risk potential in the same environment as the time series distribution of the saliency index is obtained (see Figure 9). As shown in Figure 10, the time series distribution of the saliency index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined. The combined time series distribution of the saliency index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saliency index in each divided region (hereinafter referred to as a bin) is calculated.

[0042] As shown in Figure 11, if the number of divided areas is, for example, 10, the probability distribution of the saliency index in bin 1 to the probability distribution of the saliency index in bin 10 are all added together to calculate the final normal distribution Ba (the driver's most recent normal distribution Ba).

[0043] The amplitude characteristic generating unit 24b generates a normal distribution Bb consisting of amplitude, which is one of the saccade indices, and risk potential (RP). A saccade is a saccadic eye movement in which a driver involuntarily and reflexly moves their line of sight, and is an eye movement in which the line of sight moves from a fixation point where the line of sight remains stationary for a predetermined period of time to the next fixation point. 12, a period sandwiched between adjacent fixation periods is a saccade period. The amplitude ds of the saccade is the distance the line of sight moves during the saccade period.

[0044] A period (e.g., 0.1 seconds) during which the gaze movement speed is less than a predetermined speed threshold (e.g., 2 deg / s) and continues for a predetermined stagnation time is extracted as a gaze period, and gaze movements during the period sandwiched between adjacent gaze periods, in which the movement speed is equal to or greater than the speed threshold (e.g., 40 deg / s) and the movement distance is equal to or greater than the distance threshold (e.g., 3 deg), are extracted as saccade candidates.

[0045] 13, a regression curve L10 is derived using a plurality of saccade candidates by, for example, the least squares method. Next, a first reference curve L11 is derived by shifting the regression curve L10 in a direction in which the movement speed decreases, and a second reference curve L12 is derived by shifting the regression curve L10 in a direction in which the movement speed increases. Then, the area between the first reference curve L11 and the second reference curve L12 is defined as a saccade range R10, and saccade candidates included in this saccade range R10 are extracted as selected saccades.

[0046] The amplitude characteristic creation unit 24b calculates a time series distribution (not shown) of the amplitude of the saccade index in a predetermined environment, as in the case of the normal distribution Ba. Also, it calculates a time series distribution (not shown) of the risk potential in the same environment as the time series distribution of the saccade index. The time series distribution of the saccade index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined. The combined time series distribution of the saccade index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saccade index for each divided region is calculated. If the number of divided regions is, for example, 10, the probability distributions of the saccade index from bin 1 to bin 10 are all combined to calculate the normal distribution Bb for saccades.

[0047] The frequency characteristic creation unit 24c creates a normal distribution Bc consisting of a frequency, which is one of the saccade indices, and a risk potential (RP). The frequency, which is one of the saccade indices, is a value obtained by dividing the number of saccades in a predetermined time (e.g., 30 seconds) by the predetermined time. As with the normal distribution Ba, the frequency characteristic creation unit 24c calculates the normal distribution Bc for saccades using a time series distribution (not shown) relating to the frequency of saccade indicators in a predetermined environment and a time series distribution (not shown) of the risk potential.

[0048] Next, the correction unit 25 will be described. The correction unit 25 calculates the first corrected gaze distribution X by correcting the reference gaze distribution A using the degree of influence of the first disturbance factor caused by the environment and vehicle characteristics. The basic reference gaze distribution A is created in advance before the vehicle is shipped from the factory, specifically, by using the gaze behavior of a normal third party through a preliminary experiment, etc. Furthermore, a reference gaze distribution generating means for generating the reference gaze distribution A is provided in the testing site where the preliminary experiment is conducted. The data (reference line of sight distribution A, first correction coefficient α) created in advance is stored in the vehicle in advance.

[0049] Human gaze behavior has the characteristic of changing under the influence of N factors, mainly environmental factors, vehicle factors, and human factors, among the disturbance factors. The accuracy of estimating the driver's state changes due to the influence of these N factors. The first disturbance factors are environmental factors and vehicle factors, which are less influenced by individual differences among the N factors. Examples of environmental factors are as follows: Roads (road type, shape, number of lanes, number) Road boundaries (road: type, relative position, TTC) Local transport (type: number, relative location, TTC) Pedestrians (type: number, relative position, TTC) Structures (blind spots, predicted collision distance) Weather (sunny, rainy) Lighting (day and night) In this embodiment, the number of pedestrians, the degree of congestion of other vehicles, etc. are considered as one of the environmental factors that pose a risk.

[0050] The vehicle factors are, for example, as follows: Vehicle behavior (vehicle speed, acceleration, deceleration, yaw rate) Vehicle characteristics (response to operation / road input) Window frame structure (shape / decoration, instrument panel depth) Eye point (height) HMI (HUD and other layouts) For the environmental factors and vehicle factors, one or more factors are set from the above items, and the following processing is performed for the set factors.

[0051] The correction unit 25 takes into consideration the levels of the environmental factors and vehicle factors and calculates the first correction coefficients α (αa, αb, αc) according to the environmental factors and vehicle factors using a known statistical method. After calculating the first correction coefficient α, the correction unit 25 obtains the first corrected gaze distributions Xa, Xb, and Xc by the following equation. X = α × A … (1) Since the first correction coefficient α can be pre-calculated by arbitrarily changing the environmental factors and vehicle factors, in this embodiment, the reference gaze distribution A, the first correction coefficient α, and the first corrected gaze distribution X are stored in the vehicle before it is shipped from the factory.

[0052] Next, the learning gaze distribution generating unit 26 will be described. The learning gaze distribution generation unit 26 generates the second corrected gaze distribution Y using the second correction coefficient β related to the human factors of the driver. As shown in Fig. 1, the learning gaze distribution generation unit 26 includes a learning information generation unit 26a. This learning information generation unit 26a generates the second correction coefficient β that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver based on the field of view image. Among N factors, human factors have the characteristic of being highly influenced by individual differences.

[0053] Human factors are, for example: Physical condition (illness, fatigue, alertness, drunkenness) Psychological (pleasure / discomfort; anxiety / tension / excitement, activity, expectation) Experience (previous short-term, long-term) Driving skills (mental and physical functions, metacognition) Driving style (values, personality, preferences) Driving awareness (safety awareness, driving motivation, self-awareness) Intention (driving purpose, presence of other passengers) culture Age (young / old, values) sex The human factor is set to one or more factors from the above items, and the following processing is performed for the set factors.

[0054] The learning information generating unit 26a calculates the second correction coefficient β (βa, βb, βc) according to the human factor by taking into consideration the level of the human factor and using a known statistical method. After calculating the second correction coefficient β, the learning gaze distribution generating unit 26 obtains second corrected gaze distributions Ya, Yb, Yc by the following equation. Y = β × X … (2)

[0055] Next, the learning limiting unit 27 will be described. The learning restriction unit 27 is configured to perform learning based on the field of view image when the reliability N as learning data is equal to or greater than a predetermined judgment threshold, and to restrict learning based on the field of view image when the reliability N as learning data is less than the judgment threshold. The learning limiting unit 27 has a reliability setting unit 27a (reliability setting means).

[0056] The reliability setting unit 27a is configured to set the reliability N of the field of view image included in the driver's field of view as learning data using the first reliability index Ne and the second reliability index Nc. The reliability setting unit 27a calculates a first reliability index Ne as learning data of the field of view image based on a first disturbance factor related to the surrounding conditions of the vehicle acquired by the surrounding information acquisition means. Specifically, when the risk is Ne1, the road type is Ne2, the number of lanes is Ne3, and the coefficients are Ae1 to Ae3, the first reliability index Ne is calculated by the following formula. Ne=Ae1×Ne1+Ae2×Ne2+Ae3×Ne3 …(3)

[0057] Furthermore, the reliability setting unit 27a calculates a second reliability index Nc as learning data of the field of view image based on a second disturbance factor related to the physical state of the vehicle acquired by the vehicle state acquisition means. Specifically, when the vehicle speed is Nc1, the acceleration is Nc2, the yaw rate variation is Nc3, and the coefficients are Ac1 to Ac3, the second reliability index Nc is calculated by the following formula. Nc=Ac1×Nc1+Ac2×Nc2+Ac3×Nc3 …(4)

[0058] As shown in FIG. 14, the reliability setting unit 27a has a reliability setting table. In the reliability setting table, the first reliability index Ne and the second reliability index Nc are each classified into levels (for example, large, medium, and small). The first reliability index Ne is set so that the value of the medium level is larger than the value of the large and small level, because the personal characteristic of the medium level is smaller than the personal characteristic of the large and small level. The second reliability index Nc is set so that the value of the medium level is larger than the value of the large and small level, because the personal characteristic of the medium level is smaller than the personal characteristic of the large and small level.

[0059] The reliability setting unit 27a sets the reliability N using the level of the first reliability index Ne, the level of the second reliability index Nc, and the reliability setting table. When the reliability N is equal to or greater than a predetermined judgment threshold (e.g., 0.50), the reliability N of the learning data related to the human factors is high, and the learning limiting unit 27 executes learning. When the reliability N is less than the judgment threshold, the reliability N of the learning data related to the human factors is low, and the learning limiting unit 27 limits learning.

[0060] Next, the abnormality estimation unit 28 will be described. The abnormality estimation unit 28 is configured to be able to determine whether the driver is abnormal or not. The abnormality estimation unit 28 includes a learning progress calculation unit 28a. The learning progress calculation unit 28a calculates the progress of learning about the individual characteristics of the driver, that is, the learning progress level St. This learning progress level St is calculated using the following formula, where F1 is the number of frames of image data that are necessary and sufficient to identify the driver's personal characteristics, and F2 is the number of image frames related to the driver that were used to generate the second corrected gaze distribution Y. St = F2 / F1 …(5)

[0061] The abnormality estimation unit 28 has three determination functions: a first abnormality determination function using the degree of divergence of the probability distribution, a second abnormality determination function using a saliency index (see FIG. 7), and a third abnormality determination function using attentiveness, in order to determine whether the driver is abnormal.

[0062] When the evaluation value, which is the difference between the recent line-of-sight behavior distribution B (for example, within a 30-second period) of the driver and the second corrected line-of-sight distribution Y, is greater than or equal to the determination threshold, the first abnormality determination function determines that the driver is in an abnormal state. In the present embodiment, as shown in FIG. 2, the abnormality estimation unit 26 calculates, using the Kullback-Leibler information amount as the evaluation value, the degree of divergence between the line-of-sight behavior distributions Ba, Bb, Bc and the second corrected line-of-sight distributions Ya, Yb, Yc, respectively.

[0063] The evaluation value Da of the distributions Ba, Ya, the evaluation value Db of the distributions Bb, Yb, and the evaluation value Dc of the distributions Bc, Yc are respectively obtained and substituted into the following formula to calculate the total evaluation value D. D = W × (Da + Dc) + (1 - W) × Db …(6) Here, W (0 < W < 1) is a weighting coefficient. When the learning progress degree St is small (the learning period is short), the total evaluation value D is set such that the weights of the evaluation values Da and Dc are greater than the weight of the evaluation value Db. When the learning progress degree St is large (the learning period is long), the total evaluation value D is set such that the weights of the evaluation values Da and Dc are smaller than the weight of the evaluation value Db. When the total evaluation value D is greater than or equal to the determination threshold (for example, 0.6), it is determined that the driver's state is abnormal.

[0064] The weighting coefficient W can be obtained by the following formula when P, Q, and R are coefficients. W = 1 / (1 + e -P×(St×Q-R) ) …(7) This weighting coefficient W is a monotonically increasing function that is point-symmetric like the sigmoid function, which is one of the activation functions, and the coefficients P, Q, and R can be arbitrarily set.

[0065] The second abnormality determination function estimates that the driver's attention is declining when the saliency index (the integral value of the lower right part of the ROC curve) is equal to or greater than a predetermined determination threshold. The third abnormality judgment function estimates that the driver's attention has decreased if the attention level detection unit 22 detects a high level of attention and the saccade amplitude ds has decreased significantly, and estimates that the driver's attention has decreased if the attention level detection unit 22 detects a low level of attention and the frequency of saccades has decreased significantly.

[0066] An example of a driver's state determination process will be described with reference to the flowcharts of Figures 15 to 19. In the figures, Si (i=1, 2, . . . ) indicates each step.

[0067] An example of the advance learning process is shown in the flowchart of Fig. 15. The advance learning process is basically performed before the vehicle is shipped from the factory, for example, at a testing site. First, various information such as information from the information acquisition unit 1 and various experimental results is read (S1), and then the process proceeds to S2. In S2, the standard saliency index and standard saccade index (amplitude, frequency) for a normal third person are calculated, and then the process proceeds to S3.

[0068] In S3, it is determined whether sufficient data regarding standard saliency indices and standard saccade indices for normal third parties has been obtained. If the result of the determination in S3 is that sufficient data is secured, the reference gaze distributions Aa, Ab, and Ac are calculated (S4), and the process moves to S5. If the result of the determination in S3 is that sufficient data is not secured, the process returns to S1.

[0069] In S5, the first corrected gaze distribution X is obtained, and the first correction coefficient α is calculated using equation (1), and then the process proceeds to S6. Since the first corrected gaze distribution X corresponds to an index of a normal state, the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index correspond to the first corrected gaze distribution X. Therefore, the first correction coefficient α is calculated by substituting the reference gaze distribution A and the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index, which correspond to the first corrected gaze distribution X, into equation (1). In S6, the reference gaze distribution A and the first correction coefficient α are stored in the control unit 2, and then the process ends.

[0070] Next, an example of the driver's state determination process will be shown with reference to the flowchart of FIG. The state determination process is performed when the driver operates the vehicle. First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S11), and the process proceeds to S12. In S12, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S13.

[0071] In S13, it is determined whether the second correction coefficient β is held. If the result of the determination in S13 is that the second correction coefficient β is held, the second corrected gaze distribution Y can be calculated, and the process proceeds to S14. In S14, the reference gaze distribution A and the first correction coefficient α are substituted into equation (1) to calculate the first corrected gaze distribution X, and then the first corrected gaze distribution X and the second correction coefficient β are substituted into equation (2) to determine the second corrected gaze distribution Y.

[0072] In S15, the difference between the gaze behavior distribution B and the second corrected gaze distribution Y is calculated, and the process proceeds to S16. The difference between the gaze behavior distribution B and the second corrected gaze distribution Y is calculated as a deviation (separation distance) using Kullback-Leibler divergence. Each deviation is represented by an evaluation value Da for distributions Ba and Ya, an evaluation value Db for distributions Bb and Yb, and an evaluation value Dc for distributions Bc and Yc.

[0073] In S16, the final total evaluation value D is calculated using the evaluation values ​​Da, Db, and Dc obtained in S15, and the process proceeds to S17. In S17, it is determined whether the total evaluation value D is equal to or greater than the determination threshold value. If the result of the judgment in S17 is that the total evaluation value D is equal to or greater than the judgment threshold, the gaze behavior distribution B representing the driver's current state is deviated from the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, so an abnormality judgment is made that the driver's state is abnormal (S18), and the process ends. If the result of the judgment in S17 is that the total evaluation value D is less than the judgment threshold, the gaze behavior distribution B representing the driver's current state is close to the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, so a normal judgment is made that the driver's state is normal (S19), and the process ends.

[0074] If the result of the determination in S13 is that the second correction coefficient β is not held at the time of the determination, the second corrected gaze distribution Y cannot be calculated, and the process proceeds to S20. In S20, since the second correction coefficient β is not held, the driver's condition is estimated using biometric indicators other than saliency and saccades related to gaze behavior, such as heart rate and blood pressure, and then the process proceeds to S21.

[0075] In S21, it is determined whether the driver's condition is normal or not using biological information indicators other than saliency and saccades related to gaze behavior. If the result of the determination in S21 is that the driver is in a normal state, the process proceeds to S22 to acquire the second correction coefficient β. In S22, the gaze behavior distribution B is accumulated and learned until a highly accurate second correction coefficient β can be acquired, and then the process ends. If the result of the determination in S21 is that the driver is not in a normal state, the second correction coefficient β cannot be acquired, and the process ends.

[0076] Next, an example of the total evaluation value calculation process (S16) will be shown with reference to the flowchart in FIG. The total evaluation value calculation process is performed when calculating the total evaluation value D from the gaze behavior distribution B and the second corrected gaze distribution Y. First, the number of image frames F1 necessary and sufficient for identifying the driver's personal characteristics and the number of image frames F2 used to generate the second corrected gaze distribution Y are substituted into equation (5) to calculate the learning progress level St (S31), and the process proceeds to S32.

[0077] In S32, it is determined whether the learning progress level St is equal to or greater than a predetermined determination threshold value. If the result of the determination in S32 is that the learning progress level St is equal to or greater than a preset determination threshold, the process proceeds to S33 because image data necessary and sufficient to identify the driver's personal characteristics has been acquired and highly accurate learning has already been performed. If the result of the determination in S32 is that the learning progress level St is less than the preset determination threshold, the process returns to S31 because image data necessary and sufficient to identify the driver's personal characteristics has not been acquired and there is insufficient information on human factors related to the driver.

[0078] In S33, a weighting factor W is calculated using the learning progress level St and equation (7), and the process proceeds to S34. In S34, a total evaluation value D is calculated using the weighting factor W, evaluation values ​​Da, Db, and Dc, and equation (6), and the process ends.

[0079] Next, an example of the learning phase process (S22) will be shown with reference to the flowchart of FIG. The learning phase process is performed when learning the gaze behavior distribution B in order to acquire the second correction coefficient β. First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S41), and the process proceeds to S42. In S42, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S43.

[0080] In S43, it is determined whether the driver's condition is normal. If the result of the determination in S43 is that the driver's condition is normal, data on gaze behavior distribution B is accumulated in order to learn gaze behavior distribution B, and the process proceeds to S44. If the result of the determination in S43 is that the driver's condition is not normal, data regarding the gaze behavior distribution B cannot be accumulated, and the process returns to S41. In S44, the reliability setting unit 27a sets the reliability N of the image data to be stored, and the learning limiting unit 27 limits the learning of the image data to be stored in accordance with the reliability N, and the process proceeds to S45.

[0081] In S45, it is determined whether or not a necessary and sufficient amount of data regarding the gaze behavior distribution B has been secured in order to calculate the second correction coefficient β. If the result of the determination in S45 is that the amount of data regarding gaze behavior distribution B has been secured, the process proceeds to S46. If the result of the determination in S45 is that the amount of data regarding gaze behavior distribution B has not been secured, the process returns to S41 to secure more data.

[0082] In S46, the second correction coefficient β is calculated, and after saving the second correction coefficient β (S47), the process ends. Since the second corrected gaze distribution Y corresponds to an index of the normal state, the gaze behavior distribution B in the normal state corresponds to the most recent second corrected gaze distribution Y. Therefore, the second correction coefficient β is calculated by substituting the first corrected gaze distribution X (reference gaze distribution A, first correction coefficient α) and the gaze behavior distribution B corresponding to the most recent second corrected gaze distribution Y into equation (2).

[0083] Next, an example of the reliability determination process (S44) will be shown with reference to the flowchart of FIG. The reliability determination process determines whether or not the stored image data can be stored as learning data for a determination index.

[0084] First, various information such as a reliability setting table is read (S51), and the process proceeds to S52. In S52, the reliability setting unit 27a calculates the first reliability index Ne using equation (3) and calculates the second reliability index Nc using equation (4), and then the process proceeds to S53. In S53, the reliability N is set using the first and second reliability indices Ne and Nc and the reliability setting table, and then the process proceeds to S54.

[0085] In S54, it is determined whether the reliability N is equal to or greater than a determination threshold. If the result of the determination in S54 is that the reliability N is equal to or greater than the determination threshold, the situation is not complex in which personal characteristics are significantly reflected, so the image data is accumulated as learning data for the determination index (S55), and the process ends.If the result of the determination in S54 is that the reliability N is less than the determination threshold, the situation is complex in which personal characteristics are significantly reflected, so the image data is discarded without being used as learning data for the determination index (S56), and the process ends.

[0086] Next, the operation and effects of the driver state determination system S will be described. This driver state determination method includes a driver abnormality estimation process S17 that compares the gaze behavior distribution B with a reference gaze distribution A corrected using a second correction coefficient β, which is learning information, to determine whether the driver is abnormal, and therefore the driver's abnormal state can be estimated based on the gaze behavior. Since the system has a reliability setting process S53 for setting the reliability N of the field of view image included in the driver's field of view and used as learning data for generating the second correction coefficient β based on the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle conditions obtained from the vehicle's physical conditions, it is possible to set the reliability N of the learning data representing the driver's personal characteristics for the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle conditions obtained from the vehicle's physical conditions. When the reliability N as learning data is equal to or greater than the judgment threshold, learning based on the field of view image is executed in step S55, and when the reliability N as learning data is less than the judgment threshold, learning based on the field of view image is restricted in step S56. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

[0087] The method includes a first reliability index calculation step S52 for calculating a first reliability index Ne as learning data of the field of view image based on a first disturbance factor related to the vehicle's surrounding conditions acquired in the surrounding information acquisition step, and a second reliability index calculation step S52 for calculating a second reliability index Nc as learning data of the field of view image based on a second disturbance factor related to the vehicle's physical state acquired in the vehicle state acquisition step, and a reliability setting step S53 for setting the reliability N of the field of view image included in the driver's field of view as learning data using the first reliability index Ne and the second reliability index Nc. As a result, by calculating the first reliability index Ne, which is the reliability N of the learning data related to environmental factors, and calculating the second reliability index Nc, which is the reliability of the learning data related to vehicle factors, the reliability N of the learning data can be set with high accuracy from the perspectives of both environmental factors and vehicle factors.

[0088] This driver condition determination system S has an abnormality estimation unit 28 that compares the gaze behavior distribution B with a reference gaze distribution A corrected using a second correction coefficient β, which is learning information, to determine whether the driver is abnormal, and therefore can estimate whether the driver is in an abnormal state based on the gaze behavior. Since the system has a reliability setting unit 27a that sets the reliability N of the field of view image included in the driver's field of view and used to generate the second correction coefficient β as learning data based on the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle state acquired from the vehicle's physical state, it is possible to set the reliability N of the learning data that represents the driver's personal characteristics for the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle state acquired from the vehicle's physical state. The system has a learning restriction unit 27 that executes learning based on field of view images when the reliability of the learning data is equal to or greater than the judgment threshold, and restricts learning based on field of view images when the reliability N of the learning data is less than the judgment threshold. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

[0089] Next, a modified example in which the above embodiment is partially modified will be described. 1) In the above embodiment, an example was described in which the reference gaze distribution A was corrected using the first correction coefficient α due to the environment and vehicle characteristics. However, the reference gaze distribution A may also be corrected using the first correction coefficient α due to only the environment, or the reference gaze distribution A may also be corrected using the first correction coefficient α due to only the vehicle characteristics.

[0090] 2) In the above embodiment, an example has been described in which the reference gaze distribution generating means is provided in the testing site where the preliminary experiment is conducted, but the reference gaze distribution generating means may also be provided in the control unit 2 of the vehicle.

[0091] 3) In the above embodiment, an example was described in which the learning progress level St was calculated using the number of image frames and equation (3), but it is sufficient to be able to detect at least the learning period, and the learning progress level St may also be calculated using the travel distance or driving time as a parameter.

[0092] 4) In addition, a person skilled in the art may implement the present invention in a form in which various modifications are added to the above-described embodiment without departing from the spirit of the present invention, and the present invention also includes such modifications. [Explanation of symbols]

[0093] 11 External Camera 12 Internal Camera 16 Vehicle condition sensor 17 Driving operation sensor 23 Visual field image creation unit 24 Gaze behavior distribution generation unit 26 Learning gaze distribution generation unit 26a Learning information generation unit 27 Anomaly estimation part 27a Learning progress calculation unit 28 Anomaly estimation section S Driver status determination system

Claims

1. A driver state determination method for determining the state of a driver of a vehicle, comprising: a surrounding information acquisition step of acquiring information about a surrounding situation of the vehicle; a vehicle state acquisition step of acquiring physical states of the vehicle, including a speed, an acceleration, and a yaw rate; a visual field image creation step of creating a visual field image that appears in the driver's field of view; a reference gaze distribution generating step of generating a reference gaze distribution based on a standard field of view image in advance; a gaze behavior distribution generating step of generating a gaze behavior distribution based on the gaze behavior of the driver; a learning information generating step of generating learning information that associates a surrounding situation of the vehicle, a physical state of the vehicle, and a gaze behavior of the driver; a driver abnormality inference step of comparing the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal; a reliability setting step of setting the reliability of a field of view image included in the driver's field of view and used to generate the learning information as learning data based on a traffic situation estimated based on a surrounding situation of the vehicle and a vehicle state acquired from a physical state of the vehicle; a learning limiting step of executing learning based on the field of view image when the reliability as the learning data is equal to or greater than a judgment threshold, and limiting learning based on the field of view image when the reliability as the learning data is less than the judgment threshold; A driver state determination method comprising:

2. A driver state determination system for determining the state of a vehicle driver, a surrounding information acquisition means for acquiring information about the surrounding conditions of the vehicle; a vehicle state acquisition means for acquiring a physical state of the vehicle including a speed, an acceleration, and a yaw rate; a field of view image creating means for creating a field of view image that appears in the driver's field of view; a reference gaze distribution generating means for generating a reference gaze distribution based on a standard field of view image in advance; a gaze behavior distribution generating means for generating a gaze behavior distribution based on the gaze behavior of the driver; a learning information generating means for generating learning information that associates a surrounding situation of the vehicle, a physical state of the vehicle, and a gaze behavior of the driver; a driver abnormality estimation means for comparing the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal; a reliability setting means for setting the reliability of a field of view image included in the driver's field of view and used to generate the learning information as learning data based on a traffic situation estimated based on a surrounding situation of the vehicle and a vehicle state acquired from a physical state of the vehicle; a learning restriction means for executing learning based on the field of view image when the reliability as the learning data is equal to or greater than a judgment threshold, and for restricting learning based on the field of view image when the reliability as the learning data is less than the judgment threshold; A driver state determination system comprising:

Images