Driver condition determination device

Abstract

To provide a driver state determination device capable of appropriately determining whether or not to issue an alarm to a driver.SOLUTION: A driver state determination device comprises: an arithmetic logical unit for calculating a region toward which a visual line of a driver is directed, on the basis of an image where a driver 200 of a vehicle 100 is imaged; a measurement unit which measures an elapsed time which has elapsed with the visual line of the driver unchanged; and a determination unit which determines whether or not to issue an alarm to the driver on the basis of the elapsed time. The determination unit applies different weights between to the elapsed time as of when the visual line of the driver is directed toward a first region R1, and to the elapsed time as of when the visual line of the driver is directed toward a region different from the first region, and, then, determines whether or not to issue an alarm. The first region is a partial region of a windshield 101 of the vehicle, toward which the visual line of the driver is directed when the driver gazes at the front of the vehicle.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driver condition determination device. [Background technology]

[0002] Conventionally, there is a technology for making a judgment based on the direction of the driver's face, etc. Patent Document 1 discloses a distraction judgment device that judges that the driver is looking away when a visual target judgment unit judges that the environment is one in which a visual target exists outside the vehicle and a gaze judgment unit judges that the driver is gazing inside the vehicle.

[0003] Patent document 2 discloses a distracted driving detection device that includes a main camera installed in the driver's seat of the vehicle to capture an image of the driver's face, and an auxiliary camera installed at a location where a specific on-board component is installed that is visible to the driver while the vehicle is moving, to capture an image of the driver's face from a location different from the main camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-021399 [Patent Document 2] Japanese Patent Publication No. 2020-024532 Summary of the Invention [Problem to be solved by the invention]

[0005] There is still room for improvement in determining whether to issue a warning to the driver. For example, it is thought that the impact on safety varies depending on the area in which the driver's line of sight is directed. It is therefore desirable to be able to appropriately determine whether to issue a warning to the driver.

[0006] An object of the present invention is to provide a driver condition determination device that can appropriately determine whether or not to issue a warning to the driver. [Means for solving the problem]

[0007] The driver condition determination device of the present invention comprises a calculation unit that calculates which area the driver's line of sight is directed toward based on an image of the driver of a vehicle, a measurement unit that measures the elapsed time that has passed without the direction of the driver's line of sight changing, and a determination unit that determines whether or not to issue an alert to the driver based on the elapsed time, wherein the determination unit weights the elapsed time differently when the driver's line of sight is directed toward a first area and when the driver's line of sight is directed toward an area other than the first area, and the first area is a part of the windshield of the vehicle, and is the area toward which the driver's line of sight is directed when the driver looks ahead of the vehicle. [Effects of the Invention]

[0008] In the driver condition determination device according to the present invention, the determination unit determines whether to issue a warning by assigning different weights to elapsed time when the driver's line of sight is directed toward a first area and when the driver's line of sight is directed toward an area other than the first area. The first area is a part of the windshield of the vehicle, and is the area toward which the driver's line of sight is directed when the driver looks ahead of the vehicle. The driver condition determination device according to the present invention has the effect of being able to appropriately determine whether to issue a warning to the driver. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a driver condition determination device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing each region of the embodiment. [Figure 3] FIG. 3 is a diagram showing each region of the embodiment. [Figure 4] FIG. 4 is a diagram showing the movement of the line of sight. [Figure 5] FIG. 5 is a diagram showing variations in line of sight. [Figure 6] FIG. 6 is a diagram showing tracking of a gaze target. [Figure 7] FIG. 7 is a flowchart showing the operation of the driver condition determination device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing a boundary region according to a first modified example of the embodiment. [Figure 9] FIG. 9 is a diagram illustrating the probability distribution. [Figure 10] FIG. 10 is a diagram illustrating the calculation of the position coefficient in the first modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] A driver condition determination device according to an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this embodiment. Furthermore, the components in the following embodiments include those that can be easily imagined by a person skilled in the art or that are substantially the same.

[0011] [Embodiment] An embodiment will be described with reference to Fig. 1 to Fig. 7. The embodiment relates to a driver condition determination device. Fig. 1 is a block diagram of the driver condition determination device according to the embodiment, Fig. 2 and Fig. 3 are diagrams showing each area of ​​the embodiment, Fig. 4 is a diagram showing line-of-sight movement, Fig. 5 is a diagram showing line-of-sight variation, Fig. 6 is a diagram showing tracking of a gaze target, and Fig. 7 is a flowchart showing the operation of the driver condition determination device according to the embodiment.

[0012] As shown in Fig. 1, a driver condition determination device 1 of this embodiment is mounted on a vehicle 100 such as an automobile. The driver condition determination device 1 determines the state of the driver based on an image of the driver of the vehicle 100. The driver condition determination device 1 has a camera 2, a processing unit 3, and an alarm output device 4.

[0013] Camera 2 is a camera for acquiring images for calculating the driver's facial orientation or line of sight. An infrared camera may be used as camera 2, taking into consideration the effects of external light. Camera 2 may also be a stereo camera. In this case, measuring the distance between the driver and camera 2 makes it possible to more accurately detect the gaze position, which will be described later.

[0014] The installation position of the camera 2 is, for example, the position of an existing in-vehicle component such as the inside of the meter, the instrument panel, or the steering wheel of the vehicle 100. In other words, the camera 2 is placed in a position where it can capture an image of the driver's face. The camera 2 may be installed as a retrofit to the vehicle 100. When the camera 2 is installed as a retrofit, it is necessary to provide the processing unit 3 with information about the installation position of the camera 2.

[0015] The processing unit 3 is, for example, a computer including an arithmetic device such as a CPU, a communication interface, and a memory. The processing unit 3 acquires images captured by the camera 2. The processing unit 3 also outputs commands to the alarm output device 4. The processing unit 3 has a gaze direction calculation unit 31, a face orientation calculation unit 32, a gaze position calculation unit 33, a measurement unit 34, a determination unit 35, and a position coefficient recording unit 36.

[0016] The gaze direction calculation unit 31 calculates the direction in which the driver's gaze is directed based on the image acquired from the camera 2. Methods for gaze detection using the camera 2 include the pupil-corneal reflex method, model-based estimation, and appearance-based estimation, and any of these methods may be used. The gaze direction calculation unit 31 calculates, for example, the angle between the direction in which the driver's face faces and the direction in which the driver's gaze is directed. The calculated angles are, for example, the angle in the vertical direction and the angle in the vehicle width direction.

[0017] The face direction calculation unit 32 calculates the direction in which the driver's face is facing based on the image acquired from the camera 2. Methods for detecting the face direction using the camera 2 include facial feature point detection and machine learning, and any of these methods may be used.

[0018] The gaze position calculation unit 33 calculates the position at which the driver is looking, based on the gaze direction calculated by the gaze direction calculation unit 31 and the facial direction calculated by the facial direction calculation unit 32. In other words, the gaze position calculation unit 33 calculates the direction in which the driver's gaze is directed in a coordinate system based on the vehicle 100.

[0019] The gaze position calculation unit 33 according to this embodiment calculates which of a plurality of predetermined regions the driver's line of sight is directed to. FIG. 2 shows a diagram of each region when looking forward from inside the vehicle 100. The vehicle 100 has a windshield 101 disposed in front of the driver 200. The vehicle 100 also has door mirrors 102. The door mirrors 102 are disposed on the left door 103L and the right door 103R relative to the driver 200.

[0020] As shown in Figures 2 and 3, the windshield 101 has a first region R1, a second region R2, and a third region R3. The first region R1 is a portion of the windshield 101 and is the region to which the driver 200's line of sight is directed when the driver 200 gazes ahead of the vehicle 100. Figure 2 shows a reference line SH in the vehicle width direction and a reference line SV in the up-down direction. The intersection SX of the two reference lines SH and SV is the position to which the driver 200's line of sight is directed when the driver 200 naturally faces forward. The first region R1 is the region that includes the intersection SX of the two reference lines SH and SV when viewed from the vehicle longitudinal direction. The center of the first region R1 coincides with the intersection SX when viewed from the vehicle longitudinal direction, for example.

[0021] The first region R1 is, for example, a region corresponding to the effective field of view of the driver 200 when looking directly ahead. The illustrated first region R1 has a rectangular shape. In the illustrated first region R1, the length in the vehicle width direction is greater than the length in the up-down direction.

[0022] The third region R3 is an edge portion of the windshield 101. The illustrated third region R3 has a frame shape and includes an upper end, a lower end, a left end, and a right end of the windshield 101.

[0023] The second region R2 is a region between the first region R1 and the third region R3. The second region R2 extends in the vehicle width direction from near the left edge of the windshield 101 to near the right edge. The second region R2 also extends in the vertical direction from near the top edge of the windshield 101 to near the bottom edge. The right end of the second region R2 is recessed so as to surround the first region R1. The area of ​​the second region R2 is larger than the area of ​​the first region R1.

[0024] The vehicle 100 is further defined with a fourth area R4 and a fifth area R5. The fourth area R4 is an area corresponding to the reflective surface of the door mirror 102. The fifth area R5 is an area surrounding the windshield 101 and excluding the fourth area R4. The fifth area R5 is an area outside the windshield 101 in the vehicle width direction H and an area outside the windshield 101 in the vertical direction V. The interior devices and interior wall surfaces are included in the fifth area R5.

[0025] Gaze position calculation unit 33 calculates which of first region R1 to fifth region R5 driver 200's line of sight is directed to. Gaze position calculation unit 33, for example, calculates the direction of driver 200's line of sight at predetermined time intervals and determines which region driver 200's line of sight is directed to. For example, each time one frame is captured by camera 2, gaze position calculation unit 33 calculates the direction driver 200's line of sight is directed to based on the image of the latest frame.

[0026] The measurement unit 34 measures the elapsed time T0. The elapsed time T0 is the time that has passed without any change in the line of sight of the driver 200. The measurement unit 34 resets the elapsed time T0 when the line of sight of the driver 200 changes significantly or when the driver 200 moves his / her line of sight to a different area.

[0027] To determine whether the direction of the gaze has changed significantly, for example, the value of the eyeball angular velocity that results in saccade or the change in gaze position in the time difference between acquired images can be used. When the direction of the gaze of driver 200 changes significantly as shown by arrow Ar1 in FIG. 4, elapsed time T0 is reset. The threshold value of the eyeball angular velocity that determines saccade is, for example, 100° / sec. The reason for adopting such a threshold is, for example, the variation in accuracy of gaze measurement and the existence of movements such as pursuit movements. Pursuit movements are movements in which the eyes continue to follow an object that moves relatively due to the movement of the vehicle. Note that the change in gaze position may also be used to determine saccade.

[0028] As will be explained with reference to Fig. 5, if the change in gaze direction is within the range of variation, it is determined that the gaze direction has not changed. Fig. 5 shows the distribution of the gaze direction D0 calculated last and the gaze direction D1 calculated thereafter. If the amount of change in the gaze directions D0 and D1 is equal to or less than a threshold, it is determined that the gaze direction has not changed. This determination is made, for example, by the gaze position calculation unit 33.

[0029] As will be described with reference to Fig. 6, when the line of sight of driver 200 is following a gaze target, it is not determined that the line of sight direction has changed. Fig. 6 shows the movement of the line of sight when driver 200 is gazing at sign Ts ahead. The time interval from the upper to lower sections of Fig. 6 is, for example, the interval between images captured by camera 2. As vehicle 100 moves forward, the line of sight direction of driver 200 changes along vehicle width direction H. When the amount of change ΔH in the line of sight direction has a magnitude corresponding to the traveling speed of vehicle 100, gaze position calculation unit 33 does not determine that the line of sight direction has changed.

[0030] The determination unit 35 determines, based on the elapsed time T0, whether or not to issue a warning to the driver 200. The warning to the driver 200 is issued, for example, in a situation where it is determined that the driver 200 is looking away or in a situation where it is determined that the driver 200 is distracted.

[0031] The determination unit 35 of this embodiment weights the elapsed time T0 to calculate a corrected elapsed time T1. The weighting value is determined according to the area to which the driver 200 is looking. If the corrected elapsed time T1 is equal to or greater than a threshold, the determination unit 35 determines to issue a warning to the driver 200.

[0032] In this embodiment, the following weighting values ​​are determined for each of the regions R1, R2, R3, R4, and R5. The weighting value for the elapsed time T0 is a position coefficient according to the position where the driver 200 is looking. The magnitude of the position coefficient is determined according to, for example, the effect on the safety level. First area R1:0.10 Second area R2:0.25 Third area R3:0.50 Fourth area R4:0.50 Fifth area R5:1.00

[0033] As described above, the weighting value for the fifth region R5 is greater than the weighting values ​​for any of the other regions R1, R2, R3, and R4. The fifth region R5 is an area that is unlikely to be seen while driving, and therefore has the largest position coefficient. Therefore, if the driver 200 continues to direct their gaze toward the same area, the fifth region R5 is the area that will be the first to be judged as a warning.

[0034] The weighting value for the first region R1 is smaller than the weighting values ​​for any of the other regions R2, R3, R4, and R5. However, the weighting value for the first region R1 is greater than 0. In other words, if a long time passes without a change in the gaze direction in the first region R1, a warning determination is made. An example of a situation in which a warning is issued when the driver 200 is directing his or her gaze toward the first region R1 is when the driver 200's attention is distracted. When the driver 200 is thinking about something, he or she may not be paying sufficient attention to the road ahead. In such a case, if the corrected elapsed time T1 reaches a threshold value without a change in the gaze direction, a warning is issued.

[0035] The weighting value increases as the area to which the driver 200 is directed becomes farther from the first area R1. The third area R3 and the fourth area R4 are areas that are farther from the first area R1 than the second area R2. Therefore, the position coefficients of the third area R3 and the fourth area R4 are greater than the position coefficient of the second area R2. Therefore, the driver condition determination device 1 according to this embodiment can appropriately determine whether to issue a warning to the driver 200.

[0036] The operation of the driver condition determination device 1 according to this embodiment will be described with reference to the flowchart of Fig. 7. In step S10, the processing unit 3 acquires an image of the face of the driver 200 from the camera 2. After step S10 is executed, the process proceeds to step S20.

[0037] In step S20, gaze direction calculation unit 31 calculates the gaze direction of driver 200. Based on the image acquired in step S10, gaze direction calculation unit 31 calculates the gaze direction relative to the orientation of the face of driver 200. After step S20 is executed, the process proceeds to step S30.

[0038] In step S30, face direction calculation unit 32 calculates the face direction of driver 200. Face direction calculation unit 32 calculates the face direction of driver 200 based on the image acquired in step S10. After step S30 is executed, the process proceeds to step S40.

[0039] In step S40, gaze position calculation unit 33 calculates the direction of the gaze of driver 200. Gaze position calculation unit 33 calculates the position of the gaze of driver 200 based on the gaze direction calculated in step S20 and the facial orientation calculated in step S30. After step S40 is executed, the process proceeds to step S50.

[0040] In step S50, the determination unit 35 reads the position coefficient. The value of the position coefficient corresponding to each position is stored in advance in the position coefficient recording unit 36. The position coefficient recording unit 36 ​​is, for example, a ROM, and holds position coefficients for weighting. As described above, a position coefficient value is determined for each of the regions R1, R2, R3, R4, and R5. The determination unit 35 reads the value of the position coefficient corresponding to the region to which the driver 200 is looking from the position coefficient recording unit 36. After step S50 is executed, the process proceeds to step S60.

[0041] In step S60, the processing unit 3 turns on time measurement by the measurement unit 34. After the time measurement is turned on, the measurement unit 34 continues to measure the elapsed time T0 until the time measurement is reset. After step S60 is executed, the process proceeds to step S70.

[0042] In step S70, the determination unit 35 determines whether the line of sight of the driver 200 has changed. The determination unit 35 makes the determination in step S70 by, for example, comparing the image of the latest frame with the image of the previous frame. If the determination in step S70 is affirmative, that is, that the line of sight has not changed, the process proceeds to step S80. On the other hand, if the determination is negative, that is, that the line of sight has changed, the process proceeds to step S140.

[0043] In step S80, the determination unit 35 calculates the corrected elapsed time T1. The determination unit 35 weights the elapsed time T0 by the position coefficient read in step S50. That is, the determination unit 35 calculates the corrected elapsed time T1 by multiplying the elapsed time T0 acquired from the measurement unit 34 by the value of the position coefficient. After step S80 is executed, the process proceeds to step S90.

[0044] In step S90, the determination unit 35 determines whether the corrected elapsed time T1 is equal to or greater than the threshold value Th. The value of the threshold value Th is, for example, 1 [sec]. This value is calculated, for example, from the frequency distribution of inattentive time while driving in a city. When the driver 200's line of sight is directed toward the fifth region R5, the value of the position coefficient is 1.00. Therefore, if 1 [sec] passes without a change in the line of sight, a warning determination is made. When the driver 200's line of sight is directed toward the third region R3 or the fourth region R4, the value of the position coefficient is 0.50. In this case, if the elapsed time T0 becomes 2 [sec] without a change in the line of sight, the corrected elapsed time T1 becomes 1 [sec], and a warning determination is made.

[0045] When the line of sight of driver 200 is directed toward second region R2, the value of the position coefficient is 0.25. In this case, if the elapsed time T0 becomes 4 [sec] without a change in the direction of the line of sight, the corrected elapsed time T1 becomes 1 [sec], and a warning determination is made. When the line of sight of driver 200 is directed toward first region R1, the value of the position coefficient is 0.10. In this case, if the elapsed time T0 becomes 10 [sec] without a change in the direction of the line of sight, the corrected elapsed time T1 becomes 1 [sec], and a warning determination is made. If the result of the determination in step S90 is that the corrected elapsed time T1 is equal to or greater than threshold value Th and a positive determination is made, the process proceeds to step S100. If a negative determination is made, the process proceeds to step S10.

[0046] In step S100, the determination unit 35 determines to issue a warning to the driver 200. The determination unit 35, for example, commands the warning output device 4 to output a warning. After step S100 is executed, the process proceeds to step S110.

[0047] In step S110, the warning output device 4 outputs a warning to the driver 200. The warning output device 4 outputs a warning to the driver 200 by, for example, sound, light, image, vibration, or the like. The warning output device 4 may be an independent buzzer, speaker, or video display device. Existing in-vehicle components such as a buzzer, speaker, or meter may also be used as the warning output device 4. After step S110 is executed, the process proceeds to step S120.

[0048] In step S120, determination unit 35 determines whether the direction of driver 200's line of sight has changed significantly. In this case, determination unit 35 obtains from camera 2 an image after warning output device 4 has issued a warning, and makes the determination in step S120. The threshold value for the determination in this case may be, for example, the same value as the threshold value used to determine whether the amount of change in the line of sight direction is within the range of variation. If the determination in step S120 results in a positive determination that the line of sight has changed significantly, the process proceeds to step S120, and if a negative determination is made, the process proceeds to step S100.

[0049] In step S130, the determination unit 35 stops the alarm and instructs the alarm output device 4 to stop the alarm. After step S130 is executed, the process proceeds to step S10.

[0050] In step S140, the determination unit 35 resets the time measurement. The determination unit 35 outputs a reset signal to the measurement unit 34, for example. Upon receiving the reset signal, the measurement unit 34 resets the elapsed time T0 to its initial value of 0 and stops measuring the elapsed time T0. After step S140 is executed, the process proceeds to step S10.

[0051] As described above, the driver state determination device 1 according to this embodiment includes the calculation unit 33, the measurement unit 34, and the determination unit 35. The calculation unit 33 calculates the area in which the line of sight of the driver 200 of the vehicle 100 is directed based on an image of the driver 200. The measurement unit 34 measures the elapsed time T0 during which the direction of the line of sight of the driver 200 remains unchanged. The determination unit 35 determines whether or not to issue a warning to the driver 200 based on the elapsed time T0.

[0052] The determination unit 35 determines whether to issue a warning by assigning different weights to the elapsed time T0 when the line of sight of the driver 200 is directed toward the first region R1 and when the line of sight of the driver 200 is directed toward a region other than the first region R1. The first region R1 is a partial region of the windshield 101 of the vehicle 100, and is the region toward which the line of sight of the driver 200 is directed when the driver 200 gazes ahead of the vehicle 100. The driver condition determination device 1 according to the present embodiment can appropriately determine whether to issue a warning to the driver 200 by weighting the elapsed time T0 according to the region.

[0053] In this embodiment, the determination unit 35 determines to issue a warning when the corrected elapsed time T1 is equal to or greater than the threshold value Th. The corrected elapsed time T1 is a weighted elapsed time. The weighting value when the driver 200's line of sight is directed to areas R2, R3, R4, and R5 that are different from the first area R1 is greater than the weighting value when the driver 200's line of sight is directed to the first area R1. Therefore, an appropriate determination can be made depending on the magnitude of the influence that the direction of the line of sight has on the safety level.

[0054] The windshield 101 of this embodiment has a first region R1, a third region R3, and a second region R2 between the first region R1 and the third region R3. The third region R3 is an edge of the windshield 101. The weighting value when the driver 200's line of sight is directed toward the third region R3 is greater than the weighting value when the driver 200's line of sight is directed toward the second region R2. Therefore, an appropriate determination can be made depending on the magnitude of the influence that the direction of the line of sight has on the safety level.

[0055] When the line of sight of driver 200 is directed toward door mirror 102 of vehicle 100, determination unit 35 of this embodiment weights elapsed time T0 with the same value as when the line of sight of driver 200 is directed toward third region R3. This allows appropriate determination to be made depending on the magnitude of the influence that the direction of the line of sight has on the safety level.

[0056] In this embodiment, the weighting value when the line of sight of driver 200 is not directed toward windshield 101 is larger than the weighting value when the line of sight of driver 200 is directed toward windshield 101. This allows an appropriate determination to be made depending on the magnitude of the influence that the direction of the line of sight has on the safety level.

[0057] The driver state determination device 1 according to this embodiment can determine whether the driver is looking away from the driver without requiring measurements using multiple sensors or an expensive calculation CPU.

[0058] [First Modification of the Embodiment] A first modified example of the embodiment will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is a diagram showing a boundary region according to the first modified example of the embodiment, Fig. 9 is a diagram explaining the probability distribution, and Fig. 10 is a diagram explaining the calculation of the position coefficient in the first modified example of the embodiment. The first modified example of the embodiment differs from the above embodiment in that, for example, the position coefficient is rounded based on the existence probability at the boundary part of the region.

[0059] As shown in FIG. 8, boundary regions R11 and R12 are set in the first region R1. The boundary regions R11 and R12 are edge regions in the first region R1. In other words, the boundary regions R11 and R12 are regions in the boundaries of the first region R1 with other regions R2 and R3. The boundary region R12 is the outermost region in the first region R1. The boundary region R12 is in contact with the second region R2 and the third region R3. The boundary region R11 is continuous with the boundary region R12 and is a region located more inward than the boundary region R12.

[0060] The second region R2 has boundary regions R21 and R22. The boundary regions R21 and R22 are edge regions of the second region R2 and are regions at the boundaries with the other regions R1 and R3. The boundary region R22 is the outermost region of the second region R2. The boundary region R22 is in contact with the first region R1 and the third region R3. The boundary region R21 is continuous with the boundary region R22 and is a region located more inward than the boundary region R22.

[0061] The third region R3 has boundary regions R31 and R32. The boundary regions R31 and R32 are edge regions of the third region R3 and are regions at the boundary with the fifth region R5. The boundary region R32 is the outermost region of the third region R3. The boundary region R32 is in contact with the fifth region R5. The boundary region R31 is continuous with the boundary region R32 and is a region inside the boundary region R32.

[0062] The fourth region R4 has boundary regions R41 and R42. The boundary regions R41 and R42 are edge regions of the fourth region R4 and are regions at the boundary with the fifth region R5. The boundary region R42 is the outermost region of the fourth region R4. The boundary region R42 is in contact with the fifth region R5. The boundary region R41 is continuous with the boundary region R42 and is a region inside the boundary region R42.

[0063] The boundary region will be further described with reference to Figures 9 and 10. The gaze direction Dc calculated by the gaze position calculation unit 33 has an error with respect to the actual gaze direction. As shown in Figure 9, it is assumed that the error in the calculated gaze direction Dc with respect to the actual gaze direction follows a normal distribution. If the accuracy in detecting the gaze direction is ±3[deg], the standard deviation σ is approximately 1[deg]. The probability that the actual gaze direction is within ±1[deg] of the calculated gaze direction Dc is 68[%]. Furthermore, the probability that the actual gaze direction is within ±2[deg] is 95[%], and the probability that the actual gaze direction is within ±3[deg] is approximately 100[%].

[0064] In a normal probability distribution, if the probability of existence near the center (within σ) is 1, the probability of existence between σ and 2σ is 0.400, and the probability of existence between 2σ and 3σ is 0.074.

[0065] As shown in FIG. 10, the first region R1 has boundary lines B11, B12, and B13. The boundary line B13 is the boundary line between the first region R1 and the other regions R2 and R3. The boundary line B12 is the boundary line between the two boundary regions R11 and R12. Therefore, the boundary line B12 is located inside the boundary line B13. The width W12 of the boundary region R12 corresponds to the angle of the standard deviation σ described above. In other words, when the standard deviation σ is 1 [deg], the value of the width W12 corresponds to 1 [deg] in the line of sight direction.

[0066] The boundary line B11 is an inner boundary line in the boundary region R11. Therefore, the boundary line B11 is located more inward than the boundary line B12. The width W11 of the boundary region R11 corresponds to the angle of the standard deviation σ. In other words, when the standard deviation σ is 1 [deg], the value of the width W11 corresponds to 1 [deg] of the line of sight direction. In the illustrated first region R1, the boundary line B13 corresponds to 3σ, the boundary line B12 corresponds to 2σ, and the boundary line B11 corresponds to σ. In other words, the boundary region R11 is a region from σ to 2σ, and the boundary region R12 is a region from 2σ to 3σ.

[0067] 10 is a position inside the boundary line B11. When the calculated gaze direction Dc is a position inside the boundary line B11 like position P1, the actual gaze direction is considered to be the first region R1. In other words, if the calculated gaze direction Dc is inside the boundary line B11, the probability that driver 200's gaze is directed toward first region R1 (probability of presence) is set to 1.

[0068] Position P2 shown in FIG. 10 is a position inside boundary region R11. When the calculated gaze direction Dc is position P2, the probability that the actual gaze direction is in first region R1 is 0.400. Here, position P2 is adjacent to second region R2. Therefore, when the calculated gaze direction Dc is position P2, the probability that the actual gaze direction is in second region R2 is 0.600 (= 1 - 0.400).

[0069] 10 is a position inside boundary region R12. When the calculated gaze direction Dc is at position P3, the probability that the actual gaze direction is in first region R1 is 0.074. Furthermore, when the calculated gaze direction Dc is at position P3, the probability that the actual gaze direction is in second region R2 is 0.926 (= 1 - 0.074).

[0070] The second region R2 has boundary lines B21, B22, and B23. The boundary lines B21, B22, and B23 are defined in the same manner as the boundary lines B11, B12, and B13 of the first region R1. The boundary line B23 is the boundary line between the second region R2 and the third region R3. The boundary line B22 is the boundary line between the boundary regions R21 and R22. The boundary line B21 is the inner boundary line of the boundary region R21.

[0071] If the calculated gaze direction Dc is inside the boundary line B21, the probability that the actual gaze direction is in the second region R2 is considered to be 1. If the calculated gaze direction Dc is in the boundary region R21, the probability that the actual gaze direction is in the second region R2 is considered to be 0.400. If the calculated gaze direction Dc is in the boundary region R22, the probability that the actual gaze direction is in the second region R2 is considered to be 0.074.

[0072] The third region R3 has boundary lines B31, B32, and B33. The boundary lines B31, B32, and B33 are defined in the same manner as the boundary lines B11, B12, and B13 of the first region R1. The boundary line B33 is the boundary line between the third region R3 and the fifth region R5. The boundary line B32 is the boundary line between the boundary regions R31 and R32. The boundary line B31 is the inner boundary line of the boundary region R31.

[0073] If the calculated gaze direction Dc is inside the boundary line B31, the probability that the actual gaze direction is in the third region R3 is considered to be 1. If the calculated gaze direction Dc is in the boundary region R31, the probability that the actual gaze direction is in the third region R3 is considered to be 0.400. If the calculated gaze direction Dc is in the boundary region R32, the probability that the actual gaze direction is in the third region R3 is considered to be 0.074.

[0074] A similar boundary line is defined for the fourth region R4. If the calculated gaze direction Dc is inside the boundary region R41, the probability that the actual gaze direction is in the fourth region R4 is considered to be 1. If the calculated gaze direction Dc is in the boundary region R41, the probability that the actual gaze direction is in the fourth region R4 is considered to be 0.400. If the calculated gaze direction is in the boundary region R42, the probability that the actual gaze direction is in the fourth region R4 is considered to be 0.074.

[0075] The determination unit 35 apportions the position coefficient based on the above-mentioned existence probability. For example, the position coefficient for the above-mentioned position P1 is calculated by multiplying the position coefficient of the first region R1, 0.1, by the existence probability 1.0 using the following formula (1). Therefore, the value of the position coefficient for position P1 is 0.1. Position coefficient of position P1 = 0.1 × 1 (1)

[0076] The position coefficient for position P2 is calculated using the following formula (2) based on the position coefficient of the first region R1 of 0.1, the existence probability of the boundary region R11 of 0.400, and the position coefficient of the second region R2 of 0.25. Therefore, the value of the position coefficient for position P2 is 0.190. Position coefficient of position P2 = 0.1 × 0.400 + 0.25 × (1 - 0.400) (2)

[0077] Position P4 in Figure 10 is located within boundary region R12 and is adjacent to both second region R2 and third region R3. The position coefficient of position P4 is calculated using equation (3) below based on the position coefficient of first region R1 (0.1), the existence probability of boundary region R12 (0.074), the position coefficient of second region R2 (0.25), the existence probability of boundary region R22 (0.074), and the position coefficient of third region R3 (0.5). Therefore, the value of the position coefficient of position P4 is 0.4519. Position coefficient of position P4 = 0.1 × 0.074 + 0.25 × 0.074 + 0.5 × (1 - 0.074 - 0.074) (3)

[0078] Position P5 in Figure 10 is located inside boundary region R22 and adjacent to third region R3. The position coefficient of position P5 is calculated using the following formula (4) based on the position coefficient of second region R2 (0.25), the existence probability of boundary region R22 (0.074), and the position coefficient of third region R3 (0.5). Therefore, the value of the position coefficient of position P5 is 0.4815. Position coefficient of position P5 = 0.25 × 0.074 + 0.5 × (1 - 0.074) (4)

[0079] Position P6 in Figure 10 is located inside boundary region R31 and adjacent to fifth region R5. The position coefficient of position P6 is calculated using the following formula (5) based on the position coefficient of third region R3 (0.5), the existence probability of boundary region R31 (0.4), and the position coefficient of fifth region R5 (1.0). Therefore, the value of the position coefficient of position P6 is 0.800. Position coefficient of position P6 = 0.5 × 0.400 + 1 × (1 - 0.400) (5)

[0080] As described above, when the line of sight of the driver 200 is directed toward the boundary of a plurality of adjacent regions, the determination unit 35 according to the first modification of the embodiment weights the elapsed time T0 based on the probability distribution. The weighting value at this time is a value between the maximum and minimum weighting values ​​for the plurality of regions. For example, the position P2 is the boundary between the first region R1 and the second region R2. In this case, the maximum weighting value for the regions R1 and R2 is 0.25, which is the position coefficient of the second region R2, and the minimum weighting value is 0.1, which is the position coefficient of the first region R1. The weighting value for the position P2 is 0.190, which is calculated by equation (2). In other words, the weighting value for the position P2 is greater than the minimum value 0.1 and less than the maximum value 0.25.

[0081] By setting the weighting values ​​as described above, the change in the weighting values ​​at the boundary between the multiple regions is mitigated. Note that an intermediate region may be set at the boundary between the multiple regions. For example, an intermediate region may be set between the first region R1 and the second region R2. In this case, the weighting value in the intermediate region is greater than the weighting value in the first region R1 and less than the weighting value in the second region R2.

[0082] [Second Modification of the Embodiment] A second modified example of the embodiment will be described. The threshold value Th for the corrected elapsed time T1 may be determined depending on the driving environment and the driving speed. For example, the threshold value Th may be smaller on an ordinary road than on a motorway. For example, the threshold value Th may be smaller when the driving speed is high than when the driving speed is low. The threshold value Th may be variable depending on other factors. For example, a correction constant may be added to the formula for determining the position coefficient, taking into account the traffic conditions around the vehicle 100 or the situation when parking.

[0083] The number and shapes of the regions in the windshield 101 are not limited to the illustrated number and shapes. For example, the windshield 101 may be divided into a first region R1 and a region other than the first region R1. The windshield 101 may be provided with four or more regions.

[0084] The shape of the first region R1 is not limited to a rectangle, and may be, for example, a rhombus or a trapezoid, a polygon such as a hexagon, or a circle.

[0085] The driver state determination device 1 can also be applied to visual evaluation of a display system projected on a windshield, such as a head-up display (HUD) or an augmented reality head-up display (AR-HUD). For example, it can be applied to determining whether the driver is driving safely or evaluating the displayed display.

[0086] The contents disclosed in the above-described embodiments and modifications can be implemented in appropriate combinations. [Explanation of symbols]

[0087] 1. Driver status determination device 2 Cameras 3 Processing Unit 4 Alarm output device 31: Gaze direction calculation unit, 32: Face direction calculation unit, 33: Gaze position calculation unit 34: Measurement unit, 35: Determination unit, 36: Position coefficient recording unit 100: vehicle, 101: windshield, 102: door mirror 103L, 103R: Door B11,B12,B13: Boundary line B21,B22,B23:Border line B31,B32,B33:Border line R1: First area, R11, R12: Boundary area R2: Second area, R21, R22: Boundary area R3: Third area, R31, R32: Boundary area R4: Fourth area, R41, R42: Boundary area R5: Fifth area T0: elapsed time, T1: corrected elapsed time, Th: threshold

Claims

1. a calculation unit that calculates, based on an image of the driver of the vehicle, to which area the driver's line of sight is directed; a measuring unit that measures an elapsed time during which the driver's line of sight remains unchanged; a determination unit that determines whether to issue a warning to the driver based on the elapsed time; Equipped with the elapsed time is a time during which a change in the direction of the driver's line of sight remains equal to or less than a threshold, the measurement unit resets measurement of the elapsed time when the amount of change exceeds the threshold; the determination unit determines whether to issue the warning by assigning different weights to the elapsed time depending on whether the driver's line of sight is directed toward a first area or a area different from the first area; the first area is a part of a windshield of the vehicle, and is an area to which the driver's line of sight is directed when the driver looks ahead of the vehicle, the determination unit weights the elapsed time based on a probability distribution when the driver's line of sight is directed toward a boundary portion of a plurality of adjacent regions, and the weighting value is a value between a maximum value and a minimum value of weighting values ​​for the plurality of regions; The probability distribution is a distribution of errors between the calculated value of the driver's line of sight and the actual line of sight. A driver state determination device characterized by:

2. the determination unit determines that the warning should be issued when the weighted elapsed time is equal to or greater than a threshold value; The weighting value when the driver's line of sight is directed toward a region different from the first region is greater than the weighting value when the driver's line of sight is directed toward the first region. The driver condition determination device according to claim 1 .

3. The windshield has the first region, a third region which is an edge of the windshield, and a second region between the first region and the third region, The weighting value when the driver's line of sight is directed toward the third area is greater than the weighting value when the driver's line of sight is directed toward the second area. The driver condition determination device according to claim 2 .

4. When the driver's line of sight is directed toward a door mirror of the vehicle, the determination unit weights the elapsed time with the same value as when the driver's line of sight is directed toward the third area. The driver condition determination device according to claim 3.

5. The weighting value when the driver's line of sight is not directed toward the windshield is greater than the weighting value when the driver's line of sight is directed toward the windshield. The driver condition determination device according to any one of claims 1 to 3.

Images