Opened / closed eye estimation device, method for estimating opened / closed eye, and program

The eye opening/closing estimation device uses facial feature detection and eyelid movement analysis to correct misclassifications, ensuring accurate eye state determination even in challenging conditions.

JP2024162514APending Publication Date: 2024-11-21TOYOTA JIDOSHA KK +1

Patent Information

Application Number
JP2023078078
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing eye opening/closing estimation systems struggle to accurately determine the eye state when the subject is wearing eyelash extensions or in a downcast position, leading to misclassification between open and closed eyes.

Method used

The system employs an eye opening/closing estimation device that includes a facial feature point detection unit, a reliability calculation unit, an eyelid movement detection unit, and a correction unit to refine eye state determinations by considering eyelid movement and face orientation, using techniques such as differential image processing and machine learning to correct erroneous results.

Benefits of technology

The system effectively distinguishes between open and closed eyes even when the subject is wearing eyelash extensions or in a downcast position, ensuring accurate eye state estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162514000001_ABST
    Figure 2024162514000001_ABST
Patent Text Reader

Abstract

To properly determine whether an opened / closed eye estimation target person is in a state of an opened eye even when the person is drooping.SOLUTION: An opened / closed eye estimation device 10 includes: an opened / closed eye determination unit 234 for determining whether an opened / closed eye estimation target person is in a state of an opened eye on the basis of an image of the region of the eyes of the opened / closed eye estimation target person cut off from an image including the face of the opened / closed eye estimation target person; a reliability degree calculation unit 235 for calculating the reliability of the result of determination by the opened / closed eye determination unit 234; an eyelid movement detection unit 236 for detecting a movement of the eyelids of the opened / closed eye estimation target person when the reliability is not larger than a threshold value; and a correction unit 237 for correcting the result of the determination on the basis of the result of detection by the eyelid movement detection unit 236.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an open / closed eye estimation device, an open / closed eye estimation method, and a program. [Background technology]

[0002] Patent Document 1 describes a drowsiness level estimation device in which an eye-closed / opened determination unit detects an eye-open state and an eye-closed state based on the degree of eye-opening. In the technology described in Patent Document 1, an image sensor captures an image of the driver's face, a facial feature point detection unit detects feature points of the driver's face (measurement subject) from the driver's facial image, an eye-opening degree calculation unit projects multiple curves onto an edge image based on the feature points, detects the upper and lower eyelid positions from the strength of the curved edges, calculates the degree of eye-opening of the measurement subject from the difference in Y coordinate between the upper and lower eyelid positions, and a closed / opened eye determination unit performs threshold processing of the degree of eye-opening to determine whether the eyes are open or closed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-229741 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the open / closed eyes presumption subject has their eyes open and their eyes downcast (including a state in which the open / closed eyes presumption subject is facing forward and wearing eyelash extensions, and part of the eyes of the open / closed eyes presumption subject is not visible from the driver monitor camera), the open / closed eyes presumption subject appears to have their eyes closed. Therefore, it is difficult to distinguish whether the open / closed eyes presumption subject has their eyes open and their eyes downcast, or whether the open / closed eyes presumption subject has their eyes closed.

[0005] On the other hand, Patent Document 1 does not describe the processing required to appropriately determine that an open / closed eye estimation subject is in an open-eye state when the open-eye estimation subject has his / her eyes open and his / her eyes downcast. Therefore, the technology described in Patent Document 1 may not be able to appropriately determine that an open / closed eye estimation subject is in an open-eye state when the open-eye estimation subject has his / her eyes open and his / her eyes downcast.

[0006] In view of the above-mentioned points, the present invention aims to provide an open / closed eye estimation device, an open / closed eye estimation method, and a program that can appropriately determine whether a subject of open / closed eye estimation has their eyes open even when the subject has their eyes closed. [Means for solving the problem]

[0007] (1) One aspect of the present invention is an open / closed eye estimation device that includes an open / closed eye determination unit that determines whether the open / closed eye estimation subject has their eyes open based on an image of the eye area of ​​the open / closed eye estimation subject cut out from an image including the face of the subject, a reliability calculation unit that calculates the reliability of the determination result of the open / closed eye determination unit, an eyelid movement detection unit that detects eyelid movement of the open / closed eye estimation subject if the reliability is below a threshold value, and a correction unit that corrects the determination result based on the detection result of the eyelid movement detection unit.

[0008] (2) In the open / closed eye estimation device of (1), when the open / closed eye determination unit determines that the open / closed eye estimation subject is in a closed eye state and the eyelid movement detection unit detects eyelid movement of the open / closed eye estimation subject, the correction unit may correct the determination result indicating that the open / closed eye estimation subject is in a closed eye state to the determination result indicating that the open / closed eye estimation subject is in an open eye state.

[0009] (3) In the open / closed eye estimation device of (1) or (2), the correction unit may, when the open / closed eye determination unit determines that the open / closed eye estimation subject is in an open-eyed state and no eyelid movement of the open / closed eye estimation subject is detected by the eyelid movement detection unit, correct the determination result indicating that the open / closed eye estimation subject is in an open-eyed state to the determination result indicating that the open / closed eye estimation subject is in a closed-eyed state.

[0010] (4) Any of the open / closed eye estimation devices of (1) to (3) includes a facial feature point detection unit that detects facial feature points of the open / closed eye estimation subject from an image including the face of the open / closed eye estimation subject, and the reliability calculation unit calculates the facial direction of the open / closed eye estimation subject by using the detection result of the facial feature point detection unit, and the reliability calculation unit may calculate a lower value for the reliability as the facial direction of the open / closed eye estimation subject is more downward.

[0011] (5) In any of the open / closed eye estimation devices of (1) to (4), the reliability calculation unit calculates the reliability by using a pre-trained machine learning model, the machine learning model is trained using a predetermined image, the reliability calculation unit calculates a similarity of an image to be judged by the open / closed eye determination unit to the predetermined image, and the higher the similarity of the image to be judged by the open / closed eye determination unit to the predetermined image, the lower the reliability value calculated.

[0012] (6) In any of the open / closed eye estimation devices (1) to (5), the eyelid movement detection unit may detect the eyelid movement of the open / closed eye estimation subject using a differential image between a first frame image that constitutes part of a moving image including the face of the open / closed eye estimation subject, and a second frame image that is different from the first frame image.

[0013] (7) In any of the open / closed eye estimation devices (1) to (6), the eyelid movement detection unit may calculate an optical flow of an image including a face of the open / closed eye estimation subject, and detect eyelid movement of the open / closed eye estimation subject using the optical flow.

[0014] (8) In any of the open / closed eye estimation devices (1) to (7), the eyelid movement detection unit may detect eyelid movement of the open / closed eye estimation subject using a time series model from a moving image including the face of the open / closed eye estimation subject as time series data.

[0015] (9) One aspect of the present invention is an open / closed eye estimation method including an open / closed eye determination step in which an open / closed eye estimation device determines whether the open / closed eye estimation subject has their eyes open based on an image of the eye area of ​​the open / closed eye estimation subject cut out from an image including the face of the open / closed eye estimation subject, a reliability calculation step in which a reliability of the determination result of the open / closed eye determination step is calculated, an eyelid movement detection step in which an eyelid movement of the open / closed eye estimation subject is detected if the reliability is below a threshold value, and a correction step in which the determination result is corrected based on the detection result of the eyelid movement detection step.

[0016] (10) One aspect of the present invention is a program for causing a processor to execute an open / closed eye determination step of determining whether the open / closed eye estimation subject has their eyes open based on an image of the eye area of ​​the open / closed eye estimation subject cut out from an image including the face of the open / closed eye estimation subject, a reliability calculation step of calculating the reliability of the determination result of the open / closed eye determination step, an eyelid movement detection step of detecting eyelid movement of the open / closed eye estimation subject if the reliability is below a threshold value, and a correction step of correcting the determination result based on the detection result of the eyelid movement detection step. Effect of the Invention

[0017] According to the present invention, even when the open / closed eye estimation subject has his / her eyes closed, it is possible to appropriately determine whether the open / closed eye estimation subject has his / her eyes open. [Brief description of the drawings]

[0018] [Figure 1] 1 is a diagram showing an example of a schematic configuration of a vehicle 1 to which an open / closed eye estimation device 10 of a first embodiment is applied. [Diagram 2] 2 is a diagram illustrating an example of a specific configuration of the open / closed eye estimation device 10 illustrated in FIG. [Diagram 3] FIG. 2 is a diagram showing an example of the arrangement of the driver monitor camera 7 shown in FIG. [Figure 4] 4 is a flowchart for explaining an example of processing executed by a processor 23 of the open / closed eye estimation device 10 of the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of a schematic configuration of a terminal device 51 to which an open / closed eye estimation device 60 according to a fifth embodiment is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of an open / closed eye estimation device, an open / closed eye estimation method, and a program of the present invention will be described with reference to the drawings.

[0020] First Embodiment Fig. 1 is a diagram showing an example of a schematic configuration of a vehicle 1 to which an open / closed eye estimation device 10 according to a first embodiment is applied. Fig. 2 is a diagram showing an example of a specific configuration of the open / closed eye estimation device 10 shown in Fig. 1.

[0021] In the example shown in FIG. 1 and FIG. 2, a vehicle 1 includes an exterior camera 2, a radar 3, a LiDAR (Light Detection And Ranging) 4, and an autonomous driving ECU (Electronic Control Unit) 12. The exterior camera 2 captures images showing vehicles around the vehicle 1 and the road environment around the vehicle 1 (e.g., road structure, rules, etc.), generates image data showing the vehicles around the vehicle 1 and the road environment around the vehicle 1, and transmits the image data to the autonomous driving ECU 12. The radar 3 is, for example, a millimeter wave radar, and detects the relative positions and relative speeds of the vehicles around the vehicle 1 and the road structure around the vehicle 1, and transmits the detection results to the autonomous driving ECU 12. The LiDAR 4 detects the relative positions and relative speeds of the vehicles around the vehicle 1 and the road structure around the vehicle 1, and transmits the detection results to the autonomous driving ECU 12.

[0022] The vehicle 1 also includes a GPS (Global Positioning System) unit 5 and a map information unit 6. The GPS unit 5 acquires location information indicating the current location of the vehicle 1 based on a GPS signal, and transmits the location information of the vehicle 1 to the autonomous driving ECU 12. The map information unit 6 is formed in a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) mounted on the vehicle 1. The map information held by the map information unit 6 includes various information such as road structure (road position, road shape, lane structure, etc.), rules, etc. Furthermore, the vehicle 1 is equipped with a driver monitor camera 7, an HMI (Human Machine Interface) 8, and an open / closed eye estimation device 10. The driver monitor camera 7 captures an image (more specifically, a moving image) including the face of the driver of the vehicle 1 as a subject of open / closed eye estimation.

[0023] FIG. 3 is a diagram showing an example of the arrangement of the driver monitor camera 7 shown in FIG. In the example shown in FIGS. 1 to 3, the driver monitor camera 7 is disposed above the steering column (not shown) of the vehicle 1 so as to be able to capture an image of the face and part of the upper body of the driver of the vehicle 1. In other examples, the driver monitor camera 7 may be disposed in a center cluster of the vehicle 1, or the driver monitor camera 7 may be disposed in a rearview mirror, a meter panel, a meter hood, or the like of the vehicle 1. In these examples, the driver monitor camera 7 can also capture an image of the face and part of the upper body of the driver of the vehicle 1.

[0024] 1 and 2, the HMI 8 is an interface for inputting and outputting information between the open / closed eye estimation device 10 and the driver. The HMI 8 is an information providing device for providing various information to the driver, specifically, a display for displaying characters, images, etc., a speaker for outputting sound, etc. The HMI 8 also includes operation buttons, a touch panel, etc. for accepting input operations by the driver. The exterior camera 2, radar 3, LiDAR 4, GPS unit 5, map information unit 6, driver monitor camera 7, HMI 8, open / closed eye estimation device 10 and autonomous driving ECU 12 are connected via an in-vehicle network 13.

[0025] The vehicle 1 also includes a steering actuator 14, a braking actuator 15, and a drive actuator 16. The steering actuator 14 includes, for example, a power steering system, a steer-by-wire steering system, a rear-wheel steering system, etc. The braking actuator 15 has a function of decelerating the vehicle 1. The braking actuator 15 includes, for example, a hydraulic brake, a regenerative brake, etc. The drive actuator 16 has a function of accelerating the vehicle 1. The drive actuator 16 includes, for example, an engine, an EV (electric vehicle) system, a hybrid system, a fuel cell system, etc.

[0026] 1 and 2, the autonomous driving ECU 12 can control the vehicle 1 at a driving control level of Level 3 defined by the Society of Automotive Engineers (SAE), i.e., a driving control level that does not require the driver to operate the steering actuator 14, the braking actuator 15, and the drive actuator 16, and does not require monitoring of the surroundings of the vehicle 1. The autonomous driving ECU 12 also has a function of controlling the vehicle 1 at a driving control level where the driver is involved in driving the vehicle 1, for example, at driving control levels of Levels 0 to 2 defined by the SAE. In the example shown in Figures 1 and 2, vehicle 1 is an autonomous vehicle and is equipped with an autonomous driving ECU 12, but in other examples, vehicle 1 may be a manually driven vehicle and be equipped with an ECU that does not have an autonomous driving control function (e.g., a driving assistance ECU, etc.) instead of autonomous driving ECU 12.

[0027] Level 3 autonomous driving as defined by the SAE does not tolerate drivers falling asleep or other conditions that would prevent them from smoothly responding to a request to take over driving. Therefore, in the example shown in Figures 1 and 2, the open / closed eye estimation device 10 uses the driver monitor camera 7 to estimate whether the driver has his / her eyes open in order to monitor whether the driver is dozing off.

[0028] When the driver monitor camera 7 is disposed on top of the steering column of the vehicle 1 (example shown in FIG. 3) or when the driver monitor camera 7 is disposed in the center cluster of the vehicle 1, when the driver of the vehicle 1 as the open / closed eyes estimation subject has his / her eyes open and his / her eyes downcast (including a state in which the open / closed eyes estimation subject facing forward is wearing eyelash extensions and part of the eyes of the open / closed eyes estimation subject is not visible from the driver monitor camera 7), the open / closed eyes estimation subject appears from the position of the driver monitor camera 7 as if he / she had his / her eyes closed. Therefore, it is difficult to distinguish whether the open / closed eyes estimation subject has his / her eyes open and his / her eyes downcast or whether the open / closed eyes estimation subject has his / her eyes closed. In consideration of this point, the open / closed eye estimation device 10 of the first embodiment has measures to be described below.

[0029] In the example shown in Figures 1 and 2, an open / closed eye estimation device 10 estimates whether the driver of a vehicle 1, who is a subject of open / closed eye estimation, has his / her eyes open. The open / closed eye estimation device 10 is configured by a microcomputer having a communication interface (I / F) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the open / closed eye estimation device 10 to the in-vehicle network 13. The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores programs and various data used in the processing executed by the processor 23. The processor 23 has a function of determining whether the driver of the vehicle 1 has his / her eyes open. 1 and 2, the open / closed eye estimation device 10 includes one processor 23, but in other examples, the open / closed eye estimation device 10 may include multiple processors. Also, in the example shown in Fig. 1 and 2, the open / closed eye estimation device 10 is configured by one ECU (e.g., a driver monitor ECU), but in other examples, the open / closed eye estimation device 10 may be configured by multiple ECUs (e.g., a driver monitor ECU, an alarm ECU, etc.).

[0030] In the example shown in Figures 1 and 2, the processor 23 includes an acquisition unit 231, a facial feature point detection unit 232, an eye area image cutting unit 233, an open / closed eye determination unit 234, a reliability calculation unit 235, an eyelid movement detection unit 236, and a correction unit 237. The acquisition unit 231 acquires an image including the face of the driver of the vehicle 1 as a subject of open / closed eyes estimation captured by the driver monitor camera 7.

[0031] The facial feature point detection unit 232 extracts the outer and inner corners of the eyes as facial feature points of the driver of vehicle 1 from an image including the face of the driver of vehicle 1 acquired by the acquisition unit 231, for example by using a publicly known facial feature point detection (extraction) technique.

[0032] The eye area image cutting-out unit 233 cuts out an image of the eye area from an image including the face of the driver of vehicle 1 using the facial feature points of the driver of vehicle 1 extracted by the facial feature point detection unit 232 (i.e., the outer corner and inner corner of the eye as the detection result of the facial feature point detection unit 232), for example by using a publicly known eye area image cutting-out technique.

[0033] The open / closed eyes determination unit 234 determines whether the driver of vehicle 1 has his / her eyes open based on the image of the eye area of ​​the driver of vehicle 1 cut out by the eye area image cutout unit 233, for example by using a publicly known open / closed eyes determination technique.

[0034] The reliability calculation unit 235 calculates the reliability of the judgment result of the open / closed eyes judgment unit 234. Specifically, the reliability calculation unit 235 calculates the facial orientation of the driver of the vehicle 1 using the detection result of the facial feature point detection unit 232 (i.e., facial feature points such as the outer corners, inner corners, and corners of the mouth of the driver of the vehicle 1) by using, for example, a known facial orientation judgment technique. Furthermore, the reliability calculation unit 235 calculates a lower value as the reliability of the judgment result of the open / closed eyes judgment unit 234, the greater the deviation of the facial orientation of the driver of the vehicle 1 from the front of the face. Specifically, the reliability calculation unit 235 calculates a lower value as the reliability of the judgment result of the open / closed eyes judgment unit 234, the greater the downward facing face orientation of the driver of the vehicle 1. For example, when the driver of vehicle 1 has his / her eyes downcast, the reliability calculation unit 235 calculates the facial direction of the driver of vehicle 1 as “downward” and calculates a low value as the reliability of the judgment result of the open / closed eyes judgment unit 234.

[0035] When the reliability calculated by the reliability calculation unit 235 is equal to or less than a predetermined threshold, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 (specifically, the presence or absence of eye movement such as blinking). In detail, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 by using a differential image between a first frame image (e.g., a current image) constituting a part of a moving image including the face of the driver of the vehicle 1 captured by the driver monitor camera 7, and a second frame image (e.g., a past image) different from the first frame image, for example, by using a known differential image processing technique. In other words, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 by emphasizing a part with a large change between the first frame image and the second frame image.

[0036] In the example shown in Figures 1 and 2, the eyelid movement detection unit 236 does not use the detection results of the facial feature point detection unit 232 (facial feature points of the driver of vehicle 1 detected by the facial feature point detection unit 232) when detecting the eyelid movement of the driver of vehicle 1, but in other examples, the eyelid movement detection unit 236 may use the detection results of the facial feature point detection unit 232 when detecting the eyelid movement of the driver of vehicle 1.

[0037] In the example shown in FIG. 1 and FIG. 2, the correction unit 237 corrects the determination result of the eye open / closed eye determination unit 234 based on the detection result of the eyelid movement detection unit 236. Specifically, when the driver of vehicle 1 has his eyes open (a state in which the eyelid movement of the driver of vehicle 1 is detected by the eyelid movement detection unit 236) and his eyes are downcast, if the open / closed eye judgment unit 234 erroneously judges that the driver of vehicle 1 has his eyes closed, the reliability calculation unit 235 calculates the facial direction of the driver of vehicle 1 to be "downward" and calculates a low value as the reliability of the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes closed, and the correction unit 237 corrects the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes closed to the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes open. In addition, when the driver of vehicle 1 has his eyes closed (a state in which the eyelid movement of the driver of vehicle 1 is not detected by the eyelid movement detection unit 236) and his eyes are downcast, if the open / closed eye judgment unit 234 erroneously judges that the driver of vehicle 1 has his eyes open, the reliability calculation unit 235 calculates the facial direction of the driver of vehicle 1 to be "downward" and calculates a low value as the reliability of the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes open, and the correction unit 237 corrects the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes open to the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes closed.

[0038] In the example shown in Figures 1 and 2, the correction unit 237 performs a correction to correct the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes open to a judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes closed, but in other examples, the correction unit 237 does not need to perform a correction to correct the judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes open to a judgment result of the open / closed eye judgment unit 234 indicating that the driver of vehicle 1 has his eyes closed.

[0039] FIG. 4 is a flowchart for explaining an example of processing executed by the processor 23 of the open / closed eye estimation device 10 of the first embodiment. In the example shown in FIG. 4, in step S10, the acquisition unit 231 acquires an image including the face of the driver of the vehicle 1 as a subject of open / closed eyes estimation captured by the driver monitor camera 7. In step S11, the facial feature point detection unit 232 extracts the outer corners and inner corners of the eyes as facial feature points of the driver of the vehicle 1 from the image including the face of the driver of the vehicle 1 acquired in step S10. In step S12, the eye region image cutout unit 233 cuts out an image of the eye region from the image including the face of the driver of the vehicle 1, using the facial feature points (the outer corner and inner corner of the eye) of the driver of the vehicle 1 extracted in step S11.

[0040] In step S13, the eye open / closed determination unit 234 determines whether the driver of the vehicle 1 has his / her eyes open based on the image of the eye area of ​​the driver of the vehicle 1 cut out in step S12. In step S14, the reliability calculation unit 235 calculates the reliability of the determination result as to whether or not the driver of the vehicle 1 has his eyes open in step S13. In step S15, for example, the reliability calculation unit 235 determines whether the reliability calculated in step S14 is equal to or less than a threshold value. If YES, the process proceeds to step S16, and if NO, the routine shown in FIG. 4 is terminated. In step S16, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1.

[0041] In step S17, the correction unit 237 corrects the determination result of step S13 as to whether or not the driver of the vehicle 1 has his / her eyes open, based on the detection result of the eyelid movement of the driver of the vehicle 1 in step S16. In detail, if it is determined in step S13 that the driver of vehicle 1 has his eyes closed, and if eyelid movement of the driver of vehicle 1 is detected in step S16, the correction unit 237 corrects the determination result in step S13 indicating that the driver of vehicle 1 has his eyes closed to a determination result indicating that the driver of vehicle 1 has his eyes open, based on the detection result in step S16 indicating that eyelid movement of the driver of vehicle 1 has been detected. On the other hand, if it is determined in step S13 that the driver of vehicle 1 has his eyes closed, and no eyelid movement of the driver of vehicle 1 is detected in step S16, the correction unit 237 maintains the determination result in step S13 indicating that the driver of vehicle 1 has his eyes closed without correcting it, based on the detection result in step S16 indicating that no eyelid movement of the driver of vehicle 1 was detected. In addition, if it is determined in step S13 that the driver of vehicle 1 has his eyes open, and no eyelid movement of the driver of vehicle 1 is detected in step S16, the correction unit 237 corrects the determination result in step S13 indicating that the driver of vehicle 1 has his eyes open to a determination result indicating that the driver of vehicle 1 has his eyes closed, based on the detection result in step S16 indicating that no eyelid movement of the driver of vehicle 1 was detected. On the other hand, if it is determined in step S13 that the driver of vehicle 1 has his eyes open, and eyelid movement of the driver of vehicle 1 is detected in step S16, the correction unit 237 maintains the determination result in step S13 indicating that the driver of vehicle 1 has his eyes open without correcting it, based on the detection result in step S16 indicating that eyelid movement of the driver of vehicle 1 has been detected.

[0042] In other words, in the first embodiment of the open / closed eye estimation device 10, the idea is adopted that when the driver of the vehicle 1 has his / her eyes open, eyelid movements of the driver of the vehicle 1, such as blinking, can be detected, but when the driver of the vehicle 1 has his / her eyes closed, eyelid movements of the driver of the vehicle 1 cannot be detected (the driver of the vehicle 1 does not blink, etc.).

[0043] As described above, in the first embodiment of the open / closed eye estimation device 10, when the driver of vehicle 1 has his / her eyes downcast (when the driver of vehicle 1 looks down), which is a time when it is easy to erroneously determine whether the driver of vehicle 1 has his / her eyes open (when the driver of vehicle 1 appears to have his / her eyes closed even though the driver of vehicle 1 actually has his / her eyes open), the determination result of whether the driver of vehicle 1 has his / her eyes open is corrected based on the presence or absence of eyelid movement of the driver of vehicle 1. Therefore, the open / closed eye estimation device 10 of the first embodiment can appropriately determine whether or not the driver of the vehicle 1 has his / her eyes open even when the driver of the vehicle 1 has his / her eyes downcast. 4, when the reliability is equal to or less than a threshold, the determination result of whether or not the driver of vehicle 1 has his / her eyes open is corrected based on the detection result of the eyelid movement of the driver of vehicle 1. In another example, when it is determined that the driver of vehicle 1 has his / her eyes closed, the presence or absence of eyelid movement of the driver of vehicle 1 is detected, and when the eyelid movement of the driver of vehicle 1 is detected, the determination result that the driver of vehicle 1 has his / her eyes closed may be corrected to the determination result that the driver of vehicle 1 has his / her eyes open (that is, the determination result may be corrected based on the detection result of the eyelid movement without using the reliability).

[0044] <Second embodiment> The vehicle 1 to which the open / closed eye estimation device 10 of the second embodiment is applied is configured similarly to the vehicle 1 to which the open / closed eye estimation device 10 of the first embodiment shown in Fig. 1 is applied, except for the points described below. Moreover, the open / closed eye estimation device 10 of the second embodiment is configured similarly to the open / closed eye estimation device 10 of the first embodiment shown in Fig. 2, except for the points described below.

[0045] As described above, in the first embodiment of the open / closed eye estimation device 10, the reliability calculation unit 235 calculates the facial direction of the driver of vehicle 1 by using the detection results of the facial feature point detection unit 232 (i.e., facial feature points of the driver of vehicle 1, such as the outer corners of the eyes, the inner corners of the eyes, the corners of the mouth, etc.), and the greater the deviation of the facial direction of the driver of vehicle 1 from the front of his / her face (for example, the more downward the facial direction of the driver of vehicle 1 is), the lower the reliability value of the judgment result of the open / closed eye judgment unit 234 is calculated. On the other hand, in the open / closed eye estimation device 10 of the second embodiment, the reliability calculation unit 235 calculates the reliability of the determination result of the open / closed eye determination unit 234 by using a machine learning model that has been trained in advance.

[0046] In the second embodiment of the open / closed eye estimation device 10, a machine learning model is trained using a predetermined image (for example, an image of the eye area of ​​the driver of vehicle 1 when the driver of vehicle 1 is in a downcast state, an image of the eye area of ​​the driver of vehicle 1 when the driver of vehicle 1 is looking aside, etc., which tends to reduce the judgment accuracy of the open / closed eye judgment unit 234). Moreover, the reliability calculation unit 235 uses a trained machine learning model and, for example, a known similarity calculation technique to calculate the similarity of the image to be determined by the open / closed eye determination unit 234 to the above-mentioned predetermined image. Furthermore, the reliability calculation unit 235 calculates a lower value as the reliability of the determination result of the open / closed eye determination unit 234, as the similarity of the image to be determined by the open / closed eye determination unit 234 to the predetermined image is higher. In the second embodiment of the open / closed eye estimation device 10, when the state of the driver of the vehicle 1 is one in which the judgment accuracy of the open / closed eye judgment unit 234 tends to be low (i.e., when the scene is one in which the open / closed eye judgment unit 234 has difficulty judging whether the driver of the vehicle 1 has his / her eyes open or not), the reliability calculation unit 235 calculates a low value as the reliability of the judgment result of the open / closed eye judgment unit 234, and the correction unit 237 corrects the judgment result of the open / closed eye judgment unit 234 based on the detection result of the eyelid movement detection unit 236. Therefore, in the second embodiment of the open / closed eye estimation device 10, even when the judgment accuracy of the open / closed eye judgment unit 234 tends to be low, such as when the driver of the vehicle 1 has his / her eyes down or is looking aside, the open / closed eye estimation device 10 can appropriately determine whether the driver of the vehicle 1 has his / her eyes open or not.

[0047] <Third embodiment> The vehicle 1 to which the open / closed eye estimation device 10 of the third embodiment is applied is configured similarly to the vehicle 1 to which the open / closed eye estimation device 10 of the first embodiment shown in Fig. 1 is applied, except for the points described below. Moreover, the open / closed eye estimation device 10 of the third embodiment is configured similarly to the open / closed eye estimation device 10 of the first embodiment shown in Fig. 2, except for the points described below.

[0048] As described above, in the open / closed eye estimation device 10 of the first embodiment, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 by using the difference image between the first frame image and the second frame image. On the other hand, in the open / closed eye estimation device 10 of the third embodiment, the eyelid movement detection unit 236 calculates the optical flow of an image including the face of the driver of the vehicle 1, for example, by using a known video analysis technique. Furthermore, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 using the optical flow. As with the open / closed eye estimation device 10 of the first embodiment, the open / closed eye estimation device 10 of the third embodiment can appropriately determine whether the driver of the vehicle 1 has his / her eyes open, even when the driver of the vehicle 1 has his / her eyes lowered.

[0049] <Fourth embodiment> The vehicle 1 to which the open / closed eye estimation device 10 of the fourth embodiment is applied is configured similarly to the vehicle 1 to which the open / closed eye estimation device 10 of the first embodiment shown in Fig. 1 is applied, except for the points described below. Moreover, the open / closed eye estimation device 10 of the fourth embodiment is configured similarly to the open / closed eye estimation device 10 of the first embodiment shown in Fig. 2, except for the points described below.

[0050] In the open / closed eye estimation device 10 of the fourth embodiment, the eyelid movement detection unit 236 uses a time series model such as LSTM (Long Short-Term Memory), which is a known image processing technique. In detail, the eyelid movement detection unit 236 detects the eyelid movement of the driver of the vehicle 1 from a moving image including the face of the driver of the vehicle 1 as time series data, using the time series model. As with the open / closed eye estimation device 10 of the first embodiment, the open / closed eye estimation device 10 of the fourth embodiment can appropriately determine whether the driver of the vehicle 1 has his / her eyes open, even when the driver of the vehicle 1 has his / her eyes lowered.

[0051] <Fifth embodiment> FIG. 5 is a diagram showing an example of a schematic configuration of a terminal device 51 to which an open / closed eye estimation device 60 according to the fifth embodiment is applied. 5, the terminal device 51 is used, for example, for online classes, online conferences, etc. The terminal device 51 includes a camera 57, an HMI 58, and an open / closed eye estimation device 60. The camera 57 captures an image (more specifically, a moving image) including the face of the user of the terminal device 51 as a subject of open / closed eye estimation. The HMI 58 is an interface for inputting and outputting information between the open / closed eye estimation device 60 and the like and the user of the terminal device 51. The HMI 58 is an information providing device for providing various information to the user of the terminal device 51, specifically, it is equipped with a display for displaying characters, images, etc., a speaker for outputting sound, etc. In addition, the HMI 58 is equipped with a keyboard, a mouse, a touch panel, etc. for accepting input operations by the user of the terminal device 51. The camera 57 , the HMI 58 , and the open / closed eye estimation device 60 are connected via a signal line 63 .

[0052] The open / closed eye estimation device 60 estimates whether the user of the terminal device 51 has his / her eyes open. The open / closed eye estimation device 60 is configured by a computer including an input interface 74, an output interface 75, a communication interface 71, a memory 72, and a processor 73. The input interface 74, the output interface 75, the communication interface 71, the memory 72, and the processor 73 are connected via a signal line 76. The memory 72 stores programs and various data used in the processing executed by the processor 73. The processor 73 has a function of determining whether the user of the terminal device 51 has his / her eyes open. The processor 73 includes an acquisition unit 731 having the same function as the acquisition unit 231 (see Figure 2), a facial feature point detection unit 732 having the same function as the facial feature point detection unit 232 (see Figure 2), an eye area image cut-out unit 733 having the same function as the eye area image cut-out unit 233 (see Figure 2), an open / closed eye determination unit 734 having the same function as the open / closed eye determination unit 234 (see Figure 2), a reliability calculation unit 735 having the same function as the reliability calculation unit 235 (see Figure 2), an eyelid movement detection unit 736 having the same function as the eyelid movement detection unit 236 (see Figure 2), and a correction unit 737 having the same function as the correction unit 237 (see Figure 2). The acquisition unit 731 acquires an image including the face of the user of the terminal device 51 as the open / closed eye estimation subject captured by the camera 57.

[0053] The facial feature point detection unit 732 extracts the outer and inner corners of the eyes as facial feature points of the user of the terminal device 51 from the image including the face of the user of the terminal device 51 acquired by the acquisition unit 731. The eye region image cutout section 733 cuts out an image of the eye region from an image including the face of the user of the terminal device 51 using the facial feature points of the user of the terminal device 51 extracted by the facial feature point detection section 732 . The open / closed eyes determination section 734 determines whether or not the user of the terminal device 51 has his / her eyes open based on the image of the eye region of the user of the terminal device 51 cut out by the eye region image cutout section 733 . The reliability calculation unit 735 calculates the reliability of the determination result of the open / closed eyes determination unit 734. Specifically, the reliability calculation unit 735 calculates the face orientation of the user of the terminal device 51 using the detection result of the face feature point detection unit 732 (i.e., face feature points such as the outer corners of the eyes, the inner corners of the eyes, and the corners of the mouth of the user of the terminal device 51). Furthermore, the reliability calculation unit 735 calculates a lower value as the reliability of the determination result of the open / closed eyes determination unit 734, the greater the deviation of the face orientation of the user of the terminal device 51 from the front of the face. Specifically, the reliability calculation unit 735 calculates a lower value as the reliability of the determination result of the open / closed eyes determination unit 734, the greater the downward facing face orientation of the user of the terminal device 51. For example, when the user of the terminal device 51 has their eyes downcast, the reliability calculation unit 735 calculates the facial direction of the user of the terminal device 51 as “downward” and calculates a low value as the reliability of the judgment result of the open / closed eyes judgment unit 734.

[0054] When the reliability calculated by the reliability calculation unit 735 is equal to or less than a predetermined threshold, the eyelid movement detection unit 736 detects the eyelid movement of the user of the terminal device 51 (more specifically, the presence or absence of eye movement such as blinking). In detail, the eyelid movement detection unit 736 detects the eyelid movement of the user of the terminal device 51 by using a difference image between a first frame image (e.g., a current image) constituting a part of a moving image including the face of the user of the terminal device 51 captured by the camera 57, and a second frame image (e.g., a past image) different from the first frame image. In other words, the eyelid movement detection unit 736 detects the eyelid movement of the user of the terminal device 51 by emphasizing a part with a large change between the first frame image and the second frame image.

[0055] The correction section 737 corrects the determination result of the eye open / closed determination section 734 based on the detection result of the eyelid movement detection section 736 . Specifically, when the user of terminal device 51 has his / her eyes open (a state in which the eyelid movement of the user of terminal device 51 is detected by the eyelid movement detection unit 736) and his / her eyes are downcast, if the open / closed eye judgment unit 734 erroneously judges that the user of terminal device 51 has his / her eyes closed, the reliability calculation unit 735 calculates the facial direction of the user of terminal device 51 to be "downward" and calculates a low value as the reliability of the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes closed, and the correction unit 737 corrects the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes closed to the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes open. In addition, when the user of terminal device 51 has his / her eyes closed (a state in which the eyelid movement of the user of terminal device 51 is not detected by the eyelid movement detection unit 736) and his / her eyes are downcast, if the open / closed eye judgment unit 734 erroneously judges that the user of terminal device 51 has his / her eyes open, the reliability calculation unit 735 calculates the facial direction of the user of terminal device 51 to be "downward" and calculates a low value as the reliability of the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes open, and the correction unit 737 corrects the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes open to the judgment result of the open / closed eye judgment unit 734 indicating that the user of terminal device 51 has his / her eyes closed.

[0056] As described above, the open / closed eye estimation device, the open / closed eye estimation method, and the program of the present invention have been described with reference to the drawings, but the open / closed eye estimation device, the open / closed eye estimation method, and the program of the present invention are not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit of the present invention. The configurations of the examples of the above-described embodiments can be combined as appropriate. In each of the above-described embodiments, the processing performed in the open / closed eye estimation device 10, 60 has been described as software processing performed by executing a program, but the processing performed in the open / closed eye estimation device 10, 60 may be processing performed by hardware. Alternatively, the processing performed in the open / closed eye estimation device 10, 60 may be processing that combines both software and hardware. In addition, the program stored in the memory 22, 72 of the open / closed eye estimation device 10, 60 (a program that realizes the functions of the processor 23, 73 of the open / closed eye estimation device 10, 60) may be recorded in a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, etc., and provided, distributed, etc. [Explanation of symbols]

[0057] 1 vehicle 2. Exterior camera 3. Radar 4. LiDAR 5 GPS unit 6 Map Information Unit 7 Driver monitor camera 8. Human Machine Interface 10 Eye opening / closing estimation device 12 Autonomous Driving ECU 14 Steering actuator 15 Braking Actuator 16 Drive Actuator 21 Communication Interface 22 Memory 23 Processors 231 Acquisition Department 232 Facial Feature Point Detection Unit 233 Eye Region Image Extraction 234 Open / close eye determination unit 235 Reliability Calculation Unit 236 Eyelid movement detector 237 Correction Section

Claims

1. an open / closed eye determination unit that determines whether the open / closed eye estimation subject has their eyes open based on an image of an eye region of the open / closed eye estimation subject cut out from an image including a face of the open / closed eye estimation subject; A reliability calculation unit that calculates the reliability of the determination result of the open / closed eye determination unit; an eyelid movement detection unit that detects an eyelid movement of the open / closed eye estimation subject when the reliability is equal to or less than a threshold; A correction unit that corrects the determination result based on the detection result of the eyelid movement detection unit.

2. The correction unit is When the open / closed eye judgment unit judges that the open / closed eye estimation subject is in an eye-closed state, and the eyelid movement detection unit detects an eyelid movement of the open / closed eye estimation subject, The open / closed eye estimation device according to claim 1 , wherein the determination result indicating that the open / closed eye estimation subject is in an eye-closed state is corrected to the determination result indicating that the open / closed eye estimation subject is in an eye-open state.

3. The correction unit is When the open / closed eye judgment unit judges that the open / closed eye estimation subject is in an open eye state, and the eyelid movement detection unit does not detect an eyelid movement of the open / closed eye estimation subject, The open / closed eye estimation device according to claim 1 or 2, wherein the judgment result indicating that the open / closed eye estimation subject is in an open-eye state is corrected to the judgment result indicating that the open / closed eye estimation subject is in a closed-eye state.

4. a face feature point detection unit that detects face feature points of the open / closed eyes presumption subject from an image including a face of the open / closed eyes presumption subject, The reliability calculation unit is Calculating a face direction of the subject person whose eyes are open or closed by using a detection result of the face feature point detection unit; The open / closed eye estimation device according to claim 1 , wherein the reliability is calculated to be lower as the face orientation of the open / closed eye estimation subject is more downward.

5. The reliability calculation unit calculates the reliability by using a pre-trained machine learning model, The learning of the machine learning model is performed using a predetermined image, The reliability calculation unit is Calculating a similarity between the predetermined image and an image to be determined by the open / closed eye determination unit; The open / closed eye estimation device according to claim 1 , wherein the reliability calculated by the open / closed eye determination unit is lower as the similarity of the image to be determined by the open / closed eye determination unit with respect to the predetermined image increases.

6. The open / closed eye estimation device of claim 1 or 2, wherein the eyelid movement detection unit detects the eyelid movement of the open / closed eye estimation subject using a differential image between a first frame image that constitutes part of a moving image including the face of the open / closed eye estimation subject and a second frame image that is different from the first frame image.

7. The open / closed eye estimation device according to claim 1 or 2, wherein the eyelid movement detection unit calculates an optical flow of an image including a face of the open / closed eye estimation subject, and detects the eyelid movement of the open / closed eye estimation subject using the optical flow.

8. The open / closed eye estimation device according to claim 1 or 2, wherein the eyelid movement detection unit detects the eyelid movement of the open / closed eye estimation subject from a moving image including a face of the open / closed eye estimation subject as time-series data, using a time-series model.

9. an open / closed eye determination step in which the open / closed eye estimation device determines whether the open / closed eye estimation subject has their eyes open based on an image of an eye region of the open / closed eye estimation subject cut out from an image including a face of the open / closed eye estimation subject; A reliability calculation step of calculating a reliability of a determination result of the open / closed eye determination step; an eyelid movement detection step of detecting an eyelid movement of the open / closed eye estimation subject when the reliability is equal to or less than a threshold; A method for estimating whether an eye is open or closed, comprising: a correction step of correcting the determination result based on the detection result of the eyelid movement detection step.

10. The processor: an open / closed eyes determination step of determining whether the open / closed eyes estimation subject has their eyes open based on an image of an eye region of the open / closed eyes estimation subject cut out from an image including a face of the open / closed eyes estimation subject; A reliability calculation step of calculating a reliability of a determination result of the open / closed eye determination step; an eyelid movement detection step of detecting an eyelid movement of the open / closed eye estimation subject when the reliability is equal to or less than a threshold; and a correction step of correcting the determination result based on the detection result of the eyelid movement detection step.

Citation Information

Patent Citations

  • Instrument for estimating sleepiness and method for estimating sleepiness

    JP2011229741A

Cited By

  • Dynamic lacrimal river parameter measurement method for dry eye assessment

    CN119417886A

  • Eye closing defect repairing method and device, electronic equipment and computer program product

    CN122554723A

  • Wakefulness estimation device and wakefulness estimation method

    WO2026140051A1