A method and system for detecting closed eyes in a dim environment

Abstract

The application provides a dim environment closed-eye detection method and system, and belongs to the technical field of computer vision detection.The dim environment closed-eye detection method is characterized in that the following steps are included: setting a lower threshold value and an upper threshold value, wherein the lower threshold value and the upper threshold value are both positive numbers, and the lower threshold value is smaller than the upper threshold value; receiving a video frame, calculating an EAR according to eye key point coordinate information; judging the EAR according to a double threshold method, and when a plurality of continuous frames meet the set condition, the closed-eye state is determined, wherein the execution method of the double threshold method includes: judging the EAR value, if the EAR value is smaller than the lower threshold value, starting counting, and if the EAR value is smaller than the upper threshold value, the number of times is accumulated, if the EAR value is not smaller than the upper threshold value, stopping and resetting the counting, and when the number of times reaches the set threshold number of times, the closed-eye state is determined.The application has the beneficial effect that the closed-eye state can be stably and accurately detected in a dim environment such as a mine, thereby providing support for a fatigue driving detection system.

Description

Technical Field

[0001] This invention relates to a visual detection technology, and more particularly to a method and system for detecting closed eyes in dimly lit environments. Background Technology

[0002] A key indicator of fatigue driving detection systems is the driver's eye opening and closing status, using the open, closed, and blinking states as a basis for further analysis of the driver's fatigue level. In dimly lit environments such as mines, where there is significant dust at the workface, workers need to wear protective masks, essentially exposing only their eyes. Therefore, accurate assessment of eye condition is crucial.

[0003] There are two main computer vision (CV) techniques for detecting open and closed eyes. One method trains a classification model to directly classify the Region of Interest (ROI), i.e., the eye image, in the video. The other method first detects facial feature points, then calculates the eye aspect ratio (EAR) using eye keypoints, and finally compares the EAR with a threshold to determine the eye state. The former requires a large amount of real-world facial data and incurs significant annotation costs; the quality of the training data directly affects the model's performance, and its general applicability is not very good. The latter method, while not requiring a large amount of training data, suffers from larger detection errors in harsh environments. However, regardless of the approach used, both face a common challenge: in complex and harsh environments like mines, it is impossible to guarantee the system's stability and real-time performance.

[0004] In dimly lit environments such as mines, maintaining a good mental state is crucial whether driving or working, otherwise, accidents can easily occur. Therefore, a method is needed to accurately determine whether a person's eyes are closed in dimly lit conditions, thereby further determining whether they are fatigued or unwell.

[0005] However, the working environment in mines, with its dim visibility, high dust levels, strong light pollution, and constantly changing environments, poses a challenge to this target. Furthermore, the detection method must possess high stability, as this is crucial to the physical safety and psychological well-being of the workers. Therefore, it must be able to detect the problem accurately without generating false alarms.

[0006] Therefore, accurately and promptly detecting whether the driver's eyes are closed is not only a key point but also a challenge; it is the key to the success of the entire system. Summary of the Invention

[0007] To address the problems in the prior art, this invention provides a method and system for detecting closed eyes in dimly lit environments, which is particularly suitable for dimly lit environments such as mines and can greatly improve the accuracy of detection.

[0008] The present invention provides a method for detecting closed eyes in dimly lit environments, characterized by comprising the following steps:

[0009] S1: Set a lower threshold and an upper threshold, where both the lower threshold and the upper threshold are positive numbers, and the lower threshold is less than the upper threshold;

[0010] S2: Receive video frames and calculate EAR based on the coordinate information of key eye points;

[0011] S3: EAR is determined based on the dual threshold method. If multiple consecutive frames meet the set conditions, it is considered to be in a closed-eye state.

[0012] The execution method for determining eye state based on the dual threshold method includes the following sub-steps:

[0013] S31: Determine if the EAR value is less than the lower threshold.

[0014] S32: If the value is less than the lower threshold, start counting and increment the count if the value is less than the upper threshold; if the value is not less than the upper threshold, stop and reset the count.

[0015] S33: Determine if the number of attempts has reached the set threshold. If yes, determine the state as closed eyes; otherwise, determine the state as open eyes.

[0016] The present invention is further improved by including an upper and lower limit threshold update step after step S2 is executed and before step S3 is executed, which is used to dynamically update the lower limit threshold and the upper limit threshold according to the calculated EAR value.

[0017] The present invention is further improved, and the execution method of the step of updating the upper and lower limit thresholds includes the following sub-steps:

[0018] Determine the size of the EAR cache used to store EAR values;

[0019] If it is less than the preset value, then the EAR of the current video frame is put into the EAR buffer, and then step S3 is executed.

[0020] If it is greater than the preset value, then proceed directly to step S3.

[0021] If it equals the preset value, the EAR of the current video frame is put into the EAR cache, and the lower threshold and upper threshold are calculated based on all the EAR values ​​in the EAR cache.

[0022] The present invention is further improved by including a filtering step before determining the size of the EAR cache used to store the EAR value: when the EAR value is less than a first set value, it is directly filtered out and the next EAR value is determined. The first set value is a positive number and less than the lower limit threshold.

[0023] The present invention is further improved by calculating the lower and upper thresholds based on all EAR values ​​in the EAR cache as follows:

[0024] Calculate the mean and standard deviation (delta) of all EAR values;

[0025] The formula for calculating the lower threshold (threshold_low) is:

[0026] threshold_low=mean / 2.0+delta+bias

[0027] The formula for calculating the upper limit threshold_high is:

[0028] threshold_high=threshold_low+a

[0029] Where bias is the deviation factor and a is a real number greater than 0.

[0030] The present invention is further improved by including a setting step before step S2 is executed: setting the initial value of the frame number to 0, and setting the frame number threshold and EAR buffer;

[0031] In step S2, when receiving video frames, the frame count is incremented. When the frame count exceeds the frame count threshold, the frame count is reset to zero, and the subsequent EAR buffer is cleared.

[0032] In a further improvement to this invention, in step S2, the deep learning model YuNet is used to perform face detection on the video frame. If a face is detected, the face region is cropped to obtain the coordinate information of the face key points and the coordinate information of the eye key points.

[0033] The present invention is further improved by obtaining the coordinate information of the key point of a single eye of a person near the video acquisition device based on the installation position of the video acquisition device, and calculating the EAR based on the key point coordinate information of that eye.

[0034] The present invention also provides a system for implementing the method for detecting closed eyes in dimly lit environments as described in any one of claims 1-9, characterized in that it comprises:

[0035] Setting module: Sets the lower threshold and the upper threshold, where both the lower threshold and the upper threshold are positive numbers, and the lower threshold is less than the upper threshold;

[0036] Receiver module: Used to receive video frames;

[0037] Calculation module: Used to calculate the EAR based on the coordinate information of key eye points;

[0038] Judgment Module: Used to determine EAR based on the dual threshold method. If multiple consecutive frames meet the criteria, it is considered as closed eyes.

[0039] The execution method of the dual threshold method includes:

[0040] Determine the EAR value.

[0041] If the value is less than the lower threshold, a counter is started. Each time the value is less than the lower threshold, the count is incremented by 1. When the count reaches the set threshold, the eyes are determined to be closed.

[0042] If the value exceeds the upper limit threshold, stop and reset the count.

[0043] The invention is further improved by including an upper and lower limit threshold update module: used to dynamically update the lower limit threshold and the upper limit threshold based on the calculated EAR value.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. This invention uses EAR as the basis for judgment, which does not require a large amount of real face data, has good versatility, and is more flexible and efficient in operation;

[0046] 2. This invention employs a dual-threshold method for judgment, enabling rapid and accurate assessment within a certain fluctuation range. Extensive testing has demonstrated excellent performance in dimly lit environments such as mines, demonstrating real-time and accurate identification of whether a person's eyes are closed. It maintains stable and accurate detection even in dimly lit environments like mines, thus providing support for systems such as fatigue driving detection.

[0047] 3. By dynamically calculating and updating the upper and lower thresholds, this invention is applicable to real-time detection in different background environments and by different people, and achieves higher detection accuracy in constantly changing backgrounds and scenarios. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method of the present invention;

[0049] Figure 2 This is a flowchart of a method according to an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the coordinate information of key points in the human eye;

[0051] Figure 4 This is a flowchart of the dual threshold method execution method of the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0053] Since the EAR-based model for detecting right eye closure does not have high requirements for training data, and excellent open-source deep learning models can be used, this example uses this technology as the starting point for the invention, making it more flexible to operate.

[0054] like Figure 1 As shown, the present invention provides a method for detecting closed eyes in dimly lit environments, characterized by comprising the following steps:

[0055] S1: Set a lower threshold and an upper threshold, where both the lower threshold and the upper threshold are positive numbers, and the lower threshold is less than the upper threshold;

[0056] S2: Receive video frames and calculate EAR based on the coordinate information of key eye points;

[0057] S3: EAR is determined based on the dual threshold method. If multiple consecutive frames meet the set conditions, it is considered to be in a closed-eye state.

[0058] The execution method for determining eye state based on the dual threshold method includes the following sub-steps:

[0059] S31: Determine if the EAR value is less than the lower threshold.

[0060] S32: If the value is less than the lower threshold, start counting and increment the count if the value is less than the upper threshold; if the value is not less than the upper threshold, stop and reset the count.

[0061] S33: Determine if the number of attempts has reached the set threshold. If yes, determine the state as closed eyes; otherwise, determine the state as open eyes.

[0062] Current technology for determining closed eyes directly sets a fixed threshold, such as 0.2. If the Earnings Expiration (EAR) is less than 0.2 for five consecutive frames (the consecutive frames are to filter out abnormal fluctuations), it is considered a closed eye. While this fixed threshold method is simple and direct, it lacks stability and applicability. Due to the complexity of the environment and the capabilities of deep learning models, the calculated EAR value fluctuates constantly, and in some scenarios, the fluctuation is significant. This volatility means that when the threshold is too small, it is difficult to have multiple consecutive frames below the threshold, thus failing to quickly detect closed eyes. Conversely, when the threshold is too large, it leads to false alarms. This volatility makes it impossible to find a perfect single threshold.

[0063] This invention employs a dual-threshold method for judgment, enabling rapid and accurate assessment within a certain fluctuation range. The invention uses a lower threshold as a start switch and an upper threshold as a stop switch, effectively preventing misjudgments caused by squinting or blinking. Especially in the dimly lit environment of mines, the strong light from a miner's lamp can trigger an involuntary conditioned reflex, causing the eyes to squint. This invention, by setting an upper threshold, stops counting and resets the count when it is reached, effectively filtering out misjudgments in such situations.

[0064] Through numerous experiments, this invention demonstrates excellent performance in dimly lit environments such as mines, enabling real-time and accurate identification of whether a person's eyes are closed. In particular, it maintains stable and accurate detection even in dimly lit environments like mines, thus providing support for systems such as fatigue driving detection.

[0065] However, this method is limited to situations where the driver remains the same, the operation is single-line, and the work environment is unchanged. If a different person with smaller eyes is used for testing, or if the same person is placed in a test environment that differs significantly from their current work environment, the results may deteriorate considerably, indicating that the algorithm's general applicability is still insufficient. This is because different people have different eye shapes, and naturally, each person's EAR (Earning Eye Range) will be different. The previously set thresholds cannot be applied to new testers. In particular, when testers have smaller eyes, the EAR value is very low, close to the EAR of a normal person with their eyes closed, which was a major challenge for the development team at the time.

[0066] Therefore, through extensive experiments, this invention proposes a method that combines dynamic thresholding and dual thresholding, which we call the dynamic dual thresholding method.

[0067] Specifically, in order to adapt to various environments and detection after personnel changes, this example also includes an update of upper and lower limit thresholds after step S2 is executed and before step S3 is executed. This update is used to dynamically update the lower and upper limit thresholds based on the calculated EAR value.

[0068] In addition to setting the lower and upper thresholds in step S1, this example also includes setting the initial frame count to 0, and setting the frame count threshold and EAR buffer. This enables dynamic updating of the upper and lower thresholds.

[0069] like Figure 2 As shown, the detailed implementation process of the present invention is as follows, as an embodiment of the present invention:

[0070] 1. Use an infrared camera as the hardware input interface to input video frames. Of course, other video capture devices can also be used in this example.

[0071] 2. The video stream is fed frame by frame into the algorithm module of this invention. The frame count (frameCount) is incremented. If the frame count exceeds a frame count threshold (e.g., 300), the frame count is reset to zero, and the subsequent EAR cache is cleared. The purpose of this step is to periodically update the EAR cache for later dynamic updates of the upper and lower thresholds. The threshold of 300 is used to control the update period and can be adjusted according to the actual frame rate of the device and the scenario.

[0072] 3. Use the deep learning model YuNet to perform face detection on the video frames. If a face is detected, proceed to the next step; otherwise, return to step 1.

[0073] 4. Extract the ROI (Region of Interest) of the face and input it into the Landmarks coordinate model to obtain the coordinate information of the facial key points.

[0074] 5. Calculate the Eye Aspect Ratio (EAR) based on the eye key point coordinates. EAR represents the aspect ratio of the eye; it's a ratio and therefore unaffected by the distance of the face. It's the fundamental parameter for determining whether the eyes are closed. Existing technology calculates the EAR of both eyes and obtains the average value. However, since the camera is not installed directly in front of the face, and the eye on the side closest to the camera will have significant distortion, this invention uses single-eye calculation. In this example, based on the camera's installation position (assuming the camera is to the right of the driver), since the left eye will have significant distortion in the image due to angle, using only the right eye EAR will be more accurate. Eye key point coordinates are shown in Figures 43-48. Figure 3 As shown, the formula for calculating EAR in this example is as follows:

[0075]

[0076] Among them, ||P 44 -P 48 || represents the height between key points 44 and 48, ||P 45 -P 47 || represents the height between key points 45 and 47, ||P 43 -P 46 || represents the width between key points 43 and 46.

[0077] 6. Determine the EAR cache size. If it's less than the preset `buffer_size` (e.g., 50), add the calculated EAR from the previous step to the EAR cache and proceed to the next step. If it's greater than `buffer_size`, proceed directly to the next step. If it's equal to `buffer_size`, add the calculated EAR from the previous step to the EAR cache, then calculate and update the upper and lower thresholds using the formula. Then proceed to the next step. In this example, `buffer_size` is also a preset threshold. The purpose is to use such a large cache size to calculate the EAR threshold. Obviously, the larger this value, the more accurate the calculation result, but it will take longer to fill the cache. Therefore, a trade-off must be made in terms of time.

[0078] The method for dynamically calculating the threshold in this example is as follows:

[0079] (1) The threshold is dynamically calculated based on real data, rather than a preset fixed value. The purpose is to find the most suitable threshold.

[0080] This involves statistically analyzing a continuous video segment and storing the eye EAR (Eye Scale) of N consecutive frames into an EAR cache. Before storing the EAR in the cache, this example also includes a filtering strategy to filter out EARs that are too small (e.g., below 0.2). This is because EARs that are too small are either due to closed eyes or abnormal data, which are not statistically significant and will only affect the calculation of the effective threshold.

[0081] When the EAR cache is full, the mean and standard deviation (delta) of the EARs within the cache are calculated. Clearly, the mean represents the EAR of the target person in a normal, open-eye state, and delta represents the fluctuation range.

[0082] The formula for calculating the lower limit threshold_low in this invention is as follows:

[0083] threshold_low=mean / 2.0+delta+bias

[0084] The formula for calculating the upper limit threshold_high is:

[0085] threshold_high=threshold_low+a

[0086] Where 'a' is a real number greater than 0. In this example, 'a' is 0.06. 'Bias' is a small deviation factor used to further adjust the threshold to debug the system to achieve optimal performance.

[0087] (2) Periodically update the EAR to adapt to the changing environment and human eye objects.

[0088] If the dual thresholds are only calculated once at the initial stage of system startup and then not changed, there will be two shortcomings. First, if the operator changes midway, the upper and lower thresholds will no longer be applicable. Second, the calculated EAR may be different for different background environments, and our vehicle-mounted camera is constantly moving, which may lead to completely different working environments over time. Therefore, this invention will periodically recalculate to adapt to the constantly changing scenarios.

[0089] 7. Based on the EAR's dual threshold method, the eyes are considered closed when multiple consecutive frames meet the condition.

[0090] like Figure 3 As shown, the dual threshold method of the present invention is as follows:

[0091] (1) Input the EAR of the current frame.

[0092] (2) Determine if the current EAR is less than the lower threshold. If yes, start counting, increment the count close_count, and then execute step (3). If no, determine if counting is enabled. If yes, determine if it is less than the upper threshold. If it is less than the upper threshold, increment the count close_count. If it is greater than or equal to the upper threshold, reset the count close_count to zero and disable the counting state.

[0093] (3) Check if close_count is greater than the set threshold. If it is, determine that the eyes are closed; otherwise, determine that the eyes are open. In this example, the threshold is 4, but it can be set to other values ​​according to the actual situation.

[0094] Compared with existing technologies, this invention can stably, accurately, and in real-time detect the opening and closing states of different human eyes in dimly lit environments such as mines. Through extensive experimentation and practical application in mines, this invention has achieved highly satisfactory results.

[0095] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A method for detecting closed eyes in dimly lit environments, characterized in that, Includes the following steps: S1: Set a lower threshold and an upper threshold, where both the lower threshold and the upper threshold are positive numbers, and the lower threshold is less than the upper threshold; S2: Receive video frames and calculate EAR based on the coordinate information of key eye points; S3: EAR is determined based on the dual threshold method. If multiple consecutive frames meet the set conditions, it is considered to be in a closed-eye state. The execution method for determining eye state based on the dual threshold method includes the following sub-steps: S31: Determine if the EAR value is less than the lower threshold. S32: If the value is less than the lower threshold, start counting and increment the count if the value is less than the upper threshold; if the value is not less than the upper threshold, stop and reset the count. S33: Determine if the number of attempts has reached the set threshold. If yes, determine the state as closed eyes; otherwise, determine the state as open eyes.

2. The method for detecting closed eyes in a dimly lit environment according to claim 1, characterized in that: After step S2 is executed and before step S3 is executed, there is also a step to update the upper and lower limit thresholds, which is used to dynamically update the lower limit threshold and the upper limit threshold based on the calculated EAR value.

3. The method for detecting closed eyes in a dimly lit environment according to claim 2, characterized in that: The execution method for the step of updating the upper and lower thresholds includes the following sub-steps: Determine the size of the EAR cache used to store EAR values; If the value is less than the preset value, the EAR of the current video frame is placed into the EAR buffer, and then step S3 is executed. If the value is greater than the preset value, step S3 is executed directly. If it equals the preset value, the EAR of the current video frame is put into the EAR cache, and the lower threshold and upper threshold are calculated based on all the EAR values ​​in the EAR cache.

4. The method for detecting closed eyes in a dimly lit environment according to claim 3, characterized in that: Before determining the size of the EAR cache used to store EAR values, a filtering step is also included: when the EAR value is less than a first set value, it is directly filtered out, and the next EAR value is determined. The first set value is a positive number and less than the lower threshold.

5. The method for detecting closed eyes in a dimly lit environment according to claim 3, characterized in that: The calculation method for the lower and upper thresholds based on all EAR values ​​in the EAR cache is as follows: Calculate the mean and standard deviation (delta) of all EAR values; The formula for calculating the lower threshold (threshold_low) is: threshold_low=mean / 2.0+delta+bias The formula for calculating the upper limit threshold_high is: threshold_high=threshold_low+a Where bias is the deviation factor and a is a real number greater than 0.

6. The method for detecting closed eyes in dimly lit environments according to any one of claims 1-5, characterized in that: Before step S2 is executed, a setup step is also included: setting the initial frame number to 0, and setting the frame number threshold and EAR buffer; In step S2, when receiving video frames, the frame count is incremented. When the frame count exceeds the frame count threshold, the frame count is reset to zero, and the subsequent EAR buffer is cleared.

7. The method for detecting closed eyes in dimly lit environments according to any one of claims 1-5, characterized in that: In step S2, the deep learning model YuNet is used to perform face detection on the video frame. If a face is detected, the face region is cropped to obtain the coordinate information of the face key points. Based on the coordinate information of the face key points, the coordinate information of the eye key points is obtained.

8. The method for detecting closed eyes in a dimly lit environment according to claim 7, characterized in that: Based on the installation location of the video acquisition device, obtain the key point coordinates of a single eye of a person near the video acquisition device, and calculate the EAR based on the key point coordinates of that eye.

9. A system for implementing the method for detecting closed eyes in a dimly lit environment as described in any one of claims 1-8, characterized in that, include: Setting module: Sets the lower threshold and the upper threshold, where both the lower threshold and the upper threshold are positive numbers, and the lower threshold is less than the upper threshold; Receiver module: Used to receive video frames Calculation module: Used to calculate the EAR based on the coordinate information of key eye points; Judgment Module: Used to determine EAR based on the dual threshold method. If multiple consecutive frames meet the criteria, it is considered as closed eyes. The execution method of the dual threshold method includes: Determine the EAR value. If the value is less than the lower threshold, a counter is started. Each time the value is less than the lower threshold, the count is incremented by 1. When the count reaches the set threshold, the eyes are determined to be closed. If the value exceeds the upper limit threshold, stop and reset the count.

10. The system according to claim 9, characterized in that: It also includes a module for updating upper and lower thresholds: used to dynamically update the lower and upper thresholds based on the calculated EAR value.

Images