Eye state detection method and device, display interface, equipment and medium

Abstract

The invention relates to an eye state detection method and apparatus, a display interface, a device and a medium. The method comprises the steps of obtaining multiple frames of target eye images; inputting each frame of the target eye image into a trained preset classification network to obtain a probability of an eye state corresponding to the target eye image; recognizing a blink event based on the probability of the eye state; wherein the preset classification network is a lightweight convolutional neural network, the lightweight convolutional neural network comprises a feature extraction module, and the feature extraction module comprises three convolutional layers and two maximum pooling layers. According to comparison between the eye state probability and the preset threshold value, accurate judgment of the blink state is achieved, the sensitivity and accuracy of the detection method are enhanced, the normal state and the blink signal are distinguished through fine logic calculation, and misjudgment is effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of eye detection technology, and in particular to an eye state detection method, device, display interface, equipment and medium. Background Art

[0002] Blinking is an important physiological response for maintaining eye health, which involves cleaning and moisturizing the surface of the eyeball. In addition, the frequency of blinking is closely related to an individual's attention, fatigue level, and ophthalmic diseases. Therefore, automated blink detection technology has a wide range of applications, including driver fatigue monitoring, attention status assessment, etc. Existing blink detection methods can be divided into two categories: biosignal-based and video-based detection. Biosignal-based methods usually rely on electrooculograms (EOGs). Although the detection is accurate, they require complex sensor equipment and have limited usage scenarios. In contrast, video-based methods use image processing technology to detect blinks by analyzing eye images or videos. Existing video methods are mainly based on deep learning models such as convolutional neural networks (CNNs).

[0003] However, in the existing technology, video-based blink detection has the following major problems: first, many deep learning models, such as those based on long short-term memory (LSTM) or complex convolutional networks, have large computational complexity and high hardware requirements, and cannot meet the needs of real-time detection; second, in complex environments such as lighting changes and head movements, the detection accuracy of existing methods decreases, especially in the case of rapid blinking; finally, some models have too many parameters, high training and deployment costs, and are difficult to implement on resource-constrained devices. Summary of the invention

[0004] In order to solve the problems of complex model, high computational cost, insufficient detection robustness and low detection accuracy, the present application provides an eye state detection method, device, display interface, equipment and medium.

[0005] The first aspect of the present application provides an eye state detection method, the method comprising:

[0006] Acquire multiple frames of target eye images;

[0007] Input each frame of the target eye image into a trained preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers;

[0008] Blink events are identified based on the probabilities of the eye states.

[0009] In some embodiments of the present application, the preset classification network includes a global average pooling layer; the step of inputting the target eye image into the preset classification network to obtain the probability of the eye state corresponding to the target eye image includes:

[0010] Inputting the target eye image into the feature extraction module to extract features and obtain a first feature map;

[0011] Inputting the first feature map into the global average pooling layer for pooling operation to obtain a second feature map;

[0012] A normalization operation is performed based on the second feature map to obtain the probability of the eye state corresponding to the target eye image.

[0013] In some embodiments of the present application, the preset classification network further includes a linear module, and the linear module includes 2 consecutive fully connected layers; the normalization operation based on the second feature map to obtain the probability of the eye state corresponding to the target eye image includes:

[0014] Input the second feature map into the linear module and output a target vector;

[0015] The target vector is normalized to obtain the probability of the eye being in different eye states.

[0016] In some embodiments of the present application, the identifying a blink event based on the probability of the eye state includes:

[0017] For the target eye image of the current frame, compare the probability of the eye state corresponding to the target eye image of the current frame with a first probability threshold, and if the probability of the eye state is greater than or equal to the first probability threshold, determine that the eye state is a closed state;

[0018] Determine a second probability threshold by using N frames of images before the target eye image of the current frame, where N is greater than or equal to 1;

[0019] The probability of the eye state is compared with the second probability threshold, and a blink event is identified based on the comparison result.

[0020] In some embodiments of the present application, determining the second probability threshold through the previous N frames of the target eye image of the current frame includes:

[0021] According to the probability of the eye state corresponding to each frame in the previous N frames of images, the average of the eye state probabilities of the previous N frames is calculated;

[0022] Based on the average value and a preset offset, the second probability threshold is obtained.

[0023] In some embodiments of the present application, comparing the probability of the eye state with the second probability threshold, and identifying a blink event according to the comparison result, includes:

[0024] If the probability of the eye state is greater than or equal to the second probability threshold, the target eye image of the current frame is determined to be in a closed state, and it is determined that the target eye image of the current frame corresponds to a blink event.

[0025] In some embodiments of the present application, the training method of the preset classification network includes:

[0026] Obtaining a data set, wherein the data set includes a training set and a validation set;

[0027] Training the preset classification network through the training set, and updating the parameters of the preset classification network by using a back propagation algorithm;

[0028] Using a preset optimizer to optimize the weights of the preset classification network and minimize the prediction error calculated by the cross entropy loss function;

[0029] After each training cycle, the performance of the preset classification network is evaluated using the validation set, and the training is terminated after a preset number of cycles are completed and the preset classification network reaches convergence.

[0030] In some embodiments of the present application, acquiring the target eye image includes:

[0031] receiving eye video captured by a camera;

[0032] Extract multiple frames of continuous eye images from the eye video, and use the multiple frames of continuous eye images as multiple frames of target eye images.

[0033] A second aspect of the present application provides an eye state detection device, the device comprising:

[0034] An acquisition module, used for acquiring a target eye image;

[0035] A classification module, used for inputting the target eye image into a preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers;

[0036] A recognition module is used to recognize a blink event based on the probability of the eye state.

[0037] A third aspect of the present application provides a display interface, which is used to display the detection results of the eye state detection device described in the embodiment, and the detection results include the number of blinks and the blink rate within a preset time period.

[0038] A fourth aspect of the present application provides an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the eye state detection method described in each embodiment of the present application.

[0039] A fifth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the eye state detection method described in each embodiment of the present application is implemented.

[0040] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0041] The eye state detection method in each embodiment of the present application obtains multiple frames of target eye images, inputs each frame of the target eye image into a trained preset classification network, and obtains the probability of the eye state corresponding to the target eye image, wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolution layers and 2 maximum pooling layers, and recognizes blinking events based on the probability of the eye state. In this way, the present application uses a lightweight convolutional neural network for feature extraction, accurately predicts the opening and closing state of the eyes for each frame of the image, and accurately distinguishes natural blinking from non-blinking activities on the basis of real-time monitoring, thereby improving the practicality of the method. In addition, the present application realizes accurate judgment of the blinking state based on the comparison of the eye state probability with the preset threshold, enhances the sensitivity and accuracy of the detection method, and distinguishes normal eye movement and blinking signals through sophisticated logical calculations, effectively reducing misjudgment.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0044] Figure 1 is a schematic diagram of the steps of an eye state detection method in an exemplary embodiment of the present application;

[0045] Figure 2 is a schematic diagram of the structure of a preset classification network in an exemplary embodiment of the present application;

[0046] Figure 3 is a flow chart of a probability recognition of blink events based on eye states in an exemplary embodiment of the present application;

[0047] Figure 4 is a schematic diagram of the overall process of an eye state detection method in an exemplary embodiment of the present application;

[0048] Figure 5 is a schematic diagram of an eye state detection device in an exemplary embodiment of the present application;

[0049] Figure 6 is a schematic diagram of a display result of a display interface in an exemplary embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0052] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0053] Blinking is a natural response that is essential to eye health. It not only helps clean and moisturize the surface of the eyeball, but is also closely linked to a person's attention level, fatigue, and certain eye diseases. For these reasons, developing technology for automatically detecting blinks has many practical applications, such as monitoring whether a driver is overly tired and assessing a person's concentration. Currently, there are two main types of technologies for detecting blinks: one is based on biological signals, and the other is based on video images. Biosignal-based detection mainly relies on electrooculogram (EOG) technology. Although this type of method can provide accurate detection results, the sensors it uses are relatively complex, which limits its use scenarios. At the same time, the video image method uses image processing technology to detect blinks by analyzing images or videos of the eyes. This method usually uses deep learning models such as convolutional neural networks (CNNs) for this purpose.

[0054] However, video-based detection techniques face some challenges. First, many advanced deep learning models, such as those using long short-term memory networks (LSTM) or advanced convolutional networks, require a lot of computing resources and are difficult to meet the requirements of real-time detection. Second, when encountering complex environmental conditions such as changes in lighting conditions or head movements, the detection accuracy of these technologies will be greatly reduced, and rapid blinking movements are particularly difficult to accurately identify. Finally, because these models often have a large number of parameters, the training and deployment costs are high, and implementing these technologies is a major challenge for devices with limited resources.

[0055] Therefore, in some embodiments of the present application, an eye state detection method is provided, such as Figure 1 As shown, the method includes:

[0056] S1. Acquire multiple frames of target eye images.

[0057] In a specific implementation, an eye video captured by a camera may be first received, and then multiple frames of continuous eye images in the eye video may be extracted, and the multiple frames of continuous eye images may be used as multiple frames of target eye images. Specifically, a high-resolution camera may be set to continuously shoot the user's eyes to obtain a continuous eye video including blinking movements. Considering the dynamic characteristics of blinking, the frame rate of the video should be at least 30fps to ensure that the moment of rapid closing and reopening of the eyelids during blinking can be captured.

[0058] S2. Input each frame of the target eye image into a trained preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers.

[0059] Figure 2 The structure of the preset classification network is shown as follows: Figure 2 As shown in the figure, the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers, and batch normalization and ReLU activation functions are also applied between layers. The 3 convolutional layers are (3*3, 8), (3*3, 16), and (3*3, 32), that is, the convolution kernel size is 3x3, and the depth is 8, 16, and 32 respectively; the pooling window size of the 2 maximum pooling layers is 2×2.

[0060] The preset classification network also includes a global average pooling layer, which is Figure 2In the (GAP, 128), the feature extraction module is next to the global average pooling layer, and the linear module is followed by the global average pooling layer, and the linear module includes 2 consecutive fully connected layers. The present application introduces the global average pooling layer and adds the linear module, which improves the effectiveness of feature extraction, reduces the computational complexity, speeds up the response speed of the network, makes the classification results more accurate, helps to enhance the generalization ability of the model, and improves the accuracy of blink state recognition.

[0061] For each frame of the target eye image input, it is first processed by the feature extraction module to capture the main features of the eye image. Then, these features are sent to the global average pooling layer and the subsequent linear module to finally obtain a normalized probability vector indicating whether the eyes are open or closed. In a specific implementation, the target eye image is input into a preset classification network to obtain the probability of the eye state corresponding to the target eye image, including steps S21-S23.

[0062] S21. Input the target eye image into the feature extraction module to extract features and obtain a first feature map.

[0063] A 64*64 target eye image is input into the feature extraction module for feature extraction. It first enters the first convolution layer, which uses a 3x3 convolution kernel with a depth of 8 to perform a convolution operation on the 64x64 image. This operation uses the convolution kernel to slide on the image, calculate the dot product between the convolution kernel and the local image, and generate a new feature map. Since the convolution kernel depth is 8, this operation will generate 8 64x64 feature maps. These feature maps represent low-level features in the image, such as edges or simple textures. The ReLU activation function is then used to increase nonlinearity to help the model capture more complex features. The second convolution layer uses a deeper convolution kernel (depth of 16) to process a feature map of size 32x32. This further increases the depth of the network, allowing the model to extract higher-level features such as corners and complex textures. The 16 32x32 feature maps will more accurately represent the abstract features of the image. The output also maintains the size of 64x64, but the depth is increased to 16, which represents more complex features being extracted.

[0064] Then, the first 2x2 maximum pooling layer is used for pooling. This pooling operation downsamples the output of the previous convolutional layer, using a 2x2 window and a step size of 2. The role of the pooling layer is to reduce the feature dimension, enhance the robustness of the feature and the generalization ability of the feature extraction. The size of the output image is reduced to 32x32, and the depth is kept at 16. This step effectively reduces the amount of calculation and extracts more significant features. Next, the third convolutional layer (3x3, 32) is used for convolution operation, that is, a convolution kernel of 3x3 size and depth of 32 is further used for feature extraction. This layer enables the network to capture higher-level details in the image. The size of 32x32 is still maintained, but the depth is doubled to 32, which means that more complex image features are encoded in these feature maps.

[0065] Furthermore, the output of the third convolutional layer is downsampled again through the second 2x2 maximum pooling layer, further compressing the spatial size of the feature map and improving the abstraction ability of the model. The result of the pooling operation is that the size of the feature map is halved to 16x16, while the depth 32 remains unchanged. The pooling operation further simplifies the features, that is, the first feature map is obtained, which helps the model to be more efficient and accurate in the final classification.

[0066] S22. Input the first feature map into the global average pooling layer for pooling operation to obtain a second feature map.

[0067] Specifically, the size of the first feature map is adjusted to 8x8 and the depth is 128, and the global average pooling layer is directly applied to the feature map. Unlike the traditional pooling layer, GAP calculates the average value of the entire feature map of each channel, and each channel is finally converted into a single value, so that the 8x8 feature map with a depth of 128 is converted into a 128-dimensional vector, that is, the second feature map is obtained. It can be understood that GAP not only significantly reduces the number of parameters and avoids overfitting, but also maintains the spatial hierarchical information in the feature map, so that the model can better understand the spatial patterns in the image.

[0068] S23. Perform a normalization operation based on the second feature map to obtain the probability of the eye state corresponding to the target eye image.

[0069] Furthermore, the linear module includes two fully connected layers (FC). For the first fully connected layer (FC, 64), the 128-dimensional vector after GAP, i.e., the second feature map, is fed into a linear module with 64 neurons. The task of this layer is to map the 128-dimensional feature vector to a new 64-dimensional space. This step can be regarded as further refinement and compression of the feature vector. At the same time, in order to improve the nonlinear expression ability of the model, the ReLU activation function is usually applied. For the second fully connected layer (FC, 2), the 64-dimensional vector is further passed through another fully connected layer containing 2 output units to adapt to the binary classification task (eyes open or closed). This transformation aims to transform the complex feature relationship of the image into the ultimate concern of the model: the different states of the eyes, i.e., open or closed.

[0070] For the probability of the eye state corresponding to the target eye image, the 2D output vector is processed by the Softmax function and converted into a probability form. The Softmax function ensures that the two output values ​​are non-negative and sum to 1, and each value represents the probability of the eye in the image being closed or open. This probability output is the direct target of model training and optimization. It is compared and calculated with the true label through the cross entropy loss function, and the internal parameters are continuously adjusted to reduce the error and improve the classification performance of the model.

[0071] The above preset classification model demonstrates excellent performance in eye state detection. Through continuous convolution and pooling layers, the model can extract meaningful spatial features from the original eye image, and reduce the computational complexity and overfitting risk through GAP and linear modules, and finally output highly reliable eye state probabilities through the Softmax layer. This not only demonstrates the powerful capabilities of deep learning in visual tasks, but also provides an efficient and accurate technology for future applications in related fields such as eye tracking, fatigue monitoring, and human-computer interaction. Specifically, the pooling operation result is input into the linear module, the target vector is output, and the target vector is normalized to obtain the probability that the eye is in the eye state.

[0072] S3. Identify a blink event based on the probability of the eye state.

[0073] For the target eye image of the current frame, compare the probability of the eye state corresponding to the target eye image of the current frame with the first probability threshold, and if the probability of the eye state is greater than or equal to the first probability threshold, determine that the eye state is a closed state; determine the second probability threshold through the first N frames of the target eye image of the current frame, where N is greater than or equal to 1; compare the probability of the eye state with the second probability threshold, and identify the blink event according to the comparison result. Wherein, when determining the second probability threshold through the first N frames of the target eye image of the current frame, the average value of the probability of the eye state of the first N frames can be calculated based on the probability of the eye state corresponding to each frame in the first N frames, and then based on the average value, combined with a preset offset, the second probability threshold is obtained.

[0074] Specifically, for each frame of the target eye image, the preset classification network will output a probability representing the eye state. Figure 3 , set a first probability threshold to distinguish whether the eyes are in an open state or a closed state. If the eye closure probability of the current frame is greater than or equal to the first probability threshold, the eye state of this frame is determined to be closed. Figure 3 As shown, the algorithm used to identify the blink event as a whole is used as the recognition algorithm, and the probability of the current input is the current frame probability P i , the first probability threshold is Figure 3 The basic threshold TH in B , if P i >TH B , then the eye state of this frame is considered to be closed. Assuming that the first probability threshold is set to 0.5, if the closed probability of a frame is 0.6, then the eye state of this frame is considered to be closed. If Pi does not exceed this basic threshold, the i-th frame cannot be recognized as a blink.

[0075] Intuitively, a new blink event can be described as the eye state of the current frame being “more closed” compared to the previous state. Therefore, an adaptive threshold TH is defined based on the average eye closure probability of the previous N frames. i , the adaptive threshold TH i That is the second probability threshold. i The formula is:

[0076]

[0077] Where N is the number of previous images of the target eye image in the current frame, N is greater than or equal to 1, n is used for value selection, P i-n is the probability corresponding to the nth image preceding the target eye image in the current frame, and k is a parameter reflecting the degree of significance.

[0078] It can be understood that by analyzing the closure probability of the N frames before the current frame, the average of the closure probabilities of these frames is calculated. This average reflects the general probability level of the eye closure state in the recent period. Specifically, based on the average of the closure probabilities of the previous N frames, combined with a preset offset (for example, 0.1), the offset can be obtained using k in the above formula to determine the second probability threshold. This threshold is used to identify sudden blinking events, and the setting of the offset reflects the probability change amplitude required from the normal closed state to the real blinking state. During the recognition process, the result of the previous frame needs to be considered. If the i-1 frame is not recognized as a blink, the current frame is considered to be a new blink event; otherwise, it is considered to be still in the state of the previous blink and is not a new event. In other words, the eye closure probability of the current frame is compared with the calculated second probability threshold. If the closure probability of the current frame is greater than or equal to the second probability threshold, the current frame is determined to be part of a blink. It can be seen that the recognition algorithm proposed in the present application is based on a probability threshold to identify new blinking events, which not only ensures the accuracy of the blinking detection results, but also enhances the robustness of detection under different environments (such as different lighting conditions).

[0079] In actual use, in addition to simple probability comparison, continuity also needs to be considered. If several consecutive frames all meet the closure probability greater than the second probability threshold, and the duration of these frames is within the normal human blinking time range (e.g., 100 milliseconds to 400 milliseconds), then this series of consecutive frames is identified as a complete blink event.

[0080] Therefore, the eye state detection method of the present application can refer to Figure 4 , Figure 4 This is a schematic diagram of the overall process of the eye state detection method. Figure 4 The paper presents a coherent and efficient eye monitoring, analysis and result display process. First, a high-definition camera is used to capture the user's eye image in real time, which is the data input source of the entire system, ensuring that the eye status information is clear and coherent enough, laying the foundation for subsequent processing. Secondly, the classification network works. The collected video stream is transmitted to the classification network, which is a preset lightweight convolutional neural network. This network is responsible for in-depth analysis of the target eye image of each frame, and extracts eye features through the feature extraction module (including 3 convolutional layers and 2 maximum pooling layers). Next, the recognition algorithm is used to identify new blink events. The algorithm considers continuity and probability thresholds (including a dynamically adjusted second probability threshold) to accurately determine and count blinks. The final "result display" shows the output results of the detection device, that is, the number of blinks and blink rate in a preset time period are presented through the display interface, and interactive elements such as blink indicators and eye fatigue indicators are also included.

[0081] In some other embodiments of the present application, a training method for a preset classification model is provided. During training, image data containing various eye states are first collected and divided into a training set and a validation set. The training set is used to train the network, while the validation set is used to evaluate the generalization ability of the model. The preset classification network is trained with the data on the training set. During the training process, each input image is marked as an eye open or closed state, and a prediction is generated by the network. The difference between the predicted output of the network and the true label is quantified by the cross entropy loss function, and the formula is:

[0082]

[0083] Where b is the batch size, y is the expected value, and a is the actual output value.

[0084] Preferably, the gradient of the loss function with respect to the network parameters is calculated using a back-propagation algorithm, and the weights of the network are updated using a preset optimizer (e.g., Adam or SGD) to minimize the loss function. During the optimization process, some strategies may be adopted to adjust the learning rate, such as learning rate decay, to improve the efficiency and effectiveness of training. After each round of training, the model performance is evaluated using an independent validation set to monitor overfitting during the training process. Performance can be measured by indicators such as accuracy, recall, and F1 score.

[0085] Repeat the training and validation steps until the performance of the model on the validation set is no longer significantly improved, or the preset number of training cycles is reached. At this point, the model is considered to have reached convergence and the training process can be stopped. After the model training is completed, an independent test set is used to perform a final evaluation of the model. After ensuring that the model has good generalization capabilities, it can be deployed in a blink detection system to detect and analyze real-time blink events. Through this series of detailed training steps, not only can the model achieve high accuracy in the blink detection task, but the model structure and parameters can also be optimized through timely feedback, ultimately achieving an eye state detection effect that is both robust and accurate.

[0086] This application also conducted relevant experiments to verify the technical effect of the method of this application. Public datasets and specially constructed custom datasets were used during the experiment. On the one hand, the MRL Eye Dataset dataset was used, which is a large-scale human eye image dataset containing 84,898 images of 37 different subjects (33 men and 4 women). These images cover the opening and closing states of the eyes under various acquisition conditions, covering different situations of whether glasses are worn, the amount of reflection (none, a small amount, a large amount) and the quality of light (good, poor). In addition, a custom dataset was prepared. Through a head-mounted device equipped with a high-quality eye camera, we collected 1,972 eye images from 7 volunteers, including 217 closed-eye images. In order to comprehensively evaluate the performance of the blink detection system, a continuous eye video with a total length of 27 minutes was also recorded.

[0087] During the network training process of this experiment, the eye state classification network went through 150 training cycles, and the Adam optimizer was used to automatically adjust the learning rate to improve training efficiency. In order to enhance the adaptability and generalization of the model to different environments, data enhancement techniques including random cropping, rotation, and brightness adjustment were implemented. Performance evaluation After rigorous training, our model performed well on MRL and custom datasets. The model achieved an accuracy of 98.14% on the MRL dataset and 96.70% on the custom dataset, with recall rates of 99.69% and 98.16% respectively, showing strong strength in the eye state classification task.

[0088] Regarding the overall performance of the experiment, this application comprehensively considers the complexity of blink detection, especially the sample imbalance problem, and uses the F1 score as the evaluation criterion, which is calculated by the harmonic mean of precision and recall. B =0.15, k=3, N=6, equivalent to a time span of 0.2 seconds), the system showed good performance, with an F1 score close to 0.94. In addition, the system processing speed reached 109.1 frames per second, which can meet the needs of real-time monitoring.

[0089] In some other embodiments of the present application, an eye state detection device is provided, such as Figure 5 As shown, the device includes: an acquisition module 401, used to acquire a target eye image; a classification module 402, used to input the target eye image into a preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolution layers and 2 maximum pooling layers; an identification module 403, used to identify blinking events based on the probability of the eye state.

[0090] The acquisition module 401 is responsible for capturing the user's target eye image in real time. This module uses high-resolution camera technology to ensure that the acquired eye image is clear enough to accurately reflect the opening and closing state of the eye and other details. This module works in a real-time video stream mode and can maintain good image quality under different ambient light conditions, providing high-quality input data for subsequent image processing.

[0091] The core of the classification module 402 is a preset lightweight convolutional neural network (CNN), which is specially designed to extract features from the target eye image and determine the state of the eye (open or closed). The lightweight convolutional neural network includes 3 convolutional layers and 2 maximum pooling layers. Through the superposition of these layers, the network can extract enough features from the original pixels to capture eye details. In addition, the classification module also includes a global average pooling layer and a linear module, which includes 2 consecutive fully connected layers.

[0092] The recognition module 403 performs the recognition of blink events based on the eye state probability output by the classification module. In other words, this module integrates Figure 4 Through sophisticated algorithm calculation, it determines which consecutive eye closure states constitute actual blink events, thereby accurately recording the number and frequency of blinks.

[0093] In addition, a display interface is provided. Figure 4 , the display interface ( Figure 4 The result display in the interface is used to display the detection results of the eye state detection device, and the detection results include the number of blinks and the blink rate within a preset time period. This interface not only directly reflects the numerical results of eye monitoring, but also instantly reminds the user of the eye condition through visual signals, which increases user friendliness and interactivity. Figure 6 is a schematic diagram of a display result of a display interface in an exemplary embodiment, showing the display of the GUI under several typical conditions. In the upper left corner of the interface is a blink indicator, which lights up when a new blink event is detected ( Figure 6 b). The total number of blinks and blink rate indicators are displayed and updated in real time. In the upper right corner of the interface is the eye fatigue indicator, which is usually displayed in green. When it detects that you have not blinked for a long time, it will turn red to remind the user to relax their eyes ( Figure 6 c). The time threshold is set to 10 seconds and can be adjusted according to specific needs. The fatigue indication signal can also be used as a control signal to trigger other software or devices, such as a buzzer. These intuitive interface designs enable users to easily understand and utilize the results of blink detection, both in daily life and in professional situations that require continuous monitoring of eye status.

[0094] Please refer to the following Figure 7, which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 7 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the eye state detection method provided in any of the aforementioned embodiments of the present application when running the computer program.

[0095] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0096] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The control method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0097] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the eye state detection method in combination with its hardware.

[0098] An embodiment of the present application also provides a computer-readable storage medium corresponding to the eye state detection method provided in the aforementioned embodiment, on which a computer program is stored. When the computer program is run by a processor, it will execute the eye state detection method provided in any of the aforementioned embodiments.

[0099] In addition, examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0100] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the eye state detection method provided by any of the aforementioned embodiments, the method comprising: acquiring multiple frames of target eye images; inputting each frame of the target eye image into a trained preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers; and identifying blinking events based on the probability of the eye state.

[0101] Those skilled in the art will appreciate that the various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components of the virtual machine creation device according to the embodiments of the present application.

[0102] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting eye status, characterized in that: The method comprises: Acquire multiple frames of target eye images; Input each frame of the target eye image into a trained preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers; Blink events are identified based on the probabilities of the eye states.

2. The eye state detection method according to claim 1, characterized in that: The preset classification network includes a global average pooling layer; The step of inputting the target eye image into a preset classification network to obtain a probability of an eye state corresponding to the target eye image includes: Inputting the target eye image into the feature extraction module to extract features and obtain a first feature map; Inputting the first feature map into the global average pooling layer for pooling operation to obtain a second feature map; A normalization operation is performed based on the second feature map to obtain the probability of the eye state corresponding to the target eye image.

3. The eye state detection method according to claim 2, characterized in that: The preset classification network further includes a linear module, and the linear module includes two consecutive fully connected layers; the normalization operation based on the second feature map is performed to obtain the probability of the eye state corresponding to the target eye image, including: Input the second feature map into the linear module and output a target vector; The target vector is normalized to obtain the probability of the eye being in different eye states.

4. The eye state detection method according to claim 1, characterized in that: The method of identifying a blink event based on the probability of the eye state comprises: For the target eye image of the current frame, compare the probability of the eye state corresponding to the target eye image of the current frame with a first probability threshold, and if the probability of the eye state is greater than or equal to the first probability threshold, determine that the eye state is a closed state; Determine a second probability threshold by using N frames of images before the target eye image of the current frame, where N is greater than or equal to 1; The probability of the eye state is compared with the second probability threshold, and a blink event is identified based on the comparison result.

5. The eye state detection method according to claim 4, characterized in that: The determining the second probability threshold through the previous N frames of the target eye image of the current frame includes: According to the probability of the eye state corresponding to each frame in the previous N frames of images, the average of the eye state probabilities of the previous N frames is calculated; Based on the average value and a preset offset, the second probability threshold is obtained.

6. The eye state detection method according to claim 4, characterized in that: The step of comparing the probability of the eye state with the second probability threshold and identifying a blink event according to the comparison result includes: If the probability of the eye state is greater than or equal to the second probability threshold, the target eye image of the current frame is determined to be in a closed state, and it is determined that the target eye image of the current frame corresponds to a blink event.

7. The eye state detection method according to claim 1, characterized in that: The training method of the preset classification network includes: Obtaining a data set, wherein the data set includes a training set and a validation set; Training the preset classification network through the training set, and updating the parameters of the preset classification network by using a back propagation algorithm; Using a preset optimizer to optimize the weights of the preset classification network and minimize the prediction error calculated by the cross entropy loss function; After each training cycle, the performance of the preset classification network is evaluated using the validation set, and the training is terminated after a preset number of cycles are completed and the preset classification network reaches convergence.

8. The eye state detection method according to claim 1, characterized in that: The step of acquiring a target eye image comprises: receiving eye video captured by a camera; Extract multiple frames of continuous eye images from the eye video, and use the multiple frames of continuous eye images as multiple frames of target eye images.

9. An eye state detection device, characterized in that: The device comprises: An acquisition module, used for acquiring a target eye image; A classification module, used for inputting the target eye image into a preset classification network to obtain the probability of the eye state corresponding to the target eye image; wherein the preset classification network is a lightweight convolutional neural network, and the lightweight convolutional neural network includes a feature extraction module, and the feature extraction module includes 3 convolutional layers and 2 maximum pooling layers; A recognition module is used to recognize a blink event based on the probability of the eye state.

10. A display interface, characterized in that: The display interface is used to display the detection result of the eye state detection device according to claim 9, and the detection result includes the number of blinks and the blink rate within a preset time period.

11. An electronic device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the eye state detection method as described in any one of claims 1-8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the eye state detection method as described in any one of claims 1-8 is implemented.

Images