Blink fatigue detection method and system based on deep learning

Abstract

The invention discloses a blink fatigue detection method and system based on deep learning, and the method comprises the steps: obtaining an eye image data set with an eye opening and closing state label, carrying out the preprocessing, and dividing a training set, a verification set, and a calibration data set; performing training and performance verification on the convolutional neural network model by using the training set and the verification set to obtain a blink detection model; the calibration data set is used for conducting INT8 quantification processing on the blink detection model, and the blink detection model is converted into a light-weight blink detection model and then deployed to an embedded device; taking a continuous video frame sequence which is acquired by a camera in real time and contains a face image as input data of a lightweight blink detection model for detection, and obtaining an effective blink frequency; the number of effective blinking times is counted, the blinking frequency in unit time is calculated, the blinking frequency is compared with a preset multi-level fatigue level threshold value level by level to determine the corresponding fatigue level, and whether the user is in a fatigue state or not is judged. According to the invention, blink fatigue detection can be efficiently and accurately carried out on low-power-consumption equipment.

Description

Technical Field

[0001] This invention relates to the technical field of image recognition and embedded intelligent sensing, and in particular to a method, system, storage medium and computing device for blink fatigue detection based on deep learning. Background Technology

[0002] Blinking fatigue detection methods determine the fatigue level of human eyes by capturing abnormal changes in blinking behavior. This is of great significance for preventing excessive eye fatigue and reducing the incidence of problems such as myopia and dry eye.

[0003] Currently, there are three main methods for detecting blink fatigue: 1. Methods based on grayscale thresholding and region segmentation: This method determines the approximate eye area through face localization, then uses the grayscale difference between the pupil, iris, and sclera to extract the eye contour through thresholding, and statistically analyzes the grayscale changes during eyelid opening and closing to determine blinking. 2. Methods based on Haar-like features and an Adaboost classifier: This method quickly detects the face and locates the eye area using a trained Adaboost classifier, then tracks changes in the area and aspect ratio of the eye region to identify blinking, and then statistically analyzes the frequency and duration to determine fatigue. 3. Methods based on eye feature point matching: This method captures changes in the photoelectroelectric volume of the skin around the eyes using a camera. Fatigue manifests as a slowed heart rate and abnormal pulse rhythm, which, combined with visual blinking features, provides a comprehensive assessment of fatigue levels. Overall, existing blink fatigue detection methods, such as those based on grayscale thresholding and region segmentation, suffer from high computational complexity and a high false detection rate. The drawback of methods based on Haar-like features and the Adaboost classifier is that they are easily affected by other facial expression changes, leading to low accuracy in blink detection. In summary, high-precision blink fatigue detection methods require high-end hardware, increasing costs and limiting the widespread application of the technology. Summary of the Invention

[0004] The primary objective of this invention is to provide a deep learning-based method for detecting blink fatigue, which effectively addresses the high equipment and environmental requirements of traditional blink fatigue detection methods. By constructing a custom convolutional neural network model, it maintains high accuracy even in environments with smaller size, lower computational load, and lower hardware configuration. It can be quickly deployed without high-end hardware, solving the problems of high hardware requirements and low detection accuracy in existing blink fatigue detection methods. It can perform blink fatigue detection efficiently and accurately on low-power devices, demonstrating high practicality and engineering application value.

[0005] The second objective of this invention is to provide a blink fatigue detection system based on deep learning.

[0006] A third objective of this invention is to provide a storage medium.

[0007] A fourth objective of this invention is to provide a computing device.

[0008] The first objective of this invention is achieved through the following technical solution: a method for detecting blink fatigue based on deep learning, comprising the following steps:

[0009] S1: Obtain an eye image dataset with labels for open and closed eye states and preprocess it to obtain an eye image dataset containing eye images of uniform size and corresponding labels for open and closed eye states. Divide the preprocessed eye image dataset into a training set, a validation set, and a calibration dataset for subsequent INT8 quantization processing according to the proportion.

[0010] S2: Construct a custom convolutional neural network (CNN) model. This CNN model uses a three-layer concatenated architecture to extract hierarchical features from the input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the CNN model is the probability value of the eyes being open, which represents the likelihood of the eyes being open. Iterative training and performance verification of the CNN model are performed using training and validation sets to obtain the optimal CNN model as the blink detection model.

[0011] S3: Use the calibration dataset to perform INT8 quantization on the blink detection model, and then use the TensorFlow Lite converter to convert the blink detection model after INT8 quantization into a lightweight blink detection model suitable for inference on embedded devices, and deploy it to the embedded device.

[0012] S4: After deploying the lightweight blink detection model in the embedded device, the embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than the preset threshold for more than a set number of times, it is determined that the eye is currently in an open state; otherwise, it is in a closed state. When the eye successively experiences the open state, the closed state and returns to the open state, and the duration of the closed state conforms to the time characteristics of natural human blinking, it is recorded as a valid blink count.

[0013] S5: According to the preset time statistics period, the effective blink count obtained in step S4 is counted, and the blink frequency per unit time is calculated. The blink frequency is compared with the preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

[0014] Furthermore, in step S1, the preprocessing is specifically as follows:

[0015] Load raw eye images from an eye image dataset containing image files labeled with two eye states: "awake" and "sleepy," where "awake" is the label for open eyes and "sleepy" is the label for closed eyes.

[0016] The pixel values ​​of the loaded original eye image are normalized from an integer range using a normalization formula. Linear transformation to floating point range This can be achieved through the following formula:

[0017] ;

[0018] In the formula, This represents the tensor of the original eye image. This represents a normalized floating-point tensor;

[0019] A data generator is constructed with a fixed batch size of a preset threshold and a uniform target size of 48×48 pixels. First, the image resizing operation is implemented using a bilinear interpolation algorithm, as shown in the following formula:

[0020] ;

[0021] In the formula, These represent the width and height of the original eye image, respectively. and These represent the coordinates of the target pixel in the scaled eye image. This represents the horizontal coordinate value of the target pixel. This represents the coordinate value of the target pixel in the vertical direction. This is an index variable used to iterate through four neighboring pixels. For two points in the corresponding horizontal direction, For two points in the corresponding vertical direction, These are the bilinear interpolation weighting coefficients. Indicates the image in coordinates Pixel value at;

[0022] After unifying the image size, the data generator parameters are further configured, that is, the category mode of the dataset is set to binary classification to adapt to the binary classification task with eyes open and closed.

[0023] Furthermore, in step S2, the convolutional neural network model includes:

[0024] The input layer receives eye image data of size 48×48×3 and serves as the entry point for the model to pass data to the next layer.

[0025] The first convolutional layer consists of eight 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to extract primary features, and reduces the spatial dimension through a 2×2 max pooling layer to output a 23×23×8 feature map.

[0026] The second convolutional layer consists of 16 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to capture intermediate semantic features. It is followed by a 2×2 max pooling layer to further reduce the feature map size and output a 10×10×16 feature map.

[0027] The third convolutional layer consists of 32 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to learn high-level abstract features. It is followed by a 2×2 max pooling layer to reduce the size of the feature map, and finally outputs a 4×4×32 feature map.

[0028] The classification head first flattens the 4×4×32 feature map output from the third convolutional layer into a 512-dimensional vector through a flattening layer. Then, the feature is combined through a fully connected layer. After the fully connected layer, there is a regularization layer to improve the model's generalization ability. Finally, the output layer outputs the probability value of the eye being in an open state.

[0029] Furthermore, in step S2, the convolutional neural network model employs a binary cross-entropy loss function and the Adam optimizer during training to minimize prediction error. It also terminates training when model performance reaches its optimal state through an early stopping strategy and a model checkpoint callback mechanism, thereby obtaining the optimal convolutional neural network model as the blink detection model. The optimized configuration used by the convolutional neural network model during training is as follows:

[0030] The Adam optimizer with optimized parameters is used, and the loss function is binary cross-entropy. Its complete mathematical expression is defined as follows:

[0031] ;

[0032] In the formula, N represents the total number of samples in the current training batch. Used to iterate through each sample in the eye image dataset, from the 1st to the Nth sample; Representing the The true label of each sample As a representative model for the first The predicted probability of a sample.

[0033] Furthermore, step S3 includes the following steps:

[0034] S31: Load the blink detection model using the TensorFlow Lite converter, enable the default optimization strategy for model quantization, and configure the model operation unit to use the built-in operator that supports INT8 integer operations to ensure that all computational operations in the model meet the integer quantization inference requirements; at the same time, configure the inference input data type and output data type of the blink detection model to unsigned 8-bit integer type to obtain the blink detection model after INT8 quantization.

[0035] S32: After completing the configuration in step S31, the blink detection model after INT8 quantization is calibrated using the calibration dataset. The dynamic range of the weight parameters and the output feature values ​​of each convolutional layer after the activation function is applied is statistically analyzed. Based on the dynamic range, the corresponding quantization scaling factor and zero offset are determined.

[0036] S33: Quantize the weight parameters and intermediate activation values ​​in the blink detection model, which are represented by 32-bit floating-point numbers, into 8-bit integers to output a lightweight blink detection model suitable for inference on embedded devices.

[0037] Furthermore, the cropped eye image undergoes preprocessing, as follows:

[0038] The cropped eye image is normalized in size and then scaled to the standard input size of the lightweight blink detection model using an interpolation algorithm.

[0039] Subsequently, the eye image is processed for format conversion. Based on the preset weights of the three color components (red, green, and blue), the color information of each pixel in the color image is weighted and fused to convert the color image into a single-channel grayscale image.

[0040] After format conversion, the pixel values ​​of the obtained single-channel grayscale image are copied to the red, green, and blue channels simultaneously, and the output is an RGB three-channel image that meets the input requirements of the lightweight blink detection model.

[0041] Furthermore, in step S5, when counting the number of effective blinks per unit time, the time period for counting the number of effective blinks can be adjusted according to actual needs. The blink frequency per unit time is calculated using the following formula:

[0042] ;

[0043] In the formula, The set time period for statistics, in seconds. In the time statistical period The number of valid blinks detected internally. Indicates the time statistical period The frequency of blinking within the eye.

[0044] The second objective of this invention is achieved through the following technical solution: a deep learning-based blink fatigue detection system, used to implement the aforementioned deep learning-based blink fatigue detection method, comprising:

[0045] The data acquisition and preprocessing module is used to acquire and preprocess eye image datasets with eye open and closed state labels to obtain eye image datasets containing eye images of uniform size and corresponding eye open and closed state labels. The preprocessed eye image dataset is divided into training set, validation set and calibration dataset for subsequent INT8 quantization processing according to the proportion.

[0046] A blink detection model acquisition module is used to construct a custom convolutional neural network (CNN) model. This CNN model employs a three-layer concatenated architecture to hierarchically extract input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the CNN model is the probability value of the eyes being open, representing the likelihood of the eyes being open. The CNN model is iteratively trained and its performance verified using a training set and a validation set to obtain the optimal CNN model as the blink detection model.

[0047] The lightweight blink detection model acquisition module uses a calibration dataset to perform INT8 quantization on the blink detection model, and then uses the TensorFlow Lite converter to convert the blink detection model after INT8 quantization into a lightweight blink detection model suitable for inference on embedded devices, and deploys it to the embedded device.

[0048] The effective blink count definition module is used after the lightweight blink detection model is deployed in the embedded device. The embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than a preset threshold for more than a set number of times, the eye is determined to be in an open state. Otherwise, it is in a closed state. When the eye goes through an open state, a closed state and returns to an open state in sequence, and the duration of the closed state conforms to the time characteristics of a natural human blink, it is recorded as one effective blink count.

[0049] The fatigue assessment module is used to accumulate and count the number of valid blinks obtained according to a preset time statistical period, calculate the blink frequency per unit time, and compare the blink frequency with a preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

[0050] The third objective of this invention is achieved through the following technical solution: a storage medium storing a program, which, when executed by a processor, implements the above-mentioned deep learning-based blink fatigue detection method.

[0051] The fourth objective of this invention is achieved through the following technical solution: a computing device, including a processor and a memory for storing processor-executable programs, wherein when the processor executes the program stored in the memory, it implements the above-mentioned deep learning-based blink fatigue detection method.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] 1. This invention designs a custom convolutional neural network model for the application scenario of blink fatigue detection. By performing targeted modeling of local eye features, it improves the accuracy and stability of eye opening and closing state discrimination and has stronger task adaptability compared with general network models.

[0054] 2. This invention uses INT8 quantization to compress the convolutional neural network model and combines it with the TensorFlow Lite converter to generate a lightweight model suitable for embedded devices. While ensuring detection accuracy, it significantly reduces the model's storage footprint and computational complexity, and improves the model's running efficiency and hardware adaptability.

[0055] 3. This invention effectively reduces the accuracy loss caused by model fixed-pointing through quantization calibration and inference optimization, enabling the quantized model to maintain stable and reliable blink detection performance in resource-constrained embedded environments.

[0056] 4. The blink fatigue detection method constructed in this invention adopts a modular design concept, decoupling functions such as image acquisition, eye state recognition, blink statistics and fatigue assessment, which improves portability and scalability, and can flexibly adapt to different hardware platforms and various application scenarios.

[0057] 5. This invention combines real-time status analysis and time-series statistics to continuously track and analyze the frequency of blinking behavior, avoiding misjudgment problems caused by single-frame judgment, making fatigue detection results more stable and reliable, and suitable for long-term, continuous monitoring needs.

[0058] 6. This invention effectively reduces power consumption and resource consumption while ensuring real-time performance. It can operate stably in environments with limited hardware conditions, such as mobile terminals, embedded devices, and edge computing devices, and has high engineering practical value and promising prospects for widespread application.

[0059] In summary, this invention combines a custom convolutional neural network model with INT8 quantization to improve the accuracy and real-time performance of blink detection while also taking into account model compression and deployment efficiency. It effectively solves the problems of insufficient accuracy or excessive resource consumption of traditional methods in embedded and mobile scenarios. It can also achieve efficient and reliable blink fatigue monitoring under low hardware configuration conditions, which has practical application value and is worth promoting. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention.

[0061] Figure 2 This is a diagram of a custom convolutional neural network model architecture.

[0062] Figure 3 This is an architecture diagram of the system of the present invention. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0064] Example 1

[0065] This embodiment discloses a deep learning-based method for detecting blink fatigue, such as... Figure 1 As shown, it includes the following steps:

[0066] 1) Obtain an eye image dataset with labels for open and closed eye states and preprocess it to obtain an eye image dataset containing eye images of uniform size and corresponding labels for open and closed eye states. Divide the preprocessed eye image dataset into a training set, a validation set, and a calibration dataset for subsequent INT8 quantization processing according to the proportion.

[0067] 2) Construct a custom convolutional neural network model. This model uses a three-layer concatenated architecture to extract hierarchical features from the input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the convolutional neural network model is the probability value of the eyes being open, which represents the likelihood of the eyes being open. Iterative training and performance verification of the convolutional neural network model are performed using training and validation sets to obtain the optimal convolutional neural network model as the blink detection model.

[0068] 3) The blink detection model is quantized using the calibration dataset, and then the blink detection model after INT8 quantization is converted into a lightweight blink detection model suitable for inference on embedded devices using the TensorFlow Lite converter, and then deployed to the embedded device.

[0069] 4) After deploying the lightweight blink detection model in the embedded device, the embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than the preset threshold for more than a set number of times, it is determined that the eye is currently in an open state; otherwise, it is in a closed state. When the eye successively experiences the open state, the closed state and returns to the open state, and the duration of the closed state conforms to the time characteristics of natural human blinking, it is recorded as a valid blink count.

[0070] 5) According to the preset time statistics period, the effective blink count obtained in step 4) is counted, and the blink frequency per unit time is calculated. The blink frequency is compared with the preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

[0071] Specifically, the eye image dataset labeled with eye open / closed status is preprocessed as follows:

[0072] Load raw eye images from an eye image dataset containing image files labeled with two eye states: "awake" and "sleepy," where "awake" is the label for open eyes and "sleepy" is the label for closed eyes.

[0073] The pixel values ​​of the loaded original eye image are normalized from an integer range using a normalization formula. Linear transformation to floating point range This can be achieved through the following formula:

[0074] ;

[0075] In the formula, This represents the tensor of the original eye image. This represents a normalized floating-point tensor;

[0076] A data generator is constructed with a fixed batch size of a preset threshold and a uniform target size of 48×48 pixels. First, the image resizing operation is implemented using a bilinear interpolation algorithm, as shown in the following formula:

[0077] ;

[0078] In the formula, These represent the width and height of the original eye image, respectively. and These represent the coordinates of the target pixel in the scaled eye image. This represents the horizontal coordinate value of the target pixel. This represents the coordinate value of the target pixel in the vertical direction. This is an index variable used to iterate through four neighboring pixels. For two points in the corresponding horizontal direction, For two points in the corresponding vertical direction, These are the bilinear interpolation weighting coefficients. Indicates the image in coordinates Pixel value at;

[0079] After unifying the image size, the data generator parameters are further configured, that is, the category mode of the dataset is set to binary classification to adapt to the binary classification task with eyes open and closed.

[0080] like Figure 2 As shown, the designed custom convolutional neural network model is described in detail below:

[0081] The convolutional neural network model architecture adopts a three-layer convolutional layer cascaded architecture, uses a binary cross-entropy loss function and Adam optimizer to minimize the prediction error, and uses the optimal custom convolutional neural network model as the final blink detection model.

[0082] The convolutional neural network model has a total of 71,954 learnable parameters. The specific parameters of each layer and their corresponding operator types are shown in Table 1.

[0083] Table 1 Detailed parameter information

[0084]

[0085] Conv2D is the core feature extraction operator. It performs sliding weighted operations on the input image through a specific convolution kernel, aiming to capture basic visual features such as edges and textures, and extract advanced semantic features as the layers deepen.

[0086] MaxPooling2D is a spatial downsampling operator that follows each convolutional layer. It compresses the feature map dimension by extracting the maximum value in a local region. This layer reduces computational complexity, enhances the model's robustness to changes in feature location, and effectively alleviates memory pressure on embedded systems.

[0087] Flatten is a format conversion operator responsible for flattening the multidimensional feature matrix output by the terminal pooling layer into a one-dimensional vector, thereby achieving a smooth transition from spatial feature extraction logic to fully connected classification logic.

[0088] Dense is a feature combination and classification operator that learns complex nonlinear relationships between features through weighted connections between neurons.

[0089] Dropout is a regularization operator that suppresses overfitting by randomly deactivating some neurons during the training phase, thus enabling the network to learn features with better generalization performance and stability.

[0090] The convolutional neural network model specifically includes:

[0091] The input layer is used to receive eye image data of size 48×48×3.

[0092] The first convolutional layer consists of eight 3×3 convolutional kernels, using the ReLU activation function and valid padding, followed by a 2×2 max pooling layer, outputting a 23×23×8 feature map.

[0093] The second convolutional layer consists of 16 3×3 convolutional kernels, using the ReLU activation function and valid padding, followed by a 2×2 max pooling layer, outputting a 10×10×16 feature map.

[0094] The third convolutional layer consists of 32 3×3 convolutional kernels, using the ReLU activation function and valid padding, followed by a 2×2 max pooling layer, outputting a 4×4×32 feature map.

[0095] The classification head flattens the features into a 512-dimensional vector through a flattening layer, followed by a 128-node fully connected layer, a regularization layer with a 0.5 dropout rate, and finally generates open / closed state probability values ​​through the output layer.

[0096] The model training process uses the Adam optimizer, with the following core parameters configured: initial learning rate set to 0.001, and first-order momentum decay coefficient... Set to 0.9, second-order momentum decay coefficient Setting it to 0.99 and epsilon to 1e-07, this parameter combination ensures sufficient convergence speed in the early stages of training, while maintaining stable parameter updates in the later stages of training.

[0097] The Adam optimizer is an optimization algorithm used in the model training phase. It combines the momentum method with an adaptive learning rate strategy to achieve efficient updates of model parameters. During parameter updates, it considers both the direction of the gradient and the magnitude of historical gradients, thereby dynamically assigning different learning rates to each parameter. This mechanism accelerates model convergence and reduces reliance on hyperparameter tuning.

[0098] The loss function uses binary cross-entropy. Its complete mathematical expression is defined as follows:

[0099] ;

[0100] In the formula, N represents the total number of samples in the current training batch. Used to iterate through each sample in the eye image dataset, from the 1st to the Nth sample; Representing the The true label of each sample As a representative model for the first The predicted probability of a sample.

[0101] Through the binary cross-entropy It can effectively penalize the model for misclassifying positive and negative samples, especially applying greater gradient correction to high-confidence mispredictions; it monitors validation set metrics through early stopping and terminates training when the convolutional neural network model reaches its optimal performance, preventing overfitting and preserving the generalization ability to new open and closed eye images; and it saves its optimal parameters through a model checkpoint callback mechanism, improving the efficiency and reliability of the model.

[0102] The training cycle is configured to 50 rounds. This round setting is based on early observations of validation set performance saturation, which can ensure that the model converges fully while avoiding overfitting. The batch size is fixed at 32.

[0103] Specifically, the blink detection model is quantized using INT8, and a lightweight blink detection model suitable for inference on embedded devices is generated by utilizing a calibration dataset and a TensorFlowLite converter, as detailed below:

[0104] a. The blink detection model is quantized and adapted for embedded deployment using the TensorFlow Lite model converter. During this process, TensorFlow Lite's default overall optimization strategy is enabled to reduce the model's storage size and computational complexity. Specific steps include:

[0105] First, the model's input and output data types are uniformly set to unsigned 8-bit integers to ensure end-to-end integer quantization inference throughout the entire process of input, computation, and output.

[0106] Ultimately, the model is limited to using only the basic set of operators that support bitwise integer operations to ensure that all computational operations in the model can be performed in 8-bit integer form;

[0107] b. Calibrate the quantization parameters using the calibration dataset. This is mainly done by determining the dynamic range of the 32-bit floating-point weights and activation values ​​of the blink detection model to ensure minimal loss of numerical accuracy. The specific process is as follows:

[0108] The maximum and minimum values ​​of the 32-bit floating-point weights and activation values ​​of the blink detection model are statistically analyzed and then converted into 8-bit integer values ​​through a linear mapping. The mathematical expression for this process is as follows:

[0109] ;

[0110] in, `q` represents the original 32-bit floating-point value, i.e., the input weight or activation value; `q` represents the quantized 8-bit integer value; the `round()` function rounds a number to the nearest integer; and `scale` is the scaling factor, used to control the scaling ratio from floating-point to integer. Its corresponding mathematical expression is as follows:

[0111] ;

[0112] In the formula, max is the maximum value in the original floating-point dataset, min is the minimum value in the original floating-point dataset, and Q... max Q is the maximum value of the quantized integer. min This is the minimum value of the quantized integer;

[0113] zero_point is the zero-point offset, which ensures the symmetry of the quantization process and reduces quantization error;

[0114] ;

[0115] In the formula, min has the same meaning as in the scale formula, representing the minimum value in the original floating-point dataset, Q. min The meaning is consistent with the scale formula, which expresses the minimum value of the quantized integer. The round() function is used to round a number to the nearest integer.

[0116] c. After quantizing all 32-bit floating-point weights and activation values ​​in the blink detection model into 8-bit integers, a lightweight blink detection model suitable for embedded devices is generated using the TensorFlow Lite converter. Compared with the full-precision model, the quantized model saves approximately 71.8% of storage space, significantly reducing the model's storage space and memory usage.

[0117] Specifically, after deploying the lightweight blink detection model on an embedded device, the blink detection process is as follows:

[0118] a. Real-time facial video stream is captured via a camera. Eye key point information is obtained using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye region is precisely cropped based on this key point information. The cropped eye image is then preprocessed and input into a lightweight blink detection model for inference to obtain the probability value of the eyes being open. The specific preprocessing of the cropped eye image is as follows:

[0119] The cropped eye image is scaled to a standard input size of 48×48 pixels using a bilinear interpolation algorithm. The pixel value calculation during the scaling process follows the same bilinear interpolation algorithm formula as described above.

[0120] Then, pixel format conversion is performed to convert the original image to a grayscale image according to the following conversion formula, which adopts the ITU-R BT.601 standard. The specific formula is as follows:

[0121] ;

[0122] In the formula, H, as the final output, is an 8-bit unsigned integer ranging from [0, 255], representing the final grayscale value perceived by the human eye; R, G, and B are the three primary color components, and are used as input parameters, representing the red, green, and blue intensity values ​​derived from the RGB565 format, respectively, and precision conversion is achieved through specific bit operations; specifically, the red component is mapped from 5 bits to 8 bits through (r << 3) | (r >> 2), the green component is converted from 6 bits to 8 bits through (g << 2) | (g >> 4), and the blue component is converted through (b << 3) | (b >> 3) | (b >> 3) | (b >> 3) | (b >> 3) | (b >> 3) | (b << ... 2) Expand from 5 bits to 8 bits; the three floating-point constants 0.299, 0.587, and 0.114 are standard weighting coefficients, which accurately reflect the physiological characteristics of the human visual system, which is most sensitive to green, followed by red, and least sensitive to blue; 77, 150, and 29 are fixed-point approximation coefficients, which are optimized values ​​obtained by multiplying the original floating-point coefficients by 256 and rounding them down, so that 77 is approximately equal to 0.299×256, 150 is approximately equal to 0.587×256, and 29 is approximately equal to 0.114×256;

[0123] The right shift by 8 bits is an efficient alternative to division. In embedded systems, it achieves the mathematical effect of division by 256 through simple bit operations, converting the result of fixed-point calculation back to the standard numerical range.

[0124] Subsequently, the single-channel grayscale image is converted to a three-channel RGB888 format using a channel copying method. The specific formula is as follows:

[0125]

[0126] In the formula, , , These represent the intensity values ​​Y of red, green, and blue, respectively, derived from the RGB565 format. gray This refers to the grayscale brightness value.

[0127] Finally, the preprocessed eye image is input into a lightweight blink detection model for real-time inference to obtain the probability value of the eye being in an open state.

[0128] b. The eye opening and closing status in each frame of the image is initially determined by a preset threshold of 0.9. Based on this, the detected open eye status must last for at least 2 frames to be confirmed as a valid open eye, so as to avoid errors caused by misjudgment in a single frame.

[0129] c. A state machine monitoring method is used to continuously track changes in eye state. The state machine monitoring method is: when the eye sequentially goes through an open state, a closed state and returns to an open state, it is considered a blinking action.

[0130] d. Measure the duration of the closed eye state. Only when the duration of the closed eye state is within the range of [100ms, 400ms] is it considered as a blink action, so as to ensure that it conforms to the time characteristics of the natural human blink and exclude excessively fast blinking or excessively long eye closing behavior.

[0131] e. Only if steps c and d are satisfied simultaneously will the action be considered a valid blink event, i.e., one valid blink count.

[0132] Specifically, the time statistics period is preset to 60 seconds. The number of valid blinks is counted, and the blink frequency per unit time is calculated. The time statistics period for counting the number of valid blinks can be adjusted according to actual needs. The blink frequency per unit time is calculated using the following formula:

[0133] ;

[0134] In the formula, The set time period for statistics, in seconds. In the time statistical period The number of valid blinks detected internally. Indicates the time statistical period The frequency of blinking within the eye;

[0135] The calculated blink frequency is compared step-by-step with preset multi-level fatigue thresholds, following a descending order of comparison logic. The blink frequency is matched with each threshold to determine the corresponding fatigue level. The multi-level fatigue thresholds and their corresponding fatigue levels are shown in Table 2. When the fatigue level is greater than or equal to 1, the user is determined to be fatigued, and the corresponding fatigue level and frequency information are recorded, completing the full closed loop from blink detection to fatigue assessment.

[0136] Table 2. Correspondence between multi-level fatigue thresholds and fatigue grades

[0137]

[0138] Experimental conclusion:

[0139] To address the shortcomings of existing blink fatigue detection algorithms, such as insufficient attention to key information across modalities, high computational cost, high false positive rate, susceptibility to facial expressions, and demanding hardware requirements, this invention employs a fully custom network model structure. This results in smaller size, less computation, and lower power consumption while maintaining high accuracy, effectively solving the problem of high equipment requirements in traditional fatigue detection. Experiments show that this method can be quickly deployed without high-end hardware, improving the accuracy and convenience of fatigue detection compared to existing methods. Future research will explore the adaptability of this method under complex environments with varying lighting and occlusion conditions, as well as its potential for integration with physiological signals beyond visual blink characteristics. This approach shows great promise and is worthy of widespread adoption.

[0140] Example 2

[0141] This embodiment discloses a deep learning-based blink fatigue detection system to implement the deep learning-based blink fatigue detection method described in Embodiment 1, such as... Figure 3 As shown, it includes the following functional modules:

[0142] The data acquisition and preprocessing module is used to acquire and preprocess eye image datasets with eye open and closed state labels to obtain eye image datasets containing eye images of uniform size and corresponding eye open and closed state labels. The preprocessed eye image dataset is divided into training set, validation set and calibration dataset for subsequent INT8 quantization processing according to the proportion.

[0143] A blink detection model acquisition module is used to construct a custom convolutional neural network (CNN) model. This CNN model employs a three-layer concatenated architecture to hierarchically extract input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the CNN model is the probability value of the eyes being open, representing the likelihood of the eyes being open. The CNN model is iteratively trained and its performance verified using a training set and a validation set to obtain the optimal CNN model as the blink detection model.

[0144] The lightweight blink detection model acquisition module uses a calibration dataset to perform INT8 quantization on the blink detection model, and then uses the TensorFlow Lite converter to convert the blink detection model after INT8 quantization into a lightweight blink detection model suitable for inference on embedded devices, and deploys it to the embedded device.

[0145] The effective blink count definition module is used after the lightweight blink detection model is deployed in the embedded device. The embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than a preset threshold for more than a set number of times, the eye is determined to be in an open state. Otherwise, it is in a closed state. When the eye goes through an open state, a closed state and returns to an open state in sequence, and the duration of the closed state conforms to the time characteristics of a natural human blink, it is recorded as one effective blink count.

[0146] The fatigue assessment module is used to accumulate and count the number of valid blinks obtained according to a preset time statistical period, calculate the blink frequency per unit time, and compare the blink frequency with a preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

[0147] Example 3

[0148] This embodiment discloses a storage medium storing a program, which, when executed by a processor, implements the deep learning-based blink fatigue detection method described in Embodiment 1.

[0149] The storage medium in this embodiment can be a disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), USB flash drive, portable hard drive, etc.

[0150] Example 4

[0151] This embodiment discloses a computing device, including a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the deep learning-based blink fatigue detection method described in Embodiment 1.

[0152] The computing device described in this embodiment may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer, programmable logic controller (PLC), or other terminal device with processor function.

[0153] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for detecting blink fatigue, characterized in that, Includes the following steps: S1: Obtain an eye image dataset with labels for open and closed eye states and preprocess it to obtain an eye image dataset containing eye images of uniform size and corresponding labels for open and closed eye states. Divide the preprocessed eye image dataset into a training set, a validation set, and a calibration dataset for subsequent INT8 quantization processing according to the proportion. S2: Construct a custom convolutional neural network (CNN) model. This CNN model uses a three-layer concatenated architecture to extract hierarchical features from the input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the CNN model is the probability value of the eyes being open, which represents the likelihood of the eyes being open. Iterative training and performance verification of the CNN model are performed using training and validation sets to obtain the optimal CNN model as the blink detection model. S3: Use the calibration dataset to perform INT8 quantization on the blink detection model, and then use the TensorFlow Lite converter to convert the blink detection model after INT8 quantization into a lightweight blink detection model suitable for inference on embedded devices, and deploy it to the embedded device. S4: After deploying the lightweight blink detection model in the embedded device, the embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than the preset threshold for more than a set number of times, it is determined that the eye is currently in an open state; otherwise, it is in a closed state. When the eye successively experiences the open state, the closed state and returns to the open state, and the duration of the closed state conforms to the time characteristics of natural human blinking, it is recorded as a valid blink count. S5: According to the preset time statistics period, the effective blink count obtained in step S4 is counted, and the blink frequency per unit time is calculated. The blink frequency is compared with the preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

2. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, In step S1, the preprocessing is specifically as follows: Load raw eye images from an eye image dataset containing image files labeled with two eye states: "awake" and "sleepy," where "awake" is the label for open eyes and "sleepy" is the label for closed eyes. The pixel values ​​of the loaded original eye image are normalized from an integer range using a normalization formula. Linear transformation to floating point range This can be achieved through the following formula: ； In the formula, This represents the tensor of the original eye image. This represents a normalized floating-point tensor; A data generator is constructed with a fixed batch size of a preset threshold and a uniform target size of 48×48 pixels. First, the image resizing operation is implemented using a bilinear interpolation algorithm, as shown in the following formula: ； In the formula, These represent the width and height of the original eye image, respectively. and These represent the coordinates of the target pixel in the scaled eye image. This represents the horizontal coordinate value of the target pixel. This represents the coordinate value of the target pixel in the vertical direction. This is an index variable used to iterate through four neighboring pixels. For two points in the corresponding horizontal direction, For two points in the corresponding vertical direction, These are the bilinear interpolation weighting coefficients. Indicates the image in coordinates Pixel value at; After unifying the image size, the data generator parameters are further configured, that is, the category mode of the dataset is set to binary classification to adapt to the binary classification task with eyes open and closed.

3. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, In step S2, the convolutional neural network model includes: The input layer receives eye image data of size 48×48×3 and serves as the entry point for the model to pass data to the next layer. The first convolutional layer consists of eight 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to extract primary features, and reduces the spatial dimension through a 2×2 max pooling layer to output a 23×23×8 feature map. The second convolutional layer consists of 16 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to capture intermediate semantic features. It is followed by a 2×2 max pooling layer to further reduce the feature map size and output a 10×10×16 feature map. The third convolutional layer consists of 32 3×3 convolutional kernels. It uses the ReLU activation function and valid padding to learn high-level abstract features. It is followed by a 2×2 max pooling layer to reduce the size of the feature map, and finally outputs a 4×4×32 feature map. The classification head first flattens the 4×4×32 feature map output from the third convolutional layer into a 512-dimensional vector through a flattening layer. Then, the feature is combined through a fully connected layer. After the fully connected layer, there is a regularization layer to improve the model's generalization ability. Finally, the output layer outputs the probability value of the eye being in an open state.

4. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, In step S2, the convolutional neural network model employs a binary cross-entropy loss function and the Adam optimizer during training to minimize prediction error. It also terminates training when model performance reaches its optimal state through an early stopping strategy and a model checkpoint callback mechanism, thereby obtaining the optimal convolutional neural network model as the blink detection model. The optimized configuration used by the convolutional neural network model during training is as follows: The Adam optimizer with optimized parameters is used, and the loss function is binary cross-entropy. Its complete mathematical expression is defined as follows: ； In the formula, N represents the total number of samples in the current training batch. Used to iterate through each sample in the eye image dataset, from the 1st to the Nth sample; Representing the The true label of each sample As a representative model for the first The predicted probability of a sample.

5. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, Step S3 includes the following steps: S31: Load the blink detection model using the TensorFlow Lite converter, enable the default optimization strategy for model quantization, and configure the model operation unit to use the built-in operator that supports INT8 integer operations to ensure that all computational operations in the model meet the integer quantization inference requirements; at the same time, configure the inference input data type and output data type of the blink detection model to unsigned 8-bit integer type to obtain the blink detection model after INT8 quantization. S32: After completing the configuration in step S31, the blink detection model after INT8 quantization is calibrated using the calibration dataset. The dynamic range of the weight parameters and the output feature values ​​of each convolutional layer after the activation function is applied is statistically analyzed. Based on the dynamic range, the corresponding quantization scaling factor and zero offset are determined. S33: Quantize the weight parameters and intermediate activation values ​​in the blink detection model, which are represented by 32-bit floating-point numbers, into 8-bit integers to output a lightweight blink detection model suitable for inference on embedded devices.

6. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, In step S4, the cropped eye image is preprocessed as follows: The cropped eye image is normalized in size and then scaled to the standard input size of the lightweight blink detection model using an interpolation algorithm. Subsequently, the eye image is processed for format conversion. Based on the preset weights of the three color components (red, green, and blue), the color information of each pixel in the color image is weighted and fused to convert the color image into a single-channel grayscale image. After format conversion, the pixel values ​​of the obtained single-channel grayscale image are copied to the red, green, and blue channels simultaneously, and the output is an RGB three-channel image that meets the input requirements of the lightweight blink detection model.

7. The method for detecting blink fatigue based on deep learning according to claim 1, characterized in that, In step S5, when counting the number of effective blinks per unit time, the time period for counting the number of effective blinks can be adjusted according to actual needs. The blink frequency per unit time is calculated using the following formula: ； In the formula, The set time period for statistics, in seconds. In the time statistical period The number of valid blinks detected internally. Indicates the time statistical period The frequency of blinking within the eye.

8. A deep learning-based blink fatigue detection system, characterized in that, The method for implementing the deep learning-based blink fatigue detection method according to any one of claims 1 to 7 includes: The data acquisition and preprocessing module is used to acquire and preprocess eye image datasets with eye open and closed state labels to obtain eye image datasets containing eye images of uniform size and corresponding eye open and closed state labels. The preprocessed eye image dataset is divided into training set, validation set and calibration dataset for subsequent INT8 quantization processing according to the proportion. A blink detection model acquisition module is used to construct a custom convolutional neural network (CNN) model. This CNN model employs a three-layer concatenated architecture to hierarchically extract input data. The first convolutional layer extracts primary features, the second convolutional layer captures intermediate semantic features, and the third convolutional layer learns high-level abstract features. The output of the CNN model is the probability value of the eyes being open, representing the likelihood of the eyes being open. The CNN model is iteratively trained and its performance verified using a training set and a validation set to obtain the optimal CNN model as the blink detection model. The lightweight blink detection model acquisition module uses a calibration dataset to perform INT8 quantization on the blink detection model, and then uses the TensorFlow Lite converter to convert the blink detection model after INT8 quantization into a lightweight blink detection model suitable for inference on embedded devices, and deploys it to the embedded device. The effective blink count definition module is used after the lightweight blink detection model is deployed in the embedded device. The embedded device uses a camera to collect a continuous video frame sequence containing face images in real time as the original input image data of the lightweight blink detection model. The eye key point information in the original input image data is obtained by using a cascaded classifier of HumanFaceDetectMSR01 and HumanFaceDetectMNP01. The eye key point information is the two-dimensional coordinate information of the pupil in the original input image data. Based on the eye key point information, the eye image in the original input image data is accurately cropped, and the cropped eye image is preprocessed and then input into the lightweight blink detection model for inference to obtain the probability value of the eye being in an open state. When the probability value is detected to be greater than a preset threshold for more than a set number of times, the eye is determined to be in an open state. Otherwise, it is in a closed state. When the eye goes through an open state, a closed state and returns to an open state in sequence, and the duration of the closed state conforms to the time characteristics of a natural human blink, it is recorded as one effective blink count. The fatigue assessment module is used to accumulate and count the number of valid blinks obtained according to a preset time statistical period, calculate the blink frequency per unit time, and compare the blink frequency with a preset multi-level fatigue level threshold to determine the corresponding fatigue level. When the fatigue level is greater than or equal to the preset minimum fatigue level threshold, the user is determined to be in a fatigued state.

9. A storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the deep learning-based blink fatigue detection method according to any one of claims 1 to 7.

10. A computing device, comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the deep learning-based blink fatigue detection method according to any one of claims 1 to 7.

Images