A Fatigue Driving Detection Method Based on Personalized Dual-Threshold Filtering and Adaptive Threshold Mechanism

Abstract

This invention proposes a fatigue driving detection method based on personalized dual-threshold filtering and an adaptive threshold mechanism. The method embeds the geometric relationships of facial key points into a heatmap through matrix operations and combines this with specific key point information from the eye and mouth regions to construct a complete feature representation. An improved FaRL model based on ViT-B / 16 is employed, and a novel CNN structure is designed as the feature parameter head to accurately fit the aspect ratios of the eyes and mouth from the fused heatmap. A personalized dual-threshold filtering method is introduced to replace traditional peak detection. Personalized feature parameters are extracted as baseline values ​​by analyzing keyframes in the video, while an adaptive threshold mechanism dynamically adjusts the detection standard to address environmental and individual differences. This method improves the accuracy, real-time performance, and environmental adaptability of fatigue detection, providing an efficient and reliable solution for intelligent driving safety monitoring, and has significant application value and prospects.

Description

Technical Field

[0001] This invention relates to the field of driver fatigue monitoring technology, specifically to a fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism. Background Technology

[0002] Recent analyses of the causes of road traffic accidents in my country have shown that fatigued driving has become one of the main risk factors leading to traffic accidents. According to current road traffic regulations, driving continuously for four hours constitutes fatigued driving. Fatigued driving has become a widespread problem that cannot be ignored. After driving for extended periods, drivers experience a decline in driving skills, leading to slowed reaction times, inattention, and impaired judgment. In severe cases, they may even lose control of the vehicle, greatly increasing the risk of road traffic accidents. In the field of fatigued driving detection technology, the extraction of facial key points and the calculation of feature parameters such as eye aspect ratio (EAR), mouth aspect ratio (MAR), blink frequency, and yawning frequency are crucial. Predicting and warning of fatigued driving conditions are the core of effective prevention and control. Modern fatigued driving detection systems, by monitoring driver status in real time and combining timely warning mechanisms, provide technical support for preventing traffic accidents caused by fatigued driving and are of great significance for improving road traffic safety.

[0003] Currently, with the continuous development of fields such as artificial intelligence and biomedical engineering, fatigue driving detection technology is also constantly innovating and progressing. For example, fatigue driving detection systems based on in-vehicle cameras and physiological signal acquisition devices can accurately assess a driver's fatigue state and issue timely warnings by monitoring information such as eye movements, facial expressions, and physiological indicators. In addition, there are fatigue driving early warning and management platforms based on intelligent in-vehicle systems and internet big data technologies. These platforms analyze driver behavior patterns and traffic environment factors to predict potential fatigue driving risks and take corresponding measures to reduce the risk of accidents and improve driving safety.

[0004] However, traditional fatigue driving detection methods based on physiological signal acquisition, vehicle driving behavior, and single facial features have some limitations. For example, these methods usually require drivers to wear additional equipment, which is not only inconvenient to use, but also easily affected by environmental factors. In adverse weather or complex road conditions, vehicle driving behavior itself can be interfered with by environmental changes, significantly reducing the accuracy of fatigue detection based on driving behavior. At the same time, a single feature cannot comprehensively reflect the driver's fatigue state and is easily affected by external environmental interference. For example, blinking frequency may fluctuate due to changes in light or road bumps, leading to false positives. To overcome the limitations of traditional detection methods, fatigue driving detection based on deep learning has gradually emerged. Compared with traditional methods, deep learning technology has many advantages, such as not relying on external sensors, high stability, and less susceptibility to environmental interference. However, this technology also faces certain challenges. Deep learning models may perform well on specific datasets, but their performance may significantly decrease when generalized to other datasets or real-world application scenarios. This insufficient generalization ability may lead to false positives or false negatives, especially when various scenarios and conditions are not fully considered during model training. Furthermore, deep learning models are often considered "black box models," as their internal decision-making processes are difficult to interpret. This lack of interpretability limits users' understanding and trust in the model's output, and also poses a certain obstacle to further optimization and improvement of the model.

[0005] Therefore, overcoming the limitations of existing technologies to achieve accurate, stable, and real-time facial landmark detection, and efficiently calculating characteristic parameters such as eye aspect ratio (EAR), mouth aspect ratio (MAR), blink frequency, and yawn frequency, has become a key challenge in improving the monitoring performance of fatigue driving systems. Summary of the Invention

[0006] To address the limitations of existing technologies, this invention innovatively proposes a fatigue driving detection method based on personalized dual-threshold filtering and an adaptive threshold mechanism. This method achieves significant breakthroughs in several key technical indicators. Compared to traditional detection methods, this invention successfully overcomes inherent shortcomings such as cumbersome operation, weak anti-interference ability, and single evaluation dimension. Compared to current mainstream deep learning solutions, it effectively solves technical bottlenecks such as high data dependence, high computational resource consumption, and limited model generalization ability. This method achieves significant improvements in detection accuracy, generalization ability, and real-time performance, and effectively reduces the risk of drivers deceiving the system. By innovatively integrating multiple technical advantages, this method not only achieves comprehensive improvements in technical indicators but also provides a reliable and practical solution for the field of fatigue driving detection, demonstrating broad application prospects and practical promotion value.

[0007] The present invention adopts the following technical solution:

[0008] A fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism includes the following:

[0009] Step S1: Image acquisition, acquiring facial image data of the driver to be detected;

[0010] Step S2: Image preprocessing. The collected face image data is standardized, including size unification, cropping, color channel conversion, and normalization.

[0011] Step S3: Input the preprocessed image into the trained FaRL (Facial Representation Learning) model (its basic model is Vision Transformer (ViT), specifically using the ViT-B / 16 basic structure);

[0012] Step S4: Face recognition, using the face detection model MTCNN to extract the face region;

[0013] Step S5: Key point recognition. Facial key point recognition is performed using the FaRL model's visual module. Facial image data is input into a preset face detection model for processing. This innovative approach avoids traditional point-by-point regression and geometric calculation methods, instead using heatmap fusion and coordinate embedding techniques to transform from point-by-point coordinate regression to overall feature extraction.

[0014] Step S6: Based on the facial key point coordinate data, calculate the two important facial feature parameters, the eye aspect ratio EAR and the mouth aspect ratio MAR.

[0015] Step S7: Extract personalized facial feature parameters of the driver and set action thresholds and value thresholds for the eye aspect ratio EAR and mouth aspect ratio MAR. When the system is used for the first time, the driver's facial video is sampled and analyzed, and personalized facial feature parameters of the driver are extracted from key frames as dual-threshold filtering reference values. Based on these reference values, action thresholds and value thresholds for the eye aspect ratio EAR and mouth aspect ratio MAR are set respectively.

[0016] Step S8: Process the eye aspect ratio EAR and compare it with the adaptive eye aspect ratio EAR threshold (action threshold) to determine whether a blinking action has occurred. Compare it with the eye aspect ratio EAR threshold value to determine whether the blinking action has been completed. Process the mouth aspect ratio MAR and compare it with the adaptive mouth aspect ratio MAR (action threshold) threshold value to determine whether a yawning action has occurred. Compare it with the mouth aspect ratio MAR threshold value to determine whether the yawning action has been completed.

[0017] Step S9, Frequency Statistics: Count the number of blinks over a period of time according to the blinking action determination method in S8, and thus obtain the blinking frequency; Count the number of yawns over a period of time according to the yawning action determination method in S8, and thus obtain the yawning frequency.

[0018] Step S10: Determining Fatigue Driving Status. The blinking frequency and yawning frequency per unit time in S9 are compared with preset blinking frequency thresholds and yawning frequency thresholds, respectively, to determine whether the driver is in a fatigue driving state. Alternatively, the driver's fatigue driving status can be determined by whether the time the driver is in a closed-eye state exceeds a preset fatigue-related closed-eye time threshold.

[0019] Furthermore, the specific steps of step S1 are as follows:

[0020] Step S11: Image acquisition uses an independent image acquisition module, including a conventional image sensor and an infrared image sensor. The infrared image sensor includes an infrared fill light and an infrared camera. Initially, the conventional image sensor is used for acquisition. The image acquisition module determines whether the lighting conditions are acceptable. If not, the infrared fill light is activated, and then the module switches to the infrared camera for acquisition to obtain the facial image data of the driver to be detected.

[0021] Furthermore, the specific steps of step S2 are as follows:

[0022] Step S21: Image preprocessing, which involves extracting and processing the face image data such as size unification, cropping, color channel conversion and normalization. First, a uniform image size of 447×447 is generated.

[0023] Step S22: Perform color channel conversion on the image. If the input image is grayscale or BGR format, convert the image format to RGB format.

[0024] Step S23: Normalize the image to map the pixel values ​​from the range [0,255] to [0,1].

[0025] Step S23: Standardize the normalized image by subtracting the channel's training set mean from each pixel value and dividing the result by the channel's training set standard deviation. This unifies the numerical ranges of different features, adjusting the image distribution to the standardized state used during model training, thereby improving the model's performance and stability.

[0026] Furthermore, the specific steps of step S4 are as follows:

[0027] Step S41: Use the MTCNN face detection model to detect the face locations in the image; output the bounding box of each face and return the coordinates (x, y) of the face bounding box.box ,y box (width, height);

[0028] Step S42: To ensure the complete face region is included, expand the width and height of the bounding box by 10%-20%; the formula for calculating the new face bounding box is as follows:

[0029]

[0030] Where new_width and new_height are the width and height of the face bounding box after expanding by 10%-20%, respectively, p is the expansion ratio, with a value range of [0.1, 0.2], and x... box_new and y box_new These are the x and y coordinates of the top left corner of the new face bounding box, respectively.

[0031] Step S43: Face cropping. Based on the updated face bounding box, crop out the sub-region containing the face from the original image.

[0032] Furthermore, the specific steps of step S5 are as follows:

[0033] Step S51: Use the trained FaRL visual encoder as the backbone;

[0034] Step S52: Select the feature maps of layers 4, 6, 7, and 12 for multi-level feature fusion, and use UpperNet to integrate the multi-level feature maps;

[0035] Step S53: Use 1×1 convolution to generate a predicted heatmap of key points as the output layer;

[0036] Step S54: Render the groundtruth key points into a Gaussian heatmap of size 127×127, model the key point positions using Gaussian distribution and set its standard deviation to 1 pixel, and the value range of the heatmap is [0,1].

[0037] Step S55: Based on the relationship between the detected key points and the pre-set standard template, use affine transformation to apply geometric transformations such as translation, rotation, and scaling to adjust the face to the position of the target template.

[0038] Step S56: Heatmap Fusion. The two-dimensional heatmap regression method soft-argmax is used, defined as follows:

[0039]

[0040] In the formula, d represents a given component x or y, and P is the weight matrix of W×H×2 corresponding coordinates (x, y). Matrix P can be expressed using its components P x and P y This indicates that both components are two-dimensional discrete normalized linear mappings, defined as follows:

[0041]

[0042] Here, Φ(h) i,j The softmax result of a single heatmap is defined as follows:

[0043]

[0044] This method can be viewed as a convolution with a kernel size of H×W, where the kernel contains x or y position information, and is ordered according to pixel position Φ(h i,j The simplified soft-argmax operation performs a convolution on the weighted heatmap of keypoints based on the variation characteristics of the x and y axes, and after normalization and other processing, obtains the x and y coordinates of the keypoints.

[0045] Step S57: Analyze the geometric relationships between facial keypoints. Skip the regression and geometric calculations for each keypoint and analyze the overall facial features. Taking the definition of facial keypoints in the 300-W dataset as an example, markers 36-47 are classified as eye feature keypoints, and inner lip markers 61-67 are classified as mouth feature keypoints. Define the overall facial feature weight Φ(h') i,j )for:

[0046]

[0047] In the formula, n and m represent the serial numbers of the key points of the eye features and the key points of the mouth features, respectively.

[0048] The overall heatmap M is defined as:

[0049]

[0050] Step S58: Coordinate Embedding. When processing the overall heatmap extracted from multiple images, it was found that relying solely on direct convolution operations was insufficient to effectively extract feature information for distinguishing fatigue states. To optimize the relative geometric relationships between key points after fusion, more complex matrix operations are required on the heatmap. This invention innovatively borrows the core idea of ​​the soft-argmax algorithm and designs a dual coordinate embedding mechanism to enhance the feature representation in the heatmap matrix. Specifically, the overall heatmap matrix is ​​first pre-processed with zero-padding. When H ≤ W, zeros are padded in the (WH) column to the right of M; when H > W, zeros are padded in the (HW) row at the bottom of M, converting it into a normal square matrix structure, laying the foundation for subsequent feature enhancement operations.

[0051]

[0052] Next, a diagonal matrix diag(0,1,...K-1) is constructed. This diagonal matrix represents the coordinate information of each column or row in the heatmap. Finally, by multiplying the diagonal matrix by the square matrix M′ in both the left and right directions, linear weighting by row and column is achieved to obtain the horizontal and vertical coordinate information, respectively. This positional information is then embedded into the feature map, realizing coordinate embedding on the overall heatmap.

[0053]

[0054] Where H and W represent the height and width of the heatmap, respectively, and M x M represents the result of embedding the heatmap in the x-direction. y This represents the result of embedding the heatmap in the y-direction. In the formula, K = max(W,H) ensures that the diagonal matrix dimension matches the square matrix M′. This method emphasizes the geometric relationships between the relative coordinates of key points in the overall heatmap. After coordinate embedding, facial feature parameters required for driver fatigue detection can be extracted.

[0055] Step S59: Extract data containing facial key points from the fused heatmap after coordinate embedding. Specifically, key points 36 to 47 are classified as eye key points, and key points 61 to 67 are classified as mouth key points, and the corresponding facial image data are extracted.

[0056] Furthermore, the specific steps of step S6 are as follows:

[0057] Step S61: Dynamic feature extraction is performed on the image data of the eye opening and closing state to obtain the first dynamic feature information - the coordinates of key points of both eyes. The coordinates of key points of both eyes include the coordinates of key points of the upper eyelid (2), the coordinates of key points of the lower eyelid (2), and the coordinates of key points of the corner of the eye (2).

[0058] The expression for the first feature information (taking the left eye as an example) is as follows:

[0059] The sum of the Euclidean distances in the vertical direction of the two pairs of key points on the upper and lower eyelids is calculated based on their coordinates. The eye height is the distance from the upper eyelid to the lower eyelid. Simultaneously, the horizontal distance between the left and right eye corners is calculated based on the coordinates of the key points at the corners of the eyes. The formula for calculating the eye aspect ratio (EAR) is:

[0060]

[0061] In the formula, p1 to p6 are the six key points of the eye. The numerators ||p2-p6|| and ||p3-p5|| represent the Euclidean distances in the vertical direction between the two pairs of key points corresponding to the upper and lower eyelids, and ||p1-p4|| represent the Euclidean distances in the horizontal direction between the key points at the left and right corners of the eye. ||p1-p4|| is multiplied by 2 to ensure that the numerator and denominator maintain the same weight in the calculation.

[0062] The formula for calculating the Euclidean distance between two pairs of key points on the upper and lower eyelids is:

[0063]

[0064] x 38 x 39 x represents the x-coordinate of the key point on the upper eyelid; 42 x 41 y is the x-coordinate of the key point on the lower eyelid; 38 y 39 The y-coordinate of the key point on the upper eyelid; 42 y 41 The vertical coordinate value of the coordinate point corresponding to the key point of the lower eyelid;

[0065] To calculate the pixel width of the eyes, the corresponding pixel width of both eyes is calculated based on the coordinates of the key points at the corners of the eyes. The calculation formula is as follows:

[0066]

[0067] Where ||p1-p4|| are the pixel width values ​​of the left and right corners of the eyes; x 37 y 37 These are the x and y coordinates of the key point on the left corner of the eye; x 40 y 40 These are the x-coordinates of the key points at the right corner of the eye;

[0068] Step S62: Dynamic feature extraction is performed on the image data of the mouth opening and closing state to obtain the first dynamic feature information, the coordinates of the key points of the mouth. The coordinates of the key points of the mouth include the coordinates of the key points on the inner side of the upper lip (3), the coordinates of the key points on the inner side of the lower lip (3), and the coordinates of the key points at the corners of the mouth (2).

[0069] The expression for the coordinates of the key points of the mouth, the first feature information, is as follows:

[0070] Based on the coordinates of key points in the mouth, including the coordinates of the inner corners of the upper and lower lip, the sum of the Euclidean distances of the key points in the vertical direction is calculated. Furthermore, based on the coordinates of the inner corners of the mouth, the sum of the Euclidean distances of the left and right corners of the mouth in the horizontal direction is calculated. The formula for calculating the mouth aspect ratio (MAR) is:

[0071]

[0072] Where, p 61 to p 65 Seven key points for the mouth area. ||p 62 -p 68 ||、||p 63 -p 67 ||、||p 64 -p 66 || represents the Euclidean distance between three pairs of key points on the inner sides of the upper and lower lips; ||p 61 -p 65 || represents the Euclidean distance between key points on the inner side of the lips and the left and right corners of the mouth; ||p 61 -p 65 Multiply by 3 to ensure that the numerator and denominator have the same weight in the calculation.

[0073] The formula for calculating the Euclidean distance between the three pairs of key points on the inner sides of the upper and lower lips is:

[0074]

[0075] x 62 x 63 x 64 x represents the x-coordinate of the key point on the inner side of the upper lip; 66 x 67 x 68 The x-coordinate of the key point on the inner side of the lower lip; y 62 y 63 y 64 The y-coordinate of the key point on the inner side of the upper lip; 66 y 67 y 68 The ordinate value of the coordinate point corresponding to the key point on the inner side of the lower lip;

[0076] To calculate the width of the corners of the mouth, the formula is as follows: (The formula is missing from the original text.)

[0077]

[0078] Among them, ||p 61 -p 65 || represents the pixel width of the left and right corners of the mouth; x 61 y 61 The x and y coordinates of the key points on the left side of the inner lip corner of the mouth are respectively the x and y coordinates of the points. 65 y 65 The x-coordinate and y-coordinate values ​​are respectively the coordinates of the key points on the right side of the inner lip corner.

[0079] Furthermore, the specific steps of step S7 are as follows:

[0080] Step S71: When the driver first gets into the vehicle and starts the system, start video frame segmentation and extract the facial feature parameter sequence, denoted as P = {p1, p2, ..., p...} n}, where p i Let be the facial feature value of the i-th frame. Obtain the variation curves of key parameters EAR and MAR.

[0081] Step S72: Calculate the average values ​​of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) variation curves to reflect typical values ​​of the parameters under normal conditions, and use them as a reference benchmark for the adaptive threshold of the dual-gate filter. The formulas for calculating the average values ​​of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows:

[0082]

[0083] Where u is the average of the eye aspect ratio (EAR) and mouth aspect ratio (MAR), n is the total number of frames per unit time, and p i Let be the facial feature value of the i-th frame.

[0084] Step S73: Set action threshold values ​​to determine whether an action has ended. Select a threshold setting method based on feature recovery. During blink detection, it is stipulated that only when the eye aspect ratio (EAR) parameter drops below this threshold can newly generated peak values ​​be used as the basis for blink action counting. During yawn detection, it is stipulated that only when the mouth aspect ratio (MAR) parameter drops below this threshold can newly generated peak values ​​be used as the basis for yawn action counting. This is to prevent the influence of multiple adjacent peak values ​​with unclear upward and downward trends caused by lens shake due to road bumps on the blink and yawn action counting.

[0085] Step S74: The eye aspect ratio EAR threshold (action threshold) in S7 is improved from the traditional fixed empirical threshold by adopting a dynamic adjustment method based on the historical average eye aspect ratio EAR. This means that the threshold is dynamically adjusted using the average EAR value over a period of time, and the historical record is updated in each frame based on the current EAR value. Furthermore, an adjustment coefficient α is introduced. The average eye aspect ratio EAR value of the most recent 20 frames is calculated and multiplied by the coefficient α (typically between 1.05 and 1.20, which can be fine-tuned according to actual conditions) to achieve dynamic adjustment of the action threshold. The adaptive threshold calculation formula is as follows:

[0086] dynamic_threshold=sum(historical_ears) / len(historical_ears)×α (16)

[0087] Wherein, dynamic_threshold represents the adaptive action threshold, historical_ears represents the historical eye aspect ratio (EAR) value, sum(historical_ears) represents the sum of EAR values ​​over a period of time, len(historical_ears) represents the total number of frames recorded over a period of time, and α represents the dynamic adjustment coefficient, with a value range of [1.05, 1.20].

[0088] Step S75: Select the mouth aspect ratio (MAR) threshold from step S7. As shown in the mouth aspect ratio (MAR) calculation formula from step S62, the MAR value is positively correlated with the degree of mouth opening: when the mouth is completely closed, the distance between the upper and lower lips is minimal, corresponding to a minimum mouth aspect ratio (MAR); as the opening degree increases, the mouth aspect ratio (MAR) increases accordingly. In normal driving conditions, the mouth aspect ratio (MAR) during speech is usually between the closed state and the yawning state. The mouth aspect ratio (MAR) during speech is greater than the MAR value when the mouth is closed, but generally less than the MAR value when yawning.

[0089] Based on the driver's personalized MAR parameter baseline value obtained in step S7, a critical threshold M, i.e., the action threshold value, can be determined to distinguish between yawning and normal speaking. However, it is worth noting that in practical applications, the MAR value during speaking may briefly exceed the threshold M at certain moments. This is because the mouth exhibits periodic opening and closing characteristics during speaking; even if it occasionally exceeds the threshold M, its duration is relatively short. Therefore, to improve detection accuracy, this method employs a dual judgment mechanism, not only examining whether the MAR value exceeds the threshold M, but also using the duration of the MAR value exceeding the threshold M as an important reference indicator. Through this spatiotemporal combined evaluation method, the accuracy of yawn detection is effectively improved.

[0090] Step S76: Based on step S71, calculate the standard deviation of the facial feature parameter variation curves to reflect the fluctuation range of the parameters. The formulas for calculating the standard deviation of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows:

[0091]

[0092] Set a threshold value. This threshold is used to exclude invalid peaks caused by image jitter or minor facial movements (such as slight eyelid movements), ensuring that only peaks exceeding a certain amplitude (representing a clear blink or yawn) are counted as valid movements. Set the threshold as follows:

[0093] T υ =μ-k·σ (18)

[0094] Among them, T υ This represents a threshold value used to filter out values ​​below T in the characteristic curve. υ The invalid peak value is μ, which is the average value of the eye aspect ratio EAR curve, σ is the standard deviation of the eye aspect ratio EAR curve, and k is a hyperparameter, which is usually adjusted between [0.5, 1.5].

[0095] Furthermore, the specific steps of step S8 are as follows:

[0096] Step S81: Blinking Action Determination and Counting. Based on the eye aspect ratio (EAR) obtained in step S6, compare it with the EAR adaptive threshold (action threshold) and the preset threshold to determine whether the detected person has generated and completed a blinking action. When the EAR values ​​of both eyes are less than the preset EAR action threshold, it is determined that the detected driver has started to blink. When the EAR values ​​of consecutive image frames exceed the EAR threshold again, it is determined that the detected driver has completed this blinking action, and the blink count is incremented by one.

[0097] Step S82: Yawning Action Determination and Counting. Based on the mouth aspect ratio (MAR) obtained in step S6, compare it with the preset MAR action threshold and value threshold to determine whether the person being detected has generated and completed a yawning action. When the mouth MAR value is less than the preset MAR action threshold, it is determined that the person being detected has started to yawn. When the mouth MAR value of consecutive image frames exceeds the MAR value threshold again, it is determined that the person being detected has completed the yawning action, and the yawning action count is incremented by one.

[0098] Furthermore, the specific steps of step S9 are as follows:

[0099] Step S91, the formula for calculating blink frequency is as follows:

[0100]

[0101] Where f wink N is the blink frequency. wink The number of blinks is denoted by T, where T is the unit of time.

[0102] Step S92, the formula for calculating yawning frequency is as follows:

[0103]

[0104] In the above formula, f yawn The yawning frequency is defined as the number of yawns per unit time T, where n represents the number of yawns per unit time T. The yawning frequency can also be calculated using the number of frames.

[0105]

[0106] Where N represents the total number of frames within a unit time period, f n Indicate whether the mouth in the nth frame is in a yawning state. If f n A value of 1 indicates a yawning state, f n A value of 0 indicates a normal state.

[0107] Furthermore, the specific steps of step S10 are as follows:

[0108] Step S101: Setting the blink frequency threshold from step S10. Based on extensive driver behavior research data, this study discovered a regular change in blink characteristics as fatigue levels evolve. Under normal awake conditions, a driver's blink frequency typically remains at 12–18 times per minute. As fatigue levels increase, blink characteristics exhibit distinct phased changes. In the mild fatigue stage, drivers instinctively increase blinking to combat fatigue, leading to a significant increase in blink frequency and prolonged eye closure time. Research data shows that blink frequency under fatigue conditions increases by 50% to 60% compared to normal levels, reaching 18–27 times per minute. In the moderate to severe fatigue stage, although drivers may keep their eyes open, they often experience distraction, a blank stare, or persistent eye closure, causing a sharp decrease in blink frequency. Furthermore, regarding the blink cycle, a single blink (the process from fully open to fully closed) takes 200–400 milliseconds under normal conditions, while under fatigue conditions, this cycle significantly lengthens to 600–700 milliseconds. Considering individual differences, especially the possibility that some groups may have a higher baseline blinking frequency, this study, after in-depth analysis, determined the fatigue threshold for blinking frequency to be 25 times / minute, in order to balance the accuracy and universality of the detection.

[0109] Step S102: Fatigue determination. When the blinking frequency of the driver being tested exceeds the preset upper limit threshold or falls below the preset lower limit threshold, the driver being tested is determined to be in a state of fatigued driving; or when the yawning frequency of the driver being tested exceeds the preset yawning frequency threshold, the driver being tested is determined to be in a state of fatigued driving.

[0110] The core innovation of this method lies in the organic combination of personalized dual-threshold filtering and an adaptive threshold adjustment mechanism. Based on the driver's personalized facial feature parameters, the system can respond in real-time to changes in driver state and environmental dynamics (including lighting conditions, driving environment, and individual differences), and automatically optimize and adjust the action and value thresholds for the eye aspect ratio (EAR) and mouth aspect ratio (MAR), thus maintaining excellent detection accuracy and system stability in various application scenarios. The introduction of dual-threshold filtering effectively solves the problem of camera shake caused by road bumps, successfully eliminating the interference of adjacent invalid peaks lacking significant changing trends on blink and yawn counts. This improvement significantly enhances the overall performance of the fatigue driving detection system, achieving new heights in accuracy, stability, and environmental adaptability. Furthermore, in terms of model architecture, a novel CNN (Convolutional Neural Network) structure is designed as the facial feature parameter head, directly learning and fitting the eye aspect ratio (EAR) and mouth aspect ratio (MAR) from the fused heatmap, achieving efficient extraction of facial key point features. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism has opened up a new direction for research and application in the field of fatigue driving detection, and promoted the practical and intelligent development of the system in complex scenarios.

[0111] The beneficial effects of this invention lie in establishing a complete facial recognition and feature extraction system. Through heatmap fusion and coordinate embedding, the geometric relationships of key points are directly embedded into the heatmap, and heatmap information of specific key points in the eye and mouth regions is fused to construct a complete feature representation. This not only retains the advantages of traditional geometric features but also further integrates heatmap information, achieving a more comprehensive and in-depth state assessment. At the technical implementation level, this invention innovatively uses coordinate embedding and heatmap fusion techniques to extract the eye aspect ratio (EAR) and mouth aspect ratio (MAR) indices, and breaks through by adopting a personalized dual-threshold filtering method to replace the traditional peak detection counting mechanism, significantly improving the accuracy and stability of the counting process. Regarding system intelligence, this invention designs a unique threshold mechanism: when the driver uses the system for the first time, personalized facial feature parameters are extracted by analyzing key frames of the video and used as the benchmark value for dual-threshold filtering. Simultaneously, combined with an innovative adaptive threshold adjustment mechanism, the detection threshold can be optimized in real time based on multi-dimensional factors such as lighting conditions, driving environment complexity, and individual differences, significantly enhancing the overall accuracy and environmental adaptability of the fatigue driving detection method.

[0112] Compared to existing technologies, this invention overcomes the shortcomings of traditional detection methods, such as cumbersome operation, poor anti-interference ability, and single evaluation criteria. It also effectively solves problems in deep learning methods, such as data dependence, computational resource consumption, and limited generalization ability, achieving a comprehensive improvement in detection accuracy, real-time performance, and environmental adaptability. This comprehensive solution not only provides reliable technical support for fatigue driving detection but also makes a significant contribution to improving road traffic safety, possessing broad practical application value. Attached Figure Description

[0113] Figure 1 This is a flowchart of the fatigue driving detection method of the present invention.

[0114] Figure 2 This is a simplified soft-argmax operation process for this invention.

[0115] Figure 3 This invention provides a fine-tuning process for the training and inference of the model.

[0116] Figure 4 This is a diagram of the key points for the eyes and the inner lip of the mouth in this invention.

[0117] Figure 5 This is a diagram illustrating the process of embedding heatmap coordinates in this invention. Detailed Implementation

[0118] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0119] Figure 1 The following is a flowchart of the fatigue driving detection system of the present invention, and the present invention will be further described below with reference to the accompanying drawings.

[0120] Step 1: Image Acquisition. Acquire real-time facial image data of the driver to be detected.

[0121] Step 2: Image preprocessing. The collected face image data is standardized, including size unification, region cropping, color channel conversion and normalization.

[0122] Step 3: Input the preprocessed image into the trained FaRL model;

[0123] Step 4: Face region detection. The face region is extracted using the MTCNN face detection model.

[0124] Step 5: Key Point Recognition. Facial key point recognition is performed using the FaRL model's visual module. Facial image data is input into a pre-defined facial recognition model for processing. This innovative approach avoids traditional point-by-point regression and geometric calculation methods, instead using heatmap fusion and coordinate embedding techniques to transform from point-by-point coordinate regression to overall feature extraction.

[0125] Step 6: Based on the facial key point coordinate data, calculate the two important facial feature parameters: eye aspect ratio (EAR) and mouth aspect ratio (MAR).

[0126] Step 7: Extract personalized facial feature parameters of the driver and set action thresholds and value thresholds for the eye aspect ratio (EAR) and mouth aspect ratio (MAR). When the system is used for the first time, video frames of the driver's face are sampled and analyzed. Personalized facial feature parameters are extracted from key frames as dual-threshold filtering reference values. Based on these reference values, action thresholds and value thresholds for the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are set respectively.

[0127] Step 8: Process the eye aspect ratio EAR and compare it with the adaptive eye aspect ratio EAR threshold (action threshold) to determine whether a blinking action has occurred. Compare it with the eye aspect ratio EAR threshold value to determine whether the blinking action has been completed. Process the mouth aspect ratio MAR and compare it with the adaptive mouth aspect ratio MAR (action threshold) threshold value to determine whether a yawning action has occurred. Compare it with the mouth aspect ratio MAR threshold value to determine whether the yawning action has been completed.

[0128] Step 9: Frequency Statistics. Count the number of blinks over a period of time based on the blinking action determination method in Step 8 to obtain the blinking frequency; Count the number of yawns over a period of time based on the yawning action determination method in Step 8 to obtain the yawning frequency.

[0129] Step 10: Determine if the driver is in a state of fatigue. Compare the blinking frequency and yawning frequency per unit time from Step 9 with preset blinking frequency thresholds and yawning frequency thresholds, respectively, to determine if the driver is in a state of fatigue. Alternatively, determine if the driver is in a state of fatigue by judging whether the time the driver is in a closed-eye state exceeds a preset fatigue closed-eye time threshold.

[0130] The specific implementation of step one is as follows:

[0131] First, image acquisition uses a separate image acquisition module, including a conventional image sensor and an infrared image sensor. The infrared image sensor includes an infrared illuminator and an infrared camera. Initially, the conventional image sensor is used for acquisition. The image acquisition module determines whether the lighting conditions are suitable. If not, the infrared illuminator is activated, and then the module switches to the infrared camera to acquire facial image data of the driver to be detected.

[0132] The specific implementation of step two is as follows:

[0133] First, image preprocessing is performed, which involves extracting data from the face image data, such as size unification, cropping, color channel conversion, and normalization.

[0134] First, generate a uniform image size of 447×447. Next, perform color channel conversion on the image; if the input image is grayscale or BGR format, convert the image format to RGB format.

[0135] Then, the image is normalized to map the pixel values ​​from the range [0,255] to [0,1].

[0136] Finally, the normalized image is standardized by subtracting the channel's training set mean from each pixel value and dividing the result by the channel's training set standard deviation. This unifies the numerical ranges of different features, adjusting the image distribution to the standardized state used during model training, thereby improving the model's performance and stability.

[0137] The specific implementation of step four is as follows:

[0138] First, the MTCNN face detection model is used to detect the location of faces in the image. The bounding box for each face is output, returning the coordinates (x, y, width, height) of the bounding box.

[0139] Next, to ensure the complete face region is included, the bounding box is expanded by a certain proportion, for example, by 10%-20% in width and height. The formula for calculating the new bounding box is as follows:

[0140]

[0141] Finally, face cropping is performed, cropping out sub-regions containing faces from the original image based on the updated face bounding boxes.

[0142] The specific implementation of step five is as follows:

[0143] The first step is to use the trained FaRL visual encoder as the backbone;

[0144] The second step is to select feature maps from layers 4, 6, 7, and 12 for multi-level feature fusion, using UpperNet to integrate the multi-level feature maps.

[0145] The third step is to use 1×1 convolution to generate heat maps of key points as the output layer.

[0146] Step 4: Render the ground truth key points into a Gaussian heatmap of size 127×127, model the key point positions using a Gaussian distribution and set its standard deviation to 1 pixel, with the heatmap value range being [0,1].

[0147] Step 5: Based on the relationship between the detected key points and the pre-set standard template, use affine transformation to apply geometric transformations such as translation, rotation, and scaling to adjust the face to the position of the target template.

[0148] Step 6: Heatmap Fusion. The two-dimensional heatmap regression method soft-argmax is used, defined as follows:

[0149]

[0150] In the formula, d represents a given component x or y, and P is the weight matrix of W×H×2 corresponding coordinates (x, y). Matrix P can be expressed using its components P x and P y This indicates that both components are two-dimensional discrete normalized linear mappings, defined as follows:

[0151]

[0152] Here, Φ(h) i,j The softmax result of a single heatmap is defined as follows:

[0153]

[0154] This method can be viewed as a convolution with a kernel size of H×W, where the kernel contains x or y position information, and is ordered according to pixel position Φ(h i,jArrangement. Simplified soft-argmax operation as follows: Figure 2 As shown, based on the variation characteristics of the x-axis and y-axis, the weighted heatmap of the key points is convolved once, and after normalization and other processing, the x-coordinates and y-coordinates of the key points are obtained.

[0155] Step 7: Analyze the geometric relationships between facial key points. Skip the regression and geometric calculations for each key point and analyze the overall facial features. Figure 4 To define facial keypoints in the 300-W dataset, markers 36-47 are classified as eye keypoints, and markers 61-67 on the inner lip are classified as mouth keypoints. The overall facial feature weight Φ(h′) is defined. i,j )for:

[0156]

[0157] In the formula, n and m represent the serial numbers of the key points of the eye features and the key points of the mouth features, respectively.

[0158] The overall heatmap M is defined as:

[0159]

[0160] Step 8 Figure 5 The diagram below illustrates the heatmap coordinate embedding process of this invention, and further explanation is provided below with reference to the accompanying drawings. Although heatmap fusion can extract geometric information of key points, directly performing convolution operations on the fused heatmap often fails to effectively capture the spatial geometric relationships between key points. Furthermore, experiments show that when processing overall heatmaps extracted from multiple images, relying solely on convolution operations is insufficient to extract effective features that distinguish fatigue states. This further illustrates that convolution operations alone are insufficient to fully model the geometric relationships between key points. Therefore, this invention innovatively borrows the core idea of ​​the soft-argmax algorithm and designs a dual coordinate embedding mechanism to enhance the feature representation in the heatmap matrix. Specifically, the overall heatmap matrix is ​​first pre-processed with zero-padding. When H ≤ W, zeros are padded in the (WH) column to the right of M; when H > W, zeros are padded in the (HW) row at the bottom of M, converting it into a normal square matrix structure, laying the foundation for subsequent feature enhancement operations.

[0161]

[0162] Next, a diagonal matrix diag(0,1,...K-1) is constructed. This diagonal matrix represents the coordinate information of each column or row in the heatmap. Finally, by multiplying the diagonal matrix by the square matrix M′ in both the left and right directions, linear weighting by row and column is achieved to obtain the horizontal and vertical coordinate information, respectively. This positional information is then embedded into the feature map, realizing coordinate embedding on the overall heatmap.

[0163]

[0164] Where H and W represent the height and width of the heatmap, respectively, and M x M represents the result of embedding the heatmap in the x-direction. y This represents the result of embedding the heatmap in the y-direction. In the formula, K = max(W,H) ensures that the diagonal matrix dimension matches the square matrix M′. This method emphasizes the geometric relationships between the relative coordinates of key points in the overall heatmap. After coordinate embedding, facial feature parameters required for driver fatigue detection can be extracted.

[0165] The entire training and inference workflow of the fine-tuned model is as follows: Figure 3 As shown, the driver image initially undergoes a standard face alignment procedure. After facial region detection, a face alignment network is applied to generate a facial keypoint heatmap. This heatmap is then used in two distinct processes. One set is regressed to determine the coordinates of the keypoints, which are then geometrically computed and used as training data labels. The other set is fused and filled to generate a comprehensive heatmap, which is then used for coordinate embedding to generate a high-dimensional feature representation. These embedded features serve as input to a CNN facial feature parameter extraction module. This module first extracts salient features and reduces dimensionality through a single max-pooling layer, then effectively captures spatial feature relationships through a five-layer convolutional network. The extracted features are then processed through a series of linear layers for feature transformation and fitting calculations, finally outputting two highly generalized facial feature parameter scalars through fully connected layers, providing accurate quantitative indicators for fatigue state assessment.

[0166] Step 9: Extract data containing facial key points from the fused heatmap after coordinate embedding. Specifically, key points 36 to 47 are classified as eye key points, and key points 61 to 67 are classified as mouth key points, and the corresponding facial image data are extracted.

[0167] The specific implementation of step six is ​​as follows:

[0168] First, dynamic features are extracted from the image data of the eye opening and closing state to obtain the first dynamic feature information - the coordinates of key points of both eyes. The coordinates of key points of both eyes include the coordinates of key points of the upper eyelid (2), the coordinates of key points of the lower eyelid (2), and the coordinates of key points of the corner of the eye (2).

[0169] The expression for the first feature information (taking the left eye as an example) is as follows:

[0170] Next, based on the coordinates of the key points on the upper and lower eyelids, the sum of the Euclidean distances in the vertical direction between the two pairs of key points is calculated, which is the distance from the upper eyelid to the lower eyelid. Simultaneously, based on the coordinates of the key points at the corners of the eyes, the horizontal distance between the left and right corners is calculated. The formula for calculating the aspect ratio (EAR) of the eye is:

[0171]

[0172] In the formula, p1 to p6 are the six key points of the eye. The numerators ||p2-p6|| and ||p3-p5|| represent the Euclidean distances in the vertical direction between two pairs of key points on the upper and lower eyelids, and ||p1-p4|| represent the Euclidean distances in the horizontal direction between the key points at the left and right corners of the eye. ||p1-p4|| is multiplied by 2 to ensure that the numerator and denominator have the same weight in the calculation.

[0173] The formula for calculating the Euclidean distance between two pairs of key points on the upper and lower eyelids is:

[0174]

[0175] x 38 x 39 x represents the x-coordinate of the key point on the upper eyelid; 42 x 41 y is the x-coordinate of the key point on the lower eyelid; 38 y 39 The y-coordinate of the key point on the upper eyelid; 42 y 41 The vertical coordinate value of the coordinate point corresponding to the key point of the lower eyelid;

[0176] To calculate the pixel width of the eyes, the corresponding pixel width of both eyes is calculated based on the coordinates of the key points at the corners of the eyes. The calculation formula is as follows:

[0177]

[0178] Where ||p1-p4|| are the pixel width values ​​of the left and right corners of the eyes; x 37 y 37 These are the x and y coordinates of the key point on the left corner of the eye, respectively; x 40 y 40 These are the x and y coordinates of the key point on the right corner of the eye, respectively.

[0179] Finally, based on the calculated ||p2-p6||, ||p3-p5||, and ||p1-p4||, they are substituted into the above eye aspect ratio EAR formula to calculate the final EAR value.

[0180] Next, dynamic features are extracted from the image data of the mouth opening and closing state to obtain the first dynamic feature information, the coordinates of the key points of the mouth. The coordinates of the key points of the mouth include the coordinates of the key points on the inner side of the upper lip (3), the coordinates of the key points on the inner side of the lower lip (3), and the coordinates of the key points at the corners of the mouth (2).

[0181] The expression for the first feature information is as follows:

[0182] Based on the coordinates of key points in the mouth, including the coordinates of the inner key points of the upper and lower lip, the sum of the Euclidean distances of the key points in the vertical direction of the upper and lower lip is calculated. Furthermore, based on the coordinates of the inner corners of the mouth, the sum of the Euclidean distances of the left and right corners of the mouth in the horizontal direction is calculated. The formula for calculating the mouth aspect ratio (MAR) is:

[0183]

[0184] Where, p 61 to p 65 Seven key points for the mouth area. ||p 62 -p 68 ||、||p 63 -p 67 ||、||p 64 -p 66 || represents the Euclidean distance between three pairs of key points on the inner sides of the upper and lower lips; ||p 61 -p 65 || represents the Euclidean distance between key points on the inner side of the lips and the left and right corners of the mouth; ||p 61 -p 65 Multiply by 3 to ensure that the numerator and denominator have the same weight in the calculation.

[0185] The formula for calculating the Euclidean distance between the three pairs of key points on the inner sides of the upper and lower lips is:

[0186]

[0187] x 62 x 63 x 64 x represents the x-coordinate of the key point on the inner side of the upper lip; 66 x 67 x 68 The x-coordinate of the key point on the inner side of the lower lip; y 62 y 63 y 64 The y-coordinate of the key point on the inner side of the upper lip; 66 y 67 y 68 The ordinate value of the coordinate point corresponding to the key point on the inner side of the lower lip;

[0188] To calculate the width of the corners of the mouth, the formula is as follows: (The formula is missing from the original text.)

[0189]

[0190] Among them, ||p 61 -p 65 || represents the pixel width of the left and right corners of the mouth; x 61 y 61 The x and y coordinates of the key points on the left side of the inner lip corner of the mouth are respectively the x and y coordinates of the points. 65 y 65 The x-coordinate and y-coordinate values ​​are respectively the coordinates of the key points on the right side of the inner lip corner.

[0191] Finally, based on the calculated ||p 62 -p 68 ||、||p 63 -p 67 ||、||p 64 -p 66 ||and||p 61 -p 65 Substitute this value into the MAR formula for the mouth's aspect ratio (MAR) to calculate the final MAR value.

[0192] The specific implementation of step seven is as follows:

[0193] First, when the driver gets into the vehicle and starts the system, video frame segmentation begins, and facial feature parameter sequences are extracted, denoted as P = {p1, p2, ..., p...}. n}, where p i The facial feature values ​​are for the i-th frame. Obtain the variation curves of the key parameters: eye aspect ratio (EAR) and mouth aspect ratio (MAR).

[0194] Next, the average values ​​of the EAR and MAR variation curves are statistically analyzed to reflect the typical values ​​of the parameters under normal conditions, and these values ​​are used as the reference benchmark values ​​for the adaptive threshold of the dual-gate filter. The formulas for calculating the average values ​​of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows:

[0195]

[0196] Next, an action threshold is set to determine whether an action has ended, selecting a threshold setting method based on feature recovery. In blink detection, only newly generated peak values ​​can be used as the basis for blink action counting after the eye aspect ratio (EAR) parameter drops below this threshold. In yawn detection, only newly generated peak values ​​can be used as the basis for yawn action counting after the mouth aspect ratio (MAR) parameter drops below this threshold. This is to prevent the influence of multiple adjacent peak values ​​with unclear upward and downward trends caused by lens shake due to road bumps on the blink and yawn action counting.

[0197] Finally, the eye aspect ratio EAR threshold (action threshold) in step seven was improved upon from the traditional fixed empirical threshold by adopting a dynamic adjustment method based on the historical average EAR value. This means that the threshold is dynamically adjusted using the average EAR value over a period of time, and the historical record is updated in each frame based on the current EAR value. Furthermore, an adjustment coefficient α is introduced. This is achieved by calculating the average EAR value of the most recent 20 frames and multiplying it by the coefficient α (typically between 1.05 and 1.20, which can be fine-tuned according to actual conditions). The adaptive threshold calculation formula is as follows:

[0198] dynamic_threshold=sum(historical_ears) / len(historical_ears)×α (16)

[0199] Wherein, dynamic_threshold represents the adaptive action threshold, historical_ears represents the historical eye aspect ratio (EAR) value, sum(historical_ears) represents the sum of EAR values ​​over a period of time, len(historical_ears) represents the total number of frames recorded over a period of time, and α represents the dynamic adjustment coefficient, with a value range of [1.05, 1.20].

[0200] Regarding the setting of the mouth aspect ratio (MAR) threshold (action threshold) and value threshold, firstly, the MAR threshold in step seven is selected. According to the formula for calculating the mouth aspect ratio (MAR), the MAR value is positively correlated with the degree of mouth opening and closing: when the mouth is completely closed, the distance between the upper and lower lips is minimal, corresponding to a minimum MAR value; as the opening degree increases, the MAR value increases accordingly. In normal driving conditions, the MAR value during speaking is usually between the closed state and the yawning state. The MAR value during speaking is greater than the MAR value when the mouth is closed, but generally less than the MAR value when yawning.

[0201] Based on the driver's personalized mouth aspect ratio (MAR) parameter baseline value obtained in step seven, a critical threshold M, i.e., the action threshold, can be determined to distinguish between yawning and normal speaking. However, it is worth noting that in practical applications, the MAR value during speaking may briefly exceed the threshold M at certain moments. This is because the mouth exhibits periodic opening and closing characteristics during speaking; even if it occasionally exceeds the threshold M, the duration is relatively short. Therefore, to improve detection accuracy, this method employs a dual judgment mechanism, not only examining whether the MAR value exceeds the threshold M, but also using the duration of the MAR value exceeding the threshold M as an important reference indicator. Through this spatiotemporal combined evaluation method, the accuracy of yawn detection is effectively improved.

[0202] Next, the standard deviation of the statistical facial feature parameter variation curves is used to reflect the range of parameter fluctuations. The formulas for calculating the standard deviation of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows:

[0203]

[0204] Finally, set a threshold value. This threshold is used to exclude invalid peaks caused by image jitter or minor facial movements (such as slight eyelid movements), ensuring that only peaks exceeding a certain amplitude (representing a clear blink or yawn) are counted as valid movements. Set the threshold as follows:

[0205] T υ =μ-k·σ (18)

[0206] Where T υ This represents a threshold value used to filter out values ​​below T in the characteristic curve. υ The invalid peak value is μ, which is the average value of the EAR curve, σ is the standard deviation of the EAR curve, and k is a hyperparameter, which is usually adjusted between [0.5, 1.5].

[0207] The specific implementation of step eight is as follows:

[0208] First, blinking action detection and counting. Based on the eye aspect ratio (EAR) obtained in step six, it is compared with the EAR adaptive threshold (action threshold) and the preset threshold to determine whether the detected person has generated and completed a blinking action. When the EAR values ​​of both eyes are less than the preset EAR action threshold, it is determined that the detected driver has started blinking. When the EAR values ​​of consecutive image frames exceed the EAR threshold again, it is determined that the detected driver has completed the blinking action, and the blink count is incremented by one.

[0209] Next, the yawning action is determined and counted. Based on the mouth aspect ratio (MAR) obtained in step six, it is compared with the preset MAR action threshold and value threshold to determine whether the person being detected has generated and completed the yawning action. When the mouth MAR value is less than the preset MAR action threshold, it is determined that the person being detected has started to yawn. When the MAR value of consecutive image frames exceeds the MAR value threshold again, it is determined that the person being detected has completed the yawning action, and the yawning action count is incremented by one.

[0210] The specific implementation of step nine is as follows:

[0211] First, the formula for calculating blink frequency is as follows:

[0212]

[0213] Where f wink N is the blink frequency. wink The number of blinks is denoted by T, where T is the unit of time.

[0214] Then, the formula for calculating the frequency of yawning is as follows:

[0215]

[0216] In the above formula, f yawn The yawning frequency is defined as the number of yawns per unit time T, where n represents the number of yawns per unit time T. The yawning frequency can also be calculated using the number of frames.

[0217]

[0218] Where N represents the total number of frames within a unit time period, f n Indicate whether the mouth in the nth frame is in a yawning state. If f n A value of 1 indicates a yawning state, f n A value of 0 indicates a normal state.

[0219] The specific implementation of step ten is as follows:

[0220] First, a threshold for blink frequency was set in step ten. Based on extensive driver behavior research data, this study discovered a regular change in blink characteristics as fatigue levels evolved. Under normal alertness, a driver's blink frequency typically remains at 12–18 times per minute. As fatigue levels increase, blink characteristics exhibit distinct phases. In the mild fatigue stage, drivers instinctively increase blinking to combat fatigue, leading to a significant increase in blink frequency and prolonged eye closure time. Research data shows that blink frequency under fatigue conditions increases by 50% to 60% compared to normal levels, reaching 18–27 times per minute. In the moderate to severe fatigue stage, although drivers may keep their eyes open, they often experience distraction, a blank stare, or persistent eye closure, causing a sharp decrease in blink frequency. Furthermore, regarding the blink cycle, a single blink (the process from fully open to fully closed) takes 200–400 milliseconds under normal conditions, while under fatigue conditions, this cycle significantly lengthens to 600–700 milliseconds. Considering individual differences, especially the possibility that some groups may have a higher baseline blinking frequency, this study, after in-depth analysis, determined the fatigue threshold for blinking frequency to be 25 times / minute, in order to balance the accuracy and universality of the detection.

[0221] Next, fatigue is assessed. If the blinking frequency of the driver being tested exceeds the preset upper limit threshold or falls below the preset lower limit threshold, the driver is determined to be in a state of fatigued driving. Alternatively, if the yawning frequency of the driver being tested exceeds the preset yawning frequency threshold, the driver is determined to be in a state of fatigued driving.

[0222] Ultimately, this personalized dual-threshold filtering method uses the driver's personalized facial feature parameters as a benchmark, combined with an adaptive threshold adjustment mechanism. It can automatically adjust the action and value thresholds of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) based on the driver's real-time state and dynamic environmental changes (such as lighting conditions, driving environment, and individual driver differences). Through this dynamic optimization mechanism, the system maintains high accuracy in different scenarios. Simultaneously, the dual-threshold filtering effectively suppresses camera shake caused by road bumps, thus avoiding interference from multiple adjacent invalid peaks lacking a clear upward or downward trend in blink and yawn counts, significantly improving the overall accuracy, stability, and environmental adaptability of the vehicle fatigue driving detection system. Furthermore, by designing a novel CNN structure as the facial feature parameter head, the geometric relationships of key points are directly embedded into the heatmap, and heatmap information of specific key points in the eye and mouth regions is fused to construct a complete feature representation. Finally, the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are learned and fitted from the fused heatmap, achieving efficient extraction of facial key point features. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism has opened up a new direction for research and application in the field of fatigue driving detection, and promoted the practical and intelligent development of the system in complex scenarios.

[0223] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism, characterized in that: Includes the following steps: Step S1: Image acquisition, acquiring real-time facial image data of the driver to be detected; Step S2: Image preprocessing. The collected face image data is standardized, including size unification, region cropping, color channel conversion and normalization. Step S3: Input the preprocessed image into the trained FaRL model; Step S4: Face region detection, using the face detection model MTCNN to extract face regions; Step S5: Key point recognition. Facial key point recognition is performed using the FaRL model vision module. The facial image data is input into the preset face detection model for processing. Step S6: Based on the facial key point coordinate data, calculate the two important facial feature parameters, the eye aspect ratio EAR and the mouth aspect ratio MAR. Step S7: Extract the driver's personalized facial feature parameters and set the action threshold and value threshold values ​​for the eye aspect ratio EAR and mouth aspect ratio MAR. When using it for the first time, the driver's facial video is sampled and analyzed, and the driver's personalized facial feature parameters are extracted from the key frames as the dual-threshold filtering reference value. Based on the reference value, the action threshold and value threshold values ​​for the eye aspect ratio EAR and mouth aspect ratio MAR are set respectively. Step S8: Process the eye aspect ratio EAR, compare it with the adaptive eye aspect ratio EAR threshold to determine whether a blinking action has occurred, and compare it with the eye aspect ratio EAR threshold value to determine whether the blinking action has been completed; process the mouth aspect ratio MAR, compare it with the adaptive mouth aspect ratio MAR threshold value to determine whether a yawning action has occurred, and compare it with the mouth aspect ratio MAR threshold value to determine whether the yawning action has been completed. Step S9, Frequency Statistics: Count the number of blinks per unit time according to the blinking action determination method in S8, and thus obtain the blinking frequency; Count the number of yawns per unit time according to the yawning action determination method in S8, and thus obtain the yawning frequency. Step S10: Determine if the driver is in a state of fatigued driving. Compare the blinking frequency and yawning frequency per unit time in S9 with the preset blinking frequency threshold and yawning frequency threshold, respectively, to determine if the driver is in a state of fatigued driving. Alternatively, determine if the driver is in a state of fatigued driving by judging whether the time the driver is in a closed-eye state exceeds the preset fatigued closed-eye time threshold.

2. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S11: Image acquisition uses an independent image acquisition module, including a regular image sensor and an infrared image sensor. The infrared image sensor includes an infrared fill light and an infrared camera. At the beginning of acquisition, the regular image sensor is used for acquisition. The image acquisition module determines whether the light is qualified. If it is not qualified, the infrared fill light is activated, and then the module switches to the infrared camera for acquisition to obtain the facial image data of the driver to be detected.

3. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S2 are as follows: Step S21: Perform image preprocessing, namely, extracting face image data by unifying size, cropping, converting color channels and normalizing. First, generate a unified image size of 447×447. Step S22: Perform color channel conversion on the image. If the input image is grayscale or BGR format, convert the image format to RGB format. Step S23: Normalize the image to map the pixel values ​​from the range [0,255] to [0,1]. Step S24: Standardize the normalized image by subtracting the channel's training set mean from each pixel value and dividing the result by the channel's training set standard deviation; unify the numerical range of different features and adjust the image distribution to the standardized state during model training.

4. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S4 are as follows: Step S41: Use the MTCNN face detection model to detect the location of faces in the image; output the bounding box of each face and return the coordinates (x, y) of the face bounding box. box ,y box (width, height); Step S42: To ensure the complete face region is included, expand the width and height of the bounding box by 10%-20%; the formula for calculating the new face bounding box is as follows: Where new_width and new_height are the width and height of the face bounding box after expanding by 10%-20%, respectively, p is the expansion ratio, with a value range of [0.1, 0.2], and x... box_new and y box_new These are the x and y coordinates of the top left corner of the new face bounding box, respectively; Step S43: Face cropping. Based on the updated face bounding box, crop out the sub-region containing the face from the original image.

5. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step S51: Use the trained FaRL visual encoder as the backbone; Step S52: Select the feature maps of layers 4, 6, 7, and 12 for multi-level feature fusion, and use UpperNet to integrate the multi-level feature maps; Step S53: Use 1×1 convolution to generate heatmap predictions of key points as the output layer; Step S54: Render the ground truth key points into a Gaussian heatmap of size 127×127, model the key point positions using Gaussian distribution and set its standard deviation to 1 pixel, and the heatmap value range is [0,1]. Step S55: Based on the relationship between the detected key points and the pre-set standard template, use affine transformation and apply geometric transformations such as translation, rotation, and scaling to adjust the face to the position of the target template. Step S56: Heatmap fusion; The soft-argmax heatmap regression method is used, defined as follows: In the formula, d is a given component x or y, and P is the weight matrix of W×H×2 corresponding coordinates (x,y); Matrix P uses its components P x and P y This indicates that both components are two-dimensional discrete normalized linear mappings, defined as follows: Here, Φ(h) i,j The softmax result of a single heatmap is defined as follows: This method is considered as a convolution with a kernel size of H×W, where the kernel contains x or y position information, and is ordered according to pixel position Φ(h i,j Arrange the key points; based on the variation characteristics of the x-axis and y-axis, perform a convolution on the weighted heatmap of the key points, and after normalization and other processing, obtain the x-coordinates and y-coordinates of the key points; Step S57: Analyze the geometric relationships between facial key points; skip the regression and geometric calculations for each key point and analyze the overall facial features; Based on the definition of facial key points in the 300-W dataset, markers 36-47 were classified as eye feature key points, and inner lip markers 61-67 were classified as mouth feature key points. definition Overall facial feature weight Φ(h') i,j )for: In the formula, n and m represent the serial numbers of the key points of the eye features and the key points of the mouth features, respectively; The overall heatmap M is defined as: Step S58, coordinate embedding; When processing the overall heatmap extracted from multiple images, we draw on the core idea of ​​the soft-argmax algorithm and design a dual coordinate embedding mechanism to enhance the feature representation in the heatmap matrix. In the specific implementation process, the overall heatmap matrix is ​​first pre-processed with zero padding. When H≤W, zeros are padded in the (WH) column to the right of M. When H>W, zeros are padded in the (HW) row to the bottom of M, thus converting it into a normal square matrix structure. Next, a diagonal matrix diag(0,1,...K-1) is constructed; this diagonal matrix is ​​used to represent the coordinate information of each column or row in the heatmap; finally, by multiplying the diagonal matrix by the square matrix M′ in both the left and right directions, linear weighting by row and column is achieved to obtain the coordinate information in the horizontal and vertical directions, respectively, and the position information is embedded into the feature map, thus realizing coordinate embedding on the overall heatmap: Where H and W represent the height and width of the heatmap, respectively, and M x M represents the result of embedding the heatmap in the x-direction. y This represents the result of embedding the heatmap in the y-direction; in the formula, K = max(W,H) ensures that the diagonal matrix dimension matches the square matrix M′; in this way, the geometric relationship between the relative coordinates of key points in the overall heatmap is emphasized; after coordinate embedding, the facial feature parameters required for driver fatigue detection are extracted; Step S59: Extract data containing facial key points from the fused heatmap after coordinate embedding. Key points 36 to 47 are classified as eye key points, and key points 61 to 67 are classified as mouth key points. Extract the corresponding facial image data.

6. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 5, characterized in that: The specific steps of step S6 are as follows: Step S61: Dynamic feature extraction is performed on the image data of the eye opening and closing state to obtain the first dynamic feature information, the coordinates of key points of both eyes, including the coordinates of 2 upper eyelid key points, 2 lower eyelid key points, and 2 corners of the eyes key points. The expression for the left eye in the coordinates of the key points of both eyes in the first feature information is as follows: Based on the coordinates of the key points on the upper and lower eyelids, calculate the sum of the Euclidean distances in the vertical direction between the two pairs of key points, i.e., the distance from the upper eyelid to the lower eyelid. Simultaneously, based on the coordinates of the key points at the corners of the eyes, calculate the horizontal distance between the left and right corners of the eyes. The formula for calculating the aspect ratio (EAR) of the eye is: In the formula, p1 to p6 are the 6 key points of the eye; the numerators ||p2-p6|| and ||p3-p5|| represent the Euclidean distance between the corresponding pairs of key points of the eye on the upper and lower eyelids in the vertical direction, and ||p1-p4|| represent the Euclidean distance between the key points of the eye at the left and right corners of the eye in the horizontal direction. ||p1-p4|| is multiplied by 2 to ensure that the numerator and denominator have the same weight in the calculation. The formula for calculating the Euclidean distance between two pairs of key points on the upper and lower eyelids is: x 38 x 39 x represents the x-coordinate of the key point on the upper eyelid; 42 x 41 y is the x-coordinate of the key point on the lower eyelid; 38 y 39 The y-coordinate of the key point on the upper eyelid; 42 y 41 The vertical coordinate value of the coordinate point corresponding to the key point of the lower eyelid; The pixel width of the eyes is calculated by determining the pixel width of both eyes based on the coordinates of the key points at the corners of the eyes. The calculation formula is as follows: Where ||p1-p4|| are the pixel width values ​​of the left and right corners of the eyes; x 37 y 37 These are the x and y coordinates of the key point on the left corner of the eye, respectively; x 40 y 40 These are the x and y coordinates of the key point on the right corner of the eye, respectively. Step S62: Dynamic feature extraction is performed on the image data of the mouth opening and closing state to obtain the first dynamic feature information, the coordinates of the key points of the mouth. The coordinates of the key points of the mouth include the coordinates of 3 key points on the inner side of the upper lip, the coordinates of 3 key points on the inner side of the lower lip, and the coordinates of 2 key points at the corners of the mouth. The expression for the coordinates of the key points of the mouth, the first feature information, is as follows: Based on the coordinates of the key points of the mouth, including the coordinates of the inner corners of the upper and lower lips, the sum of the Euclidean distances of the key points of the upper and lower lips in the vertical direction is calculated. Furthermore, based on the coordinates of the inner corners of the mouth, the sum of the Euclidean distances of the left and right corners of the mouth in the horizontal direction is calculated. The formula for calculating the aspect ratio (MAR) of the mouth is: Where, p 61 to p 65 Seven key points for the mouth; ||p 62 -p 68 ||、||p 63 -p 67 ||、||p 64 -p 66 || represents the Euclidean distance between three pairs of key points on the inner sides of the upper and lower lips; ||p 61 -p 65 || represents the Euclidean distance between key points on the inner side of the lips and the left and right corners of the mouth; ||p 61 -p 65 Multiply by 3 to ensure that the numerator and denominator maintain the same weight in the calculation; The formula for calculating the Euclidean distance between the three pairs of key points on the inner sides of the upper and lower lips is: x 62 x 63 x 64 x represents the x-coordinate of the key point on the inner side of the upper lip; 66 x 67 x 68 The x-coordinate of the key point on the inner side of the lower lip; y 62 y 63 y 64 The y-coordinate of the key point on the inner side of the upper lip; 66 y 67 y 68 The ordinate value of the coordinate point corresponding to the key point on the inner side of the lower lip; Calculate the width of the corners of the mouth. The width corresponding to the corners of the mouth is calculated based on the coordinates of the key points at the corners of the mouth. The calculation formula is as follows: Among them, ||p 61 -p 65 || represents the pixel width of the left and right corners of the mouth; x 61 y 61 The x and y coordinates of the key points on the left side of the inner lip corner of the mouth are respectively the x and y coordinates of the points. 65 y 65 The x-coordinate and y-coordinate values ​​are respectively the coordinates of the key points on the right side of the inner lip corner.

7. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S7 are as follows: Step S71: When the driver first gets into the vehicle and starts the system, start video frame segmentation and extract the facial feature parameter sequence, denoted as P = {p1, p2, ..., p...} n }, where p i The facial feature values ​​for the i-th frame are obtained; the variation curves of the key parameters, eye aspect ratio (EAR) and mouth aspect ratio (MAR), are acquired. Step S72: Calculate the average values ​​of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) variation curves to reflect typical values ​​of the parameters under normal conditions, and use them as a reference benchmark for the adaptive threshold of the dual-gate filter; the formulas for calculating the average values ​​of eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows: Where u is the average of the eye aspect ratio (EAR) and mouth aspect ratio (MAR), n is the total number of frames per unit time, and p i Let be the facial feature value of the i-th frame; Step S73: Set an action threshold value to determine whether an action has ended. Select a threshold setting method based on feature recovery. During blink detection, it is stipulated that only when the eye aspect ratio EAR parameter drops below the threshold can the newly generated peak value be used as the basis for blink action counting. During yawn detection, it is stipulated that only when the mouth aspect ratio MAR parameter drops below the threshold can the newly generated peak value be used as the basis for yawn action counting. Step S74: For the eye aspect ratio EAR threshold in S7, a dynamic adjustment method based on the historical average eye aspect ratio EAR is adopted. That is, the threshold is dynamically adjusted by using the average eye aspect ratio EAR value over a period of time, and the historical record is updated according to the current eye aspect ratio EAR value in each frame. In addition, an adjustment coefficient α is introduced. By calculating the average eye aspect ratio EAR value of the most recent 20 frames and multiplying it by the coefficient α, the dynamic adjustment of the action threshold value is achieved. The adaptive threshold calculation formula is as follows: dynamic_threshold=sum(historical_ears) / len(historical_ears)×α (16) Where dynamic_threshold represents the adaptive action threshold, historical_ears represents the historical eye aspect ratio (EAR) value, sum(historical_ears) represents the sum of EAR values ​​over a period of time, len(historical_ears) represents the total number of frames recorded over a period of time, and α represents the dynamic adjustment coefficient, with a value range of [1.05, 1.20]. Step S75: Select the mouth aspect ratio (MAR) threshold from step S7. As shown in the mouth aspect ratio (MAR) calculation formula in step S62, the MAR value is positively correlated with the degree of mouth opening and closing: when the mouth is completely closed, the distance between the upper and lower lips is the smallest, and the corresponding mouth aspect ratio (MAR) value reaches its minimum; as the degree of opening increases, the mouth aspect ratio (MAR) value increases accordingly; in normal driving conditions, the mouth aspect ratio (MAR) value when speaking is usually between the closed state and the yawning state; the mouth aspect ratio (MAR) value when speaking is greater than the MAR value when closed, and generally less than the MAR value when yawning. Based on the driver's personalized mouth aspect ratio (MAR) parameter baseline value obtained in step S7, a critical threshold M, i.e., the action threshold value, is determined to distinguish between yawning and normal speaking states. It also employs a dual judgment mechanism, which not only examines whether the mouth aspect ratio (MAR) exceeds the threshold M, but also uses the duration of the MAR value exceeding the threshold M as an important reference indicator. Step S76: Based on step S71, calculate the standard deviation of the facial feature parameter variation curves to reflect the fluctuation range of the parameters; the formulas for calculating the standard deviation of the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are as follows: Set a threshold value. The threshold is used to exclude invalid peaks caused by image jitter or minor facial movements, ensuring that only peaks exceeding a certain amplitude are counted as valid movements. Set the threshold as follows: T υ =μ-k·σ (18) Where T υ This represents a threshold value used to filter out values ​​below T in the characteristic curve. υ The invalid peak value is μ, which is the average value of the eye aspect ratio EAR curve, σ is the standard deviation of the eye aspect ratio EAR curve, and k is a hyperparameter that is adjusted between [0.5, 1.5].

8. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S8 are as follows: Step S81: Blinking action determination and counting. Based on the eye aspect ratio (EAR) obtained in step S6, compare it with the EAR adaptive threshold and the preset threshold value to determine whether the detected person has generated and completed a blinking action. When the EAR values ​​of both eyes are less than the preset EAR action threshold value, it is determined that the detected driver has started to generate a blinking action. When the EAR value of consecutive image frames exceeds the EAR value threshold value again, it is determined that the detected driver has completed this blinking action, and the blink count is incremented by one. Step S82: Yawning Action Judgment and Counting. Based on the mouth aspect ratio (MAR) obtained in step S6, compare it with the preset MAR action threshold and value threshold to determine whether the person being detected has generated and completed the yawning action. When the mouth MAR value is less than the preset MAR action threshold, it is determined that the person being detected has started to generate the yawning action. When the MAR value of consecutive image frames exceeds the MAR value threshold again, it is determined that the person being detected has completed the yawning action, and the yawning action count is incremented by one.

9. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S9 are as follows: Step S91, the formula for calculating blink frequency is as follows: Where f wink N is the blink frequency. wink The number of blinks per unit time, where T is the unit time; Step S92, the formula for calculating yawning frequency is as follows: Where f yawn The yawning frequency is defined as the number of yawns per unit time T, where n represents the number of yawns. The yawning frequency can also be calculated using the number of frames. Where N represents the total number of frames within a unit time period, f n Indicate whether the mouth in the nth frame is in a yawning state. If f n A value of 1 indicates a yawning state, f n A value of 0 indicates a normal state.

10. The fatigue driving detection method based on personalized dual-threshold filtering and adaptive threshold mechanism according to claim 1, characterized in that: The specific steps of step S10 are as follows: Step S101: Set the blink frequency threshold for step S10, and determine the fatigue judgment threshold for blink frequency as 25 times / minute. Step S102: Fatigue determination. When the blinking frequency of the driver being tested exceeds the preset upper limit threshold or falls below the preset lower limit threshold, the driver being tested is determined to be in a state of fatigued driving; or when the yawning frequency of the driver being tested exceeds the preset yawning frequency threshold, the driver being tested is determined to be in a state of fatigued driving.

Images