Face fatigue dynamic detection method fusing aggregation module

Abstract

The application discloses a face fatigue dynamic detection method fusing an aggregation module, and is suitable for the field of video detection of face fatigue states. The face image collected is preprocessed, and the designed module aggregated by ordinary convolution and hollow convolution is stacked to extract features; a cross-shaped cross attention is added to extract small target detail information; a detection network for eye and mouth features is built, so that the model has good small target detection capability of eyes and mouths, so as to extract the features of eyes and mouths from the preprocessed face image. By using a boundary box regression loss function, an angle, a distance and a shape are added as penalty terms, the angle loss is designed as a linear function, the eye features and the mouth features are fused to judge the fatigue state of a person, and a dynamically adjusted closed-eye duration index is designed to improve the real-time performance of fatigue discrimination. The method has high fatigue state detection accuracy, low training time cost and good real-time performance, and can reduce accidents caused by fatigue.

Description

Technical Field

[0001] This invention belongs to the field of video detection of facial fatigue state, and particularly relates to a dynamic detection method for facial fatigue using a fusion and aggregation module. Background Technology

[0002] Prolonged fatigue not only affects physical and mental health but also increases the risk of accidents, especially in situations requiring concentration, such as driving and operating equipment. Therefore, to protect human health and improve travel and workplace safety, fatigue detection technology has broad application prospects. Compared to biomedical-based detection technologies, such as using wearable sensors to collect EEG and ECG signals to assess fatigue levels, the technology of extracting facial features from facial images or videos using neural network models for fatigue detection is more convenient and easier to promote.

[0003] The core of neural network-based facial fatigue detection methods lies in facial feature extraction and fatigue state analysis and discrimination. In the facial feature extraction stage, common methods are mainly divided into facial landmark detection models and object detection models. Facial landmark detection models can accurately capture changes in the shape of facial organs, but in poor lighting conditions, when landmarks are occluded, or in blurry frames, these methods may fail to determine fatigue states, exhibiting poor robustness. Object detection models typically use neural networks to directly provide the state of the detected target, with simple judgment logic and strong robustness. However, these methods usually target small targets such as eyes, and neural network models are prone to losing features of small targets during feature extraction, thus reducing the accuracy of object detection-based fatigue detection. Furthermore, these methods rely heavily on the quantity and quality of the dataset, resulting in high training time and labor costs. In the fatigue discrimination stage, existing methods mainly use indicators such as blink frequency, PERCLOS, and yawning frequency, but these indicators often require long statistical times (e.g., counting the number of times within one minute), leading to poor real-time performance in fatigue discrimination.

[0004] Existing neural network-based methods for detecting facial fatigue state have the following problems: (1) they lack the ability to detect small targets such as eyes; (2) the model training time is long and the labor cost is high; and (3) the fatigue discrimination method has poor real-time performance. Therefore, in order to overcome the above problems, this invention provides a new neural network-based method for detecting facial fatigue state. Summary of the Invention

[0005] To address the shortcomings of existing technologies, a dynamic facial fatigue detection method integrating aggregation modules is provided. This method has a simple structure and is easy to use. By combining multiple judgment criteria, it can quickly and accurately determine the driver's fatigue state through the collected driver's facial images. This, in conjunction with an alert scheme, can effectively reduce the possibility of accidents caused by driver fatigue.

[0006] To achieve the above objectives, this invention discloses a method for dynamic detection of facial fatigue using a fusion aggregation module, characterized by the following steps:

[0007] Step S1: Use a camera device to capture facial images in real time at a fixed frame rate;

[0008] Step S2: Preprocess the acquired face images to reduce the impact of poor lighting and noise on the detection results;

[0009] Step S3: Construct a dynamic face fatigue detection network, including a feature extraction network, a feature fusion network, and a detection end connected in sequence; the feature extraction network includes convolutional layers (CBL), max pooling layers (MP), and an ELAN-Dilated module; a cross-junction attention module (CCNet) is also included between the feature extraction network and the feature fusion network; the feature fusion network and the detection end include the structure of the Neck and Head parts of the YOLOv7-tiny network; among them, the ELAN-Dilated module includes a structure of stacked and aggregated ordinary convolutional layers (CBL) and dilated convolutional layers;

[0010] The feature extraction network includes two sequentially connected convolutional layers (CBL), four ELAN-Dilated modules, and a cross-attention network (CCNet). Max pooling layers (MP) are placed between the ELAN-Dilated modules. Each ELAN-Dilated module has two branches, the first branch passing through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. Dilated convolutions with dilation rates of 2 and 5 are used for feature extraction. Finally, the features from each convolutional layer of the module are stacked and then processed again using a... The number of channels is changed through convolution;

[0011] in:

[0012] Two convolutional layers (CBL) perform coarse feature extraction on face images, including enhancing edge features and suppressing some noise.

[0013] Four ELAN-Dilated modules extract features of the eyes and mouth from shallow to deep layers;

[0014] Max pooling layers (MP) are used to compress the feature parameters of the eyes and mouth, reduce dimensionality, and decrease computational cost.

[0015] The Cross-Attention Module (CCNet) is used to extract eye and mouth features more efficiently by acquiring rich contextual information about the eyes and mouth and their surroundings.

[0016] Step S4: Use a fully trained face fatigue dynamic detection network to extract eye and mouth features from each frame of the preprocessed face image, including open eyes, closed eyes, yawning, and no yawning;

[0017] Step S5: Based on the fixed frame rate of the face images captured by the camera device, extract the eye and mouth features of each face image according to the face fatigue dynamic detection network, and calculate the duration of closed eyes, the percentage of closed eyes, the PERCLOS value, and the yawning time interval.

[0018] Step S6: Perform fatigue state analysis by integrating the duration of eye closure, the percentage of eye closure duration in PERCLOS value, and the yawning interval, and output the fatigue state analysis results.

[0019] Furthermore, the specific steps for preprocessing the acquired facial images are as follows:

[0020] Step S21: Perform grayscale processing on the face image to reduce the running memory of the face fatigue detection method and shorten the time it takes for the system to obtain results;

[0021] Step S22: Perform median filtering on the face image to remove some noise;

[0022] Step S23: Perform block histogram equalization on the filtered face image to improve the contrast of the face image in poor lighting conditions.

[0023] Furthermore, the specific steps for constructing the dynamic facial fatigue detection network are as follows:

[0024] Step S31: Construct the convolutional layer aggregation module ELAN-Dilated to extract features of the eyes and mouth;

[0025] The ELAN-Dilated module consists of stacked dilated convolutions and regular convolutions, and includes two branches. The first branch passes through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. Dilated convolutions with dilation rates of 2 and 5 are used for feature extraction. Finally, the features from each convolutional layer of the module are stacked and then processed again using a... The convolution modulates the number of channels; this module has a large receptive field and a strong ability to learn multi-scale features, which enables the model to better preserve the spatial features of the image, reduce information loss, and effectively preserve the features of the eyes and mouth.

[0026] Step S32: Construct a feature extraction network by stacking ELAN-Dilated modules;

[0027] The feature extraction network first passes through two convolutional layers to suppress some noise and enhance the edge features of the eyes and mouth; then it passes through four ELAN-Dilated modules to extract the features of the eyes and mouth from shallow to deep; and max pooling layers are added between the ELAN-Dilated modules to compress the features and reduce the number of parameters and computation.

[0028] Step S33: Construct a feature fusion network;

[0029] The feature fusion network adopts the structure of the Neck part of the YOLOv7-tiny network, which includes a structure combining the feature pyramid FPN and PANet. Through continuous upsampling and downsampling, it performs multi-scale feature fusion on the shallow, medium and deep features of the eyes and mouth extracted by the last three ELAN-Dilated modules in the feature extraction network.

[0030] The feature fusion network uses multi-scale feature fusion to capture the features of the eyes and mouth at different scales, thereby enhancing the features of the eyes and mouth such as edges, textures, and shapes, and improving the ability of the face fatigue dynamic detection network of the fusion aggregation module to locate and detect the features of the eyes and mouth.

[0031] Step S34: Set up the detection terminal;

[0032] The detection end adopts the structure of the YOLOv7-tiny network head part, which includes constructing a detection head from three ELAN-tiny modules of the feature fusion network, and each detection head uses one Feature extraction is performed using convolutional layers, and the feature maps obtained by the three detector heads are respectively... , and These are used to detect large, medium, and small target features, respectively; ELAN-tiny includes two branches, the first branch passing through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. The convolutions are used to extract features, and finally the features from each convolutional layer of the module are stacked and then used again. The number of channels is changed through convolution;

[0033] Since the positions and sizes of the detected faces are different, the positions and sizes of the eyes and mouths in the image are also different. The detection end sets up three detection heads to independently detect the eyes and mouths from feature maps of three scales, so as to improve the detection capability of the face fatigue dynamic detection network of the fusion aggregation module for eye and mouth features.

[0034] Step S35: Add a cross-attention network;

[0035] A cross-cutting attention network (CCNet) is added after the last ELAN-Dilated module of the feature extraction network. CCNet consists of two branches. The first branch passes through two cross-cutting attention modules (Criss-Cross Attention Modules) to extract rich and dense contextual information. The second branch passes through a step size of 1 and a size of [missing information]. The number of channels in the convolutional layer is adjusted, and then the features of the two branches are concatenated to fuse dense contextual features and local features. In this case, a single cross-attention module only extracts contextual information in the horizontal and vertical directions. By concatenating two cross-attention modules, the feature map can actually collect information from all pixels at each position, thus extracting richer and denser contextual information.

[0036] Furthermore, during the training process of the facial fatigue dynamic detection network, a loss function is used to quantify the deviation between the prediction results and the actual results, so as to accurately predict the eye and mouth features in facial images, including four targets: open eyes, closed eyes, yawning mouth, and non-yawning mouth; the loss function design process is as follows.

[0037] The loss function includes bounding box regression loss, bounding box confidence loss, and class confidence loss. Bounding box regression loss measures the degree of overlap between the predicted bounding boxes and the ground truth bounding boxes of the eye and mouth targets. Bounding box confidence loss measures the confidence level of the predicted bounding boxes of the eye and mouth targets. Class confidence loss measures the degree of closeness between the predicted class and the ground truth class of the eye and mouth state.

[0038] The bounding box confidence loss and class confidence loss are calculated using the binary cross-entropy function (BCE). The bounding box regression loss is designed to integrate the direction or angle, center distance, shape, and cross-union ratio of the predicted box and the ground truth box, so that the predicted boxes of the eyes and mouth of people in the image are as close as possible to the ground truth boxes, thereby accelerating the convergence speed during network training.

[0039] The formula for the bounding box regression loss of the face fatigue dynamic detection network is as follows:

[0040] ,

[0041] ,

[0042] ,

[0043] Wherein, IoU represents the intersection-union loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. The distance loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth is represented. This represents the shape loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This represents the loss in orientation or angle between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This indicates the degree to which the shape loss of the predicted boxes for the eyes and mouth is penalized against the total loss in relation to the ground truth boxes. , These represent the weights for adjusting the distance loss and shape loss, respectively. and These represent the width and height of the prediction box, respectively. and These represent the width and height of the actual bounding boxes for the eyes and mouth, respectively. and These represent the x and y coordinates of the center points of the prediction boxes for the eyes and mouth, respectively. and Let x and y represent the x and y coordinates of the center points of the true bounding boxes for the eyes and mouth, respectively. and These represent the width and height of the minimum closure rectangle between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth, respectively.

[0044] Furthermore, since the eyes and mouth are small target features, the facial fatigue dynamic detection network still loses some information when extracting features of the eyes and mouth, and is more susceptible to interference from information around the eyes and mouth, such as misidentifying the top and bottom edges of black glasses as closed eyes. By using a loss function that fuses the direction or angle, center distance, shape, and intersection-union ratio of the predicted box and the ground truth box, the facial fatigue dynamic detection network can still be penalized when the predicted box is close to the ground truth box, reducing the interference of information around the ground truth box on the prediction result, making the predicted box closer to the ground truth box, and improving the accuracy of the result.

[0045] Specifically, regarding the angle loss, the angle loss reaches its maximum when the horizontal angle between the predicted box and the ground truth box is approximately 30° to 60°; when the horizontal angle between the predicted box and the ground truth box is in the range of 0° to 30° and 60° to 90°, the closer the predicted box is to the ground truth box, the greater the angle loss. shaft or The smaller the angle loss, the faster the predicted bounding box can quickly approximate the ground truth bounding box. shaft or The axis reduces the degrees of freedom of the prediction box, thus accelerating the convergence of the dynamic face fatigue detection network model.

[0046] The calculation methods for the duration of eye closure, the percentage of eye closure time in PERCLOS, and the yawning interval are as follows:

[0047] (1) Duration of closing eyes

[0048] Define eye closure as when the area of ​​the eyelid covering more than 80% of the pupil is considered closed. The duration of eye closure is calculated based on the number of frames continuously returned by the facial fatigue dynamic detection network to the closed-eye state. The formula is as follows:

[0049] ,

[0050] Since prolonged eye closure poses significant safety risks, the duration of eye closure is used as one of the indicators for fatigue assessment. When the eye closure duration is around 0.2 seconds, it is not included in the fatigue state analysis. As the eye closure duration increases, its weight in the fatigue analysis is gradually increased. When the eye closure duration reaches around 2 seconds, it completely dominates the fatigue state analysis. It can output a moderate fatigue state when the eye closure duration reaches around 1 second and a severe fatigue state when it reaches around 2 seconds, issuing a strong warning and improving the real-time performance of fatigue assessment.

[0051] (2) PERCLOS value

[0052] The PERCLOS value specifically refers to the time the eyes are closed per unit of time. The criterion is that the eyelids cover more than 80% of the pupil as the eyes are considered closed. The following formula is used... The PERCLOS value is expressed by the following formula:

[0053] ,

[0054] (3) Yawning interval

[0055] The feature distinguishing yawning from speaking is a wide-open mouth that lasts for 3-5 seconds. A yawn is defined as the number of consecutive frames returned by the facial fatigue dynamic detection network when the number of frames reaches a certain threshold. The interval between two yawns is recorded as... .

[0056] Furthermore, the specific steps for analyzing fatigue state by integrating the duration of eye closure, the percentage of eye closure time in PERCLOS values, and the yawning interval, and outputting the fatigue state analysis results, include:

[0057] Step S61: Normalize the duration of eye closure, PERCLOS value, and yawn interval to obtain the eye closure duration score. PERCLOS rating And the yawning interval score The specific formula is as follows:

[0058] (1) Eyes closed duration score

[0059] When the eyes are closed for a relatively long period, the duration of eye closure should be the dominant characteristic in fatigue state analysis; during normal blinking, the duration of eye closure should not be included in the fatigue state analysis. The normalization formula for the duration of eye closure is as follows:

[0060]

[0061] Among them, parameters and The eye-closing duration score was divided into three stages: normal blinking, fatigue state, and prolonged eye-closing; parameters and The settings must meet the following requirements:

[0062] a. The threshold for distinguishing between the normal blinking phase and the phase of increased fatigue should be set to the maximum duration of a normal blink, approximately 0.4 seconds;

[0063] b. The threshold for distinguishing between the fatigue state stage and the prolonged closed-eye stage should be set as the minimum time from when a person closes their eyes to when a dangerous situation occurs; this threshold setting will vary depending on the environment in which the fatigue test subject is located; for example, when driving a vehicle, the minimum time from when the person closes their eyes to when a dangerous situation occurs should be used.

[0064] c. During the normal blinking phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 0; during the fatigue phase, as the duration of eye closure increases, the score for duration of eye closure increases almost linearly; during the prolonged eye closure phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 1.

[0065] (2) PERCLOS score

[0066] Fatigue state is determined by whether the PERCLOS score has a stable fluctuation range. The PERCLOS score is normalized using the minimum-maximum normalization method, as shown in the following formula:

[0067] ,

[0068] in, The PERCLOS value detected in real time. and These represent the maximum and minimum values ​​of PERCLOS during the detection process;

[0069] Statistical experiments were conducted to divide the PERCLOS score into three stages: normal blinking, moderate fatigue, and severe fatigue. The threshold for dividing the normal blinking and moderate fatigue stages was the maximum value of PERCLOS when blinking normally, which was set to 0.15. The threshold for dividing the moderate fatigue and severe fatigue stages was the minimum value of PERCLOS when it seriously affected a person's current behavior or posed a significant safety hazard.

[0070] (3) Yawning interval score

[0071] Based on the core idea of ​​minimax normalization, an overflow boundary is set, and the output value is reversed. The formula is as follows:

[0072] ,

[0073] in, The parameter represents the real-time detected yawn interval. and The yawning interval was divided into three stages: normal fatigue, moderate fatigue, and severe fatigue; parameters and The settings must meet the following requirements:

[0074] a. Parameters It is the time threshold for dividing the ordinary fatigue state stage and the severe fatigue stage. Its value is set to be slightly larger than the interval of frequent yawning, and the value is set within 5 to 10 seconds.

[0075] b. Parameters It is the time threshold for dividing the normal state stage and the ordinary fatigue stage, and the value is set within 1 to 2 minutes;

[0076] Step S62: Fusion of the normalized eye-closing duration, PERCLOS value, and yawn interval to obtain a fatigue state score. The formula is as follows:

[0077]

[0078] Fatigue status score Always Within the specified range, values ​​exceeding 1 are still represented as 1. A fatigue score closer to 0 indicates milder fatigue, while a score closer to 1 indicates more severe fatigue. During normal blinking, eye-closing duration scores close to 0 have almost no impact on fatigue assessment; however, scores for longer eye-closing durations dominate fatigue evaluation. PERCLOS scores and yawn intervals are calculated using... and Perform weighted fusion;

[0079] Step S63: Score the fatigue state. The fatigue state is classified into different levels, and corresponding measures are set for each level.

[0080] The closer the fatigue status score is to 0, the lighter the fatigue status is; the closer it is to 1, the heavier the fatigue status is. The fatigue status score is divided into three fatigue levels and one non-fatigue level. The fatigue level from heavy to light is recorded as the first fatigue state, the second fatigue state, the third fatigue state, and the non-fatigue state.

[0081] The intervals for the fatigue status score should be determined based on the intervals for the three score values: eye-closing duration score, PERCLOS score, and yawning interval score.

[0082] The first fatigue state is a state with a high safety hazard. Under this state, at least one of the following conditions must be met:

[0083] a) The score for duration of eye closure is in the prolonged eye closure stage;

[0084] b) The PERCLOS score indicates severe fatigue.

[0085] The second state of fatigue is moderate fatigue, during which the brain's judgment and reaction are slower, posing certain safety risks. In this state, at least one of the following situations may occur:

[0086] a) The score for duration of eye closure indicates a state of fatigue;

[0087] b) The PERCLOS score indicates a moderate fatigue level;

[0088] c) The interval between yawns is during the normal or severe fatigue stage;

[0089] The third fatigue state is a mild fatigue state, in which at least one of the three rating values ​​is not in the normal state stage;

[0090] The non-fatigue state is the normal blinking state. In this state, the eye closure duration score and PERCLOS are both in the normal blinking state, and the yawning interval score is in the normal state.

[0091] When driving, measures that can be taken in the first state of fatigue include strong warning signals, such as a rapid and loud warning tone or a clear voice prompt; measures that can be taken in the second state of fatigue include a shorter warning tone; measures that can be taken in the third state of fatigue include automatically turning on the in-car music system and playing cheerful music; no measures are taken when not fatigued.

[0092] The beneficial effects of this invention are:

[0093] (1) This invention reduces the memory usage of the face fatigue detection method by image preprocessing, and reduces the impact of poor lighting and low-quality, high-noise images on the detection results.

[0094] (2) The present invention designs an ELAN-Dilated module and builds a convolutional neural network model for extracting facial fatigue features by introducing dilated convolution and cross-attention mechanism. This model has a large receptive field, strong learning ability for context features, and can retain more small target features in the feature extraction stage. It has a strong ability to extract small target features such as eyes and mouth.

[0095] (3) The present invention designs a bounding box regression loss function. In addition to the most commonly used intersection-union ratio index, it also adds the angle, distance and shape indices of the predicted box and the ground box, which can accelerate the convergence speed of the network model during training. At the same time, considering that the additional indices will also increase the amount of computation, the angle loss is designed to be calculated linearly to reduce the increase in the amount of computation.

[0096] (4) This invention designs a fatigue state discrimination criterion that integrates eye-closing duration, yawning interval time and PERCLOS to improve the classification accuracy of facial fatigue state; at the same time, since the acquisition of PERCLOS and yawning interval time indicators takes a long time, by designing a dynamic weight-adjusting eye-closing duration scoring mechanism, it can not participate in fatigue discrimination when blinking normally, but dominate fatigue discrimination when closing the eyes for a long time. While ensuring the accuracy of fatigue judgment, it also improves the real-time performance of fatigue discrimination criteria, so as to ensure that the corresponding monitoring system can take early warning and other measures in a timely manner to avoid dangerous accidents. Attached Figure Description

[0097] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0098] Figure 1 This is an overall flowchart of the face fatigue dynamic detection method of the fusion and aggregation module provided in the embodiments of the present invention;

[0099] Figure 2 This is a preprocessing flowchart provided in an embodiment of the present invention;

[0100] Figure 3 This is the network structure of the face fatigue dynamic detection network of the fusion aggregation module provided in the embodiments of the present invention;

[0101] Figure 4 This is a schematic diagram of the ELAN-Dilated module provided in an embodiment of the present invention;

[0102] Figure 5This is a schematic diagram of the CCNet module provided in an embodiment of the present invention; Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the application, and not all embodiments.

[0104] The dynamic face fatigue detection method of the fusion aggregation module of the present invention first acquires face images in real time through a camera, and then performs preprocessing to reduce the impact of images and high noise and low resolution images on the detection results; designs a bounding box loss function that fuses angle, distance and shape to improve the model convergence speed and reduce training cost; finally, designs a fatigue discrimination criterion that fuses eye closing time, yawning interval time and PERCLOS to achieve dynamic real-time early warning.

[0105] like Figure 1 As shown in the embodiment of the present invention, the method for dynamic detection of facial fatigue using a fusion aggregation module comprises the following steps:

[0106] Step S1: Use a camera device to capture facial images in real time;

[0107] Step S2: Preprocess the acquired face images to reduce the impact of poor lighting and noise on the detection results;

[0108] like Figure 2 As shown, the specific steps for preprocessing the acquired face images are as follows:

[0109] (1) Perform grayscale processing on the face image to reduce the running memory of the face fatigue detection method and shorten the speed at which the system obtains the results;

[0110] (2) Perform median filtering on the face image to remove some noise;

[0111] (3) Perform block histogram equalization on the filtered face image to improve the contrast of the face image in poor lighting conditions.

[0112] Step S3: Construct a dynamic face fatigue detection network with a fusion and aggregation module, including a feature extraction network, a feature fusion network, and a detection end. The feature extraction network is mainly composed of ELAN-Dilated modules stacked with pooling layers, which are designed to be aggregated from ordinary convolutions and dilated convolutions. A cross-interference module CCNet is added between the feature extraction network and the feature fusion network. The feature fusion network and the detection end adopt the structure of the Neck and Head parts of the YOLOv7-tiny network.

[0113] like Figure 3 As shown, the specific steps for constructing the dynamic face fatigue detection network with fusion and aggregation modules are as follows:

[0114] (1) Constructing an ELAN-Dilated convolutional layer aggregation module to extract features of the eyes and mouth.

[0115] like Figure 4 As shown, to enhance the model's ability to extract features from small targets such as eyes and mouths, a neural network module named ELAN-Dilated is designed, consisting of stacked dilated and regular convolutions. This module includes two branches; the first branch passes through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. Dilated convolutions with dilation rates of 2 and 5 are used for feature extraction. Finally, the features from each convolutional layer of the module are stacked and then processed again using a... The convolution modulates the number of channels. This module has a large receptive field and strong learning ability for multi-scale features, enabling the model to better preserve the spatial features of the image, reduce information loss, and effectively preserve the features of small targets such as eyes and mouth.

[0116] (2) A feature extraction network is built by stacking ELAN-Dilated modules.

[0117] The feature extraction network first passes through two convolutional layers to suppress some noise and enhance features such as edges; then it passes through four ELAN-Dilated modules to extract features from shallow to deep. Max pooling layers are added between the ELAN-Dilated modules to compress the features and reduce the number of parameters and computation.

[0118] (3) Building a feature fusion network

[0119] The feature fusion network adopts the structure of the Neck part of the YOLOv7-tiny network, which includes a structure combining the feature pyramid FPN and PANet. Through continuous upsampling and downsampling, it performs multi-scale feature fusion on the shallow, medium and deep features of the eyes and mouth extracted by the last three ELAN-Dilated modules in the feature extraction network.

[0120] The feature fusion network uses multi-scale feature fusion to capture the features of the eyes and mouth at different scales, thereby enhancing the features such as the edges, textures, and shapes of the eyes and mouth. This improves the ability of the face fatigue dynamic detection network in the fusion aggregation module to locate and detect the features of the eyes and mouth.

[0121] (4) Set up the detection terminal

[0122] The detection end adopts the structure of the YOLOv7-tiny network head part, which includes constructing a detection head from three ELAN-tiny modules of the feature fusion network, and each detection head uses one Feature extraction is performed using convolutional layers, and the feature maps obtained by the three detector heads are respectively... , and These are used to detect large, medium, and small target features, respectively; ELAN-tiny includes two branches, the first branch passing through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. The convolutions are used to extract features, and finally the features from each convolutional layer of the module are stacked and then used again. The number of channels is changed through convolution;

[0123] Since the positions and sizes of the detected faces are different, the positions and sizes of the eyes and mouths in the image are also different. The detection end sets up three detection heads to independently detect the eyes and mouths from feature maps of three scales, so as to improve the detection capability of the face fatigue dynamic detection network of the fusion aggregation module for eye and mouth features.

[0124] (5) Add a cross-attention network

[0125] Add a cross-attention network (CCNet) after the last ELAN-Dilated module of the feature extraction network. For example... Figure 5 As shown, the CCNet network is divided into two branches. The first branch passes through two cross-junction attention modules to extract rich and dense contextual information. The second branch passes through a step size of 1 and a size of [missing information]. The number of channels in the convolutional layer is adjusted, and then the features from the two branches are concatenated to fuse dense contextual features and local features. A single cross-attention module extracts only horizontal and vertical contextual information; however, by concatenating two cross-attention modules, the feature map effectively collects information from all pixels at each location, resulting in richer and denser contextual information.

[0126] (6) Design the bounding box regression loss function

[0127] The dynamic detection network for facial fatigue in the fusion and aggregation module predicts eye and mouth features in facial images, including four types of targets: open eyes, closed eyes, yawning mouth, and non-yawning mouth (including situations such as speaking).

[0128] In order for the face fatigue dynamic detection network of the fusion aggregation module to have a better detection capability for eye and mouth features, it needs to be trained, and a loss function is used to quantify the deviation between the prediction results and the actual results.

[0129] The loss function of the face fatigue dynamic detection network in the fusion and aggregation module includes bounding box regression loss, bounding box confidence loss, and class confidence loss. Bounding box regression loss is used to measure the degree of overlap between the predicted boxes and the ground truth boxes of the eye and mouth targets. Bounding box confidence loss is used to measure the confidence of the predicted boxes of the eye and mouth targets. Class confidence loss is used to measure the closeness between the predicted class and the ground truth class of the eye and mouth state.

[0130] The bounding box confidence loss and class confidence loss of the face fatigue dynamic detection network in the fusion aggregation module are calculated using the binary cross-entropy function BCE, while the bounding box regression loss is designed to fuse the direction (or angle), center distance, shape and cross-union ratio of the predicted box and the real box, so that the predicted boxes of the eyes and mouth are as close as possible to the real boxes and to speed up the convergence speed of the network training.

[0131] The formula for the bounding box regression loss of the face fatigue dynamic detection network in the fusion aggregation module is as follows:

[0132] ,

[0133] ,

[0134] ,

[0135] Wherein, IoU represents the intersection-union loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. The distance loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth is represented. This represents the shape loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This represents the orientation (or angle) loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This indicates the degree to which the shape loss of the predicted boxes for the eyes and mouth is penalized against the total loss in relation to the ground truth boxes. , They represent and These represent the width and height of the prediction box, respectively. and These represent the width and height of the actual bounding boxes for the eyes and mouth, respectively. and These represent the x and y coordinates of the center points of the prediction boxes for the eyes and mouth, respectively. and Let x and y represent the x and y coordinates of the center points of the true bounding boxes for the eyes and mouth, respectively. and Let represent the width and height of the minimum closure rectangle between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth, respectively.

[0136] Since the eyes and mouth are small target features, the face fatigue dynamic detection network of the fusion aggregation module still loses a small amount of information when extracting eye and mouth features, and is more susceptible to interference from information around the eyes and mouth, such as misidentifying the top and bottom edges of black glasses as closed eyes. By using a loss function that fuses the direction (or angle), center distance, shape, and intersection-union ratio of the predicted box and the ground truth box, the face fatigue dynamic detection network of the fusion aggregation module can still be penalized when the predicted box is close to the ground truth box, reducing the interference of information around the ground truth box on the prediction result, making the predicted box closer to the ground truth box, and improving the accuracy of the result.

[0137] Specifically, regarding the angle loss, the angle loss reaches its maximum when the horizontal angle between the predicted box and the ground truth box is approximately 30° to 60°; the angle loss is minimized when the horizontal angle between the predicted box and the ground truth box is in the range of 0° to 30° (the predicted box is close to the ground truth box). (axis) and the 60° to 90° range (where the predicted box is close to the true box) When the axis is (i.e., the predicted bounding box is closer to the ground truth bounding box), shaft or The smaller the angle loss, the faster the predicted bounding box can quickly approximate the ground truth bounding box. shaft or The axis reduces the degrees of freedom of the prediction box, thus accelerating the convergence of the face fatigue dynamic detection network model in the fusion aggregation module.

[0138] In summary, the bounding box loss function of the face fatigue dynamic detection network in the fusion aggregation module integrates the direction (or angle), center distance, shape, and intersection-union ratio of the predicted box and the ground truth box. This not only accelerates the convergence speed of the network (i.e., reduces the number of convergence iterations) but also improves inference accuracy. However, the calculation of multiple indicators will increase the computational cost of the loss function in a single iteration. The design of the direction (or angle) loss uses linear calculation, which can reduce the increase in computational cost and save time.

[0139] Step S4: Use the face fatigue dynamic detection network of the fusion aggregation module to extract the eye and mouth features of each frame of face image, including open eyes, closed eyes, yawning, and no yawning;

[0140] Step S5: Based on the fixed frame rate of the face images captured by the camera device, extract the eye and mouth features of each face image according to the face fatigue dynamic detection network of the fusion aggregation module, and calculate the duration of closed eyes, the percentage of closed eyes PERCLOS value, and the yawning time interval.

[0141] Specific calculation methods include:

[0142] (1) Duration of closing eyes

[0143] This invention defines eye closure as when the area of ​​the eyelid covering more than 80% of the pupil is considered closed. The duration of eye closure is calculated based on the number of consecutive frames returned by the detection model in the closed-eye state. The formula is as follows:

[0144] ,

[0145] Because prolonged eye closure poses significant safety risks, the duration of eye closure is used as one of the indicators for fatigue assessment. When the eye closure duration is around 0.2 seconds (normal blinking), it is not included in the fatigue state analysis. As the eye closure duration increases, its weight in the fatigue analysis is gradually increased. When the eye closure duration reaches around 2 seconds, it completely dominates the fatigue state analysis. It can output a moderate fatigue state when the eye closure duration reaches around 1 second and a severe fatigue state when it reaches around 2 seconds, issuing a strong warning and improving the real-time performance of fatigue assessment.

[0146] (2) PERCLOS value

[0147] The PERCLOS value specifically refers to the time the eyes are closed per unit of time. This invention uses... The PERCLOS value is expressed by the following formula:

[0148] ,

[0149] (3) Yawning interval

[0150] The feature distinguishing yawning from speaking is a wide-open mouth that lasts for 3-5 seconds. A single yawn is defined as the number of consecutive frames returned by the facial fatigue dynamic detection network of the fusion aggregation module that reach a certain threshold. The interval between two yawns is recorded as... .

[0151] Existing fatigue detection mechanisms primarily use the frequency of yawning as a criterion. However, yawning has high uncertainty; in a state of mild fatigue, there may be no yawning for up to a minute, resulting in a long fatigue detection time. Using the interval between yawns as one of the fatigue detection indicators can better track the yawning state.

[0152] Step S6: Perform fatigue state analysis by fusing the duration of eye closure, the percentage of eye closure duration in PERCLOS value, and the yawning interval, and output the fatigue state analysis results;

[0153] The specific steps include:

[0154] Step S61: Normalize the duration of eye closure, PERCLOS value, and yawn interval to obtain the eye closure duration score. PERCLOS rating And the yawning interval score The specific formulas include:

[0155] When the eyes are closed for a relatively long period, the duration of eye closure should be the dominant characteristic in fatigue state analysis; during normal blinking, the duration of eye closure should not be included in the fatigue state analysis. The normalization formula for the duration of eye closure is as follows:

[0156] ,

[0157] Among them, parameters and The eye-closing duration score was divided into three stages: normal blinking, fatigue state, and prolonged eye-closing; parameters and The settings must meet the following requirements:

[0158] a. The threshold for distinguishing between the normal blinking phase and the phase of increased fatigue should be set to the maximum duration of a normal blink, typically about 0.4 seconds;

[0159] b. The threshold for distinguishing between the fatigue state stage and the prolonged closed-eye stage should be set as the minimum time from when a person closes their eyes to when a dangerous situation occurs. The threshold setting will vary depending on the environment in which the fatigue test subject is located. For example, when driving a vehicle, the threshold should be the minimum time from when a person closes their eyes to when a dangerous situation occurs (including when they deviate from their lane or are less than the safe distance from the vehicle in front).

[0160] c. During the normal blinking phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 0; during the fatigue phase, as the duration of eye closure increases, the score for duration of eye closure increases almost linearly; during the prolonged eye closure phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 1.

[0161] (2) PERCLOS score

[0162] The fatigue state is determined by whether the PERCLOS value has a stable fluctuation range. The PERCLOS value is normalized using the minimum-maximum normalization method, as shown in the following formula:

[0163] ,

[0164] in, The PERCLOS value detected in real time. and These represent the maximum and minimum values ​​of PERCLOS during the detection process;

[0165] Statistical experiments have divided PERCLOS scores into three stages: normal blinking, moderate fatigue, and severe fatigue. The threshold for the normal blinking and moderate fatigue stages is the maximum value of PERCLOS when blinking normally, usually around 0.15. The threshold for the moderate fatigue and severe fatigue stages is the minimum value of PERCLOS when it seriously affects a person's current behavior or poses a significant safety hazard.

[0166] (3) Yawning interval score

[0167] Based on the core idea of ​​minimax normalization, an overflow boundary is set, and the output value is reversed. The formula is as follows:

[0168] ,

[0169] in, The parameter represents the real-time detected yawn interval. and The yawning interval was divided into three stages: normal fatigue, moderate fatigue, and severe fatigue; parameters and The settings must meet the following requirements:

[0170] a. Parameters It is the time threshold for dividing the stage of normal fatigue and the stage of severe fatigue. Its value is set to be slightly larger than the interval of frequent yawning, and its value is usually within 5 to 10 seconds.

[0171] b. Parameters It is the time threshold for dividing the normal state stage and the ordinary fatigue stage, and its value is set within 1 to 2 minutes;

[0172] Step S62: Fusion of the normalized eye-closing duration, PERCLOS value, and yawn interval to obtain a fatigue state score. The formula is as follows:

[0173] ,

[0174] Fatigue status score Always Within a certain range (if the score exceeds 1, it is still represented as 1), the closer the fatigue status score is to 0, the milder the fatigue status; the closer it is to 1, the more severe the fatigue status. During normal blinking, the eye-closing duration score, within a range close to 0, hardly contributes to the evaluation of fatigue status; when the eye-closing duration reaches a large value, it dominates the evaluation of fatigue status. The PERCLOS value and yawning interval are respectively... and Perform weighted fusion;

[0175] Step S63: Score the fatigue state. The fatigue state is classified into different levels, and corresponding measures are set for each level.

[0176] The closer the fatigue status score is to 0, the lighter the fatigue status is; the closer it is to 1, the heavier the fatigue status is. The fatigue status score is divided into three fatigue levels and one non-fatigue level. The fatigue level from heavy to light is recorded as the first fatigue state, the second fatigue state, the third fatigue state, and the non-fatigue (normal) state.

[0177] The intervals for the fatigue state score should be determined based on the intervals for the eye-closing duration score, the PERCLOS score, and the yawning interval score (hereinafter referred to as "the three score values").

[0178] The first fatigue state is a state with a high safety hazard. Under this state, at least one of the following conditions must be met:

[0179] a) The score for duration of eye closure is in the prolonged eye closure stage;

[0180] b) The PERCLOS score indicates severe fatigue.

[0181] The second state of fatigue is moderate fatigue, during which the brain's judgment and reaction are slower, posing certain safety risks. In this state, at least one of the following situations may occur:

[0182] a) The score for duration of eye closure indicates a state of fatigue;

[0183] b) The PERCLOS score indicates a moderate fatigue level;

[0184] c) The interval between yawns is during the normal or severe fatigue stage;

[0185] The third fatigue state is a mild fatigue state, in which at least one of the three rating values ​​is not in the normal state stage;

[0186] The non-fatigue (normal) state is the normal blinking state. In this state, the eye closure duration score and PERCLOS are both in the normal blinking state, and the yawning interval score is in the normal state.

[0187] The application scenarios of the dynamic face fatigue detection method with fusion and aggregation module disclosed in this invention include driving vehicles, driving special operation equipment, and operating special machinery;

[0188] Under driving conditions, measures that can be taken in the first state of fatigue include strong warning signals, such as a rapid and loud warning tone or a clear voice prompt; measures that can be taken in the second state of fatigue include a shorter warning tone; measures that can be taken in the third state of fatigue include automatically turning on the in-vehicle music system and playing cheerful music; no measures are taken in the non-fatigue (normal) state.

[0189] Example 1

[0190] The specific steps are as follows:

[0191] Step 1: Construct a dataset for detecting facial fatigue.

[0192] The dataset uses camera devices to collect facial images from different angles as samples. The dataset must contain a sufficient number of samples with eyes closed, eyes open, yawning, and not yawning. It must also include speaking samples to distinguish yawning, as well as facial images under different lighting conditions.

[0193] Step 2: Design a preprocessing module that can perform grayscale processing, improve image contrast, and reduce noise in the image.

[0194] First, the preprocessing module can perform grayscale processing on the face image, reducing the memory usage of the fatigue detection method. Then, the preprocessing module can perform median filtering on the face image to remove some image noise. Finally, the preprocessing module can perform block histogram equalization on the filtered face image to improve the contrast of the face image in poor lighting conditions.

[0195] Step 3: Construct a dynamic face fatigue detection network with fusion and aggregation modules.

[0196] (1) Constructing an ELAN-Dilated convolutional layer aggregation module to extract features of the eyes and mouth.

[0197] ELAN-Dilated is composed of stacked and aggregated ordinary convolutions and dilated convolutions, which has a large receptive field and can learn richer contextual information and spatial features around the eyes and mouth;

[0198] (2) A feature extraction network is built by stacking ELAN-Dilated modules.

[0199] The feature extraction network consists of two sequentially connected convolutional layers (CBL), four ELAN-Dilated modules, and a cross-connected attention network (CCNet), with a max pooling layer (MP) between the ELAN-Dilated modules.

[0200] Two convolutional layers (CBL) perform coarse feature extraction on face images, including enhancing edge features and reducing some noise.

[0201] Four ELAN-Dilated modules extract features of the eyes and mouth from shallow to deep layers;

[0202] Max pooling layers (MP) are used to compress the feature parameters of the eyes and mouth, reduce dimensionality, and decrease computational cost.

[0203] The Cross-Attention Module (CCNet) is used to extract rich contextual information about the eyes and mouth and their surroundings, extracting eye and mouth features more efficiently with lower GPU memory usage.

[0204] (2) Building a feature fusion network

[0205] The feature fusion network adopts the structure of the Neck part of YOLOv7-tiny, which includes a structure combining feature pyramid FPN and PANet; multi-scale feature fusion is performed on the shallow, medium and deep three-layer feature maps of the eyes and mouth extracted by the two ELAN-Dilated modules and CCNet module in the middle of the feature extraction network, respectively.

[0206] (3) Set up the detection terminal

[0207] The detection end adopts the structure of the YOLOv7-tiny network head part, including three detection heads constructed from three ELAN-tiny modules of the feature fusion network, which are used to detect large target features, medium target features and small target features respectively; the ELAN-tiny module is composed of stacked and aggregated convolutional layers CBL;

[0208] (4) Add a cross-attention network

[0209] A cross-attention network (CCNet) is added after the last ELAN-Dilated module of the feature extraction network. The CCNet network includes two cross-attention modules (CCA), a convolutional layer (CBL), and a concat operation. The concatenation of two consecutive CCA modules enables each pixel on the feature map of the eyes and mouth to establish a relationship with any pixel, thus extracting richer and denser contextual information of the eyes and mouth.

[0210] (5) Design the bounding box regression loss function

[0211] The loss function of the face fatigue dynamic detection network with fusion and aggregation modules includes bounding box regression loss, bounding box confidence loss, and class confidence loss, which are used to measure the degree of overlap between the predicted boxes and the ground boxes for the eyes and mouth, the confidence of the predicted boxes, and the closeness between the predicted class and the ground class, respectively.

[0212] The bounding box confidence loss and class confidence loss of the face fatigue dynamic detection network in the fusion aggregation module are calculated using the binary cross-entropy function BCE, while the bounding box regression loss is designed to fuse the direction (or angle), center distance, shape and cross-union ratio of the predicted box and the real box, so that the predicted boxes of the eyes and mouth are as close as possible to the real boxes, and to speed up the convergence speed of the network training.

[0213] The formula for the bounding box regression loss of the face fatigue dynamic detection network in the fusion aggregation module is as follows:

[0214] ,

[0215] ,

[0216] ,

[0217] The bounding box regression loss of the face fatigue dynamic detection network in the fusion aggregation module can still be penalized when the predicted box is close to the real box, reducing the interference of information around the real box on the prediction result, so that the predicted box can be closer to the real box and improve the accuracy of the result.

[0218] In particular, for angle (or orientation) loss, the predicted bounding box can quickly approximate the ground truth bounding box. shaft or The axis reduces the degrees of freedom of the prediction box, thus accelerating the convergence of the face fatigue dynamic detection network model in the fusion aggregation module.

[0219] Step 4: Use the preprocessing module to process the constructed face fatigue state detection dataset, and use the preprocessed dataset to train the face fatigue dynamic detection network of the fusion aggregation module.

[0220] Step 5: Use a camera device to capture facial images in real time.

[0221] Step 6: Preprocess the collected face images to reduce the difficulty of facial feature extraction.

[0222] First, grayscale processing is performed on the face image to reduce the memory usage of the fatigue state detection method and shorten the time it takes for the system to obtain results. Then, median filtering is applied to the driver's face image to remove some image noise. Finally, block histogram equalization is performed on the filtered face image to improve the contrast of the face image in poor lighting conditions.

[0223] Step 7: Use the trained fusion and aggregation module's facial fatigue dynamic detection network model to extract the state information of the eyes and mouth of the person in the preprocessed facial image, including whether the eyes are open, closed, yawning, or not yawning.

[0224] Step 8: Analyze the body's fatigue state by integrating eye and mouth status information.

[0225] (1) Calculate the duration of eye closure

[0226] When the eyelid covers more than 80% of the pupil, the eye is considered closed. The duration of eye closure is calculated based on the number of consecutive frames returned by the detection model in the closed-eye state. Since a typical video is shot at 30 frames per second, the formula for the duration of eye closure is:

[0227] ,

[0228] (2) Calculate PERCLOS

[0229] Set the detection time to 30 seconds and calculate PERCLOS using the following formula:

[0230] ,

[0231] (3) Calculate the yawn interval.

[0232] The feature distinguishing yawning from speaking is the large opening of the mouth for 3-5 seconds. A yawn is defined as the number of consecutive frames returned by the facial fatigue dynamic detection network of the fusion aggregation module that reach a certain threshold. The interval between two yawns is... .

[0233] (4) Normalization of eye-closing duration

[0234] A person's normal blinking duration is generally around 0.2 seconds. In industrial production, driving, and other situations, blinking for more than 2 seconds can pose significant safety hazards. Therefore, to prevent eye closure duration of around 0.2 seconds from being included in fatigue state analysis (score approaching 0), and to dominate fatigue state analysis at around 2 seconds (score approaching 1), adjustment parameters were set. , The formula for normalizing the duration of eye closure is:

[0235] ,

[0236] (5) Minimum-maximum normalization of PERCLOS

[0237] Experiments show that the PERCLOS value generally ranges from 10% to 70% in a normal or fatigued state. Therefore, the formula for the minimum-maximum normalization of PERCLOS is:

[0238] ,

[0239] (6) Normalization of minimum and maximum yawn intervals

[0240] Assuming a yawn interval of less than 10 seconds indicates severe fatigue, 10 to 60 seconds indicates moderate fatigue, and more than 60 seconds indicates mild fatigue, the formula for minimum-maximum normalization of yawn intervals is as follows:

[0241] ,

[0242] (7) The fatigue score is obtained by weighted fusion of eye-closing time, PERCLOS and yawning interval.

[0243] The score for eye-closing duration needs to be dynamically adjusted based on the magnitude of fatigue, so its weight is always 1. PERCLOS and yawning interval are adjusted using weighting coefficients. This embodiment sets... The fatigue scoring formula is as follows:

[0244] ,

[0245] (8) Establish fatigue classification and corresponding measures

[0246] The closer the fatigue status score is to 0, the lighter the fatigue status is; the closer it is to 1, the heavier the fatigue status is. The fatigue status score is divided into three fatigue levels and one non-fatigue level. The fatigue level from heavy to light is recorded as the first fatigue state, the second fatigue state, the third fatigue state, and the non-fatigue (normal) state.

[0247] A fatigue score between 0 and 0.1 indicates a normal fatigue state; a score between 0.1 and 0.4 indicates a third level of fatigue (mild fatigue); a score between 0.4 and 0.8 indicates a second level of fatigue (moderate fatigue); and a score between 0.8 and 1 indicates a first level of fatigue (severe fatigue).

[0248] The fatigue grading can be implemented using different measures in different application scenarios;

[0249] In driving scenarios, measures taken for the first fatigue state include relatively strong warning signals, such as a relatively rapid and loud warning tone or a clear voice prompt; measures taken for the second fatigue state include relatively short prompt tones; measures taken for the third fatigue state include automatically turning on the in-vehicle music system and playing relatively cheerful music; no measures are taken in the normal state.

Claims

1. A method for dynamic detection of facial fatigue using a fusion aggregation module, characterized in that... The steps are as follows: Step S1: Use a camera device to capture facial images in real time at a fixed frame rate; Step S2: Preprocess the acquired face images to reduce the impact of poor lighting and noise on the detection results; Step S3: Construct a dynamic face fatigue detection network, including a feature extraction network, a feature fusion network, and a detection end connected in sequence; the feature extraction network includes convolutional layers (CBL), max pooling layers (MP), and an ELAN-Dilated module; a cross-junction attention module (CCNet) is also included between the feature extraction network and the feature fusion network; the feature fusion network and the detection end include the structure of the Neck and Head parts of the YOLOv7-tiny network; among them, the ELAN-Dilated module includes a structure of stacked and aggregated ordinary convolutional layers (CBL) and dilated convolutional layers; The feature extraction network includes two sequentially connected convolutional layers (CBL), four ELAN-Dilated modules, and a cross-attention network (CCNet). Max pooling layers (MP) are placed between the ELAN-Dilated modules. Each ELAN-Dilated module has two branches, the first branch passing through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. Dilated convolutions with dilation rates of 2 and 5 are used for feature extraction. Finally, the features from each convolutional layer of the module are stacked and then processed again using a... The number of channels is changed through convolution; in: Two convolutional layers (CBL) perform coarse feature extraction on face images, including enhancing edge features and suppressing some noise. Four ELAN-Dilated modules extract features of the eyes and mouth from shallow to deep layers; Max pooling layers (MP) are used to compress the feature parameters of the eyes and mouth, reduce dimensionality, and decrease computational cost. The Cross-Attention Module (CCNet) is used to extract eye and mouth features more efficiently by acquiring rich contextual information about the eyes and mouth and their surroundings. Step S4: Use a fully trained face fatigue dynamic detection network to extract eye and mouth features from each frame of the preprocessed face image, including open eyes, closed eyes, yawning, and no yawning; Step S5: Based on the fixed frame rate of the face images captured by the camera device, extract the eye and mouth features of each face image according to the face fatigue dynamic detection network, and calculate the duration of closed eyes, the percentage of closed eyes, the PERCLOS value, and the yawning time interval. Step S6: Perform fatigue state analysis by integrating the duration of eye closure, the percentage of eye closure duration in PERCLOS value, and the yawning interval, and output the fatigue state analysis results.

2. The method for dynamic detection of facial fatigue using a fusion aggregation module according to claim 1, characterized in that... The specific steps for preprocessing the acquired face images are as follows: Step S21: Perform grayscale processing on the face image to reduce the running memory of the face fatigue detection method and shorten the time it takes for the system to obtain results; Step S22: Perform median filtering on the face image to remove some noise; Step S23: Perform block histogram equalization on the filtered face image to improve the contrast of the face image in poor lighting conditions.

3. The method for dynamic detection of facial fatigue using a fusion and aggregation module according to claim 1, characterized in that, The specific steps for constructing a dynamic facial fatigue detection network are as follows: Step S31: Construct the convolutional layer aggregation module ELAN-Dilated to extract features of the eyes and mouth; The ELAN-Dilated module consists of stacked dilated convolutions and regular convolutions, and includes two branches. The first branch passes through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. Dilated convolutions with dilation rates of 2 and 5 are used for feature extraction. Finally, the features from each convolutional layer of the module are stacked and then processed again using a... The number of channels is changed during convolution; Step S32: Construct a feature extraction network by stacking ELAN-Dilated modules; The feature extraction network first passes through two convolutional layers to suppress some noise and enhance the edge features of the eyes and mouth; then it passes through four ELAN-Dilated modules to extract the features of the eyes and mouth from shallow to deep; and max pooling layers are added between the ELAN-Dilated modules to compress the features and reduce the number of parameters and computation. Step S33: Construct a feature fusion network; The feature fusion network adopts the structure of the Neck part of the YOLOv7-tiny network. The structure includes a combination of Feature Pyramid Network (FPN) and PANet. Through continuous upsampling and downsampling, multi-scale feature fusion is performed on the shallow, medium and deep features of the eyes and mouth extracted by the last three ELAN-Dilated modules in the feature extraction network. Step S34: Set up the detection terminal; The detection end adopts the structure of the YOLOv7-tiny network head part, which includes constructing a detection head from three ELAN-tiny modules of the feature fusion network, and each detection head uses one Feature extraction is performed using convolutional layers, and the feature maps obtained by the three detector heads are respectively... , and These are used to detect large, medium, and small target features, respectively; ELAN-tiny includes two branches, the first branch passing through a... The convolution changes the number of channels; the second branch first goes through a... The convolution is used to change the number of channels, and then passed through two convolutions of size 1. The convolutions are used to extract features, and finally the features from each convolutional layer of the module are stacked and then used again. The number of channels is changed through convolution; Since the positions and sizes of the detected faces are different, the positions and sizes of the eyes and mouths in the image are also different. The detection end sets up three detection heads to independently detect the eyes and mouths from feature maps of three scales, so as to improve the detection capability of the face fatigue dynamic detection network of the fusion aggregation module for eye and mouth features. Step S35: Add a cross-attention network; A cross-cutting attention network (CCNet) is added after the last ELAN-Dilated module of the feature extraction network. CCNet consists of two branches. The first branch passes through two cross-cutting attention modules (Criss-Cross Attention Modules) to extract rich and dense contextual information. The second branch passes through a step size of 1 and a size of [missing information]. The number of channels in the convolutional layer is adjusted, and then the features of the two branches are concatenated to fuse dense contextual features and local features. In this case, a single cross-attention module only extracts contextual information in the horizontal and vertical directions. By concatenating two cross-attention modules, the feature map can actually collect information from all pixels at each position, thus extracting richer and denser contextual information.

4. The method for dynamic detection of facial fatigue using a fusion and aggregation module according to claim 3, characterized in that, During the training of the facial fatigue dynamic detection network, a loss function is used to quantify the deviation between the prediction results and the actual results, so as to accurately predict the eye and mouth features in the face image, including four targets: open eyes, closed eyes, yawning mouth, and non-yawning mouth; the loss function design process is as follows. The loss function includes bounding box regression loss, bounding box confidence loss, and class confidence loss. Bounding box regression loss measures the degree of overlap between the predicted bounding boxes and the ground truth bounding boxes of the eye and mouth targets. Bounding box confidence loss measures the confidence level of the predicted bounding boxes of the eye and mouth targets. Class confidence loss measures the degree of closeness between the predicted class and the ground truth class of the eye and mouth state. The bounding box confidence loss and class confidence loss are calculated using the binary cross-entropy function (BCE). The bounding box regression loss is designed to integrate the direction or angle, center distance, shape, and cross-union ratio of the predicted box and the ground truth box, so that the predicted boxes of the eyes and mouth of people in the image are as close as possible to the ground truth boxes, thereby accelerating the convergence speed during network training. The formula for the bounding box regression loss of the face fatigue dynamic detection network is as follows: , ， ， Wherein, IoU represents the intersection-union loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. The distance loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth is represented. This represents the shape loss between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This represents the loss in orientation or angle between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth. This indicates the degree to which the shape loss of the predicted boxes for the eyes and mouth is penalized against the total loss in relation to the ground truth boxes. , These represent the weights for adjusting the distance loss and shape loss, respectively. and These represent the width and height of the prediction box, respectively. and These represent the width and height of the actual bounding boxes for the eyes and mouth, respectively. and These represent the x and y coordinates of the center points of the prediction boxes for the eyes and mouth, respectively. and Let x and y represent the x and y coordinates of the center points of the true bounding boxes for the eyes and mouth, respectively. and These represent the width and height of the minimum closure rectangle between the predicted bounding boxes and the ground truth bounding boxes for the eyes and mouth, respectively.

5. The method for dynamic detection of facial fatigue using a fusion and aggregation module according to claim 4, characterized in that, The calculation methods for eye-closing duration, the percentage of eye-closing duration in PERCLOS, and the yawning interval are as follows: (1) Duration of closing eyes Define eye closure as when the area of ​​the eyelid covering more than 80% of the pupil is considered closed. The duration of eye closure is calculated based on the number of frames continuously returned by the facial fatigue dynamic detection network to the closed-eye state. The formula is as follows: ， Because prolonged eye closure poses significant safety risks, the duration of eye closure is used as one of the indicators for fatigue assessment. When the eye closure duration is around 0.2 seconds, it is not included in the fatigue state analysis. As the eye closure duration increases, its weight in the fatigue analysis is gradually increased. When the eye closure duration reaches around 2 seconds, it completely dominates the fatigue state analysis. It can output a moderate fatigue state when the eye closure duration reaches around 1 second and a severe fatigue state when it reaches around 2 seconds, issuing a strong warning and improving the real-time performance of fatigue assessment. (2) PERCLOS value The PERCLOS value specifically refers to the time the eyes are closed per unit of time. The criterion is that the eyelids cover more than 80% of the pupil as the eyes are considered closed. The following formula is used... The PERCLOS value is expressed by the following formula: ， (3) Yawning interval The feature distinguishing yawning from speaking is a wide-open mouth that lasts for 3-5 seconds. A yawn is defined as the number of consecutive frames returned by the facial fatigue dynamic detection network when the number of frames reaches a certain threshold. The interval between two yawns is recorded as... .

6. The method for dynamic detection of facial fatigue using a fusion aggregation module according to claim 5, characterized in that, The specific steps for analyzing fatigue state by integrating the duration of eye closure, the percentage of eye closure duration in PERCLOS values, and the yawning interval, and outputting the fatigue state analysis results, include: Step S61: Normalize the duration of eye closure, PERCLOS value, and yawn interval to obtain the eye closure duration score. PERCLOS rating And the yawning interval score The specific formula is as follows: (1) Eyes closed duration score When the eyes are closed for a relatively long period, the duration of eye closure should be the dominant characteristic in fatigue state analysis; during normal blinking, the duration of eye closure should not be included in the fatigue state analysis. The normalization formula for the duration of eye closure is as follows: ， Among them, parameters and The eye-closing duration score was divided into three stages: normal blinking, fatigue state, and prolonged eye-closing; parameters and The settings must meet the following requirements: a. The threshold for distinguishing between the normal blinking phase and the phase of increased fatigue should be set to the maximum duration of a normal blink, approximately 0.4 seconds; b. The threshold for distinguishing between the fatigue state stage and the prolonged closed-eye stage should be set as the minimum time from when a person closes their eyes to when a dangerous situation occurs; this threshold setting will vary depending on the environment in which the fatigue test subject is located; for example, when driving a vehicle, the minimum time from when the person closes their eyes to when a dangerous situation occurs should be used. c. During the normal blinking phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 0; during the fatigue phase, as the duration of eye closure increases, the score for duration of eye closure increases almost linearly; during the prolonged eye closure phase, as the duration of eye closure increases, the score for duration of eye closure increases slowly and approaches 1. (2) PERCLOS score Fatigue state is determined by whether the PERCLOS score has a stable fluctuation range. The PERCLOS score is normalized using the minimum-maximum normalization method, as shown in the following formula: ， in, The PERCLOS value detected in real time. and These represent the maximum and minimum values ​​of PERCLOS during the detection process; Statistical experiments were conducted to divide the PERCLOS score into three stages: normal blinking, moderate fatigue, and severe fatigue. The threshold for dividing the normal blinking and moderate fatigue stages was the maximum value of PERCLOS when blinking normally, which was set to 0.

15. The threshold for dividing the moderate fatigue and severe fatigue stages was the minimum value of PERCLOS when it seriously affected a person's current behavior or posed a significant safety hazard. (3) Yawning interval score Based on the core idea of ​​minimax normalization, an overflow boundary is set, and the output value is reversed. The formula is as follows: ， in, The parameter represents the real-time detected yawn interval. and The yawning interval was divided into three stages: normal fatigue, moderate fatigue, and severe fatigue; parameters and The settings must meet the following requirements: a. Parameters It is the time threshold for dividing the ordinary fatigue state stage and the severe fatigue stage. Its value is set to be slightly larger than the interval of frequent yawning, and the value is set within 5 to 10 seconds. b. Parameters It is the time threshold for dividing the normal state stage and the ordinary fatigue stage, and the value is set within 1 to 2 minutes; Step S62: Fusion of the normalized eye-closing duration, PERCLOS value, and yawn interval to obtain a fatigue state score. The formula is as follows: ， Fatigue status score Always Within the specified range, values ​​exceeding 1 are still represented as 1. A fatigue score closer to 0 indicates milder fatigue, while a score closer to 1 indicates more severe fatigue. During normal blinking, eye-closing duration scores close to 0 have almost no impact on fatigue assessment; however, scores for longer eye-closing durations dominate fatigue evaluation. PERCLOS scores and yawn intervals are calculated using... and Perform weighted fusion; Step S63: Score the fatigue state. The fatigue state is classified into different levels, and corresponding measures are set for each level. The closer the fatigue status score is to 0, the lighter the fatigue status is; the closer it is to 1, the heavier the fatigue status is. The fatigue status score is divided into three fatigue levels and one non-fatigue level. The fatigue level from heavy to light is recorded as the first fatigue state, the second fatigue state, the third fatigue state, and the non-fatigue state. The intervals for the fatigue status score should be determined based on the intervals for the three score values: eye-closing duration score, PERCLOS score, and yawning interval score. The first fatigue state is a state with a high safety hazard. Under this state, at least one of the following conditions must be met: a) The score for duration of eye closure is in the prolonged eye closure stage; b) The PERCLOS score indicates severe fatigue. The second state of fatigue is moderate fatigue, during which the brain's judgment and reaction are slower, posing certain safety risks. In this state, at least one of the following situations may occur: a) The score for duration of eye closure indicates a state of fatigue; b) The PERCLOS score indicates moderate fatigue. c) The interval between yawns is during the normal or severe fatigue stage; The third fatigue state is a mild fatigue state, in which at least one of the three rating values ​​is not in the normal state stage; The non-fatigue state is the normal blinking state. In this state, the eye closure duration score and PERCLOS are both in the normal blinking state, and the yawning interval score is in the normal state. When driving, the measures taken for the first fatigue state include strong warning signals, including a rapid and loud warning tone or a clear voice prompt; the measures taken for the second fatigue state include a short warning tone; the measures taken for the third fatigue state include automatically turning on the in-vehicle music system to play cheerful music; no measures are taken when not fatigued.

Images