Fatigue driving detection method and system in dynamic driving process

Abstract

The invention provides a fatigue driving detection method in a dynamic driving process, and relates to the technical field of computer vision recognition. Firstly, a to-be-detected driver image is acquired; accurate driver face detection is realized through a trained improved MTCNN face detection network, the improved MTCNN face detection network is embedded in third layers of three sub-networks P-Net, R-Net and O-Net of an original MTCNN face detection network to improve attention CA layers of face recognition precision, and batch normalization BN modules are introduced into the three sub-networks P-Net, R-Net and O-Net. Inputting the detected face of the driver into the trained PFLD face key point model, and extracting key point coordinates of the face of the driver; the eye aspect ratio and the mouth aspect ratio are respectively calculated based on the key point coordinates, and the fatigue state of the driver in the dynamic driving process is judged by using various facial features of the driver, so that the fatigue detection accuracy of the driver is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision recognition, and more specifically, relates to a method and system for detecting fatigue driving during dynamic driving. Background Art

[0002] In recent years, with the rapid development of my country's national economy, the domestic transportation industry has also made significant progress, with the per capita car ownership in my country gradually increasing. However, at the same time, the incidence of traffic accidents in my country has been increasing year by year, not only seriously threatening people's lives and safety, but also causing huge losses to the national and social economy. The causes of traffic accidents are complex and diverse, among which driver fatigue is one of the most important factors. Therefore, in order to effectively prevent the occurrence of fatigue driving, the development of a method that can quickly and accurately detect and identify driver fatigue has significant application value and practical significance.

[0003] With the rapid development of deep learning technology, fatigue detection methods based on facial features have been widely used and have achieved remarkable research results. When drivers experience fatigue, their facial features will show some obvious visual changes, mainly in the eyes, head, and face. A camera installed in front of the driver captures images and extracts facial visual features that reflect the driver's fatigue level. The driver's eye state, mouth state, and head movement are then analyzed in depth to accurately determine whether the driver is in a state of fatigue. Compared with traditional fatigue detection methods, fatigue detection methods based on computer vision have many significant advantages, such as non-contact and no interference to the driver. At the same time, they have higher detection accuracy and can more reliably detect whether the driver is driving in a fatigued state.

[0004] However, existing methods for detecting driver fatigue based on facial features still have some issues. Driving is a dynamic process, and in a dynamic driving environment, issues such as lighting changes and vehicle jolting can interfere with the accuracy of face detection models. Furthermore, existing methods for detecting driver fatigue often rely on a single feature to determine driver status. When this single feature is affected by individual driver differences or the detection environment, the accuracy of fatigue detection decreases. Summary of the Invention

[0005] In order to solve the problem of low accuracy of driver fatigue detection in current fatigue driving detection methods, the present invention proposes a fatigue driving detection method during dynamic driving, which uses multiple driver facial features to consider the driver's fatigue characteristics during dynamic driving and improve the accuracy of driver fatigue detection.

[0006] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0007] The present invention provides a method for detecting fatigue driving during dynamic driving, comprising the following steps:

[0008] Acquire an image of a driver to be detected;

[0009] Inputting the driver image to be detected into a trained improved MTCNN face detection network to perform driver face detection; the improved MTCNN face detection network embeds an attention CA layer for improving face recognition accuracy in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, and introduces a batch normalization (BN) module in the three sub-networks P-Net, R-Net, and O-Net;

[0010] The detected driver's face is input into the trained PFLD face key point model to detect the face key points and extract the key point coordinates of the driver's face;

[0011] The key point coordinates of the driver's face are used to calculate the driver's eye aspect ratio and the driver's mouth aspect ratio respectively, and whether the driver is in a fatigue state is determined based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

[0012] Furthermore, the improved MTCNN face detection network includes: a CA-P-Net subnetwork, a CA-R-Net subnetwork, and a CA-O-Net subnetwork;

[0013] The CA-P-Net subnetwork includes: a first convolutional normalization layer for extracting a first primary feature of the driver image, a second convolutional normalization layer for extracting a second primary feature of the driver image, a first attention CA layer for extracting a spatial position sensitive feature of the driver image, a third convolutional normalization layer for extracting a third primary feature of the driver image, and a first fully connected layer for fusing the extracted features, which are connected in sequence; and further includes: a first face classifier layer for determining whether the driver's face exists, a first bounding box regressor layer for determining the range of the driver's face, and a first face feature point locator layer for locating key feature points of the driver's face; the output end of the first fully connected layer is connected to the input ends of the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer, respectively; the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer are input into the input end of the CA-P-Net subnetwork;

[0014] The CA-R-Net subnetwork includes: a fourth convolutional normalization layer for extracting the first intermediate feature of the driver's face, a fifth convolutional normalization layer for extracting the second intermediate feature of the driver's face, a second attention CA layer for enhancing attention to the key area of the driver's face, a sixth convolutional normalization layer for extracting the third intermediate feature of the driver's face, and a second fully connected layer for fusing the extracted features, which are connected in sequence; and also includes: a second face classifier layer for screening the absence of the driver's face in the candidate frame, a second bounding box regressor layer for adjusting the position of the driver's face candidate frame, and a second face feature point locator layer for optimizing the positioning of the key feature points of the driver's face; the output end of the second fully connected layer is connected to the input ends of the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer, respectively; the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer are input into the input end of the CA-O-Net subnetwork;

[0015] The CA-O-Net subnetwork includes: a seventh convolutional normalization layer for extracting the first high-level feature of the driver's face, an eighth convolutional normalization layer for extracting the second high-level feature of the driver's face, a third attention CA layer for focusing on the feature area related to fatigue driving in the driver's face, a ninth convolutional normalization layer for extracting the third high-level feature of the driver's face, a tenth convolutional normalization layer for extracting the fourth high-level feature of the driver's face, and a third fully connected layer for fusing the extracted features; and also includes: a third face classifier layer for judging the driver's face, a third bounding box regressor layer for outputting the position of the driver's face bounding box, and a third face feature point locator layer for accurately locating the key feature points of the driver's face; the output end of the third fully connected layer is respectively connected to the input ends of the third face classifier layer, the third bounding box regressor layer, and the third face feature point locator layer;

[0016] Any convolution normalization layer among the first convolution normalization layer, the second convolution normalization layer, the third convolution normalization layer, the fourth convolution normalization layer, the fifth convolution normalization layer, the sixth convolution normalization layer, the seventh convolution normalization layer, the eighth convolution normalization layer, the ninth convolution normalization layer and the tenth convolution normalization layer includes: a first Conv module, a first batch normalization BN module, and a first PReLU module connected in sequence.

[0017] Furthermore, any attention CA layer in the first attention CA layer, the second attention CA layer and the third attention CA layer includes: a first Residual module for extracting multi-scale features of the driver's face, a first XAvg Pool module for extracting horizontal pooling features of the driver's face, a first YAvg Pool module for extracting vertical pooling features of the driver's face, a first Concat-Conv2d module for splicing and extracting horizontal and vertical pooling features of the driver's face and extracting high-level features of the driver's face, a first BN-Non-linear module for batch normalizing the driver's face features, a first Conv2d module for extracting horizontal hierarchical features of the driver's face, a second Conv2d module for extracting vertical hierarchical features of the driver's face, a first Sigmoid module for generating an attention weight matrix for the driver's face, a second Sigmoid module for generating an attention weight matrix for the driver's face, and a first Re-weight module for reweighting the input driver's face features; the output end of the first Residual module is connected to the input end of the first XAvg Pool module, the first YAvgPool module and the first Re-weight module respectively; the first XAvg The output ends of the Pool module and the first YAvg Pool module are both connected to the input end of the first Concat-Conv2d module; the output end of the first Concat-Conv2d module is connected to the input end of the first BN-Non-linear module; the output end of the first BN+Non-linear module is respectively connected to the input end of the first Conv2d module and the second Conv2d module; the output end of the first Conv2d module is connected to the input end of the first Sigmoid module; the output end of the second Conv2d module is connected to the input end of the second Sigmoid module; the output ends of the first Sigmoid module and the second Sigmoid module are both connected to the input end of the first Re-weight module.

[0018] Based on the above technical means, the attention CA layer is added to the original MTCNN face detection network to improve the accuracy and robustness of driver facial recognition, and the batch normalization BN module is introduced to accelerate the training efficiency of the MTCNN face detection network and improve the network performance.

[0019] Furthermore, in the attention CA layer: let X∈R C×H×W Representing the input driver’s feature map, each channel of the feature map is first encoded along the horizontal and vertical directions by the pooling kernel;

[0020] Among them, after the horizontal pooling operation, the expression of the feature map output by the cth channel with a height of h is:

[0021]

[0022] Where, Represents the output value of the cth channel with a height of h after the horizontal pooling operation, W represents the width of the feature map in the horizontal direction, and i represents the i-th feature in the horizontal direction;

[0023] After the vertical pooling operation, the expression of the feature map output by the cth channel with a width of w is:

[0024]

[0025] Where, represents the output value of the cth channel with a width of w after the vertical pooling operation, H represents the height of the feature map in the vertical direction, H represents the height of the feature map in the vertical direction, and j represents the i-th feature in the vertical direction;

[0026] Next, the feature maps after the horizontal and vertical pooling operations are spliced together, and the dimensionality reduction operation is performed through convolution. The generated feature map expression is:

[0027] f=δ(F1([z h ,z w ]))

[0028] In the formula, δ represents the nonlinear activation function, F1 represents the size of 1×1 convolution, f represents the processed feature map, z h Represents the feature map after horizontal pooling, z w Represents the feature map after vertical pooling;

[0029] Then, the obtained feature map is decomposed into two independent tensors along the spatial dimension, and the number of channels of the two tensors is transformed to the same as the input feature map through convolution operation. Figure X For the same number of channels, the expression is:

[0030] g h =σ(F h (f h ))

[0031] g w =σ(F w (f w ))

[0032] Where g h represents the vertical attention weight, σ represents the Sigmoid activation function, F h represents the convolution operation, f h Represents the tensor g that is highly correlated with the feature map f after decomposition along the spatial dimension. w represents the horizontal attention weight, F wThe convolution operation represents, f w Represents the tensor related to width after the feature map f is decomposed along the spatial dimension;

[0033] Finally, the weighted output feature map is calculated based on the calculated horizontal attention weight and vertical attention weight, and the expression is:

[0034]

[0035] Where y c (i, j) represents the output eigenvalue at the c-th channel position (i, j), x c (i, j) represents the eigenvalue of the input at the cth channel position (i, j), represents the attention weight of the c-th channel with height i, represents the attention weight of the c-th channel with width i.

[0036] Furthermore, in the batch normalization BN module: let the feature input to the batch normalization BN module be z = [z1,z2,…,z m ], first, calculate the mean and variance of each feature;

[0037] The expression of the mean is:

[0038]

[0039] In the formula, μ represents the mean, m represents the number of input features, z i Represents the eigenvalue of the i-th feature in a certain dimension;

[0040] The expression for variance is:

[0041]

[0042] Where, σ 2 represents variance;

[0043] Next, use the calculated mean and variance to normalize each feature and standardize the feature value. The expression is:

[0044]

[0045] Where, represents the normalized feature, and ε represents a constant;

[0046] Finally, two learnable parameters, scaling parameter and offset parameter, are introduced to perform linear transformation on the normalized features. The expression is:

[0047]

[0048] Where y i represents the feature after linear transformation, γ represents the scaling parameter, and β represents the scaling parameter.

[0049] Furthermore, during the training process of the PFLD facial key point model, the loss function is constructed using the driver's facial key point regression loss and the driver's head pose estimation loss;

[0050] The expression of the face key point regression loss is:

[0051]

[0052] Where, L pts represents the driver's face key point regression loss, SmoothL1 represents the smooth L1 loss, y i represents the true key point coordinates of the i-th sample, represents the predicted key point coordinates of the i-th sample, and N represents the number of samples;

[0053] The expression for estimating the driver's head posture estimation loss using the mean square error between the predicted driver's head posture and the actual driver's head posture is:

[0054]

[0055] Where, L aux represents the driver head pose estimation loss, θ i represents the true driver head pose of the i-th sample, represents the predicted driver head pose of the i-th sample;

[0056] The expression of the loss function is:

[0057] L total =L pts +λL aux

[0058] Where, L total represents the loss function for training the PFLD model, and λ represents the weight coefficient;

[0059] The PFLD face key point model is trained by the back propagation algorithm and the gradient descent strategy. When the loss function converges, the training is completed and the trained PFLD face key point model is obtained.

[0060] Furthermore, the process of detecting facial key points using the trained PFLD facial key point model is as follows:

[0061] According to the coordinates of the key points of the face, the eye positions and mouth positions of the driver's face image are located, and the local area maps of the eyes and the local area maps of the mouth are cut out.

[0062] Furthermore, based on the coordinates of the key points of the driver's face, the aspect ratio of the driver's eyes and the aspect ratio of the driver's mouth are calculated to determine whether the driver's eyes and / or mouth are in a fatigue state; if both the driver's eyes and mouth are not in a fatigue state, it means that the driver is not in a fatigue state; otherwise, it means that the driver is in a fatigue state;

[0063] The process of determining whether the driver's eyes are in a fatigue state is as follows:

[0064] First, calculating the driver's eye aspect ratio, wherein the driver's eye aspect ratio includes the driver's left eye aspect ratio and the driver's right eye aspect ratio;

[0065] The expression of the driver's left eye aspect ratio is:

[0066]

[0067] Where, P 38 Represents the coordinates of the key point of the outer corner of the left eye, P 42 Indicates the coordinates of the left eye's inner corner key point, P 39 Represents the coordinates of the key point in the middle of the upper eyelid of the left eye, P 41 Represents the coordinates of the key point in the middle of the lower eyelid of the left eye, P 37 Indicates the coordinates of the left end key point of the left eye, P 40 Represents the coordinates of the right end key point of the left eye;

[0068] The expression of the driver's right eye aspect ratio is:

[0069]

[0070] Where, P 44 Represents the coordinates of the key points of the outer corner of the right eye, P 48 Indicates the coordinates of the key point of the inner corner of the right eye, P 45 Represents the coordinates of the key point in the middle of the upper eyelid of the right eye, P 47 Represents the coordinates of the key point in the middle of the lower eyelid of the right eye, P 43 Indicates the coordinates of the left end key point of the right eye, P 46 The coordinates of the right end key point of the right eye;

[0071] When the eye aspect ratio of the driver's eyes is less than the set eye fatigue threshold, it indicates that the driver's eyes are in a closed state; the expression of the driver's eye aspect ratio is:

[0072]

[0073] Where, EAR represents the driver's eye aspect ratio;

[0074] Next, calculate the PERCLOS eyelid closure percentage, which is expressed as:

[0075]

[0076] Where PERCLOS represents the percentage of eyelid closure, N represents the total number of frames of detected driver data, and Nclose represents the number of eye closure frames in the detected driver data;

[0077] Finally, the driver's blink frequency and the number of consecutive eye closure frames are considered to determine whether the driver's eyes are in a fatigue state. The judgment formula is:

[0078]

[0079] Where B represents the actual number of blinks per minute, B threshold represents the blink threshold, C represents the actual number of consecutive eye-closing frames, and C threshold represents the threshold of the number of consecutive eye-closing frames, α represents the weight coefficient, β represents the weight coefficient, F represents the comprehensive judgment function, and Fatigue represents the fatigue judgment condition;

[0080] When the judgment value of the comprehensive judgment function exceeds 1, or the actual number of continuous eye-closing frames exceeds the continuous eye-closing frame threshold, or the actual number of blinks per minute exceeds the blink threshold, it indicates that the driver's eyes are in a state of fatigue.

[0081] Furthermore, the process of determining whether the driver's mouth is in a fatigue state is as follows:

[0082] The driver's mouth aspect ratio is calculated using the key point coordinates of the driver. When the driver's mouth aspect ratio exceeds the set mouth fatigue threshold, it indicates that the driver is yawning. The calculation expression of the driver's mouth aspect ratio is:

[0083]

[0084] Where MAR represents the driver's mouth aspect ratio, P 61 Indicates the coordinates of the key point on the left end of the mouth, P 65 Indicates the coordinates of the key point on the right end of the mouth, P 64 Represents the coordinates of the key points of the upper lip of the mouth, P 66 Represents the coordinates of the key points of the lower lip of the mouth, P 62 Represents the coordinates of the key points on the inner edge of the upper lip of the mouth, P 68 Represents the coordinates of the key points on the inner edge of the lower lip of the mouth;

[0085] The yawning frequency per unit time is calculated based on the aspect ratio of the driver's mouth. When the driver's yawning frequency per unit time exceeds the set mouth yawning frequency, it indicates that the driver's mouth is in a fatigue state. The calculation expression for the yawning frequency is:

[0086]

[0087] Where K Y Indicates the number of yawning frames per unit time T, K T Then it is the total number of frames in unit time T.

[0088] The present invention also provides a system for detecting fatigue driving during dynamic driving, comprising:

[0089] A data acquisition module, used to acquire an image of the driver to be detected;

[0090] A face detection module is configured to input a driver image to be detected into a trained improved MTCNN face detection network for driver face detection. The improved MTCNN face detection network embeds an attention (CA) layer in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, which is used to improve face recognition accuracy. Furthermore, a batch normalization (BN) module is introduced into the three sub-networks P-Net, R-Net, and O-Net.

[0091] The key point detection module is used to input the detected driver's face into the trained PFLD face key point model to perform facial key point detection and extract the key point coordinates of the driver's face;

[0092] The fatigue state judgment module is used to use the key point coordinates of the driver's face to calculate the driver's eye aspect ratio and the driver's mouth aspect ratio respectively, and judge whether the driver is in a fatigue state based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

[0093] Compared with the prior art, the beneficial effects of this method are:

[0094] This paper proposes a method and system for detecting driver fatigue during dynamic driving. In this method, accurate driver face detection is achieved through a trained improved MTCNN face detection network. The improved MTCNN face detection network embeds an attention (CA) layer in the third layer of each of the three sub-networks of the original MTCNN face detection network (P-Net, R-Net, and O-Net), designed to improve face recognition accuracy. Furthermore, a batch normalization (BN) module is introduced into each of the three sub-networks (P-Net, R-Net, and O-Net). The detected driver face is then input into a trained PFLD facial key point model to extract the coordinates of the driver's facial key points. Based on the key point coordinates, the eye and mouth aspect ratios are calculated, respectively. This method utilizes multiple facial features to discern driver fatigue during dynamic driving, improving the accuracy of driver fatigue detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 A flowchart showing a method for detecting fatigue driving during dynamic driving according to an embodiment of the present invention;

[0096] Figure 2 A structural diagram showing the P-Net subnetwork of the original MTCNN face detection network proposed in an embodiment of the present invention;

[0097] Figure 3 A structural diagram showing the R-Net sub-network of the original MTCNN face detection network proposed in an embodiment of the present invention;

[0098] Figure 4 A structural diagram showing the O-Net sub-network of the original MTCNN face detection network proposed in an embodiment of the present invention;

[0099] Figure 5 A structural diagram showing the CA-P-Net sub-network of the improved MTCNN face detection network proposed in an embodiment of the present invention;

[0100] Figure 6 A structural diagram showing the CA-R-Net sub-network of the improved MTCNN face detection network proposed in an embodiment of the present invention;

[0101] Figure 7 A structural diagram showing the CA-O-Net sub-network of the improved MTCNN face detection network proposed in an embodiment of the present invention;

[0102] Figure 8 A schematic diagram showing the structure of the attention CA layer proposed in an embodiment of the present invention;

[0103] Figure 9 A graph showing a change in the driver's eye aspect ratio EAR according to an embodiment of the present invention;

[0104] Figure 10 A graph showing a change in the driver's mouth aspect ratio MAR according to an embodiment of the present invention;

[0105] Figure 11 An example diagram showing detection results of a fatigue driving detection method during dynamic driving according to an embodiment of the present invention;

[0106] Figure 12 An example diagram showing the YawDD image dataset used in an embodiment of the present invention;

[0107] Figure 13 A comparison curve chart showing the accuracy and loss values of the improved MTCNN face detection network proposed in an embodiment of the present invention and the original MTCNN face detection network;

[0108] Figure 14 An example diagram showing a driver's face detected by the improved MTCNN face detection network proposed in an embodiment of the present invention;

[0109] Figure 15 A distribution diagram showing the coordinates of the driver's facial key points detected by the PFLD facial key point model proposed in an embodiment of the present invention;

[0110] Figure 16 The figure shows a structural diagram of a fatigue driving detection system during dynamic driving proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0111] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0112] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0113] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0114] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0115] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0116] Example 1

[0117] This embodiment proposes a method for detecting fatigue driving during dynamic driving. Figure 1 The flowchart of the method shown generally includes the following steps:

[0118] S1: Acquire the driver image to be detected;

[0119] S2: Inputting the driver image to be detected into the trained improved MTCNN face detection network to perform driver face detection; the improved MTCNN face detection network embeds an attention CA layer for improving face recognition accuracy in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, and introduces a batch normalization (BN) module in the three sub-networks P-Net, R-Net, and O-Net;

[0120] S3: Input the detected driver's face into the trained PFLD face key point model to perform face key point detection and extract the key point coordinates of the driver's face;

[0121] S4: Using the key point coordinates of the driver's face, respectively calculate the driver's eye aspect ratio and the driver's mouth aspect ratio, and determine whether the driver is in a fatigue state based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

[0122] The existing MTCNN face detection network is a multi-task convolutional neural network consisting of three subnetworks: P-Net, R-Net, and O-Net. It generates candidate rectangular boxes of varying sizes through multi-scale image transformations, then uses a sliding window method to detect facial regions and extract features from the boxes. To improve face detection accuracy, the MTCNN face detection network incorporates the non-maximum suppression (NMS) algorithm to remove highly overlapping candidate boxes within the same object and retain the optimal bounding box. After progressive processing by the three subnetworks, the predicted face box and its corresponding keypoint locations are ultimately output.

[0123] like Figure 2 The P-Net subnetwork architecture of the original MTCNN face detection network is shown. The P-Net subnetwork rapidly generates candidate face boxes and their corresponding bounding box regression vectors. The P-Net subnetwork takes a 12×12×3 RGB image as input and sequentially scans the input image using a sliding window mechanism with a fixed step size to determine whether a face exists within the current 12×12 window. If a face is detected, the P-Net subnetwork predicts the face bounding box and keypoint locations. The P-Net subnetwork's classification results are used to determine whether a candidate box contains a face, while the regression results are used to optimize the candidate box's position to more accurately cover the face area.

[0124] like Figure 3The R-Net subnetwork architecture of the original MTCNN face detection network is shown. The R-Net subnetwork filters and optimizes the face region candidates generated by the P-Net subnetwork. Through bounding box regression, the R-Net subnetwork further fine-tunes the position and size of the candidate boxes and uses NMS to filter overlapping candidate boxes to retain the optimal face region. In the R-Net subnetwork, the input image size is increased to 24×24×3, and convolutional and fully connected layers are added to the network architecture. This improves the R-Net subnetwork's accuracy in detecting and regressing face regions.

[0125] like Figure 4 The structure of the O-Net subnetwork of the original MTCNN face detection network is shown. The O-Net subnetwork uses a larger input image size and adds convolutional layers. This increased network depth and parameters allows the O-Net subnetwork to more precisely extract facial features while suppressing false positives, improving detection accuracy. Ultimately, the O-Net network generates more precise coordinates for facial landmarks, representing the two-dimensional coordinates of the five landmarks as a ten-dimensional vector.

[0126] In this embodiment, the original MTCNN face detection network has insufficient ability to extract key area features in complex scenarios. The third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network is embedded with an attention CA layer for improving face recognition accuracy. The coordinate attention (CA) mechanism is used in the attention CA layer to enhance the spatial relationship of the image. Coordinate information is introduced into the driver face detection task, and the attention distribution of the feature map is adjusted to focus on the target position and improve the detection performance. The attention CA layer decomposes the two-dimensional global pooling into two one-dimensional pooling operations, performing global average pooling along the X-axis and Y-axis respectively. Capture long-distance correlation in one coordinate direction while retaining the precise position information in the other direction. The input feature map of the attention CA layer is encoded into an attention map that is sensitive to direction and position information, and applied to the original feature map, thereby enhancing the feature expression capability of the region of interest.

[0127] In this embodiment, the improved MTCNN face detection network includes: a CA-P-Net subnetwork, a CA-R-Net subnetwork, and a CA-O-Net subnetwork;

[0128] like Figure 5The structural diagram of the CA-P-Net subnetwork of the improved MTCNN face detection network is shown. The CA-P-Net subnetwork includes: a first convolutional normalization layer for extracting the first primary feature of the driver image, a second convolutional normalization layer for extracting the second primary feature of the driver image, a first attention CA layer for extracting the spatial position sensitive feature of the driver image, a third convolutional normalization layer for extracting the third primary feature of the driver image, and a first fully connected layer for fusing the extracted features; it also includes: a first face classifier layer for judging whether the driver's face exists, a first bounding box regressor layer for determining the range of the driver's face, and a first face feature point locator layer for locating the key feature points of the driver's face; the output end of the first fully connected layer is connected to the input ends of the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer respectively, and the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer are input into the input end of the CA-P-Net subnetwork.

[0129] like Figure 6 The structural diagram of the CA-R-Net subnetwork of the improved MTCNN face detection network is shown. The CA-R-Net subnetwork includes: a fourth convolutional normalization layer for extracting the first intermediate features of the driver's face, a fifth convolutional normalization layer for extracting the second intermediate features of the driver's face, a second attention CA layer for enhancing attention to the key areas of the driver's face, a sixth convolutional normalization layer for extracting the third intermediate features of the driver's face, and a second fully connected layer for fusing the extracted features; it also includes: a second face classifier layer for screening candidate frames for the absence of the driver's face, a second bounding box regressor layer for adjusting the position of the driver's face candidate frame, and a second face feature point locator layer for optimizing the positioning of the key feature points of the driver's face; the output end of the second fully connected layer is connected to the input ends of the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer respectively, and the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer are input to the input end of the CA-O-Net subnetwork.

[0130] like Figure 7The structural diagram of the CA-O-Net subnetwork of the improved MTCNN face detection network shown in the figure, the CA-O-Net subnetwork includes: the CA-O-Net subnetwork includes: the seventh convolutional normalization layer for extracting the first high-level feature of the driver's face, the eighth convolutional normalization layer for extracting the second high-level feature of the driver's face, the third attention CA layer for focusing on the feature area related to fatigue driving in the driver's face, the ninth convolutional normalization layer for extracting the third high-level feature of the driver's face, the tenth convolutional normalization layer for extracting the fourth high-level feature of the driver's face, and the third fully connected layer for fusing the extracted features; it also includes: a third face classifier layer for judging the driver's face, a third bounding box regressor layer for outputting the position of the driver's face bounding box, and a third face feature point locator layer for accurately locating the key feature points of the driver's face; the output end of the third fully connected layer is respectively connected to the input ends of the third face classifier layer, the third bounding box regressor layer and the third face feature point locator layer.

[0131] Any convolution normalization layer among the first convolution normalization layer, the second convolution normalization layer, the third convolution normalization layer, the fourth convolution normalization layer, the fifth convolution normalization layer, the sixth convolution normalization layer, the seventh convolution normalization layer, the eighth convolution normalization layer, the ninth convolution normalization layer and the tenth convolution normalization layer includes: a first Conv module, a first batch normalization BN module, and a first PReLU module connected in sequence.

[0132] In this embodiment, if Figure 8The structural diagram of the attention CA layer shown in the figure shows that any attention CA layer among the first attention CA layer, the second attention CA layer and the third attention CA layer includes: a first residual module for extracting multi-scale features of the driver's face, a first XAvg Pool module for extracting horizontal pooling features of the driver's face, a first YAvg Pool module for extracting vertical pooling features of the driver's face, a first Concat-Conv2d module for splicing and extracting horizontal and vertical pooling features of the driver's face and extracting high-level features of the driver's face, a first BN-Non-linear module for batch normalizing the driver's face features, a first Conv2d module for extracting horizontal hierarchical features of the driver's face, a second Conv2d module for extracting vertical hierarchical features of the driver's face, a first Sigmoid module for generating an attention weight matrix for the driver's face, a second Sigmoid module for generating an attention weight matrix for the driver's face, and a first Re-weight module for reweighting the input driver's face features; the output end of the first residual module is connected to the input end of the first XAvg Pool module, the first YAvg Pool module and the first Re-weight module respectively; The output ends of the Pool module and the first YAvg Pool module are both connected to the input end of the first Concat-Conv2d module; the output end of the first Concat-Conv2d module is connected to the input end of the first BN-Non-linear module; the output end of the first BN+Non-linear module is respectively connected to the input end of the first Conv2d module and the second Conv2d module; the output end of the first Conv2d module is connected to the input end of the first Sigmoid module; the output end of the second Conv2d module is connected to the input end of the second Sigmoid module; the output ends of the first Sigmoid module and the second Sigmoid module are both connected to the input end of the first Re-weight module.

[0133] In this embodiment, the attention CA layer is a lightweight attention mechanism with a small number of parameters. It can embed position information to improve the accuracy of the model in locating structural features such as the driver's facial features. The attention CA layer improves the ability of feature learning by assigning different weights to different channels and spatial positions of the feature map. In the attention CA layer: Let X∈R C×H×W Representing the input driver’s feature map, each channel of the feature map is first encoded along the horizontal and vertical directions by the pooling kernel;

[0134] Among them, after the horizontal pooling operation, the expression of the feature map output by the cth channel with a height of h is:

[0135]

[0136] Where, Represents the output value of the cth channel with a height of h after the horizontal pooling operation, W represents the width of the feature map in the horizontal direction, and i represents the i-th feature in the horizontal direction;

[0137] After the vertical pooling operation, the expression of the feature map output by the cth channel with a width of w is:

[0138]

[0139] Where, represents the output value of the cth channel with a width of w after the vertical pooling operation, H represents the height of the feature map in the vertical direction, H represents the height of the feature map in the vertical direction, and j represents the i-th feature in the vertical direction;

[0140] Next, the feature maps after the horizontal and vertical pooling operations are concatenated and dimensionality reduction is performed through 1×1 convolution to obtain a feature map of size (C / r)×1×(W+H). The generated feature map expression is:

[0141] f=δ(F1([z h ,z w ]))

[0142] In the formula, δ represents the nonlinear activation function, F1 represents the size of 1×1 convolution, f represents the processed feature map, z h Represents the feature map after horizontal pooling, z w Represents the feature map after vertical pooling;

[0143] Then, the obtained feature map is decomposed into two independent tensors along the spatial dimension, whose sizes are (C / r)×H and (C / r)×W respectively. The number of channels of the two tensors is transformed to the same as the input feature map through a 1×1 convolution operation. Figure X For the same number of channels, the expression is:

[0144] g h =σ(F h (f h ))

[0145] g w =σ(F w (f w ))

[0146] Where g h represents the vertical attention weight, σ represents the Sigmoid activation function, F h represents the convolution operation, f hRepresents the tensor g that is highly correlated with the feature map f after decomposition along the spatial dimension. w represents the horizontal attention weight, F w The convolution operation represents, f w Represents the tensor related to width after the feature map f is decomposed along the spatial dimension;

[0147] Finally, the weighted output feature map is calculated based on the calculated horizontal attention weight and vertical attention weight, and the expression is:

[0148]

[0149] Where y c (i, j) represents the output eigenvalue at the c-th channel position (i, j), x c (i, j) represents the eigenvalue of the input at the cth channel position (i, j), represents the attention weight of the c-th channel with height i, represents the attention weight of the c-th channel with width i.

[0150] In this example, the pooling kernel size in the X-axis direction is (1, W), and the pooling kernel size in the Y-axis direction is (□, 1). After average pooling, feature maps of size C × H × 1 and C × 1 × W are obtained. This preserves long-range dependencies in the X-axis while ensuring accurate position information in the Y-axis.

[0151] The attention CA layer decomposes channel attention into one-dimensional feature encodings along the X and Y spatial directions, capturing the image's directional and positional information and effectively integrating this spatial coordinate information into the generated attention map. The attention CA layer enhances feature directionality, assigning higher attention weights to key areas such as the face and lowering the weights for non-critical areas such as the background, thereby improving the accuracy of localizing the driver's face in the image. During training, the attention CA layer improves the accuracy and robustness of driver face detection tasks, thereby enhancing the performance of driver fatigue detection.

[0152] In this embodiment, the attention CA layer continuously adjusts and optimizes its internal parameters during training, ultimately achieving optimal parameters. The detection accuracy of the MTCNN face detection network decreases when faced with lighting changes or complex backgrounds. The attention CA layer, used to improve face recognition accuracy, is embedded in the third layers of the three sub-networks of the original MTCNN face detection network, P-Net, R-Net, and O-Net. This improves the MTCNN face detection network's sensitivity to driver facial information and enhances its robustness in complex backgrounds, ensuring accurate positioning of the driver's face and other spatial locations.

[0153] In this embodiment, during training, the update of the parameters of the previous layer in the MTCNN face detection network affects the input distribution of the next layer. A low learning rate and overly sophisticated parameter initialization can slow down model training, making it difficult for the MTCNN face detection network with saturated nonlinear activation functions to converge, leading to the vanishing gradient problem.

[0154] The Batch Normalization (BN) module normalizes the output of the intermediate layer to reduce the difference in input values to the same range. While improving the degree of gradient convergence and accelerating network training, each layer faces input values with the same feature distribution, reducing the uncertainty caused by changes and the impact on the next layer of the network, making each layer of the network relatively independent, alleviating the problem of gradient disappearance during training, and improving network performance.

[0155] In the batch normalization BN module: Let the feature input to the batch normalization BN module be z = [z1,z2,…,z m ], first, calculate the mean and variance of each feature;

[0156] The expression of the mean is:

[0157]

[0158] In the formula, μ represents the mean, m represents the number of input features, z i Represents the eigenvalue of the i-th feature in a certain dimension;

[0159] The expression for variance is:

[0160]

[0161] Where, σ 2 represents variance;

[0162] Next, use the calculated mean and variance to normalize each feature and standardize the feature value. The expression is:

[0163]

[0164] Where, represents the normalized feature, and ε represents a constant;

[0165] Finally, two learnable parameters, scaling parameter and offset parameter, are introduced to perform linear transformation on the normalized features. The expression is:

[0166]

[0167] Where y irepresents the feature after linear transformation, γ represents the scaling parameter, and β represents the scaling parameter.

[0168] The original MTCNN face detection network adopts a cascade architecture, and the multi-stage processing method helps to gradually improve the detection accuracy, but also increases the inference time. This architecture shows a certain efficiency bottleneck in high-resolution images or real-time processing scenarios, and it is difficult to meet the application scenarios of driver fatigue monitoring with high speed requirements. The original MTCNN face detection network is sensitive to the scale, contrast and noise of the input image, and the input attributes need to be preprocessed. It shows insufficient robustness when facing data that has not been fully preprocessed, affecting the detection effect in complex actual scenarios. In the present invention, by adding a batch normalization BN module and using the batch normalization BN module in combination with the ReLU activation function, the training efficiency of the MTCNN face detection network is accelerated and the performance of the network is improved, especially in terms of convergence speed, stability and suppression of overfitting.

[0169] Facial key point detection can accurately locate the coordinates of facial features and characterize various parts of the face, including eyebrows, eyes, nose, mouth, and facial contours. The PFLD facial key point model has the advantages of small size, few parameters, and fast detection speed. It can detect the coordinates of the driver's facial key points with high accuracy, making it suitable for integration into mobile embedded devices.

[0170] The process of facial key point detection using the trained PFLD facial key point model is as follows: the detected driver's face image is processed and input into the trained PFLD facial key point model, the trained PFLD facial key point model is used to predict the key point coordinates of the face, and the eye and mouth positions of the driver's face image are located according to the key point coordinates of the face, and the local area maps of the eyes and the local area maps of the mouth are cut out.

[0171] In this embodiment, the trained PFLD facial key point model first preprocesses the input driver's facial image during processing. This includes using a face detection algorithm to determine the region of interest, cropping, and normalization to ensure image size consistency and quality stability in subsequent processing. First, within the backbone network, comprised of the lightweight convolutional neural network MobileNetV2 within the PFLD facial key point model, multiple layers of convolution and nonlinear activation functions are used to progressively extract low-level and high-level features, capturing local details and global semantic information at different scales. Based on the extracted feature maps, features at different scales are fused using a multi-scale fully connected layer, which is then converted into the coordinates of the driver's facial key points. Simultaneously, an auxiliary branch is introduced into the PFLD facial key point model to estimate the driver's head pose. Through a multi-task learning mechanism, the coordinated optimization of driver key point localization and pose constraints is achieved, providing geometric constraints to the backbone network and accurately locating the coordinates of the driver's facial key points. This effectively alleviates the local minima problem associated with single-task optimization.

[0172] In this embodiment, the key point coordinates predicted by the PFLD facial key point model are further processed, such as coordinate smoothing or correction based on statistical information, to further improve the stability and robustness of the detection results.

[0173] Based on the coordinates of the key points of the driver's face, the aspect ratio of the driver's eyes and the aspect ratio of the driver's mouth are calculated to determine whether the driver's eyes and / or mouth are in a fatigue state; when the driver's eyes and mouth are not in a fatigue state, it means that the driver is not in a fatigue state; otherwise, it means that the driver is in a fatigue state.

[0174] In this embodiment, the coordinates of six key points are obtained from the left eye and right eye areas respectively to determine whether the driver's eyes are in a fatigue state. The process is as follows:

[0175] First, calculating the driver's eye aspect ratio, wherein the driver's eye aspect ratio includes the driver's left eye aspect ratio and the driver's right eye aspect ratio;

[0176] The expression of the driver's left eye aspect ratio is:

[0177]

[0178] Where, P 38 Represents the coordinates of the key point of the outer corner of the left eye, P 42 Indicates the coordinates of the left eye's inner corner key point, P 39 Represents the coordinates of the key point in the middle of the upper eyelid of the left eye, P 41 Represents the coordinates of the key point in the middle of the lower eyelid of the left eye, P 37 Indicates the coordinates of the left end key point of the left eye, P40 Represents the coordinates of the right end key point of the left eye;

[0179] The expression of the driver's right eye aspect ratio is:

[0180]

[0181] Where, P 44 Represents the coordinates of the key points of the outer corner of the right eye, P 48 Indicates the coordinates of the key point of the inner corner of the right eye, P 45 Represents the coordinates of the key point in the middle of the upper eyelid of the right eye, P 47 Represents the coordinates of the key point in the middle of the lower eyelid of the right eye, P 43 Indicates the coordinates of the left end key point of the right eye, P 46 The coordinates of the right end key point of the right eye;

[0182] When the eye aspect ratio of the driver's eyes is less than the set eye fatigue threshold, it indicates that the driver's eyes are in a closed state; the expression of the driver's eye aspect ratio is:

[0183]

[0184] Where, EAR represents the driver's eye aspect ratio;

[0185] Next, calculate the PERCLOS eyelid closure percentage, which is expressed as:

[0186]

[0187] Where PERCLOS represents the percentage of eyelid closure, N represents the total number of frames of detected driver data, and Nclose represents the number of frames with eye closure in the detected driver data. As the number of frames with eye closure increases, the PERCLOS percentage of eyelid closure increases accordingly.

[0188] Finally, the driver's blink frequency and the number of consecutive eye closure frames are considered to determine whether the driver's eyes are in a fatigue state. The judgment formula is:

[0189]

[0190] Where B represents the actual number of blinks per minute, B threshold represents the blink threshold, C represents the actual number of consecutive eye-closing frames, and C threshold represents the threshold of the number of consecutive eye-closing frames, α represents the weight coefficient, β represents the weight coefficient, F represents the comprehensive judgment function, and Fatigue represents the fatigue judgment condition.

[0191] For example, α + β = 1, β = 0.6, and α = 0.4. When the number of blinks per minute exceeds 20, the driver is considered to be in a fatigued driving state and should reduce fatigue by increasing blink frequency. The blink threshold is set at 20. If the driver closes their eyes continuously for more than 2 seconds, the risk of a traffic accident increases significantly. The frame rate of the image acquisition device used in this experiment is 30 frames per second, and the threshold for continuous eye closure is set at 60.

[0192] When the judgment value of the comprehensive judgment function exceeds 1, or the actual number of continuous eye-closing frames exceeds the continuous eye-closing frame threshold, or the actual number of blinks per minute exceeds the blink threshold, it indicates that the driver's eyes are in a state of fatigue.

[0193] In this embodiment, considering that the eye shapes and sizes of different drivers are different, in order to ensure that the setting of the EAR threshold is more reasonable and accurate, the present invention selects 20 videos collected by placing them on the rearview mirror from the public dataset YawDD to conduct an EAR threshold setting experiment to set the threshold.

[0194] like Figure 9 The graph below shows the change in the driver's eye aspect ratio (EAR) over time during a 20-second blink. The horizontal axis represents the total number of image frames within that 20-second period, and the vertical axis represents the EAR value. When the driver's eyes are open, the EAR value fluctuates within a certain range. During a blink, the EAR value drops rapidly before returning to a stable range, at which point the EAR curve shows a trough. The 20 sets of EAR values obtained from the experiment were divided into five groups. The EAR values for three normal eye-open and eye-closed states were statistically analyzed, as shown in Table 1.

[0195] Table 1

[0196] Group first Second time The third time Group 1 0.24 / 0.10 0.22 / 0.05 0.21 / 0.09 Group 2 0.26 / 0.08 0.22 / 0.09 0.23 / 0.07 Group 3 0.24 / 0.11 0.20 / 0.08 0.21 / 0.07 Group 4 0.21 / 0.08 0.23 / 0.06 0.22 / 0.09 Group 5 0.25 / 0.07 0.26 / 0.10 0.25 / 0.09

[0197] According to the data analysis in the table, when the eyes are open, the eye aspect ratio fluctuates between 0.20 and 0.26. When the eyes are completely closed, the eye aspect ratio mostly fluctuates between 0.05 and 0.11. The eye aspect ratio eye fatigue threshold is set to 0.15 as the standard for determining the eye closure state.

[0198] In this embodiment, six key points of the driver's mouth are selected to determine whether the driver's mouth is in a fatigue state. The process is as follows:

[0199] The driver's mouth aspect ratio is calculated using the key point coordinates of the driver. When the driver's mouth aspect ratio exceeds the set mouth fatigue threshold, it indicates that the driver is yawning. The calculation expression of the driver's mouth aspect ratio is:

[0200]

[0201] Where MAR represents the driver's mouth aspect ratio, P 61 Indicates the coordinates of the key point on the left end of the mouth, P 65 Indicates the coordinates of the key point on the right end of the mouth, P 64 Represents the coordinates of the key points of the upper lip of the mouth, P 66 Represents the coordinates of the key points of the lower lip of the mouth, P 62 Represents the coordinates of the key points on the inner edge of the upper lip of the mouth, P 68 Represents the coordinates of the key points on the inner edge of the lower lip of the mouth;

[0202] The yawning frequency per unit time is calculated based on the aspect ratio of the driver's mouth. When the driver's yawning frequency per unit time exceeds the set mouth yawning frequency, it indicates that the driver's mouth is in a fatigue state. The calculation expression for the yawning frequency is:

[0203]

[0204] Where K Y Indicates the number of yawning frames per unit time T, K T Then it is the total number of frames in unit time T.

[0205] For example, the threshold value of the mouth yawning frequency is set to 0.2, and the unit time T is set to 60 seconds. That is, if the driver yawns more than twice in one minute, he is judged to be fatigued.

[0206] In this embodiment, the yawning phenomenon is detected by the driver's mouth aspect ratio MAR. The driver's mouth aspect ratio MAR is smallest when the mouth is closed, and when the mouth gradually opens, the driver's mouth aspect ratio MAR increases accordingly. Setting an appropriate driver's mouth aspect ratio MAR threshold can effectively identify whether the driver is in a yawning state. In order to set an appropriate driver's mouth aspect ratio MAR value threshold, 29 videos captured by the camera above the dashboard in the YawDD dataset were selected to conduct a driver's mouth aspect ratio MAR threshold setting experiment. The data contains 16 male driver videos and 13 female driver videos. Figure 10 The graph shows the changes in the driver's mouth aspect ratio (MAR). The horizontal axis of the MAR curve corresponding to yawning segments in the dataset video represents the total number of video frames, and the vertical axis represents the MAR value. When the mouth is closed, the driver's mouth aspect ratio (MAR) fluctuates within a stable range. When the driver yawns, the MAR suddenly increases, with a noticeable "peak" appearing on the line graph. When the mouth is slightly open, the MAR value increases slightly, but does not reach a peak.

[0207] The 29 groups of MAR values in the experiment were divided into 5 groups, and the MAR values from mouth closing to mouth opening yawning were counted three times, as shown in Table 2, the experimental results of the mouth aspect ratio.

[0208] Table 2

[0209]

[0210] According to the data in the table, the mouth aspect ratio fluctuates between 0.11 and 0.21 when the mouth is closed or slightly open. When the mouth is open (yawning), the aspect ratio ranges from 0.42 to 0.63. To eliminate errors caused by opening the mouth while speaking, the mouth fatigue threshold is set to 0.3.

[0211] like Figure 11 The example diagram of the detection results based on the fatigue driving detection method during dynamic driving is shown. According to the aspect ratio of the driver's eyes and / or the aspect ratio of the driver's mouth, it is possible to accurately determine whether the driver is in a fatigue state.

[0212] Example 2

[0213] In this embodiment, the process of improving the MTCNN face detection network and the training of the PFLD face key point model is described.

[0214] When training the improved MTCNN face detection network, such as Figure 12 The dataset shown in the figure is an example. This example uses the YawDD (Yawning Detection Dataset) fatigue driving dataset, which contains video footage of drivers' faces captured while driving, divided into two types: frontal and side views. By slicing and filtering these videos, over 700 frames of data were obtained for yawning and normal states, respectively. This facial data exhibits diverse variations in lighting, posture, and occlusion, providing an important and reliable data source for the research presented in this paper.

[0215] In this embodiment, the performance and accuracy of the detection model are evaluated by changes in accuracy (Precision) and loss value (Loss).

[0216] Accuracy refers to the proportion of samples that are actually positive among the samples predicted by the model to be positive. The calculation expression is:

[0217]

[0218] Where Precision represents the accuracy, TP represents the number of target frames that correctly identify the driver's face, and FP represents the number of target frames that incorrectly identify the driver's face.

[0219] like Figure 13The following graph compares the accuracy and loss of the improved MTCNN face detection network and the original MTCNN face detection network. The improved MTCNN face detection network's accuracy increases rapidly before Epoch 20, then stabilizes. The original MTCNN face detection network's accuracy curve takes longer to stabilize, and after stabilization, its accuracy is lower than that of the improved MTCNN face detection network. The loss of both the original and improved MTCNN face detection networks decreases rapidly before Epoch 20, then slowly decreases from Epoch 20 to 60, and stabilizes after Epoch 60. However, the improved MTCNN face detection network has a smoother loss curve and a lower final converged loss, demonstrating better overall convergence than the original model.

[0220] like Figure 14 The figure shows an example of a driver's face detected by the improved MTCNN face detection network. The improved MTCNN face detection network can accurately detect the driver's face, meeting the requirements of fatigue driving detection in normal environments. Furthermore, since drivers operate in environments ranging from normal to complex, the original dataset was expanded to simulate these complex environments. Testing and verification of the improved MTCNN face detection network showed that it can still accurately detect the driver's face even in complex environments such as facial occlusion and low light.

[0221] During the training process of the PFLD facial key point model, the loss function is constructed using the driver's facial key point regression loss and the driver's head posture estimation loss;

[0222] The expression of the face key point regression loss is:

[0223]

[0224] Where, L pts represents the driver's face key point regression loss, SmoothL1 represents the smooth L1 loss, y i represents the true key point coordinates of the i-th sample, represents the predicted key point coordinates of the i-th sample, and N represents the number of samples;

[0225] The expression for estimating the driver's head posture estimation loss using the mean square error between the predicted driver's head posture and the actual driver's head posture is:

[0226]

[0227] Where, L aux represents the driver head pose estimation loss, θ irepresents the true driver head pose of the i-th sample, represents the predicted driver head pose of the i-th sample;

[0228] The expression of the loss function is:

[0229] L total =L pts +λL aux

[0230] Where, L total represents the loss function for training the PFLD model, and λ represents the weight coefficient;

[0231] The PFLD face key point model is trained by the back propagation algorithm and the gradient descent strategy. When the loss function converges, the training is completed and the trained PFLD face key point model is obtained.

[0232] In this embodiment, the PFLD facial key point model is trained based on the YawDD image and video dataset. Figure 15 The PFLD facial key point model shows the distribution of the coordinates of 68 key points on the driver's face. Based on the detected key point coordinates, the eyes and mouth in the input image are precisely located, allowing for the extraction of images of these localized areas.

[0233] In this example, we selected video clips from the Mirror section of the YawDD dataset for testing. This section contains 320 videos covering three driver states: normal, conversational, and fatigued. To ensure experimental accuracy, we excluded videos used for thresholding, ultimately selecting 224 videos. These videos include volunteers of varying genders and ages, and all were recorded in simulated driving environments.

[0234] The proposed method for detecting fatigue driving during dynamic driving was used to test the 224 selected videos. The test results are shown in Table 3 as the video dataset detection results.

[0235] Table 3

[0236] Video Category Number of videos Fatigue quantity Number of tests Accuracy Normal state 64 0 2 96.88％ chat 60 0 4 93.33％ fatigue 100 100 3 97.00％

[0237] According to the test results in the table, when the driver is in a fatigue state, the method proposed in the present invention can accurately identify and issue a warning in a timely manner.

[0238] Example 3

[0239] The present invention also provides a fatigue driving detection system during dynamic driving. Figure 16 The structural diagram of the system shown includes:

[0240] A data acquisition module, used to acquire an image of the driver to be detected;

[0241] A face detection module is configured to input a driver image to be detected into a trained improved MTCNN face detection network for driver face detection. The improved MTCNN face detection network embeds an attention (CA) layer in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, which is used to improve face recognition accuracy. Furthermore, a batch normalization (BN) module is introduced into the three sub-networks P-Net, R-Net, and O-Net.

[0242] The key point detection module is used to input the detected driver's face into the trained PFLD face key point model to perform facial key point detection and extract the key point coordinates of the driver's face;

[0243] The fatigue state judgment module is used to use the key point coordinates of the driver's face to calculate the driver's eye aspect ratio and the driver's mouth aspect ratio respectively, and judge whether the driver is in a fatigue state based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

[0244] The embodiments are provided merely to illustrate the present invention and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications may be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the claims.

Claims

1. A method for detecting fatigue driving during dynamic driving, characterized in that: The following steps are involved: Acquire an image of a driver to be detected; Inputting the driver image to be detected into a trained improved MTCNN face detection network to perform driver face detection; the improved MTCNN face detection network embeds an attention CA layer for improving face recognition accuracy in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, and introduces a batch normalization (BN) module in the three sub-networks P-Net, R-Net, and O-Net; The detected driver's face is input into the trained PFLD face key point model to detect the face key points and extract the key point coordinates of the driver's face; The key point coordinates of the driver's face are used to calculate the driver's eye aspect ratio and the driver's mouth aspect ratio respectively, and whether the driver is in a fatigue state is determined based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

2. The method for detecting fatigue driving during dynamic driving according to claim 1, characterized in that: The improved MTCNN face detection network includes: CA-P-Net subnetwork, CA-R-Net subnetwork and CA-O-Net subnetwork; The CA-P-Net subnetwork includes: a first convolutional normalization layer for extracting a first primary feature of the driver image, a second convolutional normalization layer for extracting a second primary feature of the driver image, a first attention CA layer for extracting a spatial position sensitive feature of the driver image, a third convolutional normalization layer for extracting a third primary feature of the driver image, and a first fully connected layer for fusing the extracted features, which are connected in sequence; and further includes: a first face classifier layer for determining whether the driver's face exists, a first bounding box regressor layer for determining the range of the driver's face, and a first face feature point locator layer for locating key feature points of the driver's face; the output end of the first fully connected layer is connected to the input ends of the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer, respectively; the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the first face classifier layer, the first bounding box regressor layer, and the first face feature point locator layer are input into the input end of the CA-P-Net subnetwork; The CA-R-Net subnetwork includes: a fourth convolutional normalization layer for extracting the first intermediate feature of the driver's face, a fifth convolutional normalization layer for extracting the second intermediate feature of the driver's face, a second attention CA layer for enhancing attention to the key area of the driver's face, a sixth convolutional normalization layer for extracting the third intermediate feature of the driver's face, and a second fully connected layer for fusing the extracted features, which are connected in sequence; and also includes: a second face classifier layer for screening the absence of the driver's face in the candidate frame, a second bounding box regressor layer for adjusting the position of the driver's face candidate frame, and a second face feature point locator layer for optimizing the positioning of the key feature points of the driver's face; the output end of the second fully connected layer is connected to the input ends of the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer, respectively; the judgment on whether the driver's face exists, the driver's face bounding box, and the position of the driver's face feature points output by the second face classifier layer, the second bounding box regressor layer, and the second face feature point locator layer are input into the input end of the CA-O-Net subnetwork; The CA-O-Net subnetwork includes: a seventh convolutional normalization layer for extracting the first high-level feature of the driver's face, an eighth convolutional normalization layer for extracting the second high-level feature of the driver's face, a third attention CA layer for focusing on the feature area related to fatigue driving in the driver's face, a ninth convolutional normalization layer for extracting the third high-level feature of the driver's face, a tenth convolutional normalization layer for extracting the fourth high-level feature of the driver's face, and a third fully connected layer for fusing the extracted features; and also includes: a third face classifier layer for judging the driver's face, a third bounding box regressor layer for outputting the position of the driver's face bounding box, and a third face feature point locator layer for accurately locating the key feature points of the driver's face; the output end of the third fully connected layer is respectively connected to the input ends of the third face classifier layer, the third bounding box regressor layer, and the third face feature point locator layer; Any convolution normalization layer among the first convolution normalization layer, the second convolution normalization layer, the third convolution normalization layer, the fourth convolution normalization layer, the fifth convolution normalization layer, the sixth convolution normalization layer, the seventh convolution normalization layer, the eighth convolution normalization layer, the ninth convolution normalization layer and the tenth convolution normalization layer includes: a first Conv module, a first batch normalization BN module, and a first PReLU module connected in sequence.

3. The method for detecting fatigue driving during dynamic driving according to claim 2, characterized in that: Any attention CA layer in the first attention CA layer, the second attention CA layer and the third attention CA layer includes: a first Residual module for extracting multi-scale features of the driver's face, a first XAvg Pool module for extracting horizontal pooling features of the driver's face, a first YAvg Pool module for extracting vertical pooling features of the driver's face, a first Concat-Conv2d module for splicing and extracting horizontal and vertical pooling features of the driver's face and extracting high-level features of the driver's face, a first BN-Non-linear module for batch normalizing the driver's face features, a first Conv2d module for extracting horizontal hierarchical features of the driver's face, a second Conv2d module for extracting vertical hierarchical features of the driver's face, a first Sigmoid module for generating an attention weight matrix for the driver's face, a second Sigmoid module for generating an attention weight matrix for the driver's face, and a first Re-weight module for reweighting the input driver's face features; the output end of the first Residual module is connected to the input end of the first XAvg Pool module, the first YAvg Pool module and the first Re-weight module respectively; the first XAvg Pool module and the first YAvg The output end of the Pool module is connected to the input end of the first Concat-Conv2d module; the output end of the first Concat-Conv2d module is connected to the input end of the first BN-Non-linear module; the output end of the first BN+Non-linear module is connected to the input end of the first Conv2d module and the second Conv2d module respectively; the output end of the first Conv2d module is connected to the input end of the first Sigmoid module; the output end of the second Conv2d module is connected to the input end of the second Sigmoid module; the output ends of the first Sigmoid module and the second Sigmoid module are both connected to the input end of the first Re-weight module.

4. The method for detecting fatigue driving during dynamic driving according to claim 3, characterized in that: In the attention CA layer: Let X∈R C×H×W Representing the input driver’s feature map, each channel of the feature map is first encoded along the horizontal and vertical directions by the pooling kernel; Among them, after the horizontal pooling operation, the expression of the feature map output by the cth channel with a height of h is: Where, Represents the output value of the cth channel with a height of h after the horizontal pooling operation, W represents the width of the feature map in the horizontal direction, and i represents the i-th feature in the horizontal direction; After the vertical pooling operation, the expression of the feature map output by the cth channel with a width of w is: Where, represents the output value of the cth channel with a width of w after the vertical pooling operation, H represents the height of the feature map in the vertical direction, H represents the height of the feature map in the vertical direction, and j represents the i-th feature in the vertical direction; Next, the feature maps after the horizontal and vertical pooling operations are spliced together, and the dimensionality reduction operation is performed through convolution. The generated feature map expression is: f=δ(F1([z h ,z w ])) In the formula, δ represents the nonlinear activation function, F1 represents the size of 1×1 convolution, f represents the processed feature map, z h Represents the feature map after horizontal pooling, z w Represents the feature map after vertical pooling; Then, the obtained feature map is decomposed into two independent tensors along the spatial dimension, and the number of channels of the two tensors is transformed to the same number of channels as the input feature map X through the convolution operation. The expression is: g h =σ(F h (f h )) g w =σ(F w (f w )) Where g h represents the vertical attention weight, σ represents the Sigmoid activation function, F h represents the convolution operation, f h Represents the tensor g that is highly correlated with the feature map f after decomposition along the spatial dimension. w represents the horizontal attention weight, F w The convolution operation represents, f w Represents the tensor related to width after the feature map f is decomposed along the spatial dimension; Finally, the weighted output feature map is calculated based on the calculated horizontal attention weight and vertical attention weight, and the expression is: Where y c (i, j) represents the output eigenvalue at the c-th channel position (i, j), x c (i, j) represents the eigenvalue of the input at the cth channel position (i, j), represents the attention weight of the c-th channel with height i, represents the attention weight of the c-th channel with width i.

5. The method for detecting fatigue driving during dynamic driving according to claim 2, characterized in that: In the batch normalization BN module: Let the feature input to the batch normalization BN module be z = [z1,z2,…,z m ], first, calculate the mean and variance of each feature; The expression of the mean is: In the formula, μ represents the mean, m represents the number of input features, z i Represents the eigenvalue of the i-th feature in a certain dimension; The expression for variance is: Where, σ 2 represents variance; Next, use the calculated mean and variance to normalize each feature and standardize the feature value. The expression is: Where, represents the normalized feature, and ε represents a constant; Finally, two learnable parameters, scaling parameter and offset parameter, are introduced to perform linear transformation on the normalized features. The expression is: Where y i represents the feature after linear transformation, γ represents the scaling parameter, and β represents the scaling parameter.

6. The method for detecting fatigue driving during dynamic driving according to claim 1, characterized in that: During the training process of the PFLD facial key point model, the loss function is constructed using the driver's facial key point regression loss and the driver's head posture estimation loss; The expression of the face key point regression loss is: Where, L pts represents the driver's face key point regression loss, SmoothL1 represents the smooth L1 loss, y i represents the true key point coordinates of the i-th sample, represents the predicted key point coordinates of the i-th sample, and N represents the number of samples; The expression for estimating the driver's head posture estimation loss using the mean square error between the predicted driver's head posture and the actual driver's head posture is: Where, L aux represents the driver head pose estimation loss, θ i represents the true driver head pose of the i-th sample, represents the predicted driver head pose of the i-th sample; The expression of the loss function is: THE total =L pts +λL aux Where, L total represents the loss function for training the PFLD model, and λ represents the weight coefficient; The PFLD face key point model is trained by the back propagation algorithm and the gradient descent strategy. When the loss function converges, the training is completed and the trained PFLD face key point model is obtained.

7. The method for detecting fatigue driving during dynamic driving according to claim 6, characterized in that: The process of detecting facial key points using the trained PFLD facial key point model is as follows: According to the coordinates of the key points of the face, the eye positions and mouth positions of the driver's face image are located, and the local area maps of the eyes and the local area maps of the mouth are cut out.

8. The method for detecting fatigue driving during dynamic driving according to claim 1, characterized in that: Calculate the driver's eye aspect ratio and mouth aspect ratio based on the key point coordinates of the driver's face to determine whether the driver's eyes and / or mouth are in a fatigue state; When the driver's eyes and mouth are not in a fatigue state, it means that the driver is not in a fatigue state; On the contrary, it means the driver is in a fatigue state; The process of determining whether the driver's eyes are in a fatigue state is as follows: First, calculating the driver's eye aspect ratio, wherein the driver's eye aspect ratio includes the driver's left eye aspect ratio and the driver's right eye aspect ratio; The expression of the driver's left eye aspect ratio is: Where, P 38 Represents the coordinates of the key point of the outer corner of the left eye, P 42 Indicates the coordinates of the left eye's inner corner key point, P 39 Represents the coordinates of the key point in the middle of the upper eyelid of the left eye, P 41 Represents the coordinates of the key point in the middle of the lower eyelid of the left eye, P 37 Indicates the coordinates of the left end key point of the left eye, P 40 Represents the coordinates of the right end key point of the left eye; The expression of the driver's right eye aspect ratio is: Where, P 44 Represents the coordinates of the key points of the outer corner of the right eye, P 48 Indicates the coordinates of the key point of the inner corner of the right eye, P 45 Represents the coordinates of the key point in the middle of the upper eyelid of the right eye, P 47 Represents the coordinates of the key point in the middle of the lower eyelid of the right eye, P 43 Indicates the coordinates of the left end key point of the right eye, P 46 The coordinates of the right end key point of the right eye; When the eye aspect ratio of the driver's eyes is less than the set eye fatigue threshold, it indicates that the driver's eyes are in a closed state; the expression of the driver's eye aspect ratio is: Where, EAR represents the driver's eye aspect ratio; Next, calculate the PERCLOS eyelid closure percentage, which is expressed as: Where PERCLOS represents the percentage of eyelid closure, N represents the total number of frames of detected driver data, and Nclose represents the number of eye closure frames in the detected driver data; Finally, the driver's blink frequency and the number of consecutive eye closure frames are considered to determine whether the driver's eyes are in a fatigue state. The judgment formula is: Where B represents the actual number of blinks per minute, B threshold represents the blink threshold, C represents the actual number of consecutive eye-closing frames, and C threshold represents the threshold of the number of consecutive eye-closing frames, α represents the weight coefficient, β represents the weight coefficient, F represents the comprehensive judgment function, and Fatigue represents the fatigue judgment condition; When the judgment value of the comprehensive judgment function exceeds 1, or the actual number of continuous eye-closing frames exceeds the continuous eye-closing frame threshold, or the actual number of blinks per minute exceeds the blink threshold, it indicates that the driver's eyes are in a state of fatigue.

9. The method for detecting fatigue driving during dynamic driving according to claim 8, characterized in that: The process of determining whether the driver's mouth is in a fatigue state is as follows: The driver's mouth aspect ratio is calculated using the key point coordinates of the driver. When the driver's mouth aspect ratio exceeds the set mouth fatigue threshold, it indicates that the driver is yawning. The calculation expression of the driver's mouth aspect ratio is: Where MAR represents the driver's mouth aspect ratio, P 61 Indicates the coordinates of the key point on the left end of the mouth, P 65 Indicates the coordinates of the key point on the right end of the mouth, P 64 Represents the coordinates of the key points of the upper lip of the mouth, P 66 Represents the coordinates of the key points of the lower lip of the mouth, P 62 Represents the coordinates of the key points on the inner edge of the upper lip of the mouth, P 68 Represents the coordinates of the key points on the inner edge of the lower lip of the mouth; The yawning frequency per unit time is calculated based on the aspect ratio of the driver's mouth. When the driver's yawning frequency per unit time exceeds the set mouth yawning frequency, it indicates that the driver's mouth is in a fatigue state. The calculation expression for the yawning frequency is: Where K Y Indicates the number of yawning frames per unit time T, K T Then it is the total number of frames in unit time T.

10. A system for detecting fatigue driving during dynamic driving, for implementing a method for detecting fatigue driving during dynamic driving as claimed in any one of claims 1 to 9, characterized in that: include: A data acquisition module, used to acquire an image of the driver to be detected; A face detection module is configured to input a driver image to be detected into a trained improved MTCNN face detection network for driver face detection. The improved MTCNN face detection network embeds an attention (CA) layer in the third layer of the three sub-networks P-Net, R-Net, and O-Net of the original MTCNN face detection network, which is used to improve face recognition accuracy. Furthermore, a batch normalization (BN) module is introduced into the three sub-networks P-Net, R-Net, and O-Net. The key point detection module is used to input the detected driver's face into the trained PFLD face key point model to perform facial key point detection and extract the key point coordinates of the driver's face; The fatigue state judgment module is used to use the key point coordinates of the driver's face to calculate the driver's eye aspect ratio and the driver's mouth aspect ratio respectively, and judge whether the driver is in a fatigue state based on the driver's eye aspect ratio and / or the driver's mouth aspect ratio.

Images