Facial feature and Bi-LSTM-based intelligent rail driver fatigue detection method

Abstract

The invention relates to the technical field of fatigue detection, and discloses an intelligent rail driver fatigue detection method based on facial features and Bi-LSTM, and the method comprises the steps: obtaining an intelligent rail driver fatigue detection video frame, and extracting a facial key point feature vector; calculating facial features including an eye aspect ratio, a mouth aspect ratio, a head pitch angle, a head yaw angle and a head roll angle according to the facial key point feature vectors; constructing a facial feature time sequence according to the facial features in the continuous video frames, and performing multi-scale time feature extraction and time sequence signal compression coding on the facial feature time sequence to obtain a facial spatial-temporal feature time sequence; and performing intelligent rail driver fatigue state classification according to the facial spatial-temporal feature time sequence. According to the method, the eye aspect ratio, the mouth aspect ratio and the head posture index are comprehensively considered, the current state of the driver is determined through the relation between the continuous frame features, and the accuracy and the real-time performance of intelligent rail driver fatigue detection can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fatigue detection, in particular to a smart rail driver fatigue detection method based on facial features and Bi-LSTM. BACKGROUND

[0002] With the continuous expansion of smart rail travel demand and operation scale, its large-scale application in the field of public transportation is facing new safety challenges. According to statistical data, human factors account for 82.3% of the total number of transportation accidents during the operation of the smart rail system, and the safety hazards caused by the fatigue state of the driver are particularly prominent. Fatigue state will have a negative impact on the driver's alertness, perception ability, decision-making ability and other aspects, and is extremely easy to cause traffic accidents. Research shows that the degree of damage caused by the fatigue state of the driver is 3.3 times that of the sober state, therefore, it is of great significance to study fatigue detection methods for improving the safety of smart rail driving.

[0003] The fatigue detection method is an important link in the fatigue detection of smart rail drivers. On the one hand, timely identification of fatigue or distraction state can complete sound and light reminders or forced braking in combination with the alarm system to avoid accidents such as collision and deviation from the track caused by human error, and ensure driving safety. On the other hand, the system continuously collects driving behavior data to provide a basis for optimizing the scheduling system and developing a scientific rotation mechanism, reducing operation delays caused by fatigue accumulation, improving line punctuality and overall operating efficiency, and ensuring the high reliability and all-weather stable service capability of the smart rail system. In the evolution process of technology, the driver fatigue detection system can be divided into the following three categories according to the type of sensor: based on vehicle features, physiological features and facial features. However, the complex driving environment may reduce the accuracy of fatigue detection based on vehicle features. Electroencephalogram (EEG), electrocardiogram (ECG) and other physiological signals are the standard for directly measuring fatigue, but the collection of such physiological features requires sensors to be attached to the skin of the human body, which may cause discomfort to the driver and thus affect the normal driving behavior of the driver. SUMMARY

[0004] In view of the above deficiencies in the prior art, the present application provides a smart rail driver fatigue detection method based on facial features and Bi-LSTM.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0006] A smart rail driver fatigue detection method based on facial features and Bi-LSTM, comprising the following steps:

[0007] Obtain the smart rail driver fatigue detection video frame and extract the facial key point feature vector;

[0008] The face features include eye aspect ratio, mouth aspect ratio, head pitch angle, head yaw angle and head roll angle, which are calculated according to the face key point feature vector;

[0009] The face feature time sequence is constructed according to the face features in the continuous video frames, and multi-scale time feature extraction and time sequence signal compression coding are performed on the face feature time sequence to obtain a face spatio-temporal feature time sequence;

[0010] The fatigue state of the driver is classified according to the face spatio-temporal feature time sequence.

[0011] Further, a heterogeneous scale time encoder is used to perform multi-scale time feature extraction and time sequence signal compression coding on the face feature time sequence.

[0012] Further, the heterogeneous scale time encoder comprises:

[0013] The multi-branch convolutional network, the feature fusion module, the dilated convolution residual network and the one-dimensional maximum pooling module.

[0014] Further, the multi-branch convolutional network comprises:

[0015] The three convolution branches are in parallel, and the channel splicing module is connected to all the convolution branches.

[0016] Further, the dilated convolution residual network comprises:

[0017] The first dilated convolution layer and the second dilated convolution layer are cascaded, and the input feature vector of the first dilated convolution layer is connected in residual with the output feature vector of the second dilated convolution layer.

[0018] Further, the calculation method of the eye aspect ratio according to the face key point feature vector is:

[0019] ;

[0020] Wherein, EAR is the eye aspect ratio, is the Euclidean distance, p1, p4 are the horizontal key points of the eyes, and p2, p3, p5, p6 are the vertical key points of the eyes.

[0021] Further, the calculation method of the mouth aspect ratio according to the face key point feature vector is:

[0022] ;

[0023] Wherein, MAR is the mouth aspect ratio, is the Euclidean distance, M1, M5 are the horizontal key points of the mouth, and M2, M3, M4, M6, M7, M8 are the vertical key points of the mouth.

[0024] Further, the calculation manner of calculating the head pitch angle, the head yaw angle and the head roll angle according to the facial key point feature vector is as follows:

[0025] Supposing that the coordinates of a group of key points on the three-dimensional face model in the model coordinate system are:

[0026] ;

[0027] wherein is the i th three-dimensional space point, are the coordinates of the three-dimensional space point in the X, Y and Z directions in the three-dimensional space respectively, and N is the number of three-dimensional space points;

[0028] The corresponding two-dimensional image plane pixel coordinates are:

[0029] ;

[0030] wherein p i is the i th two-dimensional point on the image plane, are the horizontal and vertical coordinates of the two-dimensional point in the image coordinate system respectively;

[0031] The camera intrinsic matrix is denoted as

[0032] ;

[0033] wherein , are the focal lengths in the horizontal direction and the vertical direction respectively, is the principal point coordinate;

[0034] The projection of each three-dimensional space point P i on the image plane satisfies:

[0035] ;

[0036] wherein s i is a scaling factor, r is a rotation vector, and t is a translation vector;

[0037] The rotation vector r is converted into a rotation matrix R:

[0038] ;

[0039] wherein is the skew-symmetric matrix corresponding to the rotation vector r, and I is a unit matrix;

[0040] According to the components of the rotation matrix , the rotation matrix R is decomposed into Euler angles around the fixed axes of the camera coordinate system:

[0041] ;

[0042] wherein, theta pitch is a head pitch angle, theta roll is a head yaw angle, theta yaw is a head roll angle.

[0043] Further, a fatigue state classification model based on a bidirectional long short-term memory network and a time attention mechanism is adopted to classify the fatigue state of the driver of the intelligent rail according to the time sequence of the facial space-time features.

[0044] Further, the fatigue state classification model based on the bidirectional long short-term memory network and the time attention mechanism comprises:

[0045] a bidirectional long short-term memory network, a time attention network and a classifier;

[0046] The time attention network comprises a linear transformation layer, a normalization layer and a weighted summation module.

[0047] The classifier comprises a full connection layer, an activation layer, a regularization layer and a full connection layer.

[0048] The present application has the following beneficial effects:

[0049] The present application comprehensively considers the eye aspect ratio, the mouth aspect ratio and the head posture index, and determines the current state of the driver through the relationship between the features of consecutive frames, so that the accuracy and real-time performance of the fatigue detection of the driver of the intelligent rail can be significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 Fig. 1 is a flowchart of a fatigue detection method for a driver of an intelligent rail based on facial features and Bi-LSTM;

[0051] Figure 2 Fig. 2 is a framework diagram of a fatigue detection method for a driver of an intelligent rail based on facial features and Bi-LSTM;

[0052] Figure 3 Fig. 3 is a flowchart of a face key point detection process;

[0053] Figure 4 Fig. 4 is a schematic diagram of an eye aspect ratio;

[0054] Figure 5 Fig. 5 is a schematic diagram of a mouth aspect ratio;

[0055] Figure 6 Fig. 6 is a schematic diagram of a head posture change direction. DETAILED DESCRIPTION

[0056] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0057] The present application comprehensively considers the eye aspect ratio, mouth aspect ratio, head posture index, and proposes a smart rail driver fatigue detection algorithm based on facial features and Bi-LSTM. The present application positions the driver's face key points through Dlib, calculates 5 fatigue feature parameters such as EAR, MAR, head posture angle, and then inputs the fatigue feature information into the Bi-LSTM network to establish the time sequence dependence relationship, and finally determines the driving state through the classifier.

[0058] As shown in Figure 1 The smart rail driver fatigue detection method based on facial features and Bi-LSTM provided by the embodiment of the present application comprises the following steps S1 to S4:

[0059] S1, acquiring a smart rail driver fatigue detection video frame, and extracting a facial key point feature vector;

[0060] In an optional embodiment of the present application, step S1 acquires a smart rail driver fatigue detection video, and selects Dlib library to extract driver face fine-grained features. Dlib is a tool library developed based on modern C++, which integrates machine algorithms and tool components, and contains a 68-face key point detection model. The model is trained by millions of faces. When detecting a face, the method estimates the position of the face key point from a sparse pixel intensity data subset through cascaded shape regression, and has the characteristics of high precision and low time consumption, which is suitable for vehicle deployment. When Dlib detects the face of the smart rail driver, the above algorithm can capture the 68 key points of the driver's face in real time, and encode them into a 128-dimensional feature vector representing face information, which is used for subsequent face feature calculation.

[0061] S2, calculating facial features including eye aspect ratio, mouth aspect ratio, head pitch angle, head yaw angle and head roll angle according to the facial key point feature vector;

[0062] In an optional embodiment of the present application, step S2 calculates the geometric-based facial features according to the facial key point feature vector, which specifically includes the eye aspect ratio, the mouth aspect ratio, the head pitch angle, the head yaw angle and the head roll angle.

[0063] Eye Aspect Radio (EAR), which is the ratio of the length to the width of the eye. When the driver's eyes are open, the EAR value is larger, and when the eyes are closed, the EAR value is smaller. Therefore, the EAR value is used as a feature to measure the size of the opening and closing of the driver's eyes to detect whether the driver's eyes are closed.

[0064] First, the driver's face in the video stream is labeled using Dlib to obtain the positioning of the eye key points. Then, the Euclidean distance between the points is obtained using the eye key point coordinate information to calculate the EAR, as shown in the following formula, and the final EAR value is calculated by the same weight of the left eye and the right eye:

[0065] ;

[0066] where EAR is the eye aspect ratio, is the Euclidean distance, p1, p4 are the horizontal key points of the eye, and p2, p3, p5, p6 are the vertical key points of the eye.

[0067] Mouth Aspect Radio (MAR), which is the ratio of the length to the width of the mouth. When the driver yawns, the MAR value is larger, and when the mouth is closed, the MAR value is smaller. Therefore, the MAR value is used as a feature to measure the size of the opening and closing of the driver's mouth to determine whether the driver is yawning.

[0068] Similarly, the positioning of the mouth key points obtained using Dlib is used to calculate the MAR using the Euclidean distance between the points:

[0069] ;

[0070] where MAR is the mouth aspect ratio, is the Euclidean distance, M1, M5 are the horizontal key points of the mouth, and M2, M3, M4, M6, M7, M8 are the vertical key points of the mouth.

[0071] When the driver produces a drowsy action, i.e., nods or tilts his head, the pitch angle or roll angle will instantaneously increase. Therefore, Pitch (head pitch angle, rotation angle around X axis), Yaw (head yaw angle, rotation angle around Y axis), and Roll (head roll angle, rotation angle around Z axis) are used as features to measure the fatigue state of the driver.

[0072] The calculation of the head features is realized by the following steps:

[0073] First, Dlib is used to extract specific 2D points from the image, and a strict 2D-3D correspondence is established between the 2D points and the 3D face model.

[0074] Next, let's denote the coordinates of a set of key points (such as the tip of the nose, the corners of the left and right eyes, the chin, etc.) on a three-dimensional face model in the model coordinate system as:

[0075] ;

[0076] where represents the coordinate vector of the i-th three-dimensional space point (usually a key point in the three-dimensional face model), are the coordinates of the three-dimensional point in the X, Y, Z directions in three-dimensional space, respectively.

[0077] The corresponding two-dimensional image plane pixel coordinates (after normalization by the camera intrinsic parameters) are:

[0078] ;

[0079] where pi represents the two-dimensional point coordinate on the i-th image plane (that is, the position of the key point in the face image), are the horizontal and vertical coordinates of the key point in the image coordinate system, respectively.

[0080] The camera intrinsic parameter matrix is denoted as

[0081] ;

[0082] where , are the focal lengths in the horizontal and vertical directions, respectively, , is the principal point coordinate.

[0083] To estimate the pose of the human head in the camera coordinate system, we need to solve the rotation matrix R and the translation vector t, such that for each model point P i its projection on the image plane satisfies the classical pinhole camera model:

[0084] ;

[0085] where s i is the scaling factor, r is the rotation vector, and t is the translation vector. The above equation forms a nonlinear least squares problem based on N corresponding points, and the solvePnP algorithm in OpenCV can be used to obtain the rotation vector r and t. The rotation vector r can be converted to the rotation matrix R through the Rodrigues formula:

[0086] ;

[0087] where is the vector corresponding to the skew-symmetric matrix:

[0088] ;

[0089] After obtaining the rotation matrix R, the next step is to decompose it into Euler angles around the fixed axes of the camera coordinate system (or world coordinate system). Denote the components of the rotation matrix as:

[0090] ;

[0091] Then, the following can be calculated from the rotation matrix R: 、 、 :

[0092] ;

[0093] wherein, represents the i-th row and j-th column of the rotation matrix R.

[0094] Based on the above method, the time values of the features EAR, MAR, Pitch, Yaw and Roll are obtained, but since different indicators have different dimensions, all obtained feature values need to be standardized to eliminate the dimension difference. In this embodiment, Z-SCORE standardization is used to convert data of different scales into a standard normal distribution with a mean of 0 and a standard deviation of 1, so that the model can learn different dimensional features evenly, ensure the gradient smooth, and further improve the model convergence speed. The conversion formula is as follows:

[0095] ;

[0096] wherein, is the average value of all data points of each feature value, is the standard deviation, is the standardized data.

[0097] S3, constructing a face feature time sequence according to the face features in the continuous video frames, performing multi-scale time feature extraction and time sequence signal compression coding on the face feature time sequence, and obtaining a face space-time feature time sequence;

[0098] In an optional embodiment of the present application, step S3 uses a heterogeneous scale time encoder to perform multi-scale time feature extraction and time sequence signal compression coding on the face feature time sequence.

[0099] The heterogeneous scale time encoder comprises:

[0100] a multi-branch convolutional network, a feature fusion module, a dilated convolution residual network and a one-dimensional maximum pooling module.

[0101] The multi-branch convolutional network comprises:

[0102] three parallel convolution branches and a channel splicing module connected with all the convolution branches.

[0103] The hollow convolution residual network comprises:

[0104] The first hollow convolution layer and the second hollow convolution layer are cascaded, and the input feature vector of the first hollow convolution layer is connected in residual with the output feature vector of the second hollow convolution layer.

[0105] Before performing driving fatigue state classification, the embodiment proposes a heterogeneous scale time encoder, which is used to undertake multi-scale time feature extraction and time sequence signal compression and coding tasks, to provide more effective high-level features for the subsequent Bi-LSTM network model, and is beneficial to improve the overall performance of the driver fatigue detection model. The network architecture details are shown in Table 1.

[0106] Table 1

[0107]

[0108] The heterogeneous scale time encoder of the embodiment takes a time sequence (length 90) containing 5 facial features of EAR, MAR, Pitch, Yaw and Roll as input. First, 3 1D convolution branches are set in parallel, with convolution kernel sizes of 3, 5 and 7 respectively, and output channel numbers of 64, 128 and 256 respectively, to capture multi-scale features of local details (such as instantaneous closed eyes) and long-term context (such as continuous changes in head posture) at the same time. Each branch is followed by a ReLU activation. Subsequently, the outputs of the three branches are spliced in the channel dimension (channel number 64+128+256=448), and then compressed to 256 channels through a 1x1 convolution in the feature fusion module, to retain the original information of each branch while eliminating redundancy. Next, two layers of hollow convolution (aperture 3, expansion rate 2 and 4 respectively) are used to build a larger time receptive field, each followed by BatchNorm and ReLU, and directly added through residual connection between the two, which not only expands the time coverage of the features, but also effectively stabilizes the gradient propagation. Finally, one-dimensional maximum pooling with a step size of 3 is used to compress the sequence length from 90 to 30, obtaining a high-level spatiotemporal feature representation with a channel number of 256 and a time sequence length of 30. This structure significantly enhances the representation ability of driver facial micro-movements and long-period posture changes through multi-scale parallel extraction, fusion and hollow residual operations, providing more abundant and robust spatiotemporal feature inputs for the subsequent Bi-LSTM and attention mechanism classifier, and greatly improving the accuracy and real-time performance of fatigue detection.

[0109] S4, classifying the fatigue state of the driver on the intelligent rail according to the facial spatiotemporal feature time sequence.

[0110] In an optional embodiment of the present application, step S4 adopts a fatigue state classification model based on a bidirectional long short-term memory network and a time attention mechanism to classify the fatigue state of the driver according to the facial spatiotemporal feature time series.

[0111] The fatigue state classification model based on the bidirectional long short-term memory network and the time attention mechanism comprises:

[0112] The bidirectional long short-term memory network, the time attention network and the classifier.

[0113] The time attention network comprises a linear transformation layer, a normalization layer and a weighted summation module.

[0114] The classifier comprises a fully connected layer, an activation layer, a regularization layer and a fully connected layer.

[0115] Since the blinking, yawning or bowing of a normal person is a dynamic continuous process with a duration of about 1-2s, the facial state in a single frame of image cannot accurately determine whether the person is fatigued. Therefore, in the video data, the fatigue state of the driver is reflected in the actions in the stage time. In order to determine the current state of the driver through the relationship between the continuous frame features, the embodiment uses a special type of recurrent neural network (RNN), i.e., a long short-term memory network (LSTM), and introduces an attention mechanism to dynamically adjust the weight of different time steps on the time series data to focus on more important fatigue signals, and finally output the state of the driver as normal or fatigued.

[0116] The fatigue state classification model based on the bidirectional long short-term memory network and the time attention mechanism proposed in the embodiment mainly comprises the following parts, and the network architecture is shown in Table 2.

[0117] Table 2

[0118]

[0119] The long short-term memory network (Long Short-Term Memory, LSTM) is stacked by several recurrent unit structures. Compared with the traditional neurons, the recurrent unit structure adopted by the LSTM has stronger functional flexibility, can store the historical information up to the current time, and uses a gating mechanism to regulate the addition and deletion of information and control the transmission path of information, which enables the entire LSTM network to establish a long-distance time sequence dependency relationship.

[0120] The LSTM processing unit includes three gates, namely the forget gate, the input gate and the output gate. The forget gate controls how much information of the cell state at the previous time needs to be forgotten; the input gate determines how much information of the candidate state at the current time needs to be saved; and the output gate decides how much information of the cell state at the current time needs to be output to the hidden state. The final LSTM cell state accumulates information through linear update, and the hidden state is generated by the output gate and the tanh function, which normalizes the output value to [-1, 1], and the relevant functions are as follows:

[0121] ;

[0122] wherein, is an input vector at the input time step ; , , is a weight of ; , , , is a bias vector; is a sigmoid activation function; is vector element multiplication.

[0123] The conventional LSTM can only obtain information from the previous input data, and cannot use information from future input observations. The bidirectional long short-term memory (Bi-LSTM) as a special RNN is composed of two independent LSTM networks: one part processes data in the forward sequence order, and the other part processes data in the reverse sequence order, and jointly determines the final output value. The input data sequence obtains a series of forward and reverse hidden states through the Bi-LSTM, and the function is as follows:

[0124] ;

[0125] wherein, x t is an input vector at the input time step ; h ft is a hidden state of the forward LSTM; h bt is a hidden state of the backward LSTM; is a forward propagation network; is a backward propagation network; Y t is a merged hidden state of the forward and backward.

[0126] The fatigue state has persistence, such as the lengthening of the closed-eye duration, frequent nodding, and the like, and the characteristics of these fatigue behaviors have certain front and back correlations in the time sequence. The bidirectional long and short term memory network can capture the forward and backward information, and the data information of the future several frames can assist in verifying the decision of the current state, reduce the decision misjudgment, and significantly improve the prediction accuracy of the model.

[0127] The temporal attention module is a resource allocation method simulating human time sequence cognitive behavior. The attention mechanism can establish closer correlation between different time states. The core idea is to focus on the key fragments in the time sequence by adjusting the dynamic weight, and suppress the influence of noise or secondary information on the decision. The time sequence attention module introduced in the scheme has the following technical advantages: on the one hand, by adaptively allocating weights to the features of each time, the contribution of small but key facial movement signals (such as eyelid micro-movement, instantaneous blinking) in feature fusion can be effectively amplified, while frequent but irrelevant background jitter and collection noise are suppressed, thereby improving the accuracy and robustness of feature representation; on the other hand, the lightweight attention calculation only needs a small number of additional parameters and can be calculated in parallel, which can significantly enhance the model's ability to capture long and short time dependencies while ensuring real-time performance. The advantages of the cooperation of Bi-LSTM are as follows: Bi-LSTM is responsible for encoding the global context of the input sequence from the positive and negative directions respectively, and fully expands the time sequence information. The attention module dynamically selects the most discriminative time steps in the output of the bidirectional hidden state, forming a weighted fusion representation. This process not only takes into account the semantic coherence of the whole sequence, but also captures key details, so that the subsequent classifier can determine the fatigue state based on more accurate and discriminative spatio-temporal features, thereby significantly improving the detection accuracy and response speed.

[0128] In the detection of driver fatigue, the bidirectional hidden state can provide more rich feature information for the temporal attention mechanism, which can help focus on key frame information and avoid network attention dispersion caused by information redundancy.

[0129] The model training data of the embodiment is derived from a self-built data set and a public data set YAWDD. The public data set YAWDD contains simulated videos of 57 male volunteers and 50 female volunteers from different ages, races and facial features. The videos record the behaviors of the drivers under different light conditions, such as normal driving, closing eyes, yawning, etc. The self-built data set contains videos of 10 male drivers driving on a simulated driving platform. The images are drawn by software tools such as python.

[0130] The existing method judges the fatigue state through the facial features such as eyes, mouth and the like of the driver, and the algorithm usually uses fixed thresholds to judge the eye and mouth state, and the individual difference of the driver leads to a high misjudgment rate, if the deep learning is used to identify the behaviors such as blinking and yawning, the accuracy is higher, but the above methods only pay attention to the spatial features of the fatigue state, and ignore the time features between frames. Therefore, the network model of multi-feature fusion is designed, the fatigue state is identified by analyzing the space-time features of the face, a fatigue driving detection model with small model, high precision and fast speed is realized, and the fatigue driving detection model is more suitable for the driving scene with high real-time requirement under the condition of ensuring a certain precision.

[0131] The present application is described with reference to flowcharts and / or block diagrams that illustrate the method, apparatus (system) and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0132] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0133] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0134] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used for helping to understand the method of the present application and its core idea; meanwhile, for the ordinary skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed, and the above description should not be understood as the limitation of the present application.

[0135] Those skilled in the art will understand that the examples described herein are for the purpose of understanding the principles of the present application and should be understood as not limiting the scope of protection of the present application. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A method for detecting fatigue of a driver of a smart track based on facial features and Bi-LSTM, characterized in that, The method comprises the following steps: Obtaining a video frame for fatigue detection of a smart rail driver and extracting a facial key point feature vector; Calculating facial features including an eye aspect ratio, a mouth aspect ratio, a head pitch angle, a head yaw angle and a head roll angle according to the facial key point feature vector; Constructing a facial feature time sequence according to the facial features in continuous video frames, performing multi-scale time feature extraction and time sequence signal compression coding on the facial feature time sequence, and obtaining a facial spatiotemporal feature time sequence; Performing fatigue state classification of the smart rail driver according to the facial spatiotemporal feature time sequence. 2.The face feature and Bi-LSTM based tram driver fatigue detection method of claim 1, wherein, The multi-scale time feature extraction and time sequence signal compression coding on the facial feature time sequence are performed by using a heterogeneous scale time encoder. 3.The face feature and Bi-LSTM based tram driver fatigue detection method of claim 2, wherein, The heterogeneous scale time encoder comprises: A multi-branch convolutional network, a feature fusion module, a dilated convolution residual network and a one-dimensional maximum pooling module.

4. The method according to claim 3, wherein, The multi-branch convolutional network comprises: Three parallel convolution branches and a channel concatenation module connected with all the convolution branches.

5. The method according to claim 4, wherein, The dilated convolution residual network comprises: A first dilated convolution layer and a second dilated convolution layer are cascaded, and an input feature vector of the first dilated convolution layer is connected in residual with an output feature vector of the second dilated convolution layer.

6. The method according to claim 1, wherein the method is characterized by, The calculation method for calculating the eye aspect ratio according to the facial key point feature vector is: ； wherein EAR is the eye aspect ratio, is the Euclidean distance, p1, p4 are the eye horizontal key points, and p2, p3, p5, p6 are the eye vertical key points.

7. The method according to claim 1, wherein the method is characterized by, The calculation method for calculating the mouth aspect ratio according to the facial key point feature vector is: ； wherein MAR is a mouth aspect ratio, is the Euclidean distance, M1, M5 are the horizontal key points of the mouth, and M2, M3, M4, M6, M7, M8 are the vertical key points of the mouth. 8.The method of claim 1, wherein, The calculation method for calculating the head pitch angle, the head yaw angle and the head roll angle according to the facial key point feature vector is: Let the coordinates of a group of key points on a three-dimensional face model in a model coordinate system be: ； wherein is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i-th three-dimensional spatial point, is the i The corresponding two-dimensional image plane pixel coordinates are: ； where p i is a two-dimensional point on the i-th image plane, are the horizontal and vertical coordinates of the two-dimensional point in the image coordinate system, respectively. The camera intrinsic parameter matrix is denoted as ； wherein , are the focal lengths in horizontal and vertical direction, respectively, is the principal point coordinate; Each three-dimensional space point P i The projection on the image plane satisfies: ； where s i is a scaling factor, r is a rotation vector, and t is a translation vector. The rotation vector r is converted into a rotation matrix R: ； wherein is the skew-symmetric matrix corresponding to the rotation vector r, and I is the identity matrix; According to the components of the rotation matrix Decompose the rotation matrix R into Euler angles about fixed axes of the camera coordinate system: ； where θ pitch is the head pitch angle, θ roll is the head yaw angle, and θ yaw is the head roll angle. 9.The fatigue detection method based on facial features and Bi-LSTM for tram driver according to claim 1, characterized in that, The fatigue state classification model based on the bidirectional long short-term memory network and the time attention mechanism is used to perform fatigue state classification of the smart rail driver according to the facial spatiotemporal feature time sequence.

10. The method according to claim 9, wherein the method is characterized by, The fatigue state classification model based on the bidirectional long short-term memory network and the time attention mechanism comprises: A bidirectional long short-term memory network, a time attention network and a classifier; The time attention network comprises a linear transformation layer, a normalization layer and a weighted summation module; The classifier comprises a full connection layer, an activation layer, a regularization layer and a full connection layer.

Images