A fatigue detection method based on spatiotemporal information fusion

Abstract

The present invention discloses a fatigue detection method based on spatiotemporal information fusion, and belongs to the technical field of fatigue detection. The present invention collects user facial videos through a camera, uses a pre-trained face detection model, performs face detection on each frame of video, and marks key points of the face; calculates the eye aspect ratio and rPPG signal as timing information based on the key points of the face, selects key frames as spatial information, uses SSIM structural similarity as a measurement index, and selects key frames by gradient descent; constructs a fatigue detection model based on a spatiotemporal fusion neural network, sets model parameters for training, and inputs timing information and spatial information into corresponding modules; fuses the features extracted by the fatigue detection model, maps them to a probability space, realizes fatigue detection through a classifier, and compares the output value with the cutoff point according to the set cutoff point of the fatigue state to obtain a fatigue detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue detection, and in particular to a fatigue detection method based on spatiotemporal information fusion. Background Art

[0002] Fatigue is a decline in physical function caused by long hours of continuous work. After a long period of work, individuals generally experience physical fatigue and mental fatigue, which in turn affects decision-making ability and reaction speed. Due to factors such as insufficient sleep time and complex work, fatigue is more likely to occur, which in turn affects work safety.

[0003] Existing fatigue detection technologies are often limited to a single feature. For example, computer vision-related technologies are used to calculate eyelid closure, yawning and other fatigue features and use certain thresholds to issue warnings or use sensors to detect fatigue. Traditional detection methods have two main deficiencies: First, the accuracy of single-dimensional discrimination is not stable enough. For example, when a user wears glasses, noise is easily introduced based on the relevant features of the eyes; second, the threshold-based fatigue judgment standard has poor adaptability. Different users may have different physiological and behavioral characteristics in a fatigued state. At this time, using a fixed threshold may lead to misjudgment or missed judgment. Summary of the invention

[0004] The purpose of the present invention is to provide a fatigue detection method based on spatiotemporal information fusion to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A fatigue detection method based on spatiotemporal information fusion, the method comprising the following steps:

[0007] S100, collecting user facial videos through a camera, using a pre-trained face detection model in the Dlib library, performing face detection on each frame of video, marking facial key points, and storing the coordinates of the facial key points;

[0008] S200, calculating the eye aspect ratio and rPPG signal as timing information based on the key points of the face, selecting key frames as spatial information, using SSIM structural similarity as a measurement indicator, and selecting key frames by gradient descent;

[0009] S300, constructing a fatigue detection model based on a spatiotemporal fusion neural network, dividing a training set and a validation set, setting model parameters for training, using time series information as input data of a time series feature extraction module, and using space information as input data of a space feature extraction module;

[0010] S400, the features extracted from the fatigue detection model are fused through concat, mapped to the probability space through the Sigmoid function, fatigue detection is implemented through the classifier, and the output value of the Sigmoid function is compared with the cutoff point according to the set cutoff point of the fatigue state to obtain the fatigue detection result.

[0011] According to step S100, a camera is used to capture a user's facial video, and the video recording is performed continuously to capture facial expressions and movements at different time points. The face is detected and the key points of the face are marked by Dlib; the video frame is converted into a grayscale image, and image denoising and smoothing are performed; the pre-trained face detection model in the Dlib library is used to perform face detection on each frame of the video, and the Dlib face detector uses directional gradient histogram features and a linear classifier to detect the face position; the detected face area is represented by a rectangular box, and the face position detected in each frame is recorded;

[0012] In the detected face area, Dlib's pre-trained face key point detection model is used to mark the key points of the face. The face parts include eyebrows, eyes, nose, mouth and facial contour. Each face part has several face key points; the coordinates of the face key points detected in each frame of video are stored in the database. Dlib is a C++-based toolbox that provides a pre-trained model to mark 68 key points of the face, including 5 key points on the left and right of the eyebrows, 6 key points on the left and right of the eyes, 9 key points on the nose, 20 key points on the mouth, and 17 key points on the facial contour. According to these key point coordinates, the ROI area position can be anchored in the video.

[0013] Step S200 includes:

[0014] S201, select the eye aspect ratio EAR as a feature of fatigue detection, and calculate it through the coordinates of the key points of the face. The calculation formula is as follows:

[0015]

[0016] in, The function represents the calculation of the Euclidean distance between two points. It is the coordinates of the upper and lower key points of the first eye, corresponding to the 38th point of Dlib. It is the coordinates of the upper and lower key points of the second eye, corresponding to the 42nd point of Dlib. It is the coordinates of the upper and lower key points of the third eye, corresponding to the 39th point of Dlib. It is the coordinates of the upper and lower key points of the fourth eye, corresponding to the 41st point of Dlib. It is the left and right eye corner coordinates of the first eye, corresponding to the 37th point of Dlib. The left and right eye corner coordinates of the second eye, corresponding to the 40 point position of Dlib;

[0017] S202, since the cheek area has dense blood vessels, the heart rate information is extracted by using the mean value of the green channel pixels in the cheek area as the rPPG signal, and the rPPG signal is band-pass filtered using a Butterworth filter, and the upper and lower cut-off frequencies of the band-pass filter are set according to the frequency range of the heartbeat;

[0018] S203. Select several key frames as spatial information. On the one hand, the similarity between adjacent frames is very high, so there is often a lot of redundant information. By selecting key frames, the model can be prevented from being interfered by redundant information. The key frames can retain these key moments to ensure that the model captures the most important features related to fatigue. On the other hand, the fatigue detection task requires strong real-time performance, and deep neural network reasoning on videos will result in a large amount of computation. SSIM structural similarity is used as a measurement indicator, and key frames are selected through gradient descent.

[0019] According to step S203, SSIM is used to measure the similarity between two images. The calculation formula of SSIM is as follows:

[0020]

[0021] in, is the first image, is the second image, and are the pixel averages of the first and second images, respectively. and are the variances of the first and second images respectively, is the covariance, and They are the first constant and the second constant used to maintain the stability of the algorithm. When the two pictures are exactly the same, ;

[0022] For the captured videos, they are uniformly cropped to the same size, converted to grayscale images, and the SSIM index of two adjacent frames is calculated. Based on the corresponding gradient when the SSIM decreases, the first A frames with the largest absolute value of the gradient are selected and stacked in sequence into a vector of A channel as spatial information, where A is a positive integer.

[0023] Step S300 includes:

[0024] S301, the training data of the fatigue detection model comes from the UTA-RLDD database, the data set is divided into a training set and a validation set in proportion, several experimental subjects are selected as training data, several experimental subjects are selected as test data, the length of the time window is set, several frames are selected as training data, and data preprocessing is performed; the model learning rate, training rounds and batch size are set, the optimizer selects the stochastic gradient descent SGD optimization algorithm, and the L2 regularization coefficient and momentum coefficient are set;

[0025] The UTA-RLDD database is a dataset for user fatigue detection research. The dataset was collected and publicly released by the University of Texas at Arlington. It aims to provide a real-life user fatigue detection data resource to help researchers develop and verify fatigue detection models. The database is the largest open source fatigue detection database to date, containing 30 hours of experimental data from 60 experimental subjects.

[0026] S302, using the eye aspect ratio and the rPPG signal as input data of a temporal feature extraction module, wherein the temporal feature extraction module integrates a 1D-ResNet convolutional neural network model of a Transformer encoder; the eye aspect ratio and the rPPG signal are independently subjected to feature extraction through ResNet, and temporal features are fused through a Transformer module;

[0027] 1D-ResNet is a convolutional neural network model specifically designed for processing one-dimensional time series data. It is based on the ResNet residual network architecture and solves the gradient vanishing problem in deep networks by introducing residual connections. The input information is directly passed to the subsequent layers through residual connections, which alleviates the gradient vanishing problem in deep networks and improves the training effect and convergence speed of the model.

[0028] S303: Using the spatial information as input data of a spatial feature extraction module, the spatial feature extraction module uses a two-dimensional convolutional layer to extract features.

[0029] According to step S301, the SGD optimization algorithm formula is as follows:

[0030]

[0031] in, and are the parameter weights at time t+1 and t respectively, lr is the model learning rate, is the L2 regularization coefficient, is the momentum coefficient, is the exponential sliding average of the gradient direction at time t, is the parameter gradient at time t;

[0032] The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows:

[0033]

[0034] in, is the logical value of the original output of the model, is the true label, represents the Sigmoid function, N is the batch size, is the model prediction value of the ith sample, is the actual label of the i-th sample, which takes a value of 0 or 1.

[0035] According to step S302, the input data is subjected to low-level feature extraction through a one-dimensional convolution layer, and the BatchNorm layer performs standardization processing on the data of each batch, so that the output distribution of the convolution layer remains stable, the model convergence is accelerated and the model overfitting is prevented; a nonlinear transformation is introduced through the ReLU activation function, and the input data is sent to a high-level spatiotemporal feature extraction module composed of a number of 1DConvLayer, each convolution layer in the high-level spatiotemporal feature extraction module is composed of a number of residual convolution modules; the training data passes through each residual convolution module in turn, and the residual convolution module constructs a more abstract representation of the data by stacking a number of one-dimensional convolution operations, which is used to capture the key information in the input data, and strengthen the original features of the input through residual connections; the output data is reduced in dimension and retains the main features through an average pooling strategy;

[0036] 1DConvLayer is a convolutional layer used to process one-dimensional time series data. It extracts local patterns and features from the data by performing convolution operations on the time axis.

[0037] The outputs of the two information branches consisting of the eye aspect ratio and the rPPG signal are spliced ​​into a feature vector and fused through a Transformer encoder; the Transformer encoder based on the multi-head self-attention mechanism performs parallel calculations when processing time series signals to connect the information between any two time points; the Transformer encoder is composed of several layers of attention layers, each layer contains several heads with independent parameters, and the rPPG and EAR features output by the convolutional layer are semantically fused by weighted summation of the self-attention output of each head.

[0038] According to step S303, the spatial information is feature extracted and dimensionally upgraded through the convolution kernel, and high-dimensional features are further extracted through several 2DConvLayer to obtain a spatial feature vector; 2DConvLayer includes several residual convolution blocks, and point-to-point convolution is used in each residual convolution block to perform convolution operations on the channel dimension and dimension upgrade to achieve information fusion between channels and enhance the expression ability of the network; in order to align the outputs of the two modules of time and space, a flattening layer is used to compress the spatial feature vector;

[0039] 2DConvLayer is a convolutional layer used to process two-dimensional image data. It extracts local features and patterns in the image by performing convolution operations in the spatial dimension.

[0040] According to step S400, the outputs of the temporal feature extraction module and the spatial feature extraction module are concatenated into a feature vector through concat, and are linearly transformed through two fully connected layers to output Logits, which are mapped to the probability space through the Sigmoid function; the Sigmoid function formula is as follows:

[0041]

[0042] in, is Logits, e is the base of natural logarithm, the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, and the Logits output by the model are mapped to a probability value between 0 and 1; fatigue detection is achieved through the classifier, and the cutoff point of fatigue state is selected. The test results are analyzed according to the three indicators of accuracy Acc, precision P and recall R. The corresponding formula is as follows:

[0043] ,

[0044] ,

[0045] ,

[0046] Among them, TP means fatigue samples are correctly predicted as fatigue, FP means non-fatigue samples are incorrectly predicted as fatigue, TN means non-fatigue samples are correctly predicted as non-fatigue, and FN means fatigue samples are incorrectly predicted as non-fatigue.

[0047] The concatenated feature vector is input into two layers of fully connected layers for linear transformation. The first layer of fully connected layer maps the input feature vector to a representation of an intermediate dimension, and the nonlinear ability of the model is increased by the activation function ReLU. The second layer of fully connected layer further maps the intermediate representation to the output dimension, which is Logits. Logits are obtained through the output of the fully connected layer. Logits are the original output values ​​that have not been processed by the activation function and are allowed to be any real value.

[0048] According to the set fatigue state cutoff point, the output value of the Sigmoid function is compared with the cutoff point; if the output value is greater than or equal to the cutoff point, it is determined to be a fatigue state; otherwise, it is determined to be a non-fatigue state.

[0049] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0050] The present invention can comprehensively capture the multi-dimensional features of the user's fatigue state. The time series information can reflect the user's fatigue changes over a period of time, and the spatial information can effectively capture the subtle changes in facial expressions.

[0051] The present invention uses 1D-ResNet convolutional neural network and Transformer encoder to extract features and perform time series fusion on eye aspect ratio and rPPG signal, thus improving the model's ability to understand time series data. It uses two-dimensional convolutional layer to extract spatial features, and enhances the network's expression ability through residual convolutional block to achieve efficient spatial feature extraction.

[0052] The present invention strengthens the original features of the input through residual connections, improves the training effect of the model, standardizes the data of each batch, accelerates the convergence of the model and prevents overfitting, introduces nonlinear transformation, and improves the model's ability to express complex features. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0054] Figure 1 It is a schematic diagram of the steps of a fatigue detection method based on spatiotemporal information fusion of the present invention;

[0055] Figure 2 It is a schematic diagram of specific implementation steps of a fatigue detection method based on spatiotemporal information fusion of the present invention;

[0056] Figure 3 It is a confusion matrix diagram of experimental results of a fatigue detection method based on spatiotemporal information fusion of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0059] According to one embodiment of the present invention, Figure 1 A fatigue detection method based on time-space information fusion is shown in a schematic diagram of the steps. A fatigue detection method based on time-space information fusion includes the following steps:

[0060] S100, collecting user facial videos through a camera, using a pre-trained face detection model in the Dlib library, performing face detection on each frame of video, marking facial key points, and storing the coordinates of the facial key points;

[0061] S200, calculating the eye aspect ratio and rPPG signal as timing information based on the key points of the face, selecting key frames as spatial information, using SSIM structural similarity as a measurement indicator, and selecting key frames by gradient descent;

[0062] S300, constructing a fatigue detection model based on a spatiotemporal fusion neural network, dividing a training set and a validation set, setting model parameters for training, using time series information as input data of a time series feature extraction module, and using space information as input data of a space feature extraction module;

[0063] S400, the features extracted from the fatigue detection model are fused through concat, mapped to the probability space through the Sigmoid function, fatigue detection is implemented through the classifier, and the output value of the Sigmoid function is compared with the cutoff point according to the set cutoff point of the fatigue state to obtain the fatigue detection result.

[0064] like Figure 2As shown in the schematic diagram of the specific implementation steps of a fatigue detection method based on spatiotemporal information fusion, according to step S100, a camera is used to capture a user's facial video, the video recording is performed continuously, facial expressions and movements at different time points are captured, and faces are detected and key points of faces are marked through Dlib; the video frames are converted into grayscale images, and image denoising and smoothing are performed; the pre-trained face detection model in the Dlib library is used to detect faces for each frame of video, and the Dlib face detector uses directional gradient histogram features and linear classifiers to detect the face position; the detected face area is represented by a rectangular box, and the face position detected in each frame is recorded;

[0065] In the detected face area, Dlib's pre-trained face key point detection model is used to mark the key points of the face. The face parts include eyebrows, eyes, nose, mouth and facial contour. Each face part has several face key points; the coordinates of the face key points detected in each frame of video are stored in the database. Dlib is a C++-based toolbox that provides a pre-trained model to mark 68 key points of the face, including 5 key points on the left and right of the eyebrows, 6 key points on the left and right of the eyes, 9 key points on the nose, 20 key points on the mouth, and 17 key points on the facial contour. According to these key point coordinates, the ROI area position can be anchored in the video.

[0066] Step S200 includes:

[0067] S201, select the eye aspect ratio EAR as a feature of fatigue detection, and calculate it through the coordinates of the key points of the face. The calculation formula is as follows:

[0068]

[0069] in, The function represents the calculation of the Euclidean distance between two points. It is the coordinates of the upper and lower key points of the first eye, corresponding to the 38th point of Dlib. It is the coordinates of the upper and lower key points of the second eye, corresponding to the 42nd point of Dlib. It is the coordinates of the upper and lower key points of the third eye, corresponding to the 39th point of Dlib. It is the coordinates of the upper and lower key points of the fourth eye, corresponding to the 41st point of Dlib. It is the left and right eye corner coordinates of the first eye, corresponding to the 37th point of Dlib. The left and right eye corner coordinates of the second eye, corresponding to the 40 point position of Dlib;

[0070] S202, extracting heart rate information by taking the mean value of green channel pixels of the cheek area as the rPPG signal, performing band-pass filtering on the rPPG signal using a Butterworth filter, and setting the upper and lower cut-off frequencies of the band-pass filter to 0.5 Hz and 2 Hz according to the frequency range of the heartbeat;

[0071] S203, select 8 key frames as spatial information, use SSIM structural similarity as a measurement indicator, and select key frames through gradient descent; SSIM is used to measure the similarity between two images. Compared with the traditional method of comparing histogram distribution, SSIM can pay more attention to the difference between facial expressions and movements, so it can retain the key picture with the largest information entropy.

[0072] According to step S203, SSIM is used to measure the similarity between two images. The calculation formula of SSIM is as follows:

[0073]

[0074] in, is the first image, is the second image, and are the pixel averages of the first and second images, respectively. and are the variances of the first and second images respectively, is the covariance, and They are the first constant and the second constant used to maintain the stability of the algorithm. When the two pictures are exactly the same, ;

[0075] For the captured videos, they are uniformly cropped to a size of 224×224, converted to grayscale images, and the SSIM index of two adjacent frames is calculated. Based on the corresponding gradient when the SSIM decreases, the first 8 frames with the largest absolute value of the gradient are selected and stacked in sequence into an 8-channel vector as the spatial information.

[0076] Step S300 includes:

[0077] S301, the training data of the fatigue detection model comes from the UTA-RLDD database, the data set is divided into a training set and a validation set at a ratio of 80% and 20%, 48 experimental subjects are selected as training data, 12 experimental subjects are selected as test data, 30 seconds is used as a time window, 900 frames are selected as training data, and data preprocessing is performed; the model learning rate is set to 0.0005, the training round is 500 rounds, the batch size BatchSize is 64, the optimizer selects the stochastic gradient descent SGD optimization algorithm, in order to avoid overfitting, the L2 regularization coefficient is set to 0.01, in order to accelerate the convergence of the model and jump out of the local optimal solution, the momentum coefficient is selected to be 0.8;

[0078] The UTA-RLDD database is a dataset for user fatigue detection research. The dataset was collected and publicly released by the University of Texas at Arlington. It aims to provide a real-life user fatigue detection data resource to help researchers develop and verify fatigue detection models. The database is the largest open source fatigue detection database to date, containing 30 hours of experimental data from 60 experimental subjects.

[0079] S302, using the eye aspect ratio and the rPPG signal as input data of a temporal feature extraction module, wherein the temporal feature extraction module integrates a 1D-ResNet convolutional neural network model of a Transformer encoder; the eye aspect ratio and the rPPG signal are independently subjected to feature extraction through ResNet, and temporal features are fused through a Transformer module;

[0080] 1D-ResNet is a convolutional neural network model specifically designed for processing one-dimensional time series data. It is based on the ResNet residual network architecture and solves the gradient vanishing problem in deep networks by introducing residual connections. The input information is directly passed to the subsequent layers through residual connections, which alleviates the gradient vanishing problem in deep networks and improves the training effect and convergence speed of the model.

[0081] S303: Using the spatial information as input data of a spatial feature extraction module, the spatial feature extraction module uses a two-dimensional convolutional layer to extract features.

[0082] According to step S301, the SGD optimization algorithm formula is as follows:

[0083]

[0084] in, and are the parameter weights at time t+1 and t respectively, lr is the model learning rate, is the L2 regularization coefficient, is the momentum coefficient, is the exponential sliding average of the gradient direction at time t, is the parameter gradient at time t;

[0085] The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows:

[0086]

[0087] in, is the logical value of the original output of the model, is the true label, represents the Sigmoid function, N is the batch size, is the model prediction value of the ith sample, is the actual label of the i-th sample, which takes a value of 0 or 1.

[0088] According to step S302, the input data is subjected to low-level feature extraction through a one-dimensional convolution layer with a convolution kernel size of 3, and the BatchNorm layer performs standardization processing on the data of each batch, so that the output distribution of the convolution layer remains stable, accelerates model convergence and prevents model overfitting; a nonlinear transformation is introduced through the ReLU activation function, and the input data is sent to an advanced spatiotemporal feature extraction module composed of three 1DConvLayers, each convolution layer in the advanced spatiotemporal feature extraction module is composed of three residual convolution modules; the training data passes through each residual convolution module in turn, and the residual convolution module constructs a more abstract and advanced representation of the data by stacking several one-dimensional convolution operations, which is used to capture the key information in the input data and strengthen the original features of the input through residual connections; the output data is reduced in dimension and retains the main features through an average pooling strategy;

[0089] 1DConvLayer is a convolutional layer used to process one-dimensional time series data. It extracts local patterns and features from the data by performing convolution operations on the time axis.

[0090] The outputs of the two information branches consisting of the eye aspect ratio and the rPPG signal are spliced ​​into a feature vector and fused through the Transformer encoder; the Transformer encoder based on the multi-head self-attention mechanism performs parallel calculations when processing time series signals to connect the information between any two time points; the Transformer encoder consists of 4 attention layers, each layer contains 8 heads with independent parameters, and the rPPG and EAR features output by the convolution layer are semantically fused by weighted summation of the self-attention output of each head. The output and parameter quantity of each layer of the time series feature extraction module are shown in Table 1. The shape table of each layer output of the time series feature extraction module:

[0091] Table 1 Output shape table of each layer of temporal feature extraction module

[0092]

[0093] According to step S303, the 8×224×224 spatial information is feature extracted and dimensionally upgraded through a 3×3 convolution kernel, and high-dimensional features are further extracted through three 2DConvLayers to obtain a spatial feature vector; 2DConvLayer contains several residual convolution blocks, and a 1×1 point-to-point convolution is used in each residual convolution block to perform a convolution operation on the channel dimension and dimensionally upgrade to achieve information fusion between channels and enhance the expression ability of the network; in order to align the outputs of the two modules of time and space, a flattening layer is used to compress the spatial feature vector; the outputs and parameter quantities of each layer of the spatial feature extraction module are shown in Table 2. The output shape table of each layer of the spatial feature extraction module:

[0094] Table 2 Output shape table of each layer of spatial feature extraction module

[0095]

[0096] 2DConvLayer is a convolutional layer used to process two-dimensional image data. It extracts local features and patterns in the image by performing convolution operations in the spatial dimension.

[0097] According to another embodiment of the present invention, Figure 3 The confusion matrix diagram of the experimental results of a fatigue detection method based on spatiotemporal information fusion is shown in FIG. According to step S400, the outputs of the temporal feature extraction module and the spatial feature extraction module are concatenated into a 128-dimensional feature vector through concat, and are linearly transformed through two fully connected layers to output Logits, which are mapped to the probability space through the Sigmoid function; the Sigmoid function formula is as follows:

[0098] ,

[0099] in, is Logits, e is the base of natural logarithm, the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, and the Logits output by the model are mapped to a probability value between 0 and 1; fatigue detection is achieved through the classifier, and 0.5 is selected as the dividing point of fatigue state. The test results are analyzed according to the three indicators of accuracy Acc, precision P and recall R. The corresponding formula is as follows:

[0100] ,

[0101] ,

[0102] ,

[0103] Among them, TP means fatigue samples are correctly predicted as fatigue, FP means non-fatigue samples are incorrectly predicted as fatigue, TN means non-fatigue samples are correctly predicted as non-fatigue, and FN means fatigue samples are incorrectly predicted as non-fatigue.

[0104] The concatenated feature vector is input into two layers of fully connected layers for linear transformation. The first layer of fully connected layer maps the input feature vector to a representation of an intermediate dimension, and the nonlinear ability of the model is increased by the activation function ReLU. The second layer of fully connected layer further maps the intermediate representation to the output dimension, which is Logits. Logits are obtained through the output of the fully connected layer. Logits are the original output values ​​that have not been processed by the activation function and are allowed to be any real value.

[0105] According to the set fatigue state cutoff point, the output value of the Sigmoid function is compared with the cutoff point; if the output value is greater than or equal to the cutoff point, it is judged as fatigue state; otherwise, it is judged as non-fatigue state. The test results are accuracy rate of 89.84%, recall rate of 91.87, and precision rate of 88.28%.

[0106] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0107] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fatigue detection method based on spatiotemporal information fusion, characterized in that: The method comprises the following steps: S100, collecting user facial videos through a camera, using a pre-trained face detection model in the Dlib library, performing face detection on each frame of video, marking facial key points, and storing the coordinates of the facial key points; S200, calculating the eye aspect ratio and rPPG signal as timing information based on the key points of the face, selecting key frames as spatial information, using SSIM structural similarity as a measurement indicator, and selecting key frames by gradient descent; Step S200 includes: S201, select the eye aspect ratio EAR as a feature of fatigue detection, and calculate it through the coordinates of the key points of the face. The calculation formula is as follows: ； in, The function represents the calculation of the Euclidean distance between two points. is the coordinates of the upper and lower key points of the first eye, are the coordinates of the upper and lower key points of the second eye, are the coordinates of the upper and lower key points of the third eye, It is the coordinates of the upper and lower key points of the fourth eye. is the left and right eye corner coordinates of the first eye, is the left and right eye corner coordinates of the second eye; S202, extracting heart rate information by taking the mean value of green channel pixels in the cheek area as the rPPG signal, performing band-pass filtering on the rPPG signal using a Butterworth filter, and setting the upper and lower cut-off frequencies of the band-pass filter according to the frequency range of the heartbeat; S203, selecting several key frames as spatial information, using SSIM structural similarity as a measurement indicator, and selecting key frames by gradient descent; According to step S203, SSIM is used to measure the similarity between two images. The calculation formula of SSIM is as follows: ； in, is the first image, is the second image, and are the pixel averages of the first and second images, respectively. and are the variances of the first and second images respectively, is the covariance, and They are the first constant and the second constant used to maintain the stability of the algorithm. When the two pictures are exactly the same, ; For the collected videos, they are uniformly cropped to the same size, converted to grayscale images, and the SSIM index of two adjacent frames is calculated. Based on the corresponding gradient when the SSIM decreases, the first A frames with the largest absolute value of the gradient are selected and stacked into a vector of A channels as spatial information, where A is a positive integer; S300, constructing a fatigue detection model based on a spatiotemporal fusion neural network, dividing a training set and a validation set, setting model parameters for training, using time series information as input data of a time series feature extraction module, and using space information as input data of a space feature extraction module; S400, the features extracted from the fatigue detection model are fused through concat, mapped to the probability space through the Sigmoid function, fatigue detection is implemented through the classifier, and the output value of the Sigmoid function is compared with the cutoff point according to the set cutoff point of the fatigue state to obtain the fatigue detection result.

2. The fatigue detection method based on spatiotemporal information fusion according to claim 1 is characterized in that: According to step S100, a camera is used to capture a user's facial video, the video recording is performed continuously, facial expressions and movements at different time points are captured, and faces are detected and key points of faces are marked through Dlib; Convert video frames into grayscale images and perform image denoising and smoothing; Use the pre-trained face detection model in the Dlib library to perform face detection on each frame of video. Dlib's face detector uses directional gradient histogram features and linear classifiers to detect the face position. The detected face area is represented by a rectangular box, and the face position detected in each frame is recorded. In the detected face area, Dlib's pre-trained face key point detection model is used to annotate the key points of the face. The face parts include eyebrows, eyes, nose, mouth and facial contours. Each face part has several face key points. The coordinates of the face key points detected in each frame of video are stored in the database.

3. The fatigue detection method based on spatiotemporal information fusion according to claim 1 is characterized in that: Step S300 includes: S301, the training data of the fatigue detection model comes from the UTA-RLDD database, the data set is divided into a training set and a validation set in proportion, several experimental subjects are selected as training data, several experimental subjects are selected as test data, the length of the time window is set, several frames are selected as training data, and data preprocessing is performed; the model learning rate, training rounds and batch size are set, the optimizer selects the stochastic gradient descent SGD optimization algorithm, and the L2 regularization coefficient and momentum coefficient are set; S302, using the eye aspect ratio and the rPPG signal as input data of a temporal feature extraction module, wherein the temporal feature extraction module integrates a 1D-ResNet convolutional neural network model of a Transformer encoder; the eye aspect ratio and the rPPG signal are independently subjected to feature extraction through ResNet, and temporal features are fused through a Transformer module; S303: Using the spatial information as input data of a spatial feature extraction module, the spatial feature extraction module uses a two-dimensional convolutional layer to extract features.

4. The fatigue detection method based on spatiotemporal information fusion according to claim 3 is characterized in that: According to step S301, the SGD optimization algorithm formula is as follows: ； in, and are the parameter weights at time t+1 and t respectively, lr is the model learning rate, is the L2 regularization coefficient, is the momentum coefficient, is the exponential sliding average of the gradient direction at time t, is the parameter gradient at time t; The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows: ； in, is the logical value of the original output of the model, is the true label, represents the Sigmoid function, N is the batch size, is the model prediction value of the ith sample, is the actual label of the i-th sample, which takes a value of 0 or 1.

5. The fatigue detection method based on spatiotemporal information fusion according to claim 3 is characterized in that: According to step S302, the input data is subjected to low-level feature extraction through a one-dimensional convolution layer, and the BatchNorm layer performs standardization processing on the data of each batch, so that the output distribution of the convolution layer remains stable, the model convergence is accelerated and the model overfitting is prevented; a nonlinear transformation is introduced through the ReLU activation function, and the input data is sent to a high-level spatiotemporal feature extraction module composed of a number of 1DConvLayer, each convolution layer in the high-level spatiotemporal feature extraction module is composed of a number of residual convolution modules; the training data passes through each residual convolution module in turn, and the residual convolution module constructs a more abstract representation of the data by stacking a number of one-dimensional convolution operations, which is used to capture the key information in the input data, and strengthen the original features of the input through residual connections; the output data is reduced in dimension and retains the main features through an average pooling strategy; The outputs of the two information branches consisting of the eye aspect ratio and the rPPG signal are spliced ​​into a feature vector and fused through a Transformer encoder; the Transformer encoder based on the multi-head self-attention mechanism performs parallel calculations when processing time series signals to connect the information between any two time points; the Transformer encoder is composed of several layers of attention layers, each layer contains several heads with independent parameters, and the rPPG and EAR features output by the convolutional layer are semantically fused by weighted summation of the self-attention output of each head.

6. The fatigue detection method based on spatiotemporal information fusion according to claim 3 is characterized in that: According to step S303, the spatial information is feature extracted and dimensionally upgraded through the convolution kernel, and high-dimensional features are further extracted through several 2DConvLayer to obtain a spatial feature vector; 2DConvLayer contains several residual convolution blocks, and point-to-point convolution is used in each residual convolution block to perform convolution operations on the channel dimension and dimension upgrade to achieve information fusion between channels and enhance the expression ability of the network; in order to align the outputs of the two time and space modules, a flattening layer is used to compress the spatial feature vector.

7. The fatigue detection method based on spatiotemporal information fusion according to claim 1 is characterized in that: According to step S400, the outputs of the temporal feature extraction module and the spatial feature extraction module are concatenated into a feature vector through concat, and are linearly transformed through two fully connected layers to output Logits, which are mapped to the probability space through the Sigmoid function; the Sigmoid function formula is as follows: ； in, is Logits, e is the base of natural logarithm, the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, and the Logits output by the model are mapped to a probability value between 0 and 1; fatigue detection is achieved through the classifier, and the cutoff point of fatigue state is selected. The test results are analyzed according to the three indicators of accuracy Acc, precision P and recall R. The corresponding formula is as follows: ； ； ； Among them, TP means fatigue samples are correctly predicted as fatigue, FP means non-fatigue samples are incorrectly predicted as fatigue, TN means non-fatigue samples are correctly predicted as non-fatigue, and FN means fatigue samples are incorrectly predicted as non-fatigue.

8. The fatigue detection method based on spatiotemporal information fusion according to claim 7 is characterized in that: The concatenated feature vector is input into two layers of fully connected layers for linear transformation. The first layer of fully connected layer maps the input feature vector to a representation of an intermediate dimension, and the nonlinear ability of the model is increased by the activation function ReLU. The second layer of fully connected layer further maps the intermediate representation to the output dimension, which is Logits. Logits are obtained through the output of the fully connected layer. Logits are the original output values ​​that have not been processed by the activation function and are allowed to be any real value. According to the set fatigue state cutoff point, the output value of the Sigmoid function is compared with the cutoff point; if the output value is greater than or equal to the cutoff point, it is determined to be a fatigue state; Otherwise it is judged as a non-fatigue state.

Images