A fatigue detection network based on facial video extraction of blood flow motion features

Abstract

The application provides a fatigue detection network based on blood flow motion feature extraction of a face video, which is composed of a face video preprocessing module, a feature extraction module, a feature constraint processing module and a classifier. The fatigue detection network aims to solve the problems of the existing fatigue detection methods, such as strong invasiveness of a biochemical index method, poor real-time performance of a physiological signal method and weak reliability of a behavior characteristic method.

Description

Technical Field

[0001] This invention relates to artificial intelligence and deep learning, and to an innovative non-invasive technique for measuring physiological signals. Background Technology

[0002] Fatigue is a complex physiological and psychological phenomenon influenced by multiple factors, and it is a widely used concept in health-related disciplines. Physiologists consider fatigue to be an outward manifestation of poor physical condition caused by fatigue, emphasizing short-term effects and its relation to situational factors. The fatigue discussed in this invention primarily refers to physiological fatigue, a subjective feeling experienced by the body after prolonged periods of high-intensity activity. Its characteristics include insufficient energy, physical weakness, and poor concentration, and these adverse symptoms can be alleviated and improved through sufficient rest. Existing research shows a strong correlation between this physiological fatigue and physiological and biochemical indicators, establishing a relatively clear correspondence that can serve as a basis for detecting fatigue, reminding the body to pay attention to its fatigued state, and prompting timely rest and adjustment.

[0003] The key to fatigue detection lies in identifying features that are strongly correlated with fatigue and extracting these features using convenient and efficient methods. From these two perspectives, detection methods based on biochemical indicators and physiological signals rely on discrimination criteria related to the physiological essence of fatigue. These spontaneous bodily reactions are difficult to fake and best objectively reflect the state of fatigue. However, the cost is the need for highly invasive methods to obtain features; expensive detection equipment and complex detection techniques increase detection time and difficulty, sacrificing real-time performance. Detection methods based on behavioral representations use more direct external fatigue manifestations as distinguishing criteria, reducing the difficulty of feature recognition. Convenient feature extraction can be achieved using non-contact, low-cost methods, enabling real-time detection. However, classification accuracy decreases, and application scenarios are limited.

[0004] With the continuous advancement of computer vision technology, the value of image and video data is being explored in greater depth, enabling the extraction of physiological signal patterns from video signals. Therefore, if essential physiological characteristics correlated with fatigue can be extracted non-contactly from video, and then fatigue can be detected based on these physiological characteristics, it would offer the advantages of both reliability and real-time performance. Attempts have already demonstrated the feasibility of this approach; that is, rPPG signals can be obtained by processing pixel signals from each color channel of the facial region in a video using algorithms. Furthermore, the characteristics of rPPG signals can also help distinguish between fatigue and wakefulness. Therefore, this invention aims to analyze blood flow information in facial videos non-contactly using neural networks, and to derive a more universal fatigue detection method based on the differences in blood flow changes under fatigue conditions. Summary of the Invention

[0005] Existing fatigue detection methods have objective limitations. Biochemical and physiological parameter detection methods are highly invasive and require complex equipment, making them difficult to implement in practical applications. Behavioral parameter detection methods have low reliability, poor universality, and are easily limited by specific scenarios. This invention proposes an innovative fatigue detection method. It constructs a network model that extracts physiological signal features from facial video data. By utilizing the constraints of blood flow rhythm and motion characteristics, the network can autonomously capture facial blood flow information. Based on the physiological connection between blood flow and fatigue, effective fatigue detection is achieved.

[0006] The network can be divided into four modules: a face video preprocessing module, a feature extraction module, a feature constraint processing module, and a classifier module. First, the network preprocesses the input video data to obtain a sequence of facial video frames. Next, the facial video sequence passes through the feature extraction module, obtaining feature vectors from multiple different viewpoints. Then, these features are further processed in the feature constraint module, which establishes relationships between various features. This module guides the direction of feature extraction, ensuring similar representations in the feature space at different depths and maintaining distance in feature spaces with significant depth differences. It also establishes a connection between features and physiological pseudo-labels, preserving the physiological rhythmic expression of the features. Finally, the extracted features are further dimensionality-reduced and fused, and then processed by the classifier to obtain the predicted fatigue result. After completing one full network feedforward process on the training data, the network calculates the cross-entropy loss between the predicted result and the real data labels, thereby updating the parameters. Through repeated learning and iterative optimization using videos, the network achieves excellent training results. After training, the model performance is validated. At this point, it is no longer necessary to constrain the intermediate feature vectors of the network again, as the network already has good learning capabilities. The fatigue discrimination result can be obtained simply through the feature extraction module and the classifier. Attached Figure Description

[0007] Figure 1 This is a general structural diagram of the framework proposed in this invention.

[0008] Figure 2 This is the detailed structure of the feature extraction module.

[0009] Figure 3 This is a schematic diagram of a gated memory classifier based on information similarity. Specific implementation methods

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific examples. Figure 1 As shown, the main structure of the network consists of four modules: face video preprocessing module, feature extraction module, feature constraint processing module, and classifier module. The structure and design method of each module are described in detail below.

[0011] First, the facial video preprocessing module.

[0012] This module mainly consists of two parts. On the one hand, it crops the face region from the video to avoid noise interference in non-skin areas. On the other hand, it obtains physiological signal pseudo-labels containing blood volume pulse wave signals from the face video to help constrain features in subsequent network training. These labels cannot be obtained directly from the dataset and need to be obtained through other algorithms.

[0013] Before inputting the raw video into the network, we first perform face recognition on individual frames. This only requires selecting the face region in the first frame of the video; subsequent frames can directly reuse this face bounding box for image cropping. Next, we calculate the difference between adjacent frames to obtain a frame difference sequence for the face video. Frame differences are particularly valuable for motion feature recognition because they are more sensitive to motion. Pixels change at locations where motion occurs between adjacent frames, while the difference is zero at completely still frames, thus quickly distinguishing dynamic features. The frame difference implementation method is simple, adaptable to various lighting conditions, and exhibits strong robustness.

[0014] At the same time, we need to calculate the physiological pseudo-labels corresponding to the videos. The dataset only has simple fatigue classification labels, and it is difficult for us to constrain the physiological performance of features through external physiological signal parameters. As mentioned earlier, the network can obtain rPPG signals from the videos. These signals are highly similar to BVP signals, and blood volume pulse waves contain important physiological characteristics of blood flow. Therefore, they can be used as pseudo-labels for feature learning.

[0015] Second, the feature extraction module.

[0016] This module is a three-branch parallel convolutional neural network that shares parameters. It can guide the direction of network optimization iteration by analyzing the differences in feature performance from different perspectives, achieving efficient feature extraction from various angles. The three branches have similar structural designs. The single feature extraction structure is a convolutional neural network composed of 3D convolutional modules. The network input and output are connected by repeatedly nesting convolutional structures. Batch sample normalization, non-linear activation functions, and dropout operations for hidden neurons are interspersed between the convolutional layers.

[0017] The three networks share similar structures and aim to find blood flow features at different receptive field scales. The characteristic of convolution is that the convolutional kernel preferentially extracts detailed information features within a limited local area each time. If image downsampling and dimensionality reduction are performed using pooling layers, the actual image range perceived by the same convolutional kernel will expand, which is equivalent to the local receptive field gradually expanding, thus obtaining more abstract feature information in positional relationships. We expect the blood flow motion information we obtain to be a small motion with little change in positional relationships, existing only within a local area. Once the receptive field increases, with more spatial variations, the small blood flow motion will be hidden by more obvious motion characteristics, and the network will focus more on macroscopic spatial variation features. The difference between the three networks is that the convolutional receptive field increases from small to large, and the blood flow motion characteristics are gradually hidden. To achieve the difference in receptive field, we control the number of pooling layers, changing the spatial compression of the data. After incorporating pooling operations, the higher the layer of feature extraction, the larger the area it focuses on in the initial image, and the more complex the features obtained.

[0018] In the first structure, to maintain a small receptive field, the image size is not compressed during the entire convolution operation. The convolution kernel can only transmit the feature relationships of a small region at a time. After multiple convolution operations, the local blood flow motion characteristics can be obtained. In the second structure, to slightly expand the receptive field, a pooling layer is added after the first convolutional layer, compressing the image size to half of its original size. Then, the same convolution kernel is used for convolution. In fact, the area that the convolution kernel focuses on has expanded when returning to the original image, but we believe that the current receptive field still contains detailed blood flow motion information. In the third structure, to further expand the receptive field, pooling layers are added after the first, third, and fifth convolutional layers. The image size is gradually reduced. Under the same convolution kernel, the network focuses on the spatiotemporal variation features of a larger area of ​​the face image. This high-dimensional spatiotemporal convolution has covered most of the image. At this time, the network tends to obtain large-scale motion features, which no longer conform to our understanding of blood flow motion characteristics related to fatigue. By adding pooling layers, we can change the feature receptive field size of each branch of the network, realize the extraction of feature information from different perspectives, and use the relationship between features to guide the learning direction of the network. In this process, in addition to adding pooling layers, we also set the types of pooling layers. In the second branch, the receptive field is slightly expanded, but the blood flow motion characteristics still exist in local locations. Max pooling can preserve these local special motion features, select features with better resolution, and improve the saliency of the feature map in the region. The third branch responds better to abstract and complex features under a large receptive field. At this time, average pooling is used for image downsampling, which can ignore the feature expression of the face and only needs to transmit the spatial information completely downward. At the same time, the blood flow information is less obvious in the third branch, enhancing the difference in feature expression from different perspectives.

[0019] Third, the feature constraint processing module.

[0020] The main objectives are twofold: constraining the physiological rhythm of features and processing the relationships between multiple feature branches. After the feature extraction module, the three networks each obtain feature vectors. For the physiological constraints on the features, we apply a fixed-length sliding window to the feature outputs of the first two networks, calculating the cosine similarity of features within each window to obtain the autocorrelation matrix of the features. We aim for this matrix to be similar to the physiological pseudo-labels while preserving the physiological regularity of the feature maps, such as feature aggregation on a diagonal plot and the alternation frequency of the feature maps within a reasonable heart rate range. These constraints can be achieved through a loss function. The reason for this design is based on the prior assumption that we want to obtain minute movements that are blood flow movements, which can be viewed as the transmission of blood volume pulse waves. This feature, even in a single time dimension, maintains synchronous periodicity with the BVP signal. By constraining the physiological rhythm, we distinguish blood flow movement features from other minute movement features. Regarding the relationship between features of different branches, we have already set the following when designing the network structure for feature extraction: the motion characteristics of blood flow can exist in small and medium receptive fields, but will be significantly hidden by macroscopic motion in large receptive fields. Microscopic motion features should be significantly different from macroscopic motion features. Therefore, we designed a loss function to establish a push-pull relationship between the feature information of the three branches, so that the features of the first and second branches remain similar, and the features of the two branches and the third branch are different.

[0021] Fourth, the classifier module.

[0022] We further process the features from the first two branches and input them into the classifier, ultimately establishing a robust relationship between blood flow characteristics and fatigue. First, the feature vectors containing blood flow characteristics from the first two branches output by the feature extraction module are subjected to max pooling to reduce spatial resolution. After spatial dimensionality reduction, salient features are preserved without interfering with the extraction of blood flow characteristics in the preceding spatiotemporal dimensions. The classifier introduces an IGM structure, a classification method proposed by Zeng et al. that can replace traditional fully connected layer structures. Its structure diagram is shown below. Figure 3 As shown in the diagram. This structure introduces an intermediate memory variable. The initial value of the memory variable is a random variable. During training, the IBS similarity between the input features and the memory variable is calculated. If the two variables are similar, the channel energy and feature direction of the projection are the same; if the two variables are dissimilar, the feature directions are completely opposite. The specific formula for measuring feature similarity is as follows:

[0023]

[0024]

[0025] Where X and Y represent the encoding results of two feature vectors, IBSsub represents the similarity of the vectors, and IBSplus represents the dissimilarity of the vectors. IBSplus is used to normalize IBSsub, with a value range of 0 to 1, where the closer to 0, the more similar the features are, and the closer to 1, the more dissimilar the features are. We use the similarity calculation result between the feature vector and the memory vector each time as the classification result. The classification features with high similarity are fed back to modify the memory vector, so that the memory vector can well summarize the common features of a specific category. Features that do not belong to this category will have lower similarity with the memory and will be classified into another category after similarity calculation. This method can work in binary classification scenarios. The core of the IGM-based classifier is the gated memory unit. The IGM structure is inspired by the human brain's memory process. According to the law of human learning new things based on experience, it calculates the similarity between new information and known information to optimize the known information and better understand the new information, thereby improving the classification effect. This structure has achieved better classification results than traditional fully connected layer classifiers on some medical datasets, mainly relying on the outstanding ability of IGM to analyze fine-grained and complex features, and it can also summarize good feature commonalities on small sample datasets. Considering that blood flow motion characteristics also have high similarity and subtle differences, and that the fatigue dataset has a relatively small sample size, we chose IGM as the classifier for the network.

Claims

1. A fatigue detection method based on a fatigue detection network, characterized by, The method comprises the following steps: Step 1, a face video preprocessing module is established, which is used for processing an input face video, and outputs a face frame difference sequence and a physiological pseudo label containing a blood volume pulse wave signal; Step 2, a three-branch parallel convolution feature extraction module is established, which is connected with the face video preprocessing module in step 1, and is used for feature extraction on the input face frame difference sequence, and outputs the extracted features; Step 3, a feature constraint processing module is established, which is connected with the three-branch parallel convolution feature extraction module in step 2, and is used for constraint on the extracted three-branch features, and outputs the constrained features; first, the autocorrelation matrix of the first two branches is calculated, and the first two branches are similar to the physiological pseudo label through a loss function; second, the loss function is designed to keep the similarity between the first two branches, and the two branches are different from the third branch; Step 4, a classifier is used to obtain a fatigue detection result. 2.The fatigue detection method based on the fatigue detection network according to claim 1, wherein, The face video preprocessing module in step 1 first needs to use an existing face recognition model to perform face recognition on the start frame of the video, and returns a rectangular face region, and all subsequent frames directly reuse the face frame to crop the picture. 3.The fatigue detection method based on the fatigue detection network according to claim 1, characterized in that, After the face region selection operation in step 1, the face video preprocessing module needs to perform a frame difference operation on the video sequence, that is, the difference between each adjacent frame is obtained. 4.The fatigue detection method based on the fatigue detection network according to claim 1, wherein, The face video preprocessing module in step 1 needs to use an existing algorithm to generate a physiological pseudo label containing a blood volume pulse wave signal from the original video.

5. The fatigue detection method based on the fatigue detection network according to claim 1, characterized in that, The feature extraction module in step 2 is composed of three parallel convolutional neural networks sharing parameters, and the difference between the three networks is that the convolution receptive field changes from small to large, and the implementation method is obtained by controlling the number of pooling layers in each network. 6.The fatigue detection method based on the fatigue detection network according to claim 1, wherein, The feature constraint processing module in step 3 first calculates the cosine similarity between the features in each window by sliding the window with a fixed length on the first two branches, obtains the autocorrelation matrix of the first two branches, and designs a loss function to make the autocorrelation matrix of the first two branches similar to the physiological pseudo label; second, the constraint relationship between the three branches is established, and the loss function is used to keep the similarity between the first two branches, and the two branches are different from the third branch.

7. The fatigue detection method based on the fatigue detection network according to claim 1, characterized in that, The classifier module in step 4 only takes the first two branches as input, and introduces an IGM structure for the design of the classifier.

Images