Fatigue detection network for extracting blood flow motion features based on face video

Abstract

The invention provides a fatigue detection network for extracting blood flow motion features based on a face video. The network is composed of a face video preprocessing module, a feature extraction module, a feature constraint processing module and a classifier. The objective of the invention is to solve the problems of strong invasion of a biochemical index method, poor real-time performance of a physiological signal method, weak reliability of a behavior characterization method and the like in an existing fatigue detection method.

Description

Technical Field

[0001] The present invention involves artificial intelligence, deep learning, and innovations in measuring physiological signals using non-invasive technology. Background Art

[0002] Fatigue is a complex physiological and psychological phenomenon affected by multiple factors. It is a concept widely used in health-related disciplines. Physiologists believe that fatigue is a poor performance of an individual's external appearance. It is due to insufficient physical function caused by fatigue, emphasizing short-term and situational factors. The fatigue discussed in this invention mainly refers to physiological fatigue, which is a subjective feeling of the human body after a long period of high-intensity action. Its characteristics include insufficient energy, physical fatigue, lack of concentration, etc. These adverse symptoms can be alleviated and improved through adequate rest. Existing studies have shown that there is a strong correlation between this physiological fatigue and physiological and biochemical indicators, and a clearer corresponding relationship can be established, which can serve as a basis for detecting fatigue, reminding the human body to pay attention to the fatigued physical state, and to rest and adjust in time.

[0003] The key to fatigue detection is to find features that can establish a strong correlation with fatigue, and to extract these features using convenient and efficient means. From these two aspects, the detection methods based on biochemical indicators and physiological signals rely on the criteria that are related to the physiological nature of fatigue. This spontaneous reaction of the body is difficult to forge and can most objectively and truly reflect the state of fatigue, but the cost is that it requires highly invasive methods to obtain features. Expensive detection equipment and complex detection methods increase the time and difficulty of detection, and lose the real-time nature of fatigue detection; the detection method based on behavioral representation uses more direct external fatigue manifestations as the distinction standard, the difficulty of feature recognition is reduced, and a non-contact, low-cost method can be used to complete convenient feature extraction, which can achieve real-time effectiveness of detection, but the accuracy of classification is reduced and is limited by the usage scenario.

[0004] With the continuous advancement of computer vision technology, the value of image and video data has been deeply explored, and it is possible to obtain the physiological signal patterns of the human body from video signals. Therefore, if the physiological essential characteristics that can be correlated with fatigue are extracted from the video non-contactly, and then fatigue is detected by the physiological characteristics, this can have the advantages of both reliability and real-time performance. There have been attempts to prove that this idea is feasible, that is, the rPPG signal can be obtained after the pixel signals of each color channel in the face area of ​​the video are processed by an algorithm. At the same time, the characteristics of the rPPG signal can also help distinguish between fatigue and wakefulness. Therefore, the present invention intends to analyze the blood flow information in the face video non-contactly through a neural network, and obtain a more universal fatigue detection method from the differences in blood flow changes under fatigue conditions. Summary of the invention

[0005] Existing fatigue detection methods have objective limitations. Biochemical index and physiological parameter detection methods are highly invasive and require complex equipment, making them difficult to implement in actual application scenarios. Behavioral parameter detection methods have low reliability, poor universality, and are easily limited by scenarios. This invention mainly proposes an innovative fatigue detection method, constructs a network model, extracts physiological signal characteristics of the human body from face video data, and uses the constraints of blood flow rhythm characteristics and motion characteristics to enable the network to autonomously capture facial blood flow information, and then based on the physiological connection between blood flow and fatigue, achieves effective fatigue detection.

[0006] The network as a whole can be divided into four modules, namely, face video preprocessing module, feature extraction module, feature constraint processing module and classifier module. First, the network preprocesses the input video data to obtain the facial video frame sequence; then, the facial video sequence passes through the feature extraction module to obtain feature vectors under multiple different perspectives; then, these features are further processed in the feature constraint module and the connection between multiple features is established, which guides the feature extraction direction, ensures similar expressions in the deep feature space, and maintains the distance in the feature space with large depth differences. At the same time, the features are connected with physiological pseudo-labels to retain the physiological rhythm expression of the features; finally, the extracted features are further dimensionality reduced and fused, and processed by the classifier to obtain the predicted fatigue results. After completing a complete network feedforward process for the training data, the network will calculate the cross entropy loss between the predicted results and the real data labels, and then update the parameters. Through repeated learning and iterative optimization of the video, the network can obtain good training results. After the training is completed, the model performance is verified. At this time, it is no longer necessary to constrain the feature vectors in the middle of the network again, because the network already has good learning ability, and only the feature extraction module and classifier are needed to obtain the fatigue discrimination results. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is the overall structural diagram of the framework proposed by the present invention

[0008] Figure 2 is the detailed structure of the feature extraction module

[0009] Figure 3 This is a schematic diagram of a gated memory classifier based on information similarity. Specific implementation methods

[0011] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific examples. Figure 1 As shown in Figure 1, the main structure of the network is divided into the following four modules: face video preprocessing module, feature extraction module, feature constraint processing module and classifier module. The structure and design method of each module are introduced in detail below.

[0012] First, the face video preprocessing module.

[0013] This module mainly consists of two parts. On the one hand, it crops the face area from the video to avoid noise interference from the non-skin area. On the other hand, it obtains pseudo labels of physiological signals including blood volume pulse wave signals through face videos to help constrain features in subsequent network training. This label cannot be obtained directly from the dataset and needs to be obtained through other algorithms.

[0014] Before inputting the original video into the network, we first perform face recognition on a separate frame image. We only need to perform the face selection operation on the first frame of the video. Each subsequent frame can directly reuse the face frame to crop the image. Then, we calculate the difference between adjacent frames to obtain the frame difference sequence of the face video. Frame difference can play a greater role in motion feature recognition because frame difference is more sensitive to motion performance. The pixels at the position where the motion changes between two adjacent frames change, and the pixel difference is 0 when it is completely still, thereby quickly distinguishing dynamic features. The frame difference implementation method is simple, can adapt to a variety of scene conditions such as lighting, and has strong robustness.

[0015] At the same time, we need to calculate the physiological pseudo-labels corresponding to the video. The data set only has simple fatigue classification labels. It is difficult for us to constrain the physiological performance of the features through external physiological signal parameters. As mentioned earlier, the network can obtain rPPG signals from the video. This signal is highly similar to the BVP signal, and the blood volume pulse wave contains important physiological characteristics of blood flow, so it can be used as a pseudo-label for feature learning.

[0016] Second, feature extraction module.

[0017] This module is a three-branch parallel convolutional neural network with shared parameters. It can guide the direction of network optimization iteration by the difference of feature performance under different perspectives, and realize efficient feature extraction from different perspectives. The three networks have similar structural designs. The single feature extraction structure is a convolutional neural network composed of 3D convolution modules. The network input and output are connected by repeated application of convolution structure. Batch sample normalization, nonlinear activation function, and hidden neuron Dropout operation are interspersed in the middle of the convolution layer.

[0018] The three networks have similar structures and expect to find blood flow features at different receptive field scales. The characteristic of convolution is that the convolution kernel first extracts detailed information features within a limited local range each time. If the image is downsampled and reduced in dimension through the pooling layer, the actual image range perceived by the same convolution kernel will be expanded, which is equivalent to the gradual expansion of the local receptive field, and more abstract feature information in the position relationship can be obtained. The blood flow motion information we expect to obtain is a small motion with little change in the position relationship, which can only exist in a local range. Once the receptive field becomes larger and there are more spatial change relationships, the small blood flow motion will be hidden by more obvious motion characteristics, and the network will focus more on the macroscopic spatial change characteristics. The difference between the three networks is that the convolution receptive field changes from small to large, and the blood flow motion characteristics will gradually be hidden. In order to achieve the difference in receptive field, we control the number of pooling layers and change the spatial compression size of the data. After the pooling operation is connected, the higher the level of feature extraction, the larger the range of attention on the initial image, and the more complex the features obtained.

[0019] In order to keep the receptive field smaller in the first structure, the image size is not compressed during the entire convolution operation, and the convolution kernel can only transmit the feature relationship of a small area each time. After multiple convolution operations, the local blood flow motion characteristics can be obtained; in order to slightly expand the receptive field in the second structure, a pooling layer is added after the first convolution layer to compress the image size to half of the original size, and then the same convolution kernel is used for convolution. In fact, when returning to the original image, the area that the convolution kernel focuses on has been expanded, but we believe that the current receptive field still contains detailed blood flow motion information; in order to further expand the receptive field in the third structure, pooling layers are added after the first, third and fifth convolution layers, and the image size is gradually reduced. Under the same convolution kernel, the network focuses on the spatiotemporal variation characteristics of a larger area of ​​the face image. This high-dimensional spatiotemporal convolution has covered most of the image. At this time, the network is more inclined to obtain large-scale motion features, which no longer conform to the blood flow motion characteristics related to fatigue that we think. After adding the pooling layer, we can change the size of the feature receptive field of each network, extract feature information from different perspectives, and use the relationship between features to reversely guide the learning direction of the network. In this process, in addition to adding pooling layers, we also set the types of pooling layers. The receptive field in the second network is slightly expanded, but the blood flow motion characteristics still exist in the local position. The maximum pooling can retain this special local motion feature, select features with better resolution, and improve the significant expression of feature maps in the region; the third network responds better to abstract and complex features under a large receptive field. At this time, average pooling is used for image downsampling, which can ignore the facial feature expression. It only needs to pass the spatial information completely downward, and at the same time make the blood flow information less obvious in the third branch, enhancing the difference in feature expression from different perspectives.

[0020] Third, feature constraint processing module.

[0021] It mainly achieves two purposes, namely, the physiological rhythmic constraints of features and the relationship processing of multiple features. After the feature extraction module, the three networks obtain feature vectors respectively. For the physiological constraints of features, we slide the feature outputs of the first two networks with a fixed window length, calculate the cosine similarity of each window feature, and obtain the autocorrelation matrix of the features. We expect this matrix to be similar to the physiological pseudo-labels, while retaining the physiological laws of the feature graph, such as feature aggregation on the oblique line graph, and the alternation frequency of the feature graph is within a reasonable heart rate range. These can be constrained by the loss function. The reason for this design is that according to the prior assumption, we hope that the micro-motion we get is blood flow motion, and blood flow motion can be regarded as the conduction of blood volume pulse waves. This feature still maintains the synchronous periodicity with the BVP signal in a single time dimension. By constraining the physiological rhythmicity, we distinguish the blood flow motion features from other micro-motion features. Regarding the relationship processing of different branch features, we have already set it when designing the network structure for extracting features. The motion characteristics of blood flow can exist in small and medium receptive fields, but will be hidden by significant macroscopic motion in large receptive fields. The micro-motion features should be significantly different from the macroscopic motion features. Therefore, we design the loss function to establish a push-pull relationship between the feature information of the three networks, so that the features of the first and second branches remain similar, and the features of these two networks are different from the features of the third branch.

[0022] Fourth, the classifier module.

[0023] We further process the features of the first two branches and input them into the classifier, and finally establish a robust connection between blood flow motion characteristics and fatigue. First, the first two feature vectors containing blood flow motion characteristics output by the feature extraction module are subjected to maximum pooling to reduce the spatial resolution. After spatial dimensionality reduction, significant features are retained and will not interfere with the extraction of blood flow motion features in the previous spatiotemporal dimension. The classifier introduces the IGM structure. IGM is a classification method proposed by Zeng et al. that can replace the traditional fully connected layer structure. Its structure diagram is shown in the figure below. Figure 3 As shown. This structure introduces an intermediate variable of memory, the initial value of which is a random variable. During the training process, the IBS similarity between the input feature and the memory variable is calculated. If the two variables are similar, the energy and feature direction of each channel of the projection are the same. If the two variables are different, the feature directions are completely opposite. The specific calculation formula for measuring the degree of feature similarity is:

[0024]

[0025]

[0026] Among them, X and Y represent the encoding results of two feature vectors, IBSsub represents the similarity of vectors, and IBSplus represents the difference of vectors. IBSsub is normalized by IBSplus, and the value range is 0 to 1, where the closer to 0, the features are similar, and the closer to 1, the features are different. We use the similarity calculation results of each feature vector and memory vector as the classification result, and return the classification features with high similarity to modify the memory vector, so that the memory vector can well summarize the common characteristics of a certain classification. Features that do not belong to this classification will have lower similarity with the memory and be classified into another category after similarity calculation. This method can work in binary classification scenarios. The core of the IGM-based classifier is the gated memory unit. The IGM structure is inspired by the memory process of the human brain. According to the law of people learning new things based on experience, it calculates the similarity between new information and known information to optimize the known information and better understand the new information, thereby improving the classification effect. This structure has achieved better classification results than the traditional fully connected layer classifier on some medical data sets, mainly relying on IGM's outstanding ability to analyze fine-grained and complex features, and can also summarize good common characteristics on small sample data sets. Considering that the blood flow motion characteristics also have the characteristics of high similarity and subtle differences, and the sample size of the fatigue dataset is not large, we choose IGM as the classifier of the network.

Claims

1. A fatigue detection network based on extracting blood flow motion features from face videos, characterized in that: The following steps are involved: Step 1, using the face video preprocessing module to obtain a facial frame difference sequence; Step 2, establish three parallel convolution feature extraction modules; Step 3, establishing a feature constraint processing module; Step 4: Use the classifier to obtain fatigue detection results.

2. A fatigue detection network based on extracting blood flow motion features from face video according to claim 1, characterized in that: The face video preprocessing module in step 1 first needs to use the existing face recognition model to perform face recognition on the start frame of the video and return a rectangular face area. All subsequent frames directly reuse the face frame to crop the image.

3. The fatigue detection network based on extracting blood flow motion features from face video according to claim 1 is characterized in that: After performing the face region selection operation, the face video preprocessing module in step 1 needs to perform a frame difference operation on the video sequence, that is, to obtain the difference between each adjacent frame.

4. The fatigue detection network based on extracting blood flow motion features from face video according to claim 1 is characterized in that: The face video preprocessing module in step 1 needs to use existing algorithms to generate physiological pseudo-labels from the original video.

5. The fatigue detection network based on extracting blood flow motion features from face video according to claim 1 is characterized in that: The feature extraction module in step 2 is composed of three parallel convolutional neural networks that share parameters. The difference between the three networks is that the convolution receptive field changes from small to large, and the implementation method is obtained by controlling the number of pooling layers in each network.

6. The fatigue detection network based on extracting blood flow motion features from face video according to claim 1 is characterized in that: The feature constraint processing module in step 3 first uses physiological pseudo-labels to constrain the autocorrelation matrix calculated from the first two features; secondly, a push-pull relationship is established between the three features, so that the first and second features maintain similarity, and these two features are different from the third feature.

7. The fatigue detection network based on extracting blood flow motion features from face video according to claim 1 is characterized in that: The classifier module in step 4 only takes the first two features as input, and introduces the IGM structure to design the classifier.

Images