Face emotion perception recognition method based on rPPG signal and three-dimensional normal vector assistance

Abstract

The invention discloses a face emotion perception recognition method based on rPPG signals and three-dimensional normal vector assistance, and relates to the technical field of face recognition, and the method comprises the steps: carrying out the extraction of a face three-dimensional normal vector of a face video frame through employing a Medipe module, and then extracting a feature vector fnv of the face three-dimensional normal vector; meanwhile, an MTTS-CAN network architecture module is adopted to carry out rPPG signal extraction, and then an rPPG signal feature vector frPPG is extracted; and inputting the face three-dimensional normal vector feature vector fnv and the rPPG signal feature vector frPPG into a feature fusion classification model to obtain a final emotion perception result. According to the invention, the two different sources of face signals are ingeniously integrated and are applied to facial expression recognition of the face, so that the accuracy of face emotion perception is improved.

Description

Technical Field

[0001] The present invention relates to technical fields such as computer vision, image processing and face recognition, and in particular to a method for human face emotion perception and recognition based on rPPG signals and three-dimensional normal vector assistance. Background Art

[0002] Expression recognition is a task to identify and understand human expressions through computer vision and machine learning techniques. It aims to automatically identify and classify facial expressions, such as joy, sadness, anger, surprise, etc., from face images or videos. Expression recognition has important applications in many fields, including human-computer interaction, affective computing, psychological research, social media analysis, etc.

[0003] At present, there are many methods for video expression recognition. Commonly used frameworks include VGG-Face, ResNet, TSN (Temporal Segment Networks), 3D-CNN (Convolutional 3D), DenseNet, etc. These frameworks provide powerful feature extraction and representation capabilities, and provide strong support for video expression recognition tasks. The present invention adopts a multimodal facial expression recognition method under these existing basic frameworks to improve the accuracy of recognizing facial expressions in videos.

[0004] Although the existing facial expression recognition has made significant progress in many aspects, it is difficult for a single facial expression recognition framework to achieve multi-faceted facial expression recognition. Human expressions are rich in diversity, and the appearance and dynamic characteristics of expressions vary with individual, cultural and background differences. Traditional facial expression recognition methods may find it difficult to accurately capture and represent this diversity, so in some cases they may not be able to accurately recognize complex or subtle expressions. Expressions are often conveyed through multiple sources of information, including facial expressions, sounds, voices and gestures. However, current facial expression recognition methods are mainly based on facial images or videos, ignoring information from other modalities. Multimodal expression recognition that integrates multiple sources of information is still a challenging problem, and traditional methods have poor adaptability. In addition, the current common deep neural networks such as FNN have too many parameters and are difficult to meet real-time requirements. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a facial emotion perception and recognition method based on rPPG signals and three-dimensional normal vectors to perform multimodal fusion of facial rPPG signals and facial three-dimensional normal vector signals to identify the emotional state of the characters in the video.

[0006] The objective of the present invention is achieved through the following technical solutions: The facial emotion perception and recognition method based on rPPG signal and three-dimensional normal vector assistance includes: The face video is divided into multiple 1.2S segments and each segment is framed. The Mediapipe module is used to extract the face 3D normal vector of the face video frame. Then the 3D face normal vector information is mapped into two dimensions. The mapped image is sent to the spatial feature extraction sub-network to obtain the extracted face 3D normal vector feature vector. f nv ; The MTTS-CAN network architecture module is used to extract rPPG signals from face videos, and the rPPG signals are divided into a 1.2S single pulse model. The extracted rPPG signals are preprocessed and converted into one-dimensional signals and sent to the rPPG feature extraction subnetwork to obtain the extracted rPPG signal feature vector. f rPPG ; The three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception result.

[0007] Furthermore, the extraction of the three-dimensional normal vector of the face from the face video frame by using the Mediapipe module specifically includes: Use the pre-trained convolutional neural network (CNN) to detect facial key points and obtain the key feature points on the face; The normal vector of the face is calculated based on the key feature points on the face, and finally the normal vector coordinates of the facial feature points are obtained.

[0008] Furthermore, the two-dimensional mapping of the three-dimensional face normal vector information specifically includes: For the normal vector of the plane where each feature point is located, the cosine value of the angle between the normal vector and the horizontal plane is constructed as the first channel, and the cosine value of the angle between the normal vector and the vertical plane is constructed as the second channel. The depth map is obtained by calculating the distance from the camera to the face surface in the video frame, and then the depth map is constructed as the third channel. Finally, the three channel values ​​are normalized to [0~255] and displayed.

[0009] Furthermore, the rPPG feature extraction subnetwork includes an input layer input, a first convolutional layer C1, a first normalization layer BN1, a first pooling layer S1, a second convolutional layer C2, a second normalization layer BN2 and a second pooling layer S2 connected in sequence; the first convolutional layer C1 and the second convolutional layer C2 are both used to extract features of the input signal using a 3×1 convolutional filter to generate a feature map; the first pooling layer S1 and the second pooling layer S2 are both used to sample the output of the convolutional layer and then use maximum pooling to return the maximum value in the 2×1 filter.

[0010] Furthermore, the three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception results, including: Receive the input face 3D normal vector feature vector f nv and rPPG signal feature vector f rPPG , the feature vector f nv and the eigenvector f rPPG Through horizontal splicing, seamless connection and fusion are performed to obtain the fused feature vector f union ; The concatenated feature vector f union Send it to the classifier to get the emotion classification result.

[0011] Furthermore, the classifier is a Softmax classifier, including a fully connected layer FC and a Softmax layer, the Softmax layer is used to output the predicted classification result using the Softmax activation function, and the loss function selects the sparse classification cross entropy function.

[0012] Furthermore, the extracting of rPPG signals from face videos using the MTTS-CAN network architecture module specifically includes: Features are extracted from video sequences containing faces, and the temporal convolutional network (TCN) and the spatial convolutional network (SCN) are used to capture dynamic changes and spatial information. The attention mechanism is used to focus on areas related to heart rate estimation to learn multiple tasks simultaneously and finally output the rPPG signal for each frame.

[0013] The beneficial effects of the present invention are: 1) The Mediapipe module uses a convolutional neural network (CNN) to perform efficient facial key point detection. By training on a large amount of labeled face image data, it can accurately learn the locations of facial key points. In the present invention, the Mediapipe module is cleverly integrated to provide a reliable solution for the extraction of three-dimensional normal vector signals of the face, thereby making a significant contribution to the accuracy and comprehensiveness of emotion recognition.

[0014] 2) The present invention uses the MTTS-CAN module to extract the corresponding rPPG signal in the face video, which can accurately capture the pixels containing the skin and reduce the camera quantization error; the extracted rPPG signal is preprocessed and converted into a one-dimensional signal and sent to the rPPG feature extraction subnetwork, which can efficiently extract key features from the rPPG signal and provide a reliable basis for subsequent heart rate detection.

[0015] 3) The present invention cleverly integrates these two facial signals from different sources and applies them to facial expression recognition, thereby improving the accuracy of facial emotion perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process of extracting a three-dimensional normal vector of a face disclosed in an embodiment of the present invention; Figure 2 The main technical roadmap of the method for facial emotion perception and recognition based on rPPG signals and 3D normal vector assistance disclosed in the embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of the rPPG feature extraction subnetwork disclosed in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a feature fusion classification model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1-Figure 4 , the present invention provides a technical solution: The facial emotion perception and recognition method based on rPPG signal and three-dimensional normal vector assistance includes: S1: Segment the face video into multiple 1.2S segments and extract frames from each segment. Use the Mediapipe module to extract the face 3D normal vector from the face video frame. Then perform 2D mapping on the 3D face normal vector information. Send the mapping image to the spatial feature extraction subnetwork to obtain the extracted face 3D normal vector feature vector. f nv ; The reason for dividing the face video into multiple 1.2S segments is that the 1.2S rppg single pulse waveform can better perform rapid recognition of single pulses.

[0019] like Figure 1 , the extraction of the three-dimensional normal vector of the face from the face video frame by using the Mediapipe module includes: Use the pre-trained convolutional neural network (CNN) to detect facial key points and obtain the key feature points on the face; The normal vector of the face is calculated based on the key feature points on the face, and finally the normal vector coordinates of the facial feature points are obtained.

[0020] Specifically, the two-dimensional mapping of the three-dimensional face normal vector information includes: For the normal vector of the plane where each feature point is located, the cosine value of the angle between the normal vector and the horizontal plane is constructed as the first channel, and the cosine value of the angle between the normal vector and the vertical plane is constructed as the second channel. The depth map is obtained by calculating the distance from the camera to the face surface in the video frame, and then the depth map is constructed as the third channel. Finally, the three channel values ​​are normalized to [0~255] and displayed.

[0021] In the 3D face normal vector, each point has its position information in the three-dimensional space. The depth information can be obtained by mapping each point from the three-dimensional space to the camera coordinate system and calculating its depth in the camera coordinate system (that is, the distance from the point to the camera).

[0022] Mediapipe is an open source framework developed by Google, specifically for building media processing applications based on machine learning. It provides a series of pre-trained models and tools, including face detection and key point recognition models, which can be used to handle various visual tasks. First, using the face detection model provided by Mediapipe, you can accurately locate and delimit faces. This model can accurately detect the position and bounding box of faces in images or video streams. Next, through Mediapipe's key point recognition model, you can accurately detect key feature points on the face, such as eyes, mouth, etc. The position information of these key points can be used to calculate the normal vector of the face. The Mediapipe framework provides efficient algorithms and data structures that can quickly and accurately detect and track these key points. The surface normal vector of the face can be accurately estimated through the vectors between these feature points. In this way, the shape and position information of facial expressions can be captured. The flow chart is as follows Figure 1 shown.

[0023] S2: The MTTS-CAN network architecture module is used to extract rPPG signals from face videos, and the rPPG signals are divided into a 1.2S single pulse model. The extracted rPPG signals are preprocessed and converted into one-dimensional signals and sent to the rPPG feature extraction subnetwork to obtain the extracted rPPG signal feature vector f rPPG ; rPPG (Remote Photoplethysmography) technology is a method of measuring physiological signals remotely, mainly used to monitor physiological parameters such as heart rate and blood oxygen saturation. Unlike traditional physiological signal measurement methods, rPPG technology uses a camera to capture tiny color changes on the skin surface to infer heart rate and blood oxygen levels. The basic principle of rPPG is to obtain physiological signals through photoelectric measurement technology, especially by analyzing the way blood absorbs and reflects light. When the heart beats, blood flows with it, causing slight changes in skin color. These changes may be caused by changes in the absorption of light by hemoglobin. Therefore, by analyzing the color changes on the skin surface, heart rate and other physiological parameters can be inferred.

[0024] In this embodiment, the rPPG signal extraction of face videos is performed using the MTTS-CAN network architecture module, specifically including: extracting features from video sequences containing faces, using a temporal convolutional network TCN and a spatial convolutional network SCN to capture dynamic changes and spatial information, using an attention mechanism to focus on areas related to heart rate estimation, so as to simultaneously learn multiple tasks, and finally outputting the rPPG signal of each frame.

[0025] In the process of extracting rPPG signals using the multi-task time-shifted convolutional attention (MTTS-CAN) network architecture module, a convolutional neural network (CNN) is used as a feature extractor to extract feature representations for each video frame. Then, the extracted feature sequence is input into the temporal convolutional network (TCN) and the spatial convolutional network (SCN). TCN can capture the dynamic changes at different time scales in the video sequence, which helps to extract the pulse signal. SCN can process the spatial information in the video frame, which helps to distinguish the contribution of the face area and other areas to the pulse estimation. Using the attention mechanism, the network can focus on the areas related to heart rate estimation and better distinguish the parts of the face related to pulse changes through the learned attention weights. Multiple tasks, such as heart rate estimation and posture estimation, may be learned simultaneously to improve the accuracy of heart rate estimation. Finally, the network outputs the rPPG signal of each frame in the video sequence for heart rate estimation. Through this process, the MTTS-CAN network can effectively extract rPPG signals from videos and realize heart rate estimation.

[0026] The MTTS-CAN network architecture module cleverly removes the 3D convolution operation while modeling space-time, and introduces two key additional components: the temporal shift module (TSM) and the attention module. The TSM divides the input tensor into three blocks across the channel dimension by shifting the tensor before feeding it into the convolution layer. Specifically, the TSM shifts the first data block by one position to the left (one frame in advance), the second data block by one position to the right (one frame in delay), and the third block remains unchanged. This tensor shift operation is performed on the time axis, simulating the effect of 3D convolution, which is particularly suitable for the motion branch. At the same time, the appearance branch of the network module only takes a single average frame. Through this clever design, the model only needs to calculate the attention mask once, which significantly reduces the calculation time, while more accurately capturing the pixels containing the skin and reducing the camera quantization error, thereby providing a more accurate basis for emotion recognition in the extraction of rPPG signals.

[0027] In this embodiment, the rPPG feature extraction subnetwork is as follows: Figure 3As shown, it includes an input layer input, a first convolution layer C1, a first normalization layer BN1, a first pooling layer S1, a second convolution layer C2, a second normalization layer BN2, and a second pooling layer S2 connected in sequence; the first convolution layer C1 and the second convolution layer C2 are both used to extract features of the input signal using a 3×1 convolution filter to generate a feature map; the first pooling layer S1 and the second pooling layer S2 are both used to sample the output of the convolution layer and then use maximum pooling to return the maximum value in the 2×1 filter, and sample through a 2×1 filter to reduce the dimension of the feature map, so as to retain the main feature information. In some specific embodiments, the number of feature maps output by the first convolution layer C1 is 10, and the second convolution layer C2 uses 20 3×1 convolution filters for feature extraction.

[0028] The design of the rPPG feature extraction subnetwork aims to make full use of the characteristics of one-dimensional signals and capture key features through convolution and pooling operations. The batch normalization layer BN introduced by the rPPG feature extraction subnetwork after each convolution layer helps to accelerate the training process and improve the generalization ability of the model. This design enables the neural network to efficiently extract key features from rPPG signals, providing a reliable foundation for subsequent heart rate detection. By utilizing convolution and pooling operations, the network can automatically learn abstract features in the signal, while the batch normalization layer helps to alleviate the gradient vanishing problem, improve the stability and performance of the model, and finally output the extracted rPPG feature vector f rPPG .

[0029] S3: transform the three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception result.

[0030] like Figure 4 As shown, the three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception results, including: Receive the input face 3D normal vector feature vector f nv and rPPG signal feature vector f rPPG , the feature vector f nv and the eigenvector f rPPG Through horizontal splicing, seamless connection and fusion are performed to obtain the fused feature vectorf union ; The concatenated feature vector f union Send it to the classifier to get the emotion classification result.

[0031] In this embodiment, the rPPG signal feature vector f rPPG =[a1, a2, ..., am] and the eigenvector of the 3D normal vector of the face f nv =[b1, b2, ..., bn] are seamlessly connected through splicing and fused to form a more comprehensive and rich feature vector [a1,a2, ..., am, b1, b2, ..., bn].

[0032] In this embodiment, the classifier is a Softmax classifier, including a fully connected layer FC and a Softmax layer, and the Softmax layer is used to output the predicted classification result using the softmax activation function. The Softmax classifier essentially uses a neural network model for classification. The output layer of the model sets N units according to the type N of emotion classification, and uses the softmax activation function to ensure that the output value is normalized to a probability distribution, and the sum of the probabilities of each category is 1.

[0033] The loss function used is Sparse Categorical Crossentropy, which is suitable for multi-classification tasks.

[0034] The overall network of the present invention comprises two branches, such as Figure 2 As shown, the first branch obtains the feature vector corresponding to rPPG through the rPPG feature extraction sub-network f rPPG At the same time, the other branch extracts the 3D normal vector of the face and maps the 3D normal vector information to the 2D space for training. The feature vector corresponding to the 3D normal vector of the face is extracted through the spatial feature sub-network. f rPPG Finally, these two feature vectors are input into the feature fusion classification model to obtain the fused feature vector f union Send it into the classifier to get the final emotion perception result.

[0035] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A facial emotion perception and recognition method based on rPPG signals and 3D normal vectors. Features: include: The face video is divided into multiple 1.2S segments and each segment is framed. The Mediapipe module is used to extract the face 3D normal vector of the face video frame. Then the 3D face normal vector information is mapped into two dimensions. The mapped image is sent to the spatial feature extraction sub-network to obtain the extracted face 3D normal vector feature vector. f nv ; The MTTS-CAN network architecture module is used to extract rPPG signals from face videos, and the rPPG signals are divided into a 1.2S single pulse model. The extracted rPPG signals are preprocessed and converted into one-dimensional signals and sent to the rPPG feature extraction subnetwork to obtain the extracted rPPG signal feature vector. f rPPG ; The three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception result.

2. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 1, It is characterized in that The method of extracting the three-dimensional normal vector of a face from a face video frame by using the Mediapipe module specifically includes: Use the pre-trained convolutional neural network (CNN) to detect facial key points and obtain the key feature points on the face; The normal vector of the face is calculated based on the key feature points on the face, and finally the normal vector coordinates of the facial feature points are obtained.

3. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 1, Features: The two-dimensional mapping of the three-dimensional face normal vector information specifically includes: For the normal vector of the plane where each feature point is located, the cosine value of the angle between the normal vector and the horizontal plane is constructed as the first channel, and the cosine value of the angle between the normal vector and the vertical plane is constructed as the second channel. The depth map is obtained by calculating the distance from the camera to the face surface in the video frame, and then the depth map is constructed as the third channel. Finally, the three channel values ​​are normalized to [0~255] and displayed.

4. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 1, Features: The rPPG feature extraction subnetwork includes an input layer input, a first convolutional layer C1, a first normalization layer BN1, a first pooling layer S1, a second convolutional layer C2, a second normalization layer BN2 and a second pooling layer S2 connected in sequence; the first convolutional layer C1 and the second convolutional layer C2 are both used to extract features of the input signal using a 3×1 convolutional filter to generate a feature map; the first pooling layer S1 and the second pooling layer S2 are both used to sample the output of the convolutional layer and then use maximum pooling to return the maximum value in the 2×1 filter.

5. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 1, Features: The three-dimensional normal vector feature vector of the face f nv and rPPG signal feature vector f rPPG Input the feature fusion classification model to obtain the final emotion perception results, including: Receive the input face 3D normal vector feature vector f nv and rPPG signal feature vector f rPPG , the feature vector f nv and the eigenvector f rPPG Through horizontal splicing, seamless connection and fusion are performed to obtain the fused feature vector f union ; The concatenated feature vector f union Send it to the classifier to get the emotion classification result.

6. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 5, Features: The classifier is a Softmax classifier, including a fully connected layer FC and a Softmax layer. The Softmax layer is used to output the predicted classification result using a softmax activation function, and the loss function uses a sparse classification cross entropy function.

7. The method for facial emotion perception and recognition based on rPPG signal and three-dimensional normal vector according to claim 1, Features: The extracting of rPPG signals from face videos using the MTTS-CAN network architecture module specifically includes: Features are extracted from video sequences containing faces, and the temporal convolutional network (TCN) and the spatial convolutional network (SCN) are used to capture dynamic changes and spatial information. The attention mechanism is used to focus on areas related to heart rate estimation to learn multiple tasks simultaneously and finally output the rPPG signal for each frame.

Images