Multi-modal data emotion recognition deep learning network based on rPPG principle

Abstract

The invention provides a multi-modal data emotion recognition deep learning network based on an rPPG principle. Through introduction of multi-modal data and auxiliary tasks, end-to-end emotion recognition is realized from physiological significance. The network attempts to obtain emotion and physiological labels from a physiological perspective by observing local actions of human face blood flow instead of macroscopic muscle expression actions such as eyebrow making and the like on the basis of videos and heart rate signals. Blood flow action features containing physiological information in the video are captured through a deep learning network, time attention is provided for the features based on heart rate signal coding, and network learning of key feature representation containing physiological significance in the face video is guided. And meanwhile, heart rate self-similarity feature extraction is introduced as an auxiliary task to improve the network performance.

Description

Technical Field

[0001] The present invention involves innovations in rPPG, emotion recognition, heart rate measurement, multimodality, deep learning, signal processing, and neural network classification. Background Art

[0002] Emotion is a complex physical and mental state caused by external stimuli or specific events, which involves the higher functions of the human brain. Emotional mechanisms play a vital role in cognitive activities such as perception, decision-making, logical reasoning, and social interaction. Positive emotions can enhance personal happiness and improve work efficiency, while negative emotions may have an adverse effect on daily life and professional performance. With the acceleration of the pace of modern society, people pay more and more attention to emotional health and are eager to master more methods of understanding, analyzing and even regulating emotions. As one of the important technologies in emotion science research, emotion recognition is not only crucial to personal psychological and physical health, but also has wide applications in many fields such as human-computer interaction, instant messaging, and intelligent monitoring based on biometrics. The development and application of emotion recognition technology show that this field is not only a window to explore the essence of human nature, but also promotes the integration of different disciplines such as science and philosophy, and shows great application prospects in education, health care, and industry.

[0003] Traditional methods of measuring physiological signals of emotions, such as electroencephalogram (EEG), electrocardiogram (ECG) and electromyogram (EMG), can provide accurate data, but one of their major drawbacks is that they require sensors to directly contact the skin to collect data. This contact measurement method may limit the freedom of movement of the subject and may cause discomfort in the case of long-term monitoring. In addition, these methods are not suitable for individuals with skin sensitivity or allergic reactions, or skin surface damage. At the same time, groups that need to move frequently, such as epileptic patients, active children or patients with ADHD, may not be suitable for these technologies. When multiple individuals need to be monitored at the same time, contact methods also face technical challenges in terms of the number of devices and parallel measurements.

[0004] To overcome these limitations, remote photoplethysmography (rPPG) technology came into being. rPPG technology allows non-contact extraction of heart rate information by analyzing facial videos, thanks to the correlation between changes in light reflected from the skin and fluctuations in blood volume. This method greatly improves the convenience of data collection and reduces interference to the subjects. With the advancement of rPPG technology, researchers have also begun to explore how to use this technology for emotion recognition and emotion computing, thus opening up a new research path aimed at better understanding and interpreting human emotional states.

[0005] In recent years, deep learning techniques have been widely used in emotion recognition tasks based on facial expressions, body postures or physiological signals. However, emotion recognition based on facial expressions may be susceptible to artificial camouflage, and although the method based on physiological signals provides deeper information, it may not be able to capture subtle changes in emotions due to its relatively macroscopic observation scale. Single-modal emotion recognition methods often only consider one type of data input, which may lead to ignoring the impact of interactions between different modalities on emotion recognition results. Studies have shown that rPPG technology can be used as a reliable non-contact method to replace traditional electrocardiograms for the identification of emotional states. This method not only improves the convenience and user acceptance of emotion recognition, but also paves the way for the development of more natural and user-friendly emotion recognition systems. Summary of the invention

[0006] The present invention solves the above problems: the dual-modal emotion discrimination network model based on video and heart rate-related physiological signals proposed in the present invention solves the problem that the recognition dimension of single modality is single, and the multimodal solution involving physiological signals is also mainly concentrated in EEG signals, resulting in low accuracy of emotion classification.

[0007] The present invention solves the above problems: the rPPG-based multimodal emotion discrimination network proposed in the present invention can monitor the emotional changes of relatively vulnerable groups such as patients with mental illness and the elderly by identifying and understanding individual emotions, and effectively help subjects regulate their emotions and improve their mental health.

[0008] The present invention solves the above problems: the rPPG-based multimodal emotion discrimination network proposed in the present invention utilizes RPPG technology to provide a skin area-based non-contact measurement solution for heart rate signals, solving the problem that contact measurements of physiological signals commonly used to measure emotions, including electroencephalogram, electrocardiogram, electromyogram, etc., all require close contact with the skin of the subject, causing inconvenience to people's movements.

[0009] The present invention solves the above problems: the rPPG-based multimodal emotion discrimination network proposed in the present invention introduces heart rate self-similarity feature extraction as an auxiliary task to improve network performance.

[0010] The present invention solves the above problems: the rPPG-based multimodal emotion discrimination network proposed in the present invention captures the blood flow motion features containing physiological information in the video through the network, provides temporal attention for it based on heart rate signal encoding, and guides the network to learn the key feature representations containing physiological significance in the face video. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly and easily explain the embodiments of the present invention or the technical solutions in the prior art, I will now briefly introduce the drawings required to be used in the description of the embodiments or the prior art.

[0012] Figure 1 The overall model architecture diagram of the neural network of the rPPG-based multimodal emotion recognition network provided in the embodiment of the present invention;

[0013] Figure 2 A multi-scale feature embedding structure for physiological signals provided by an embodiment of the present invention;

[0014] Figure 3 A flow chart of making a self-similarity feature map provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is clearly and detailed described below with reference to the accompanying drawings and specific examples.

[0016] like Figure 1 As shown in Figure 1, the network architecture aims to perform emotion recognition by combining video frame difference signals with physiological signals (ECG signals by default). The spatiotemporal feature extractor is used to extract spatiotemporal features containing physiological information from the video frame difference signals to form a spatiotemporal feature map M. st After receiving the video frame difference signal and ECG physiological signal as input, the spatiotemporal feature extractor is first used to extract the basic spatiotemporal features containing physiological information. Then, the self-similarity feature map M is generated through time domain sliding window, feature space projection and self-similarity calculation based on cosine similarity. s , output heart rate feature label, and dataset self-similarity label Calculate the loss. Physiological parameters such as heart rate and heart rate variability can be calculated based on the heart rate self-similarity feature label. The default input physiological signal is ECG. When the input is an ECG signal, the signal is directly encoded; if the ECG signal is missing, the remote photoplethysmography (rPPG) signal extracted from the video is used as a supplement. After the ECG or rPPG signal is processed by multi-scale feature embedding, the corresponding physiological signal representation is generated. These representations are then processed through a temporal attention module similar to the Transformer encoder to obtain masked attention, providing physiologically meaningful time domain attention enhancement for the video sequence.

[0017] The solution adopted by the present invention to solve the problem is divided into 5 parts:

[0018] In the first part, a spatiotemporal feature extractor is constructed. After the video frame difference signal and ECG physiological signal are input, the underlying spatiotemporal features containing physiological information are first extracted through the spatiotemporal feature extractor. The present invention selects the DeePhys structure, a classic backbone network in the rPPG field, and makes modifications based on it. In order to obtain purer spatiotemporal features of physiological signals, the present invention only uses frame difference sequences during training and retains only a single structure. In addition, the present invention uses three-dimensional convolution instead of two-dimensional to better extract spatiotemporal features; adjusts the activation function to promote network convergence; modifies the network depth and the size of the convolution kernel, and adjusts the size of the input video.

[0019] The second part, such as Figure 2As shown, the input physiological signal is first processed by multi-scale feature embedding and then sent to the subsequent module. First, the signal is divided into a sequence of fixed length by sliding window technology, the original one-dimensional signal is converted into a matrix form, and input into the tokenizer for processing. The length of the sliding window is a hyperparameter. The ablation study was carried out in the experiment, and the results showed that the optimal window length was set to 60. Then a parallel multi-scale pooling structure is used to process the signal after the sliding window. Each layer can be unpooled or use pooling layers of different sizes to obtain multi-scale features of different fields of view, and then further processed by convolution layer, batch normalization layer and activation function. All processed features are finally spliced ​​and merged by cat operation, and the number of channels of the feature is adjusted by a one-dimensional convolution layer. The present invention realizes multi-scale feature extraction through a pooling structure. In addition, this structure also helps to enhance the robustness of the network to local changes (such as rotation and translation) of the input signal. The pooling operation divides the input feature map into several windows and processes the pixels in each window, thereby retaining the most significant features in the image. Among them, the maximum pooling layer enhances the feature capture capability by selecting the maximum value in the window; while the average pooling layer achieves the effect of smoothing noise by calculating the average value in the window. Considering that the original signals processed by the present invention are all positive values, in order to extract more significant and meaningful features, we choose the maximum pooling layer.

[0020] The third part is to design a temporal attention module based on physiological signals. This invention adopts a transformer-like encoder structure and uses multi-head attention. A total of 6 layers of encoders are stacked to obtain an attention score. The number of channels is then changed through linear mapping, and then the batch normalization layer and sigmoid layer are used for normalization and adjustment of the value range to obtain the final attention result. Subsequently, the features extracted from the video are enhanced by time-domain weighted attention. It can be regarded as a global remodeling based on temporal significance, and the globally fused rich information obtained through the attention mechanism guides the corresponding tasks.

[0021] The fourth part is to design a cross-modal fusion scheme and emotion recognizer. The present invention performs cross-modal fusion enhancement for spatiotemporal physiological signals based on time domain attention. The attention and feature map are multiplied by time to fully integrate the features. The present invention flattens the fusion representation after attention enhancement and feeds it into the classifier. Corresponding to the classifier, the loss function uses cross entropy, back-propagates gradients and modifies network parameters.

[0022] The fifth part, such as Figure 3As shown, the extraction of the self-similarity feature of the heart rate is selected as an auxiliary task. For the feature map extracted by the spatiotemporal feature extractor, a series of operations such as time domain sliding window, feature mapping and cosine similarity calculation are used to generate the self-similarity feature map. The present invention produces the self-similarity feature map of the heart rate after extracting the features, and the specific operation process is as follows:

[0023] (1) Perform time domain sliding window on the feature map output by the front module. Note that each sliding window must contain the signal trend information near the center point of the window.

[0024] (2) The sliding window result is projected through a linear layer and converted into a feature vector. The feature vector carries the rhythm and period information of the rPPG signal.

[0025] (3) Form vector pairs from the generated series of feature vectors one by one, calculate the cosine similarity between them, and obtain the final output matrix M s . M s The size is N×N, where N is the number of generated feature vectors. Each element m ij Represents the relationship between the i-th and j-th moments; the i-th row represents the relationship between the i-th moment and all moments; each diagonal line represents the self-similarity of the same time difference feature.

[0026] When extracting waveform and heart rate, first, the heart rate self-similarity feature map M s Compressed to a one-dimensional sequence to obtain a self-similar waveform (denoted as W s ). First, M s The elements in are grouped according to the condition of meeting the same time distance. For each specific c, all corresponding elements that meet the condition are grouped together to obtain a group {G0, G1, ..., G N-C-L Then find the average value of the group, and then sort them in order from small to large, and finally form the waveform sequence W s .W s Each element w in k The calculation formula is:

[0027] w k =menn(G k ), k∈[0,NCL]

[0028] For the labels in the data set, the corresponding self-similarity sequence is recorded as W s .

[0029] On this basis, the present invention provides two methods, frequency domain and time domain, to calculate the heart rate value. The first method is based on the frequency domain: first use FFT to find the frequency domain spectrum of the sequence, and convert the time domain signal into the frequency domain; then select the highest component in the frequency domain, and judge the frequency value of the sequence according to the peak component, which corresponds to the final heart rate value. The second method is based on the time domain, and the average period of the heart rate is judged by calculating the average distance of the sequence peak, and then the heart rate value is obtained. First, perform peak detection on the signal, and count and record the maximum value position of the sequence signal. Let the position of each peak point be p i , where i∈{0, 1, …, M}, M is the number of peak points of the sequence signal, and the signal sampling rate is s, then the time interval Δp between adjacent peaks is recorded i for:

[0030] Δp i =p i+1 -p i

[0031] The final heart rate value is calculated as follows:

[0032]

[0033] For the frequency calculation method, due to the calculation method of FFT, when the time series sampling window is short, there may be certain quantization errors.

Claims

1. A multimodal data emotion recognition deep learning network based on rPPG principle, characterized in that: The following steps are involved: Step 1: divide the data set, with most of the data used as the network training set and a small part of the data used as the network test set. The experimental objects and video clips of the network training set and the network test set do not overlap with each other. Step 2: Statistically analyze the sample label distribution of the data set and preprocess the video signal and physiological signal in the data; Step 3, establishing a spatiotemporal feature extractor; Step 4: Establish a multi-scale feature embedding structure; Step 5, establish a temporal attention module based on physiological signals; Step 6, establishing an rPPG extraction module; Step 7: Complete the training and fine-tuning of the entire module, and then perform evaluation tests on the test set.

2. According to claim 1, a multimodal data emotion recognition deep learning network based on rPPG principle is characterized in that: In step 2, the video data is verified in terms of video length, sampling rate, direction, and label, the video interval is screened, and data enhancement is performed using various means. The physiological signals are then screened and cropped in the corresponding intervals.

3. The multimodal data emotion recognition deep learning network based on rPPG principle according to claim 1, characterized in that: In step 3, the input of the spatiotemporal feature extractor is the video frame difference signal ECG physiological signal to extract the underlying spatiotemporal features containing physiological information, and its structure is a simplified DeePhys structure.

4. The multimodal data emotion recognition deep learning network based on rPPG principle according to claim 1, characterized in that: The multi-scale feature embedding structure in step 4 enables the model to better extract features, capture, process and understand key information in the data. Its structure is a parallel multi-scale pooling structure to obtain multi-scale different fields of view, and finally merge and splice, and adjust the channel through a one-dimensional convolution.

5. The multimodal data emotion recognition deep learning network based on rPPG principle according to claim 1, characterized in that: The temporal attention module based on physiological signals in step 5 is a global remodeling based on temporal significance. It obtains globally fused rich information through the attention mechanism to guide the corresponding task. Its structure is similar to that of the transformer encoder.

6. The multimodal data emotion recognition deep learning network based on rPPG principle according to claim 1, characterized in that: In step 6, the rPPG extraction module is divided into heart rate self-similarity feature extraction and waveform and heart rate extraction. The heart rate self-similarity feature extraction uses time domain sliding window, feature mapping, cosine similarity calculation and other operations to obtain the self-similarity feature map; in waveform extraction, the heart rate self-similarity feature map M S Compressed to a one-dimensional sequence to obtain a self-similarity waveform (Selfsimilarity Wave, denoted as M S ), when extracting the heart rate, first use FFT to find the frequency domain spectrum of the sequence and convert the time domain signal into the frequency domain; then select the highest component in the frequency domain, and judge the frequency value of the sequence according to the peak component, which corresponds to the final heart rate value.

7. The multimodal data emotion recognition deep learning network based on rPPG principle according to claim 1, characterized in that: The loss function of the overall network in step 7 is: LOSS=σL CLS +μL rppg +δL IBS Among them, L CLS , L rppg and L IBS represents the emotion label loss, rppg task loss and IBS loss based on information similarity, σ, μ, δ are hyperparameters. Specifically, L CLS Using the classic classification loss cross entropy: The rppg task loss includes MSE loss, negative Pearson correlation coefficient loss, standard deviation regularization loss, and regularization based on frequency domain signal-to-noise ratio: L rppg =L MSE +ɑL p +βL SD +γL SNR Among them, L MSE Defined as: L p The negative Pearson correlation coefficient calculation formula is defined as: L sD Defined as: L SNR Defined as: L IBS Defined as:

Images