A multi-modal data emotion recognition deep learning network based on rPPG principle

Abstract

The application provides a multi-modal data emotion recognition deep learning network based on an rPPG principle. Through the introduction of multi-modal data and auxiliary tasks, end-to-end emotion recognition is realized from a physiological meaning. The network attempts to obtain emotion and physiological labels from a physiological point of view based on video and heart rate signals by observing local blood flow actions of a face instead of macro muscle expression actions such as winking and so on. Blood flow action features containing physiological information in the video are captured through a deep learning network, time attention is provided based on the heart rate signals, and the network is guided to learn key feature representations containing physiological meanings in the face video. Meanwhile, heart rate self-similarity feature extraction is introduced as an auxiliary task to improve the network performance.

Description

TECHNICAL FIELD

[0001] The present invention relates to the innovation of rPPG, emotion recognition, heart rate measurement, multimodal, deep learning, signal processing, neural network classification direction. BACKGROUND

[0002] Emotion is a complex psychosomatic state triggered by external stimuli or specific events, which involves the higher functions of the human brain. In cognitive activities such as perception, decision-making, logical reasoning, and social interaction, emotional mechanisms play a crucial role. Positive emotions can enhance an individual's sense of well-being and improve work efficiency, while negative emotions can have adverse effects on daily life and professional performance. As the pace of modern society accelerates, people increasingly value emotional health and yearn for a better understanding, analysis, and even regulation of emotions. As one of the important technologies in the field of emotional science research, emotion recognition is not only crucial for personal psychological and physiological health, but also has wide applications in human-computer interaction based on biological characteristics, instant messaging, and intelligent monitoring. The development and application of emotion recognition technology show that this field is not only a window to explore the nature of humanity, but also promotes the integration of different disciplines such as science and philosophy, and has shown great application prospects in education, healthcare, and industry.

[0003] Traditional physiological signal methods for measuring emotions, such as electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG), although can provide accurate data, have a major drawback that they require direct contact with the skin through sensors to collect data. This contact-based measurement method may limit the freedom of movement of the subjects being measured and may cause discomfort in long-term monitoring. In addition, these methods are not suitable for individuals with skin sensitivity or allergic reactions, or those with damaged skin surfaces. At the same time, groups that need to move frequently, such as epilepsy patients, active children, or patients with attention deficit hyperactivity disorder, may not be suitable for these technologies. When multiple individuals need to be monitored at the same time, contact methods also face challenges in the number of devices and parallel measurement techniques.

[0004] To overcome these limitations, Remote Photoplethysmography (rPPG) technology has emerged. rPPG technology allows for the non-contact extraction of heart rate information by analyzing facial videos, thanks to the correlation between changes in reflected light from the skin and fluctuations in blood volume. This approach greatly improves the convenience of data collection and reduces interference with the subjects. With the advancement of rPPG technology, researchers have also begun to explore how to use this technology for emotion recognition and affective computing, opening up a new research path aimed at better understanding and interpreting human emotional states.

[0005] In recent years, deep learning techniques have been widely applied to emotion recognition tasks based on facial expressions, body postures, or physiological signals. However, emotion recognition based on facial expressions can be susceptible to human deception, while methods based on physiological signals, although providing more in-depth information, may not capture subtle emotional changes due to their macroscopic observation scale. Single-modal emotion recognition methods often only consider one type of data input, which can lead to the neglect of the influence of interactions between different modalities on emotion recognition results. Studies have shown that rPPG technology can serve as a reliable non-contact method to replace traditional electrocardiograms for emotion state recognition. This approach not only improves the convenience of emotion recognition and user acceptance, but also paves the way for the development of more natural and user-friendly emotion recognition systems. SUMMARY

[0006] The present invention solves the above problems: the emotion recognition network model based on video and heart rate related physiological signals proposed in the present invention solves the problem of single recognition dimension of single modal and the problem of low emotion classification accuracy caused by the fact that multi-modal solutions involving physiological signals mainly focus on electroencephalogram signals.

[0007] The present invention solves the above problems: the multi-modal emotion recognition network based on rPPG proposed in the present invention can monitor the emotional changes of relatively vulnerable groups such as patients with psychological diseases and the elderly by recognizing and understanding individual emotions, and effectively help subjects regulate emotions and improve mental health.

[0008] The present invention solves the above problems: the multi-modal emotion recognition network based on rPPG proposed in the present invention uses RPPG technology to provide a non-contact measurement scheme for heart rate signals based on skin regions, solving the problem that physiological signals commonly used to measure emotions, including electroencephalogram, electrocardiogram, electromyogram, etc., require close contact with the skin of the subject, causing inconvenience to the person.

[0009] This invention solves the above problems: The multimodal emotion discrimination network based on rPPG proposed in this invention introduces heart rate self-similarity feature extraction as an auxiliary task to improve network performance.

[0010] This invention solves the above problems: The multimodal emotion discrimination network based on rPPG proposed in this invention captures blood flow motion features containing physiological information in videos through the network, provides temporal attention based on heart rate signal encoding, and guides the network to learn key feature representations containing physiological meaning in face videos. Attached Figure Description

[0011] In order to more clearly and understandably illustrate the technical solutions in the embodiments of the present invention or the prior art, I will now briefly introduce the accompanying drawings used in the description of the embodiments or the prior art.

[0012] Figure 1 The overall model architecture diagram of the neural network based on rPPG multimodal emotion recognition network provided in the embodiments of the present invention is shown.

[0013] Figure 2 This is a physiological signal multi-scale feature embedding structure provided in an embodiment of the present invention;

[0014] Figure 3 A flowchart illustrating the creation of a self-similarity feature map provided in this embodiment of the invention; Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described clearly and in detail below with reference to the accompanying drawings and specific examples.

[0016] like Figure 1 As shown, the network architecture aims to perform emotion recognition by combining video frame difference signals with physiological signals (ECG signals by default). A spatiotemporal feature extractor is used to extract spatiotemporal features containing physiological information from the video frame difference signals, forming a spatiotemporal feature map M. st After receiving video frame difference signals and ECG physiological signals as input, the system first extracts basic spatiotemporal features containing physiological information using a spatiotemporal feature extractor. Then, a self-similar feature map M is generated through temporal sliding windowing, feature space projection, and self-similarity calculation based on cosine similarity. s Output heart rate feature labels and self-similarity labels from the dataset. Calculate the loss. Based on the heart rate self-similarity feature label, physiological parameters such as heart rate and heart rate variability can be calculated. The default input physiological signal is ECG, when the input is ECG signal, the signal is directly encoded for processing; if the ECG signal is missing, the remote photoplethysmography (rPPG) signal extracted from the video is used as a supplement. After the ECG or rPPG signal is processed by the multi-scale feature embedding, the corresponding physiological signal representation is generated. These representations are then processed by a time attention module similar to the Transformer encoder, obtaining the mask attention, providing the video sequence with time domain attention enhancement with physiological significance.

[0017] The scheme adopted by the present application to solve the problem is divided into 5 parts:

[0018] The first part is to build a spatio-temporal feature extractor. After inputting the video frame difference signal and the ECG physiological signal, the bottom layer spatio-temporal features containing physiological information are first extracted by the spatio-temporal feature extractor. The present application selects the classic backbone network DeePhys structure in the rPPG field, and modifies it based on it. In order to obtain purer spatio-temporal features of physiological signals, the present application only uses frame difference sequences during training, and only retains a single branch structure. In addition, the present application uses three-dimensional convolution instead of two-dimensional to better extract spatio-temporal features; the activation function is adjusted to promote network convergence; the network depth and the size of the convolution kernel are modified, and the size of the input video is adjusted.

[0019] The second part is, as Figure 2As shown, the input physiological signal is first processed by multi-scale feature embedding, and then sent to the subsequent module. First, the signal is segmented into a fixed length sequence by the sliding window technique, the original one-dimensional signal is converted into a matrix form, and input to the tokenizer for processing. The length of the sliding window is a hyperparameter, and the ablation study is conducted in the experiment, and the results show that the optimal window length is set to 60. Then a parallel multi-scale pooling structure is used to process the signal after sliding window. Each layer can not be pooled or use different size of pooling layer to obtain multi-scale features of different fields, and then further process through convolution layer, batch normalization layer and activation function. All processed features are finally spliced and merged through cat operation, and a one-dimensional convolution layer is used to adjust the channel number of the features. The present application realizes multi-scale feature extraction through the pooling structure. In addition, this structure also helps to enhance the robustness of the network to local changes (such as rotation and translation) of the input signal. The pooling operation divides the input feature map into several windows and processes the pixels in each window, thereby retaining the most significant features in the image. Among them, the maximum pooling layer enhances the feature capture ability by selecting the maximum value in the window; while the average pooling layer calculates the average value in the window to achieve the effect of smoothing noise. Since the original signal processed by the present application is positive, in order to extract more significant and meaningful features, we choose the maximum pooling layer.

[0020] The third part is to design a time attention module based on physiological signals. The present application adopts a structure similar to the encoder of the transformer encoder, and uses multi-head attention. There are 6 layers of encoder stacked together to obtain the attention score. Then, the channel number is changed through linear mapping, and then normalized and adjusted by using the batch normalization layer and the sigmoid layer. Finally, the attention result is obtained. The subsequent features extracted from the video are time-weighted and attention-enhanced. It can be regarded as a global remodeling based on time sequence, which obtains rich information through attention mechanism to guide the corresponding task.

[0021] The fourth part is to design a cross-modal fusion scheme and an emotion recognizer. The present application performs cross-modal fusion enhancement on the spatiotemporal physiological signal based on time domain attention. The attention and feature map are multiplied by time to fully fuse the features. The present application flattens the fusion representation enhanced by attention and sends it to the classifier. Corresponding to the classifier, the loss function uses cross-entropy, and the gradient is propagated in the reverse direction to modify the network parameters.

[0022] The fifth part is as follows Figure 3As shown, heart rate self-similarity feature extraction is selected as an auxiliary task. For the feature map extracted by the spatiotemporal feature extractor, a series of operations such as temporal sliding windowing, feature mapping, and cosine similarity calculation are used to generate a self-similarity feature map. This invention creates the heart rate self-similarity feature map after feature extraction; the specific operation process is as follows:

[0023] (1) Perform time-domain sliding windowing on the feature map output by the front-end module. Note that each sliding window must contain signal trend information near the center point of the window.

[0024] (2) The result after sliding windowing is transformed into a feature vector by passing it through a linear layer. The feature vector carries the rhythm and period information of the rPPG signal.

[0025] (3) Form vector pairs from each of the generated feature vectors, calculate their cosine similarity, and obtain the final output matrix M. s M s The size is N×N, where N is the number of generated feature vectors. Each element m ij The first row represents the relationship between the i-th and j-th time points; the i-th row represents the relationship between the i-th time point and all other time points; each diagonal line represents the self-similarity of features with the same time difference.

[0026] During waveform and heart rate extraction, firstly, the heart rate self-similarity feature map M is extracted. s Compressing to a one-dimensional sequence yields a self-similar waveform (denoted as W). s First, put M s The elements are grouped according to the condition of having the same time distance. For each specific c, all corresponding elements that satisfy the condition are grouped together, resulting in group {G0, G1, ..., G...}. N-C-L Next, calculate the average value of the group, then sort them in ascending order of c, ultimately forming the waveform sequence W. s W s Each element w in k The calculation formula is:

[0027] w k =menn(G k k∈[0, NCL]

[0028] For each label in the dataset, the corresponding self-similarity sequence is denoted as W. s .

[0029] On this basis the present application provides both frequency domain and time domain to calculate heart rate value. The first one is based on frequency domain: first through FFT to get the frequency domain spectrum of the sequence, the time domain signal is converted to frequency domain; then select the highest component of the frequency domain, according to the peak component to determine the frequency value of the sequence, corresponding to the final heart rate value. The second one is based on time domain, by calculating the average distance of the sequence peak value to determine the average period of heart rate, and then get the heart rate value. First, the signal is peak detection, statistics and record the maximum value position of the sequence signal. Set each peak point position as p i , i∈{0,1,…,M}, M is the number of peak points of the sequence signal, the signal sampling rate is s, then record the time interval Δp i between each adjacent peak as:

[0030] Δp i = p i+1 -p i

[0031] The final heart rate value calculation formula is:

[0032]

[0033] For frequency calculation method, due to the calculation method of FFT, when the time sequence sampling window is short, there may be a certain amount of quantization error.

Claims

1. A method of a multi-modal data emotion recognition deep learning network based on rPPG principle, characterized in that, Comprising the following steps: Step 1, divide the data set, most of the data as network training set, a small part of the data as network test set, the experimental objects, video clips of network training set and network test set do not coincide with each other; Step 2, statistical analysis is carried out on the sample label distribution of the data set, and the video signal and physiological signal in the data are pretreated; Step 3, a space-time feature extractor is established, the input of which is the video frame difference signal ECG physiological signal, which extracts the underlying space-time feature containing physiological information, and the structure thereof is a simplified DeePhys structure; Step 4, a multi-scale feature embedding structure is established, which can extract features, capture, process and understand key information in the data, and the structure thereof is a parallel multi-scale pooling structure to obtain multi-scale different fields, and finally merged and spliced, and adjusted through a one-dimensional convolution channel; Step 5, a time attention module based on physiological signal is established, which is a global remodeling based on time sequence, and rich information is obtained through attention mechanism to guide the corresponding task, and the structure thereof is similar to the encoder structure of transformer encoder; Step 6, establish rPPG extraction module, divided into heart rate self-similarity feature extraction and waveform and heart rate extraction, wherein the heart rate self-similarity feature extraction uses time domain sliding window, feature mapping, cosine similarity calculation and other operations to obtain a self-similarity feature map; in waveform extraction, the heart rate self-similarity feature map M S is compressed to a one-dimensional sequence to obtain a self-similarity waveform, and in heart rate extraction, the frequency domain spectrum of the sequence is calculated through FFT to convert the time domain signal into the frequency domain; then the highest component in the frequency domain is selected, the frequency value of the sequence is judged according to the peak component, and the final heart rate value is obtained. Step 7, the training and fine-tuning of the whole module are completed, and the evaluation test is carried out on the test set.

2. The method of claim 1, wherein the multi-modal data emotion recognition deep learning network based on the rPPG principle is characterized by, In step 2, the video data is verified for video length, sampling rate, direction and label, the video interval is screened, and the data is enhanced by various means, and the physiological signal is screened and cut according to the corresponding interval.

3. The method of claim 1, wherein the multi-modal data emotion recognition deep learning network based on the rPPG principle is characterized by, The loss function of the whole network in step 7 is: LOSS = σL CLS + μL rppg + δL IBS where L CLS , L rppg and L IBS represent the emotion label loss, the rppg task loss and the IBS loss based on information similarity, and σ, μ, δ are hyperparameters. Specifically, L CLS uses the classic classification loss cross-entropy: The rppg task loss includes MSE loss, negative Pearson correlation coefficient loss, standard deviation regularization loss and frequency domain signal-to-noise ratio based regularization: L rppg = L MSE + αL p + βL SD + γL SNR wherein L MSE is defined as: L p and the negative Pearson correlation coefficient is defined as: L SD is defined as: L SNR defined as: L IBS is defined as:

Images