Multi-modal physiological signal emotion recognition method based on time-frequency spatial domain feature fusion

Abstract

The invention discloses a multi-modal physiological signal emotion recognition method based on time-frequency-space feature fusion. Comprising the following steps: denoising and normalizing a database by using Butterworth filtering, extracting various time-domain features based on the characteristics of various physiological signals, and performing time-frequency analysis on the signals by adopting wavelet transform and Hilbert-Huang transform. The method comprises the following steps of: decomposing signals with different emotions by using an empirical mode decomposition method to obtain different intrinsic mode functions, and performing data analysis on the mode functions to obtain emotional characteristics of various physiological signals under different scales and different frequency components. And then a convolutional neural network is used to select and fuse various features extracted from the multi-modal signal in a spatial domain to obtain more universal effective emotional features, so that a physiological signal emotion recognition model with good generalization ability and stable and excellent effect is formed.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition, mainly to the field of physiological signal emotion recognition. Specifically, the present invention provides a multimodal physiological signal emotion recognition method that integrates time-frequency and spatial domain features. Background Art

[0002] Affective computing is a general term for human emotions, feelings and feelings, emotion recognition and emotion analysis. Since Professor Picard proposed the concept of affective computing in 1997, it has been guiding computers to recognize and express emotions and respond intelligently to human emotions. Emotions are not only closely related to human cognitive activities, but also play an important role in communication, negotiation and decision-making. Therefore, making machines understand human emotions can not only enable people to have a better emotional experience in interaction with computers, but also improve work performance and work efficiency, which has great social significance and a wide range of application scenarios.

[0003] Emotion recognition has important practical significance in many aspects, such as medical monitoring, psychological diagnosis, behavior monitoring, daily emotion control, user experience research, etc. For example, in psychological diagnosis, machine monitoring and identification of the patient's emotional state can increase the diagnostic basis of psychologists, help doctors guide the patient's emotions, and make the patient recover faster; in the field of transportation, it can identify the driver's fatigue level and mental state, and provide relevant feedback to reduce the occurrence of traffic accidents.

[0004] The materials used for emotion recognition mainly include text, language, expression, physiological signals, etc. Among them, physiological signals are only controlled by the hormone system and the nervous system, which are not easy to control subjectively. Compared with recognition materials such as expression, sound and text, they can more objectively reflect the emotional state of the human body. At the same time, modern sensing equipment can easily collect a large number of physiological signals of users under various emotions, which also provides objective conditions for research. Common physiological signals mainly include EEG, eye movement, electromyography, skin electricity, electrocardiogram and breathing signals. Feature extraction is a key step in emotion recognition. Only by extracting more effective features can the accuracy of emotion recognition be guaranteed. Common physiological signal features include time domain features, frequency domain features and time-frequency domain features.

[0005] Different physiological signals have different feature extraction methods. The time domain features of EEG are mainly concentrated in event-related potentials, the time domain features of peripheral physiological signals are mainly the statistical characteristics of the signal, the time domain features of ECG include heart rate and heart rate variability, and the time domain features of respiratory signals are mainly the time intervals between calculated peaks. In terms of frequency domain features, the frequency energy distribution of each physiological signal is also different. Considering that physiological signals are non-stationary signals with low signal-to-noise ratio and strong randomness, traditional time and frequency domain discrete analysis cannot effectively reflect the characteristics of physiological signals. It has low spectral resolution, Fourier transform-based analysis loses time domain information, smoothing measures are needed to improve the statistical characteristics of spectrum estimation, and some windowed correlation functions will have negative power spectrum estimates.

[0006] Wavelet transform and Hilbert transform can obtain the time-frequency information of the signal at the same time, and is a standard, feasible and most effective time-frequency analysis method. Because its localized analysis method uses time and frequency windows, multi-rate filtering, and multi-scale scaling, it is most suitable for non-stationary signals. The Hilbert-Huang transform consists of empirical mode decomposition (EMI) and Hilbert transform (HT). The signal is decomposed by EMI to obtain a series of intrinsic mode functions (IMFs), and then the instantaneous amplitude, frequency and phase of these IMFs are obtained by HT.

[0007] Most of the previous studies have randomly fused the multiple features extracted from various physiological signals, and then obtained a feature combination suitable for classification training through common dimensionality reduction methods, such as principal component analysis (PCA) and independent component analysis (ICA), or directly input the denoised physiological signals into the network model for feature extraction, without exploring the correlation between multimodal signal features. The representation of different human emotional information is contained in physiological signals of different modalities. The collection of physiological signals is multi-channel collection. Electrodes are placed in various parts of the human body, and brain electrodes are placed in different brain regions. The human brain regions and body parts do not exist independently, which forms a fixed spatial domain information. The correlation of these different modal information in the spatial domain can effectively improve the accuracy of emotion recognition. Considering that convolutional neural networks perform well in spatial domain analysis, and that a not overly complex structure has a faster learning and classification speed for more feature data input, the present invention uses convolutional neural networks for feature fusion in the spatial domain.

[0008] The problem of feature extraction, selection and fusion of physiological signals is a major challenge in this field. How to extract appropriate features or fuse different features will have a significant impact on the emotion computing model. To address this problem, the present invention first extracts different time domain features based on the characteristics of each physiological signal, and then performs wavelet transform and Hilbert-Huang transform on the signal to obtain the emotional characteristics of each physiological signal at different scales and different frequency components. A convolutional neural network is then used to select and fuse the features extracted from multi-channel EEG and various peripheral physiological signals in the spatial domain to obtain the effective features most relevant to induced emotions. This is a multimodal physiological signal emotion recognition method that fuses time-frequency and spatial domain features. Summary of the invention

[0009] The present invention aims to make up for the deficiencies of the prior art in the research on the fusion of time-frequency and spatial domain features. With scientific thinking and rigorous attitude, the present invention proposes a multimodal physiological signal emotion recognition method that fuses time-frequency and spatial features. After extracting various time domain features based on the characteristics of various physiological signals, the present invention uses wavelet transform and Hilbert-Huang transform to perform time-frequency analysis on the signal, and uses the empirical mode decomposition method to decompose signals with different emotions to obtain different intrinsic mode functions. These mode functions are subjected to data analysis to obtain the emotional characteristics of various physiological signals at different scales and different frequency components. After that, a convolutional neural network is used to select and fuse the various features extracted from the multimodal signal in the spatial domain, so that more universal and effective emotional features can be obtained, thereby establishing a high-precision physiological signal emotion recognition method with good generalization ability and stable and excellent results.

[0010] The technical solution of the present invention is as follows:

[0011] A multimodal physiological signal emotion recognition method based on the fusion of time-frequency and space features is characterized in that time domain features are extracted based on the characteristics of various physiological signals, time-frequency features are extracted by using wavelet transform and Hilbert-Huang transform for time-frequency analysis, and features after time-frequency and space-domain fusion are obtained by using convolutional neural network.

[0012] Time domain features are mainly statistical features of signal changes, such as the mean, median, standard deviation, maximum and minimum values, the ratio and difference of the maximum and minimum values, etc., and then the above signal features are extracted after first-order difference and second-order difference calculation, and the same statistical features, such as the amplitude peak of the P wave, Q wave, R wave, S wave, T wave of ECG, HRV (heart rate) and RSP respiratory rate, are representative and can reflect the influence of changes in emotional state on the signal. The features extracted by wavelet analysis are the features of energy changes at different frequency scales after the signal is reconstructed by wavelet transform. The features extracted by Hilbert-Huang analysis are a series of intrinsic mode functions obtained after the signal is decomposed by empirical mode, and then the instantaneous amplitude, frequency and phase of these intrinsic mode functions are obtained by Hilbert transform. There are many manually extracted features, and there is a possibility of information duplication. The time-frequency and spatial domain fusion features are deep and shallow fusion features obtained after the manually extracted features are learned and screened by convolutional neural network, which not only reduces the number of features, but also has stronger universality and effectiveness.

[0013] The present invention first performs denoising and normalization processing on the data set, and uses Buttonworth filtering to set different filtering parameters according to the characteristics of different physiological signals. The energy of the ECG signal is mainly concentrated in 0.5-45Hz, and a 1-30Hz bandpass filter is used to remove high-frequency noise and baseline drift; the energy of the electromyography signal is mainly concentrated in 10-500Hz, and a 10Hz high-pass filter is used; the energy of the skin electrical signal is mainly concentrated in 0.02-0.2Hz, and a 0.3Hz low-pass filter is used; the energy of the respiratory signal is mainly concentrated in 0.05-2Hz, and a 2Hz low-pass filter is used; the main frequency component of the photoelectric pulse signal is within 20Hz, and a 20Hz low-pass filter is used; the energy of the electrooculogram signal is mainly concentrated in 0.02-0.2Hz, and a 2Hz low-pass filter is used; the energy of the electrooculogram signal is mainly concentrated in 0.05-2Hz, and a 20Hz low-pass filter is used; the energy of the electrooculogram signal is mainly concentrated in 0.02-0.2Hz, and a 2Hz low-pass filter is used; the energy of the respiratory signal is mainly concentrated in 0.05-2Hz, and a 2Hz low-pass filter is used; the main frequency component of the photoelectric pulse signal is within 20Hz, and a 20Hz low-pass filter is used; the energy of the electrooculogram signal is mainly concentrated in 0.02-0.2Hz, and a 2 ... The energy of EEG signals is mainly concentrated in 0.05-30Hz, using a 30Hz low-pass filter; the energy of EEG signals is mainly concentrated in 0.2-45Hz, of which the 0.2Hz-3Hz band is mainly generated in the deep sleep stage, the 3Hz-8Hz band is mainly generated in the deep relaxation stage, the 8Hz-12Hz band is mainly generated in the relaxation stage, the 12Hz-27Hz band is mainly generated in the concentration stage, and the 27Hz-45Hz band is mainly generated in the high concentration stage. Since the subjects are not in the deep sleep stage, the last four frequency bands are mainly concerned.

[0014] Then there is the convolutional neural network module. The neural network structure of the present invention has three small modules. The first two modules are convolution modules. The first convolution module uses a 3×3 convolution kernel for feature extraction. The convolution layer is followed by a maximum pooling layer with a pooling window of 2×2 and a step size of 2. The second convolution module uses a 5×5 convolution kernel (padding is 2) for feature extraction. The convolution layer is followed by a maximum pooling layer with a pooling window of 2×2 and a step size of 2. Each module uses ReLU as the activation function.

[0015] Finally, the recognition and classification module inputs the fused features into the fully connected layer, uses cross entropy to calculate the loss value, uses stochastic gradient descent as the optimizer, and outputs the recognition results based on the predicted probabilities of four different emotions.

[0016] The present invention has the following beneficial effects:

[0017] The present invention provides a multimodal physiological signal emotion recognition method that fuses time-frequency and spatial domain features. The time and frequency domain features are selected and fused in the spatial domain, and the most universal and effective emotion features are obtained from a large number of features, thereby constructing a high-precision physiological signal emotion recognition network model with good generalization ability and stable and excellent effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Workflow diagram of multimodal physiological signal emotion recognition method based on time-frequency-spatial domain feature fusion;

[0019] Figure 2 The convolutional neural network structure diagram of the present invention; DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0021] The operating environment required for this example is pytorch1.4, ubuntu16.04, cuda10, and the server is V10016GB. The parameters used for training are batch_size of 64, 200 epochs, and the initial learning rate is set to 0.01. During the training process, the strategy of dynamically adjusting the learning rate is adopted. When the model performance (verification accuracy) does not improve, the learning rate is automatically reduced with the strategy of learning_rate*0.9. It takes 3.5 hours to complete a whole task.

[0022] This example provides a multimodal physiological signal emotion recognition method that integrates time-frequency and spatial features. The workflow diagram is as follows: Figure 1As shown in the figure, the first step is to denoise and extract features from the data set. The Buttonworth filter is used to denoise each physiological signal and then extract features. The mean, mean square error, median, standard deviation, minimum, maximum, maximum interval, maximum resolution, spectrum average, statistical characteristics of first-order difference, statistical characteristics of second-order difference, and statistical characteristics of each frequency scale coefficient after wavelet transformation reconstruction, as well as the instantaneous amplitude, frequency and phase of each intrinsic mode function after Hilbert-Huang transformation are extracted from the ECG signal, EMG signal, EO signal, skin electrical signal, respiratory signal, and photoelectric pulse signal. In addition, the amplitude peak and HRV of P wave, Q wave, R wave, S wave, and T wave are extracted from the ECG signal, and the statistical characteristics of the respiratory interval and respiratory rate are extracted from the respiratory signal. The mean, mean square error, median, standard deviation, minimum value, maximum value, maximum interval, maximum resolution, spectral average, statistical characteristics of first-order differences, statistical characteristics of second-order differences of the EEG signals, statistical characteristics of each frequency scaling coefficient after wavelet transform reconstruction, and the instantaneous amplitude, frequency and phase of each intrinsic mode function after Hilbert-Huang transform were extracted respectively.

[0023] Then comes the construction of the convolutional neural network structure. The neural network structure of the present invention has three small modules. The first two modules are convolution modules. The first convolution module uses a 3×3 convolution kernel for feature extraction. The convolution layer is followed by a pooling window of 2×2 and a maximum pooling layer with a step size of 2. The second convolution module uses a 5×5 convolution kernel (padding is 2) for feature extraction. The convolution layer is followed by a pooling window of 2×2 and a maximum pooling layer with a step size of 2. Each module uses ReLU as an activation function, so that the network has more nonlinear transformations, further enhancing the network's ability to learn features. The first convolution block uses a small 3×3 convolution kernel to obtain more features. The second convolution block uses a 5×5 convolution kernel, padding is 2, and the "same" format is used to fill the feature matrix boundary with 0, and the features are reduced through the pooling layer. Using convolution kernels of different sizes for convolution operations can obtain different information of input features, thereby obtaining richer features. After that, the feature matrix is ​​reduced in dimension and converted into a one-dimensional feature vector, which is input into two connection layers in sequence. The dimension of the input vector of the last fully connected layer is set according to the label type of the classification task. Finally, the probability value of each type of label is weighted by the Softmax classifier, and the label with the maximum probability value is output. Figure 2 As shown in the figure, after the 8×32 feature passes through the first convolution module, the feature is expanded to 8*32*32, and then after passing through the second convolution module, the feature becomes 4*16*64. After the flattening transition, the feature passes through two fully connected layers and is input into the loss function to obtain the final recognition accuracy.

[0024] Obviously, the described embodiment is only one possible embodiment of the present invention, rather than all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of the present invention.

Claims

1. A multimodal physiological signal emotion recognition method that fuses time-frequency and space features, characterized in that various time domain features are extracted based on the characteristics of various physiological signals, and the signal is analyzed in time and frequency using wavelet transform and Hilbert-Huang transform. The empirical mode decomposition method is used to decompose signals with different emotions to obtain different intrinsic mode functions, and these mode functions are subjected to data analysis to obtain the emotional characteristics of various physiological signals at different scales and different frequency components. Then, a convolutional neural network is used to select and fuse the various features extracted from the multimodal signals in the spatial domain, thereby obtaining more universal and effective emotional features, making the emotional state classification results more accurate. The specific steps include: First, the data set is denoised and normalized, and different filtering parameters are set according to the characteristics of different physiological signals using the Buttonworth filter. Then, various time domain features are extracted based on the characteristics of various physiological signals, and the time-frequency analysis of the signals is performed using wavelet transform and Hilbert-Huang transform. The empirical mode decomposition method is used to decompose signals with different emotions to obtain different intrinsic mode functions. These mode functions are analyzed to obtain the emotional characteristics of various physiological signals at different scales and different frequency components. Then there is the convolutional neural network module. The neural network structure of the present invention has three small modules. The first two modules are convolution modules. The first convolution module uses a 3×3 convolution kernel for feature extraction. The convolution layer is followed by a maximum pooling layer with a pooling window of 2×2 and a step size of 2. The second convolution module uses a 5×5 convolution kernel (padding is 2) for feature extraction. The convolution layer is followed by a maximum pooling layer with a pooling window of 2×2 and a step size of 2. Each module uses ReLU as the activation function. Finally, the recognition and classification module inputs the fused features into the fully connected layer, uses cross entropy to calculate the loss value, uses stochastic gradient descent as the optimizer, and outputs the recognition results based on the predicted probabilities of four different emotions.

2. The multimodal physiological signal emotion recognition method based on time-frequency-space feature fusion according to claim 1 is characterized in that Based on the characteristics of various physiological signals, various time domain features are extracted, and wavelet transform and Hilbert-Huang transform are used to perform time-frequency analysis on the signal. Then, convolutional neural network is used to select and fuse features in the spatial domain for the composed feature matrix. More universal and effective emotional features are obtained, which improves the accuracy of emotional state classification to a certain extent.

Images