Emotion recognition method based on deep neural network with dense inter-modal interaction
Through the dense interaction between modes, deep neural networks to process and splice EEG signals, eye tracking signals and eye image signals, the problem of emotional models relying on specific modal data in the existing technology is solved, and adjustment is achieved according to actual conditions Modal number of emotions recognition improves recognition accuracy and flexibility.
Patent Information
- Application Number
- CN202111356761.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-16
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2041-11-16
AI Technical Summary
The emotion models in the prior art rely on specific modal data and cannot adjust the number of modals according to actual conditions, resulting in a decrease in recognition accuracy in practical applications.
Using dense interaction deep neural network between modes, through the design of dense interaction neural network, intensive interaction processing is performed on EEG signals, eye tracking signals and eye image signals, to generate feature matrix, and to splice and classify them by identifying emotional neural networks. , realize emotional recognition.
This method does not depend on a specific modal or the number of modals. The types and quantities of modals can be adjusted according to actual conditions, improving the accuracy and flexibility of the emotion recognition system.
Smart Images
Figure CN114091599B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in the field of electroencephalogram signals, specifically to an inter-modal dense interaction deep neural network emotion recognition method that supports multi-modal data fusion of any number of modalities. Background Art
[0002] In recent years, researchers have used a variety of data sources to identify emotion, including EEG, eye tracking, facial expressions, and language. Various modal data have their own characteristics in building emotion models. Among them, the use of EEG signals to build emotion models has high accuracy and good stability, and has been widely recognized by domestic and foreign researchers, and in-depth research has been carried out. However, in practical applications, the process of collecting EEG signals is cumbersome and costly. Using other signals, such as facial expressions or language conversations, is convenient to collect, but the ability to distinguish is weak and easy to disguise. Emotion recognition research using multiple modal data fusion is gaining more and more attention. Research results from different laboratories around the world show that signals of different modalities collected simultaneously reflect different aspects of emotions. Therefore, using multimodal data fusion methods for emotion recognition can greatly improve the accuracy of the emotion recognition system.
[0003] At present, various emotion recognition methods based on multimodal data fusion have been designed for various modal data. However, although there are many emotion recognition neural network models based on multimodal data fusion, the design of most models depends on a certain modality of data itself and cannot be changed to other modalities. In addition, the design of most models depends on the number of modalities and cannot increase or decrease the number of modalities according to actual conditions. Summary of the invention
[0004] In order to solve the bottleneck problem that the emotion models used in the prior art rely on certain specific modal data and do not support adding or deleting the number of modalities, the present invention proposes an inter-modal dense interaction deep neural network emotion recognition method, which uses EEG signals, eye tracking signals and eye image signals as inputs. Through the original dense interaction neural network design, it can be adjusted according to the actual situation in actual applications without changing the model structure and without relying on certain modalities or the number of modalities. The specific type and number of modalities can be adjusted according to the actual situation during actual use.
[0005] The present invention is achieved through the following technical solutions:
[0006] The present invention relates to an inter-modal dense interaction deep neural network emotion recognition method. After preprocessing the data of each modality, dense interaction processing is performed on the data of each modality through a dense interaction neural network to obtain a feature matrix; then the data matrix of each modality is spliced through the emotion recognition neural network, that is, the output data of the last interaction layer is spliced, and finally classified to obtain the emotion category, that is, the classification result.
[0007] The data of each modality include: EEG signals, eye movement feature signals and eye images.
[0008] The preprocessing includes: for EEG signals, extracting their differential entropy features to obtain the input matrix of modality A; for eye movement feature signals, using the output data of the SMI eye tracker to classify according to the items in Table 1 to obtain the input matrix of modality B; for eye images, passing the eye movement images into the ResNet convolutional neural network, and then using the LSTM network to further extract features, and finally using the fully connected layer to output the input matrix of modality C.
[0009] Table 1
[0010]
[0011] The dense interactive neural network described above optimizes the calculation parameters therein by machine learning methods, i.e. training. In practical applications, firstly, training is carried out for a certain subject, by watching relevant video materials (comedy, tragedy, horror film, documentary, etc.) to achieve the effect of inducing emotions, and at the same time, collecting data of each modality, passing it into the model, and optimizing the training model by known emotional state, which is the training stage. In the testing or commissioning stage, the optimized model is used to collect data of the same modality of the subject, and the same calculation process is adopted to realize the recognition of emotional state.
[0012] The dense interactive neural network includes multiple stacked interactive layers, which preprocess the data matrix of each modality As input, output a matrix of the same size The i-th dense interaction layer of the stack is As input, output a matrix of the same shape Each interaction layer includes: a fusion unit, several parallel MLP layers and their softmax layers, and an output unit, where: the fusion unit calculates the fusion matrix Several parallel MLP layers and their softmax layers calculate the weights of each modality The output unit calculates the output matrix
[0013] In the dense interactive neural network, each mode is linked and fused when calculating the fusion matrix, and then the weight of the mutual influence between the modes is calculated through the softmax function of the neural network. Finally, the input data is weighted according to the influencing factors of other modes. Therefore, this calculation process is called the interaction layer. By cleverly using the matrix calculation rules and designing the calculation process, the interaction layer does not change the input matrix. shape, so that multiple interaction layers can be stacked to make the modal interaction denser to further improve the accuracy.
[0014] Preferably, in practical applications, k-Augmentation can also be performed on the input, that is, k columns of zero matrices are concatenated in the second dimension and the k columns are truncated before output. This method can store some useless information in the k columns and truncate them before output to achieve a filtering effect.
[0015] The emotion recognition neural network splices the feature matrix of each modality, that is, splices the output data of the last interaction layer, and finally passes it into the softmax classifier of the neural network to output the emotion category, which specifically includes:
[0016] 1) Fusion of each modality data to obtain Among them: concatenate means to concatenate multiple matrices according to a certain dimension, which is the second dimension in this method.
[0017] 2) Calculate the emotion category result = softmax(MLP(ALL)), where: MLP refers to a multi-layer perceptron in a neural network, which consists of several fully connected layers followed by nonlinear activation functions. In this method, there are three. The above-mentioned ReLU and softmax are both nonlinear activation functions.
[0018] The optimization / training mentioned above refers to: collecting training data, which consists of known emotional states (model output data) and input data of three modes (EEG signals, eye images and eye movement features), using the known emotional states in the training data to gradually correct the parameters in the calculation model, so as to achieve the purpose of input data passing through the calculation of the model and output data approaching the known real emotional state. The correction method is: through the reverse gradient propagation algorithm, that is, according to the known emotional state and the calculation process, the inverse is calculated step by step from back to front, and the gradient value of a certain calculation parameter in the calculation model from the optimal value of the parameter is obtained, and the parameter is corrected according to the obtained gradient. Through iterative calculation of a large amount of data, the model parameters are gradually corrected until the model parameters tend to be stable, that is, the gradient value of the parameter tends to 0.
[0019] Technical Effects
[0020] Compared with the prior art, the design of the model is strongly dependent on the type and number of the selected modalities. Once a change occurs, the recognition accuracy of the model is greatly reduced. The present invention uses a dense interactive neural network and does not rely on certain modalities or the number of modalities. In actual use, the specific type and number of modalities can be adjusted according to actual conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a schematic diagram of the process flow of the present invention;
[0022] Figure 2 A schematic diagram of a dense interaction layer in a dense interactive multimodal emotion recognition network of the present invention;
[0023] Figure 3 Schematic diagram of the dense interactive multimodal emotion recognition network structure in the present invention; DETAILED DESCRIPTION
[0024] Figure 1 This embodiment relates to an inter-modal dense interaction deep neural network emotion recognition method that supports multi-modal data fusion of any number of modalities. This embodiment includes a model training stage and an actual application stage in actual application. The calculation process of the two stages is basically the same. After preprocessing the data of each modality, the dense interaction neural network is used to perform dense interaction processing on the data of each modality to obtain a feature matrix; then the data matrix of each modality is spliced through the emotion recognition neural network, that is, the output data of the last interaction layer is spliced, and finally classified to obtain the emotion category, that is, the classification result.
[0025] The model training phase includes:
[0026] Step 1: For a certain subject, first collect training data by watching five types of emotional stimulus videos: happiness, sadness, fear, disgust and neutrality, so as to induce the corresponding emotions of the subjects (the emotions are known outputs). During the collection process, the subject wears an SMI eye tracker, and the SMI eye tracker is used to capture eye images and collect eye movement data as input signals.
[0027] Step 2: Preprocess the training data and input it into the neural network, and modify the parameters in the model according to the known emotional state, that is, model training. After the neural network parameters are stable, save the network for actual use.
[0028] The actual application stage is as follows: the subjects collect the same modal data as in the training, pass the input signal into the model modified in the training stage, and calculate and output the emotional state in exactly the same way.
[0029] The specific implementation process of the experiment is as follows: 16 healthy subjects (10 males and 6 females) were given three times to participate in the experiment, each time about three to four days apart. Each time, they watched five types of emotional stimulation videos, including happiness, sadness, fear, disgust and neutrality, to induce the corresponding emotions of the subjects. There were three clips for each emotion, and each clip was about 2-4 minutes. During the viewing process, they wore a 62-lead EEG cap and an SMI eye tracker. The EEG data of the subjects were collected according to the 10-20 international standard potential distribution, and the eye images and eye movement data were captured by the SMI eye tracker.
[0030] The experiment was set up into four groups, namely: trimodal emotion recognition using EEG signals, eye movement signals and eye images; bimodal emotion recognition using EEG signals and eye movement signals; bimodal emotion recognition using EEG signals and eye images; and bimodal emotion recognition using eye movement signals and eye images.
[0031] The use and performance evaluation of experimental data are as follows: the data of each of the 16 subjects participating in the experiment three times are divided into training set and test set in a ratio of 2:1. The training set is used to optimize the calculation model, and the test set is used to simulate the actual application. The result indicators are the recognition accuracy and variance of the test set. The recognition accuracy of the subject in each experiment is the average of the results of the three-fold cross-validation of the division method of the training set and the test set. The recognition accuracy of the subject is the average of the results of the three experiments of the subject.
[0032] The parameter adjustment method adopted in the experiment is the Adam gradient descent method.
[0033] The parameters of the experiment are: in the second step, the parameter k in the k-Augmentation method, the number of interaction layers l in the dense interaction neural network. In the third step, the width of each layer in the three-layer MLP of the neural network for emotion recognition, m1, m2, m3. Among them, the range of parameter k is 1-6, the range of parameter l is 1-6, the range of parameters m1 and m2 is 1024, 2048, 3072, and the range of parameter m3 is 64, 128, 256.
[0034] After specific actual experiments, the results obtained under the above experimental parameter settings include:
[0035] 1) For the trimodal emotion recognition experiment using EEG signals, eye movement signals and eye images, the average accuracy was 82.04% with a standard deviation of 6.03.
[0036] 2) For the bimodal emotion recognition experiment using EEG signals and eye movement signals, the average accuracy was 80.79% and the standard deviation was 5.34.
[0037] 3) For the bimodal emotion recognition experiment using EEG signals and eye images, the average accuracy was 74.44% with a standard deviation of 7.15.
[0038] 4) For the bimodal emotion recognition experiment using eye movement signals and eye images, the average accuracy is 69.95% and the variance is 7.13.
[0039] Compared with the prior art, the present invention can still achieve a higher accuracy under limited, usable data conditions.
[0040] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.
Claims
1. A method for emotion recognition using a densely interactive deep neural network between modalities, characterized in that: After preprocessing the data of each modality, the dense interactive neural network is used to perform dense interactive processing on the data of each modality to obtain a feature matrix; then the data matrix of each modality is spliced through the emotion recognition neural network, that is, the output data of the last interactive layer is spliced, and finally classified to obtain the emotion category, that is, the classification result, where The dense interactive neural network includes multiple stacked interactive layers, which preprocess the data matrix of each modality As input, output a matrix of the same size The i-th dense interaction layer of the stack is As input, output a matrix of the same shape Each interaction layer includes: a fusion unit, several parallel MLP layers and their softmax layers, and an output unit, where: the fusion unit calculates the fusion matrix Several parallel MLP layers and their softmax layers calculate the weights of each modality The output unit calculates the output matrix The data of each modality include: EEG signals, eye movement feature signals and eye images.
2. The inter-modal dense interaction deep neural network emotion recognition method according to claim 1 is characterized in that: The preprocessing includes: for EEG signals, extracting their differential entropy features to obtain the input matrix of modality A; for eye movement feature signals, using the output data of the SMI eye tracker for classification to obtain the input matrix of modality B; for eye images, passing the eye movement images into the ResNet convolutional neural network, then using the LSTM network to further extract features, and finally using the fully connected layer to output the input matrix of modality C; The classification is specifically performed according to the following output characteristics: Pupil diameter: mean, standard deviation, and PSD for four frequency bands: 0-0.2 Hz, 0.2-0.4 Hz, 0.4-0.6 Hz, and 0.6-1 Hz; Dispersion: mean, standard deviation; Blink duration: mean, standard deviation; Saccades: mean saccade duration, standard deviation, and saccade amplitude; Eye activity statistics: blink frequency, fixation frequency, maximum fixation duration, total number of fixation dispersions, maximum value of fixation dispersion, saccade frequency, average saccade duration, average saccade amplitude, average saccade latency.
3. The inter-modal dense interaction deep neural network emotion recognition method according to claim 1 is characterized in that: The dense interactive neural network described above optimizes the calculation parameters thereof through machine learning methods, specifically: training is performed for a certain subject, by inducing emotions by watching relevant video materials, while collecting data of each modality and transmitting them into the model, and optimizing the training model through known emotional states. This is the training phase, and in the testing or commissioning phase, the optimized model is used to collect data of the same modality of the subject, and the same calculation process is adopted to realize the recognition of the emotional state.
4. The inter-modal dense interaction deep neural network emotion recognition method according to claim 1 is characterized in that: Perform k-Augmentation on the input, that is, concatenate k columns of zero matrices in the second dimension and truncate the k columns before output. This method can store some useless information in the k columns and truncate them before output to achieve a filtering effect.
5. The inter-modal dense interaction deep neural network emotion recognition method according to claim 3 is characterized in that: The optimization refers to: collecting training data, which consists of known emotional states, that is, model output data and input data of EEG signals, eye images and eye movement features, using the known emotional states in the training data to gradually correct the parameters in the calculation model, so as to achieve the purpose of input data passing through the calculation of the model and output data approaching the known real emotional state, and the correction method is: through the reverse gradient propagation algorithm, that is, according to the known emotional state and the calculation process, the inverse is calculated step by step from back to front to obtain the gradient value of a certain calculation parameter in the calculation model from the optimal value of the parameter, and the parameter is corrected according to the obtained gradient, and the model parameters are gradually corrected through iterative calculation of a large amount of data until the model parameters tend to be stable, that is, the gradient value of the parameter tends to 0.