Multi-modal fusion feature emotion recognition method

Abstract

The invention discloses a multi-modal fusion feature emotion recognition method, and particularly relates to the technical field of biological recognition. Comprising the following steps: S1, establishing an emotion induction experiment task platform by screening or designing VR scene tasks; s2, collecting physiological signals of the testee under different emotions based on an emotion induction experiment task platform, and preprocessing the physiological signals; s3, performing feature extraction and dimension reduction processing on the preprocessed physiological signals; and S4, establishing an emotion classification and prediction model for the features subjected to dimension reduction processing through a recurrent neural network. By adopting the technical scheme of the invention, the problems of low recognition rate and poor robustness of an emotion recognition model in existing emotion recognition are solved, and the safety of man-machine interaction operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of biometric technology, and in particular to a multi-modal fusion feature emotion recognition method. Background Art

[0002] With the rapid development of the Internet, virtual reality, and mobile computing, new challenges and higher requirements have been raised for human-computer interaction technology, and many new opportunities have also been provided. At present, the "people-centered" interaction design principle is the main goal of developing a new generation of human-computer interaction. Therefore, if the emotional state of people during the operation can be effectively identified and timely warned, it can effectively reduce human errors in the human-computer interaction process and ensure the safety of the operation, which has important practical significance.

[0003] Emotion recognition research has long been the focus of researchers in various fields. A large amount of research work has been carried out in various related fields. There are two main research methods: non-physiological signal and physiological signal emotion recognition. Non-physiological signal recognition mainly includes facial expressions, voice intonation, text, body movements and other related recognition features. Although these methods have made great progress and breakthroughs in the field of emotion recognition, there are still some problems: (1) These most commonly used emotion recognition methods are based on individual appearance characteristics, which can be disguised by the individual's subjective will and sometimes cannot truly reflect the individual's current emotional state; (2) Each individual has different characteristics, such as facial structure characteristics, timbre characteristics, and text usage habits. These characteristics increase the difficulty of emotion recognition and lead to poor generalization ability of the recognition model; (3) The requirements for data are high, but data extraction is easily affected by the external environment. For example, images are easily affected by light and occlusion, and speech is easily affected by ambient noise.

[0004] According to Canon's theory, physiological signals are the result of the activities of the central nervous system and the peripheral nervous system. The nervous system controls and regulates physiological activities in the unconscious state of human beings, that is, physiological signals are difficult to be controlled by human subjective consciousness. Therefore, emotion recognition based on physiological signals has the greatest objectivity. At home and abroad, some scholars have also studied the role of EEG, respiration, ECG, pulse wave, and electromyography in emotion recognition. At the same time, they have also studied the comprehensive facial electromyography, EEG, ECG, pulse, respiration and skin conductance signals. Through genetic algorithms, ant colony algorithms, and particle swarm algorithms for feature extraction, Fisher discriminant is used as a classification algorithm to identify six emotions of happiness, surprise, disgust, sadness, anger, and fear, and achieved good results. At present, the research on emotion recognition is still mainly focused on the use of single-modal data for emotion recognition. Although certain results have been achieved, there are still shortcomings such as low recognition rate and poor robustness of emotion recognition models. Summary of the invention

[0005] The present invention aims to provide a multimodal fusion feature emotion recognition method, which solves the problems of low recognition rate and poor robustness of emotion recognition models in existing emotion recognition.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a multimodal fusion feature emotion recognition method, comprising the following steps:

[0007] S1. Build an emotion-inducing experimental task platform by screening or designing VR scene tasks;

[0008] S2, based on the emotion induction experimental task platform, collect the physiological signals of the subjects under different emotions and pre-process the physiological signals;

[0009] S3, extracting features and reducing dimension of the preprocessed physiological signals;

[0010] S4. Use the features after dimensionality reduction through a recurrent neural network to establish a sentiment classification and prediction model.

[0011] Furthermore, the construction method of step S1 is as follows:

[0012] S1.1. Determine the target emotion model: select six emotion states by selecting different valence and arousal levels: negative (select depression, stress, fear), positive (select sleepiness, relaxation, excitement);

[0013] S1.2, Emotion induction methods and material screening: Statistical data on the adaptability and effectiveness of different VR materials in inducing target emotions, and screen effective emotion inducing materials by setting certain thresholds;

[0014] S1.3. Cognitive experimental task design: Select the 2-back experimental paradigm;

[0015] S1.4. Experimental paradigm design: The experimental paradigm consists of multiple VR clips. After experiencing each VR video, the subjects are required to quickly make a self-assessment based on their feelings about the video they just watched, and then perform cognitive experimental tasks to examine changes in their operational performance.

[0016] Furthermore, in step S1.4, the subjects are relaxed before the new VR video starts, and data of the subjects in the relaxed state and their cognitive operation performance in the relaxed state are collected.

[0017] Furthermore, the preprocessing method of the physiological signal in step S2 is as follows:

[0018] S2.1, denoising the EEG signal;

[0019] S2.2, denoising the ECG signal;

[0020] S2.3, use a smoothing filter to remove high-frequency glitches in the electrodermal signal;

[0021] S2.4, baseline signal removal.

[0022] Furthermore, the denoising method of step S2.1 is as follows:

[0023] S2.1.1. Based on the frequency range of normal EEG signals, use a filter to perform 0.5-40 Hz bandpass filtering and 50 Hz notch processing on the EEG;

[0024] S2.1.2. Use independent component analysis to remove oculoscopic artifacts.

[0025] Furthermore, the denoising method of step S2.2 is as follows:

[0026] S2.2.1. Filter out by designing 50Hz and 100Hz power frequency notch filters;

[0027] S2.2.2, use wavelet transform denoising method to remove burrs;

[0028] S2.2.3. Use the median filter method to extract the shape of the baseline drift, and then subtract the extracted baseline from the original signal.

[0029] Furthermore, the specific method of step S2.4 is as follows: collect the baseline physiological signal of each subject in a relaxed state, and then subtract each person's baseline physiological signal from the physiological signal in different emotional states to obtain the actual physiological change value of the subject after emotion induction.

[0030] Furthermore, the feature extraction method of step S3 is as follows:

[0031] Emotion recognition feature extraction of EEG signals: The emotion recognition features of EEG signals are extracted using the wavelet packet transform method;

[0032] Extraction of emotion recognition features of ECG signals: Extract emotion recognition features of ECG signals using QRS wave groups;

[0033] Emotion recognition feature extraction of skin electrodermal signals: Extract skin electrodermal signal sequence: skin conductance, first-order difference and second-order difference of skin conductance, and on this basis, calculate the time domain feature parameters related to emotions.

[0034] Furthermore, the feature dimension reduction method of step S3 is as follows:

[0035] S3.1. Analyze the correlation between each attribute and the category of the final data point through the one-way variance analysis algorithm, eliminate irrelevant or low-correlation attribute features, and achieve feature dimensionality reduction in attribute selection;

[0036] S3.2. Use the principal component analysis algorithm on linear transformation to project the original data onto a low dimension that can better distinguish these attribute features and categories through mean comparison and variance analysis, thereby achieving feature dimensionality reduction.

[0037] Furthermore, the specific method of step S4 is as follows: sort the LSTM prediction models from high to low according to the percentage of prediction results, select the models with the top accuracy, and use the prediction accuracy of each model as the weight of the prediction result category. By counting the corresponding category weights of each record, the final judgment result of the record is determined.

[0038] Compared with the prior art, this solution has the following beneficial effects:

[0039] 1. This scheme will use multimodal physiological signal fusion of EEG signals reflecting changes in the central nervous system and peripheral physiological signals (electrocardiogram, skin conductance) of changes in the autonomic nervous system as research clues for emotion recognition, and use more immersive and interactive VR scene tasks as a means of emotion induction to establish an effective emotion model. On this basis, multi-sample, high-quality physiological data will be collected for analysis. By exploring the intrinsic connection between multimodal physiological signals and emotion processing and expression, and the interactive relationship between emotion and cognition, and extracting effective feature subsets, an emotion recognition model based on multi-source physiological signals will be established using deep learning methods.

[0040] 2. This solution relies on high-quality physiological data as the basis for emotion recognition and analysis. Therefore, how to obtain high-quality physiological data is very critical. Compared with traditional emotion induction methods such as pictures, music, and videos, this solution introduces VR technology, which not only incorporates the subject's visual, auditory and other senses and cognition into the virtual situation, but also can solve the direct interaction between the subject and the environment, increase their "sense of presence", and the social pressure experienced by the subject is greater, the stimulus response is stronger, and emotions closer to daily life can be induced.

[0041] 3. In the prior art, research on emotion recognition mostly relies on physical signals of the human body, such as facial expressions, voice, gestures, and postures. However, in the process of social interaction, people can easily hide their true emotions by controlling facial expressions or speech, so the reliability cannot be guaranteed. This solution uses physiological signals with high reliability to identify emotions, and avoids the problem of low accuracy caused by the evaluation of a single physiological signal. It selects three physiological signals, namely, EEG signals reflecting emotional changes in the central nervous system and peripheral physiological signals (electrocardiogram and skin conductance) reflecting emotional changes in the autonomic nervous system, to comprehensively identify emotions. The emotion recognition of multimodal physiological signals achieves good information complementarity, greatly improving the accuracy of emotion recognition.

[0042] 4. This scheme uses multi-physiological signal fusion for emotion recognition, which is easy to cause feature redundancy. Some features have little or no effect on emotion recognition. These features will affect the classification effect of the classifier. Therefore, it is often necessary to select and reduce the dimension of features before classification. Based on the traditional dimensionality reduction algorithm, this scheme proposes a feature dimensionality reduction method that combines one-way variance analysis with principal component analysis. This method can effectively reduce the feature dimensionality of high-dimensional attributes while ensuring a certain attribute contribution of the original valid data, and meet the feature dimensionality requirements of subsequent emotion learning and emotion recognition.

[0043] 5. Regarding the establishment of emotion classification models, deep learning methods are superior to traditional machine learning methods, and the learned emotion recognition models are more accurate. As one of the deep learning algorithms, recurrent neural networks (RNN) have been widely used in many fields. The LSTM algorithm adds short-term memory function fragments to the RNN algorithm, and its algorithm execution results are better than the traditional RNN algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of this embodiment;

[0045] Figure 2 is the emotional two-dimensional space model in this embodiment;

[0046] Figure 3 is a schematic diagram of the stimulus presentation sequence of the 1 / 2 / 3-back paradigm of letters in this embodiment;

[0047] Figure 4 is the experimental paradigm design and flow chart of this embodiment;

[0048] Figure 5 is a diagram of the placement of different physiological sensors on the human body in this embodiment;

[0049] Figure 6 is the emotion self-rating scale in this embodiment;

[0050] Figure 7 is a comparison diagram before and after removing electrooculogram artifacts in this embodiment;

[0051] Figure 8 is a comparison diagram of the ECG signal before and after filtering in this embodiment;

[0052] Fig. 9 is a comparison diagram of the skin electrical signal before and after filtering in this embodiment;

[0053] Fig.10 : is a graph of the corresponding relationship between each node of the EEG wavelet packet decomposition and four EEG rhythm frequency bands in this embodiment;

[0054] Fig.11 is a diagram of the composition of the ECG signal in this embodiment;

[0055] Fig.12 is a comparison diagram of the ECG signal before and after filtering in this embodiment;

[0056] Fig.13 is the R wave check flow chart of this embodiment;

[0057] Fig.14 is a flow chart of feature dimensionality reduction processing in this embodiment;

[0058] Fig.15 It is a flow chart of the LSTM algorithm in this embodiment. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below through specific embodiments:

[0060] Example

[0061] like Figure 1 As shown, a multimodal fusion feature emotion recognition method includes the following steps:

[0062] S1. Build an emotion induction experimental task platform by screening or designing VR scene tasks. The specific construction method is as follows:

[0063] S1.1. Determine the target emotion model: Select a two-dimensional space model, which consists of valence and arousal, such as Figure 2 As shown. Among them, valence is the degree of pleasure felt by an individual, ranging from unpleasant (negative) to pleasant (positive); arousal is the intensity of the emotion felt by an individual, ranging from passive (low) to active (high). This embodiment considers the two dimensions of valence and arousal, and combines the actual application scenarios to select six emotional states that can better represent natural attributes and universal recognition: negative dimensions (depression, stress, fear) and positive dimensions (sleepy, relaxed, excited).

[0064] S1.2, Emotion induction means and material screening: Statistics on the adaptability and effectiveness of different VR materials in inducing target emotions, and screen effective emotion inducing materials by setting certain thresholds. In this embodiment, adaptability refers to the percentage of subjects reporting that a given stimulus appropriately induces the expected emotion; effectiveness is determined by self-report results, and subjects are asked to verbally evaluate the intensity of the emotions they feel from the five levels of stimulation in the scale.

[0065] The specific screening method for emotion-inducing materials is as follows: the intensity and differentiation of target emotions are used as evaluation indicators for the emotions induced by video materials. Emotional intensity refers to the score given by the subjects on the intensity of emotions during the subjective evaluation process. The higher the score, the stronger the emotion induced. In the early screening process of video clips, although the emotional theme is clear, other emotional dimensions are also mixed. Therefore, in addition to analyzing the differences in emotions in different dimensions, the hit rate is introduced to characterize the differentiation of emotion induction. Specifically, it refers to the ratio of the number of subjects whose scores on target emotions are higher than those on non-target emotions by ≥1 point to the total number of subjects. The larger the ratio, the more concentrated and single the target emotion induction is.

[0066] S1.3, Cognitive experimental task design: By designing cognitive experimental tasks, we can test the effects of different emotional states and different emotional intensities on cognitive operations and strategies. This example selects the 2-back experimental paradigm, which represents a short-term memory task, to explore the effects of different emotional states on memory cognition.

[0067] For the n-back experimental paradigm, n is an uncertain value, which represents different:

[0068] Load level, the value can be 0, 1, 2, 3...N, n = 0, the subject is required to compare whether the current content is the same as the target stimulus; n = 1, the subject is required to compare whether the current stimulus is the same as the previous adjacent stimulus; n = 2 or 3, the subject is required to compare whether the current stimulus is the same as the second or third stimulus from it...Numbers, letters or words can be used as stimulus content, see Figure 3 shown.

[0069] S1.4、Experimental paradigm design: Figure 4As shown in the figure, the experimental paradigm consists of multiple VR clips. After experiencing each VR video, the subjects need to quickly make a self-assessment based on their feelings about the video just watched, and then perform cognitive experimental tasks to test the changes in their operational performance. In order to avoid the subjects failing to adjust their state between experiencing different VR videos, the subjects were relaxed before the new VR video started (such as listening to soothing music to recover). At the same time, for comparison, the data of the subjects in a relaxed state and their cognitive operational performance in a relaxed state were collected. Three physiological signals, self-assessment values, and 2-back operational performance values ​​were collected throughout the experiment.

[0070] In this embodiment, during the formal experiment, the subjects were required to self-evaluate from the two aspects of validity and arousal (intensity) based on their own feelings after experiencing each VR segment, combined with the subjective evaluation scale, such as Figure 5 As shown. At the same time, after the self-assessment, the subjects also need to quickly perform the 2-back task, and the software automatically records the subjects' operation performance (average reaction time RT and operation error rate ACC). Based on the statistics and analysis of subjective self-assessment data and operation performance data, the effectiveness of the emotion induction model is finally verified based on the self-assessment emotion label and intensity, combined with the changes in operation performance.

[0071] S2. Based on the emotion induction experimental task platform, the physiological signals of the subjects under different emotions are collected and pre-processed. In this embodiment, the VR virtual reality technology equipment is used to induce emotions, and a multi-channel physiological acquisition system is used to realize the synchronous acquisition of three physiological signals of EEG, ECG and skin electricity during the experiment. The position of each signal acquisition sensor (electrode) is as follows: Figure 6 shown.

[0072] At the same time, the requirements for the subjects are as follows: the male-female ratio is 1:1, and the age is between 23 and 35 years old. All subjects are in good health, without any cardiovascular disease or mental illness, and have no record of drinking stimulating beverages such as alcohol and coffee within 24 hours before the experiment.

[0073] The preprocessing method for the collected physiological signals is as follows:

[0074] S2.1. De-noising the EEG signal. The de-noising method is as follows:

[0075] S2.1.1. Based on the frequency range of normal EEG signals, use a filter to perform 0.5-40 Hz bandpass filtering and 50 Hz notch processing on the EEG to remove myoelectric artifacts, slow voltage drift phenomenon artifacts and power frequency interference;

[0076] S2.1.2. Since the frequency bands of EOG signals and EEG signals overlap, independent component analysis (ICA) is used to remove EOG artifacts. The main idea of ​​ICA is to classify mixed signals composed of several relatively independent signal sources, and then implement non-Gaussian measurement on the results. If the non-Gaussianity is the largest, the mixed signal is considered to have achieved optimal separation. Since the generation of artifacts such as EOG signals is a different physiological process from brain activity, that is, it meets the requirements of independent signal sources, ICA can be used to remove EOG artifacts. Figure 7 Shown are the before and after comparisons of removing electrooculogram artifacts.

[0077] S2.2. De-noising the ECG signal. The denoising method is as follows:

[0078] S2.2.1. ECG signals contain a large amount of 50Hz and 100Hz power frequency noise, which can be filtered out by designing 50Hz and 100Hz power frequency notch filters.

[0079] S2.2.2. After removing the power frequency interference, the wave still contains many burrs caused by myoelectric interference and other reasons. Considering that the traditional low-pass filter is easy to weaken the ECG R wave while removing the burrs, the wavelet transform noise reduction method is used to remove the burrs;

[0080] S2.2.3. Sometimes, there will be serious baseline drift in the ECG signal. The shape of the baseline drift is extracted using the median filter method, and then the extracted baseline is subtracted from the original signal. Figure 8 shown.

[0081] S2.3, use a smoothing filter (such as a 3rd order Butterworth filter) to remove high frequency glitches in the skin electrical signal. Since the skin electrical signal generally changes slowly, the cutoff frequency of the low pass filter is usually set to 0.3 Hz (such as Fig. 9 shown).

[0082] S2.4. Baseline signal removal. The specific method is as follows: In order to eliminate the errors caused by individual differences, the baseline signal elimination method is adopted: the baseline physiological signal of each subject in a relaxed state is collected, and then the physiological signals in different emotional states are subtracted from each person's baseline physiological signal to obtain the actual physiological change value of the subject after emotion induction.

[0083] S3. Perform feature extraction and dimensionality reduction on the preprocessed physiological signals.

[0084] The feature extraction method is as follows:

[0085] Emotion recognition feature extraction of EEG signals: The wavelet packet transform method is used to extract the emotion recognition features of EEG signals; the specific method is as follows:

[0086] The wavelet packet transform method is used to obtain multiple characteristic parameters such as the rhythmic waves {γ, β, α, θ}, power spectrum density and wavelet entropy of the EEG signal. When performing wavelet packet decomposition, the number of decomposition layers is determined according to the frequency division range of the four rhythmic waves in the EEG and the sampling frequency. This embodiment uses a 6-layer decomposition, such as Fig.10 shown.

[0087] Since the strength of the signal can be represented by the wavelet packet decomposition coefficient, the wavelet packet decomposition coefficients contained in each rhythm signal can be squared separately and then summed to obtain the energy of each frequency band. Assume that the wavelet decomposition coefficient corresponding to the i-th wavelet node in the j-th layer is The energy of a specific frequency band can be calculated using the following formula:

[0088]

[0089] The wavelet entropy analysis method based on EEG rhythm obtains wavelet entropy based on the relative energy of each rhythm wave. The wavelet entropy obtained in this way can reflect the distribution of the energy of the four rhythm waves in the total energy. The logarithmic energy entropy in the wavelet packet entropy is used here, that is, the wavelet entropy is defined by combining the energy distribution of the wavelet packet decomposition coefficients with the information entropy. The algorithm formula is as follows:

[0090] SE=-∑RE m oeLh m (2)

[0091] Among them, m represents the four waves of δ, θ, α, and β, and RE m is the relative energy value of each rhythmic wave.

[0092] Extraction of emotion recognition features of ECG signals: Extract the emotion recognition features of ECG signals using QRS wave groups. Fig.11 As shown in FIG, the extraction of QRS complex information is the key part of ECG feature information extraction. First, the position of the R wave is required before subsequent parameter measurement. Considering the real-time and accuracy of the system, this embodiment uses the differential threshold method and the amplitude threshold method to locate the R wave. The process is shown in FIG. Fig.12As shown. Among them, X(n) is the voltage value of the nth sampling point in the ECG signal segment. According to the characteristic of the largest wave slope, the ECG amplitude array is differentially calculated to generate the differential sequence Z(n): Z(n) = X(n+1)-X(n-1). The differential value is judged point by point. If the differential value of three consecutive points is greater than K1 (R wave rising slope threshold), then from 100ms after this point forward, it is judged whether there is such a point n, so that X(n) is greater than Xh (amplitude threshold), Z(n), Z(n-1) are less than K2 (R wave falling slope threshold). If it exists, the point with the largest amplitude between these two points is the R wave peak. In order to avoid interference from large T waves, a relative refractory period (200ms) is skipped after the wave is detected, and the next detection is performed. At the same time, in order to prevent missed detection, the RR interval should be judged for each R wave detected. If the RR interval is greater than the sampling period, there may be missed detection and recheck is required. The backcheck starts from the last R wave skipping the refractory period, and checks point by point whether there is such a point: the amplitude of this point is greater than the threshold, and higher than the amplitude of the previous and next points. If such a point is found, it is marked as the R wave peak, and returns to the main program to identify the next R wave. The R wave backcheck process is as follows Fig.13 shown.

[0093] The RR interval sequence was calculated based on the R wave peak, and then the RR interval was subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain time domain characteristics (RR interval mean and STD, coefficient of variation between RRs, STD of continuous differences between RRs, etc.) and frequency domain characteristic parameters (LF, HF, LF / HF ratio, etc.), sample entropy, etc.

[0094] Emotion recognition feature extraction of skin electrodermal signals: Since GSR does not have significant characteristic waveform points like electrocardiogram signals, this embodiment extracts time domain features by extracting skin electrodermal signal sequences: skin conductance, first-order difference and second-order difference of skin conductance, and on this basis, calculates the time domain feature parameters related to emotions (mean, median, mean square error, minimum value, maximum value, ratio of the total number of data points equal to the minimum value / maximum value to the sequence length).

[0095] like Fig.14 As shown, the feature dimensionality reduction method is as follows:

[0096] S3.1. Analyze the correlation between each attribute and the category of the final data point through the one-way ANOVA algorithm, remove irrelevant or low-correlation attribute features, and achieve feature dimensionality reduction in attribute selection;

[0097] S3.2. Use the principal component analysis (PCA) algorithm on linear transformation to project the original data onto a low dimension that can better distinguish these attribute features and categories through mean comparison and variance analysis, thereby achieving feature dimensionality reduction.

[0098] S4. Use the features after dimensionality reduction to establish a sentiment classification and prediction model through a recurrent neural network. The specific method is as follows: Sort the LSTM prediction models from high to low according to the percentage of prediction results, select the models with the highest accuracy, and use the prediction accuracy of each model as the weight of the prediction result category. By counting the corresponding category weights of each record, the final judgment result of the record is determined. Fig.15 The figure shows the LSTM flow chart of this embodiment. The input Train and Test feature data sets shown in the figure both include low-dimensional attribute feature sets and corresponding emotion categories after dimensionality reduction. Emotion model training is emotion learning, and multiple emotion models are obtained for storage. Emotion recognition and classification is to call the stored emotion model to perform emotion classification on the input test data set respectively. Xa>Xb, Xc outputs the judgment result A, Xa is the probability description of the category A of the emotion record evaluated by weight calculation, and the final category of the record is determined by the joint decision of the top few models with higher accuracy on a certain emotion record.

[0099] The above are only embodiments of the present invention, and the common knowledge such as the known specific structures and / or characteristics in the scheme are not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A multimodal fusion feature emotion recognition method, characterized by: The steps include: S1. Build an emotion-inducing experimental task platform by screening or designing VR scene tasks; S2, based on the emotion induction experimental task platform, collect the physiological signals of the subjects under different emotions and pre-process the physiological signals; S3, extracting features and reducing dimension of the preprocessed physiological signals; S4. Use the features after dimensionality reduction through a recurrent neural network to establish a sentiment classification and prediction model.

2. The multimodal fusion feature emotion recognition method according to claim 1, characterized in that: The construction method of step S1 is as follows: S1.

1. Determine the target emotion model: select six emotion states by selecting different valence and arousal levels: negative dimension (select depression, stress, fear), positive dimension (select sleepiness, relaxation, excitement); S1.2, Emotion induction means and material screening: Statistical data on the adaptability and effectiveness of different VR materials in inducing target emotions, and screen effective emotion inducing materials by setting certain thresholds; S1.

3. Cognitive experimental task design: Select the 2-back experimental paradigm; S1.

4. Experimental paradigm design: The experimental paradigm consists of multiple VR clips. After experiencing each VR video, the subjects are required to quickly make a self-assessment based on their feelings about the video they just watched, and then perform cognitive experimental tasks to examine changes in their operational performance.

3. The multimodal fusion feature emotion recognition method according to claim 2, characterized in that: In step S1.4, the subjects are relaxed before a new VR video starts, and data of the subjects in a relaxed state and their cognitive operation performance in a relaxed state are collected.

4. The multimodal fusion feature emotion recognition method according to claim 3, characterized in that: The preprocessing method of the physiological signal in step S2 is as follows: S2.1, denoising the EEG signal; S2.2, denoising the ECG signal; S2.3, use a smoothing filter to remove high-frequency glitches in the electrodermal signal; S2.4, baseline signal removal.

5. The multi-modal fusion feature emotion recognition method according to claim 4, characterized in that: The denoising method of step S2.1 is as follows: S2.1.

1. Based on the frequency range of normal EEG signals, use a filter to perform 0.5-40 Hz bandpass filtering and 50 Hz notch processing on the EEG; S2.1.

2. Use independent component analysis to remove oculoscopic artifacts.

6. The multi-modal fusion feature emotion recognition method according to claim 4, characterized in that: The denoising method of step S2.2 is as follows: S2.2.

1. Filter out by designing 50Hz and 100Hz power frequency notch filters; S2.2.2, use wavelet transform denoising method to remove burrs; S2.2.

3. Use the median filter method to extract the shape of the baseline drift, and then subtract the extracted baseline from the original signal.

7. The multimodal fusion feature emotion recognition method according to claim 4, characterized in that: The specific method of step S2.4 is as follows: collect the baseline physiological signal of each subject in a relaxed state, and then subtract each person's baseline physiological signal from the physiological signals in different emotional states to obtain the actual physiological change value of the subject after emotion induction.

8. The multimodal fusion feature emotion recognition method according to claim 1, characterized in that: The feature extraction method of step S3 is as follows: Emotion recognition feature extraction of EEG signals: The emotion recognition features of EEG signals are extracted using the wavelet packet transform method; Extraction of emotion recognition features of ECG signals: Extract emotion recognition features of ECG signals using QRS wave groups; Emotion recognition feature extraction of skin electrodermal signals: Extract skin electrodermal signal sequence: skin conductance, first-order difference and second-order difference of skin conductance, and on this basis, calculate the time domain feature parameters related to emotions.

9. The multi-modal fusion feature emotion recognition method according to claim 8, characterized in that: The feature dimensionality reduction method of step S3 is as follows: S3.

1. Analyze the correlation between each attribute and the category of the final data point through the one-way variance analysis algorithm, eliminate irrelevant or low-correlation attribute features, and achieve feature dimensionality reduction in attribute selection; S3.

2. Use the principal component analysis algorithm on linear transformation to project the original data onto a low dimension that can better distinguish these attribute features and categories through mean comparison and variance analysis, thereby achieving feature dimensionality reduction.

10. The multi-modal fusion feature emotion recognition method according to claim 1, characterized in that: The specific method of step S4 is as follows: sort the LSTM prediction models from high to low according to the percentage of prediction results, select the models with the top accuracy, and use the prediction accuracy of each model as the weight of the prediction result category. The final judgment result of the record is determined by statistics of the corresponding category weights of each record.

Images