Emotion recognition method and system based on multimodal spatiotemporal feature fusion

Abstract

The present disclosure belongs to the technical field of emotion recognition, and provides an emotion recognition method and system based on multimodal spatiotemporal feature fusion, comprising the following steps: acquiring original physiological data; preprocessing the acquired original physiological data to obtain multimodal physiological data; extracting spatial characteristics and temporal characteristics of the multimodal data respectively based on the obtained multimodal physiological data; performing feature-level fusion on the extracted spatial characteristics and temporal characteristics of the multimodal data to obtain fusion features; and classifying according to the obtained fusion features to obtain emotion recognition results.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of emotion recognition, and specifically relates to an emotion recognition method and system based on multimodal spatiotemporal feature fusion. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Emotional changes are usually caused by the stimulation of the external environment, accompanied by changes in individual representation and psychological response, so they can be measured and simulated by scientific methods. With the development and promotion of portable and wearable sensors, the difficulty of obtaining physiological signals has been reduced, and emotion recognition methods based on physiological signals have attracted more and more attention from researchers. Emotion recognition methods based on physiological signals refer to analyzing the emotional changes in human hearts by processing physiological signals. It has been widely used in many fields, such as fatigue driving detection and medical care. However, many studies have shown that physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), galvanic skin response (GSR), respiration (RSP) and electrooculogram (EOG) have a certain correlation with specific emotions, but each physiological signal has different characteristics and performs differently in emotion recognition tasks. Each physiological signal needs to be studied separately.

[0004] At present, emotion recognition methods based on physiological signals can be roughly divided into two categories: unimodal emotion recognition methods and multimodal emotion recognition methods. Among unimodal emotion recognition methods, the most commonly used one is the emotion recognition method based on EEG signals. In addition, physiological signals such as electrocardiogram, skin electricity and breathing also have good effects in the field of emotion recognition. Multimodal emotion recognition methods refer to methods that combine data and features of multiple modalities to obtain the final classification results. They mainly include three levels: data-level fusion, feature-level fusion and decision-level fusion. Whether it is a unimodal or multimodal emotion recognition method, it mainly includes several steps such as data preprocessing, feature extraction, feature optimization, feature fusion and emotion classification, with the focus on feature engineering. Therefore, how to extract features with strong emotion representation capabilities and apply them to emotion recognition tasks has become a key challenge.

[0005] According to the inventors, the existing emotion recognition methods have the following technical problems:

[0006] (1) Traditional emotion recognition methods based on physiological signals mainly extract artificial features such as statistical features and frequency domain features from data based on professional knowledge and experience. They are highly interpretable, but require a high level of professional knowledge and may cause problems such as information loss, thereby affecting recognition accuracy. Some researchers have proposed that using neural networks to extract high-level features of data is another feasible method. However, different network structures have shown great differences in the effects of feature extraction. How to use neural networks to extract high-level features with high emotion representation capabilities has become a major technical difficulty.

[0007] (2) Multimodal technology mainly includes three aspects: data-level fusion, feature-level fusion and decision-level fusion. In the field of emotion recognition, research mainly focuses on decision-level fusion and feature-level fusion. Decision-level fusion is to integrate the results based on training a classifier in each modality. It can make full use of the beneficial information of each modality and achieve simplicity and strong interpretability, but it will also lead to the loss of complementary information between modalities. Feature-level fusion is to extract features from the original data, fuse the features between the modalities to obtain the fused features for recognition tasks, and make full use of the emotional complementarity between the modalities. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes an emotion recognition method and system based on multimodal spatiotemporal feature fusion, which takes ECG, RSP and eye movement signals as input, performs linear filling and denoising on the original physiological data to eliminate the influence of outliers and noise on the recognition accuracy; uses convolutional neural network (CNN) and long short-term memory neural network (LSTM) to extract the temporal and spatial features of physiological data to characterize emotions; uses a multimodal compact bilinear pooling layer to fuse the temporal and spatial features of physiological data, and fully utilizes the complementary information between different physiological data on the basis of retaining effective information and reducing dimensionality, solves the related technical problems of complementary information fusion between different modalities, and improves the recognition accuracy of the model.

[0009] According to some embodiments, the first solution of the present disclosure provides an emotion recognition method based on multimodal spatiotemporal feature fusion, which adopts the following technical solutions:

[0010] An emotion recognition method based on multimodal spatiotemporal feature fusion includes the following steps:

[0011] Obtaining raw physiological data;

[0012] Preprocessing the acquired raw physiological data to obtain multimodal physiological data;

[0013] Extracting spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data;

[0014] Perform feature-level fusion on the spatial characteristics and temporal characteristics of the extracted multimodal data to obtain fused features;

[0015] Classification is performed based on the obtained fusion features to obtain the result of emotion recognition.

[0016] Here, the acquired raw physiological data at least includes electrocardiogram, respiration and eye movement signals.

[0017] As a further technical limitation, the original emotion data set contains the emotion stimulation stage and the self-assessment stage. The physiological signals in the original emotion data set are cropped to extract the emotion stimulation stage data. The truncated data are linearly interpolated to eliminate the influence of missing values ​​during data collection and processing. The wavelet denoising method is used to denoise the data to eliminate the influence of noise on the recognition effect.

[0018] As a further technical limitation, the preprocessing includes:

[0019] The original emotion data set contains the emotion stimulation stage and self-assessment truncation. The physiological signals in the original emotion data set are trimmed to extract the emotion stimulation stage data.

[0020] Linear interpolation is performed on the truncated data to eliminate the impact of missing values ​​during data collection and processing;

[0021] The wavelet denoising method is used to denoise the data and eliminate the influence of noise on the recognition effect.

[0022] Furthermore, the preprocessed multimodal physiological data are converted into grayscale images and input into the neural network to extract the spatial features of the multimodal data from the grayscale images.

[0023] Furthermore, the preprocessed multimodal physiological data are respectively input into the neural network to extract the temporal features of the multimodal physiological data.

[0024] As a further technical limitation, the temporal features and spatial features of the multimodal physiological data extracted from the neural network are fused at the feature level to obtain fused features for the emotion recognition task. The specific process is as follows: the CountSktech algorithm is used to count the frequency of occurrence of each element to achieve mapping from high dimension to low dimension; the bilinear pooling method is used to fuse the features after dimensionality reduction to obtain fused features.

[0025] As a further technical limitation, a classification task is performed based on the obtained fusion features to obtain the final emotion recognition result. The specific process is: training an SVM classifier; inputting the fusion features into the classifier to obtain the final recognition result.

[0026] According to some embodiments, the second solution of the present disclosure provides an emotion recognition system based on multimodal spatiotemporal feature fusion, which adopts the following technical solutions:

[0027] An emotion recognition system based on multimodal spatiotemporal feature fusion, comprising:

[0028] An acquisition module is configured to acquire original physiological data, pre-process the acquired original physiological data, and obtain multimodal physiological data;

[0029] A fusion module is configured to extract spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data, perform feature-level fusion on the extracted spatial characteristics and temporal characteristics of the multimodal data, and obtain fusion features;

[0030] The recognition module is configured to perform classification according to the obtained fusion features to obtain the result of emotion recognition.

[0031] According to some embodiments, a third solution of the present disclosure provides a computer-readable storage medium, which adopts the following technical solution:

[0032] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in the emotion recognition method based on multimodal spatiotemporal feature fusion as described in the first aspect of the present disclosure.

[0033] According to some embodiments, a fourth solution of the present disclosure provides an electronic device, which adopts the following technical solution:

[0034] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the emotion recognition method based on multimodal spatiotemporal feature fusion as described in the first aspect of the present disclosure are implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention first preprocesses the original physiological signal, and the preprocessing steps include data truncation, linear interpolation and noise reduction processing; second, a spatial feature extraction module based on a two-dimensional convolutional neural network (Two Dimensional Convolutional Neural Network, referred to as 2D-CNN) is proposed, and the preprocessed multimodal physiological data is converted into a 1×120×120 image (1 represents the number of layers, and 120×120 represents the pixel size of the image), and then the image is input into the 2D-CNN to extract the spatial features of the multimodal physiological data; third, a temporal feature extraction module based on a long short-term memory neural network (LSTM) is proposed, and the preprocessed multimodal physiological data are respectively input into the LSTM to extract the temporal features of the multimodal physiological data; fourth, a multimodal compact bilinear pooling method is used to perform feature-level fusion of temporal features and spatial features to obtain fused features; fifth, the fused features are input into a trained classifier to obtain the final emotion recognition result.

[0037] 2. The system described in this disclosure consists of five parts: data preprocessing module, spatial feature extraction module, temporal feature extraction module, feature fusion module and emotion classification module. Through analysis, it is found that when human emotions change or fluctuate greatly, EEG and RSP will also change, such as increasing amplitude and frequency. By analyzing the regularity of these waveforms, the current emotions can be accurately identified; the spatial feature extraction module proposed in this disclosure converts the preprocessed multimodal physiological data into a 1×120×120 image (1 represents the number of layers, and 120×120 represents the pixel size of the image), uses a 2D-CNN network to extract the spatial features of physiological signals, and maximizes the use of the waveform regularity of physiological signals.

[0038] 3. In order to extract the time information in the physiological signals in a targeted manner, the present disclosure proposes a time feature extraction module, which inputs the preprocessed physiological signals into the LSTM to extract the time features and maximizes the use of the time information in the physiological signals.

[0039] 4. In order to fully integrate temporal features and spatial features, and utilize the complementary information between different modalities, the present invention proposes a feature fusion module, which uses a multi-modal compact bilinear pooling method to fuse temporal features and spatial features to obtain fused features for emotion recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0041] Figure 1 is a flow chart of an emotion recognition method based on multimodal spatiotemporal feature fusion in Embodiment 1 of the present disclosure;

[0042] Figure 2 is a specific working principle diagram of the emotion recognition method based on multimodal spatiotemporal feature fusion in the first embodiment of the present disclosure;

[0043] Figure 3 is a flowchart of an emotion recognition method based on multimodal spatiotemporal feature fusion in the first embodiment of the present disclosure;

[0044] Figure 4 is a schematic diagram of the structure of spatial feature extraction in the first embodiment of the present disclosure;

[0045] Figure 5 is a schematic diagram of the structure of time feature extraction in the first embodiment of the present disclosure;

[0046] Figure 6 is a structural diagram of feature fusion in Embodiment 1 of the present disclosure;

[0047] Figure 7 It is a structural block diagram of the emotion recognition system based on multimodal spatiotemporal feature fusion in the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0052] Embodiment 1

[0053] Embodiment 1 of the present disclosure introduces an emotion recognition method based on multimodal spatiotemporal feature fusion.

[0054] like Figure 1 The emotion recognition method based on multimodal spatiotemporal feature fusion is characterized by comprising the following steps:

[0055] Obtaining raw physiological data;

[0056] Preprocessing the acquired raw physiological data to obtain multimodal physiological data;

[0057] Extracting spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data;

[0058] Perform feature-level fusion on the spatial characteristics and temporal characteristics of the extracted multimodal data to obtain fused features;

[0059] Classification is performed based on the obtained fusion features to obtain the result of emotion recognition.

[0060] like Figure 2 and Figure 3 As shown, this embodiment provides an emotion recognition method based on the fusion of multimodal spatiotemporal features. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0061] Step S01: For the original emotion dataset D, the physiological signals in the dataset D are subjected to data cropping, linear interpolation and noise reduction to obtain the dataset D. * ;

[0062] Step S02: For data set D * The physiological signal S in the image is converted into a grayscale image I_s, and the grayscale image I_s is input into the 2D-CNN to obtain the spatial feature F spatio ;

[0063] Step S03: for data level D * The physiological signal S is input into LSTM to obtain the time feature F temporal ;

[0064] Step S04: transform the spatial feature F spatio and time feature F temporal Input into the multi-mode compact bilinear pooling layer for feature fusion to obtain the fused feature F fusion ;

[0065] Step S05: The fused feature F fusion Input into the classifier to get the final recognition result.

[0066] In step S01 of the embodiment, the original emotion data set includes an emotion stimulation stage and a self-assessment stage. The physiological signals in the original emotion data set are cropped to extract the emotion stimulation stage data. Linear interpolation is performed on the truncated data to eliminate the influence of missing values ​​during data collection and processing. Wavelet denoising is used to perform denoising on the data to eliminate the influence of noise on the recognition effect.

[0067] Initialize the input original emotion dataset D = [S 1 ,S 1 ,…,S n ], where S n is the physiological signal sequence collected in the nth emotional stimulation experiment, where n∈N, N represents the number of experiments. The physiological signal sequence S collected in each experiment n =[ECG 1,n ,ECG 2,n ,ECG 3,n ,RSP n ,Eye_Data n ], where ECG i,n represents the ECG signal of the ith channel collected in the nth emotional stimulation experiment, i = 1, 2, 3, RSP n Indicates the breathing data collected in the nth emotional stimulation experiment, Eye_Data n Represents the eye movement data collected in the nth emotional stimulation experiment. For each experiment in the dataset D, the physiological signal sequence S n Data truncation is performed to extract the physiological signals of the emotional stimulation part, and then linear interpolation and noise reduction are performed to obtain the preprocessed data set D * .

[0068] In step S02 of this embodiment, Figure 4 As shown, the present disclosure uses the matplotlib toolkit to plot the data set D *Each physiological signal sequence in the image is converted into a grayscale image (1×120×120, 1 represents the number of layers, and 120×120 represents the pixel size of the image). Then, the Pytorch toolkit is used to build a 2D-CNN network, and the grayscale image is input into the network to extract the spatial feature F. spatio .

[0069] For the design of the spatial feature extraction module, first, the grayscale image I_s with a dimension of 120×120×1 is input into the 2D-CNN. The 2D-CNN includes two 5×5×32 and two 5×5×64 convolution operations to obtain a 30×30×64 feature map. Then, two fully connected layers are used to obtain a spatial feature F with a dimension of 1×50. spatio The convolution operation can be expressed as:

[0070] s(t)=(x×ω)(t) (1)

[0071] Among them, the first parameter x becomes the "input data", the second parameter ω is called the "kernel function", and s(t) is the output, that is, the feature map.

[0072] Taking the grayscale image I_s of a certain experiment as an example, the calculation process of extracting spatial features using 2D-CNN is as follows:

[0073] (1) The first convolutional layer: The input is a grayscale image I_s of size 120×120×1 (height×width×number of color channels). This layer has 32 convolution kernels, each with a size of 7×7×1, and the output matrix has a size of 60×60×32.

[0074] (2) The second convolutional layer: The input is a 60×60×32 matrix. This layer has 32 convolution kernels, each with a size of 7×7×32, and the output matrix has a size of 60×60×32.

[0075] (3) The third convolutional layer: The input is a 60×60×32 matrix. This layer has 64 convolution kernels, each with a size of 7×7×32, and the output matrix has a size of 30×30×64.

[0076] (4) The fourth convolutional layer: The input is a 30×30×64 matrix. This layer has 64 convolution kernels, each with a size of 7×7×64, and the output matrix has a size of 30×30×64.

[0077] (5) The first fully connected layer: the input is a 30×30×64 matrix and the output is a 1×512 matrix;

[0078] (6) The second fully connected layer: the input is a 1×512 matrix and the output is a 1×50 feature matrix.

[0079] In step S03 of this embodiment, Figure 5 As shown, this paper uses the Pytorch toolkit to build an LSTM network from the dataset D * Extract the temporal features F from each physiological signal sequence temporal The concept of gate is needed in LSTM network, which is essentially a fully connected layer and can be expressed as:

[0080] g(x)=σ(wx+b) (2)

[0081] Here, x is the input vector, w is the gate weight vector, b is the bias term, and g(x) is the output vector.

[0082] LSTM uses two gates to control the content of the cell state c. One is the forget gate, which determines the cell state c at the previous moment. t-1 How many are retained to the current time c t , the forget gate can be expressed as:

[0083] f 1 =σ(W f ·[h t-1 ·x t ]+b f ) (3)

[0084] Among them, W f is the weight matrix of the forget gate, [h t-1 ·x t ] means connecting two vectors into a longer vector, b f is the bias term of the forget gate, and σ is the sigmoid function.

[0085] The other is the input gate, which determines the input x of the network at the current moment. t How much is saved to the cell state c t The input gate can be expressed as:

[0086] i t =σ(W i ·[h t-1 ·x t ]+b i ) (4)

[0087] Among them, W i is the weight matrix of the forget gate, b i is the bias term of the forget gate, and σ is the sigmoid function.

[0088] The current input cell status It is calculated based on the previous output and this input, and the calculation formula is as follows:

[0089]

[0090] Next, calculate the current cell state c t , which is the last cell state c t-1 Multiply the forget gate f element-wise t , and then use the current input unit state Element-wise multiplication of input gate i t , and then the two products are added together to produce the following calculation formula:

[0091]

[0092] Here, “.” means element-wise multiplication.

[0093] LSTM uses the output gate to control how much of the cell state c is output to the current output value h of the LSTM. t , the calculation is shown as follows:

[0094] o t =σ(W o ·[h t-1 ·x t ]+b o ) (7)

[0095] The final output of LSTM is determined by the output gate and the cell state. The calculation is shown below:

[0096] h t =o t ·tanh(c t ) (8)

[0097] In step S04 of the embodiment, as Figure 6 As shown, the present disclosure uses the CountSktech algorithm to count the frequency of occurrence of each element to achieve data F spatio and F temporal Mapping from high dimension to low dimension to obtain the features after dimensionality reduction and This reduces the amount of computation required for subsequent feature fusion. Then, the fusion feature F is obtained through Fast Fourier Transform (FFT), dot multiplication, and Inverse Fast Fourier Transform (IFFT). fusion .

[0098] In step S05 of this embodiment, a classification task is performed based on the obtained fusion features to obtain the final emotion recognition result, which is specifically:

[0099] Train the SVM classifier;

[0100] The fused features are input into the classifier to obtain the final recognition result.

[0101] Embodiment 2

[0102] Embodiment 2 of the present disclosure introduces an emotion recognition system based on multimodal spatiotemporal feature fusion.

[0103] like Figure 7 An emotion recognition system based on multimodal spatiotemporal feature fusion is shown, comprising:

[0104] An acquisition module is configured to acquire original physiological data, pre-process the acquired original physiological data, and obtain multimodal physiological data;

[0105] A fusion module is configured to extract spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data, perform feature-level fusion on the extracted spatial characteristics and temporal characteristics of the multimodal data, and obtain fusion features;

[0106] The recognition module is configured to perform classification according to the obtained fusion features to obtain the result of emotion recognition.

[0107] The detailed steps are the same as those of the emotion recognition method based on multimodal spatiotemporal feature fusion provided in Example 1, and will not be repeated here.

[0108] Embodiment 3

[0109] A third embodiment of the present disclosure provides a computer-readable storage medium.

[0110] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in the emotion recognition method based on multimodal spatiotemporal feature fusion as described in the first embodiment of the present disclosure.

[0111] The detailed steps are the same as those of the emotion recognition method based on multimodal spatiotemporal feature fusion provided in Example 1, and will not be repeated here.

[0112] Embodiment 4

[0113] A fourth embodiment of the present disclosure provides an electronic device.

[0114] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the emotion recognition method based on multimodal spatiotemporal feature fusion as described in the first embodiment of the present disclosure are implemented.

[0115] The detailed steps are the same as those of the emotion recognition method based on multimodal spatiotemporal feature fusion provided in Example 1, and will not be repeated here.

[0116] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. An emotion recognition method based on multimodal spatiotemporal feature fusion, characterized in that: The following steps are involved: Obtain raw physiological data; Preprocessing the acquired raw physiological data to obtain multimodal physiological data; Extracting spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data; Perform feature-level fusion on the spatial characteristics and temporal characteristics of the extracted multimodal data to obtain fused features; Classify according to the obtained fusion features to obtain the result of emotion recognition; For the original emotion dataset, perform data cropping, linear interpolation and noise reduction on the physiological signals in the dataset; The original emotion data set contains the emotion stimulation stage and self-assessment truncation. The physiological signals in the original emotion data set are trimmed to extract the emotion stimulation stage data. The preprocessed multimodal physiological data is converted into a grayscale image and input into the neural network, and the spatial features of the multimodal data are extracted from the grayscale image; The preprocessed multimodal physiological data are respectively input into the neural network to extract the temporal features of the multimodal physiological data; The temporal and spatial features of the multimodal physiological data extracted from the neural network are fused at the feature level to obtain fused features for the emotion recognition task. The specific process is as follows: the Count Sktech algorithm is used to count the frequency of occurrence of each element to achieve mapping from high dimension to low dimension; the bilinear pooling method is used to fuse the features after dimensionality reduction to obtain fused features; Initialize the input original emotion dataset D = [S1, S1, ..., S n ], where S n is the physiological signal sequence collected in the nth emotional stimulation experiment, where n∈N, N represents the number of experiments; the physiological signal sequence collected in each experiment ECG i，n represents the ECG signal of the ith channel collected in the nth emotional stimulation experiment, i = 1, 2, 3, RSP n Indicates the breathing data collected in the nth emotional stimulation experiment, Eye_Data n Represents the eye movement data collected in the nth emotional stimulation experiment.

2. The emotion recognition method based on multimodal spatiotemporal feature fusion as claimed in claim 1, characterized in that: The original emotion data set contains the emotion stimulation stage and the self-assessment stage. The physiological signals in the original emotion data set are cropped to extract the emotion stimulation stage data. Linear interpolation is performed on the truncated data to eliminate the influence of missing values ​​during data collection and processing. Wavelet denoising method is used to denoise the data to eliminate the influence of noise on the recognition effect.

3. The emotion recognition method based on multimodal spatiotemporal feature fusion as claimed in claim 1, characterized in that: The pre-processing comprises: The original emotion data set contains the emotion stimulation stage and self-assessment truncation. The physiological signals in the original emotion data set are trimmed to extract the emotion stimulation stage data. Linear interpolation is performed on the truncated data to eliminate the impact of missing values ​​during data collection and processing; The wavelet denoising method is used to denoise the data and eliminate the influence of noise on the recognition effect.

4. The emotion recognition method based on multimodal spatiotemporal feature fusion as claimed in claim 1, characterized in that: According to the obtained fusion features, the classification task is carried out to obtain the final emotion recognition result. The specific process is as follows: training the SVM classifier; The fused features are input into the classifier to obtain the final recognition result.

5. An emotion recognition system based on multimodal spatiotemporal feature fusion, characterized in that: The following steps are involved: An acquisition module is configured to acquire original physiological data, pre-process the acquired original physiological data, and obtain multi-modal physiological data; A fusion module is configured to extract spatial characteristics and temporal characteristics of the multimodal data based on the obtained multimodal physiological data, perform feature-level fusion on the extracted spatial characteristics and temporal characteristics of the multimodal data, and obtain fusion features; For the original emotion data set, data clipping, linear interpolation and noise reduction processing are performed on the physiological signals in the data set. The original emotion data set contains the emotion stimulation stage and self-assessment truncation. The physiological signals in the original emotion data set are clipped to intercept the emotion stimulation stage data; The preprocessed multimodal physiological data is converted into a grayscale image and input into the neural network, and the spatial features of the multimodal data are extracted from the grayscale image; The preprocessed multimodal physiological data are respectively input into the neural network to extract the temporal features of the multimodal physiological data; The temporal and spatial features of the multimodal physiological data extracted from the neural network are fused at the feature level to obtain fused features for the emotion recognition task. The specific process is as follows: the Count Sktech algorithm is used to count the frequency of occurrence of each element to achieve mapping from high dimension to low dimension; the bilinear pooling method is used to fuse the features after dimensionality reduction to obtain fused features; The recognition module is configured to perform classification according to the obtained fusion features to obtain the result of emotion recognition.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the emotion recognition method based on multimodal spatiotemporal feature fusion as described in any one of claims 1 to 4 are implemented.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the emotion recognition method based on multimodal spatiotemporal feature fusion as described in any one of claims 1 to 4 are implemented.

Images