Emotion recognition method and system based on multi-modal feature fusion

Abstract

The invention discloses an emotion recognition method and system based on multi-modal feature fusion, and the method comprises the steps: constructing an emotion recognition model suitable for time sequence feature extraction, training the weight of each modal through a quantized emotion label and an attention mechanism, carrying out the fusion, and finally carrying out the emotion quantification classification through an output layer. Meanwhile, data of three modes of electroencephalogram signals, pulse signals and voice signals are collected to serve as input data of an emotion recognition model. According to the invention, through combination with the multi-modal data, the emotion state of a person can be reflected more comprehensively, and the accuracy and robustness of emotion recognition are improved; meanwhile, real-time monitoring and early warning of the emotional state of the subject are achieved by collecting and processing data in real time; in addition, the features are automatically extracted and classified by using a deep learning algorithm, and complexity and subjectivity of artificial feature extraction in a traditional method are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition method and system based on multimodal feature fusion. Background Art

[0002] As human-computer interaction and intelligent technology are developing day by day, emotion recognition system, as an emerging technical means, is gradually changing our lives. Emotion recognition refers to judging people's emotional state by identifying relevant data such as voice, images, physiological signals, etc., including different emotional states such as happiness, sadness, anger, surprise, etc.

[0003] At present, traditional emotion recognition systems mainly use single-modal data for feature extraction and classification, and use a single pre-trained model for prediction. However, human emotions are diverse and closely related to individual experience, age, preferences, etc. The emotional patterns of only some groups cannot well predict the emotional patterns of unknown individuals. In addition, if only words such as happy and sad are used to describe emotions, it has a subjective language understanding color, and emotional changes are a gradual process. They cannot be simply divided by simple concepts such as "yes" and "no". The translation of vocabulary corresponding to different languages ​​is a difficult task, so a more objective method is needed to describe them. Many emotion algorithms use computer vision methods such as videos and pictures. The models are relatively complex, the number of parameters is extremely large, and the training cost is very high. Time series is a one-dimensional data. At this stage, there are many mature network models, which are suitable as an emotion data set.

[0004] It is urgent to improve the existing emotion recognition methods to achieve objective and accurate emotion recognition. In view of this, the present invention proposes an emotion recognition method and system based on multimodal feature fusion. Summary of the invention

[0005] The object of the present invention is to provide an emotion recognition method and system based on multimodal feature fusion.

[0006] In a first aspect, the present invention provides an emotion recognition method based on multimodal feature fusion, which comprises the following steps: Step 1: Collect different types of modal data of the subject, and perform label quantification processing on the collected modal data; Step 2: preprocess different types of modal data respectively, and construct a data set with the preprocessed modal data; Step 3: Construct an emotion recognition model; the emotion recognition model includes multiple gated recurrent units, a feature fusion module and an output layer; the gated recurrent unit is used to extract features of different types of modal data respectively; the feature fusion module is used to fuse the modal data features output by multiple gated recurrent units; the output layer is used to receive the fused features and output the final classification results; Step 4: Use the training set to train the emotion recognition model, and use the trained emotion recognition model to perform emotion recognition on the subjects.

[0007] Preferably, in step one, two or more sequences are selected from EEG signal sequences, speech signal sequences, and pulse signal sequences as modal data.

[0008] Preferably, the method for preprocessing the modal data is: using a bandpass filter to remove background noise, low-frequency trends and baseline drift in the modal data; if the modal data includes an EEG signal sequence, then after the bandpass filter processing, the EEG signal sequence is decomposed by an independent component analysis method to separate and remove artifact components; if the modal data includes a pulse signal sequence, then after the bandpass filter processing, the pulse signal sequence is smoothed to eliminate short-term fluctuations in the signal, and the artifact portion of the pulse signal sequence is removed by template matching; the modal data processed by the above processing methods are standardized and normalized respectively, and the amplitude of the modal data is scaled to a preset range.

[0009] Preferably, in the step 1, a self-emotion rating scale is used to perform label quantization on the collected modal data.

[0010] Preferably, the output layer comprises two fully connected layers connected in series and a Softmax function.

[0011] Preferably, in the step 4, the cross entropy loss is used as the loss function of the emotion recognition model, and the expression of the loss function L is: in, is the predicted probability of the true class.

[0012] In the second aspect, the present invention provides an emotion recognition system based on multimodal feature fusion, which is used to execute the above-mentioned emotion recognition method; the emotion recognition system includes a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module and an output layer module; the data acquisition module is used to collect the modal data of the subject; the preprocessing module is used to preprocess the collected modal data; the feature extraction module is used to extract features from the modal data processed by the preprocessing module; the feature fusion module is used to model the correlation between each modal data and obtain global features; the output layer module is used to obtain output results according to the global features output by the feature fusion module.

[0013] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described.

[0014] In a fourth aspect, the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the described method is implemented.

[0015] The present invention has the following beneficial effects: 1. The present invention can more comprehensively reflect a person's emotional state and improve the accuracy and robustness of emotion recognition by combining multimodal data such as EEG signals, pulse signals and voice signals. At the same time, the present invention can achieve real-time monitoring and early warning of the subject's emotional state by real-time data collection and processing; 2. The present invention uses a deep learning algorithm to automatically extract and classify features, avoiding the complexity and subjectivity of manual feature extraction in traditional methods; at the same time, the deep learning algorithm has strong learning and generalization capabilities, and can adapt to the emotion recognition needs of different individuals and situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the structure of the emotion recognition system of the present invention.

[0017] Figure 2 This is an expression of the self-emotion rating in the emotion recognition method of the present invention.

[0018] Figure 3 This is a structural diagram of the emotion recognition model in the emotion recognition method of the present invention.

[0019] Figure 4 This is a structural diagram of the gated recurrent unit in the emotion recognition method of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] like Figure 1 As shown, an emotion recognition system adopted by an emotion recognition method based on multimodal feature fusion includes a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module and an output layer module. The data acquisition module is used to collect the subject's EEG signal, pulse signal and voice signal; the preprocessing module is used to perform denoising and smoothing on the three collected signals respectively; the feature extraction module is used to extract features from the signal preprocessed by the preprocessing module and unify the different input dimensions of the data; the feature fusion module is used to model the correlation between each modality and obtain global features; the output layer module is used to extract common features and generate output results through dimensionality reduction operations.

[0022] The emotion recognition method comprises the following steps: Step 1: Collect modal data EEG caps, wearable devices, micro piezoelectric sensors and signal amplifiers were used to collect EEG signal sequences, speech signal sequences and pulse signal sequences of different subjects. During signal collection, emotional stimulation short films were played synchronously, which mainly consisted of movie slices. Each slice had a fixed length of 2 minutes and was classified for each slice. The SAM scale (Self-Assessment Manikin) was used in combination with subjective emotional judgment to quantify the labels of the collected signal sequences, and the emotions of the subjects were evaluated through three dimensions: pleasure, arousal and dominance. Each dimension contained 5 puppet pictures and 9 level options, such as Figure 2 As shown. Pleasantness (emotional valence) is used to measure positive and negative emotions; among them, "1" represents "very pleasant", "5" represents "neutral", and "9" represents "very unpleasant". Arousal (emotional arousal) represents the degree to which emotions are aroused and is used to measure the strength of emotions; among them, "1" represents "very excited", "5" represents "neutral", and "9" represents "very calm". Dominance (emotional dominance) represents the degree of control over the emotion and is used to measure the subject's ability to control emotional responses; among them, "1" represents "completely controlled", "5" represents "neutral", and "9" represents "completely controlled". The SAM scale is easy to understand and use, and can be used in emotion measurement situations under different regional cultural backgrounds.

[0023] Step 2: Data preprocessing Bandpass filters are used to remove background noise, low-frequency trends, and baseline drift in EEG signal sequences, speech signal sequences, and pulse signal sequences. After bandpass filter processing, the EEG signal sequence is decomposed by independent component analysis to separate and remove artifact components; the pulse signal sequence is smoothed to eliminate short-term fluctuations in the signal, highlight long-term trends, and the artifact part in the pulse signal sequence is removed by template matching. The processed signal sequence is standardized and normalized to scale the amplitude of the signal to a specific range (e.g., 0 to 1). A data set is constructed with the processed data, and the data set is divided into a training set and a test set.

[0024] Step 3: Build an emotion recognition model like Figure 3 As shown in the figure, the emotion recognition model includes three gated recurrent units (GRUs), a feature fusion module, and an output layer. Since the time series is used as the input of the emotion recognition model, the gated recurrent unit is selected as the feature extraction network to extract features from the single-modal time series. The long-term and short-term memory and gated unit update capabilities of the gated recurrent unit are used to extract the previous and next dependencies and other deep markers on the time series, which can effectively capture the temporal features in the time series and provide basic features for subsequent multimodal feature fusion. The network architecture of the gated recurrent unit is shown in the figure. Figure 4 As shown, the principle is as follows:

[0025] Reset the output of the gate t It is expressed as: r t =σ(W r x t +U r h t-1 +b r ) Where σ represents the sigmoid function; W r and U r is the weight parameter; x t is the input of the current time step; h t-1 is the hidden state of the previous time step; b r is the bias parameter.

[0026] Update the output z of the gate t It is expressed as: z t =σ(W z x t +U z h t-1 +b z ) Among them, W z and U z is the weight parameter; b zis the bias parameter.

[0027] Candidate hidden states It is expressed as: =tanh(Wx t +r t ⊙Uh t-1 +b) Among them, tanh represents the hyperbolic tangent function; W and U are weight parameters; b is the bias parameter; ⊙ represents element-by-element multiplication.

[0028] Hidden state h t It is expressed as: h t =(1-z t )⊙h t-1 +z t ⊙ The features of the EEG signal sequence, speech signal sequence and pulse signal sequence are extracted respectively through three gated recurrent units to obtain the modal features of the three modal data.

[0029] The feature fusion module is used to fuse the three modal features extracted by the gated recurrent unit. It uses the Transformer's multi-head self-attention mechanism to perform weighted calculations on the input modal features, which can capture the global correlation between input features and allow the model to focus on different feature subspaces. After the multi-head self-attention mechanism, the fully connected layer is used to further extract global features to complete the interaction between modalities and generate a unified hidden representation to provide input for subsequent classification tasks. The multi-head self-attention mechanism is used to extract modal features of different modal data. Perform a linear transformation to obtain the query, key, and value matrices:

[0030] Among them, Q i , K i and V i They are query matrix, key matrix and value matrix respectively; is the weight matrix of linear transformation; .

[0031] Get the attention score between different modal features based on the query matrix and the key matrix ij , whose expression is: Among them, softmax is a normalized exponential function; is the query matrix of the i-th signal sequence; is the key matrix of the j-th signal sequence; is the dimension of the key matrix; .

[0032] According to the attention score ij Get the corresponding weighted value matrix , whose expression is: in, is the value matrix of the j-th signal sequence.

[0033] Fuse the weighted value matrices between different modal features and output the fused feature representation .

[0034] The output layer is used to receive the fused feature representation and output the final classification result, which is the probability distribution of the three emotion labels. The output layer includes two fully connected layers and a Softmax function; the feature representation is sequentially processed through two series of fully connected layers. Processing to obtain the quantization range mapped to each label :

[0035] in, is the output score vector of the mth emotion label, indicating the score of each discrete value 0-9 for this emotion label; is the weight matrix; is the bias matrix; .

[0036] The Softmax function is used to convert the output score vector Convert to probability distribution , whose expression is: Among them, y m is the prediction result of the mth emotion label.

[0037] Finally, the score with the highest probability is taken as the final predicted value y: In some embodiments, two or more signal sequences among an EEG signal sequence, a speech signal sequence, and a pulse signal sequence are used as input features of the emotion recognition model.

[0038] Step 4: Use the training set to train the emotion recognition model, and use the cross entropy loss as the loss function. The expression of the loss function L is: in, is the predicted probability of the true category, and the average is taken to calculate the overall loss of the three labels.

[0039] Step 5: Use the test set to test the trained emotion recognition model. If it fails to meet the preset standard, retrain it; otherwise, use the tested emotion recognition model to recognize the subject's emotions.

[0040] To sum up, the present invention is an innovative technical means with broad application prospects and huge potential. By combining multimodal data, deep learning algorithms and real-time processing technology, the system can realize accurate identification and real-time monitoring of emotional states, and provide strong support for mental health monitoring, human-computer interaction, intelligent education and other fields. In the future, with the continuous advancement and improvement of technology, the system is expected to play a greater role in more fields and bring more convenience and intelligence to people's lives.

[0041] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. The replacement may be a replacement of a part of the structure, device, method step, or a complete technical solution. Any equivalent replacement or change according to the technical solution and the inventive concept of the present invention shall be covered within the protection scope of the present invention.

Claims

1. An emotion recognition method based on multimodal feature fusion, characterized in that: The following steps are involved: Step 1: Collect different types of modal data of the subject, and perform label quantification processing on the collected modal data; Step 2: preprocess different types of modal data respectively, and construct a data set with the preprocessed modal data; Step 3: Construct an emotion recognition model; the emotion recognition model includes multiple gated recurrent units, a feature fusion module and an output layer; the gated recurrent units are used to extract features of different types of modal data respectively; feature The fusion module is used to fuse the modal data features output by multiple gated recurrent units; The output layer is used to receive the fused features and output the final classification results; Step 4: Use the training set to train the emotion recognition model, and use the trained emotion recognition model to perform emotion recognition on the subjects.

2. The emotion recognition method based on multimodal feature fusion according to claim 1, characterized in that: In the step 1, two or more signal sequences are selected from the EEG signal sequence, the speech signal sequence, and the pulse signal sequence as modal data.

3. The emotion recognition method based on multimodal feature fusion according to claim 2, characterized in that: The method for preprocessing the modal data is as follows: using a bandpass filter to remove background noise, low-frequency trends and baseline drift in the modal data; if the modal data includes an EEG signal sequence, then after the bandpass filter is processed, the EEG signal sequence is decomposed by an independent component analysis method to separate and remove artifact components; if the modal data includes a pulse signal sequence, then after the bandpass filter is processed, the pulse signal sequence is smoothed to eliminate short-term fluctuations in the signal, and the artifact part in the pulse signal sequence is removed by template matching; the processed modal data are standardized and normalized respectively, and the amplitude of the modal data is scaled to a preset range.

4. The emotion recognition method based on multimodal feature fusion according to claim 1, characterized in that: In the step 1, a self-emotion rating scale is used to perform label quantification on the collected modal data.

5. The emotion recognition method based on multimodal feature fusion according to claim 1, characterized in that: The output layer includes two fully connected layers connected in series and a Softmax function.

6. The emotion recognition method based on multimodal feature fusion according to claim 1, characterized in that: In the step 4, the cross entropy loss is used as the loss function of the emotion recognition model, and the expression of the loss function L is: in, is the predicted probability of the true class.

7. An emotion recognition system based on multimodal feature fusion, characterized in that: Used to execute the emotion recognition method based on multimodal feature fusion described in claim 1; the emotion recognition system includes a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module and an output layer module; the data acquisition module is used to collect the modal data of the subject; the preprocessing module is used to preprocess the collected modal data; the feature extraction module is used to extract features from the modal data processed by the preprocessing module; the feature fusion module is used to model the correlation between the modal data and obtain global features; The output layer module is used to obtain the output results based on the global features output by the feature fusion module.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images