A speech emotion recognition method based on double-layer fusion deep network
By decomposing the combination method of bilinear pooling fusion module and multi-layer neural network, the problem of insufficient feature fusion in multimodal speech emotion recognition is solved, and a higher recognition rate is achieved, which significantly improves the accuracy of speech emotion recognition. .
Patent Information
- Application Number
- CN202210419568.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-21
AI Technical Summary
The existing multimodal speech emotion recognition methods are insufficient in terms of feature fusion, and simple addition or splicing methods cannot fully utilize the complementarity between speech and text information, resulting in insufficient recognition rate.
The decomposed bilinear pooling fusion module (FBP) is used to perform primary feature fusion, and a primary feature encoding network composed of sub-models such as LSTM, GRU, DNN is combined with the hadmard product and BiLSTM for secondary fusion, and finally connected to the classification output layer. Predict emotional categories.
The hierarchical structural relationship between high and low-level features is effectively established, and the complementarity of speech and text information is fully utilized, which significantly improves the accuracy of speech emotion recognition.
Smart Images

Figure CN114743569B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of speech emotion recognition and relates to a speech emotion recognition algorithm of multimodal feature fusion. Background Art
[0002] At present, there are three main research directions in the field of speech emotion recognition. The first is based on traditional machine learning methods, manually extracting emotion-related features and using machine learning methods to classify emotions. The second is an emotion recognition method based on deep neural networks, which implements an end-to-end emotion recognition system through neural networks. The third is to combine speech and text for multimodal emotion recognition.
[0003] Multimodal speech emotion recognition has been a new research hotspot in recent years and has attracted widespread attention. The current main research area of multimodal speech emotion recognition is to combine speech and text for multimodal emotion recognition. Considering that different speech features have different effects on emotions, in order to improve the recognition rate of the system, the future trend of speech emotion recognition is the multimodal recognition system framework.
[0004] Speech emotion recognition is a challenging task because emotional expression is complex and multimodal. The key to improving speech emotion recognition performance lies first in the extraction of emotional features and second in the fusion method of multimodal features. At present, most mainstream methods for feature extraction use deep neural networks to extract emotional features, and many studies have verified that the effect is significantly improved. However, innovative research in feature fusion is still not deep enough. The survey found that most methods use simple fusion methods such as addition and splicing. Obviously, these methods cannot fully utilize the complementarity between the information of each modality.
[0005] In the current research on multimodal emotion recognition, Hu Yaxin et al. published an article titled "Multimodal Emotion Recognition Based on Speech-Text Fusion" in which the feature fusion method uses simple fusion methods such as addition and concatenation. Obviously, these methods cannot fully utilize the complementarity of the emotional features between the modal information. Summary of the invention
[0006] The main problem solved by the present invention is to improve the existing speech emotion recognition method and system and propose a multimodal feature fusion solution that can improve the recognition rate. To achieve this technical purpose, the technical solution process adopted by the present invention is as follows:
[0007] Step 1: Preprocess the speech and text data to make them meet the input requirements of the network model;
[0008] Step 2: Input the speech and text feature vectors preprocessed in step 1 into the decomposition bilinear pooling fusion module (FBP) for primary feature fusion;
[0009] Step 3: The fused primary features output by the FBP module in step 2 are passed through the level-1 primary feature encoding network composed of three sub-models: LSTM, GRU, and DNN;
[0010] Step 4: Then the outputs of the three sub-networks of level-1 are fused twice to encode high-level features. The fusion method is the hadmard product. The fused features are then input into the BiLSTM encoding network of level-2, and finally connected to the classification output layer to predict the emotion category.
[0011] Step 5: Finally train the network model.
[0012] Furthermore, the text signal preprocessing refers to adopting a word embedding method, using a pre-trained Glove model to represent each word with a vector.
[0013] Furthermore, the speech signal preprocessing refers to windowing and framing the audio signal, using a Hamming window with a window length of 25 ms and a frame shift of 10 ms, then performing Fourier transform on it, and finally performing a Mel filter operation to obtain Mel spectrum features.
[0014] Furthermore, the alignment operation is performed by merging speech frames related to the words to obtain speech features corresponding to each word.
[0015] Furthermore, the structure of the network model includes a total of four layers. The first layer is the FBP primary feature fusion layer; the second layer is the level-1 primary feature encoding network layer composed of three sub-models: LSTM, GRU, and DNN; the third layer is the hadmard product fusion layer; and the fourth layer is the advanced encoding network layer composed of BiLSTM.
[0016] Furthermore, the fusion adopts a fusion algorithm process, specifically inputting audio and text feature vectors, first cross-fusing the audio features and text features through the FBP fusion module, and passing the fused features through the level-1 primary feature encoding network composed of three sub-models: LSTM, GRU, and DNN. Then, the outputs of the three sub-networks of level-1 are secondary fused to encode high-level features. The fusion method is hadmard product, and then the fused features are input into the level-2 BiLSTM encoding network, and finally connected to the classification output layer to predict the emotion category.
[0017] Furthermore, the network model training method is to use the Adam optimizer to minimize the cross entropy loss function, the learning rate is set to 0.0001, the batch size is 100, and L2 regularization is used to prevent the model from overfitting.
[0018] Beneficial effects of the present invention:
[0019] (1) The network structure of the present invention adopts a hierarchical approach. The first layer sub-network is used to encode primary features, and the second layer network is used to map the primary features to high-level features, which can effectively establish a hierarchical structure relationship between high-level and low-level features.
[0020] (2) The present invention makes full use of the feature complementarity between speech and text information, and can capture complex associations between audio and text. Compared with the prior art methods, the recognition rate of the model of the present invention is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a system model diagram of the present invention;
[0022] Figure 2 It is a decomposed bilinear pooling feature fusion module (FBP) graph;
[0023] Figure 3 It is the confusion matrix of the algorithm proposed in this paper on the IEMOCAP dataset;
[0024] Figure 4 Model loss function change diagram;
[0025] Figure 5 Precision, recall, and F-value curves. DETAILED DESCRIPTION
[0026] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.
[0027] System Model
[0028] The model structure proposed in the present invention includes three layers. The first layer is low-level feature fusion, which uses a decomposition bilinear pooling fusion module (FBP) to first fuse the speech feature vector and the text feature vector. The second layer is high-level feature fusion. The low-level features output by the FBP module are respectively output by the three sub-models of LSTM, GRU, and DNN for secondary fusion, and the high-level features are encoded. The fusion method is to splice two by two hadmard products. The third layer is the sentiment classification layer. The obtained high-level feature vector is input into the fully connected layer. The fused high-level sentiment features are reduced in dimension in the fully connected layer. Finally, the sentiment coefficient at this time is output through the softmax function and sentiment judgment is made. The system model diagram is as follows Figure 1 shown.
[0029] Speech signal preprocessing
[0030] The speech emotion feature of the present invention adopts Mel spectrum, which is a general spectrum diagram plus Mel filter function. The extraction process is as follows.
[0031] Given a speech signal S, after windowing and framing, it is represented as S = {s1, s2, s3..., s n}, where n represents the total number of speech frames. The window length used in this part of the topic is a 25ms Hamming window with a frame shift of 10ms. After that, the Fourier transform of each frame in S is calculated to obtain the frequency domain representation x of each frame. t . As the formula shows:
[0032]
[0033] Where M represents the number of Fourier transforms. t (k) represents x t The kth value in the vector.
[0034]
[0035] Then perform Mel transformation according to the above formula and convert x t The frequency of is transformed from the linear scale to the Mel scale, where f represents the frequency scale after Fourier transform. Then a set of filters are designed on the Mel scale to filter the spectrum of each frame, and finally the Mel spectrum features are obtained.
[0036] Text signal preprocessing
[0037] The present invention uses the famous GloVe algorithm to preprocess text information by representing each word with a vector. The main idea of the GloVe algorithm is to intuitively represent the relationship between words i and j through the co-occurrence matrix. The words are vectorized so that the vectors contain as much semantic and grammatical information as possible. The algorithm flow is as follows:
[0038] 1. Construct the co-occurrence matrix X and calculate the probability. Its mathematical expression is:
[0039]
[0040] where p ij Represents the co-occurrence probability of words i and j appearing in the context.
[0041] 2. Construct an approximate relationship between the word vector and the co-occurrence matrix X. Given a word k that may appear near them, determine its correlation with i and j.
[0042]
[0043] The correlation between k and i and j can be obtained. If the correlation is small, the result of the formula is close to 1, and if the correlation is large, the result of the formula is far away from 1. ik represents the co-occurrence probability of words i and k, p jk represents the co-occurrence probability of words j and k.
[0044] 3. Construct loss function
[0045]
[0046] Where V represents the size of the vocabulary, w i and w j is the word vector, b i and b i is the bias, f(x ij ) is the weighting function.
[0047] The weight function is expressed as follows:
[0048]
[0049] Based on experience, Glove authors believe that x max =100, is a better choice. If two words have never appeared together, the weight is 0; f(x) must be a non-decreasing function.
[0050] 4. Training Model
[0051] For the text data in the IEMOCAP dataset, the trained GloVe word embedding model is used to vectorize the text information with a dimension of 300. The maximum sequence length of the text data is kept at 500, and finally a (500, 300)-dimensional word vector is obtained for the study of text sentiment analysis.
[0052] Speech to text alignment
[0053] This section mainly introduces the implementation method of speech and text alignment. Given a speech representation S = {s1,...,s i ,...,s n}, where n represents the number of speech frames. Given a text word vector representation T = {t1,...,t j ,...,s m}, where m represents the number of words in the text. Since one word corresponds to multiple frames, the speech features of each word can be obtained by merging the speech frames related to the word, which means that there is a strong connection between each word and its related speech frames. From this point of view, we first calculate the similarity between the word and the speech frame to obtain the relationship between the word and the speech frame, as shown in the formula:
[0054]
[0055] where t j represents the jth word vector, s i represents the i-th speech frame, a ij Represents the similarity between the jth word vector and the i-th speech frame. The larger the value, the more similar the two vectors are.
[0056] Finally, a ij Perform normalization operation to obtain the probability values of word j and speech frame i.
[0057]
[0058] Finally, the weighted sum of all frames is taken to obtain the speech feature u aligned with the jth word. j , the aligned speech is represented as U = {u1,...u j ,...,u m}.
[0059]
[0060] Dataset Introduction
[0061] The IEMOCAP interactive emotion binary motion capture dataset is a widely used English multimodal emotion recognition dataset recorded by the SALL laboratory of the University of Southern California. It is divided into 5 sessions, each of which consists of 5 groups of 10 people, 5 men and 5 women, and is widely cited in the field of speech recognition. The IEMOCAP multimodal dataset includes a total of about 12 hours of audio and video files, including data information in three modes: audio, semantic text and facial expressions. The speech in the dataset is divided into ten emotions: happy, angry, sad, frustrated, natural, excited, fearful, disgusted, surprised and others. In order to be consistent with existing work, we only use four emotion categories in this work: angry, sad, happy and natural. The distribution of the four emotion categories is shown in the following table.
[0062] The fusion algorithm flow of the present invention
[0063] 1. Decompose the bilinear pooling fusion module (FBP) as follows Figure 2 As shown, x and y represent the preprocessed speech features and text features, x∈R m ,y∈R n The specific operations are as follows.
[0064]
[0065] angry happy nature sad total total 1103 1636 1707 1084 5531
[0066]
[0067] K is the potential dimension of the decomposition matrix, and the matrix U i =[u1,…,u k ]∈R m×k , V i =[v1,…,v k ]∈R n×k ,1∈R k . is the hadmard product operation.
[0068] In order to obtain the output feature z∈R O , the weights to be learned are two third-order tensors U = [U1,…,U O ]∈R m×k×o and V=[V1,…,VO ] ∈R n×k×o We can reformulate U and V as two-dimensional matrices and Then the reshape operation is performed, and the formula is as follows.
[0069]
[0070] The function SumPooling(x,k) represents the use of a one-dimensional non-overlapping window of size k to perform SumPooling on x. Dropout is used to prevent overfitting. Because the output amplitude changes dramatically due to the introduction of element-by-element multiplication, L2 normalization is used to normalize the energy of Z to 1.
[0071]
[0072] 2. The second part of the network framework of the present invention uses three different LSTM, GRU, and DNN neural networks to extract deep features and obtain different representations. The primary fusion feature vector z output by the FBP module t After LSTM, GRU and DNN network output respectively and The formula is shown below.
[0073]
[0074]
[0075]
[0076] 3. Then the high-level feature fusion output part integrates the three types of features into a vector through the pairwise hadmard product and outputs the high-level fusion feature C at that moment. t, as shown below.
[0077]
[0078] Among them C t ={C1,C2,…,C m}, m represents the number of words. Here, the symbol ⊙ represents the multiplication of the corresponding elements of the vector.
[0079] 4. Then we merge the vector C from the previous step t Feed it into the BiLSTM network for encoding. The specific operation is as follows. Reverse eigenvector
[0080]
[0081]
[0082]
[0083] 5. Finally, the feature vector is imported into the fully connected layer for dimensionality reduction, and the Softmax function is used to obtain It is used for sentiment classification of discourse and uses the cross entropy loss function. The specific operation is as follows.
[0084]
[0085]
[0086] where y k represents the standard distribution of sample categories, is the predicted distribution of sample categories. θ (.) represents a fully connected layer network with parameter θ.
[0087] Network training
[0088] When training the model, the Adam optimizer was used to minimize the cross entropy loss function, the learning rate was set to 0.0001, the batch size was 100, and L2 regularization was used to prevent the model from overfitting.
[0089] The Adam optimization algorithm estimates the first-order moment and second-order moment of the gradient of the cross entropy loss function and applies it to parameter update. The specific operation is as follows.
[0090] m t =β1·m t-1 +(1-β1)·g t
[0091]
[0092]
[0093] m t and v t They are estimates of the first-order moment and second-order moment of the gradient of the cross entropy loss function, and are attenuated by hyperparameters β1 and β2. t represents the gradient of the function, θ t Updates the value of the variable, and Ir represents the learning rate.
[0094] The network parameter settings are shown in the following table:
[0095] Simulation parameter table
[0096]
[0097]
[0098] Experimental results:
[0099] The confusion matrix of the experimental results of the efficient fusion algorithm model we proposed on the IEMOCAP dataset is as follows: Figure 3 As shown in the figure, the confusion matrix shows that the accuracy rates of the four emotions of happy, sad, natural and angry are 91.69%, 73.95%, 73.68% and 75.16% respectively, and the overall accuracy rate reaches 80.38%. This verifies the effectiveness and feasibility of our proposed method on the IEMOCAP dataset.
[0100] This paper mainly uses WA weighted accuracy and UA unweighted accuracy to measure the performance of the model, where WA is used to evaluate the overall performance of the model and UA is used to evaluate the model's classification results for each type of emotion. The specific calculation formula is shown below:
[0101]
[0102]
[0103] Where L is the number of emotion categories, TP i Indicates the number of correctly predicted samples of the i-th category, FN i Indicates the number of samples of the i-th class that are predicted incorrectly.
[0104] In order to further measure the performance of the speech emotion recognition model proposed in the present invention, precision, recall and F1 score are used to highlight the performance of the model. The calculation formulas of precision, recall and F1 score are as follows.
[0105]
[0106]
[0107]
[0108] Among them, TP means that both the prediction and the label are positive; FP means that the predicted positive label is negative; TN means that both the prediction and the label are negative; FN means that the predicted negative label is positive. Precision and Recall refer to precision and recall respectively, and the F1 score is the harmonic mean of the two. For the F1 score, the larger the value, the better. The experimental indicators related to precision, recall and F1 score are shown in the following table.
[0109] TP FN FP precision recall F1-score happy 397 36 93 0.8102 0.9169 0.8602 sad 176 62 31 0.8502 0.7395 0.7910 neutral 280 100 82 0.7735 0.7368 0.7547 anger 118 39 31 0.7919 0.7516 0.7712 Average 0.8065 0.7862 0.7943
[0110] Comparison with other methods
[0111] In order to better verify the effectiveness and scientificity of the improved algorithm in this paper, the experimental results of the algorithm in this paper are compared with other algorithm models on the IEMOCAP dataset, and the results are shown in the following table.
[0112] 1. Hengshun Zhou et al. used the bilinear pooling algorithm as a feature fusion method. (ACM2021)
[0113] 2. Haiyang Xu et al. used speech and text feature concatenation and fusion. (INTERSPEECH2019)
[0114] 3. Md Asif Jalal et al. use multiple attention output splicing fusion method (INTERSPEECH2020)
[0115] 4. Ming Chen et al. adopted a mixed splicing and then a separate splicing method. (INTERSPEECH2020)
[0116] 5. Qi Cao et al. use speech and text consistency and contextual attention mechanism to operate the fusion method. (2021)
[0117] Comparison table with different models
[0118]
[0119] As shown in the table above, the method proposed in the present invention achieved 80.38% WA and 78.62% UA in the four-category emotion recognition experiment of the IEMOCAP dataset. WA is the best experimental result in all comparative experiments. The worst result is that Haiyang Xu et al. used a simple linear splicing method to perform multimodal fusion, with only 70.4% WA and 69.5% UA. This may be because this simple splicing method failed to fully capture the complementary relationship between the two modalities. MingChen et al. added two separate features in parallel on the basis of Haiyang Xu et al., taking into account the correlation between the two modalities and within the modality, and obtained 71.06% WA and 72.05% UA, and the performance was improved. Yuanyuan Zhang et al. used a bilinear pooling method to fuse the two features and achieved a good effect of 75.49% WA. The present invention performs feature fusion again on the basis of bilinear pooling feature fusion to obtain rich cross-modal feature representation, thereby improving 4.89% WA compared to the previous method. Qi Cao et al. used the fusion method of speech and text consistency and context attention mechanism to achieve 78.74% WA and 79.77% UA, which is slightly higher than our method in UA. In summary, the above comparative experiments verify that the algorithm proposed in this invention has obvious advantages over other algorithms.
[0120] In view of the shortcomings of the current feature fusion methods in the field of multimodal emotion recognition, a speech emotion recognition method based on a two-layer fusion deep network is proposed, which can capture complex associations between audio and text. It is proposed to first perform the first fusion based on the decomposition bilinear pooling fusion module (FBP), and then perform the second fusion. It is better than the concatenation and summation methods, and can make the features of each modality more fully integrated with lower time complexity. The network structure adopts a hierarchical approach. The first layer of sub-network is used to encode the primary features, and the second layer of network is used to map the primary features to high-level features, which can effectively establish the hierarchical structure relationship between high-level and low-level features. Finally, the experiment was conducted on the IEMOCAP dataset, achieving 80.38% WA and 78.62% UA. Compared with the related work of other researchers, the recognition rate has been greatly improved, which verifies the feasibility and effectiveness of the efficient fusion algorithm we proposed.
[0121] The protection scope of the present invention shall be based on the description of the claims.
Claims
1. A speech emotion recognition method based on a double-layer fusion deep network, characterized in that: The following steps are involved: Step 1: Preprocess the speech signal and the text signal, and perform alignment operations to make them meet the input requirements of the network model; Step 2: Input the speech and text feature vectors preprocessed in step 1 into the decomposition bilinear pooling fusion module FBP for primary feature fusion; Step 3: The fused primary features output by the bilinear pooling fusion module FBP in step 2 are respectively passed through the level-1 primary feature encoding network composed of three sub-models: LSTM, GRU and DNN; Step 4: Then perform secondary fusion on the outputs of the three sub-models of the level-1 primary feature encoding network to encode high-level features. The fusion method is the hadmard product. The fused features are then input into the level-2 BiLSTM encoding network and finally connected to the classification output layer to predict the sentiment category. Step 5: Finally train the network model; The structure of the network model includes four layers in total, the first layer is the FBP primary feature fusion layer; The second layer is the level-1 primary feature encoding network layer composed of three sub-models: LSTM, GRU and DNN; The third layer is the hadmard product fusion layer; the fourth layer is the advanced encoding network layer composed of BiLSTM; The network model training method described is to use the Adam optimizer to minimize the cross entropy loss function, the learning rate is set to 0.0001, the batch size is 100, and L2 regularization is used to prevent the model from overfitting.
2. The speech emotion recognition method based on a double-layer fusion deep network according to claim 1 is characterized in that: The text signal preprocessing refers to adopting a word embedding method and using a pre-trained Glove model to represent each word with a vector.
3. The speech emotion recognition method based on a double-layer fusion deep network according to claim 1 is characterized in that: The speech signal preprocessing refers to windowing and framing the audio signal, using a Hamming window with a window length of 25ms and a frame shift of 10ms, then performing Fourier transform, and finally performing a Mel filter operation to obtain Mel spectrum features.
4. The speech emotion recognition method based on a double-layer fusion deep network according to claim 1 is characterized in that: The alignment operation is to obtain the speech features corresponding to each word by merging the speech frames related to the words.
