A speech emotion recognition method based on fusion of dynamic and static features by bilinear pooling

Abstract

The application discloses a speech emotion recognition method based on bilinear pooling fusion of dynamic and static features. The method first extracts 24-dimensional static features from the collected speech data, and then calculates dynamic features based on the static features. Then, the LSTM network is used to extract the time sequence feature representation of the dynamic and static features, and after the bilinear fusion, the attention mechanism is used for fusion to obtain the utterance-level features. After power normalization, the full connection layer is input to identify the emotion category. The method fuses the static features and dynamic features of the speech data through bilinear pooling, so that they can learn from each other, thereby generating more rich feature representation. At the same time, the attention mechanism is used to replace the pooling operation in the traditional bilinear pooling method, and the recognition result is more advantageous.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology and relates to speech emotion recognition, specifically to a speech emotion recognition method based on bilinear pooling fusion of dynamic and static features. Background Technology

[0002] As the most natural and effective means of communication for humans, speech not only contains semantic information but also a wealth of emotional information, reflecting the speaker's inner mental or emotional state. With the rapid development of computer technology, human-computer interaction scenarios are becoming increasingly diverse, making speech emotion recognition increasingly important in these interactions. For example, in robot communication, if robots can understand human emotional states through speech, they can choose appropriate tones to communicate with the speaker; in intelligent driving, if in-vehicle intelligent driving systems can detect the driver's mental and emotional state in real time through speech, many traffic accidents caused by road rage can be avoided.

[0003] Speech emotion recognition methods consist of two stages—feature extraction and feature classification. In recent years, researchers have conducted extensive research and achieved fruitful results in acquiring features suitable for speech emotion recognition. Commonly used features in existing technologies include prosodic features, phonological features, and spectral correlation features. However, due to the strong subjectivity of speech emotion, finding suitable features remains a significant challenge. Many researchers only select static features for emotion discrimination, ignoring the changes in the emotion expression process, i.e., dynamic features. Some researchers also choose to directly concatenate or combine static and dynamic features into multi-channel features for emotion discrimination, but they ignore the mutual interference between features, all of which lead to poor emotion recognition results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a speech emotion recognition method based on bilinear pooling fusion of dynamic and static features. By fusing dynamic and static features through bilinear pooling, complementary learning between features is achieved to generate a more comprehensive and unified feature representation, effectively improving the speech emotion recognition performance.

[0005] A speech emotion recognition method based on bilinear pooling fusion of dynamic and static features includes the following steps:

[0006] Step 1: Collect voice data samples, set sample labels according to voice emotion, and perform equalization processing on data of different categories to serve as training set.

[0007] Step 2: Preprocess the speech data in the training set to obtain training samples.

[0008] Step 3: Extract the static features of the training samples, including the ISO9 feature set and 8 log Mel frequency bands. Then, calculate the first-order difference for the static features as the dynamic features of the training samples.

[0009] Step 4: Preprocess the static and dynamic features separately to ensure that the number of frames for the sample features is consistent, and then perform normalization processing.

[0010] Step 5: Construct a parallel single-layer LSTM network. Input the static and dynamic features processed in Step 4 into the LSTM network to extract temporal features. Then, use bilinear pooling to fuse the temporal features into higher-dimensional features, and use the attention mechanism to further fuse them into discourse-level features. After power normalization, input the features into a fully connected layer and use the Softmax function for sentiment discrimination.

[0011] Step 6: Optimize the network structure from Step 5, then input the static and dynamic features of the processed speech data to identify the corresponding emotions.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] This method fuses static and dynamic features using bilinear pooling to generate richer feature representations, allowing them to learn complementarily. Furthermore, it replaces the traditional pooling operation with an attention mechanism, assigning weights to each speech frame and then weighting and summing them to create speech-level features, offering advantages over sum pooling or max pooling. Compared to existing techniques using single static features, single dynamic features, spliced ​​features, or multi-channel features, this method significantly improves the accuracy of emotion recognition in speech samples. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the dynamic and static feature fusion network structure constructed in the embodiment;

[0015] Figure 2 This is the speech emotion recognition confusion matrix obtained in the example. Detailed Implementation

[0016] The present invention will be further explained below with reference to the accompanying drawings;

[0017] A speech emotion recognition method based on bilinear pooling fusion of dynamic and static features includes the following steps:

[0018] Step 1: Conduct emotion recognition experiments using speech data from the publicly available dataset IEMOCAP. Speech data representing four different emotions—anger, sadness, happiness, and neutral—were selected, ensuring a balanced number of samples across each category, to construct the training set.

[0019] Step 2: Sample the speech data in the training set at a frequency of 16KHz, and then perform pre-emphasis, framing, and windowing operations, with a frame length of 25ms and a frame shift of 10ms, to obtain the training samples.

[0020] Step 3: Use the OpenSmile toolkit to extract features from the training samples, including zero-crossing rate, square root of energy, F0 (pitch), HNR (signal-to-noise ratio), MFCC-12, and 8 log Mel frequency bands, obtaining 24-dimensional data as the static features of the samples. Then, calculate the first difference of these 24-dimensional features as the dynamic features:

[0021]

[0022] Where c t Let x represent the dynamic characteristics of the sample in frame t, N represent the differential frame interval, and n = 1, 2, ..., N. t This represents the static features of the sample in frame t.

[0023] Step 4: Perform truncation and padding operations on static and dynamic features respectively, ensuring that the maximum sequence length of the data is 500 frames. The size of the processed sample features is T×D, where T is the number of frames and D is the feature dimension. Then, normalize the sample features using Z-Score standardization.

[0024] Step 5, as follows Figure 1 As shown, a parallel single-layer LSTM network is constructed with a hidden layer size of 80. The static and dynamic features processed in step four are input as inputs to obtain the network's output representation. Then the static and dynamic temporal features of frame t are respectively Where M1 and M2 are the dimensions of the static and dynamic temporal features output by the LSTM network, respectively. The static and dynamic features of frame t are fused according to the following formula:

[0025]

[0026] The fused features are represented as h = [h1, h2, ..., h...] T ] T ,

[0027] Traditional bilinear pooling methods require pooling of the fused features after bilinear fusion, typically using sum pooling or max pooling. This application employs an attention mechanism, assigning weights to the fused features based on the different contributions of each speech frame to emotion classification. The size of the attention mechanism is set to 16.

[0028] s t =V T tanh(Wh t +b)

[0029]

[0030]

[0031] Where V and W are h t The weight matrix, b represents the bias, s t h t The sentiment score, α t Let represent the weights assigned to the features of frame t, and u represent the fused discourse-level features.

[0032] The utterance-level feature u is input into a fully connected layer after power normalization, and finally sentiment is determined by the Softmax function. The power normalization formula is as follows:

[0033]

[0034] Where sign() represents the sign function and z represents the power normalization result.

[0035] Cross-entropy was chosen as the loss function, and the Adam optimizer was used to optimize the above network structure. The learning rate was set to 0.00005, and ten-fold cross-validation was adopted. Weighted precision (WA) and unweighted precision (UA) were used as evaluation metrics. The experimental results of the LLDs static features extracted by this method were compared with ISO9 features and 8 log Mel-spectrogram features, as shown in Table 1.

[0036]

[0037]

[0038] Table 1

[0039] Ablation experiments were conducted on the differential frame interval N when calculating dynamic features. The experimental results are shown in Table 2.

[0040]

[0041] Table 2

[0042] The data in the table shows that the experimental results are optimal when the interval for calculating the differential frame is 2. The corresponding confusion matrix is ​​as follows: Figure 2 As shown.

Claims

1. A speech emotion recognition method based on bilinear pooling fusion of dynamic and static features, characterized in that, Specifically, the following steps are included: Step 1: Collect voice data samples, set sample labels according to voice emotion, perform equalization processing on data of different categories, and construct a training set; Step 2: Preprocess the speech data in the training set to obtain training samples; Step 3: Extract the static features of the training samples, including ISO9 features and 8 log Mel frequency bands; then calculate the first-order difference for the static features as the dynamic features of the training samples. Step 4: Preprocess the static and dynamic features separately to ensure that the number of frames for the sample features is consistent, and then perform normalization. Step 5: Construct a parallel single-layer LSTM network. Input the static and dynamic features processed in Step 4 into the LSTM network to extract temporal features, and then use bilinear pooling to combine the temporal features. , Integration into higher-dimensional features: Where T represents the number of training sample frames, M1 and M2 are the dimensions of the static and dynamic temporal features output by the LSTM network, respectively; for the fused feature vector... Pooling is performed using an attention mechanism, with the attention size set to 16. The attention mechanism is as follows: in, and for The weight matrix, Indicates bias. express Emotional score, This represents the weight assigned to the features of frame t. This represents the discourse-level features after fusion; After power normalization, the data is input into a fully connected layer and sentiment is determined using the Softmax function. Step 6: Optimize the network structure from Step 5, then input the static and dynamic features of the processed speech data to identify the corresponding emotions.

2. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1, characterized in that: The speech data in the training set is sampled at a frequency of 16KHz, and then pre-emphasis, framing, and windowing operations are performed, with a frame length of 25ms and a frame shift of 10ms, to obtain the training samples.

3. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1, characterized in that: The static characteristics include zero-crossing rate, square root of energy, F0 (pitch), HNR (signal-to-noise ratio), MFCC-12, and 8 logMel frequency bands.

4. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1, characterized in that: The calculation method for the dynamic features is as follows: in Indicates the first The dynamic characteristics of frame samples, where N represents the differential frame interval, n=1,2,…N. Indicates the first Static features of frame samples.

5. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1, characterized in that: Step four involves truncating and padding static and dynamic features to ensure a maximum sequence length of 500 frames. Then, Z-Score standardization is used to normalize the sample features.

6. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1, characterized in that: The hidden layer size of the single-layer LSTM network is 80.

7. The speech emotion recognition method based on bilinear pooling fusion of dynamic and static features as described in claim 1 or 6, characterized in that: Cross-entropy was chosen as the loss function, the learning rate was set to 0.00005, and the Adam optimizer was used to optimize the network structure in step five.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 6.

Images