A speech emotion recognition method that integrates emotion-related features of historical sentences

Abstract

This invention discloses a speech emotion recognition method that integrates emotion-related features from historical statements. First, features are extracted from the collected speech data, resulting in 45-dimensional features including MFCC (Multi-Folded Cognitive Capabilities). Then, two historical statements from the speaker and the interlocutor are processed using an IFCN-LSTM network to extract historical emotion-related features. The speaker's current statement is then processed by an LSTM network. Finally, the historical emotion-related features are integrated into the current statement as the final judgment statement. This invention employs the method of integrating historical emotion-related features into the current statement. By grasping the fine granularity of historical emotion, the network focuses more on the emotionally prominent parts of the current statement, thereby improving recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, and relates to artificial intelligence-based speech emotion recognition, specifically to a speech emotion recognition method that integrates emotion-related features of historical sentences. Background Technology

[0002] Artificial intelligence primarily uses three methods to determine emotions: speech-based, facial expression-based, and body language-based. Speech signals are undoubtedly the easiest information to collect. Emotion determination through speech data collection is easy to implement, highly portable, and widely applicable, finding applications in sectors such as service, education, and healthcare. Speech not only contains the semantic information the speaker intends to express but also implies their emotional state. Speech-based emotion recognition determines a speaker's emotional state by collecting acoustic feature information such as tone, intonation, and keywords. Computer speech emotion recognition capabilities are a crucial component of computational emotional intelligence and an important research direction for improving human-computer interaction.

[0003] The process of speech emotion recognition generally consists of three modules: feature extraction, emotion classification, and emotion discrimination. Commonly used speech emotion features include spectral features, prosodic features, and phonological features. However, most current speech emotion classification methods only discriminate the emotion of the speaker's current utterance, ignoring emotion-related features in historical utterances, which significantly reduces the accuracy of emotion analysis. Speech emotion recognition plays an irreplaceable role in human-computer interaction, therefore, more advanced and accurate speech emotion recognition methods are of great significance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a speech emotion recognition method that integrates emotion-related features from historical sentences. This method extracts emotion-related features from historical sentences and integrates these features into the current sentence, thereby improving the accuracy of emotion recognition for the current sentence.

[0005] A speech emotion recognition method that integrates emotion-related features of historical statements specifically includes the following steps:

[0006] Step 1: Collect voice data, including at least two historical sentences in addition to one sentence to be recognized.

[0007] Step 2: Create a conversation model. Designate the sender of the statement to be identified as speaker T, and the others as conversation partners Q. Designate the statement to be identified as the current statement, denoted as T0, and the other statements as historical statements, denoted as Tn and Qn respectively, based on the number of statements between them and the current statement and their senders.

[0008] Step 3: Extract 45-dimensional acoustic features from the current statement and historical statements, including 12 Mel cepstral coefficients (MFCC), 26 Mel-derived parameters, 3 sound quality features, and 6 short-time statistical features.

[0009] Step 4: Construct a cascaded IFCN network and LSTM network. The IFCN network replaces the fully connected layers of the CNN network with two convolutional layers and adds three deconvolutional layers after the convolutional layers. The upsampling results of the deconvolutional layers are added to the features of the same dimension output by the pooling layers. The acoustic features of the historical sentences extracted in Step 3 are input into the cascaded IFCN and LSTM networks to obtain the historical sentiment-related features of the historical sentences. Then, the acoustic features of the current sentence are input into the LSTM network for aggregation to obtain the current sentiment features of the current sentence.

[0010] Step 5: Fuse the historical sentiment-related features obtained in Step 4 with the current sentiment features, and use them as discriminative features. Input these features into the softmax function for sentiment classification.

[0011] The present invention has the following beneficial effects:

[0012] 1. The IFCN network is used to extract relevant spatial features. The last fully connected layer in the traditional CNN network is replaced with a convolutional layer to solve the problem of inputting sentences of different lengths. Upsampling is performed by deconvolutional layers after the convolutional layers, which can restore the speech sequence to the sequence length at the time of input. While extracting relevant spatial features, the original temporal feature sequence is not disrupted.

[0013] 2. A combination of IFCN and LSTM is used to extract historical sentiment-related features. IFCN can extract spatially relevant features and remove redundant features while restoring the speech time series to the input state. LSTM can associate the correlation between sentiment features in the time series during the processing of the time series, thus effectively extracting spatial features and associating the temporal correlation of time features, and extracting the best sentiment-related features from the historical sentences.

[0014] 3. By integrating historical emotion-related features with current emotion features and using attention mechanisms to grasp the fine granularity of historical emotions, the network can focus more on the emotionally prominent parts of the current statement, thereby effectively identifying the current emotional state. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the IFCN network structure;

[0016] Figure 2 A schematic diagram illustrating the principle of a speech emotion recognition method that integrates emotion-related features from historical statements. Detailed Implementation

[0017] The present invention will be further explained below with reference to the accompanying drawings;

[0018] A speech emotion recognition method that integrates emotion-related features of historical statements specifically includes the following steps:

[0019] Step 1: In this embodiment, the IEMOCAP public dataset is used as the source of speech data for emotion recognition.

[0020] Step 2: Create a conversation model. Designate the sender of the statement to be identified as Speaker T, and the others as Practitioners Q. Designate the statement to be identified as the current statement, denoted as T0, and the other statements as historical statements. Mark these historical statements according to the number of statements preceding the current statement and the sender. For example, the previous statement for Practitioner Q is denoted as Q1, the previous statement for Speaker T as T2, and so on.

[0021] Step 3: Sample the current and historical statements at a frequency of 16kHz, and then use the OpenSmile toolkit to extract features by frame segmentation, with a frame length of 25ms and a frame shift of 10ms. This yields 45-dimensional acoustic features, including 12 MFCC features, 26 Mel-derived parameters, 3 phonology features, and 6 short-time statistical features.

[0022] Step 4: Construct a cascaded IFCN and LSTM network, such as... Figure 1 As shown, the IFCN network comprises 5 convolutional layers, 3 pooling layers, and 3 deconvolutional layers. It retains the 3 convolutional layers + pooling layers structure of a CNN network, then replaces the subsequent fully connected layers with two convolutional layers. The input data is mapped to two-dimensional spectral features according to the time frame order. The two replaced convolutional layers ensure that the IFCN network is suitable for inputs of different lengths. Then, upsampling is performed through deconvolutional layers. The upsampling process restores the feature sequence to its original spatial structure without disrupting its temporal sequence. After each upsampling, the resulting feature sequence is added to the features with the same dimensional structure after pooling. After two more upsampling operations, the feature sequence is directly mapped back to its input size through deconvolutional layers according to the dimensional structure at the time of input.

[0023] like Figure 2 As shown, inputting historical statements Q1 and T2 into the recognition model yields historical sentiment-related features h. c h o Then, the current statement T0 is input into the optimized LSTM network to obtain the current sentiment features.

[0024] Step 5: Then, the historical sentiment-related features h obtained in Step 4 are fused using the Bahdanau attention mechanism. c h oBy fusing with current sentiment features, discriminative features h are obtained. t :

[0025] f i =tanh(Wh) i +W c h c +W o h o +b) (1)

[0026]

[0027]

[0028] Where W, Wc, and Wo are respectively h i h c h o The weights, b is the bias, h i This represents the feature of the i-th time step after the current statement is divided into 1 to T time steps, α. t The weight is calculated by representing the proportion of time i to the total time.

[0029] Then the discriminant feature h t After passing through the ReLU activation layer, the data is input into the softmax function for sentiment classification.

[0030] The cross-entropy after softmax regression is used as the loss function, and the network parameters are adjusted using the Adam optimization algorithm to train the entire network. The optimized network is then used as the recognition model.

[0031] The classification results of this method are compared with those of two other classification methods using weighted accuracy (WA) and unweighted accuracy (UA) as evaluation criteria, as shown in the table below:

[0032]

[0033] Method 1 involves classifying historical sentiment features directly by fusing them with the current statement using an attention mechanism, without extracting historical sentiment-related features. Method 2 involves classifying historical sentiment-related features by linearly concatenating them with the current statement. The data in the table shows that the weighted accuracy (WA) and unweighted accuracy (UA) of this method are significantly higher than the other two methods, indicating that this method has a higher recognition accuracy.

Claims

1. A speech emotion recognition method that integrates emotion-related features of historical statements, characterized in that: Step 1: Collect voice data, including at least two historical sentences in addition to one sentence to be recognized; Step 2: Create a conversation model, designating the sender of the statement to be identified as speaker T, and the others as conversation partners Q; The statement to be identified is taken as the current statement and denoted as T0. Other statements are taken as historical statements and denoted as Tn and Qn respectively according to the number of statements between them and the current statement and the issuer. Step 3: Extract 47-dimensional acoustic features from the current statement and historical statements, including 12 Mel cepstral coefficients, 26 Mel derived parameters, 3 sound quality features, and 6 short-time statistical features; Step 4: Construct a cascaded IFCN network and LSTM network. The IFCN network is formed by replacing the fully connected layers of the CNN network with two convolutional layers and adding three deconvolutions after the convolutional layers. The upsampling results of the deconvolution layers are added to the features of the same dimension output by the pooling layers. The acoustic features of the historical sentences extracted in step three are input into the cascaded IFCN network and LSTM network to obtain historical sentiment-related features. The acoustic features of the current sentence are then input into the LSTM network for aggregation to obtain the current sentiment features; Step 5: Fuse the historical sentiment-related features obtained in Step 4 with the current sentiment features, and use them as discriminative features. Input these features into the softmax function for sentiment classification.

2. The speech emotion recognition method that integrates historical sentence emotion-related features as described in claim 1, characterized in that: Speech data was sampled at a frequency of 16kHz, and then the Opensmile toolkit was used to extract features by frame segmentation, with a frame length of 25ms and a frame shift of 10ms.

3. The speech emotion recognition method that integrates historical sentence emotion-related features as described in claim 1, characterized in that: The IFCN network includes 5 convolutional layers, 3 pooling layers, and 3 deconvolutional layers. The deconvolutional layers are used for upsampling to restore the feature sequence to its original spatial structure without disrupting its temporal sequence. After each upsampling, the obtained feature sequence is added to the features of the same dimensional structure output by the pooling layer. Finally, the output data is mapped to the input size according to the dimensional structure at the time of input.

4. The speech emotion recognition method that integrates historical sentence emotion-related features as described in claim 1, characterized in that: Discriminant features h are obtained through an attention mechanism. t : f i =tanh(Wh i +W c h c +W o h o +b) (1) Where W, Wc, and Wo are respectively h i h c h o The weights, b is the bias, h i This represents the feature of the i-th time step after the current statement is divided into 1 to T time steps, α. t The weight is calculated by representing the proportion of time i to the total time.

5. The speech emotion recognition method that integrates historical sentence emotion-related features as described in claim 1, characterized in that: The network was trained by using the cross-entropy after softmax regression as the loss function and the Adam optimization algorithm to adjust the network parameters. The optimized network was then used as the recognition model for sentiment classification.

Images