A speech emotion recognition system based on hybrid entropy down-sampling and ensemble classifier

Abstract

This invention discloses a speech emotion recognition system based on hybrid entropy downsampling and an ensemble classifier. The system includes the following steps: In a preprocessing stage, the speech signal of the training data is segmented into segments, and spectrograms are extracted. These spectrograms are used to train a base classifier, and the deep features and confidence scores of each speech segment are obtained. In a training stage, the hybrid entropy of all speech segments is calculated, and its weighted sum with the confidence scores is used as the ranking value. Subsequently, the spectrograms of speech segments with ranking values ​​greater than a set threshold are used to retrain a base classifier. The ranking values ​​of all speech segments are calculated again, and the base classifier is trained again. This operation is repeated for a given number of rounds, and the base classifiers trained in each round form an ensemble classifier. Finally, the test speech is segmented, its spectrogram is extracted, and then input into the ensemble classifier to calculate the emotion recognition result of the speech. This invention significantly reduces the influence of speech segments with ambiguous emotions and unstable distribution structures, effectively improving the accuracy of speech emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of speech emotion recognition technology, and more specifically, to a speech emotion recognition system based on hybrid entropy downsampling and an ensemble classifier. Background Technology

[0002] Speech is the most direct and natural form of communication between people and a primary form of human-computer interaction. However, the emotions expressed in real-life speech are often complex, subtle, and constantly changing. Therefore, detecting and recognizing emotions in speech has become a challenging task. In recent years, speech emotion recognition has made significant research progress and has been widely applied in many fields such as virtual customer service, intelligent assistants, and medical auxiliary diagnosis. Speech emotion recognition systems typically consist of two parts: feature extraction and classifier training. Traditional methods involve segmenting the original speech waveform before manually extracting features. Commonly used classifiers in speech emotion recognition include Gaussian mixture classifiers and support vector machines. In recent years, with the development of deep learning, many methods based on deep learning classifiers have emerged, such as recurrent neural network classifiers and convolutional neural network classifiers.

[0003] Previous studies have found that the confidence level of each emotion varies depending on its position within different segments of the speech. For example, the true emotion label of a speech might be happiness, but the trained classifier results show that the confidence level for neutral emotions is highest in the first half of the speech, while the confidence level for happiness is highest in the second half. Clearly, the intensity of happiness in the first half is weaker, which is detrimental to classifier training. Speech segments with ambiguous emotions introduce noise into the classifier training process and degrade the performance of speech emotion recognition systems. Therefore, speech emotion recognition at the segment level remains challenging. While some methods exist to address this problem, such as attention mechanisms and multi-instance learning, these methods rely on deep learning classifiers to autonomously learn how to weight different parts of the speech, which is theoretically difficult to analyze and interpret. Summary of the Invention

[0004] This invention proposes a speech emotion recognition system based on hybrid entropy downsampling and an ensemble classifier. In each training round, speech segments with clear emotions are selected from all training data segments for the next round of training. This involves downsampling all training data segments, generating a base classifier in each round. These base classifiers form an ensemble classifier. During each training round, the hybrid entropy and confidence of the speech segments are calculated, and a ranking value is used to select samples with clear emotion categories. The ensemble classifier uses the base classifiers trained through multiple iterations to predict the emotion of the entire speech segment, effectively improving the accuracy of speech emotion recognition.

[0005] The speech emotion recognition system based on hybrid entropy downsampling and ensemble classifier in this project includes the following steps: 1) Divide the dataset into two parts: training data and test data. After dividing the speech signal of the training data into segments, extract the spectrograms and use them to train the base classifier to obtain the deep features and confidence of each speech segment. 2) Calculate the mixture entropy of all speech segments and use the weighted sum of the entropy and confidence as the ranking value; 3) The spectrograms of speech segments with ranking values ​​greater than a set threshold are used to retrain a base classifier, and the ranking values ​​of all speech segments and the base classifier are recalculated. This operation is repeated for a given number of rounds, and the base classifiers trained in each round form an ensemble classifier. 4) After testing the speech segmentation and extracting the spectrogram, input the data into the ensemble classifier to calculate the emotion recognition result of the speech.

[0006] The technical solution adopted in this invention can be further refined. The label of each speech segment is the true label of the entire speech segment in the dataset. In step 2), the mixed entropy of the speech segment consists of sentiment certainty entropy and structural distribution entropy. The sentiment certainty entropy is used to measure the salience of the sentiment expressed by the speech segment. The formula is: Where i is the ID of the speech segment in the training data, C is the number of emotion categories in the dataset, and k is the set number of nearest neighbors. The basic degree entropy of the i-th speech segment; Structural distribution entropy is used to measure the stability of the distribution structure of speech segments in the deep feature space. The formula is: Where i is the index of the speech segment in the training data, k is the set number of nearest neighbors, and d i,q Let represent the Euclidean distance between the deep features of the i-th speech segment and the q-th speech segment in the training data, and ln represents the calculation of the logarithm to the base e.

[0007] During the training of the base model, the ranking value calculated by the weighted sum of the mixture entropy and confidence is used as the basis for downsampling speech segments in each round. The mixture entropy of each speech segment is calculated by the sentiment certainty entropy and the structure distribution entropy. The formula for mixture entropy is: Where i is the number of the speech segment in the training data. Entropy represents the degree of certainty in sentiment. Represents the structural distribution entropy, where NOR is the Min-Max normalization function, and MIE is the structural distribution entropy.i Represents the mixing entropy of the i-th speech segment; The ranking value of each speech segment is defined as the weighted sum of the mixture entropy and the confidence obtained in step 1). The formula for the ranking value is: Rnak i =(1-λ)nor(conf) i )+λnor(-MIE i (4) Where i is the number of the speech segment in the training data, conf i MiE represents the confidence level of the i-th speech segment. i Let λ represent the mixing entropy of the i-th speech segment, λ be the weighting coefficient, and nor be the Min-Max normalization function. Rank i This represents the ranking value of the i-th speech segment.

[0008] In each round, the base model updates its parameters by minimizing the cross-entropy loss between the speech segment label and the speech segment sentiment classification results using gradient descent. Finally, the ensemble classifier, composed of the base classifiers generated in each round, calculates the sentiment category predicted by the system based on the output of each speech segment in the entire test speech.

[0009] The beneficial effects of this invention are as follows: This invention proposes a speech emotion recognition system based on hybrid entropy downsampling and an ensemble classifier. By selecting speech segments for training in multiple rounds of base classifier training, and combining the classifiers from each iteration into an ensemble classifier, the accuracy of speech emotion recognition is effectively improved. Compared with existing classifiers and base classifiers, it can significantly reduce the influence of speech segments with unclear emotion categories. This invention introduces the concept of hybrid entropy. The hybrid entropy of a speech segment includes emotion certainty entropy and structural distribution entropy. Using the ranking value calculated by hybrid entropy and confidence as a standard, samples with clear emotion categories and stable distribution structures can be effectively selected for training the ensemble classifier. Attached Figure Description

[0010] Figure 1 This is a structural diagram of a speech emotion recognition system based on hybrid entropy downsampling and an integrated classifier, according to the present invention. Detailed Implementation

[0011] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0012] Step 1: In the preprocessing stage, all raw emotional speech signals in the training data are divided into speech segments of 2 seconds each, for a total of N speech segments. There is no overlap between two speech segments. Speech segments shorter than 2 seconds are padded with zeros in the read signal values. Then, the signal values ​​of each speech segment are framed and windowed according to the requirements to extract the spectrogram as new training data. Where f is the number of frames, w is the length of the speech feature in each frame, and w is the corresponding training label for each speech segment. The true sentiment label of the entire speech in the training data;

[0013] Step 2: In each iteration l, train a new base classifier m. l The input is the training data consisting of spectrograms of speech segments in the l-th round. Its corresponding label in Y is Where n is the number of speech segments participating in the training of the base classifier in each round; when l=1, the number of speech segments in X1 is n=N, that is, the spectrograms of all speech segments participate in the training of the base classifier; each speech segment in the base classifier m l The final output is Where C represents the number of sentiment categories in the dataset, indicating the probability that the speech segment is predicted as each sentiment category by the base classifier. Then, the base classifier m... l Predicted labels During training, the loss function for the i-th speech segment is the true sentiment label y. i The cross-entropy loss -y′ between the base classifier output and the base classifier output i ·log(y i )-(1-y′ i )·log(1-y i The loss is minimized by gradient descent, and the parameters of the base classifier are updated. After a given number of gradient descent iterations, the trained base classifier m for this round can be obtained. l ;

[0014] Step 3: Input all training data, i.e., the spectrograms X of all speech segments, into the trained classifier m. l Among them, the spectrogram x of each speech segment i A depth feature of size z can be obtained in the penultimate fully connected layer of the classifier. Where i is the number of the speech segment, and the deep features corresponding to the spectrogram X of all speech segments can be denoted as F = {f i |i=1,2,…,N};This speech segment is in the base classifier m l The confidence level can be achieved through yy=y i Perform calculations;

[0015] Step 4: Calculate a k-nearest neighbor Euclidean distance matrix among the deep features F of the spectrograms of speech segments in all training data. and a k-nearest neighbor speech segment numbering matrix Used to calculate the sentiment certainty entropy and structural distribution entropy in mixed entropy;

[0016] Step 5: Calculate the mixture entropy on the deep features F of the spectrograms of speech segments in all training data:

[0017] Step 5.1: Calculate the sentiment certainty entropy: Emotional Certainty Entropy The formula is: Where i is the ID of the speech segment in the training data, C is the number of emotion categories in the dataset, and k is the set number of nearest neighbors. Let ln be the base degree entropy of the i-th speech segment, and let ln denote the calculation of the logarithm to the base e, as follows: Specifically, the entropy of sentiment certainty In the formula, This represents the number of segments corresponding to the sentiment category label with the largest number of segments among the k segments with the closest Euclidean distance to the i-th speech segment in the deep features of the training data. The formula is as follows: in, This represents the number of speech segments with emotion category label j among the k speech segments that have the closest Euclidean distance to the i-th speech segment in the training data, calculated from the deep features. It is represented by matrix M. ind Calculate, for the i-th speech segment, For M ind The actual label y of the speech segment corresponding to the k nearest neighbor speech segment number in the i-th row. i The number of segments for j.

[0018] Step 5.2: Calculate the structural distribution entropy: Where d i,q ∈M dis The deep feature f represents the deep feature f of the i-th speech segment on the deep feature F. i The deep features f of the q-th speech segment q The Euclidean distance.

[0019] Step 5.3: The mixture entropy of each speech segment is calculated from the sentiment certainty entropy in Step 5.1 and the structural distribution entropy in Step 5.2: Where nor is the Min-Max normalization function.

[0020] Step 6: Calculate the rank value of the deep feature F on the spectrogram of speech segments in all training data, which is a weighted sum of the mixture entropy calculated in Step 5 and the confidence obtained in Step 3: Rank i =(1-λ)nor(conf) i )+λnor(-MIE i (10)

[0021] Step 7: Downsample the audio segments used in training and assign Rank to them. i The spectrograms of n speech segments that exceed a specified threshold are used as new training data X. l+1 That is, selecting spectrograms of speech segments with clear emotions and strong structural stability in the deep feature space as new training data X. l+1 ;

[0022] Step 8: Repeat steps 2 to 7 for a total of L rounds, obtaining the base classifier m in each round. l Add them to a set M as an ensemble classifier;

[0023] Step 9: During the testing process, a complete speech is divided into E speech segments. The output of each speech segment on the base classifier obtained in the l-th round of training is... Where e is the segment number and C is the number of sentiment categories in the dataset, the output of the complete speech in the ensemble classifier M can be defined as: Where e is the segment number, and the output of each speech segment on the base classifier obtained in the l-th round of training is: E represents the number of speech segments divided into the complete speech, and L represents the total number of training rounds. Therefore, the final recognized emotion index R of the complete speech in the speech emotion recognition system based on hybrid entropy downsampling and ensemble classifier described in this patent is... ind The calculation formula is: Where C is the number of sentiment categories in the dataset, then R ind The corresponding emotion category is the final recognition result of the system. Experimental Design

[0024] Experimental Dataset Selection: This invention uses the IEMOCAP speech dataset. It contains 12 hours of audio recordings performed by 10 actors in a conversational format. The recordings are divided into five sections, with two actors per section, for a total of 10 actors. In this invention's experiments, only four common emotions are considered: anger, happiness, neutrality, and sadness. The true emotion label for excited audio recordings in the dataset is also considered to be happiness. The dataset contains a total of 5531 audio recordings, including 1,103 classified as angry, 1,636 as happy, 1,708 as angry, and 1,084 as sad.

[0025] We use two metrics, Weighted Accuracy (WA) and Unweighted Accuracy (UA), to measure the accuracy of the classifier on the test data, defined as follows: N = WA. c r represents the number of sentiment samples of type c. c This represents the number of samples correctly classified in terms of sentiment for class c: The base classifier used in the experiment was a ResNet18 convolutional neural network. Ablation and comparison experiments were conducted based on this classifier. In each experiment, the speech data corresponding to each person in the dataset was used alternately as the test data, and the average of the results was taken. The ablation experiment compared the original base classifier, an ensemble learning classifier using confidence-based downsampling with the original base classifier, an ensemble learning classifier using mixed entropy-based downsampling with the original base classifier, and the method proposed in this invention to reveal the effectiveness of each of the proposed rights. The comparison experiment compared the method with popular speech emotion recognition methods in recent years. In both experiments, the nearest neighbor number k=5, the total number of iterations L=5, the weight coefficient λ=0.6, the downsampling threshold t=0.6, the base classifier training performed 8 gradient descent iterations, the speech frame duration was 16ms, and the overlapping portion duration was 8ms.

[0026] Ablation experiment results: Table 1. Speech emotion recognition accuracy in ablation experiments on the IEMOCAP dataset. Base classifier 54.95 56.42 Base classifier + confidence 57.26 58.49 Base classifier + hybrid entropy 57.76 58.32 This invention 58.72 58.79

[0027] Each row in the table represents a set of ablation experiments, and each column represents the WA and UA of the current ablation experiment, listed as a percentage.

[0028] It can be seen that when using only confidence scores for downsampling ensemble learning, the classifier's WA and UA improved by 2.31% and 2.07%, respectively, indicating that confidence scores can effectively measure the intensity of sentiment in each segment. When using only mixture entropy as the basis for downsampling, the results showed even greater improvement compared to the base classifier, with improvements of 2.81% and 1.90% in WA and UA, respectively. The results showed the greatest improvement when both confidence scores and mixture entropy were used in downsampling. The addition of mixture entropy allowed the calculated ranking value to take into account the characteristics of deep features in the sample space, further improving the accuracy of the results.

[0029] Ablation experiments show that when using only confidence scores for downsampling ensemble learning, the classifier's WA and UA improved by 2.31% and 2.07%, respectively, indicating that confidence scores can effectively measure the intensity of sentiment across different segments. When using only mixture entropy as the basis for downsampling, the results show even greater improvement compared to the base classifier, with improvements of 2.81% and 1.90% in WA and UA, respectively. The greatest improvement is achieved when both confidence scores and mixture entropy are used in downsampling. The addition of mixture entropy allows the calculated ranking value to consider the characteristics of deep features in the sample space, further improving the accuracy of the results.

[0030] Comparative experimental results: Table 2. Accuracy of speech emotion recognition in comparative experiments on the IEMOCAP dataset.

[0031] Each row in the table represents an experiment of a classifier, and each column represents the WA and UA of the current experiment, listed as a percentage.

[0032] Comparative experiments show that our classifier achieves high accuracy on both WA and UA. Classifiers 2 and 4 are both deep learning classifiers, and the comparison results demonstrate that our proposed ensemble learning method effectively improves the accuracy of speech emotion recognition.

Claims

1. A speech emotion recognition system based on hybrid entropy downsampling and an ensemble classifier, characterized in that, Includes the following steps: 1) Divide the dataset into training data and test data. After dividing the speech signal of the training data into segments, extract the spectrogram. Use the spectrogram to train the base classifier and obtain the deep features and confidence of each speech segment. 2) Calculate the mixture entropy of all speech segments and use the weighted sum of the entropy and confidence as the ranking value. The mixture entropy consists of sentiment certainty entropy and structural distribution entropy. Sentiment certainty entropy is used to measure the significance of the sentiment expressed by the speech segment, and structural distribution entropy is used to measure the stability of the distribution structure of the speech segment in the deep feature space. Emotional Certainty Entropy The formula is: ， in, The numbers of the speech segments in the training data. The number of sentiment categories in the dataset. For the set number of nearest neighbors, For the first The basic degree of certainty entropy of a speech segment Indicates calculation The formula for the logarithm with base 0 is as follows: ， Specifically, the entropy of sentiment certainty In the formula, This indicates that the deep features of the training data are comparable to the first... Calculate the Euclidean distance from each speech segment. The formula for determining the number of segments corresponding to the sentiment category label with the most segments in a given audio segment is as follows: ， in, This indicates that in the training data, the first... Calculate the Euclidean distance between the deep features of each speech segment. The emotion category labels in each audio segment are: The number of segments; Structural distribution entropy The formula is: in, The numbers of the speech segments in the training data. For the set number of nearest neighbors, This indicates the first [number]th [item] in the training data. The first audio segment and the first Euclidean distance between deep features of speech segments Indicates calculation Logarithm to base 0; 3) The spectrograms of speech segments with ranking values ​​greater than a set threshold are used to retrain a base classifier, and the ranking values ​​of all speech segments and the base classifier are recalculated. This operation is repeated for a given number of rounds, and the base classifiers trained in each round form an ensemble classifier. 4) After testing the speech segmentation and extracting the spectrogram, input the data into the ensemble classifier to calculate the emotion recognition result of the speech.

2. The speech emotion recognition system based on hybrid entropy downsampling and ensemble classifier according to claim 1, characterized in that, In step 2), the mixture entropy of each speech segment is calculated from the emotion certainty entropy and the structure distribution entropy; The formula for mixed entropy is: ， in, The numbers of the speech segments in the training data. Entropy represents the degree of certainty in sentiment. Represents the structural distribution entropy. It is the Min-Max normalization function. Indicates the first Mixed entropy of a speech segment.

3. The speech emotion recognition system based on hybrid entropy downsampling and ensemble classifier according to claim 2, characterized in that, In step 2), the ranking value of each speech segment is defined as the weighted sum of the mixture entropy and confidence. The formula for the ranking value is: ， in, The numbers of the speech segments in the training data. Indicates the first Confidence level of each speech segment Indicates the first Mixed entropy of a speech segment These are weighting coefficients. It is the Min-Max normalization function. Indicates the first The ranking value of each speech segment is used as the basis for downsampling the speech segment.

4. The speech emotion recognition system based on hybrid entropy downsampling and ensemble classifier according to claim 1, characterized in that, The ensemble classifier obtained in step 3) is ,in Indicates the first The base classifier trained in each round, For the set total number of training rounds, a complete voice message was divided into segments during the testing process. There are 1 audio segments, and each audio segment is in the 1st... The output of the base classifier obtained from the training rounds is ,in Number the segments. Given the number of sentiment categories in the dataset, then the complete speech in the ensemble classifier... The output on can be defined as: ， in, Number the segments, each speech segment is in the [number]th [section]. The output of the base classifier obtained from the training rounds is , The number of speech segments into which this complete speech is divided. Given the set total number of training rounds, the final recognized sentiment index of this complete speech segment... The calculation formula is: ， in, Let be the number of sentiment categories in the dataset. The corresponding emotion category is the final recognition result of the system.

Images