A speech emotion recognition method and system based on double features

Abstract

The application discloses a speech emotion recognition method and system based on double features, and the method comprises the following steps: S1, acquiring a speech signal with the same sampling frequency and time length; S2, extracting wavelet scattering features and MFCC related features of the speech signal; S3, expanding the wavelet scattering features into scale features according to the scale dimension, and calculating the permutation entropy of the scale features as a weight; S4, using the permutation entropy to weight the posterior probability of the emotion category predicted by the scale features, and summing according to the emotion category to obtain the permutation entropy weighted posterior probability; S5, using the MFCC related features to predict the posterior probability of the emotion category; S6, using a bias adjustment rule to fuse the posterior probabilities based on the wavelet scattering features and the MFCC features to obtain an emotion classification result. The application utilizes the complementarity of the wavelet scattering scale expansion features and the MFCC related features, and improves the accuracy and robustness of speech emotion recognition through permutation entropy weighting and bias adjustment.

Description

Technical Field

[0001] This invention belongs to the field of speech emotion recognition technology, specifically relating to a speech emotion recognition method and system based on dual features of wavelet scattering features and Mel frequency cepstral coefficients, using permutation entropy weighting and bias adjustment rules. Background Technology

[0002] Human emotions can be expressed and recognized through various information sources, such as speech, text, facial expressions, and physiological signals. Methods that utilize two or more information sources for emotion recognition are called multimodal emotion recognition. Among these sources, speech is one of the most promising methods for expressing emotions and is also an important foundation for human-computer interaction.

[0003] Mel-frequency cepstral coefficients (MFCCs), as a sound feature representation method simulating the auditory characteristics of the human ear, have long been widely used in speech emotion recognition research. With the increase in data volume and computing power, deep learning methods have also demonstrated powerful performance in speech emotion recognition. While customized neural network architectures are needed for each task, wavelet scattering transform (WST) has emerged as a new feature extraction technique. Its advantages lie in its rigorous theoretical proof and its proven superior recognition rate compared to convolutional neural networks (CNNs) in small sample scenarios. Therefore, it can provide high-quality input for specific classifiers or CNNs in biological signals, especially sound signals. However, explorations based on the differences in wavelet scattering scale features are limited to the feature level; a search revealed no relevant literature at the fractional level. Integrating speech WST features with other time-frequency scale features has the potential to further improve the classification performance of sound signals.

[0004] Based on the current state of existing technologies, this invention aims to improve the accuracy of speech emotion recognition. Based on the differences in wavelet scattering transform scale features and the complementarity of different time-frequency expressions, it proposes a speech emotion recognition method and system based on WST and MFCC permutation entropy weighting and deviation adjustment rules. Summary of the Invention

[0005] The purpose of this invention is to provide a speech emotion recognition method and system based on dual features. This invention can effectively utilize the complementarity of wavelet scattering scale extension features and MFCC correlation features, and improve the accuracy and robustness of speech emotion recognition through permutation entropy weighting and bias adjustment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A speech emotion recognition method based on dual features includes the following steps:

[0008] S1. Preprocessing: Acquire speech signals with the same sampling frequency and duration;

[0009] S2. Extract two types of features from the speech signal in step S1: one is wavelet scattering features, and the other is Mel frequency cepstral coefficient (MFCC) correlation features.

[0010] S3. Extend the wavelet scattering features into scale features according to the scale dimension, and calculate the permutation entropy of each scale feature as the weight;

[0011] S4. Use permutation entropy to weight the posterior probabilities of each sentiment category predicted by each scale feature, and sum them by sentiment category to obtain the permutation entropy weighted posterior probability.

[0012] S5. Use Mel frequency cepstral coefficient (MFCC) related features to predict the posterior probability of each sentiment category;

[0013] S6. Use the bias adjustment rule to fuse the posterior probabilities based on the two types of features to obtain the final sentiment classification result.

[0014] Preferably, in step S1, voice emotion data with different sampling frequencies are unified into samples with a fixed sampling frequency, and a fixed length is truncated. The insufficient part is filled with zeros, and finally normalized to obtain a voice signal with the same sampling frequency and duration.

[0015] Preferably, in step S2, the Mel frequency cepstral coefficient correlation feature is the statistical value of the Mel frequency cepstral coefficient and its first and second order difference features.

[0016] Preferably, in step S2, wavelet scattering features of the preprocessed speech signal are first obtained based on three steps of convolution, nonlinearity, and averaging; then, statistical values ​​are obtained from the extracted MFCC and its first and second order difference features as MFCC-related features.

[0017] Preferably, in step S3, the obtained scattering features are expanded into N features of the same length according to the scale dimension and assigned labels of the same category; the time delay and embedding dimension are set to obtain the reconstructed phase space of the input scale dimension features, and then each embedding vector is arranged in ascending order, and the probability of each arrangement pattern is calculated. Finally, the arrangement entropy is obtained as the weight of the scale feature based on the probability of the arrangement pattern.

[0018] Preferably, in step S4, a support vector machine is used to predict the posterior probabilities of each category for scale dimension features; for all scale features of each predicted data, the corresponding posterior probabilities are weighted by multiplying them with weights, and then all scale-weighted probabilities are summed according to sentiment category, and finally normalized to obtain the permutation entropy-weighted posterior probability.

[0019] Preferably, in step S5, another support vector machine is used to predict the posterior probabilities p of each class of MFCC-related features. m .

[0020] Preferably, in step S6, the posterior probability predicted by MFCC is subtracted from a pre-set bias adjustment coefficient; according to the decision rule, the weighted posterior probability based on WST and the posterior probability based on MFCC bias adjustment are fused to obtain the final predicted probability. The decision rule is as follows:

[0021] Rule 1: When and When that happens, the input is divided into class C1.

[0022] Rule 2: When and At that time, the input is divided into class C2.

[0023] Rule 3: When and At that time, if The input is classified into class C1; otherwise, it is classified into class C2.

[0024] Rule 4: When and At that time, if The input is classified into class C2; otherwise, it is classified into class C1.

[0025] Among them, symbols This indicates classification based on the highest probability value, where C1 and C2 are the predicted categories, and th is the set bias adjustment coefficient.

[0026] The present invention also provides a speech emotion recognition system based on dual features, which is used to perform the above-mentioned speech emotion recognition method and includes the following modules:

[0027] Preprocessing module: used to unify datasets with different sampling frequencies to obtain speech samples of the same duration and sampling frequency;

[0028] Feature extraction module: used to extract wavelet scattering feature matrix from speech samples and expand it into scale features according to the scale dimension; and to obtain MFCC related features;

[0029] The permutation entropy weighting module is used to expand wavelet scattering features into scale features according to the scale dimension, calculate the permutation entropy of the scale features, and perform weighted summation of the posterior probabilities of the sentiment categories predicted by each scale feature based on the permutation entropy. The normalized result is the permutation entropy weighted posterior probability, and the posterior probability of the sentiment category is predicted using Mel frequency cepstral coefficient correlation features.

[0030] Bias adjustment module: Based on the bias adjustment rules, the posterior probabilities of two types of features, namely wavelet scattering features and Mel frequency cepstral coefficients, are adjusted, and the final sentiment classification result is obtained according to the decision rules.

[0031] As a preferred approach, the permutation entropy weighting module uses SVM1 to predict the posterior probability of each sentiment category for each scale feature; and multiplies and sums the posterior probabilities of each scale feature based on the permutation entropy of each scale feature as weights; finally, the weighted result is normalized to obtain the permutation entropy weighted posterior probability.

[0032] The bias adjustment module uses SVM2 to predict the posterior probability of each sentiment category for MFCC-related features; and calculates the difference between the prediction result and the SVM2 prediction result based on the preset bias adjustment coefficient; and obtains the final sentiment classification result according to the decision rule.

[0033] As a preferred solution, the permutation entropy weighting module and the deviation adjustment module based on dual features constitute the permutation entropy weighting and deviation adjustment structure for speech emotion recognition.

[0034] In summary, this invention discloses a dual-feature speech emotion recognition method and system based on wavelet scattering transform and Mel-frequency cepstral coefficients. In this invention, the input is preprocessed and then fed into two parallel model branches. First, a wavelet scattering feature-based model extracts the wavelet scattering features and expands them into scale feature vectors along the scale dimension. It also calculates the permutation entropy of each scale feature and performs a weighted summation of the predicted probabilities based on the scale features. Second, a Mel-frequency cepstral coefficient-based recognition model extracts the Mel-frequency cepstral coefficient-related features and predicts the posterior probability of each emotion category. Finally, a bias adjustment rule is used to fuse the prediction results of the two branches, thereby obtaining the final emotion prediction result.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) This invention utilizes the scale dimension of wavelet scattering transform to expand features without additional computational overhead, thereby increasing the amount of training data for emotional speech and improving the expressive power of features.

[0037] (2) The present invention uses permutation entropy to weight the recognition results of features at different scales, effectively utilizing the different scale features of wavelet scattering transform to represent the differences in speech emotion. At the same time, the model only performs a dot product operation once during weighting, ensuring that the addition of this module will not significantly increase the complexity of the entire system or affect the system's operating efficiency.

[0038] (3) The present invention designs a parallel structure based on different time-frequency scale features and performs fusion at the fractional level through deviation adjustment rules. The structure adopts a simple system composition, makes full use of the complementarity of different time-frequency scale features, and further improves the accuracy and robustness of speech emotion recognition. Attached Figure Description

[0039] Figure 1 This is a flowchart of a preferred embodiment of the speech emotion recognition method of the present invention;

[0040] Figure 2 This is a block diagram of the permutation entropy weighted model of a preferred embodiment of the present invention;

[0041] Figure 3 This is a flowchart of the deviation adjustment rule according to a preferred embodiment of the present invention;

[0042] Figure 4 This is a module architecture diagram of a voice emotion recognition system according to a preferred embodiment of the present invention. Detailed Implementation

[0043] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, a preferred embodiment of the present invention provides a speech emotion recognition method based on permutation entropy weighting and deviation adjustment rules of wavelet scattering transform features and MFCC correlation features. The method includes the following steps:

[0045] S1. Preprocessing: The input speech signal is downsampled to a sampling frequency of 16kHz. Then, the first 3 seconds of each speech signal are extracted as emotional speech samples. For speech signals that are less than 3 seconds long, zero padding is used to fill them. Finally, each speech sample is normalized to obtain speech signals with the same sampling frequency and duration.

[0046] S2. Extract two types of features: wavelet scattering features and Mel frequency cepstral coefficient (MFCC) correlation features; the specific steps are as follows:

[0047] S21. Perform wavelet scattering transform on the preprocessed speech signal. Set the number of wavelets per octave of the first-order filter bank Q1 to 4, the number of wavelets per octave of the second-order filter bank Q2 to 1, and the invariance scale T of the scattering transform to 0.75s. A 200×24 second-order wavelet scattering transform feature matrix is ​​obtained.

[0048] S22. Perform MFCC feature extraction on the preprocessed speech signal. Set a Hamming window with a window length of 30ms for frame division and a frame shift of 10ms. Extract 13 MFCC coefficients and log energy in each frame. Then calculate five types of statistical values, such as minimum, maximum, mean, median and standard deviation, of MFCC and its first and second order difference features to obtain a 210-dimensional MFCC related feature vector.

[0049] S3. Expand the wavelet scattering features by scale and calculate the permutation entropy of the scale features as weights; specifically as follows:

[0050] S31. Expand the wavelet scattering feature matrix into 24 scale feature vectors of length 200 according to the scale dimension, and assign them labels of the same category;

[0051] S32. For each scale feature vector f n By setting the same time delay and embedding dimension, a reconstructed phase space is obtained. Then, each embedding vector is arranged in ascending order, and the probability of each arrangement pattern is calculated. Finally, the arrangement entropy is obtained based on the probability of the arrangement pattern.

[0052]

[0053] Where d is the embedding dimension, i is the permutation pattern index, and π is the permutation pattern index. i f n The pattern of reconstructing phase space, where p(·) represents the probability.

[0054] S33. Since PE∈[0,log2 d! ], the normalized permutation entropy can be obtained:

[0055]

[0056] The calculated PE value is used as the scale weight. To express.

[0057] S4. Use permutation entropy to weight the posterior probabilities of each sentiment category predicted by the scale features, and sum them by sentiment category to obtain the permutation entropy-weighted posterior probabilities; as follows:

[0058] S41. Use the SVM1 classifier to estimate the features f at each scale. n The posterior probability of the sentiment category, expressed as p n =(p n1 ,p n2 ,…,p nC ) represents, where C represents the number of sentiment categories. The scale weights w obtained in step S33 are used to adjust p. n We perform weighting to obtain the scale-weighted prediction probability. Calculated by the following formula:

[0059]

[0060] Where l = 1, 2, ..., C.

[0061] S42. Scale-weighted prediction probability for all scale features Sum the results by sentiment category to obtain the weighted prediction probability p. w =(p w1 ,p w2 ,…,p wC ) Calculated by the following formula:

[0062]

[0063] S43. To facilitate model fusion, p w Normalization yields the normalized probability. Calculated by the following formula:

[0064]

[0065] S5. Using MFCC-related features, SVM2 predicts the posterior probability of each sentiment category, and p... m =(p m1 ,p m2 ,…,p mC )express;

[0066] S6. Use the bias adjustment rule to fuse the probabilities obtained from the two types of features to obtain the final sentiment classification result.

[0067] like Figure 3 As shown, in step S6, the permutation entropy weighted result based on wavelet scattering features is... Use deviation adjustment rules to fuse p m The emotion category of the input speech is obtained according to the following rules.

[0068] Rule 1: When and When that happens, the input is divided into class C1.

[0069] Rule 2: When and At that time, the input is divided into class C2.

[0070] Rule 3: When and At that time, if The input is classified into class C1; otherwise, it is classified into class C2.

[0071] Rule 4: When and At that time, if The input is classified into class C2; otherwise, it is classified into class C1.

[0072] like Figure 4 As shown, a preferred embodiment of the present invention provides a speech emotion recognition system based on dual features, which, based on the speech emotion recognition method of the above embodiment, includes the following modules:

[0073] Preprocessing module: The sampling frequency of the speech signal is reduced to 16kHz by downsampling, and the first 3 seconds of the downsampled signal are uniformly extracted and normalized as the input for feature extraction.

[0074] Feature extraction module: Input the preprocessed signal, extract wavelet scattering transform features and MFCC features, and obtain the scale expansion features of wavelet scattering features and the statistical values ​​of MFCC and its first and second order differences to obtain the two types of features required in the end.

[0075] The permutation entropy weighting module sets the time delay and embedding dimension, calculates the permutation entropy of the scale features, uses support vector machines to predict the posterior probability of each category, and uses the permutation entropy of the corresponding scale features as weights to sum the posterior probabilities of all scale features in a weighted manner, normalizes them to obtain the permutation entropy weighted prediction probability, and uses Mel frequency cepstral coefficient correlation features to predict the posterior probability of the sentiment category.

[0076] Bias adjustment module: Based on the weighted prediction results of the scale features of the test speech, a bias adjustment coefficient is set, and the posterior probabilities of each category predicted based on the MFCC-related features of the speech are fused to obtain the final predicted sentiment category.

[0077] The permutation entropy weighting module uses SVM1 to predict the posterior probability of sentiment categories for scale features; it then multiplies and sums the posterior probabilities of scale features using the permutation entropy as weights; finally, it normalizes the weighted result to obtain the permutation entropy weighted posterior probability. The bias adjustment module uses SVM2 to predict the posterior probability of each sentiment category for Mel frequency cepstral coefficient-related features, and calculates the difference between the prediction result and the predefined bias adjustment coefficient; finally, it obtains the final sentiment classification result according to the decision rule. The permutation entropy weighting module and the bias adjustment module constitute the permutation entropy weighting and bias adjustment structure.

[0078] In summary, this invention proposes a speech emotion recognition method and system based on permutation entropy weighting and bias adjustment rules, using two types of features: scale-expanded features based on WST and correlation features based on MFCC. Two support vector machines are used to predict the posterior probabilities of various emotion categories for each type of feature. Then, the predicted probabilities of wavelet scattering scale features are weighted according to permutation entropy. Finally, a bias adjustment coefficient is used to correct the predicted posterior probabilities based on the two types of features, thereby reducing the bias between the two feature extraction methods and improving the accuracy of emotion recognition.

[0079] The foregoing has provided a detailed description of the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the embodiments and descriptions above, but includes various possible variations and modifications within the spirit and scope of the invention. These variations and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on dual features, characterized in that, Includes the following steps: S1. Acquire speech signals with the same sampling frequency and duration; S2. Extract wavelet scattering features and Mel frequency cepstral coefficient correlation features from the speech signal in step S1; S3. Extend the wavelet scattering features into scale features according to the scale dimension, and calculate the permutation entropy of the scale features as weights; S4. Use permutation entropy to weight the posterior probability of the sentiment category predicted by the scale feature, and sum them by sentiment category to obtain the permutation entropy weighted posterior probability. S5. Use Mel frequency cepstral coefficient correlation features to predict the posterior probability of sentiment category; S6. Use the bias adjustment rule to fuse the posterior probabilities based on wavelet scattering features and Mel frequency cepstral coefficient features to obtain the sentiment classification results; In step S6, the posterior probability predicted by the Mel frequency cepstral coefficients is subtracted from a pre-set deviation adjustment coefficient. Based on the decision rule, the weighted posterior probability based on wavelet scattering transform and the posterior probability based on the Mel frequency cepstral coefficient deviation adjustment are fused to obtain the final predicted probability. The decision rule is as follows: Rule 1: When ,and At that time, the input is divided into kind; Rule 2: When ,and At that time, the input is divided into kind; Rule 3: When ,and At that time, if Divide the input into Class, otherwise, divide into kind; Rule 4: When ,and At that time, if Divide the input into Class, otherwise, divide into kind; in, The posterior probability is the weighted posterior probability of the permutation entropy. For each category, This indicates classification based on the highest probability value. and To predict the category, This is the set deviation adjustment factor.

2. The speech emotion recognition method based on dual features according to claim 1, characterized in that, In step S2, the Mel frequency cepstral coefficient correlation feature is the statistical value of the Mel frequency cepstral coefficient and its first-order difference feature and second-order difference feature.

3. The speech emotion recognition method based on dual features according to claim 1, characterized in that, In step S3, the obtained scattering features are expanded into N features of the same length according to the scale dimension and assigned labels of the same category; the time delay and embedding dimension are set to obtain the reconstructed phase space of the input scale dimension features, then each embedding vector is arranged in ascending order and the probability of each arrangement pattern is calculated. Finally, the arrangement entropy is obtained as the weight of the scale feature based on the probability of the arrangement pattern.

4. The speech emotion recognition method based on dual features according to claim 1, characterized in that, In step S4, the permutation entropy weighted posterior probability is obtained by weighting the posterior probability predicted by the scale feature using the permutation entropy of the scale feature as the weight, summing it according to the sentiment category, and finally normalizing it.

5. The speech emotion recognition method based on dual features according to claim 1, characterized in that, In step S5, a support vector machine is used to predict the class posterior probability of the Mel frequency cepstral coefficient related features.

6. A speech emotion recognition system based on dual features, characterized in that, The system is used to perform the speech emotion recognition method as described in any one of claims 1-5, and the system includes the following modules: Preprocessing module: used to acquire speech samples with the same sampling frequency and duration; Feature extraction module: used to extract wavelet scattering feature matrix from speech samples and expand it into scale features; And obtain the correlation characteristics of Mel frequency cepstral coefficients; Permutation Entropy Weighted Module: This module expands wavelet scattering features into scale features along the scale dimension, calculates the permutation entropy of the scale features, and performs a weighted summation of the posterior probabilities of the sentiment categories predicted by the scale features based on the permutation entropy to obtain the permutation entropy weighted posterior probability. The posterior probability of the sentiment category is then predicted using Mel frequency cepstral coefficient correlation features. Bias adjustment module: Based on the bias adjustment rules, the posterior probability based on wavelet scattering features and Mel frequency cepstral coefficient features is adjusted, and the final sentiment classification result is obtained according to the decision rules.

7. A speech emotion recognition system based on dual features according to claim 6, characterized in that, The permutation entropy weighting module uses SVM1 to predict the posterior probability of the sentiment category of the scale feature; and multiplies and adds the posterior probabilities of the scale feature according to the permutation entropy of the scale feature as weight; finally, the weighted result is normalized to obtain the permutation entropy weighted posterior probability.

8. A speech emotion recognition system based on dual features according to claim 6 or 7, characterized in that, The bias adjustment module uses SVM2 to predict the posterior probability of each sentiment category of the Mel frequency cepstral coefficient related features, and calculates the difference between the prediction result and the SVM2 prediction result according to the preset bias adjustment coefficient; the final sentiment classification result is obtained according to the decision rule.

Images