Speech emotion recognition method based on M-LFCC feature fusion
By fusing MFCC and LFCC features and utilizing LSTM networks and an improved ECAPA-TDNN model, the problems of high-frequency information loss and excessive model complexity in speech emotion recognition are solved, achieving more efficient speech emotion recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-03
AI Technical Summary
In existing speech emotion recognition methods, high-frequency information is lost in single MFCC features, and the model complexity is too high, making it difficult to balance recognition accuracy and efficiency. The feature fusion mechanism is also imperfect.
We employ MFCC and LFCC feature fusion, combine it with LSTM network for temporal feature fusion, and improve the ECAPA-TDNN model by replacing the compressed excitation module with an efficient channel attention module to construct an improved ECAPA-TDNN model for sentiment classification.
It enhances the ability to capture high-frequency emotional features, reduces model complexity and computational cost, and improves recognition accuracy and efficiency.
Smart Images
Figure CN121789722A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary fields of speech signal processing, pattern recognition, and artificial intelligence, specifically to a speech emotion recognition method, system, electronic device, and computer-readable storage medium based on the fusion of Mel-frequency cepstral coefficients and linear-frequency cepstral coefficients. This invention is particularly suitable for practical applications requiring high-precision and high-efficiency emotion recognition, such as human-computer interaction systems, intelligent customer service, mental health monitoring, and emotion perception in autonomous driving cabins. Background Technology
[0002] Speech, as the most natural and direct form of human communication, not only conveys semantic information but also carries rich emotional content. Speech emotion recognition aims to automatically identify the speaker's emotional state through computational models and is a core research direction in the fields of affective computing and human-computer interaction. Accurate speech emotion recognition can greatly improve the naturalness of interaction and user experience in intelligent systems, and has broad application prospects in fields such as telemedicine, intelligent education, in-vehicle systems, and entertainment games.
[0003] Traditional speech emotion recognition methods primarily rely on manually designed acoustic features and shallow machine learning models. Commonly used artificial features include prosody-based features, timbre-based features, and spectral-based features. Among these, MFCC (Multi-Functional Microsystem Convergence) has achieved widespread success in speech and emotion recognition tasks due to its ability to simulate the nonlinear perceptual characteristics of the human auditory system. These traditional features are often combined with classifiers such as Gaussian mixture models, Hidden Markov Models (HMMs), and Support Vector Machines (SVMs). However, these methods have significant limitations: First, the design of artificial features is highly dependent on domain knowledge, and the feature engineering process is cumbersome; second, these features often only capture the shallow, local characteristics of speech signals, making it difficult to comprehensively represent the complex and dynamic acoustic patterns corresponding to emotional states; finally, shallow models have limited learning capabilities and cannot fully exploit the nonlinear relationships in high-dimensional features, leading to performance bottlenecks.
[0004] With the rapid development of deep learning technology, speech emotion recognition methods based on deep neural networks have gradually become mainstream. Convolutional neural networks can automatically learn local correlations and frequency domain patterns in speech spectrograms; recurrent neural networks and their variants excel at modeling temporal dependencies in speech signals; and the introduction of attention mechanisms allows models to focus on speech segments that are more critical to emotion classification. These deep learning methods significantly reduce the reliance on manual feature engineering and achieve more powerful feature representation capabilities through end-to-end learning, resulting in superior performance compared to traditional methods on multiple public datasets.
[0005] Despite this, current deep learning-based speech emotion recognition methods still face several challenges: 1. Feature level: Most studies still use MFCC as the primary or sole acoustic feature input. While MFCC aligns with human hearing characteristics, its Mel frequency scale has low resolution in the high-frequency region. This is because the Mel scale is denser in the low-frequency band and sparser in the high-frequency band, causing spectral details in the high-frequency band to be smoothed or lost during filtering. Many emotional states' acoustic representations contain rich high-frequency components. Therefore, relying solely on MFCC features may not be sufficient to capture these key high-frequency emotional cues, limiting the model's ability to distinguish specific emotions. 2. Model level: Many high-performance models have complex structures and a large number of parameters. For example, the ECAPA-TDNN model performs well in speaker recognition tasks, significantly improving feature representation capabilities by introducing a compressed excitation attention module and a Res2Net multi-scale structure. However, the SE module requires dimensionality reduction and expansion operations in fully connected layers, which not only increases the number of model parameters and computational burden but may also lead to information loss due to dimensionality reduction. Directly applying it to speech emotion recognition, especially on resource-constrained edge devices, may face deployment difficulties. 3. Fusion Strategies: Some studies have attempted to fuse multiple features or models, but the fusion methods are often relatively simple and fail to fully consider the complementarity of different types of features in the time-frequency domain and the long-term contextual dependencies of speech signals. How to design an effective fusion mechanism that deeply integrates the perceptual advantages of MFCC with the frequency domain accuracy advantages of LFCC, and utilizes the temporal modeling capabilities of sequence models, is a problem that needs further exploration.
[0006] In summary, existing technologies suffer from shortcomings such as insufficient high-frequency information capture, excessive model complexity, and imperfect feature fusion mechanisms, making it difficult to balance accuracy and efficiency in speech emotion recognition systems. Therefore, there is an urgent need for an innovative solution that can effectively integrate multi-dimensional complementary features, optimize model efficiency, and simultaneously enhance the ability to perceive high-frequency information. Summary of the Invention
[0007] In view of this, the purpose of this invention is to overcome the shortcomings of the prior art and provide a speech emotion recognition method, system, device, and storage medium based on the fusion of MFCC and LFCC features, so as to solve the problem of difficulty in balancing recognition accuracy and efficiency caused by the loss of high-frequency information in a single MFCC feature and excessive model complexity. To achieve the above objective, this invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a speech emotion recognition method based on the fusion of MFCC and LFCC features, comprising the following steps:
[0009] Step S1: Preprocess the input speech signal and extract the Mel frequency cepstral coefficient features and the linear frequency cepstral coefficient features respectively.
[0010] Step S2: Input the extracted MFCC features and LFCC features into the Long Short-Term Memory network to fuse temporal features and output the fused feature vector.
[0011] Step S3: Input the fused feature vector into the improved ECAPA-TDNN model for sentiment classification, wherein the improved ECAPA-TDNN model uses an efficient channel attention module to replace the compressed excitation module in the original ECAPA-TDNN model.
[0012] Step S4: Output the emotion recognition result of the speech signal.
[0013] Furthermore, in step S1, a linear filter bank is used to extract LFCC features to enhance the resolution of the speech signal in the high-frequency region; a Mel filter bank is used to extract MFCC features to simulate the characteristics of human hearing. By fusing the two, low-frequency information that conforms to human hearing is preserved, while high-frequency details are supplemented.
[0014] Furthermore, in step S2, the LSTM network is used to capture the temporal dependencies of the speech signal and to concatenate and fuse the MFCC features and LFCC features in the time dimension to form a fused feature vector containing rich contextual information.
[0015] Furthermore, the improved ECAPA-TDNN model in step S3 sequentially includes: at least one time-delay neural network layer for feature mapping and context information extraction; at least one ECA-Res2Block module for cross-channel attention interaction and multi-scale feature enhancement; an attention statistical pooling layer for extracting global statistical features; and a fully connected layer for mapping features to the sentiment category space and outputting a probability distribution.
[0016] Furthermore, the ECA-Res2Block module comprises a TDNN sublayer, an ECA attention sublayer, and a Res2Net structure. The ECA module achieves cross-channel interaction through one-dimensional convolution, avoiding the side effects of dimensionality reduction in the original SE module, and capturing the importance between channels with lower complexity.
[0017] Secondly, the present invention provides a voice emotion recognition system, comprising:
[0018] The feature extraction module is used to extract MFCC and LFCC features from the input speech signal.
[0019] The feature fusion module includes an LSTM network, which is used to perform temporal fusion of the MFCC features and LFCC features to output a fused feature vector.
[0020] The sentiment classification module includes an improved ECAPA-TDNN model for sentiment classification of the fused feature vectors.
[0021] The output module is used to output the emotion recognition results.
[0022] Furthermore, the improved ECAPA-TDNN model in the sentiment classification module replaces the SE module with the ECA module.
[0023] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect above.
[0024] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.
[0025] The beneficial effects of this invention are as follows:
[0026] By integrating MFCC and LFCC features, the advantages of MFCC's low-frequency response that aligns with human hearing and LFCC's high-frequency resolution are combined to provide a more comprehensive and refined speech spectrum representation, especially enhancing the ability to capture high-frequency emotional features.
[0027] By introducing an LSTM network to perform temporal fusion of dual features, the long-term dependencies in speech signals are effectively modeled, improving the model's understanding of the continuous changes in emotion.
[0028] An improved ECAPA-TDNN model is adopted, replacing the SE module with the ECA module. While maintaining or even enhancing the channel attention effect, the model's parameter complexity and computational cost are significantly reduced, making the model easier to train and deploy.
[0029] Experiments have verified that the method of this invention achieves excellent recognition accuracy on public datasets and exhibits stable recognition performance for various emotion categories, demonstrating high practical value in speech emotion recognition tasks. Attached Figure Description
[0030] Figure 1 A schematic diagram of the overall architecture of the improved ECAPA-TDNN-LSTM model provided by this invention.
[0031] Figure 2 This is a schematic diagram of the LSTM network unit structure in an embodiment of the present invention.
[0032] Figure 3 This is a schematic diagram of the high-efficiency channel attention module structure in an embodiment of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0034] This embodiment provides a speech emotion recognition method based on the fusion of MFCC and LFCC features. The specific steps are as follows:
[0035] Speech signal preprocessing and feature extraction:
[0036] The audio signal is acquired and preprocessed, including pre-emphasis, framing, and windowing.
[0037] MFCC Feature Extraction: The spectrum of each preprocessed speech frame is obtained by performing a Fast Fourier Transform (FFT). This spectrum is then filtered using a Mel filter bank, and after taking the logarithmic energy, a Discrete Cosine Transform (DCT) is performed to extract the MFCC coefficients and their first and second-order differences, forming the MFCC feature vector. The Mel filter bank simulates the nonlinearity of human hearing and is sensitive to low frequencies. The conversion relationship between the Mel frequency M(f) and the actual frequency f is as follows:
[0038] LFCC Feature Extraction: Similar to the MFCC process, but using a linearly spaced filter bank instead of a Mel filter bank. The linear filter bank provides higher frequency resolution in the high-frequency region, enabling more precise capture of detailed changes in the high-frequency components of speech.
[0039] The extracted MFCC feature sequences and LFCC feature sequences are used as subsequent inputs.
[0040] LSTM-based temporal feature fusion:
[0041] The MFCC and LFCC feature sequences are respectively input into an LSTM network. The LSTM unit learns and memorizes the long-term temporal patterns of the speech signal through its gating mechanism (input gate, forget gate, output gate).
[0042] Two LSTM networks output high-level feature sequences after time series modeling of MFCC and LFCC, respectively. The computation process of the LSTM unit at each time step t is as follows:
[0043]
[0044] Where, x t Let h be the input feature vector at time t. t For the hidden state, σ represents the sigmoid activation function, ⊙ represents element-wise multiplication (Hadamard product), and W and b are the weights and biases to be learned.
[0045] These two high-level feature sequences are concatenated in the time dimension to form a fusion feature vector that integrates the auditory characteristics of MFCC and the high-frequency details of LFCC and is rich in contextual information.
[0046] Improved ECAPA-TDNN model for sentiment classification:
[0047] An improved ECAPA-TDNN classification model is constructed. The main structure of this model includes: a. A front-end TDNN layer: receiving the fused feature vector and performing preliminary feature mapping and contextual information aggregation. b. Multiple ECA-Res2Block modules: this is the core of the improvement. Each module first performs feature transformation through a TDNN sub-layer; then, an ECA module is introduced, which learns channel attention weights by performing one-dimensional convolution on channel features, efficiently enhancing the feature response of important channels without dimensionality reduction; finally, combined with a Res2Net structure, multi-scale feature extraction is implemented within the module, enhancing the model's ability to perceive sentiment features at different scales.
[0048] c. Attention Statistical Pooling Layer: Weighted pooling is performed on the frame-level features after processing by multiple modules, and their weighted mean and standard deviation are calculated and aggregated into a fixed-dimensional global discourse-level feature vector.
[0049] d. Fully connected classification layer: Maps the global feature vector to the dimension of the number of target sentiment categories, and outputs the probability distribution of each category through the Softmax function.
[0050]
[0051] Among them, z k The output of the fully connected layer corresponds to the k-th type of emotion.
[0052] The model is trained end-to-end using training strategies such as additive angular interval loss function to optimize parameters.
[0053] Emotion recognition results output
[0054]
[0055] Where N is the batch size, K is the number of sentiment categories, s is the scaling factor, m is the angular interval, and θ is the angle. j,i This represents the angle between the feature vector of the i-th sample and the weight vector of the j-th class.
[0056] Based on the probability distribution output by the model, the sentiment category with the highest probability is selected as the final recognition result.
[0057] Experiments show that the method in this embodiment significantly improves recognition accuracy on publicly available speech emotion datasets such as RAVDESS, compared to baseline methods using only MFCC features or a single model, and also has higher model efficiency. The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A speech emotion recognition method based on MFCC and LFCC feature fusion, characterized in that, Includes the following steps: Step S1: Preprocess the input speech signal and extract Mel-frequency cepstral coefficient (MFCC) features and linear-frequency cepstral coefficient (LFCC) features respectively; Step S2: Input the extracted MFCC and LFCC features into a Long Short-Term Memory (LSTM) network to fuse temporal features and output a fused feature vector; Step S3: Input the fused feature vector into the improved ECAPA-TDNN model for sentiment classification, wherein the improved ECAPA-TDNN model uses the efficient channel attention (ECA) module to replace the compressed excitation (SE) module in the original ECAPA-TDNN model; Step S4: Output the emotion recognition result of the speech signal.
2. The method according to claim 1, characterized in that, In step S1, a linear filter bank is used to extract LFCC features to enhance the resolution of the speech signal in the high-frequency region; a Mel filter bank is used to extract MFCC features to simulate the characteristics of human hearing.
3. The method according to claim 1, characterized in that, In step S2, the LSTM network is used to capture the temporal dependencies of the speech signal and to concatenate and fuse the MFCC features and LFCC features in the time dimension.
4. The method according to claim 1, characterized in that, The improved ECAPA-TDNN model in step S3 includes, in sequence: at least one time-delayed neural network (TDNN) layer for feature mapping and context information extraction; at least one ECA-Res2Block module for cross-channel attention interaction and feature enhancement; an attention statistical pooling layer for extracting global statistical features; and a fully connected layer for mapping features to the sentiment category space and outputting a probability distribution.
5. The method according to claim 4, characterized in that, The ECA-Res2Block module includes a TDNN sublayer, an ECA attention sublayer, and a Res2Net structure.
6. A voice emotion recognition system, characterized in that, include: The feature extraction module is used to extract MFCC and LFCC features from the input speech signal; The feature fusion module includes an LSTM network, which is used to temporally fuse the MFCC features and LFCC features to output a fused feature vector. The emotion classification module includes an improved ECAPA-TDNN model for emotion classification of the fused feature vector; the output module is used to output the emotion recognition results.
7. The system according to claim 6, characterized in that, Improvements in the sentiment classification module The ECAPA-TDNN model replaces the SE module with the ECA module.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.