Multi-feature fusion speech emotion recognition method and system based on deep learning
By adopting a multi-feature fusion method based on deep learning in speech emotion recognition technology, combining attention mechanism and cross-receptive field convolution network, the problem of singleness and limitations of feature extraction in traditional methods is solved, and higher pronunciation is achieved Emotional recognition accuracy and generalization performance.
Patent Information
- Application Number
- CN202411626749.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-25
AI Technical Summary
Existing speech emotion recognition technology is difficult to extract and integrate emotional features efficiently and accurately in complex audio data environments. Traditional methods are limited by the singularity and limitations of feature extraction, making it difficult to fully capture emotional details in speech.
Using a multi-feature fusion method based on deep learning, the original audio is preprocessed, the Mel frequency cepspectral coefficients, spectral diagrams and waveform features are extracted, and the models such as Wav2Vec2.0, AlexNet and cross-receptive field convolution network CRF are used for Feature extraction. Combined with the feature fusion strategy of attention mechanisms, multiple features are integrated to enhance the accuracy of emotion recognition.
It realizes all-round analysis of emotional signals, enhances feature extraction capabilities, improves the accuracy and generalization performance of emotional recognition, effectively suppresses noise, and highlights key emotional clues.
Smart Images
Figure CN119517097A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to a multi-feature fusion speech emotion recognition method and system based on deep learning. Background Art
[0002] As an important technology in the field of human-computer interaction, speech emotion recognition is becoming increasingly important. When building an intelligent and highly empathetic human-computer interaction system, speech emotion recognition can give machines the ability to understand and perceive human emotions, thereby providing more intimate and personalized services. However, in a complex audio data environment, how to efficiently and accurately extract and integrate emotional features from a large amount of information has always been a major challenge facing this field.
[0003] Traditional speech emotion recognition methods mainly rely on manually designed features, such as Mel-Frequency Cepstral Coefficients (MFCCs) and spectrograms. Although these features can reflect the emotional information in speech to a certain extent, they are often limited by the singleness and limitations of feature extraction and are difficult to fully capture the emotional details in speech. In addition, with the increasing richness and diversity of audio data, traditional methods are unable to cope with large-scale and complex audio data.
[0004] In order to solve the above problems, deep learning technology has been widely used in the field of speech emotion recognition in recent years. Deep learning models have powerful feature extraction and pattern recognition capabilities, and can automatically learn and extract high-level emotional features from raw audio data. However, a single deep learning model can often only capture some features in the audio data, and it is difficult to fully reflect the emotional information in the speech.
[0005] Therefore, how to integrate multiple audio features and make full use of the advantages of deep learning technology has become the key to improving the performance of speech emotion recognition. Summary of the invention
[0006] In order to solve the problems existing in the background technology and improve the accuracy of emotion recognition, one aspect of the present invention provides a multi-feature fusion speech emotion recognition method based on deep learning, comprising:
[0007] S1: preprocess the original audio to extract the Mel frequency cepstrum coefficient and spectrum of the original audio;
[0008] S2: Use the Wav2Vec2.0 model to extract features from the original audio waveform to obtain the waveform features of the original audio;
[0009] S3: Use the AlexNet network to extract features from the spectrum of the original audio to obtain the spectrum features of the original audio;
[0010] S4: Use the cross-receptive field convolutional network CRF to extract the Mel frequency cepstral coefficients of the original audio to obtain the MFCC enhanced features of the original audio;
[0011] S5: The feature fusion strategy based on the attention mechanism fuses the waveform features, spectrum features and MFCC enhanced features of the original audio to obtain the fused features;
[0012] S6: Input the obtained fusion features into the classifier to identify the emotion category of the original audio.
[0013] Preferably, the cross-receptive field convolutional network CRF includes: three feature extraction branches of different scales and N cascaded feature fusion modules; wherein the three feature extraction branches all include cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP; the convolutions of the three feature extraction branches are convolutions of three different scales respectively; the feature fusion module includes: cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP.
[0014] Preferably, the convolutions of the three feature extraction branches are respectively 3×3 convolution, 9×1 convolution kernel and 1×11 convolution.
[0015] Preferably, the use of a cross-receptive field convolutional network CRF to extract features from the Mel-frequency cepstral coefficients of the original audio includes: inputting the Mel-frequency cepstral coefficients of the original audio into three feature extraction branches of different scales for feature extraction, respectively, to obtain feature maps of three different scales; splicing the feature maps of the three different scales, and inputting the spliced feature maps into N cascaded feature fusion modules for feature fusion to obtain MFCC enhanced features of the original audio.
[0016] Preferably, the feature fusion strategy of the attention mechanism performs feature fusion on the waveform features, spectrum features and MFCC enhancement features of the original audio, including:
[0017] S51: The waveform features, spectrum features and MFCC enhancement features of the original audio are preprocessed by a cascaded Dropout layer, a 1D Conv layer and a RELU activation function to obtain corresponding intermediate waveform features, intermediate spectrum features and intermediate MFCC enhancement features;
[0018] S52: Concatenate the intermediate spectrum feature and the intermediate MFCC enhancement feature to obtain a first intermediate feature;
[0019] S53: performing an attention mechanism on the waveform feature of the original audio according to the first intermediate feature to obtain a second intermediate feature;
[0020] S54: Concatenate the second intermediate feature, the spectrum feature, and the MFCC enhanced feature to obtain a fused feature.
[0021] Another aspect of the present invention provides a multi-feature fusion speech emotion recognition system based on deep learning, the system comprising a memory and a processor; the memory is used to store applications; the processor is used to run the applications and execute the multi-feature fusion speech emotion recognition method based on deep learning.
[0022] On one hand, the present invention provides a computer storage medium, on which a remote monitoring program is stored. When the remote monitoring program is executed by a processor, the multi-feature fusion speech emotion recognition method based on deep learning is implemented.
[0023] The present invention has at least the following beneficial effects
[0024] The present invention not only integrates traditional features such as classic Mel-frequency cepstral coefficients and spectrograms through a composite feature parallel processing strategy, but also introduces the deep features of the original audio waveform, thus achieving a comprehensive analysis of emotional signals. At the same time, the cross-receptive field convolutional network designed by the present invention can capture the local details and global context of emotional expression across different scales, further enhancing the feature extraction capability. In addition, the feature fusion strategy based on the attention mechanism enables the present method to dynamically adjust the weights according to the contribution of each feature to the emotion classification, effectively suppress noise, and highlight key emotional clues, thereby improving the generalization performance and recognition accuracy of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the framework of the method flow of the present invention. DETAILED DESCRIPTION
[0026] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0027] See also Figure 1 One aspect of the present invention provides a multi-feature fusion speech emotion recognition method based on deep learning, comprising:
[0028] S1: preprocess the original audio to extract the Mel frequency cepstrum coefficient and spectrum of the original audio;
[0029] In this embodiment, step S1 includes:
[0030] Extract the Mel-frequency cepstral coefficients of the original audio: Use an audio processing library (such as librosa, scipy.io.wavfile, or pydub in Python) to load the audio file and obtain the audio data (usually a time series) and the sampling rate. Normalize the audio data to ensure that the amplitude is within a reasonable range, which helps with subsequent feature extraction. If there is background noise in the audio, it can be denoised using filters (such as bandpass filters, high-pass filters, or low-pass filters) or preprocessing algorithms (such as pre-emphasis, spectral subtraction). Audio signals are continuous, but feature extraction usually needs to be performed in short time periods (frames). Therefore, it is necessary to divide the audio signal into multiple overlapping frames and apply a window function (such as a Hamming window or Hanning window) to each frame to reduce spectral leakage. Apply STFT (Short Time Fourier Transform) to each frame to obtain a spectrogram (a complex matrix) in which each element represents the spectral amplitude and phase of the corresponding frequency and time. Square the amplitude of the spectrogram to obtain the power spectrum (or the square of the amplitude spectrum). The power spectrum is passed through a set of Mel-scale triangular filters, and the logarithm of the output of each filter is taken to obtain the logarithmic energy; the logarithmic energy is DCTed to obtain MFCCs. Usually, only the first few coefficients (such as 13) of the DCT are taken, because these coefficients contain most of the information of the audio signal.
[0031] Extract the spectrogram of the original audio: Same as the first step of extracting MFCCs, apply STFT to each frame to get the spectrogram. Take the magnitude of the STFT result to get the magnitude spectrum. For visualization purposes, the magnitude spectrum is usually converted to decibel (dB) scale. This can be done by taking the logarithm of the magnitude spectrum and multiplying it by a constant. Plot the spectrogram using a visualization library such as matplotlib.
[0032] S2: Use the Wav2Vec2.0 model to extract features from the original audio waveform to obtain the waveform features of the original audio;
[0033] S3: Use the AlexNet network to extract features from the spectrum of the original audio to obtain the spectrum features of the original audio;
[0034] S4: Use the cross-receptive field convolutional network CRF to extract the Mel frequency cepstral coefficients of the original audio to obtain the MFCC enhanced features of the original audio;
[0035] S5: The feature fusion strategy based on the attention mechanism fuses the waveform features, spectrum features and MFCC enhanced features of the original audio to obtain the fused features;
[0036] S6: Input the obtained fusion features into the classifier to identify the emotion category of the original audio.
[0037] In this embodiment, a variety of feature extraction methods (such as MFCC, spectrogram, Wav2Vec2.0, AlexNet, etc.) are used to extract richer and more accurate feature information from the original audio, providing strong support for subsequent classification tasks. By introducing advanced technologies such as cross-receptive field convolutional networks (CRF) and attention mechanisms, the generalization ability of the model can be further enhanced, enabling it to perform well on different audio data sets. By effectively fusing multiple features and using advanced classifiers to identify emotion categories, the accuracy of emotion recognition can be significantly improved, providing more reliable technical support for fields such as audio emotion analysis.
[0038] Preferably, the cross-receptive field convolutional network CRF includes: three feature extraction branches of different scales and N cascaded feature fusion modules; wherein the three feature extraction branches all include cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP; the convolutions of the three feature extraction branches are convolutions of three different scales respectively; the feature fusion module includes: cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP.
[0039] Preferably, the convolutions of the three feature extraction branches are respectively 3×3 convolution, 9×1 convolution kernel and 1×11 convolution.
[0040] Preferably, the use of a cross-receptive field convolutional network CRF to extract features from the Mel-frequency cepstral coefficients of the original audio includes: inputting the Mel-frequency cepstral coefficients of the original audio into three feature extraction branches of different scales for feature extraction, respectively, to obtain feature maps of three different scales; splicing the feature maps of the three different scales, and inputting the spliced feature maps into N cascaded feature fusion modules for feature fusion to obtain MFCC enhanced features of the original audio.
[0041] In this embodiment, by adopting convolution kernels of different scales, the CRF network can capture feature information of different scales in the audio signal and improve the accuracy of feature extraction; by fusing feature information from different feature extraction branches, the CRF network can generate a more comprehensive and accurate audio feature representation, thereby enhancing the generalization ability of the model; when the features extracted by the CRF network are used for subsequent audio classification, emotion recognition and other tasks, the performance of the task can be significantly improved.
[0042] Preferably, the feature fusion strategy of the attention mechanism performs feature fusion on the waveform features, spectrum features and MFCC enhancement features of the original audio, including:
[0043] S51: The waveform features, spectrum features and MFCC enhancement features of the original audio are preprocessed by a cascaded Dropout layer, a 1D Conv layer and a RELU activation function to obtain corresponding intermediate waveform features, intermediate spectrum features and intermediate MFCC enhancement features;
[0044] S52: Concatenate the intermediate spectrum feature and the intermediate MFCC enhancement feature to obtain a first intermediate feature;
[0045] S53: performing an attention mechanism on the waveform feature of the original audio according to the first intermediate feature to obtain a second intermediate feature;
[0046] S54: Concatenate the second intermediate feature, the spectrum feature, and the MFCC enhanced feature to obtain a fused feature.
[0047] In this embodiment, through the feature preprocessing process in the step, the waveform features, spectrum features and MFCC enhanced features of the original audio are effectively extracted and denoised through the cascaded Dropout layer, 1DConv layer and RELU activation function. The Dropout layer helps prevent the model from overfitting and improves the generalization ability of the model; the 1D Conv layer can further extract key information from the features; the RELU activation function can introduce nonlinear factors and enhance the expression ability of the model.
[0048] In step S52, the intermediate spectrum feature and the intermediate MFCC enhancement feature are spliced to obtain a first intermediate feature containing two types of feature information. This splicing method realizes the complementarity of features, making the fused features more comprehensive and accurate.
[0049] In step S53, the waveform feature of the original audio is processed by the attention mechanism according to the first intermediate feature to obtain the second intermediate feature. The attention mechanism can focus on the more important parts of the input data and ignore unimportant information, thereby enhancing the expressiveness and robustness of the waveform feature.
[0050] This feature fusion strategy extracts a variety of key information from the audio signal by comprehensively utilizing waveform features, spectrum features, and MFCC enhancement features, providing a richer and more accurate feature representation for subsequent emotion recognition tasks. Therefore, when the fused features are input into the classifier for emotion recognition, the accuracy and robustness of emotion recognition can be significantly improved.
[0051] Another aspect of the present invention provides a multi-feature fusion speech emotion recognition system based on deep learning, the system comprising a memory and a processor; the memory is used to store applications; the processor is used to run the applications and execute the multi-feature fusion speech emotion recognition method based on deep learning.
[0052] On one hand, the present invention provides a computer storage medium, on which a remote monitoring program is stored. When the remote monitoring program is executed by a processor, the multi-feature fusion speech emotion recognition method based on deep learning is implemented.
[0053] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0054] In summary, the present invention not only integrates traditional features such as classic Mel-frequency cepstral coefficients and spectrograms through a composite feature parallel processing strategy, but also introduces the deep features of the original audio waveform, thereby achieving a comprehensive analysis of emotional signals. At the same time, the cross-receptive field convolutional network designed by the present invention can capture local details and global context of emotional expression across different scales, further enhancing the feature extraction capability. In addition, the feature fusion strategy based on the attention mechanism enables the method to dynamically adjust the weights according to the contribution of each feature to the emotion classification, effectively suppress noise, and highlight key emotional clues, thereby improving the generalization performance and recognition accuracy of the method.
[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A multi-feature fusion speech emotion recognition method based on deep learning, characterized in that: include: S1: pre-process the original audio to extract the Mel frequency cepstrum coefficient and spectrum of the original audio; S2: Use the Wav2Vec2.0 model to extract features from the original audio waveform to obtain the waveform features of the original audio; S3: Use the AlexNet network to extract features from the spectrum of the original audio to obtain the spectrum features of the original audio; S4: Use the cross-receptive field convolutional network CRF to extract the Mel frequency cepstral coefficients of the original audio to obtain the MFCC enhanced features of the original audio; S5: The feature fusion strategy based on the attention mechanism fuses the waveform features, spectrum features and MFCC enhanced features of the original audio to obtain the fused features; S6: Input the obtained fusion features into the classifier to identify the emotion category of the original audio.
2. According to the multi-feature fusion speech emotion recognition method based on deep learning in claim 1, it is characterized in that: The cross-receptive field convolutional network CRF includes: three feature extraction branches of different scales and N cascaded feature fusion modules; wherein the three feature extraction branches all include cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP; the convolutions of the three feature extraction branches are convolutions of three different scales respectively; the feature fusion module includes: cascaded convolutions, BN layers, RELU activation functions and global average pooling layers GAP.
3. According to the multi-feature fusion speech emotion recognition method based on deep learning in claim 2, it is characterized in that: The convolutions of the three feature extraction branches are 3×3 convolution, 9×1 convolution kernel and 1×11 convolution respectively.
4. The multi-feature fusion speech emotion recognition method based on deep learning according to claim 2 is characterized in that: The method of using a cross-receptive field convolutional network (CRF) to extract features from the Mel-frequency cepstral coefficients of the original audio includes: inputting the Mel-frequency cepstral coefficients of the original audio into three feature extraction branches of different scales for feature extraction, and obtaining feature maps of three different scales; splicing the feature maps of the three different scales, and inputting the spliced feature maps into N cascaded feature fusion modules for feature fusion to obtain the MFCC enhanced features of the original audio.
5. The multi-feature fusion speech emotion recognition method based on deep learning according to claim 1 is characterized in that: The feature fusion strategy of the attention mechanism performs feature fusion on the waveform features, spectrum features and MFCC enhancement features of the original audio, including: S51: The waveform features, spectrum features and MFCC enhancement features of the original audio are preprocessed through a cascaded Dropout layer, a 1DConv layer and a RELU activation function to obtain corresponding intermediate waveform features, intermediate spectrum features and intermediate MFCC enhancement features; S52: Concatenate the intermediate spectrum feature and the intermediate MFCC enhancement feature to obtain a first intermediate feature; S53: performing an attention mechanism on the waveform feature of the original audio according to the first intermediate feature to obtain a second intermediate feature; S54: Concatenate the second intermediate feature, the spectrum feature, and the MFCC enhanced feature to obtain a fused feature.
6. A multi-feature fusion speech emotion recognition system based on deep learning, characterized in that: The system includes a memory and a processor; the memory is used to store applications; the processor is used to run the applications and execute a multi-feature fusion speech emotion recognition method based on deep learning as described in any one of claims 1-5.
7. A computer storage medium, characterized in that: The computer storage medium stores a remote monitoring program, which, when executed by a processor, implements a multi-feature fusion speech emotion recognition method based on deep learning as described in any one of claims 1 to 5.