A speech emotion recognition method based on conformer structure and multi-task learning framework
By constructing a multi-task learning model based on the Conformer structure, combining speech recognition and emotion recognition tasks, and utilizing the correlation between tasks for deep feature extraction and sentence embedding loss, the problems of resource waste and insufficient recognition accuracy caused by independent model execution in existing technologies are solved, achieving higher recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV OF TECH
- Filing Date
- 2023-05-17
- Publication Date
- 2026-05-19
AI Technical Summary
In existing technologies, speech recognition and speech emotion recognition models are usually executed as independent models, which leads to system redundancy and waste of computing resources, as well as insufficient recognition accuracy and robustness.
We adopt a Conformer-based approach and a multi-task learning framework to construct a multi-task learning model that combines a shared encoder, an emotion recognition decoder, and a speech recognition decoder. We use the shared encoder for deep feature extraction and leverage the correlation between tasks for training. We also introduce sentence embedding loss to improve the model's generalization and recognition accuracy.
It improves the accuracy and generalization performance of speech recognition and emotion recognition models, reduces the waste of computing resources, and enhances the accuracy and robustness of recognition.
Smart Images
Figure CN116524962B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer speech recognition and speech emotion recognition technology, specifically to a speech emotion recognition method based on the Conformer structure and a multi-task learning framework. Background Technology
[0002] In recent years, with the development of internet and artificial intelligence technologies, human-computer interaction (HCI) has become increasingly widespread. There are various methods of HCI, with voice interaction being one of the most extensive and direct. The first step in HCI voice interaction is for the computer to recognize and understand human speech; speech recognition is one of the important methods for computer semantic understanding. However, in practical applications, simply obtaining text information through speech recognition is insufficient for semantic understanding. Humans generally speak with emotion, and the same sentence spoken with different emotions may convey completely different meanings. Therefore, emotional information is equally important for semantic understanding. Introducing speech emotion recognition into HCI applications can provide emotional information for subsequent semantic understanding tasks, minimizing misunderstandings by the computer.
[0003] Early speech emotion recognition models were mostly based on traditional machine learning algorithms, most notably Hidden Markov Models (HMMs), Gaussian Mixture Models (GMMs), and Support Vector Machines (SVMs). These methods trained the models to determine the relationship between speech features and emotion classification. However, these methods performed poorly in terms of both recognition accuracy and model robustness. In recent years, with the development of deep learning and the enhancement of computing power, deep neural network-based models have been widely used in the speech field. Research shows that deep learning-based speech emotion recognition models have better performance than traditional machine learning models.
[0004] Currently, there are many application scenarios that combine speech recognition and voice emotion recognition, including intelligent customer service, intelligent voice assistants, smart healthcare, and smart cockpits. Smart terminals have comprehensive voice products like Xiaomi's Xiao Ai and Baidu's Xiao Du, which incorporate various voice technologies such as speech recognition, voice emotion recognition, and speech synthesis. However, most applications execute speech recognition and voice emotion recognition as two independent models, which leads to system redundancy and a waste of computing resources. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a speech emotion recognition method based on the Conformer structure and a multi-task learning framework. This method combines a shared encoder based on the Conformer structure, an emotion recognition decoder, and a speech recognition decoder to form a multi-task learning model, which can improve the generalization performance and recognition accuracy of the model.
[0006] This invention provides a speech emotion recognition method based on the Conformer structure and a multi-task learning framework, comprising:
[0007] S1. Obtain the speech emotion dataset, extract features from the audio data of the training and validation sets in the speech emotion dataset, and generate triplet samples. The triplet samples include FBank features, emotion labels, and text labels.
[0008] S2. Construct a network model for multi-task learning. The network model includes a shared encoder, an emotion recognition decoder, and a speech recognition decoder. The shared encoder includes a Conformer model for deep feature extraction. The emotion recognition decoder includes an averaging layer and a fully connected linear layer for outputting emotion vectors. The speech recognition decoder includes an averaging layer and a CTC decoder for outputting text vectors.
[0009] S3. Train the network model using the triplet samples corresponding to the training set;
[0010] S4. Adjust the parameters of the network model using the triplet samples corresponding to the validation set;
[0011] S5. Input the audio data to be recognized into the network model to achieve speech recognition and emotion recognition of the audio data to be recognized.
[0012] The above method constructs a multi-task network model that simultaneously performs speech recognition and emotion recognition. Combining the speech recognition and emotion recognition sub-tasks leverages the correlation between tasks, allowing the model to learn additional information during training and thus improving the performance of each sub-task. The shared encoder includes a Conformer model for deep feature extraction, which effectively enhances the feature representation capability of the network model and improves the accuracy of speech recognition and emotion recognition.
[0013] Furthermore, the Conformer model includes multiple stacked Conformer modules, which are constructed by introducing the Transformer module into the CNN module.
[0014] By stacking Conformer modules, hidden features can be generated. By adding a CNN structure to the Transformer structure, the Conformer module can take advantage of the CNN module's ability to capture local features and its time-domain invariance, as well as the Transformer module's long-range modeling advantage, to link the extracted similar features together. This allows the Conformer model to both utilize local features and establish relationships between local features.
[0015] Furthermore, the Conformer module includes a first feedforward module, a self-attention module, a convolution module, a second feedforward module, and a normalization module. The calculation formula for the Conformer module is as follows:
[0016]
[0017]
[0018] x″ i =x′ i +Conv(x′ i )
[0019]
[0020] Where, x i As the input to the first feedforward module, y i is the output of the normalization module, FFN is the feedforward module, MHSA is the self-attention module, Conv is the convolution module, and LN is the normalization module.
[0021] Furthermore, the first feedforward module, self-attention module, convolution module, and second feedforward module in the Conformer module are connected through residuals.
[0022] Furthermore, multiple stacked Conformer modules are connected via residuals.
[0023] Furthermore, the shared encoder includes a convolutional downsampling layer, a fully connected layer, a Dropout layer, and a Conformer model.
[0024] Further, step S3 includes: calculating the loss function of the network model, the loss function including emotion recognition task loss, speech recognition task loss and sentence embedding loss.
[0025] Introducing sentence embedding loss can improve the generalization of speech emotion recognition models.
[0026] Furthermore, the loss function for calculating the network model includes: the sentiment vector f obtained after the triplet samples pass through the network model. e (X n ) and text vector f t (X n ) and the shared encoder output g(X) n ), where X n Let L represent the FBank feature vector, and calculate the loss function L = L1 + αL2 + βL3, where α and β are hyperparameters;
[0027] Loss in emotion recognition task Where MSELoss represents the mean squared error loss function, e n This represents the sentiment label in the triplet sample;
[0028] Speech recognition task loss Where CTCLoss represents the CTC loss function, t n This represents the text labels in the triplet sample;
[0029] Sentence embedding loss Here, CrossEntropy represents the cross-entropy loss function, pool represents the average pooling calculation, and sentence embedding loss is used to constrain the integrity of deep information in speech at the sentence level.
[0030] By using sentence embedding loss to constrain the integrity of sentiment information at the sentence level, the model's recognition accuracy and generalization can be further improved.
[0031] Further, step S1 includes: changing the speed of the audio data by a speed factor, adding noise interference to the audio data, and adding the new audio data to the training set.
[0032] The above methods can increase the amount of data in the original audio data.
[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program, wherein the computer program is stored in the memory and configured to be executed by the processor to implement the speech emotion recognition method based on the Conformer structure and multi-task learning framework.
[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the speech emotion recognition method based on the Conformer structure and multi-task learning framework.
[0035] The beneficial effects of this invention are as follows: This invention constructs a multi-task network model that simultaneously performs speech recognition and emotion recognition. Combining the speech recognition sub-task and the emotion recognition sub-task allows the model to learn additional information during training by utilizing the correlation between tasks, thereby improving the performance of each sub-task. The shared encoder includes a Conformer model for deep feature extraction, which can effectively improve the feature representation ability of the network model and improve the accuracy of speech recognition and emotion recognition. Attached Figure Description
[0036] Figure 1 This is a flowchart illustrating the training process of the network model of this invention.
[0037] Figure 2This is a structural diagram of the Conformer module of the present invention;
[0038] Figure 3 This is a structural diagram of the shared encoder of the present invention;
[0039] Figure 4 This is a flowchart of the speech representation extraction process of the present invention;
[0040] Figure 5 This is a flowchart of the training process for the emotion recognition decoder of the present invention;
[0041] Figure 6 This is a flowchart illustrating the training process of the speech recognition decoder of this invention. Detailed Implementation
[0042] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0043] The environment used in this embodiment is a GeForce GTX Titan X GPU, an Intel Core i7-5930K, a 3.50GHz CPU, 64GB of RAM, and a Linux operating system. It is developed using Python and the open-source library PyTorch.
[0044] A speech emotion recognition method based on Conformer architecture and multi-task learning framework includes:
[0045] Step S1: Divide the speech emotion dataset into training, testing, and validation sets. Extract features from the audio data in the training and validation sets to generate triplet samples. The triplet samples include FBank features, emotion labels, and text labels. Figure 4 This is a flowchart for speech representation extraction.
[0046] This embodiment uses the open-source dataset IEMOCAP, short for The Interactive Emotional Dyadic Motion Capture, an open-source emotion recognition dataset. The IEMOCAP dataset consists of 151 recorded dialogue videos, each with two speakers, for a total of 302 videos. Each segment is labeled with nine emotion categories (anger, excitement, fear, sadness, surprise, frustration, happiness, disappointment, and neutral) and their corresponding text. This invention selects four relatively typical emotions (anger, sadness, happiness, and neutral) as training data. The dataset is integrated and divided into training, validation, and test sets, and the data sample distribution is shown in Table 1 below.
[0047]
[0048] Next, feature extraction is performed on the dataset. The specific steps are as follows:
[0049] The original audio is segmented, pre-emphasized, and windowed: This embodiment divides the original WAV file into multiple small segments of fixed length by using a frame window size of 25ms and a frame shift size of 10ms; pre-emphasis is used to enhance the high-frequency part of each frame of the speech signal to improve its high-frequency signal resolution; and windowing is used to make the time-domain signal better meet the periodicity requirements of the Fast Fourier Transform (FFT) and reduce frequency leakage.
[0050] Speech signals are non-stationary and time-varying. However, within a short time frame, they can be considered stationary and time-invariant. This short time frame is typically 10ms-30ms. Therefore, in speech signal processing, to reduce the overall non-stationary and time-varying effects of the speech signal, it is segmented, with each segment called a frame, typically 25ms in length. To ensure smooth transitions between frames and maintain continuity, overlapping segmentation is generally used to ensure that adjacent frames overlap. The time difference between the start positions of two adjacent frames is called the frame shift, typically 10ms. During transmission, high-frequency signals are more prone to attenuation, and the pronunciation of vowels and other elements contains many high-frequency components. The loss of high-frequency signals may result in indistinct formants for phonemes, weakening the acoustic model's ability to model these phonemes. Pre-emphasis is a first-order high-pass filter that can increase the energy of the high-frequency components of the signal. In the real world, it is impossible to collect signals with a time span from -∞ to +∞; only signals of finite duration can be acquired. Since the framed signal is non-periodic, frequency leakage will occur after FFT transformation. To minimize this leakage error, a weighting function, also called a window function, is needed. Windowing is mainly used to make the time-domain signal appear to better meet the periodicity requirements of FFT processing, thereby reducing frequency leakage.
[0051] Extracting Fbank features as input to the model: In this embodiment, a Fast Fourier Transform is performed on the speech signal to obtain its frequency domain representation. The obtained linear frequency f is then converted into the Mel frequency in the cepstral domain, as shown in the following formula:
[0052] Eighty triangular bandpass filters with equal bandwidth were set up in the Mel spectrum range. Then, the spectral features after discrete Fourier transform were filtered to obtain the energy of the 80 filter banks. After performing a log operation, the 80-dimensional Fbank features were obtained.
[0053] Next, the acoustic features are augmented: In this embodiment, the original audio data is sped up by 0.9x and 1.1x speed factors, increasing the original audio data volume by 3 times. Additionally, some noise interference is added to the original audio data, and the resulting new audio data is added to the training set. This process can be performed using open-source speech processing frameworks such as Kaldi or audio processing tools such as librosa.
[0054] Finally, the features are processed into data files for model training: After feature extraction, they are stored in the form of files. Each feature file corresponds to a sentiment label. The paths of the feature files and the corresponding sentiment labels are organized into a text file, which includes multiple triplet samples as input for training the network model.
[0055] Step S2: Construct a network model for multi-task learning.
[0056] like Figure 2 As shown, the shared encoder includes a convolutional downsampling layer, a linear layer, a Dropout layer, and a Conformer model, which together form an encoder serving as a shared encoder for the multi-task model. In this embodiment, acoustic features are trained by stacking 15 Conformer modules to generate hidden layer features.
[0057] The specific method for constructing the Conformer model is as follows:
[0058] The Conformer module incorporates the CNN module into the Transformer module. It leverages the CNN module's ability to capture local features and its time-domain invariance, along with the Transformer module's long-range modeling advantage, to link extracted similar features together. This allows the Conformer model to both utilize local features and establish relationships between them, thereby achieving deep feature extraction.
[0059] In this embodiment, the Conformer module adds a self-attention module and a convolution module between the first and second feedforward modules. Finally, it undergoes normalization through a normalization module. The first four modules are connected via residuals; that is, the first feedforward module, self-attention module, convolution module, and second feedforward module are connected via residuals. The calculation formula for the Conformer model is as follows:
[0060]
[0061]
[0062] x″ i =x′ i+Conv(x′ i )
[0063]
[0064] Where, x i As the input to the first feedforward module, y i The output of the normalization module is FFN, the feedforward module is MHSA, the self-attention module is Conv, and the normalization module is LN (layer normalization).
[0065] The constructed Conformer modules are combined into a Conformer encoder, and there are residual connections between the Conformer modules.
[0066] Step S3: Train the network model using the triplet samples corresponding to the training set.
[0067] Using the above network model, speech recognition and emotion recognition tasks are performed based on the Conformer model, as follows:
[0068] like Figure 5 , 6 As shown, the emotion recognition decoder includes an averaging layer and a linear fully connected layer to output emotion vectors; the speech recognition decoder includes an averaging layer and a CTC decoder to output text vectors.
[0069] like Figure 1 As shown, each triplet sample obtains a speech representation after passing through a shared encoder. The speech representation is then passed through an emotion recognition decoder, a speech recognition decoder, and average pooling to obtain the emotion vector f. e (X n ), text vector f t (X n ), shared encoder output g(X) n ).
[0070] Loss in emotion recognition task Where MSELoss represents the mean squared error loss function, e n This represents the sentiment label in the triplet sample;
[0071] Speech recognition task loss Where CTCLoss represents the CTC loss function, t n This represents the text labels in the triplet sample;
[0072] The sentence embedding loss is calculated from the output of the speech recognition task. Where CrossEntropy represents the cross-entropy loss function, pool represents the average pooling calculation, sentence embedding loss is used to constrain the integrity of deep information in speech at the sentence level, and N is the number of triplet samples.
[0073] The loss function is L = L1 + αL2 + βL3, where α and β are hyperparameters.
[0074] Step S4: Adjust the parameters of the network model using the triplet samples corresponding to the validation set.
[0075] By calculating the loss gradient of the total loss function L, joint optimization and updates for the two tasks are achieved. During training, hyperparameters such as training batch size, number of network layers, and learning rate can be adjusted to fine-tune the optimization model. This embodiment obtains the following set of hyperparameter values through training and hyperparameter tuning:
[0076]
[0077] The network trained above is used to predict the sentiment category of audio in the test set, and the F1-score (a commonly used evaluation metric for classification problems, where a higher value indicates better performance) is calculated based on the true labels of the test set.
[0078] To evaluate the effectiveness of the method of this invention, the recognition performance of this embodiment is compared with several state-of-the-art multi-task speech emotion recognition methods, including MTL with self-attention, MTL with wav2vec and TIM-net. In this experiment, these models are trained on the same training set and under the same training environment and conditions. The F1-score of the trained models is calculated on the IEMOCAP test set. In order to compare the generalization of different methods, the audio of the test set is also noise-added to obtain two sets of test data: "no noise" and "noisy". The specific test results are shown in the table below.
[0079]
[0080]
[0081] The encoder in the network model of this invention acts as an intermediate embedding layer to extract information from the input speech data. This intermediate embedding layer is then used by a backend classifier to identify the corresponding sentiment category. For multi-task learning models that include both speech recognition and sentiment recognition tasks, the embedding obtained by the encoder needs to contain both semantic and sentiment information. Simultaneously, the model needs to have good generalization ability to recognize data from different scenarios. This invention proposes an improved encoder based on the Conformer structure, introducing residual and Dropout layers to retain more overall information when extracting sentiment information, thereby improving the model's generalization performance and recognition accuracy.
[0082] In practical applications, speech emotion recognition is often used in conjunction with speech recognition. To combine these two tasks and leverage their correlation, this paper combines ASR (Automatic Speech Recognition) and SER (Speech Emotion Recognition) tasks into a multi-task learning model based on multi-task learning theory. A hidden embedding is obtained through a shared encoder at the front end, and this hidden embedding is then simultaneously input into both the speech recognition and speech emotion recognition modules. During training, the inter-task correlation helps the network learn better internal representations, enabling the shared encoder to learn more shared features, thereby improving the performance of each task. Furthermore, sentence embedding loss is used to calculate the distance between the hidden embedding and the output of the speech recognition module, constraining the integrity of the information in the hidden embedding at the sentence level.
[0083] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A speech emotion recognition method based on Conformer structure and multi-task learning framework, characterized in that: include: S1. Obtain the speech emotion dataset, extract features from the audio data of the training and validation sets in the speech emotion dataset, and generate triplet samples. The triplet samples include FBank features, emotion labels, and text labels. S2. Construct a network model for multi-task learning. The network model includes a shared encoder, an emotion recognition decoder, and a speech recognition decoder. The shared encoder includes a Conformer model for deep feature extraction. The emotion recognition decoder includes an averaging layer and a linear fully connected layer for outputting an emotion vector. The speech recognition decoder includes an averaging layer and a CTC decoder for outputting text vectors; S3. Train the network model using the triplet samples corresponding to the training set; S4. Adjust the parameters of the network model using the triplet samples corresponding to the validation set; S5. Input the audio data to be recognized into the network model to realize speech recognition and emotion recognition of the audio data to be recognized; The Conformer model comprises multiple stacked Conformer modules, which are constructed by introducing a Transformer module into a CNN module; The Conformer module includes a first feedforward module, a self-attention module, a convolution module, a second feedforward module, and a normalization module. The calculation formula for the Conformer module is as follows: ， Where, x i As the input to the first feedforward module, y i is the output of the normalization module, FFN is the feedforward module, MHSA is the self-attention module, Conv is the convolution module, and LN is the normalization module.
2. The speech emotion recognition method based on the Conformer structure and multi-task learning framework according to claim 1, characterized in that: The first feedforward module, self-attention module, convolution module, and second feedforward module in the Conformer module are connected through residuals.
3. The speech emotion recognition method based on the Conformer structure and multi-task learning framework according to claim 1, characterized in that: Multiple stacked Conformer modules are connected via residuals.
4. The speech emotion recognition method based on the Conformer structure and multi-task learning framework according to claim 1, characterized in that: The shared encoder includes a convolutional downsampling layer, a fully connected layer, a Dropout layer, and a Conformer model.
5. The speech emotion recognition method based on the Conformer structure and multi-task learning framework according to claim 1, characterized in that: S3 includes: calculating the loss function of the network model, which includes emotion recognition task loss, speech recognition task loss and sentence embedding loss.
6. The speech emotion recognition method based on the Conformer structure and multi-task learning framework according to claim 5, characterized in that: The loss function for calculating the network model includes: obtaining the sentiment vector after the triplet samples pass through the network model. and text vector and shared encoder output ,in, Let L represent the FBank feature vector, and calculate the loss function L = L1 + αL2 + βL3, where α and β are hyperparameters; Loss in emotion recognition task Where MSELoss represents the mean squared error loss function, e n This represents the sentiment label in the triplet sample; Speech recognition task loss Where CTCLoss represents the CTC loss function, t n This represents the text labels in the triplet sample; Sentence embedding loss Where CrossEntropy represents the cross-entropy loss function, pool represents the average pooling calculation, sentence embedding loss is used to constrain the integrity of deep information in speech at the sentence level, and N is the number of triplet samples.
7. An electronic device, characterized in that: It includes a memory, a processor, and a computer program, wherein the computer program is stored in the memory and configured to be executed by the processor to implement the speech emotion recognition method based on the Conformer structure and multi-task learning framework as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: It stores a computer program; the computer program is executed by a processor to implement the speech emotion recognition method based on the Conformer structure and multi-task learning framework as described in any one of claims 1 to 6.