A method and apparatus for cross-database speech emotion recognition based on deep transductive transfer networks

Abstract

This invention discloses a cross-database speech emotion recognition method and apparatus based on a deep transductive transfer network. The method includes: (1) acquiring a source speech emotion database and a target speech emotion database; (2) processing the emotional audio data from the source and target databases into spectrograms; (3) establishing a deep regression neural network; (4) inputting the spectrograms from the source and target databases into the deep regression neural network for training, calculating the maximum mean difference at different scales between the source and target data based on the features obtained from the network, and fine-tuning the neural network; (5) inputting the speech to be recognized as speech data from the target database into the trained deep convolutional neural network to obtain the speech emotion type. This invention achieves higher recognition accuracy.

Description

Technical Field

[0001] This invention relates to speech emotion recognition technology, and more particularly to a cross-database speech emotion recognition method and apparatus based on a deep transductive transfer network. Background Technology

[0002] Speech is one of the most natural forms of human expression. Compared to other common forms of communication in daily life, it is also more revealing of natural human emotional states. Emotions help people understand each other better. For this reason, in order to help humans understand and recognize speech emotions more efficiently, automatic speech emotion recognition using computer programs and artificial intelligence algorithms has become a popular research direction in recent years in the fields of pattern recognition, computer vision, and affective computing.

[0003] In recent years, researchers have proposed many effective methods based on machine learning and deep learning for automatically recognizing speech emotions. For example, traditional machine learning methods typically extract handcrafted features, such as ISO9 and IS10, and then construct various types of classifiers, such as SVM, K-NN, and Bayesian classifiers, specifically for speech emotion recognition tasks. Meanwhile, some deep learning methods have also been used for speech emotion recognition tasks, such as LSTM, pre-trained CNNs (e.g., ResNet, VggNet, and DenseNet), and CapsuleNet. These networks can generally improve the representation of speech emotions and learn emotion features and classifiers in an end-to-end manner to classify speech emotions.

[0004] The methods described above assume the ideal scenario where test and training samples come from the same database. However, in many practical applications, test and training samples typically come from different databases, which can easily lead to significant domain differences, resulting in unsatisfactory performance for most speech emotion recognition methods across different databases. Recently, many researchers have attempted to address the cross-database speech emotion recognition problem. For example, Zong et al. proposed a domain adaptation method based on least squares regression (DaLSR) to handle cross-database speech emotion recognition tasks. Hassan et al. proposed an importance-weighted support vector machine (IW-SVM) to eliminate feature distribution mismatches between different samples and improve classification accuracy across different databases. Long et al. proposed applying transfer kernel learning (TKL) to learn a domain-invariant kernel to eliminate feature distribution differences between samples from different databases. Gong et al. proposed a method called geodesic flow kernel (GFK), using a well-designed GFK on a Grassmann manifold to connect two different databases and reduce the feature distribution differences between them. Deng et al. proposed a universal autoencoder (UAE) to learn a database-independent feature space, aiming to leverage the powerful mapping capabilities of the UAE to map test and training samples to a domain-independent feature space. Fernando et al. proposed a subspace alignment (SA) method to find a mapping function that aligns the subspace of the source sample with that of the target sample. Pan et al. proposed a Transition Component Analysis (TCA) method based on the reproducing kernel Hilbert space, which eliminates the distribution differences between samples from different domains by finding some cross-domain transition components. Gideon et al. proposed a method based on Adversarial Discriminative Domain Generalization (ADDoG), which can learn more generalized speech emotion features from more database samples. Most of the above research on cross-database speech emotion recognition is mainly based on traditional machine learning methods, relying on domain-invariant spatiotemporal descriptors and machine learning classifiers to handle cross-database speech emotion recognition tasks. The recognition rate is not ideal, and it is still some distance from practical application. In addition, with the expansion of speech emotion datasets, deep learning-based methods will be one of the main research directions for cross-database speech emotion recognition in the future, but there are currently few related studies, and the overall progress is relatively slow. Summary of the Invention

[0005] Purpose of the invention: This invention addresses the problems existing in the prior art by providing a cross-database speech emotion recognition method and apparatus based on a deep transductive transfer network with higher recognition accuracy.

[0006] Technical solution: The cross-database speech emotion recognition method based on deep transductive transfer networks described in this invention includes:

[0007] (1) Obtain two different speech emotion databases, namely the source database and the target database. The source database stores emotional speech audio and corresponding emotion type labels, while the target database only stores emotional speech audio.

[0008] (2) Process the emotional speech audio from the source database and the target database into spectrograms;

[0009] (3) Establish a deep regression neural network;

[0010] (4) Input the spectrograms and corresponding labels of the source database, the spectrograms and randomly initialized pseudo-labels of the target database as samples into the deep regression neural network for pre-training; then input the obtained deep features as samples into the deep regression neural network for fine-tuning to complete the training.

[0011] (5) The audio of the speech emotion to be identified is preprocessed into a phrasal graph and used as a sample in the target database to input into the trained deep regression neural network to obtain the emotion type of the speech.

[0012] Furthermore, step (2) specifically includes: using the Python librosa toolkit to process the emotional speech audio from the source database and the target database into spectrograms.

[0013] Furthermore, the deep regression neural network established in step (3) includes, in order from front to back, a first convolutional layer, a first maximum pooling layer, a second convolutional layer, a second maximum pooling layer, a third convolutional layer, a fourth convolutional layer, a third maximum pooling layer, a fifth convolutional layer, a sixth convolutional layer, a fourth maximum pooling layer, a seventh convolutional layer, an eighth convolutional layer, a fifth maximum pooling layer, an adaptive average pooling layer, and a fully connected layer.

[0014] Furthermore, each neuron in all convolutional and fully connected layers uses the ReLU (Recursive Linearity Unit) as its activation function.

[0015] Furthermore, the neuron output of the fully connected layer uses Dropout=0.5 to prevent overfitting.

[0016] Furthermore, step (4) includes:

[0017] (4-1) The spectrograms and corresponding labels from the source database, and the spectrograms and randomly initialized pseudo-labels from the target database are simultaneously input into the deep regression neural network as samples. The loss function L used during training is:

[0018]

[0019] Where n is the source database sample ID, N is the number of source database samples, m is the target database sample ID, M is the target database sample number, j is the voice emotion category ID, and J is the number of voice emotion categories. Let be the probability that the actual output sentiment feature of the nth sample in the source database is classified as j. The expected probability of the output sentiment feature of the nth sample in the source database being divided into j is given. Let be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j. Let j be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j in the previous training round.

[0020] (4-2) The deep features obtained from the source database and target database through the deep regression neural network are used as sample inputs to the trained deep regression neural network for fine-tuning training. The loss function used during training is L. total for:

[0021] L total =αL mmd +βL

[0022]

[0023] Among them, MMD(X) s , Y t X represents the maximum mean difference (MMD) of the sentiment features output by samples from the source and target databases on a deep regression neural network. s Y represents the sentiment features of the source database samples output by the deep regression neural network. t This represents the sentiment features of the target database samples as output by the deep regression neural network. The maximum mean difference (MMD) represents the sentiment features of category j output by samples from the source and target databases on a deep regression neural network. This represents the distribution of sentiment features of category j in the output of a deep regression neural network for samples from the source database. This represents the sentiment feature of category j in the output of a deep regression neural network for a sample from the target database. The maximum mean difference (MMD) represents the difference between positive and negative sentiment features in the output of a deep regression neural network, representing samples from the source and target databases. This represents the positive and negative sentiment characteristics of the source database samples as output by the deep regression neural network. The positive and negative sentiment characteristics of the target database samples in the output of the deep regression neural network are represented by MMD, which is the mean distance between two sets of data in the regenerated Hilbert space, and α and β are the binding strength coefficients obtained through training.

[0024] The cross-library speech emotion recognition device based on deep pushover transfer network of the present invention includes a processor and a computer program stored in memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method.

[0025] Beneficial effects: Compared with the prior art, the significant advantage of this invention is that it has a higher recognition accuracy. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the cross-database speech emotion recognition method based on deep transductive transfer networks provided by the present invention.

[0027] Figure 2 This is a schematic diagram of the cross-database speech emotion recognition architecture of the Deep Transmission Transfer Network (DTTRN).

[0028] Figure 3 This is the detailed structure of the backbone network of the Deep Push Transmission Network (DTTRN);

[0029] Figure 4 This is a comparison of experimental results of the Deep Transductive Transfer Network (DTTRN) in cross-database speech emotion recognition. Detailed Implementation

[0030] This embodiment provides a cross-database speech emotion recognition method based on a deep transductive transfer network, such as... Figure 1 Shown, including:

[0031] (1) Obtain two different voice emotion databases, namely the source database and the target database. The source database stores emotional voice audio and corresponding emotion type labels, while the target database only stores emotional voice audio.

[0032] In this invention, the training samples (source database or source domain) and the test samples (target domain or target database) belong to different speech emotion datasets, and the two have obvious differences in feature distribution.

[0033] (2) Process the emotional speech audio from the source database and the target database into spectrograms.

[0034] In this embodiment, spectrogram processing is implemented using the librosa toolkit in Python.

[0035] (3) Establish a deep regression neural network (CNN, DTTRN), the network including the CNN backbone network and the fully connected layer FC1 connecting the backbone network.

[0036] like Figure 2As shown, the structure of the deep convolutional neural network includes, from front to back, the following layers connected in sequence: first convolutional layer conv1, first max pooling layer Maxpool1, second convolutional layer conv2, second max pooling layer Maxpool2, third convolutional layer conv3, fourth convolutional layer conv4, third max pooling layer Maxpool3, fifth convolutional layer conv5, sixth convolutional layer conv6, fourth max pooling layer Maxpool4, seventh convolutional layer conv7, eighth convolutional layer conv8, fifth max pooling layer Maxpool5, adaptive average pooling layer Adavepool1, and fully connected layer FC1. Each neuron in the convolutional and fully connected layers uses the ReLU (Reduced Linearity Unit) as the activation function, and the outputs of all fully connected layer neurons are Dropout = 0.5 to prevent overfitting.

[0037] like Figure 3 As shown, the local receptive field size of each of the eight convolutional layers in the network is set to 3×3, and the stride is set to 1. Zero-padding is used after convolution to maintain the feature map size. For the first convolutional layer, 64 kernels are used; for the second, 128 kernels; for the third and fourth, 256 kernels; and for the last four, 512 kernels. For the two pooling layers, the window size of the max pooling layer is set to 2×2, and the window size of the adaptive average pooling layer is set to 7×7, with a stride of 2. Each pooling operation reduces the feature dimension of the output map to half its original size.

[0038] The network activation function is set as follows: the Rectified Linear Unit (ReLU) is used as the activation function for each neuron in the DTTRN network, and its definition is as follows:

[0039]

[0040] (4) Input the spectrograms and corresponding labels of the source database, the spectrograms and randomly initialized pseudo-labels of the target database as samples into the deep regression neural network for pre-training; then input the obtained deep features as samples into the deep regression neural network for fine-tuning to complete the training.

[0041] This step specifically includes:

[0042] (4-1) The spectrograms and corresponding labels from the source database, and the spectrograms and randomly initialized pseudo-labels from the target database are simultaneously input into the deep regression neural network as samples. The loss function L used during training is:

[0043]

[0044] Where n is the source database sample ID, N is the number of source database samples, m is the target database sample ID, M is the target database sample number, j is the voice emotion category ID, and J is the number of voice emotion categories. Let be the probability that the actual output sentiment feature of the nth sample in the source database is classified as j. The expected probability of the output sentiment feature of the nth sample in the source database being divided into j is given. Let be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j. Let j be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j in the previous training round.

[0045] (4-2) The deep features obtained from the source database and target database through the deep regression neural network are used as sample inputs to the trained deep regression neural network for fine-tuning training. The loss function used during training is L. total for:

[0046] L total =αL mmd +βL

[0047]

[0048] Among them, MMD(X) s , Y t X represents the maximum mean difference (MMD) of the sentiment features output by samples from the source and target databases on a deep regression neural network. s Y represents the sentiment features of the source database samples output by the deep regression neural network. t This represents the sentiment features of the target database samples as output by the deep regression neural network. The maximum mean difference (MMD) represents the sentiment features of category j output by samples from the source and target databases on a deep regression neural network. This represents the distribution of sentiment features of category j in the output of a deep regression neural network for samples from the source database. This represents the sentiment feature of category j in the output of a deep regression neural network for a sample from the target database. The maximum mean difference (MMD) represents the difference between positive and negative sentiment features in the output of a deep regression neural network, representing samples from the source and target databases. This represents the positive and negative sentiment characteristics of the source database samples as output by the deep regression neural network. The positive and negative sentiment characteristics of the target database samples in the output of the deep regression neural network are represented by MMD, which is the mean distance between two sets of data in the regenerated Hilbert space, and α and β are the binding strength coefficients obtained through training.

[0049] (5) The audio of the speech emotion to be identified is preprocessed into a phrasal graph and used as a sample in the target database to input into the trained deep regression neural network to obtain the emotion type of the speech.

[0050] The training optimizer employs a stochastic gradient descent (SGD) algorithm with a correction factor momentum (Nesterov Momentum). It calculates the loss function to continuously update the weights. Nesterov Momentum, by simulating the concept of momentum in physics, suppresses oscillations in the gradient direction and accelerates convergence. If the historical gradient is aligned with the current gradient direction, the momentum term increases; otherwise, it decreases. The Nesterov term adds a correction during gradient updates to prevent the gradient from advancing too rapidly, while also making the gradient update more flexible. The iterative process is as follows:

[0051]

[0052] θ = 0 - v t

[0053] Where η represents the learning rate, which was set to 10 in the experiment. -3 In addition, the weight decay is set to 10. -5 The correction factor is 0.9.

[0054] To accelerate training and improve the reliability of recognition results, the weights of each layer in the DTTRN network model pre-trained with the source database are frozen. Then, by combining target domain samples and source domain samples, only the weights of the last fully connected layer of the DTTRN are trained and fine-tuned to achieve the expected cross-database speech emotion recognition task. The maximum training epoch is set to 200.

[0055] This embodiment also provides a cross-library speech emotion recognition device based on a deep pushover migration network, including a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, it implements the above-described method.

[0056] To verify this invention, simulations were conducted. Simulating cross-database speech emotion recognition experiments requires sample data from different databases with the same sample labels. EmoDB, eNTERFACE, and CASIA were selected, and we use B, E, and C to represent the corresponding databases, respectively. In the transfer between EmoDB and CASIA, we used five emotions: anger, sadness, fear, happiness, and indifference; in the transfer between EmoDB and eNTERFACE, we used anger, sadness, fear, happiness, and disgust; and in the transfer between CASIA and eNTERFACE, we used anger, sadness, fear, happiness, and surprise. To verify the effectiveness and necessity of the Deep Transmission Transfer Learning Network (DTTRN) in this invention, we conducted cross-database speech emotion experiments on EmoDB, eNTERFACE, and CASIA. UAR (Unweighted Average Recall) was selected as the evaluation metric in the experiments, and the results are as follows: Figure 4 As shown, the Transductive Transfer Deep Regression Network (DTTRN) in this invention achieves excellent recognition results in cross-database speech emotion recognition tasks. DTTRN borrows the idea of ​​transductive transfer and utilizes unlabeled target domain data for better network pre-training. Furthermore, the designed loss function constrains the feature distribution differences between the two databases on the network during DTTRN fine-tuning training, playing a crucial role in improving cross-database recognition performance.

[0057] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A cross-database speech emotion recognition method based on deep transductive transfer networks, characterized in that... The method includes: (1) Obtain two different speech emotion databases, namely the source database and the target database. The source database stores emotional speech audio and corresponding emotion type labels, while the target database only stores emotional speech audio. (2) Process the emotional speech audio from the source database and the target database into spectrograms; (3) Establish a deep regression neural network; (4) Input the spectrograms and corresponding labels of the source database, the spectrograms and randomly initialized pseudo-labels of the target database as samples into the deep regression neural network for pre-training; then input the obtained deep features as samples into the deep regression neural network for fine-tuning to complete the training. (5) The audio of the speech emotion to be identified is preprocessed into a spectrogram and used as a sample in the target database to input into the trained deep regression neural network to obtain the emotion type of the speech; Step (4) includes: (4-1) The spectrograms and corresponding labels from the source database, and the spectrograms and randomly initialized pseudo-labels from the target database are simultaneously input into the deep regression neural network as samples. The loss function L used during training is: Where n is the source database sample ID, N is the number of source database samples, m is the target database sample ID, M is the target database sample number, j is the voice emotion category ID, and J is the number of voice emotion categories. Let be the probability that the actual output sentiment feature of the nth sample in the source database is classified as j. The expected probability of the output sentiment feature of the nth sample in the source database being divided into j is given. Let be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j. Let j be the probability that the actual output sentiment feature of the m-th sample in the target database is classified as j in the previous training round. (4-2) The deep features obtained from the source database and target database through the deep regression neural network are used as sample inputs to the trained deep regression neural network for fine-tuning training. The loss function used during training is L. total for: L total =αL mmd +βL Among them, MMD(X) s , Y t X represents the maximum mean difference (MMD) of the sentiment features output by samples from the source and target databases on a deep regression neural network. s Y represents the sentiment features of the source database samples output by the deep regression neural network. t This represents the sentiment features of the target database samples as output by the deep regression neural network. The maximum mean difference (MMD) represents the sentiment features of category j output by samples from the source and target databases on a deep regression neural network. This represents the distribution of sentiment features of category j in the output of a deep regression neural network for samples from the source database. This represents the sentiment feature of category j in the output of a deep regression neural network for a sample from the target database. The maximum mean difference (MMD) represents the difference between positive and negative sentiment features in the output of a deep regression neural network, representing samples from the source and target databases. This represents the positive and negative sentiment characteristics of the source database samples as output by the deep regression neural network. The positive and negative sentiment characteristics of the target database samples in the output of the deep regression neural network are represented by MMD, which is the mean distance between two sets of data in the regenerated Hilbert space, and α and β are the binding strength coefficients obtained through training.

2. The cross-database speech emotion recognition method based on deep transductive transfer networks according to claim 1, characterized in that: Step (2) specifically includes: using the librosa toolkit in Python to process the emotional speech audio from the source database and the target database into spectrograms.

3. The cross-database speech emotion recognition method based on deep transductive transfer networks according to claim 1, characterized in that: The deep regression neural network established in step (3) includes the following layers connected from front to back: first convolutional layer, first maximum pooling layer, second convolutional layer, second maximum pooling layer, third convolutional layer, fourth convolutional layer, third maximum pooling layer, fifth convolutional layer, sixth convolutional layer, fourth maximum pooling layer, seventh convolutional layer, eighth convolutional layer, fifth maximum pooling layer, adaptive average pooling layer, and fully connected layer.

4. The cross-database speech emotion recognition method based on deep transductive transfer networks according to claim 3, characterized in that: Each neuron in all convolutional and fully connected layers uses the ReLU (Recursive Linear Unit) as its activation function.

5. The cross-database speech emotion recognition method based on deep transductive transfer networks according to claim 3, characterized in that: The neuron output of the fully connected layer uses Dropout=0.5 to prevent overfitting.

6. A cross-database speech emotion recognition device based on a deep transductive transfer network, comprising a processor and a computer program stored in memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method described in any one of claims 1-5.

Images