A cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment

Abstract

The present invention discloses a cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment. The present invention comprises the following steps: first, a deep network model based on deep denoising autoencoder and deep neural network is constructed to compress feature redundant information and improve feature representation capability; then, a global domain and subdomain adaptive method is used to realize feature migration, while reducing the impact of sample imbalance on model recognition performance; finally, in the training stage, a dynamic weight factor is constructed to adjust the contribution of different loss functions to achieve model optimization. The method proposed by the present invention can effectively learn the common emotion information of sample imbalance corpus and reduce feature distribution differences.

Description

[0001] Technology Neighborhood

[0002] The present invention belongs to the technical field of speech signal processing, and in particular relates to a cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment. Background Art

[0003] Speech emotion recognition is an important technical foundation for human-computer interaction. Traditional speech emotion recognition research is often based on the same corpus for training and testing, and has achieved very good recognition results. However, due to the differences in recording environment, gender, age distribution, language, etc. of different corpora, there are great differences in the distribution of speech features in different corpora, which is a typical cross-corpus speech emotion recognition problem. Therefore, how to effectively deal with the feature distribution differences caused by cross-corpus training is a very important and challenging problem in speech emotion recognition research.

[0004] Inspired by the successful application of transfer learning in text classification and clustering, image classification, sensor localization, collaborative filtering, etc., domain adaptation is introduced in the research of cross-database speech emotion recognition to reduce the differences in feature distribution between different fields.

[0005] Therefore, the present invention mainly focuses on cross-database speech emotion recognition between different corpora. First, the present invention obtains low-dimensional speech emotion features based on a deep network model built based on a deep denoising autoencoder and a deep neural network. Among them, the deep denoising autoencoder can effectively compress feature redundant information and improve the robustness of the model; the deep neural network has a strong nonlinear fitting ability, which can effectively improve the emotion representation ability of speech features. Then, MMD and LMMD are introduced to simultaneously reduce the feature distribution distance and alleviate the impact of sample imbalance on model recognition performance. Finally, in the training stage, the dynamic weight factor is used to dynamically adjust the contribution of different loss functions to model optimization. Summary of the invention

[0006] In order to solve the problem of feature distribution differences between different corpus databases, better transfer the knowledge of labeled source domain data to unlabeled target domain, and achieve accurate classification of unlabeled data, a cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment is proposed. The specific steps are as follows:

[0007] (1) Prepare corpora: Obtain two unbalanced corpora, which are used as the source domain database and the target domain database, respectively. The source domain database includes a number of speech signals and corresponding emotion category labels, and the target domain database includes a number of speech signals.

[0008] (2) Speech preprocessing: preprocess the speech signals in the source domain database and the target domain database to prepare for the next step of feature extraction;

[0009] (3) Speech feature extraction: extract speech emotion features from the speech signal after preprocessing in step (2), including but not limited to MFCC, short-time average zero-crossing rate, fundamental frequency, mean, standard deviation, maximum and minimum values, etc.;

[0010] (4) Feature processing: First, the source domain features obtained in step (3) are processed. Labels corresponding to source domain features and target domain features Then in X s and X T After adding noise that follows normal distribution, input it into the deep autoencoder for feature reconstruction:

[0011]

[0012]

[0013] in and is the sample feature after decoding and reconstruction by the deep autoencoder. Then, the encoding output of the deep autoencoder is input into the deep neural network for further processing, thereby respectively obtaining the low-dimensional sentiment features of the source domain and the target domain. and Finally, use the true label Y of the source domain s And the source domain feature probability predicted by the softmax classifier Perform cross entropy calculation:

[0014]

[0015] (5) Feature migration: First, the maximum mean discrepancy (MMD) algorithm is used to reduce X′ S and X′ T The global domain feature distribution distance of:

[0016]

[0017] Where H is the reproducing kernel hillbert space (RKHS), δ(·) is the feature mapping function (Gaussian kernel function). Then, the local maximum mean discrepancy (LMMD) is used to reduce X′ at the same time. S and X′ T The global feature distribution distance of:

[0018]

[0019] in The source domain sample The weight of each sample belonging to the sentiment category C, The target domain sample The weight of each sample belonging to the sentiment category C;

[0020] (6) Model training: Based on the five loss functions obtained in steps (4) and (5), the dynamic weight factor w i To adjust the contribution of different loss functions to model optimization, the overall optimization goal of the model is obtained as follows:

[0021]

[0022] The dynamic weight factor is expressed as:

[0023]

[0024] where i∈{S,T,y,MMD,LMMD},α i >0.

[0025] (7) Repeat steps (4) and (5), iteratively train the network model through the gradient descent method, and continuously update the dynamic weight factor of step (6) until the model is optimal;

[0026] (8) Using the network model trained in step (7), use the sofmatx classifier to predict the target domain feature labels in step (3), and finally realize the emotion recognition of speech emotion under cross-corpus conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] As shown in the attached figure, Figure 1 This is a training flowchart of a cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment. Figure 2 This is a test flowchart of a cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with specific implementation methods.

[0029] (1) The open source toolkit Opensmile was used to extract the standard feature set of the 2010 International Speech Emotion Recognition Challenge from speech. The features extracted from each speech were 1582 dimensions. Therefore, the EMO-DB database has 368 speech records in 5 categories of emotional speech, with a total data size of 368*1582; ​​the eNTERFACE database has 1072 speech records in 5 categories of emotional speech, with a total data size of 1072*1582; ​​the CASIA Chinese Speech Emotion Database has 1200 speech records in 6 categories of emotional speech, with a total data size of 1200*1582.

[0030] (2) Normalize the source domain features and target domain features and add noise that follows the normal distribution.

[0031] (3) A deep network model based on deep autoencoder and deep neural network was established. The number of hidden layers of DAE was 5, and the number of hidden layer neuron nodes was set to 1200, 900, 500, 900, and 1200 respectively. The activation function of the encoding stage used the ELU function, and the activation function of the decoding stage used the Sigmoid function. In addition, BatchNorm layer and Dropout layer were added to each layer of DAE. The number of hidden layers of DNN was set to 2, and the number of hidden layer neuron nodes of DNN was 600 and 256 respectively, and the activation function used the Sigmoid function.

[0032] (4) The source domain and target domain dataset features obtained in (2) are respectively input into a deep network model based on a deep autoencoder and a deep neural network to extract low-dimensional sentiment features.

[0033] (5) In the low-dimensional sentiment space, MMD and LMMD are used to measure the feature distribution distance of the low-dimensional sentiment features from the source and target domains. The calculation of MMD only requires the low-dimensional sentiment features of the source and target domains, while when calculating the sub-domain adaptive loss function based on LMMD, in addition to using the low-dimensional features of the source domain and the corresponding true labels and low-dimensional features of the target domain, the probability distribution calculated by softmax is also used as the label of the target domain, which is a pseudo label.

[0034] (6) The network loss of the model is:

[0035]

[0036] in

[0037] and represents the metric feature distribution loss function, represents the source domain reconstruction loss, represents the target domain reconstruction loss, represents the source domain classification loss, α i is a hyperparameter used to enhance the contribution of different losses in the overall optimization loss.

[0038] (7) The learning rate and batch size of the model are set to 0.00001 and 100, and the Adam gradient descent method is used to train the network model. The model is iterated for 500 times, and the classifier uses softmax. In MMD and LMMD, the feature mapping function uses a multi-core Gaussian function, and the number of Gaussian cores is set to 5. At the end of each round of training, a set of loss function values ​​is generated to update the weights w in (6) i , to achieve dynamic adjustment of loss weights.

[0039] (8) The speech signal to be recognized is normalized and input into the trained deep network model. The category with the highest output probability using the softmax classifier is the recognized emotion category.

[0040] The following verification is carried out using the eNTERFACE database (abbreviated as E), MO-DB database (abbreviated as B) and CASIA database (abbreviated as C) as examples. The four cross-library experiments listed are all sample imbalance experiments, and the verification results are shown in Table 1.

[0041] Table 1 Experimental results with comparison methods (UAR / WAR)

[0042]

[0043] The accuracy measurement methods used as models are UAR, which is the unweighted average recall, and WAR, which is the weighted average recall. PCA+SVM, DoSL, TSDSL and JDAR are all cross-database speech emotion recognition methods that use the IS10 feature set and cross-database recognition of the same emotion category. The invention is a cross-database speech emotion recognition method built on the above claims.

[0044] The experimental results show that the proposed invention can achieve the best cross-database speech recognition rate in the sample imbalance experiment. The model proposed by the present invention combines the deep network model and two transfer learning algorithms, and also uses dynamic weight factors to adjust the contribution of different loss functions to achieve the best model training.

[0045] The scope of protection sought for the present invention is not limited to the description of this specific embodiment.

Claims

1. A cross-database speech emotion recognition method based on transfer learning and multi-loss dynamic adjustment, characterized in that: The following steps are involved: (1) Prepare a corpus: Obtain a corpus with unbalanced samples as a source domain database and a target domain database, respectively. The source domain database includes a number of speech signals and corresponding emotion category labels, and the target domain database includes a number of speech signals. (2) Speech preprocessing: preprocess the speech signals in the source domain database and the target domain database to prepare for the next step of feature extraction; (3) Speech feature extraction: extracting speech emotion features from the speech signal after preprocessing in step (2), the features including but not limited to MFCC, short-time average zero-crossing rate, fundamental frequency, mean, standard deviation, maximum value, and minimum value; (4) Feature processing: First, the source domain features are obtained through step (3) Labels corresponding to source domain features and target domain features Then in X s and X T After adding noise that follows normal distribution, input it into the deep autoencoder for feature reconstruction: in and is the sample feature after decoding and reconstruction by the deep autoencoder; then, the encoding output of the deep autoencoder is input into the deep neural network for further processing, so as to obtain the low-dimensional sentiment features of the source domain and the target domain respectively. and Finally, use the true label Y of the source domain s And the source domain feature probability predicted by the softmax classifier Perform cross entropy calculation: (5) Feature migration: First, the maximum mean difference (MMD) algorithm is used to reduce X′ S and X′ T The global feature distribution distance of: Where H is the reproducing kernel Hilbert space RKHS, δ(·) is the feature mapping function, i.e., the Gaussian kernel function; then, the local maximum mean difference LMMD is used to simultaneously reduce X′ S and X′ T The global feature distribution distance of: in The source domain sample The weight of each sample belonging to the sentiment category C, The target domain sample The weight of each sample belonging to the sentiment category C; (6) Model training: Based on the five loss functions obtained in steps (4) and (5), the dynamic weight factor w i To adjust the contribution of different loss functions to model optimization, the overall optimization goal of the model is obtained as follows: minL sum =∑ i In i L i ,In i >0 (6) The dynamic weight factor is expressed as: where i∈{S,T,y,MMD,LMMD},α i >0; (7) Repeat steps (4) and (5), iteratively train the network model through the gradient descent method, and continuously update the dynamic weight factor of step (6) until the model is optimal; (8) Using the network model trained in step (7), use the sofmatx classifier to predict the target domain feature labels in step (3), and finally realize the emotion recognition of speech emotion under cross-corpus conditions.

Images