A cross-database speech emotion recognition method based on one-dimensional convolutional autoencoder and adversarial domain adaptation

Abstract

The present invention discloses a cross-database speech emotion recognition method based on one-dimensional convolutional autoencoder and adversarial domain adaptation. This method first uses a one-dimensional convolutional autoencoder based on the encoding and decoding style to process the features to obtain more representative features. Then, the maximum mean difference (MMD) is used as a source domain target domain distance measurement method and incorporated into the adversarial domain adaptation architecture to alleviate the balance challenges faced in its learning process and promote the effective alignment of the source domain and target domain feature distributions. Finally, the model comprehensively considers the feature extraction network and feature distribution difference loss to improve the recognition effect.

Description

Technical Field

[0001] The present invention relates to a cross-database speech emotion recognition method based on a one-dimensional convolutional autoencoder and adversarial domain adaptation. Background Art

[0002] In recent years, speech emotion recognition has become a hot topic in human-computer interaction, pattern recognition and emotional computing. Giving computers the ability to recognize emotions is a key technology to make machines intelligent. In traditional speech emotion recognition, many algorithms have achieved remarkable recognition performance, but these algorithms are often trained and tested on the same speech emotion database. In practical applications, training data and test data often come from different databases, with different languages, cultures, speaker pronunciation styles and classification methods. These differences cause a mismatch in the distribution of training and test data, and also cause the performance of the originally well-trained model to drop severely when applied to an unknown test set. As a result, cross-database speech emotion recognition, a very challenging problem, came into being, and many researchers began to work on improving the recognition performance of cross-database speech emotions.

[0003] To address this problem, a large number of algorithms based on feature processing and feature distribution alignment have been proposed. Feature processing mainly involves building a new network model to extract more representative emotional features from speech signals, which is conducive to the next step of feature distribution alignment. Feature distribution alignment mainly uses domain adaptation methods to reduce the data distribution differences between the training library and the test library, so that the good performance of the model on the training library can be transferred to the test library.

[0004] Therefore, the present invention mainly focuses on how to effectively extract emotional features with strong representation ability in speech signals and reduce the difference in feature distribution between the training speech library and the test speech library. A one-dimensional convolution layer is used to explore the connection between neighboring features and emotional information after convolution, and a neural network based on the encoding and decoding style is used to ensure the effectiveness of the proposed features. After effective feature processing, adversarial domain adaptation is used to complete feature distribution alignment, and MMD is used to reduce the balance challenge in adversarial learning, so as to better realize knowledge transfer and improve the cross-library speech emotion recognition performance. Summary of the invention

[0005] For cross-database speech emotion recognition, effective extraction of emotional features and alignment of feature distributions between different libraries are key technologies to improve performance. This paper constructs a new one-dimensional convolutional autoencoder to extract emotional features with strong representation capabilities, and uses adversarial domain adaptation to complete feature distribution alignment. In particular, MMD is used to alleviate the balance challenge in adversarial learning and better complete knowledge transfer. The specific steps are as follows:

[0006] (1) Speech preprocessing: Extract speech features such as fundamental frequency (F0), Mel-frequency cepstrum (MFCC) and line spectral pair (LSP) from the original speech signal as input to the feature processing network;

[0007] (2) Feature processing: The one-dimensional statistical features extracted in step (1) are sent to a one-dimensional convolutional autoencoder. The features are first processed using a convolutional layer, and then restored using a transposed convolutional layer. The restored speech features and the original features are combined to construct a reconstruction error. The training library classification loss, convolution process, deconvolution process, and reconstruction loss are defined as follows:

[0008]

[0009] where y p Predict the sentiment label for the network output, y s is the real emotion label, L ce is the cross entropy function,

[0010]

[0011]

[0012] Where [X j1 ,X j2 ...X js ] are adjacent statistical features, Y(j) is the result obtained after convolution, K and b are the weight and bias in the convolution process, respectively.

[0013]

[0014] Where X R and X are the reconstructed features and original features respectively;

[0015] (3) Feature distribution alignment: The features output by the encoder in step (2) are fed into the domain discriminator of the adversarial domain adaptation module. The feature distributions of the training and test libraries are aligned by confusing the domain discriminator, and the statistical differences between the training and test libraries are simultaneously measured using MMD. The domain discriminator loss and MMD distance are as follows:

[0016]

[0017] Where F is the feature processing network, D is the domain discriminator, and d i is the real domain label of the test library,

[0018]

[0019] Where H is the regenerated Hilbert space, which is used to find the feature mapping function φ so that the feature distribution distance between the training library and the test library is minimized after mapping;

[0020] (4) Model training: The one-dimensional statistical features obtained in step (1) are input into the feature processing network to obtain the classification loss and reconstruction loss , the features with strong representation ability extracted by the encoder in the convolutional autoencoder are sent to the domain discriminator to obtain the adversarial loss , and use MMD to minimize the statistical difference and get the MMD loss , combine these losses to optimize the model:

[0021]

[0022] (5) Repeat steps (2), (3), (4) to optimize the network model parameters;

[0023] (6) After a certain number of iterations, the optimal model under the guidance of this method is obtained, achieving improved cross-database speech emotion recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] As shown in the attached figure, Figure 1 It is a flow chart for implementing the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with specific implementation methods.

[0026] (1) Feature extraction: extract speech features such as fundamental frequency (F0), Mel-frequency cepstrum (MFCC) and line spectral pair (LSP) from the original speech signal, and use multiple statistical functions to obtain a total of 1582-dimensional speech statistical features as the input of the feature processing network.

[0027] (2) Feature processing: The one-dimensional speech features obtained in step (1) are input into a one-dimensional convolutional autoencoder. The autoencoder includes four one-dimensional convolutional modules and four one-dimensional transposed modules. The one-dimensional convolutional module consists of a convolutional layer, a batch normalization (BN) layer, a dropout layer and an activation layer (Relu function). The parameters of the four one-dimensional convolutional layers are set to {16×9×2, 32×9×2, 64×9×2, 128×9×2} (number of convolution kernels × convolution kernel size × step size). The one-dimensional deconvolution module consists of a one-dimensional transposed convolutional layer, a batch normalization (BN) layer and an activation layer (Relu function). The parameters of the four one-dimensional transposed convolutional layers are set to {64×9×2, 32×9×2, 16×9×2, 1×9×2} (number of convolution kernels × convolution kernel size × step size).

[0028] (3) The features with strong representation ability obtained after feature processing are input into the adversarial domain adaptation module, and the confusion domain discriminator method is used to reduce the difference between the processed training speech library features and the test speech library features. The domain discriminator uses three Dense layers to classify the input features. The first Dense layer is set to a fully connected layer (1582, 1024), an activation layer (Relu function) and a dropout (0.5) layer. The second Dense layer is set to a fully connected layer (1024, 1024), an activation layer (Relu function) and a dropout (0.5) layer. The third Dense layer is set to (1024, 1) and the activation layer (sigmoid function) for domain classification. When the domain discriminator cannot distinguish whether the feature comes from the training library or the test library, the effect of promoting the alignment of feature distribution is achieved. However, due to the balance challenge in adversarial learning, the confused domain discriminator cannot well guarantee the alignment of feature distribution. Therefore, the distance measurement method MMD is incorporated into the adversarial learning framework. While confusing the domain discriminator, the statistical differences between the training library and the test library are aligned, which well bridges the balance challenge in adversarial learning and promotes knowledge transfer in the process of cross-library speech emotion recognition.

[0029] (4) The learning rate and batch size of the model are set to 0.001 and 16, and the network model is trained using the steepest gradient descent method. The model is iterated for 2000 rounds, and the classifier uses softmax.

[0030] (5) Combined with the source domain classification loss in the model , reconstruction loss of training and test libraries and Fighting Losses and MMD loss The model is trained by back-propagation and the network parameters are iteratively optimized to improve the cross-database speech emotion recognition performance. The recognition performance of the proposed method (CAEADA) is compared with other cross-database speech emotion recognition methods and domain adaptation methods on six tasks set on three benchmark speech emotion libraries. These methods are SVM (a non-deep classification method), TCA (transfer component analysis), DoSL (domain adaptive subspace learning), JDAR (joint distribution adaptive regression), DAAN (adversarial domain adaptation), and MDD (maximum density divergence domain adaptation). The unweighted average recall rate is used as the evaluation indicator.

[0031] (6) Experimental settings,To verify the effectiveness of the model, we selected three widely used emotion speech libraries in speech emotion recognition, EmoDB (B), eNTERFACE (E), and CASIA (C) as benchmark libraries, and selected their common emotion categories for experiments. Six groups of tasks were set as shown in the following table:

[0032]

[0033] (7) The experimental results are as follows. The overall recognition effect proves the effectiveness of the disclosed method.

[0034]

[0035] The scope of protection sought for the present invention is not limited to the description of this specific embodiment.

Claims

1. A cross-database speech emotion recognition method based on one-dimensional convolutional autoencoder and adversarial domain adaptation, comprising the following steps: (1) Speech preprocessing: Extract the fundamental frequency, Mel-frequency cepstrum and line spectrum of the original speech signal as the input of the feature processing network; (2) Feature processing: The one-dimensional statistical features extracted in step (1) are sent to a one-dimensional convolutional autoencoder. The features are first processed using a convolutional layer, and then restored using a transposed convolutional layer. The restored speech features and the original features are combined to construct a reconstruction error. The training library classification loss, convolution process, deconvolution process, and reconstruction loss are defined as follows: L cls =L ce (y p ,y s ) (1) where y p Predict the sentiment label for the network output, y s is the real emotion label, L ce is the cross entropy function, Where [X j1 , X j2 , ..., X js ] are adjacent statistical features, Y(j) is the result obtained after convolution, K and b are the weight and bias in the convolution process, respectively. Where X R and X are the reconstructed features and original features respectively; (3) Feature distribution alignment: The features output by the encoder in step (2) are fed into the domain discriminator of the adversarial domain adaptation module. The feature distributions of the training and test libraries are aligned by confusing the domain discriminator, and the statistical differences between the training and test libraries are simultaneously measured using MMD. The domain discriminator loss and MMD distance are as follows: Where F is the feature processing network, D is the domain discriminator, and d i is the real domain label of the test library, Where H is the regenerated Hilbert space, which is used to find the feature mapping function φ so that the feature distribution distance between the training library and the test library is minimized after mapping; (4) Model training: The one-dimensional statistical features obtained in step (1) are input into the feature processing network to obtain the classification loss L cls and the reconstruction loss L reloss , the features with strong representation ability extracted by the encoder in the convolutional autoencoder are sent to the domain discriminator to obtain the adversarial loss L adv , and use MMD to minimize the statistical difference, and get the MMD loss L MMD , combine these losses to optimize the model: L all =L cls +L adv +L MMD +L reloss_s +L reloss_t (7) L reloss_s is the reconstruction loss of the training library, L reloss_t Reconstruction loss for the training and testing libraries; (5) Repeat steps (2), (3), (4) to optimize the network model parameters; (6) After iteration, the optimal model is obtained, achieving improved cross-database speech emotion recognition performance.

Images