A domain adaptation cross-library speech emotion recognition method based on causal feature extraction

Abstract

This invention discloses a domain-adaptive cross-database speech emotion recognition method based on causal feature extraction. The method includes: (1) selecting training speech data and test speech data, wherein the training speech data contains multiple speech segments and their corresponding emotion category labels, and the test speech data contains only the speech to be classified; (2) processing speech segments using acoustic low-dimensional descriptors and expanding the obtained values ​​using statistical functions, and concatenating the expanded data into a vector as speech emotion features; (3) establishing a domain-adaptive model based on causal feature extraction, training the model using training speech emotion features and test speech emotion features, and obtaining a robust model that generalizes from training data to test data through domain adaptation and causal feature extraction; (4) performing backpropagation based on classification loss, domain adaptation loss and causal feature loss to optimize the model network parameters and classify the test data.

Description

Technical Field

[0001] This invention relates to speech emotion recognition methods, and more particularly to a domain-adaptive cross-database speech emotion recognition method based on causal feature extraction. Background Technology

[0002] Emotion recognition is an interdisciplinary research field encompassing psychology, biology, and computer science, playing a crucial role in human-computer interaction and artificial intelligence, and has attracted considerable attention from researchers in recent years. The ability of computers to understand human emotions is a vital step in the development of artificial intelligence. In early research on speech emotion recognition, various traditional classifiers, such as Gaussian Mixture Models (GMMs), Hidden Markov Models (HMMs), Support Vector Machines (SVMs), and various hybrid models, were used to construct emotion recognition models and achieved good recognition performance. For cross-database speech emotion recognition, importance-weighted SVMs have also been proposed for emotion feature classification to mitigate the problem of different distributions between training and test sets.

[0003] Compared to the shallow structure of traditional machine learning, deep learning methods, due to their deep network structures and large number of parameters, achieve better performance in mining more effective feature representations and simulating more complex data distributions. Furthermore, the flexibility of deep learning methods allows researchers to create feature sets for input and perform end-to-end learning directly using raw speech. Using Convolutional Neural Networks (CNNs) to learn from spectrograms or using Recurrent Neural Networks (RNNs) to mine sentiment information from frame-level features have yielded superior performance. However, these experiments are based on training and test data being identically distributed. In real-world applications, training and test data are highly likely to come from different cultures, environments, or speakers. Therefore, the generalization ability of the model is difficult to guarantee, which is a key issue limiting the application of speech emotion recognition in many fields.

[0004] Researchers have proposed numerous domain adaptation algorithms to address the issue of differing distributions between training and test data. These include representation learning based on autoencoder architectures, domain adversarial approaches to mitigate domain discrepancies, and feature distribution alignment algorithms based on statistical metrics. Among these, the maximum mean difference algorithm is widely used for feature distribution measurement. However, simply aligning feature distributions to mitigate domain discrepancies cannot effectively extract domain-invariant representations. Therefore, this invention focuses on how to find key sentiment features during feature extraction while aligning features. Primarily, based on deep learning for feature representation and the use of maximum mean difference for measurement, a causal metric matrix is ​​constructed to model the correlation and independence of various dimensions within the features, thereby training a better feature processing network to improve the model's sentiment recognition performance. Summary of the Invention

[0005] Purpose of the invention: This paper addresses the problem that existing models cannot effectively generalize to test data. It constructs a domain-adaptive cross-database speech emotion recognition model based on causal feature extraction. While aligning feature distributions, it models the correlation and independence of features to achieve a high-quality domain adaptation process.

[0006] Technical solution: The domain-adaptive cross-library speech emotion recognition method based on causal feature extraction described in this invention includes:

[0007] (1) Two speech databases are used as training speech database and test speech database respectively. The training speech database contains multiple speech segments and their corresponding emotion category labels, while the test database contains only the speech data to be recognized.

[0008] (2) Use acoustic low-dimensional descriptors to process speech segments and use statistical functions to expand the obtained values. Then, concatenate the expanded data into a vector as speech emotion features.

[0009] (3) Establish a domain adaptive model based on causal feature extraction, train the model using training speech emotion features and test speech emotion features, and obtain a robust model that generalizes from training data to test data through domain adaptation and causal feature extraction. Backpropagate the model loss and optimize the network parameters.

[0010] (4) Repeat step (3) until the training reaches the preset number of rounds, complete the model optimization, and perform emotion classification on the training speech data.

[0011] Furthermore, step (2) specifically includes:

[0012] (2-1) For training and testing speech data, calculate the values ​​of its acoustic low-dimensional descriptors, including zero-crossing rate, root mean square frame energy, gene frequency, and Mel-spectral frequency cepstral coefficients;

[0013] (2-2) The values ​​of the acoustic low-dimensional descriptors calculated in (2-1) are expanded by applying statistical functions, including the mean, standard deviation, maximum value and skewness.

[0014] Furthermore, step (3) specifically includes:

[0015] (3-1) The overall loss design of the domain adaptive model based on causal feature extraction is as follows:

[0016] L = L cls +L MMD +L cau

[0017] In the formula, L cls For the classification loss of the training data, L MMD L is the feature distribution alignment loss between the training and test data. cau For causal loss;

[0018] (3-2) Obtain the speech emotion features extracted in step (2), input them into a one-dimensional convolutional neural network for feature processing, and use the corresponding feature representations to complete the emotion classification in the source domain to guide network training. The source domain classification loss of the model is designed as follows:

[0019]

[0020] In the formula, B represents the batch size during training, and y ic The value can be 1 or 0. This represents the predicted probability that a sample belongs to the c-th sentiment type;

[0021] (3-3) The maximum mean discrepancy (MMD) is used to measure the feature distribution distance between the training and test data, and alignment is performed to adapt the model to the test data. The feature distribution distance loss is designed as follows:

[0022]

[0023] In the formula, H is the regenerated Hilbert space, φ is the eigenmap function, and x s and x t These are the features of the processed training and test data, respectively;

[0024] (3-4) The source domain features are deconvolved to obtain dimensions consistent with the original features. A correlation matrix is ​​constructed to measure the correlation of features in the same dimension before and after reconstruction, as well as the independence of features in different dimensions. The correlation matrix is ​​designed as follows:

[0025]

[0026] In the formula fo and f r Let represent the original features and the features reconstructed after convolution, respectively, and D represent the feature dimension. The correlation matrix is ​​optimized to an identity matrix I to obtain the causal relationship between features and labels. The causal loss is designed as follows:

[0027]

[0028] Furthermore, step (4) optimizes the network parameters, including:

[0029] (4-1) The optimizer is set to stochastic gradient descent (SGD), and the initial learning rate is set to 0.1-0.001;

[0030] (4-2) After calculating the loss function in each round, the parameters are updated according to the gradient of the loss function with respect to each parameter in the network, and the feature processing network is optimized. The update process is as follows:

[0031]

[0032] In the formula, W represents the network parameters, and η represents the learning rate. The gradient is the parameter.

[0033] Beneficial effects: Compared with the prior art, the significant advantage of this invention is that it models the statistical relationship between labels and features in the cross-database adaptive process, enhances the generalization of the model, and improves the recognition accuracy of the model. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the domain-adaptive cross-library speech emotion recognition method based on causal feature extraction provided by the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to specific embodiments.

[0036] (1) Two speech databases are used as training speech database and test speech database respectively. The training speech database contains multiple speech segments and their corresponding emotion category labels, while the test database contains only the speech data to be recognized.

[0037] In this embodiment, three widely used public datasets in speech emotion recognition, EmoDB(B), eNTERFACE(E), and CASIA(C), were selected as benchmark datasets. Based on the common number of emotion categories, six sets of experiments were set up, and the number of emotion categories and speech entries selected are shown in Table 1.

[0038] Table 1

[0039]

[0040] (2) Use acoustic low-dimensional descriptors to process speech segments and use statistical functions to expand the obtained values. Then, concatenate the expanded data into a vector as speech emotion features.

[0041] Step (2) specifically includes:

[0042] (2-1) For training and testing speech data, calculate the values ​​of its acoustic low-dimensional descriptors, including zero-crossing rate, root mean square frame energy, gene frequency, and Mel-spectral frequency cepstral coefficients;

[0043] (2-2) The values ​​of the acoustic low-dimensional descriptors calculated in (2-1) are expanded by applying statistical functions, including the mean, standard deviation, maximum value and skewness.

[0044] (3) Establish a domain adaptive model based on causal feature extraction;

[0045] Step (3) specifically includes:

[0046] (3-1) The model is trained using training speech emotion features and test speech emotion features. Through domain adaptation and causal feature extraction, a robust model that generalizes from training data to test data is obtained. The model loss is backpropagated to optimize the network parameters. The model loss is:

[0047] L = L cls +L MMD +L cau

[0048] In the formula, L cls For the classification loss of the training data, L MMD L is the feature distribution alignment loss between the training and test data. cau For causal loss;

[0049] (3-2) Obtain the speech emotion features extracted in step (2), input them into a one-dimensional convolutional neural network for feature processing, and use the corresponding feature representations to complete the emotion classification in the source domain to guide network training. The source domain classification loss of the model can be defined as follows:

[0050]

[0051] In the formula, B represents the batch size during training, and y ic The value can be 1 or 0. This represents the predicted probability that a sample belongs to the c-th sentiment type;

[0052] (3-3) The maximum mean discrepancy (MMD) is used to measure the feature distribution distance between the training and test data, and alignment is performed to adapt the model to the test data. The feature distribution distance loss is designed as follows:

[0053]

[0054] In the formula, H is the regenerated Hilbert space, φ is the eigenmap function, and x s and x t These are the features of the processed training and test data, respectively;

[0055] (3-4) Feature Causality Extraction: During steps (3-1) and (3-2), the feature representation ability extracted by the network is key to improving the model's generalization performance. In order to extract the causal relationship between features and the true labels during training, the source domain features are deconvolved to obtain the same dimensions as the original features. A correlation matrix is ​​constructed to measure the correlation of features in the same dimension before and after reconstruction, as well as the independence of features in different dimensions. The correlation matrix is ​​as follows:

[0056]

[0057] In the formula f o and f r Let represent the original features and the features reconstructed after convolution, respectively, and D represent the feature dimension. Optimizing the correlation matrix into an identity matrix I allows us to obtain the causal relationship between features and labels. The causal loss can be defined as:

[0058]

[0059] Step (3-2) uses a one-dimensional convolutional neural network to process features, specifically including:

[0060] (3-2-1) A one-dimensional convolutional neural network setup includes four convolutional modules. Each module consists of a one-dimensional convolutional layer (Conv1D layer), a batch normalization layer, and an activation layer. The parameter settings for the four one-dimensional convolutional layers are {16×9×2, 32×9×2, 64×9×2, 128×9×2} (number of convolutional kernels × kernel size × stride). After inputting sentiment features into the one-dimensional convolutional neural network, the output features obtained are:

[0061]

[0062] Where [X] j1 ,Xj2 ...X js [Statistical characteristics of adjacent pairs] ， Y(j) is the result obtained after the convolution operation. ， W and b are the weights and biases in the convolution process, respectively;

[0063] (3-2-2) After processing the features, the feature dimensions output by the convolutional neural network are mapped to the number of sentiment categories using a fully connected layer, and then processed using the Softmax function to perform the classification loss calculation in step (3-1).

[0064] (4) Repeat step (3) until the training reaches the preset number of rounds, complete the model optimization, and perform emotion classification on the training speech data.

[0065] Furthermore, step (4) optimizes the network parameters, including:

[0066] (4-1) The optimizer is set to stochastic gradient descent (SGD), and the initial learning rate is set to 0.1-0.001;

[0067] (4-2) After calculating the loss function in each round, update the parameters based on the gradient of the loss function with respect to each parameter in the network, and optimize the feature processing network:

[0068] (5) To verify the effectiveness of the present invention, based on the six different cross-database recognition schemes in Table 1, several other classic and high-performance algorithms in cross-database speech emotion recognition were selected, namely: SVM, TCA, DoSL, JDAR, and MDD. The experimental results were compared, as shown in Table 2:

[0069] (6) The experimental results of the disclosed algorithm are compared with those of other algorithms. The results are evaluated using the accuracy (WAR). The experimental results show that the speech emotion recognition algorithm proposed in this invention has achieved high recognition performance.

[0070] Table 2

[0071]

Claims

1. A domain-adaptive cross-library speech emotion recognition method based on causal feature extraction, characterized in that... The method includes: (1) Two speech databases are used as training speech database and test speech database respectively. The training speech database contains multiple speech segments and their corresponding emotion category labels, while the test database contains only the speech data to be recognized. (2) Use acoustic low-dimensional descriptors to process speech segments, and use statistical functions to expand the obtained values. Then, concatenate the expanded data into a vector as speech emotion features. (3) Establish a domain adaptive model based on causal feature extraction, use a one-dimensional convolutional neural network for feature processing, measure the distribution distance between training speech emotion features and test speech emotion features by the maximum mean difference metric and align them, extract causal features by deconvolution to reconstruct features, construct a correlation matrix and optimize it into an identity matrix; train the model using training speech emotion features and test speech emotion features, obtain a robust model through domain adaptation and causal feature extraction, and optimize network parameters based on overall loss backpropagation. The specific steps are as follows: The overall loss design of the domain adaptive model based on causal feature extraction is as follows: In the formula, The classification loss for the training data, The loss is the feature distribution alignment loss between the training and test data. For causal loss; The speech emotion features extracted in step (2) are obtained and input into a one-dimensional convolutional neural network for feature processing. The corresponding feature representations are used to complete the emotion classification in the source domain to guide network training. The source domain classification loss of the model is designed as follows: In the formula Indicates the batch size during the training process. The value can be 1 or 0. Indicates that the sample belongs to the first Predictive probability of sentiment type; The maximum mean difference is used to measure the feature distribution distance between the training and test data, and alignment is performed to adapt the model to the test data. The feature distribution distance loss is designed as follows: In the formula To regenerate Hilbert space, For feature mapping function, and These are the features of the processed training and test data, respectively; The source domain features are deconvolved to obtain dimensions consistent with the original features. A correlation matrix is ​​constructed to measure the correlation of features in the same dimension before and after reconstruction, as well as the independence of features in different dimensions. The correlation matrix is ​​designed as follows: In the formula and These represent the original features and the features reconstructed by convolution, respectively. Representing the feature dimension. Optimizing the correlation matrix into an identity matrix. To obtain the causal relationship between features and labels, the causal loss is designed as follows: (4) Repeat step (3) until the training reaches the preset number of rounds, complete the model optimization, and perform emotion classification on the training speech data.

2. The domain-adaptive cross-library speech emotion recognition method based on causal feature extraction according to claim 1, characterized in that: Step (2) specifically includes: (2-1) For training and testing speech data, calculate the values ​​of its acoustic low-dimensional descriptors, including zero-crossing rate, root mean square frame energy, gene frequency, and Mel-spectral frequency cepstral coefficients; (2-2) The values ​​of the acoustic low-dimensional descriptors calculated in (2-1) are expanded by applying statistical functions, including the mean, standard deviation, maximum value and skewness.

3. The domain-adaptive cross-library speech emotion recognition method based on causal feature extraction according to claim 1, characterized in that: Step (4) Optimizing network parameters includes: (3-1) The optimizer is set to stochastic gradient descent, and the initial learning rate is set to 0.1-0.001; (3-2) After calculating the loss function in each round, the parameters are updated according to the gradient of the loss function on each parameter in the network, and the feature processing network is optimized. The update process is as follows: In the formula For network parameters, For learning rate, For the gradient of the parameters.

Images