Cross-library speech emotion recognition method and device based on PDAN

Abstract

The application discloses a cross-library speech emotion recognition method and device based on progressive distribution adapted neural networks (PDAN), which comprises the following steps: (1) obtaining two speech emotion databases as a training database and a test database respectively; (2) extracting speech samples with the same emotion label in the training database and the test database as source domain samples and target domain samples respectively, and extracting the spectrogram features of the samples; (3) establishing a cross-library speech emotion recognition model based on the progressive distribution adapted neural network, wherein the model takes a convolutional neural network as a backbone network, adds a progressive distribution adapted regularization term in an original classification loss function to guide network training, and thus eliminates the distribution difference between different fields; (4) training the above model by using the spectrogram features of the source domain and the target domain and the emotion label of the source domain; and (5) extracting the spectrogram features of a speech to be recognized, inputting the trained model, and recognizing the emotion category. The application has higher recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to speech emotion recognition technology, and particularly to a cross-corpus speech emotion recognition method and device based on progressive distribution adaptation neural network BACKGROUND

[0002] Speech is a major means of communication in human daily life, which contains rich emotional information. Imagine if computers could understand the emotional state from human speech signals, human-computer interaction would undoubtedly be more natural. Therefore, the research on automatically recognizing emotional states from speech signals, such as speech emotion recognition (SER), has attracted widespread attention in the fields of affective computing, human-computer interaction and speech signal processing. In the past few decades, many well-performing SER methods have been proposed and achieved satisfactory performance on widely used publicly available speech emotion databases. However, it is worth noting that most of them do not consider the realistic scenario that training and testing speech signals may be recorded by different identical microphones or in different environments. In this case, there may be a feature distribution mismatch between training and testing speech samples, so the performance of these originally well-performing SER methods may drop sharply, which brings a meaningful and more challenging task in SER, namely cross-corpus SER. Unlike traditional SER, the labeled training and unlabeled testing samples in cross-corpus SER come from different speech database libraries. Following the naming convention in cross-corpus SER, the training and testing samples / databases / feature sets will be referred to as source set and target set in this document. SUMMARY

[0003] The present application aims at the problems existing in the prior art, and provides a cross-corpus speech emotion recognition method based on progressive distribution adaptation neural network with higher recognition accuracy.

[0004] Technical scheme: The cross-corpus speech emotion recognition method based on progressive distribution adaptation neural network provided by the present application comprises:

[0005] (1) Obtain two speech emotion databases as training database and testing database, respectively;

[0006] (2) Extract speech samples with the same emotional label in the training database and the testing database as source domain samples and target domain samples, respectively, and extract the spectrogram features of the source domain samples and the target domain samples;

[0007] (3) Establish a cross-corpus speech emotion recognition model based on progressive distribution adaptation neural network, which takes a convolutional neural network as a backbone network, and guides network training by adding a progressive distribution adaptation regularization term in the original classification loss function, so as to eliminate the distribution difference between different domains;

[0008] (4) training the above model using the spectrogram features of the source domain and the target domain, and the emotion labels of the source domain;

[0009] (5) for the speech to be recognized, extracting the spectrogram features, inputting the trained model, and obtaining the emotion category of the speech to be recognized.

[0010] Further, step (2) specifically comprises:

[0011] (2-1) extracting speech samples with the same emotion labels in the training database and the test database as source domain samples and target domain samples, respectively;

[0012] (2-2) sequentially performing frame division and windowing preprocessing on the source domain samples and the target domain samples;

[0013] (2-3) performing short-time discrete Fourier transform on the preprocessed samples to obtain spectrogram features.

[0014] Further, the cross-library speech emotion recognition model based on the progressive distribution adaptation neural network specifically comprises:

[0015] a convolutional neural network comprising a plurality of layers of convolutional layers, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence;

[0016] a loss calculation module comprising a first loss calculation unit for calculating an emotion recognition loss, a second loss calculation unit for calculating an edge distribution difference loss between source domain features and target domain features, a third loss calculation unit for calculating a fine-grained emotion label information guided conditional distribution adaptation loss, a fourth loss calculation unit for calculating a coarse emotion category perception conditional distribution adaptation regularization term, and a total loss calculation unit for calculating the sum of the four losses.

[0017] Further, the first loss calculation unit is connected to the third fully connected layer and is specifically configured to calculate the emotion recognition loss based on the following formula:

[0018]

[0019] In the formula, is the emotion recognition loss, is the cross-entropy loss connected to the spectrogram features of the source domain samples and their corresponding emotion labels, N s represents the number of source domain samples, g1, g2, and g3 represent the parameters of the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively, represents the spectrogram features of the i-th source domain sample, f represents the parameters of the convolutional neural network, represents the emotion label of the i-th source domain sample.

[0020] Further, the second loss calculation unit is connected to the first fully connected layer, and is configured to calculate an edge distribution difference loss based on the following formula:

[0021]

[0022] In the formula, is an edge distribution loss, N s represents the number of source domain samples, N t represents the number of target domain samples, g1 represents the first fully connected layer parameter, represents the i-th source domain sample spectrogram feature, represents the i-th target domain sample spectrogram feature, f represents the parameter of the convolutional neural network, and Φ(·) is a kernel mapping operator, represents a reproducing kernel Hilbert space.

[0023] Further, the third loss calculation unit is connected to the third fully connected layer, and is configured to calculate a conditional distribution adaptation loss based on the following formula:

[0024]

[0025] In the formula, represents a conditional distribution adaptation loss, and respectively represent the i-th sample belonging to the j-th emotion class in the source domain and the target domain, and C represents the number of emotion classes, and respectively represent the number of samples belonging to the j-th emotion class in the source domain and the target domain, and satisfy and N s represents the number of source domain samples, N t represents the number of target domain samples, f represents the parameter of the convolutional neural network, g1, g2, and g3 respectively represent the parameters of the first fully connected layer, the second fully connected layer, and the third fully connected layer, represents a reproducing kernel Hilbert space.

[0026] Further, the fourth loss calculation unit is connected to the second fully connected layer, and is configured to calculate an emotion class perception conditional distribution adaptation regularization term based on the following formula:

[0027]

[0028] In the formula, C r is the number of coarse emotion classes, C r is less than the number of emotion classes C, and respectively represent the i-th sample belonging to the j-th emotion class in the source domain and the target domain, and Φ(·) is a kernel mapping operator, and respectively represent the number of samples belonging to the jth emotion in the source domain and the target domain, and satisfy and N s represents the number of source domain samples, N t represents the number of target domain samples, f represents the parameters of the convolutional neural network, g1 and g2 respectively represent the parameters of the first full connection layer and the second full connection layer, represent the reproducing kernel Hilbert space.

[0029] Further, the total loss calculation unit is configured to calculate the total loss according to the following formula:

[0030]

[0031] In the formula, is the total loss, λ1, λ2 and λ3 are weighting coefficients that make minimum.

[0032] Further, step (4) specifically comprises:

[0033] (4-1) randomly initializing the parameters of the cross-library speech emotion recognition model;

[0034] (4-2) predicting the pseudo emotion label of the target domain sample

[0035] (4-3) calculating the total loss

[0036] (4-4) updating the pseudo emotion label using the classic stochastic gradient descent optimization algorithm according to the total loss

[0037] (4-5) judging whether the total loss converges, if not, updating the pseudo emotion label according to the current parameters of the cross-library speech emotion recognition model, and returning to step (4-3), if the total loss converges, the network training is completed.

[0038] The cross-library speech emotion recognition device based on the progressive distribution adaptation neural network comprises a processor and a computer program stored on a memory and executable on the processor, and the processor implements the above method when executing the program.

[0039] Advantages: Compared with the prior art, the present application has the following advantages:

[0040] ​1. The application provides a new end-to-end deep transfer learning model, called Progressive Distribution Adapted Neural Networks (PDAN), to cope with cross-corpus SER tasks. Unlike most existing methods, PDAN can utilize the deep neural network and the nonlinear mapping ability of the speech spectrum to directly learn cross-corpus invariant and emotion discriminative speech features from the original speech signal.

[0041] 2. The application gradually adapts the feature distribution between source speech samples and target speech samples by adding three different MMD loss functions to different fully connected layers.

[0042] 3. The application has higher accuracy. DETAILED DESCRIPTION

[0043] Figure 1 is a flowchart of an embodiment of the cross-corpus speech emotion recognition method based on the progressive distribution adapted neural network provided by the application;

[0044] Figure 2 is a structural diagram of the progressive distribution adapted neural network. DETAILED DESCRIPTION

[0045] The embodiment provides a cross-corpus speech emotion recognition method based on a progressive distribution adapted neural network, as shown in Figure 1 , which comprises the following steps:

[0046] (1) Obtain two speech emotion databases as a training database and a test database, respectively.

[0047] (2) Extract speech samples with the same emotion label in the training database and the test database as source domain samples and target domain samples, respectively, and extract the spectrogram features of the source domain samples and the target domain samples.

[0048] This step comprises the following steps:

[0049] (2-1) Extract speech samples with the same emotion label in the training database and the test database as source domain samples and target domain samples, respectively.

[0050] (2-2) Frame and window the source domain samples and the target domain samples for preprocessing in sequence.

[0051] (2-3) Perform short-time discrete Fourier transform on the preprocessed samples to obtain spectrogram features.

[0052] (3) a cross-library speech emotion recognition model based on a progressive distribution adaptation neural network is established, the model takes a convolutional neural network as a backbone network, a progressive distribution adaptation regularization term is added in an original classification loss function to guide network training, and thus distribution differences between different fields are eliminated.

[0053] The cross-library speech emotion recognition model based on the progressive distribution adaptation neural network established by the application is as shown in Figure 2 The cross-library speech emotion recognition model based on the progressive distribution adaptation neural network established by the application is as shown in The loss calculation module includes a first loss calculation unit for calculating an emotion recognition loss, a second loss calculation unit for calculating an edge distribution difference loss between source domain features and target domain features, a third loss calculation unit for calculating a fine-grained emotion label information guided conditional distribution adaptation loss, a fourth loss calculation unit for calculating a coarse emotion class perception conditional distribution adaptation regularization term, and a total loss calculation unit for calculating a sum of the four losses.

[0054] The first loss calculation unit is connected with the third full connection layer and is specifically used for calculating the emotion recognition loss based on the following formula, which is used to realize the target of making the network have emotion recognition:

[0055]

[0056] In the formula, is the emotion recognition loss, is a cross-entropy loss connected with a source domain sample spectrogram feature and a corresponding emotion label, N s represents a source domain sample number, g1, g2 and g3 represent parameters of the first full connection layer, the second full connection layer and the third full connection layer respectively, represents a spectrogram feature of an i-th source domain sample, f represents a parameter of the convolutional neural network, represents an emotion label of the i-th source domain sample, for y i If a label of a corresponding speech sample is a j-th emotion, only the j-th item is set to 1, and other items are set to 0.

[0057] The second loss calculation unit is connected with the first full connection layer and is used for calculating the edge distribution difference loss based on the following formula:

[0058]

[0059] In the formula, is the edge distribution loss, N s represents a source domain sample number, N t represents a target domain sample number, g1 represents a first full connection layer parameter, denotes the spectrogram feature of the i-th source domain sample, denotes the spectrogram feature of the i-th target domain sample, f denotes the parameters of the convolutional neural network, and Φ(·) is a kernel mapping operator, denotes a reproducing kernel Hilbert space (RKHS).

[0060] The third loss calculation unit is connected to the third fully connected layer, and is configured to calculate a conditional distribution adaptation loss based on the following formula:

[0061]

[0062] In the formula, denotes the conditional distribution adaptation loss, and denote the i-th sample belonging to the j-th emotion class in the source domain and the target domain respectively, and C denotes the number of emotion classes, and denote the number of samples belonging to the j-th emotion class in the source domain and the target domain respectively, and satisfy and N s denotes the number of source domain samples, N t denotes the number of target domain samples, f denotes the parameters of the convolutional neural network, g1, g2, and g3 denote the parameters of the first fully connected layer, the second fully connected layer, and the third fully connected layer respectively, denotes a reproducing kernel Hilbert space (RKHS).

[0063] The fourth loss calculation unit is connected to the second fully connected layer, and is configured to calculate an emotion class perception conditional distribution adaptation regularization term based on the following formula:

[0064]

[0065] In the formula, C r is the number of coarse emotion classes, C r is less than the number of emotion classes C, and denote the i-th sample belonging to the j-th emotion class in the source domain and the target domain respectively, and Φ(·) is a kernel mapping operator, and denote the number of samples belonging to the j-th emotion class in the source domain and the target domain respectively, and satisfy and N s denotes the number of source domain samples, N t denotes the number of target domain samples, f denotes the parameters of the convolutional neural network, g1, g2 denote the parameters of the first fully connected layer and the second fully connected layer respectively, denotes a reproducing kernel Hilbert space (RKHS).

[0066] look like an upgrade, but they are actually very different. Specifically, is to divide them into high valence group and low valence group according to their distribution along valence dimension in the valence-arousal emotion wheel, and then compute the conditional MMD according to the new grouping. The design The main reason for the design is that fine-grained emotions are not well distinguishable along the arousal dimension, because most of the existing typical emotions are high arousal, only a few emotions, such as "sadness", are low arousal. In other words, since most of these emotions are difficult to distinguish, it can be a difficult task to directly align the fine-grained emotion class perception conditional distribution gap between the source and target speech features in the last fully connected layer. However, interestingly, along the valence dimension, the separability between these emotions will significantly improve. Because according to the grouping, "anger", "disgust", and "fear" belong to the low valence group, while "surprise" and "happy" belong to the high valence group, even though they all belong to the high arousal group along the arousal dimension. For this reason, the coarse-grained emotion information guided conditional distribution (i.e., alignment along the valence dimension) is applied in the second fully connected layer, so the design is to facilitate the reduction of inter-domain differences. According to the complexity of the emotion class information used in the computation of the feature distribution adaptation, it can be seen that the feature distribution adaptation operation in the model proposed in the present application presents a gradual way, so it is called a gradual distribution adaptation neural network.

[0067] and mainly aims to eliminate the inter-domain differences of the model. The three loss functions are all based on the computation of the maximum mean discrepancy (MMD). The specific formula for computing MMD is as follows:

[0068]

[0069] where k(·) is a kernel function, which replaces the inner product operation between vectors in the RKHS generated by Φ(·) with the computation of a pre-defined function. and represent the i-th column in X s , X t .

[0070] The total loss calculation unit is configured to calculate the total loss according to the following formula:

[0071]

[0072] In the formula, For the total loss, λ1, λ2, and λ3 are the weighting coefficients that are set to make the minimum trade-off.

[0073] (4) Training the above model using the source domain and target domain spectrogram features, and the emotion label of the source domain.

[0074] This step specifically includes:

[0075] (4-1) Randomly initializing the parameters (i.e., f, g1, g2, g3) of the cross-library speech emotion recognition model;

[0076] (4-2) Predicting the pseudo emotion label of the target domain sample

[0077] (4-3) According to calculating the total loss

[0078] (4-4) According to the total loss updating the parameters of the cross-library speech emotion recognition model using the classic Stochastic Gradient Descent (SGD) algorithm;

[0079] (4-5) Judging whether the total loss converges, if not, updating the pseudo emotion label according to the current parameters of the cross-library speech emotion recognition model, and returning to step (4-3), if it converges, the network training is completed.

[0080] (5) For the speech to be recognized, extract the spectrogram feature, input it into the trained model, and obtain the emotion category of the speech to be recognized.

[0081] The embodiment also provides a cross-library speech emotion recognition device based on a progressive distribution adaptation neural network, which comprises a processor and a computer program stored on a memory and capable of running on the processor, and the processor implements the above method when executing the program.

[0082] To evaluate the performance of the proposed PDAN model in the cross-corpus speech emotion recognition task, a large number of experiments were conducted based on the EMO-DB, CASIA, and eNTERFACE three databases. Specifically, any two of the above speech emotion databases were used as the source and target sample set, thereby obtaining 6 typical cross-corpus SER tasks, as shown in Table 1, wherein B, E, C are the abbreviations of EmoDB, eNTERFACE, CASIA, respectively, and the left and right databases of the arrow correspond to the source domain database and the target domain database, respectively. In addition, since these databases have different emotions, in each cross-corpus SER task, speech samples with the same emotion label are extracted to ensure the consistency of the label. The detailed sample statistics of the selected databases are shown in Table 1. In addition, for the constraint term of the conditional probability distribution adaptation based on the high and low valence dimension information guidance in the method, it is still necessary to redivide the discrete emotion labels in the database according to the emotion valence dimension in the valence-arousal emotion wheel model, see Table 2 for details. In the experiment, in order to highlight the generality of the method design, a variety of deep learning-based domain adaptation methods are selected as a comparison, including DAN (Deep Adaptation Networks), DANN (Domain-Adversarial Neutral Network), Deep-CORAL, DSAN (Deep Subdomain Adaptation Network) and other methods, and AlexNet is selected as their backbone network. In order to simultaneously reflect the superiority of the deep neural network compared with the traditional method, SVM is also used as a benchmark method here, and a series of excellent and classic domain adaptation methods are selected for comparison, including Transfer Component Analysis (TCA), Subspace Alignment (SA), Domain Adaptive Subspace Learning (DoSL), Geodesic Flow Kernel (GFK), and Joint Distribution Adaptive Regression (JDAR). Please note that the feature set used by the traditional method is INTERSPEECH 2009 Emotion Challenge and INTERSPEECH 2010 Paralinguistic Challenge. As for the evaluation index of the experimental results, the unweighted average recall (UAR) is used as the evaluation standard in the experiment. All the verification results are shown in Table 3.

[0083] Table 1

[0084]

[0085] Table 2

[0086]

[0087] Table 3

[0088]

[0089] The experimental results show that the speech emotion recognition method based on the application achieves a high cross-lingual speech emotion recognition rate.

[0090] The above only discloses a preferred embodiment of the application, and cannot limit the scope of the application. Any equivalent changes made according to the claims of the application are still within the scope of the application.

Claims

1. A cross-library speech emotion recognition method based on a progressively distributed adaptive neural network, characterized in that... include: (1) Obtain two speech emotion databases, which will be used as the training database and the test database respectively; (2) Extract speech samples with the same emotion label from the training database and the test database, and use them as source domain samples and target domain samples respectively, and extract the spectrogram features of the source domain samples and target domain samples. (3) Establish a cross-database speech emotion recognition model based on a progressively distributed adaptive neural network. This model uses a convolutional neural network as the backbone network and guides network training by adding a progressively distributed adaptive regularization term to the original classification loss function, thereby eliminating distribution differences between different domains. Specifically, the cross-database speech emotion recognition model based on a progressively distributed adaptive neural network includes: A convolutional neural network comprises several convolutional layers, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The loss calculation module includes a first loss calculation unit for calculating emotion discrimination loss, a second loss calculation unit for calculating the marginal distribution difference loss between source domain features and target domain features, a third loss calculation unit for calculating the conditional distribution adaptation loss guided by fine-grained emotion label information, a fourth loss calculation unit for calculating the coarse emotion-class perception conditional distribution adaptation regularization term, and a total loss calculation unit for calculating the sum of the four losses; the progressive distribution adaptation regularization term includes marginal distribution difference loss, conditional distribution adaptation loss, and conditional distribution adaptation regularization term; (4) The above model is trained using spectrogram features of the source and target domains, as well as sentiment labels of the source domain; (5) For the speech to be identified, extract spectrogram features, input them into the trained model, and obtain the emotion category of the speech to be identified.

2. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: Step (2) specifically includes: (2-1) Extract speech samples with the same emotion labels from the training database and the test database, and use them as source domain samples and target domain samples, respectively. (2-2) Perform frame segmentation and windowing preprocessing on the source domain samples and the target domain samples in sequence; (2-3) Perform short-time discrete Fourier transform on the preprocessed samples to obtain spectrogram features.

3. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: The first loss calculation unit is connected to the third fully connected layer and is specifically used to calculate the sentiment discrimination loss based on the following formula: In the formula, It is a loss of emotional discrimination. It is the cross-entropy loss that connects the spectrogram features of the source domain samples and their corresponding sentiment labels, N s The number of samples in the source domain is represented by g1, g2, and g3, which represent the parameters of the first, second, and third fully connected layers, respectively. Let f represent the spectrogram features of the i-th source domain sample, and let f represent the parameters of the convolutional neural network. Let represent the sentiment label of the i-th source domain sample.

4. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: The second loss calculation unit is connected to the first fully connected layer and is used to calculate the edge distribution difference loss based on the following formula: In the formula, It is the marginal distribution loss, N s N represents the number of samples in the source domain. t This represents the number of samples in the target domain, and g1 represents the parameters of the first fully connected layer. This represents the spectrogram features of the i-th source domain sample. Let f represent the spectrogram features of the i-th target domain sample, f represent the parameters of the convolutional neural network, and Φ(·) be the kernel mapping operator. Represents the regenerating nucleus Hilbert space.

5. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: The third loss calculation unit is connected to the third fully connected layer and is used to calculate the conditional distribution adaptation loss based on the following formula: In the formula, This represents the conditional distribution adaptation loss. and Let C represent the i-th sample belonging to the j-th sentiment category in the source and target domains, respectively. and Let represent the number of sentiment samples belonging to the j-th class in the source domain and the target domain, respectively, and satisfy . and N s N represents the number of samples in the source domain. t denoted by , where f represents the number of samples in the target domain, and g1, g2, and g3 represent the parameters of the first, second, and third fully connected layers, respectively. Let Φ represent the regenerated kernel Hilbert space, and Φ(·) be the kernel mapping operator.

6. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: The fourth loss calculation unit is connected to the second fully connected layer and is used to calculate the adaptive regularization term of the emotional perception conditional distribution based on the following formula: In the formula, For the conditional distribution adaptation regularization term of emotion perception, C r It is the number of coarse emotion categories, C r Less than the number of emotion categories C, and Let Φ(·) represent the source and target domains, respectively, representing the i-th sample belonging to the j-th sentiment class. and Let represent the number of sentiment samples belonging to the j-th class in the source domain and the target domain, respectively, and satisfy . and N s N represents the number of samples in the source domain. t denoted by , where f represents the number of samples in the target domain, and g1 and g2 represent the parameters of the first and second fully connected layers, respectively. Represents the regenerating nucleus Hilbert space.

7. The cross-library speech emotion recognition method based on a progressively distributed adaptive neural network according to claim 1, characterized in that: The total loss calculation unit is used to calculate the total loss according to the following formula: In the formula, For the total loss, λ1, λ2, and λ3 are such that... The smallest trade-off factor, It is a loss of emotional discrimination. It is a marginal distribution loss. This represents the conditional distribution adaptation loss. This is a regularization term for the conditional distribution of emotion perception.

8. The cross-library speech emotion recognition method based on progressively distributed adaptive neural networks according to claim 1, characterized in that: Step (4) specifically includes: (4-1) Randomly initialize the parameters of the cross-database speech emotion recognition model; (4-2) Predicting pseudo-sentiment labels for target domain samples (4-3) According to Calculate total loss (4-4) Based on the total loss The parameters of the cross-database speech emotion recognition model are updated using the classic stochastic gradient descent optimization algorithm. (4-5) Determine if the total loss has converged. If it has not converged, update the pseudo-emotion labels based on the parameters of the current cross-database speech emotion recognition model. Then return to step (4-3). If convergence is achieved, the network training is complete.

9. A cross-library speech emotion recognition device based on a progressively distributed adaptive neural network, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-8.

Images