Cross-database speech emotion recognition method and device based on CDAR model

Abstract

The present invention discloses a cross-database speech emotion recognition method and device based on a CDAR model, the method comprising: (1) obtaining two speech emotion databases, respectively serving as a training database and a test database; (2) extracting speech samples with the same emotion category label from the training database and the test database, respectively serving as source domain samples and target domain samples, and extracting IS09 features of the source domain samples and the target domain samples as sample features; (3) establishing a collaborative distribution adaptive regression CDAR model; (4) solving the CDAR model using an alternating direction method and an inexact augmented Lagrange multiplier to obtain an optimal value of a projection matrix U; (5) extracting IS09 features for the speech to be recognized, and using the learned projection to obtain the corresponding emotion category. The present invention has a higher recognition accuracy.

Description

Technical Field

[0001] The present invention relates to speech emotion recognition technology, and in particular to a cross-database speech emotion recognition method and device based on a CDAR model. Background Art

[0002] Speech is one of the most natural behaviors for humans to communicate emotional information in their daily lives. Therefore, the study of Speech Emotion Recognition (SER) has received extensive attention in the fields of affective computing, pattern recognition, and speech signal processing, which enables computers to automatically understand emotional states such as happiness, fear, and sadness from speech signals. In the past few decades, many well-performing SER methods have been proposed and have achieved good performance on widely used speech emotion databases. However, the existing SER methods are still far from being practically available. One of the main reasons is that these SER methods do not consider the scenario in practice where the training and test speech signals may come from different databases. In this case, due to the differences in databases, such as different recording devices and environments, and different languages, there is inevitably a significant mismatch in feature distribution between the training speech signals and the test speech signals. Therefore, the performance of the original SER methods on a single speech database will drop significantly. This gives rise to a challenging SER task, namely cross-database SER. Formally speaking, in the cross-database SER task, the emotion labels of the training database samples are given, while the emotion labels of the target database samples are completely unavailable. It is necessary to learn a classifier from the given source label information to predict the emotion of unlabeled test speech samples. The recognition accuracy of existing technologies needs to be improved. Summary of the invention

[0003] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a cross-database speech emotion recognition method and device based on the CDAR model with higher recognition accuracy.

[0004] Technical solution: The cross-database speech emotion recognition method based on the CDAR model described in the present invention includes:

[0005] (1) Obtain two speech emotion databases, one as a training database and the other as a test database. The training database includes a number of speech samples and corresponding emotion category labels, and the test database includes only speech samples:

[0006] (2) Extract speech samples with the same emotion category labels from the training database and the test database as source domain samples and target domain samples, respectively, and extract IS09 features of the source domain samples and the target domain samples as sample features;

[0007] (3) Establish a CDAR model, specifically:

[0008]

[0009] Where U is the projection matrix to be learned, Represents the feature matrix formed by the sample features of all samples in the source domain, d is the dimension of the feature matrix, n s is the number of source domain samples, Represents the sentiment category label matrix of all samples in the source domain, Represents the feature matrix formed by the sample features of all samples in the target domain, n t is the number of samples in the target domain, λ and μ are weight parameters, c is the total number of emotion categories, and c r is the total number of coarse-grained sentiment categories, Represent the feature matrices of speech samples belonging to the i-th emotion in the source domain and the target domain respectively, Represents the feature matrix of the speech sample belonging to the jth class of coarse-grained emotion in the source domain and the target domain, 1 s ,1 t , and are all vectors with a value of 1 and dimensions of n s 、n t , are the number of speech samples belonging to the i-th emotion in the source domain and the target domain, respectively. Respectively represent the number of speech samples belonging to the jth coarse-grained emotion in the source domain and the target domain. Several emotion categories can be classified into one coarse-grained emotion category;

[0010] (4) Using the alternating direction method and inexact augmented Lagrange multipliers to solve the CDAR model, we can obtain the optimal value of the projection matrix U.

[0011] (5) For the speech to be recognized, extract the IS09 features and use the learned Projection obtains the corresponding emotion category.

[0012] Furthermore, step (4) specifically includes:

[0013] (4-1) Initialize the projection matrix U;

[0014] (4-2) According to the projection matrix U, the sentiment category labels of all target domain samples are predicted using the following formula;

[0015]

[0016] emo_label(k) represents the emotion category label of the kth target domain sample. for The i-th item of is the sample feature of the kth target domain sample;

[0017] (4-3) According to the predicted emotion category label, all samples in the target domain that belong to the same emotion category are divided into one category, and the sample features of each category form a feature vector Then, the target domain samples belonging to the same coarse-grained emotion category are divided into one category, and the sample features of each coarse-grained category form a feature vector

[0018] (4-4) According to the determined The original edge Δ is calculated as follows m and two perceived conditional feature distribution gaps

[0019]

[0020]

[0021]

[0022] (4-5) Convert the CDAR model into a constrained optimization problem:

[0023]

[0024] stP=U

[0025] Where, L = [Y s ,0] and is the intermediate variable, is a zero matrix,

[0026] (4-6) Solve the optimization problem by alternately minimizing the Lagrangian function of the variable and obtain the projection matrix value U of the current cycle l ;

[0027] (4-7) Determine whether ||U is satisfied l -U l-1 || F <∈,∈ represents the preset threshold, U l-1 Indicates the projection matrix value of the previous cycle, l indicates the number of iterations, if it is not satisfied, the current projection matrix value U l Return to step (4-2). If satisfied, set the current projection matrix value U l As the optimal value

[0028] Further, steps (4-6) specifically include:

[0029] (4-6-1) Convert the optimization problem into Lagrangian function:

[0030]

[0031] Where T is the multiplier matrix, κ is the trade-off parameter, and Tr[] represents the trace of the matrix;

[0032] (4-6-2) Fix U, T and κ, and update P according to the following formula

[0033]

[0034] Derived P = (2ZZ T +κI) -1 (κU-T-ZL T ), I represents the identity matrix;

[0035] (4-6-3) Fix P, T and κ, and update U:

[0036]

[0037] The solution is Get, where p i and T i are the i-th row of P and T respectively, otherwise u i =0;

[0038] (4-6-4) Update T and κ according to the following formula:

[0039] T=T+κ(PU),κ=min{ρκ,κ max}; (Formula 13)

[0040] In the formula, ρκ represents the updated k value, κ max represents the maximum κ value;

[0041] (4-6-5) Check whether ||PU|| F <∈, if not, return to execute step (4-6-2), if so, end the loop and output the current U value.

[0042] Furthermore, step (5) specifically includes:

[0043] According to the projection matrix U, the emotion category of the speech to be recognized is predicted by the following formula:

[0044]

[0045] emo_label(te) represents the emotion category of the speech to be recognized. for The i-th item of is the sample feature of the speech to be recognized.

[0046] The cross-database speech emotion recognition device based on the CDAR model described in the present invention includes a processor and a computer program stored in a memory and executable on the processor, and the above method is implemented when the processor executes the program.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0048] 1. This paper proposes a new transfer subspace learning method CDAR to handle cross-database SER tasks. The basic idea of ​​CDAR is very simple, that is, to learn emotional discriminant representations and cross-database invariant representations for both source and target speech samples, and the classifier learned based on labeled source speech samples can also be used to predict the emotion of the target speech signal.

[0049] 2. This paper proposes a regularization based on distribution adaptation (CDA) to identify the emotion of speech signals by guiding the learning of database invariant features. CDA synergistically aligns the conditional distribution guided by fine-grained emotion information and coarse-grained emotion information between the source speech sample and the target speech sample.

[0050] 3. A large number of experiments have proved the effectiveness and superiority of the algorithm in processing cross-database SER tasks, with higher recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of an embodiment of the cross-database speech emotion recognition method based on the CDAR model of the present invention. DETAILED DESCRIPTION

[0052] This embodiment provides a cross-database speech emotion recognition method based on the CDAR model. Figure 1 As shown, including:

[0053] (1) Obtain two speech emotion databases, which are used as a training database and a test database respectively. The training database includes a number of speech samples and corresponding emotion category labels, and the test database only includes speech samples.

[0054] (2) Speech samples with the same emotion category labels are extracted from the training database and the test database as source domain samples and target domain samples, respectively, and the IS09 features of the source domain samples and the target domain samples are extracted as sample features.

[0055] (3) Establish a Collaborative Distribution Adapted Regression (CDAR) model, specifically:

[0056]

[0057] Where U is the projection matrix to be learned, Represents the feature matrix formed by the sample features of all samples in the source domain, d is the dimension of the feature matrix, n s is the number of source domain samples, Represents the sentiment category label matrix of all samples in the source domain, Represents the feature matrix formed by the sample features of all samples in the target domain, n t is the number of samples in the target domain, λ and μ are weight parameters, c is the total number of emotion categories, and c r is the total number of coarse-grained sentiment categories, Represent the feature matrices of speech samples belonging to the i-th emotion in the source domain and the target domain respectively, Represents the feature matrix of the speech sample belonging to the jth class of coarse-grained emotion in the source domain and the target domain, 1 s , 1 t , and are all vectors with a value of 1 and dimensions of n s 、n t , are the number of speech samples belonging to the i-th emotion in the source domain and the target domain, respectively. They represent the number of speech samples belonging to the jth coarse-grained emotion in the source domain and the target domain respectively. Several emotion categories can be classified into one coarse-grained emotion category.

[0058] The idea of ​​establishing the CDAR model is as follows:

[0059] The basic idea of ​​CDAR is to establish a subspace learning model to learn emotional discriminant representation and cross-library invariant representation (i.e., eliminate inter-domain differences) for source and target speech samples. To achieve this goal, it is proposed to use the feature space guided by label information as this subspace, and then learn a projection matrix to establish the relationship between this subspace and the original feature space, which can be expressed as a simple linear regression problem.

[0060]

[0061] Where U is such a satisfactory projection matrix that can easily transform the speech sample from the original feature space to the label space.

[0062] In order to give the projection matrix U the ability to eliminate inter-domain differences, a regularization term is designed, which is composed of l 2，1 The norm and the combination of the edge of U, the conditional distribution guided by fine-grained sentiment information and the conditional distribution guided by coarse-grained sentiment information are expressed as

[0063]

[0064] Where λ1 is a trade-off parameter controlling the balance between loss functions. Note that the first term in the formula, namely ||U T || 2，1 , is about U 2，1 norm. Minimize ||U T || 2，1 This helps the proposed CDAR learn a row sparse projection matrix, which will suppress features that contribute less to emotion discrimination during model training and help distinguish speech features with different emotions.

[0065] In order to help the cross-database SER task eliminate the difference in the distribution of source and target speech features, this paper proposes a collaborative distribution adaptation regression model CDAR. According to the common "valence-arousal" emotion wheel, it is found that common basic emotions such as fear, disgust, anger, surprise and happiness belong to different emotion categories, but they all have high arousal. In other words, from the dimension of arousal, these five types of emotions have high similarity. However, using valence as a scale, the three emotions of fear, disgust and anger can be well distinguished from surprise and happiness. Inspired by this phenomenon, this chapter proposes to divide the emotion category into high valence group and low valence group according to valence, and consider the conditional distribution of adapting the edge distribution and the conditional distribution guided by the emotion label to eliminate the distribution difference of the source domain and the target domain speech feature set. Specifically, along the valence dimension, emotions are first divided into two mixed types, including high valence (surprise, happiness, excitement and neutrality) and low valence (anger, disgust, fear, frustration and sadness). Then, for a specific cross-library SER task, several emotional elements involved can be selected, such as fear from the low-valence mixed emotion group, and then reallocated to another group (high-valence). Finally, based on the improved mixed emotion group, the conditional distribution adaptation term guided by coarse-grained emotional information is calculated.

[0066] Finally, combining the above two formulas, we get the final optimization problem of the CDAR method, which is expressed as:

[0067]

[0068] where λ and μ = λ × λ1 are trade-off parameters that balance all terms.

[0069] (4) Using the Alternated Direction Method (ADM) and the Inexact Augmented Lagrange Method (IALM) to solve the CDAR model, we can get the optimal value of the projection matrix u:

[0070] Step (4) specifically includes:

[0071] (4-1) Initialize the projection matrix U; specifically, randomly initialize it;

[0072] (4-2) According to the projection matrix U, the sentiment category labels of all target domain samples are predicted using the following formula;

[0073]

[0074] emo_label(k) represents the emotion category label of the kth target domain sample. for The i-th item of is the sample feature of the kth target domain sample;

[0075] (4-3) According to the predicted emotion category label, all samples in the target domain that belong to the same emotion category are divided into one category, and the sample features of each category form a feature vector Then, the target domain samples belonging to the same coarse-grained emotion category are divided into one category, and the sample features of each coarse-grained category form a feature vector

[0076] (4-4) According to the determined The original edge Δ is calculated as follows m and two perceived conditional feature distribution gaps

[0077]

[0078]

[0079]

[0080] (4-5) Convert the CDAR model into a constrained optimization problem:

[0081]

[0082] stP=U

[0083] Where, L = [Y s , 0] and is the intermediate variable, is a zero matrix,

[0084] (4-6) Solve the optimization problem by alternately minimizing the Lagrangian function of the variable and obtain the projection matrix value U of the current cycle l ;

[0085] (4-7) Determine whether ||U is satisfied l -U l-1 || F <∈,∈ represents the preset threshold, U l-1 Indicates the projection matrix value of the previous cycle, l indicates the number of iterations, if it is not satisfied, the current projection matrix value U l Return to step (4-2). If satisfied, take the current projection matrix value Ul as the optimal value.

[0086] Steps (4-6) specifically include:

[0087] (4-6-1) Convert the optimization problem into Lagrangian function:

[0088]

[0089] Where T is the multiplier matrix, κ is the trade-off parameter, and Tr[] represents the trace of the matrix;

[0090] (4-6-2) Fix U, T and κ, and update P according to the following formula

[0091]

[0092] Derived P = (2ZZ T +κI) -1 (κU-T-ZL T ), I represents the identity matrix;

[0093] (4-6-3) Fix P, T and κ, and update U:

[0094]

[0095] The solution is Get, where p i and t i are the i-th row of P and T respectively, otherwise u i =0;

[0096] (4-6-4) Update T and κ according to the following formula:

[0097] T=T+κ(PU),κ=min{ρκ,κ max}; (Formula 13)

[0098] In the formula, ρκ represents the updated k value, κ max represents the maximum κ value;

[0099] (4-6-5) Check whether ||PU|| F <∈, if not, return to execute step (4-6-2), if so, end the loop and output the current U value.

[0100] (5) For the speech to be recognized, extract the IS09 features and use the learned Projection obtains the corresponding emotion category.

[0101] The following formula is used to predict the emotional category of the speech to be recognized;

[0102]

[0103] emo_label(te) represents the emotion category of the speech to be recognized. for The i-th item of is the sample feature of the speech to be recognized.

[0104] This embodiment also provides a cross-database speech emotion recognition device based on the CDAR model, including a processor and a computer program stored in a memory and executable on the processor, and the processor implements the above method when executing the program.

[0105] To verify the effectiveness of the present invention, a cross-database SER experiment was conducted to evaluate the proposed CDAR method. Three commonly used speech emotion databases, EmoDB (Berlin), eNTERFACE and CASIA, were used to design a cross-database SER experiment. Specifically, any two of the above-mentioned speech emotion databases were used as the source database library and the target database library to obtain 6 typical cross-database SER tasks, as shown in Table 1, where B, E, and C are the abbreviations of EmoDB, eNTERFACE, and CASIA, respectively, and the left and right databases of the arrows correspond to the source domain database and the target domain database, respectively. In addition, since these databases have different emotions, in each cross-database SER task, speech samples with the same emotion label are extracted to ensure the consistency of the labels. The detailed sample statistics of the selected databases are shown in Table 1. The performance indicator selected is the unweighted average recall (UAR), which is defined as the accuracy of each class divided by the average of the total number of emotion classes. The feature sets provided by INTERSPEECH 2009Emotion Challenge (IS09) and INTERSPEECH 2010 Paralinguistic Challenge (IS10) are used to describe the speech samples in the experiment. Recent well-performing learning methods such as Deep Adaptation Network (DAN), Adversarial Neural Network (DANN), Deep-coral, Deep Subdomain Adaptation Network (DSAN), Deep Transductive Transfer Regression Network (DT TRN) are compared. The best UAR among all results is reported, and the validation results are shown in Table 2.

[0106] Table 1

[0107]

[0108] Table 2

[0109]

[0110] Experimental results show that the speech emotion recognition method proposed in the present invention achieves a higher cross-database speech emotion recognition rate.

[0111] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A cross-database speech emotion recognition method based on the CDAR model, characterized in that The method includes: (1) Obtain two speech emotion databases, one as a training database and the other as a test database. The training database includes a number of speech samples and corresponding emotion category labels, and the test database includes only speech samples: (2) Extract speech samples with the same emotion category labels from the training database and the test database as source domain samples and target domain samples, respectively, and extract IS09 features of the source domain samples and the target domain samples as sample features; (3) Establish a CDAR model, specifically: Where U is the projection matrix to be learned, Represents the feature matrix formed by the sample features of all samples in the source domain, d is the dimension of the feature matrix, n s is the number of source domain samples, Represents the sentiment category label matrix of all samples in the source domain, Represents the feature matrix formed by the sample features of all samples in the target domain, n t is the number of samples in the target domain, λ and μ are weight parameters, c is the total number of emotion categories, and c r is the total number of coarse-grained sentiment categories, Represent the feature matrices of speech samples belonging to the i-th emotion in the source domain and the target domain respectively, Represents the feature matrix of the speech sample belonging to the jth class of coarse-grained emotion in the source domain and the target domain, 1 s ,1 t , and are all vectors with a value of 1 and dimensions of n s 、n t , are the number of speech samples belonging to the i-th emotion in the source domain and the target domain, respectively. Respectively represent the number of speech samples belonging to the jth coarse-grained emotion in the source domain and the target domain. Several emotion categories can be classified into one coarse-grained emotion category; (4) Using the alternating direction method and inexact augmented Lagrange multipliers to solve the CDAR model, we can obtain the optimal value of the projection matrix U. (5) For the speech to be recognized, extract the IS09 features and use the learned Projection obtains the corresponding emotion category; Wherein, step (4) specifically includes: (4-1) Initialize the projection matrix U; (4-2) According to the projection matrix U, the sentiment category labels of all target domain samples are predicted using the following formula; emo_label(k) represents the emotion category label of the kth target domain sample. for The i-th item of is the sample feature of the kth target domain sample; (4-3) According to the predicted emotion category label, all samples in the target domain that belong to the same emotion category are divided into one category, and the sample features of each category form a feature vector Then, the target domain samples belonging to the same coarse-grained emotion category are divided into one category, and the sample features of each coarse-grained category form a feature vector (4-4) According to the determined The original edge Δ is calculated as follows m and two perceived conditional feature distribution gaps (4-5) Convert the CDAR model into a constrained optimization problem: stP=U Where, L = [Y s ,0] and is the intermediate variable, is a zero matrix, (4-6) Solve the optimization problem by alternately minimizing the Lagrangian function of the variable and obtain the projection matrix value U of the current cycle l ; (4-7) Determine whether ∥U is satisfied l -U l-1 ∥0<∈, ∈ represents the preset threshold, U l-1 Indicates the projection matrix value of the previous cycle, l indicates the number of iterations. If it is not satisfied, the current projection matrix value I l Return to step (4-2). If satisfied, set the current projection matrix value U l As the optimal value Wherein, steps (4-6) specifically include: (4-6-1) Convert the optimization problem into Lagrangian function: Where T is the multiplier matrix, κ is the trade-off parameter, and Tr[] represents the trace of the matrix; (4-6-2) Fix U, T and κ, and update P according to the following formula Derived P = (2ZZ T +κI) -1 (κU-T-ZL T ), I represents the identity matrix; (4-6-3) Fix P, T and κ, and update U: The solution is Get, where p i and t i are the i-th row of P and T respectively, otherwise u i =0; (4-6-4) Update T and k according to the following formula: T=T+k(PU),κ=min{ρκ,κ max }; In the formula, ρκ represents the updated k value, κ max represents the maximum κ value; (4-6-5) Check whether ∥PU∥ is satisfied F <∈, if not, return to step (4-6-2), if yes, end the loop and output the current U value; Wherein, step (5) specifically includes: According to the projection matrix U, the emotion category of the speech to be recognized is predicted by the following formula: emo_label(te) represents the emotion category of the speech to be recognized. for The i-th item of is the sample feature of the speech to be recognized.

2. A cross-database speech emotion recognition device based on a CDAR model, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to claim 1 is implemented.

Images