Cross-database speech emotion recognition method and device based on implicit alignment subspace learning

Abstract

The present invention discloses a cross-database speech emotion recognition method and device based on implicit aligned subspace learning, the method comprising: (1) obtaining two speech emotion databases, respectively serving as a training database and a test database; (2) extracting speech features from each speech sample in the training database and the test database; (3) establishing a joint distribution implicit aligned subspace learning model; (4) learning the learning model according to the speech features of the speech samples to obtain the optimal value of the corpus invariant projection matrix U; (5) for the speech to be recognized, obtaining the speech features according to step (2), and using the learned features to obtain the corresponding emotion category. The present invention has a higher recognition accuracy.

Description

Technical Field

[0001] The present invention relates to speech emotion recognition technology, and in particular to a cross-database speech emotion recognition method and device based on implicit aligned subspace learning. Background Art

[0002] The research on speech emotion recognition (SER) aims to enable computers to automatically understand emotional states from speech signals, thereby providing more natural human-computer interaction. Therefore, SER research has attracted widespread attention from researchers in the fields of affective computing and speech signal processing. In the past few decades, many well-performing SER methods have been proposed and achieved good performance on widely used speech emotion corpora. However, it should be noted that these SER methods do not consider the real scenario, that is, the training and testing speech samples may be recorded by different devices, in different environments, or belong to different languages. In this case, the training and testing speech samples may have inconsistent feature distributions, which greatly reduces the performance of the originally well-performing SER methods. Therefore, it is meaningful to study the more challenging but more interesting SER task, namely cross-corpus SER.

[0003] Traditional speech emotion recognition is often trained and tested on a single speech emotion database. However, in real life, the training database and the test database often have large differences. For example, the speech emotion samples are unbalanced in categories, the samples come from different races, etc., so the recognition results are inaccurate. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a cross-database speech emotion recognition method and device based on implicit aligned subspace learning with higher recognition accuracy.

[0005] Technical solution: The cross-database speech emotion recognition method based on implicit aligned subspace learning of the present invention comprises:

[0006] (1) obtaining two speech emotion databases, one as a training database and the other as a test database, wherein the training database contains a number of speech samples and corresponding emotion category labels, and the test database contains a number of speech samples;

[0007] (2) extracting speech features from each speech sample in the training database and the test database;

[0008] (3) Establishing a joint distribution implicit alignment subspace learning model:

[0009]

[0010] Where U is the corpus-invariant projection matrix to be learned, W represents the reconstruction coefficient matrix, and W (i)represents the reconstruction coefficient matrix of the i-th emotion category, i represents the emotion category number, f1(U) represents the simple group sparse linear regression loss related to emotion recognition ability, μ represents the trade-off parameter controlling the balance between the two items, and f2(U, W, W (i) ) represents the invariance capability function of the corpus;

[0011] (4) The learning model is learned according to the speech features of the speech sample to obtain the optimal value of the corpus-invariant projection matrix U

[0012] (5) For the speech to be recognized, obtain the speech features according to step (2) and use the learned , and get the corresponding emotion category.

[0013] Furthermore, the function of f1(U) in the joint distribution implicit alignment subspace learning model is specifically:

[0014]

[0015] Where λ is a trade-off parameter, Represents the feature matrix formed by the speech features of the speech samples in the training database, represents the speech feature of the kth speech sample in the training database, d is the dimension of the acoustic feature vector describing the speech signal, and n s is the number of speech samples in the training database, yes The label matrix formed by the corresponding labels, c represents the total number of emotions involved in cross-corpus emotion recognition, L s Each column of is a one-hot encoded vector, with l = [l1, ..., l c ] means that if the speech sample corresponding to the column belongs to the i-th emotion, then the i-th entry l i The value of is set to 1, and the rest are set to 0;

[0016] f2(U,W,W (i) ) is as follows:

[0017]

[0018] In the formula, Represents the feature matrix formed by the speech features of the test database speech samples, represents the speech features of the jth speech sample in the test database, n t is the number of speech samples in the test database, τ is a trade-off parameter, is the jth test sample The corresponding reconstruction coefficient vector; Represents X sThe feature vector composed of the speech features belonging to the i-th emotion category in, Represents X s The number of speech features belonging to the i-th emotion category in , and Represents X t The feature vector composed of the speech features belonging to the i-th emotion category in, Represents X t The number of speech features belonging to the i-th emotion category in , and

[0019] Furthermore, step (4) specifically includes:

[0020] (4-1) Initialize the corpus invariant projection matrix U;

[0021] (4-2) Based on the corpus-invariant projection matrix U, the following formula is used to predict the emotion category label emotion_label of the speech sample in the test database, and then the emotion category number i to which the predicted emotion category label emotion_label belongs is determined:

[0022]

[0023] Speech emotion category label ith item of ;

[0024] (4-3) Solve the joint distribution implicit alignment subspace learning model. The process is as follows:

[0025] A. According to the determined and Fixed W and W (i) , simplify the learning model into the following form and solve it to get U;

[0026]

[0027] In the formula, and L = [L s , 0 (0) , 0 (1) , …, 0 (c) ],0 (0) and 0 (i) is a string with all entries equal to Δ (0) and Δ (i) Same matrix, i=1,...,c;

[0028] B. According to the obtained U, fix U, divide the learning model into the following two independent optimization models and solve them to obtain W and W (i) :

[0029]

[0030]

[0031] In this formula, W and W (i) The jth column w j and The optimal solution of is obtained by solving the following two typical LASSO problems using the SLEP package:

[0032]

[0033]

[0034] In the formula, and

[0035] C. Check W and W (i) Whether it converges, if it converges, the U value at this time is output as the optimal result, otherwise return to step (4-2) to loop.

[0036] Furthermore, step (5) specifically includes:

[0037] For the speech to be recognized, follow step (2) to obtain the speech features The emotion category is obtained according to the following formula:

[0038]

[0039] In the formula, Speech emotion category label The i-th item of .

[0040] Furthermore, step (2) specifically includes:

[0041] The openSMILE toolkit is used to extract speech features from speech samples, and the speech features include 32 acoustic low-level descriptors (LLDs) and their corresponding 12 features.

[0042] The cross-database speech emotion recognition device based on implicit aligned subspace learning described in the present invention includes a processor and a computer program stored in a memory and executable on the processor, and the above method is implemented when the processor executes the program.

[0043] Beneficial effect: Compared with the prior art, the present invention has the significant advantage that the recognition accuracy of the present invention is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Technical solution: The cross-database speech emotion recognition method based on implicit aligned subspace learning of the present invention comprises:

[0046] (1) Obtain two speech emotion databases, which are used as a training database and a test database respectively. The training database contains a number of speech samples and corresponding emotion category labels, and the test database contains a number of speech samples.

[0047] (2) Extract speech features for each speech sample in the training database and the test database.

[0048] The feature set is extracted by the openSMILE toolkit, which consists of 384 elements, including 32 acoustic low-level descriptors (LLDs) and their corresponding 12 features.

[0049] Finally, the feature matrix formed by the speech features of the speech samples in the training database is obtained And the feature matrix formed by the speech features of the test database speech samples represents the speech features of the kth speech sample in the training database, d is the total number of emotion categories involved in cross-database speech emotion recognition, and n s is the number of speech samples in the training database, represents the speech features of the jth speech sample in the test database, n t To test the number of voice samples in the database, Yes X s The label matrix formed by the corresponding labels, c represents the total number of emotions involved in cross-corpus emotion recognition, L s Each column of is a one-hot encoded vector, with l = [l1, ..., l c ] means that if the speech sample corresponding to this column belongs to the i-th emotion, then the i-th entry l i The value of is set to 1, and the rest are set to 0.

[0050] (3) Establishing a joint distribution implicit alignment subspace learning model:

[0051]

[0052] Where U is the corpus-invariant projection matrix to be learned, W represents the reconstruction coefficient matrix, and W (i) represents the reconstruction coefficient matrix of the i-th emotion category, i represents the emotion category number, f1(U) represents the simple group sparse linear regression loss related to emotion recognition ability, μ represents the trade-off parameter controlling the balance between the two items, and f2(U, W, W (i) ) represents the invariance capability function of the corpus.

[0053] From formula (1), we can see that the objective function of the proposed model has two main terms. Both terms actually correspond to the desired abilities described in the basic idea of ​​implicitly aligned subspace learning of joint distributions, namely, emotion discrimination ability and corpus invariance ability. To this end, f1(U) is designed as a simple group sparse linear regression loss related to emotion recognition ability, and the formula is

[0054]

[0055] Where λ is a trade-off parameter. The learned projection matrix establishes the relationship between the speech features of the training set and its label information, thus having the ability to distinguish different emotions.

[0056] The second term f2(U, W, W (i) ) represents the invariance ability of the corpus. To achieve this goal, we develop it from two aspects: First, we propose an idea to implicitly alleviate the mismatch between the feature distributions of the source and target domain corpora, rather than directly minimizing their statistical moments, such as mean, covariance, and MMD. Specifically, following the common subspace learning work, we adopt the strategy of reconstructing the target domain speech samples using part of the source domain speech samples, and implement projection matrix learning in the model, which can be expressed as the following sparse optimization problem

[0057]

[0058] In the formula, τ is the trade-off parameter, represent The jth column is the jth test library speech sample The corresponding reconstruction coefficient vector. By considering the reconstruction of all target domain speech samples, the overall reconstruction optimization problem is as follows:

[0059]

[0060] in

[0061] Secondly, the model also combines the idea of ​​jointly aligning the edge and class-aware conditional feature distribution differences, which has been proven to be effective in handling domain adaptation tasks. By introducing this idea into equation (4), the objective function of the above design reconstruction can be generalized to the following formula, and finally f2(U, W, W (i) )for

[0062]

[0063] Where τ is the trade-off parameter, is the jth test sample The corresponding reconstruction coefficient vector; Represents X sThe feature vector composed of the speech features belonging to the i-th emotion category in, Represents X s The number of speech features belonging to the i-th emotion category in , and Represents X t The feature vector composed of the speech features belonging to the i-th emotion category in, Represents X t The number of speech features belonging to the i-th emotion category in , and Similar to W, ||W (i) ||1 can also be written as in Yes||W (i) ||The j-th column of 1.

[0064] Finally, substituting equations (2) and (5) into equation (1), we can get the optimization problem of the proposed model, which is expressed as

[0065]

[0066] (4) The learning model is learned according to the speech features of the speech sample to obtain the optimal value of the corpus-invariant projection matrix U

[0067] Specifically include:

[0068] (4-1) Initialize the corpus invariant projection matrix U;

[0069] (4-2) Based on the corpus-invariant projection matrix U, the following formula is used to predict the emotion category label emotion_label of the speech sample in the test database, and then the emotion category number i to which the predicted emotion category label emotion_label belongs is determined: and

[0070]

[0071] Speech emotion category label ith item of ;

[0072] (4-3) Solve the joint distribution implicit alignment subspace learning model. The process is as follows:

[0073] A. According to the determined and Fixed W and W (i) , simplify the learning model into the following form and solve it to get U;

[0074]

[0075] In the formula, and L = [L s , 0 (0) , 0 (1) , …, 0 (c) ],0 (0) and 0 (i) is a string with all entries equal to Δ (0) and Δ (i) Same matrix, i=1,...,c;

[0076] B. According to the obtained U, fix U, divide the learning model into the following two independent optimization models and solve them to obtain W and W (i) :

[0077]

[0078]

[0079] In this formula, W and W (i) The jth column w j and The optimal solution of is obtained by solving the following two typical LASSO problems using the SLEP package:

[0080]

[0081]

[0082] In the formula, and

[0083] C. Check W and W (i) Whether it converges, if it converges, the U value at this time is output as the optimal result, otherwise return to step (4-2) to loop.

[0084] (5) For the speech to be recognized, obtain the speech features according to step (2) and use the learned Get the corresponding emotion category.

[0085] Specifically, for the speech to be recognized, the speech features are obtained according to step (2): The emotion category is obtained according to the following formula:

[0086]

[0087] In the formula, Speech emotion category label The i-th item of .

[0088] This embodiment also provides a cross-database speech emotion recognition device based on implicit aligned subspace learning, including a processor and a computer program stored in a memory and executable on the processor, and the processor implements the above method when executing the program.

[0089] To verify the effectiveness of the present invention, extensive cross-corpus emotion recognition tasks were designed on three commonly used speech emotion corpora, EmoDB (Berlin), eNTERFACE and CASIA, to evaluate the proposed model. EmoDB is a German speech emotion corpus containing 535 speech samples from seven emotions: happiness (HA), sadness (SA), disgust (DI), anger (AN), boredom (BO), fear (FE) and neutral (NE). eNTERFACE is an English bimodal emotion database, and its audio part was extracted in our experiment. eNTERFACE has 1257 samples, each of which is labeled as one of the six emotions, namely HA, SA, DI, FE, AN and surprise (SU). CASIA is a Chinese speech emotion corpus containing 1200 samples and six emotions (HA, SA, NE, FE, AN and SU). By using the above two corpora alternately, we can design 6 cross-corpus emotion recognition tasks, as shown in Table 1, where B, E, and C are the abbreviations of EmoDB, eNTERFACE, and CASIA, the left side of the arrow is the source speech corpus, and the other side corresponds to the target domain. It should be noted that since the label information of these three corpora is inconsistent, in each task we selected speech samples with the same emotion label from them. We summarize the sample statistics of all 6 cross-corpus emotion recognition tasks in Table 1.

[0090] In the experiments, we choose the official feature set of INTERSPEECH 2009Emotion Challenge (IS09) to describe speech signals. The IS09 feature set consists of 384 elements, including 32 acoustic low-level descriptors (LLDs) and their corresponding 12 features, which can be extracted by the openSMILE toolkit. In addition, the unweighted average recall (UAR), which is widely used in random information retrieval, is adopted as the performance indicator of the model. According to the works done, UAR is defined as the accuracy of each class averaged by the total number of emotion classes. The proposed model is compared with recent well-performing subspace learning-based cross-corpus emotion recognition methods, including DaLSR, DoSL, and Joint Distribution Adaptive Regression (JDAR). In addition, several representative transfer subspace learning methods, namely transfer component analysis (TCA), geodesic flow kernel (GFK), subspace alignment (SA), transfer kernel learning (TKL), and baseline method (SVM), are also included in the comparison. We set the trade-off parameters of all compared methods by searching from a preset parameter interval, and then report the best UAR corresponding to the best parameters. Our model selects λ and τ from {0.0001, 0.001, 0.01, 1, 10, 100, 1000}. The verification results are shown in Table 2:

[0091] Table 1. Sample statistics for all 6 cross-corpus sentiment tasks

[0092]

[0093] Table 2. Experimental results of the cross-corpus sentiment recognition task

[0094]

[0095]

[0096] Experimental results show that the cross-corpus emotion recognition method proposed in this invention achieves a higher cross-database recognition rate.

[0097] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A cross-database speech emotion recognition method based on implicitly aligned subspace learning, characterized in that The method includes: (1) obtaining two speech emotion databases, one as a training database and the other as a test database, wherein the training database contains a number of speech samples and corresponding emotion category labels, and the test database contains a number of speech samples; (2) extracting speech features from each speech sample in the training database and the test database; (3) Establishing a joint distribution implicit alignment subspace learning model: Where U is the corpus-invariant projection matrix to be learned, W represents the reconstruction coefficient matrix, and W (i) represents the reconstruction coefficient matrix of the i-th emotion category, i represents the emotion category number, f1(U) represents the simple group sparse linear regression loss related to emotion recognition ability, μ represents the trade-off parameter controlling the balance between the two items, and f2(U,W,W (i) ) represents the invariance capability function of the corpus; (4) The learning model is learned according to the speech features of the speech sample to obtain the optimal value of the corpus-invariant projection matrix U (5) For the speech to be recognized, obtain the speech features according to step (2) and use the learned Get the corresponding emotion category; Among them, the function of f1(U) in the joint distribution implicit alignment subspace learning model is specifically: Where λ is a trade-off parameter, Represents the feature matrix formed by the speech features of the speech samples in the training database, represents the speech feature of the kth speech sample in the training database, d is the dimension of the acoustic feature vector describing the speech signal, and n s is the number of speech samples in the training database, Yes X s The label matrix formed by the corresponding labels, c represents the total number of emotions involved in cross-corpus emotion recognition, L s Each column of is a one-hot encoded vector, with l = [l1,…,l c ] means that if the speech sample corresponding to the column belongs to the i-th emotion, then the i-th entry l i The value of is set to 1, and the rest are set to 0; f2(U,W,W (i) ) is as follows: In the formula, Represents the feature matrix formed by the speech features of the test database speech samples, represents the speech features of the jth speech sample in the test database, n t is the number of speech samples in the test database, τ is a trade-off parameter, is the jth test sample The corresponding reconstruction coefficient vector; Represents X s The feature vector composed of the speech features belonging to the i-th emotion category in, Represents X s The number of speech features belonging to the i-th emotion category in , and Represents X t The feature vector composed of the speech features belonging to the i-th emotion category in, Represents X t The number of speech features belonging to the i-th emotion category in , and Wherein, step (4) specifically includes: (4-1) Initialize the corpus invariant projection matrix U; (4-2) Based on the corpus-invariant projection matrix U, the following formula is used to predict the emotion category label emotion_label of the speech sample in the test database, and then the emotion category number i to which the predicted emotion category label emotion_label belongs is determined: and Speech emotion category label ith item of ; (4-3) Solve the joint distribution implicit alignment subspace learning model. The process is as follows: A. According to the determined and Fixed W and W (i) , simplify the learning model into the following form and solve it to get U; In the formula, Δ (0) =X t -X s W, and L = [L s , 0 (0) , 0 (1) , …, 0 (c) ],0 (0) and 0 (i) is a string with all entries equal to Δ (0) and Δ (i) Same matrix, i=1,...,c; B. According to the obtained U, fix U, divide the learning model into the following two independent optimization models and solve them to obtain W and W (i) : In this formula, W and W (i) The jth column w j and The optimal solution of is obtained by solving the following two typical LASSO problems using the SLEP package: In the formula, and C. Check W and W (i) Whether it converges, if it converges, the U value at this time is output as the optimal result, otherwise return to step (4-2) to loop; Wherein, step (5) specifically includes: For the speech to be recognized, follow step (2) to obtain the speech features The emotion category is obtained according to the following formula: In the formula, Speech emotion category label The i-th item of .

2. The cross-database speech emotion recognition method based on implicit aligned subspace learning according to claim 1, characterized in that: Step (2) specifically includes: The openSMILE toolkit is used to extract speech features from speech samples, and the speech features include 32 acoustic low-level descriptors (LLDs) and their corresponding 12 features.

3. A cross-database speech emotion recognition device based on implicit aligned subspace learning, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 2 is implemented.

Images