Speech emotion recognition method and device, electronic equipment and computer storage medium

Abstract

The invention provides a speech emotion recognition method and device, electronic equipment and a computer storage medium, and aims to recognize the emotion of speech data through a speech emotion recognition model integrating a feature extractor, a target emotion representation adapter and a target emotion distributor to obtain an emotion classification result. In the model training process, diversified scene task construction is utilized, the domain distribution difference condition in an actual complex scene is simulated, and a strategy based on scene learning is adopted to improve the domain generalization ability in the actual complex scene. Specifically, local domain distribution alignment loss is introduced in a pre-training stage, distribution deviation of inter-domain emotion features is reduced, and cross-domain alignment and fusion of the emotion features are realized. In a fine tuning stage, a decoupling learning method is used to promote an emotion representation adapter and a classifier to gradually adapt to a new emotion expression scene. Therefore, the influence of the inter-domain difference on the emotion recognition performance is effectively reduced, and the robustness and generalization of the emotion features are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent speech processing technology, and in particular to a speech emotion recognition method, device, electronic device and computer storage medium. Background Art

[0002] Speech Emotion Recognition (SER) aims to use computers to automatically analyze and understand the emotions conveyed in speech signals. Due to its broad application prospects in the field of human-computer interaction (HCI), it has become a hot research topic in recent years. Thanks to the rapid development of deep learning in the field of pattern recognition, speech emotion recognition methods based on deep neural networks have achieved good results.

[0003] As a subjective psychological state, emotion is expressed in a complex and diverse way. Affected by multiple factors such as speaker characteristics, conversation topics, and language, emotion characteristics inevitably have significant domain specificity. Specifically, there are deviations in the emotional expressions of different speakers, which causes the performance of existing speech emotion recognition methods to drop significantly when testing unknown speakers. In addition, even the same speaker will have certain differences in emotional expression due to differences in topics and speech content. Due to the high cost of emotion data annotation, it is often only possible to perform simple emotion category annotation on segment-level speech. The differences in subjective judgments between different annotators further affect the annotation quality of the data, which also greatly limits the modeling ability of the model and the domain robustness of the model. Summary of the invention

[0004] In view of this, the present application provides a speech emotion recognition method, device, electronic device and computer storage medium, which effectively reduce the impact of inter-domain differences on emotion recognition performance and greatly improve the robustness and generalizability of emotion features.

[0005] The first aspect of the present application provides a method for recognizing speech emotion, comprising:

[0006] receiving voice data;

[0007] The speech data is input into a speech emotion recognition model, and an emotion classification result is output; wherein the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into a feature extractor, and outputs speech features; the speech features are input into a target emotion representation adapter, and a mapping of local features is output; the mapping of local features is input into a target emotion distributor, and an emotion classification result is output.

[0008] Optionally, the method for constructing the speech emotion recognition model includes:

[0009] Randomly select an attribute information in the multi-domain sentiment dataset;

[0010] Extracting data of the first domain and data of the second domain from the attribute information;

[0011] Inputting the data of the first domain and the data of the second domain into the feature extractor respectively, and outputting the first speech feature and the second speech feature;

[0012] Training the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter;

[0013] The pre-trained emotion representation adapter and the initial emotion distributor are trained according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor.

[0014] Optionally, the training of the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter includes:

[0015] Constructing a local attribute set according to the first speech feature and the second speech feature; wherein the local attribute set is obtained by training using a contrast loss function;

[0016] Mapping the data of the first domain and the data of the second domain to the local attribute set respectively to obtain a first local feature map and a second local feature map;

[0017] A local inter-domain distribution loss is determined according to the first local feature map and the second local feature map, and the initial emotion representation adapter is pre-trained using the local inter-domain distribution loss to obtain a pre-trained emotion representation adapter.

[0018] Optionally, the pre-trained emotion representation adapter and the initial emotion distributor are trained according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor, including:

[0019] Inputting the data of the first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator, and calculating a first sentiment classification loss;

[0020] Update the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter;

[0021] Inputting the data of the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculating a second emotion classification loss;

[0022] Update the parameters in the initial sentiment allocator according to the second sentiment classification loss to obtain a target sentiment allocator;

[0023] Inputting data from the first domain and data from the second domain into the pre-trained sentiment representation adapter, and calculating a global inter-domain distribution loss;

[0024] Inputting the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculating a third emotion classification loss;

[0025] The parameters in the pre-trained emotion representation adapter are updated according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

[0026] Optionally, the feature extractor includes a waveform encoder and a BERT encoder.

[0027] Optionally, the initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer and a local attribute comparison module.

[0028] Optionally, the initial emotion allocator includes a global maximum pooling layer, a waveform encoder and a fully connected layer.

[0029] The second aspect of the present application provides a speech emotion recognition device, comprising:

[0030] A receiving unit, used for receiving voice data;

[0031] The recognition unit is used to input the speech data into the speech emotion recognition model and output the emotion classification result; wherein the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor and outputs speech features; inputs the speech features into the target emotion representation adapter and outputs a mapping of local features; inputs the mapping of local features into the target emotion distributor and outputs an emotion classification result.

[0032] Optionally, the construction unit of the speech emotion recognition model includes:

[0033] Random unit, used to randomly select an attribute information in the multi-domain sentiment dataset;

[0034] An extraction unit, configured to extract data of the first domain and data of the second domain according to the attribute information;

[0035] A feature extraction unit, used for inputting the data of the first domain and the data of the second domain into a feature extractor respectively, and outputting a first speech feature and a second speech feature;

[0036] A first training unit is used to train the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter;

[0037] The second training unit is used to train the pre-trained emotion representation adapter and the initial emotion distributor according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor.

[0038] Optionally, the first training unit includes:

[0039] A local attribute construction unit, configured to construct a local attribute set according to the first speech feature and the second speech feature; wherein the local attribute set is obtained by training using a contrast loss function;

[0040] A local feature map training unit maps the data of the first domain and the data of the second domain to the local attribute set respectively to obtain a first local feature map and a second local feature map;

[0041] The pre-training unit is used to determine the local inter-domain distribution loss according to the first local feature map and the second local feature map, and use the local inter-domain distribution loss to pre-train the initial emotion representation adapter to obtain a pre-trained emotion representation adapter.

[0042] Optionally, the second training unit includes:

[0043] A first calculation unit is used to input the data of the first domain into the pre-trained emotion representation adapter and the initial emotion allocator to calculate a first emotion classification loss;

[0044] A first updating unit, configured to update parameters in a pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter;

[0045] A second calculation unit is used to input the data of the second domain into the updated emotion representation adapter and the initial emotion allocator to calculate a second emotion classification loss;

[0046] A second updating unit, used for updating the parameters in the initial emotion allocator according to the second emotion classification loss to obtain a target emotion allocator;

[0047] A third computing unit is used to input the data of the first domain and the data of the second domain into the pre-trained emotion representation adapter to calculate the global inter-domain distribution loss;

[0048] a fourth computing unit, configured to input the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculate a third emotion classification loss;

[0049] The third updating unit is used to update the parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

[0050] Optionally, the feature extractor includes a waveform encoder and a BERT encoder.

[0051] Optionally, the initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer and a local attribute comparison module.

[0052] Optionally, the initial emotion allocator includes a global maximum pooling layer, a waveform encoder and a fully connected layer.

[0053] A third aspect of the present application provides an electronic device, including:

[0054] one or more processors;

[0055] a storage device having one or more programs stored thereon;

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the speech emotion recognition method as described in any one of the first aspects.

[0057] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for recognizing speech emotions as described in any one of the first aspects is implemented.

[0058] It can be seen from the above scheme that the present application provides a method, device, electronic device and computer storage medium for recognizing speech emotions, which recognizes the emotions of speech data through a speech emotion recognition model that integrates a feature extractor, a target emotion representation adapter and a target emotion distributor, and obtains emotion classification results. During the training process, a variety of scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, the local domain distribution alignment loss is introduced in the pre-training stage, and the cross-domain alignment and fusion of emotional features are achieved by finely analyzing and reducing the distribution deviation of inter-domain emotional features. In the fine-tuning stage, the decoupled learning method is used to enable the emotional representation adapter and classifier to gradually adapt to new emotional expression scenarios. It effectively reduces the impact of inter-domain differences on emotion recognition performance and greatly improves the robustness and generalizability of emotional features. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 A specific flow chart of a method for recognizing speech emotion provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of a method for constructing a speech emotion recognition model provided in another embodiment of the present application;

[0062] Figure 3 A flowchart of a method for training an initial emotion representation adapter provided in another embodiment of the present application;

[0063] Figure 4 A flowchart of a method for training a pre-trained emotion representation adapter and an initial emotion distributor provided in another embodiment of the present application;

[0064] Figure 5 An overall structural diagram of a speech emotion recognition model training process provided by another embodiment of the present application;

[0065] Figure 6 A schematic diagram of a speech emotion recognition device provided in another embodiment of the present application;

[0066] Figure 7 A schematic diagram of an electronic device for implementing a method for recognizing speech emotions provided in another embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0069] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0070] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0071] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0072] The present application embodiment provides a method for recognizing speech emotion. Figure 1 As shown, the specific steps include:

[0073] S101: Receive voice data.

[0074] S102: Input the speech data into a speech emotion recognition model, and output the emotion classification result.

[0075] Among them, the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training the initial emotion representation adapter and the initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor and outputs the speech features; the speech features are input into the target emotion representation adapter and output the mapping of local features; the mapping of local features is input into the target emotion distributor and outputs the emotion classification result.

[0076] Optionally, in another embodiment of the present application, an implementation method of a method for constructing a speech emotion recognition model is as follows: Figure 2 As shown, including:

[0077] S201. Randomly select an attribute information in the multi-domain sentiment dataset.

[0078] Among them, there are various data with domain deviations in the multi-domain sentiment dataset; attribute information includes but is not limited to "speaker", "Chinese", "English", "topic", etc., which are not limited here.

[0079] S202: Extract the data of the first domain and the data of the second domain based on the attribute information.

[0080] In the specific implementation process of the present application, two domain data with existence and deviation can be randomly selected as the first domain data and the second domain data, which is not limited here.

[0081] This application uses the first domain data and the second domain data as a scenario task to perform domain adaptive training on the model.

[0082] For example, in the first iteration, the data of speaker 1 and speaker 2 are selected and sampled to obtain domain A and domain B respectively. In the second iteration, the data of Chinese and English are selected and sampled to obtain domain A and domain B respectively.

[0083] S203, inputting the data of the first domain and the data of the second domain into a feature extractor respectively, and outputting a first speech feature and a second speech feature.

[0084] In the actual application process of this application, the feature extractor can use but is not limited to the HuBERT-large network that has been unsupervised pre-trained with Libri-Light60k data to convert the input data into general speech features.

[0085] In the specific implementation process of the present application, the feature extractor includes two parts: a waveform encoder and a BERT encoder. The waveform encoder may include a seven-layer convolutional neural network with step sizes of (5, 2, 2, 2, 2, 2, 2), convolution kernel sizes of (10, 3, 3, 3, 3, 2, 2), and the number of channels is 512. The BERT encoder may include 24 Transformer layers, using 16 attention heads and a feature dimension of 1024. In the emotion recognition task, the output result of the 17th Transformer layer of the network can be used as a general speech feature, but is not limited to, and emotion recognition processing is performed in subsequent steps, which is not limited here.

[0086] S204: Train the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter.

[0087] In the specific implementation process of the present application, the emotion representation adapter used may include but is not limited to two local adaptation layers, a 1x1 convolutional layer and a local attribute comparison module, which is used to adapt the speech features output by the feature extractor to the speech emotion task.

[0088] Optionally, in another embodiment of the present application, an implementation of step S204 is as follows: Figure 3 As shown, including:

[0089] S301: Construct a local attribute set according to a first speech feature and a second speech feature.

[0090] Among them, the local attribute set is obtained by training using the contrast loss function.

[0091] Continuing with the above example, the structure of the local adaptation layer is similar to the Transformer layer, which includes 16 self-attention modules and a feedforward neural network layer to adjust the local features based on context information. The 1*1 convolutional layer is used to adjust the dimension of the local features to , is the number of frame-level features contained in the sample to avoid computational difficulties caused by too high a dimension. In the local attribute comparison module, a local attribute set is constructed ,in is the number of local attributes.

[0092] S302 , the data of the first domain and the data of the second domain are respectively mapped to the local attribute set to obtain a first local feature map and a second local feature map.

[0093] Specifically, the method of using the contrast loss function to train the local attribute set can be as follows:

[0094] ;in, is the contrast loss, Represents the local feature of the distance in the attribute set The nearest attribute, this loss function trains the attribute set without using any labels. And the attributes are used as classification targets to classify local features. The classification scores are used as the mapping of local features. The mapping process is as follows:

[0095] ;

[0096] in, Representing local features and Attributes The cosine similarity measure between . yes The softmax value of Belongs to the target attribute The probability of . is a temperature hyperparameter, usually used to control the "smoothness" of the softmax output, and is set by default to .

[0097] S303. Determine a local inter-domain distribution loss according to the first local feature map and the second local feature map, and use the local inter-domain distribution loss to pre-train an initial emotion representation adapter to obtain a pre-trained emotion representation adapter.

[0098] In the specific implementation process of the application, the inter-class covariance of the first domain data and the second domain data can be calculated by, but not limited to, , modeling the deviation between different domains:

[0099] ;

[0100] in, Represents different dimensions of local features, and Representatives from different categories and The local feature map obtained after the sample is passed through the sentiment representation adapter. The Frobenius norm of the covariance difference between different domain classes is calculated as the local domain distribution loss :

[0101] ;

[0102] in, and Represent the inter-class covariance of domain A (first domain) and domain B (second domain). This loss function is used to train the local emotion adapter so that the learned local attributes have better cross-domain robustness.

[0103] S205. Train the pre-trained emotion representation adapter and the initial emotion distributor according to the data of the first domain and the data of the second domain to obtain a target emotion representation adapter and a target emotion distributor.

[0104] In the specific implementation process of this application, the sentiment distributor can be, but is not limited to, composed of a pooling layer and a fully connected layer. However, considering the variability of sentiment in a sentence, a global maximum pooling layer can be used to select the most representative local feature in the sample on a specific attribute as the score of the entire sample on the attribute as a segment-level embedding. Finally, a fully connected layer is used to obtain the sentiment classification result.

[0105] In the specific implementation process of this application, the decoupled learning method can be adopted but not limited to, and the model is fine-tuned using a specific speech emotion classification task. During this period, the emotion representation adapter and the emotion classifier are trained using different domain data respectively to simulate the situation in the actual classification task. This can not only allow the classifier to adapt to the local emotional features with deviations, but also make the features output by the emotion representation adapter more robust, thereby enhancing the domain generalization ability of the entire system.

[0106] This stage can use but is not limited to the following two loss functions:

[0107] 1. Classification loss function , that is, the cross entropy loss function is calculated by predicting the sentiment label score and annotating the segment-level sentiment label. 2. Global domain distribution alignment loss , the loss function is calculated in the same way as the local domain distribution alignment loss in the formula in step S303, except that the input feature here is a global feature after maximum pooling, then, an implementation method of step S205 is as follows: Figure 4 As shown, including:

[0108] S401. Input the data of the first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator, and calculate the first sentiment classification loss.

[0109] S402: Update the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter.

[0110] S403: input the data of the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculate the second emotion classification loss.

[0111] S404: Update the parameters in the initial emotion allocator according to the second emotion classification loss to obtain a target emotion allocator.

[0112] S405: Input the data of the first domain and the data of the second domain into the pre-trained emotion representation adapter, and calculate the global inter-domain distribution loss.

[0113] S406: Input the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculate the third emotion classification loss.

[0114] S407. Update the parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

[0115] The following algorithmic approach is used to explain the specific training process of sentiment classification decoupling learning:

[0116]

[0117] That is, first use the data of domain A to calculate the sentiment classification loss , at this time, the sentiment classifier is fixed , for the sentiment representation adapter parameters Make adjustments:

[0118] ;

[0119] in is the learning rate of the emotion representation adapter. Next, to simulate the difference between speakers during training and testing, we use and Perform sentiment classification on domain B and calculate sentiment classification loss , fixed emotion representation adapter , for the sentiment classifier parameters Make adjustments:

[0120] ;

[0121] because contains the distribution information of domain A. This step allows the classifier to learn the distribution migration from domain A to domain B, improving the classifier's adaptability when faced with differences in domain distribution. Finally, use and the sentiment classifier trained after domain B transfer , then perform sentiment recognition on domain A data and calculate the classification loss , and use the data from both domains to calculate the global inter-domain distribution loss Fixed sentiment classifier , re-align Optimize to obtain the emotion representation adapter parameters that can adapt to the distribution differences of different domains :

[0122] ;

[0123] By randomly selecting different domain combinations multiple times and repeating scenario training, we can finally obtain the domain-robust emotional speech recognizer we need.

[0124] Figure 5 The overall structural diagram of a speech emotion recognition model training process provided by an embodiment of the present application, for a given multi-domain emotion data set containing N domains, randomly select an attribute information in the multi-domain emotion data set, randomly extract two domain data with existence and deviation, as the first domain data and the second domain data, take the first domain data and the second domain data as a scenario task, and perform domain adaptive training on the model. The pre-trained feature extractor can use but is not limited to the HuBERT-large network that has been unsupervised pre-trained with Libri-Light 60k data to convert the input waveform signal into a general speech feature. The network is divided into two parts: a waveform encoder and a BERT encoder. The emotion representation adapter part includes two local adaptation layers, a 1x1 convolutional layer, and a local attribute comparison module, which is used to adapt the general speech features output by the pre-trained feature extractor to the speech emotion task. The emotion classifier consists of a pooling layer and a fully connected layer. Considering the variability of emotion in a sentence, a global maximum pooling layer is used to select the most representative local feature in the sample on a specific attribute as the score of the entire sample on the attribute, as a segment-level embedding. Finally, a fully connected layer is used to obtain the sentiment classification results.

[0125] In order to fully verify the effectiveness of the method proposed in the present invention, the applicant used different speakers as different domains in the experiment, and used different speakers in training and testing to test the generalization ability of emotional speech features. The substantial improvement in the experimental results proves that the method proposed in the invention successfully reduces the inter-domain distribution differences of emotional features, greatly improves the domain robustness of the emotion recognition system, and significantly improves the emotion classification effect in unknown situations. The specific experiments are described as follows:

[0126] (1) Experimental setup

[0127] Experiments are conducted on the Interactive Emotion Motion Capture Database (IEMOCAP) dataset to evaluate the effectiveness of the proposed method. The IEMOCAP dataset contains about 12 hours of English audio-visual data. The database is divided into 5 parts, each containing one male and one female actor. Depending on the recording scene, it can be further subdivided into improvisational dialogue or scripted dialogue parts. Each utterance is annotated with 8 emotion tags by multiple annotators. In order to make a fair comparison with previous work, 4 emotion types (i.e., neutral, happy, angry, and sad) are selected in the experiment. A 5-fold cross-validation strategy is adopted to evaluate the dataset, with 4 parts of sentences in each fold used for training and the remaining parts used for validation and testing.

[0128] All systems were implemented using the PyTorch deep learning framework. The SGD optimizer was used, and the mini-batch size was set to 64. The system was trained for 50 cycles, with the initial learning rate set to 0.01, which was reduced tenfold in the 21st, 31st, and 41st rounds, respectively.

[0129] (2) Evaluation indicators

[0130] For the IEMOCAP dataset, we used the following metrics to evaluate the results of four-category emotion recognition.

[0131] Weighted Accuracy (WA): This is a method of calculating the accuracy by taking into account the number of samples in each category. The calculation formula is:

[0132] ;

[0133] in Indicates category The number of samples, Indicates category The accuracy of N is the total number of samples. This calculation method makes the contribution of each category of error to the overall accuracy correspond to its proportion in the data set.

[0134] Unweighted Accuracy (UA): The average accuracy calculated for each emotion category, that is, the number of correctly predicted instances in a category divided by the total number of instances in that category, and then the accuracy of all categories is averaged. The calculation formula is:

[0135] ;

[0136] in Indicates category The accuracy of Indicates the number of categories.

[0137] F1-Score: The harmonic mean of precision and recall, calculated as:

[0138] ;

[0139] Where precision represents the accuracy and recall represents the recall rate.

[0140] (3) Experimental results

[0141] As shown in Table 1, the effect of the model on the sentiment classification results of the IEMOCAP dataset after using the meta-feature learning method is given. The indicators are WA, UA, and F1-score. The experimental results show that the proposed contextual learning method has further improved the performance of sentiment recognition. When the number of prototypes in the prototype set is set to 128, the effect is best. The weighted accuracy of the classification is increased from 74.05% to 78.38%, and the F1-score is increased from 69.10% to 76.90%.

[0142] Table 1

[0143]

[0144] Table 2 shows the ablation experiment. When the number of attributes used is 128, the contribution of different loss functions in this scheme to the experimental results is compared, and the improvement of the effect of emotion recognition in the two stages is compared. The method used in the first row of Table 2 is the experimental result when the number of prototypes is 128 without using the scenario learning strategy. The second row represents the experimental effect when only the emotion classification loss is used without adding the domain distribution alignment loss. At this time, the three indicators of WA, UA and F1 have risen to 77.10%, 74.26% and 74.98% respectively, which are 2.03%, 1.77% and 0.25% higher than the baseline respectively. This means that the decoupling training method proposed in this invention has successfully enabled the emotion representation to obtain a certain domain generalization ability. The third row is the scenario training experimental result of the domain distribution alignment loss. Based on the previous experimental configuration, the WA of this scheme has been improved by 1.28% again, while the UA indicator has been improved by 2.98% and the F1 indicator has been improved by 1.92%. This proves that the domain distribution alignment loss has successfully reduced the distribution difference between different and further improved the robustness of the model.

[0145] Table 2

[0146]

[0147] The applicant also compared the present invention with some of the best existing methods, as shown in Table 3. The results show that the robust speech emotion recognition method based on contextual learning proposed in the present invention has achieved significant performance improvement over the listed systems. All three indicators are significantly ahead of other systems. Especially in the key evaluation indicator WA accuracy, the improvement achieved by our meta-feature training strategy is particularly obvious, which is more than 4% higher than the advanced methods proposed recently.

[0148] Table 3

[0149]

[0150] It can be seen from the above scheme that the present application provides a method for recognizing speech emotions, which recognizes the emotions of speech data through a speech emotion recognition model that integrates a feature extractor, a target emotion representation adapter, and a target emotion distributor to obtain emotion classification results. During the training process, a variety of scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, the local domain distribution alignment loss is introduced in the pre-training stage, and the cross-domain alignment and fusion of emotional features are achieved by finely analyzing and reducing the distribution deviation of inter-domain emotional features. In the fine-tuning stage, the decoupled learning method is used to enable the emotional representation adapter and classifier to gradually adapt to new emotional expression scenarios. It effectively reduces the impact of inter-domain differences on emotion recognition performance and greatly improves the robustness and generalizability of emotional features.

[0151] Another embodiment of the present application provides a speech emotion recognition device, such as Figure 6 As shown, specifically including:

[0152] The receiving unit 601 is used to receive voice data.

[0153] The recognition unit 602 is used to input the speech data into the speech emotion recognition model and output the emotion classification result.

[0154] Among them, the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training the initial emotion representation adapter and the initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor and outputs the speech features; the speech features are input into the target emotion representation adapter and output the mapping of local features; the mapping of local features is input into the target emotion distributor and outputs the emotion classification result.

[0155] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0156] Optionally, in another embodiment of the present application, an implementation of a construction unit of a speech emotion recognition model includes:

[0157] Random unit, used to randomly select an attribute information in the multi-domain sentiment dataset.

[0158] The extraction unit is used to extract the data of the first domain and the data of the second domain according to the attribute information.

[0159] The feature extraction unit is used to input the data of the first domain and the data of the second domain into the feature extractor respectively, and output the first speech feature and the second speech feature.

[0160] The first training unit is used to train the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter.

[0161] The second training unit is used to train the pre-trained emotion representation adapter and the initial emotion distributor according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor.

[0162] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.

[0163] Optionally, in another embodiment of the present application, an implementation of the first training unit includes:

[0164] A local attribute construction unit is used to construct a local attribute set according to the first speech feature and the second speech feature; wherein the local attribute set is obtained by training using a contrast loss function.

[0165] The local feature map training unit is used to map the data of the first domain and the data of the second domain with the local attribute set respectively to obtain a first local feature map and a second local feature map.

[0166] The pre-training unit is used to determine the local inter-domain distribution loss according to the first local feature map and the second local feature map, and use the local inter-domain distribution loss to pre-train the initial emotion representation adapter to obtain a pre-trained emotion representation adapter.

[0167] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 3 As shown, no further description is given here.

[0168] Optionally, in another embodiment of the present application, an implementation of the second training unit includes:

[0169] The first computing unit is used to input the data of the first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator to calculate the first sentiment classification loss.

[0170] The first updating unit is used to update the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter.

[0171] The second calculation unit is used to input the data of the second domain into the updated emotion representation adapter and the initial emotion allocator to calculate the second emotion classification loss.

[0172] The second updating unit is used to update the parameters in the initial emotion allocator according to the second emotion classification loss to obtain the target emotion allocator.

[0173] The third computing unit is used to input the data of the first domain and the data of the second domain into the pre-trained sentiment representation adapter to calculate the global inter-domain distribution loss.

[0174] The fourth computing unit is used to input the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator to calculate the third emotion classification loss.

[0175] The third updating unit is used to update the parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

[0176] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.

[0177] Optionally, in another embodiment of the present application, the feature extractor includes a waveform encoder and a BERT encoder.

[0178] Optionally, in another embodiment of the present application, the initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer and a local attribute comparison module.

[0179] Optionally, in another embodiment of the present application, the initial emotion allocator includes a global maximum pooling layer, a waveform encoder and a fully connected layer.

[0180] It can be seen from the above scheme that the present application provides a speech emotion recognition device, which recognizes the emotion of speech data through a speech emotion recognition model that integrates a feature extractor, a target emotion representation adapter, and a target emotion distributor to obtain an emotion classification result. During the training process, a variety of scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, the local domain distribution alignment loss is introduced in the pre-training stage, and the cross-domain alignment and fusion of emotional features are achieved by finely analyzing and reducing the distribution deviation of inter-domain emotional features. In the fine-tuning stage, the decoupled learning method is used to enable the emotion representation adapter and classifier to gradually adapt to new emotional expression scenarios. It effectively reduces the impact of inter-domain differences on emotion recognition performance and greatly improves the robustness and generalizability of emotional features.

[0181] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0182] Another embodiment of the present application provides an electronic device, such as Figure 7 As shown, including:

[0183] One or more processors 701.

[0184] The storage device 702 stores one or more programs.

[0185] When the one or more programs are executed by the one or more processors 701, the one or more processors 701 implement the speech emotion recognition method as described in the above embodiments.

[0186] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the method for recognizing speech emotions as described in the above embodiment is implemented.

[0187] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0189] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0190] Another embodiment of the present application provides a computer program product, which, when executed, is used to perform the above-mentioned speech emotion recognition method.

[0191] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0192] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0193] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0194] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A method for recognizing speech emotion, characterized in that: include: receiving voice data; The speech data is input into a speech emotion recognition model, and an emotion classification result is output; wherein the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into a feature extractor, and outputs speech features; the speech features are input into a target emotion representation adapter, and a mapping of local features is output; the mapping of local features is input into a target emotion distributor, and an emotion classification result is output.

2. The method for recognizing speech emotion according to claim 1, characterized in that: The method for constructing the speech emotion recognition model comprises: Randomly select an attribute information in the multi-domain sentiment dataset; Extracting data of the first domain and data of the second domain from the attribute information; Inputting the data of the first domain and the data of the second domain into the feature extractor respectively, and outputting the first speech feature and the second speech feature; Training the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter; The pre-trained emotion representation adapter and the initial emotion distributor are trained according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor.

3. The method for recognizing speech emotion according to claim 2, characterized in that: The step of training the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter includes: Constructing a local attribute set according to the first speech feature and the second speech feature; wherein the local attribute set is obtained by training using a contrast loss function; Mapping the data of the first domain and the data of the second domain to the local attribute set respectively to obtain a first local feature map and a second local feature map; A local inter-domain distribution loss is determined according to the first local feature map and the second local feature map, and the initial emotion representation adapter is pre-trained using the local inter-domain distribution loss to obtain a pre-trained emotion representation adapter.

4. The method for recognizing speech emotion according to claim 2, characterized in that: The method of training the pre-trained emotion representation adapter and the initial emotion distributor according to the data of the first domain and the data of the second domain to obtain the target emotion representation adapter and the target emotion distributor comprises: Inputting the data of the first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator, and calculating a first sentiment classification loss; Update the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter; Inputting the data of the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculating a second emotion classification loss; Update the parameters in the initial sentiment allocator according to the second sentiment classification loss to obtain a target sentiment allocator; Inputting data from the first domain and data from the second domain into the pre-trained sentiment representation adapter, and calculating a global inter-domain distribution loss; Inputting the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculating a third emotion classification loss; The parameters in the pre-trained emotion representation adapter are updated according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

5. The method for recognizing speech emotion according to claim 2, characterized in that: The feature extractor includes a waveform encoder and a BERT encoder.

6. The method for recognizing speech emotion according to claim 2, characterized in that: The initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer and a local attribute comparison module.

7. The method for recognizing speech emotion according to claim 2, characterized in that: The initial emotion allocator includes a global maximum pooling layer, a waveform encoder and a fully connected layer.

8. A speech emotion recognition device, characterized in that: include: A receiving unit, used for receiving voice data; The recognition unit is used to input the speech data into the speech emotion recognition model and output the emotion classification result; wherein the speech emotion recognition model includes a feature extractor, a target emotion representation adapter and a target emotion distributor; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor and outputs speech features; inputs the speech features into the target emotion representation adapter and outputs a mapping of local features; inputs the mapping of local features into the target emotion distributor and outputs an emotion classification result.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the speech emotion recognition method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for recognizing speech emotions as described in any one of claims 1 to 7 is implemented.

Images