Method, apparatus, electronic device, and computer storage medium for recognizing speech emotion

Abstract

The present application provides a method, apparatus, electronic device, and computer storage medium for speech emotion recognition. The emotion of speech data is recognized through a speech emotion recognition model integrating a feature extractor, a target emotion representation adapter, and a target emotion allocator to obtain an emotion classification result. During the model training process, diverse scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a scenario learning-based strategy is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, a local domain distribution alignment loss is introduced in the pre-training stage to reduce the distribution deviation of inter-domain emotion features, realizing cross-domain alignment and fusion of emotion features. In the fine-tuning stage, a decoupled learning method is used to prompt the emotion representation adapter and the classifier to gradually adapt to new emotion expression scenarios. Thus, the influence of inter-domain differences on emotion recognition performance is effectively reduced, and the robustness and generalization of emotion features are greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent speech processing, and particularly to a method, device, electronic device and computer storage medium for recognizing speech emotion. Background Art

[0002] Speech Emotion Recognition (SER) aims to automatically analyze and understand the emotions conveyed in speech signals by a computer. Due to its wide application prospects in the field of Human-Computer Interaction (HCI), it has become a research hotspot attracting much attention in recent years. Thanks to the rapid development of deep learning in the field of pattern recognition, speech emotion recognition methods based on deep neural networks have achieved good results.

[0003] As a subjective mental state, emotion has complex and diverse expression forms. Affected by multiple factors such as the speaker's characteristics, conversation topics, and languages, emotion features inevitably have significant domain specificity. Specifically, there are deviations in the emotion expressions of different speakers, resulting in a large decline in the performance of existing speech emotion recognition methods when testing unknown speakers. In addition, even the same speaker will have certain differences in emotion expression due to differences in topics and speech contents. Due to the high cost of emotion data annotation, only simple emotion category annotations can be performed on segment-level speech, and the subjective judgment differences between different annotators further affect the annotation quality of the data, which also greatly limits the modeling ability of the model and the domain robustness of the model. Summary of the Invention

[0004] In view of this, the present application provides a method, device, electronic device and computer storage medium for recognizing speech emotion, which effectively reduces the influence of inter-domain differences on emotion recognition performance and greatly improves the robustness and generalization of emotion features.

[0005] The first aspect of the present application provides a method for recognizing speech emotion, including:

[0006] Receiving speech data;

[0007] Input the speech data into a speech emotion recognition model, and output an emotion classification result; wherein, the speech emotion recognition model includes a feature extractor, a target emotion representation adapter, and a target emotion allocator; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor to output a speech feature; input the speech feature into the target emotion representation adapter to output a mapping of local features; input the mapping of local features into the target emotion allocator to output an emotion classification result.

[0008] Optionally, the method for constructing the speech emotion recognition model includes:

[0009] Randomly select an attribute information from the multi-domain emotion dataset;

[0010] For the attribute information, extract the data of the first domain and the data of the second domain;

[0011] Input the data of the first domain and the data of the second domain into the feature extractor respectively, and output the first speech feature and the second speech feature;

[0012] Train the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter;

[0013] Train the pre-trained emotion representation adapter and the initial emotion allocator according to the data of the first domain and the data of the second domain to obtain a target emotion representation adapter and a target emotion allocator.

[0014] Optionally, the training of the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter includes:

[0015] Construct a local attribute set according to the first speech feature and the second speech feature; wherein, the local attribute set is obtained by training with a contrastive loss function;

[0016] The data of the first domain and the data of the second domain are respectively mapped with the local attribute set to obtain a first local feature mapping and a second local feature mapping;

[0017] Determine a local inter-domain distribution loss according to the first local feature mapping and the second local feature mapping, and use the local inter-domain distribution loss to pre-train the initial emotion representation adapter to obtain a pre-trained emotion representation adapter.

[0018] Optionally, training the pre-trained emotion representation adapter and the initial emotion allocator according to the data in the first domain and the data in the second domain to obtain a target emotion representation adapter and a target emotion allocator includes:

[0019] Input the data in the first domain into the pre-trained emotion representation adapter and the initial emotion allocator, and calculate a first emotion classification loss;

[0020] Update the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter;

[0021] Input the data in the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculate a second emotion classification loss;

[0022] Update the parameters in the initial emotion allocator according to the second emotion classification loss to obtain a target emotion allocator;

[0023] Input the data in the first domain and the data in the second domain into the pre-trained emotion representation adapter, and calculate a global inter-domain distribution loss;

[0024] Input the data in the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculate a third emotion classification loss;

[0025] Update the parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss to obtain a target emotion representation adapter.

[0026] Optionally, the feature extractor includes a waveform encoder and a BERT encoder.

[0027] Optionally, the initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer, and a local attribute contrast module.

[0028] Optionally, the initial emotion allocator includes a global maximum pooling layer, a waveform encoder, and a fully connected layer.

[0029] A second aspect of the present application provides a speech emotion recognition device, including:

[0030] A receiving unit, configured to receive speech data;

[0031] An identification unit for inputting the speech data into a speech emotion recognition model and outputting an emotion classification result; wherein, the speech emotion recognition model includes a feature extractor, a target emotion representation adapter, and a target emotion allocator; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor to output speech features; inputs the speech features into the target emotion representation adapter to output a mapping of local features; and inputs the mapping of the local features into the target emotion allocator to output an emotion classification result.

[0032] Optionally, the construction unit of the speech emotion recognition model includes:

[0033] A random unit for randomly selecting an attribute information from the multi-domain emotion dataset;

[0034] An extraction unit for extracting the data of the first domain and the data of the second domain for the attribute information;

[0035] A feature extraction unit for respectively inputting the data of the first domain and the data of the second domain into the feature extractor to output the first speech feature and the second speech feature;

[0036] A first training unit for training the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter;

[0037] A second training unit for training the pre-trained emotion representation adapter and the initial emotion allocator according to the data of the first domain and the data of the second domain to obtain a target emotion representation adapter and a target emotion allocator.

[0038] Optionally, the first training unit includes:

[0039] A local attribute construction unit for constructing a local attribute set according to the first speech feature and the second speech feature; wherein, the local attribute set is obtained by training with a contrast loss function;

[0040] A local feature mapping training unit, the data of the first domain and the data of the second domain are respectively mapped with the local attribute set to obtain a first local feature mapping and a second local feature mapping;

[0041] A pre-training unit, configured to determine an inter-domain distribution loss according to a first local feature map and a second local feature map, and pre-train an initial sentiment representation adapter by using the inter-domain distribution loss to obtain a pre-trained sentiment representation adapter.

[0042] Optionally, the second training unit includes:

[0043] A first calculation unit, configured to input data of a first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator, and calculate a first sentiment classification loss;

[0044] A first update unit, configured to update parameters in the pre-trained sentiment representation adapter according to the first sentiment classification loss to obtain an updated sentiment representation adapter;

[0045] A second calculation unit, configured to input data of a second domain into the updated sentiment representation adapter and the initial sentiment allocator, and calculate a second sentiment classification loss;

[0046] A second update unit, configured to update parameters in the initial sentiment allocator according to the second sentiment classification loss to obtain a target sentiment allocator;

[0047] A third calculation unit, configured to input data of the first domain and data of the second domain into the pre-trained sentiment representation adapter, and calculate a global inter-domain distribution loss;

[0048] A fourth calculation unit, configured to input data of the first domain into the pre-trained sentiment representation adapter and the target sentiment allocator, and calculate a third sentiment classification loss;

[0049] A third update unit, configured to update parameters in the pre-trained sentiment representation adapter according to the third sentiment classification loss and the global inter-domain distribution loss to obtain a target sentiment representation adapter.

[0050] Optionally, the feature extractor includes a waveform encoder and a BERT encoder.

[0051] Optionally, the initial sentiment representation adapter includes two local adaptation layers, a 1×1 convolutional layer, and a local attribute contrast module.

[0052] Optionally, the initial sentiment allocator includes a global maximum pooling layer, a waveform encoder, and a fully connected layer.

[0053] A third aspect of the present application provides an electronic device, including:

[0054] One or more processors;

[0055] A storage device, on which one or more programs are stored;

[0056] When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for recognizing speech emotion according to any one of the first aspect.

[0057] A fourth aspect of the present application provides a computer storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the method for recognizing speech emotion according to any one of the first aspect is implemented.

[0058] As can be seen from the above solutions, the present application provides a method, an apparatus, an electronic device and a computer storage medium for recognizing speech emotion. The emotion of speech data is recognized through a speech emotion recognition model integrating a feature extractor, a target emotion representation adapter and a target emotion allocator, and an emotion classification result is obtained. During the training process, diverse scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, a local domain distribution alignment loss is introduced in the pre-training stage, and through fine analysis and reduction of the distribution deviation of inter-domain emotion features, cross-domain alignment and fusion of emotion features are achieved. In the fine-tuning stage, a decoupled learning method is used to prompt the emotion representation adapter and the classifier to gradually adapt to new emotion expression scenarios. The influence of inter-domain differences on emotion recognition performance is effectively reduced, and the robustness and generalization of emotion features are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0060] Figure 1 It is a specific flowchart of a method for recognizing speech emotion provided by an embodiment of the present application;

[0061] Figure 2 It is a flowchart of a method for constructing a speech emotion recognition model provided by another embodiment of the present application;

[0062] Figure 3 It is a flowchart of a method for training an initial emotion representation adapter provided by another embodiment of the present application;

[0063] Figure 4 It is a flowchart of a method for training a pre-trained emotion representation adapter and an initial emotion allocator provided by another embodiment of the present application;

[0064] Figure 5 A structural diagram of the overall process of training a speech emotion recognition model provided in another embodiment of the present application;

[0065] Figure 6 A schematic diagram of a speech emotion recognition device provided in another embodiment of the present application;

[0066] Figure 7 A schematic diagram of an electronic device for implementing a speech emotion recognition method provided in another embodiment of the present application. Detailed implementation manners

[0067] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0068] The term "including" and its variations used in this article are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0069] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0070] It should be noted that the concepts such as "first" and "second" mentioned in the present application are only used to distinguish different devices, modules, or units, and are not used to limit the order or interdependence relationship of the functions performed by these devices, modules, or units.

[0071] [[ID=XX]]It should be noted that the modification of "one" and "multiple" mentioned in the present application is illustrative rather than restrictive. Those skilled in the art should understand that unless clearly stated otherwise in the context, it should be understood as "one or more".

[0072] The embodiments of the present application provide a speech emotion recognition method, as Figure 1 shown, specifically including the following steps:

[0073] S101. Receive voice data.

[0074] S102. Input the voice data into a voice emotion recognition model, and output an emotion classification result.

[0075] Among them, the voice emotion recognition model includes a feature extractor, a target emotion representation adapter, and a target emotion allocator; the voice emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the voice data, the voice emotion recognition model inputs the voice data into the feature extractor to output voice features; inputs the voice features into the target emotion representation adapter to output a mapping of local features; inputs the mapping of local features into the target emotion allocator to output an emotion classification result.

[0076] Optionally, in another embodiment of the present application, an implementation manner of the construction method of the voice emotion recognition model is as Figure 2 shown, including:

[0077] S201. Randomly select an attribute information from the multi-domain emotion dataset.

[0078] Among them, there are various domain-biased data in the multi-domain emotion dataset; the attribute information includes but is not limited to "speaker", "Chinese", "English", "topic", etc., and is not limited here.

[0079] S202. For the attribute information, extract the data of the first domain and the data of the second domain.

[0080] In the specific implementation process of the present application, two domain-biased domain data can be randomly extracted as the first domain data and the second domain data, and are not limited here.

[0081] The present application uses the data of the first domain and the data of the second domain as a scenario task to perform one-time domain adaptation training on the model.

[0082] For example: in the first iteration, select the data of speaker 1 and speaker 2, and sample to obtain domain A and domain B respectively. In the second iteration, select the data of Chinese and English, and sample to obtain domain A and domain B respectively.

[0083] S203. Input the data of the first domain and the data of the second domain into the feature extractor respectively, and output the first voice feature and the second voice feature.

[0084] In the actual application process of this application, the feature extractor can adopt, but is not limited to, the HuBERT-large network pre-trained without supervision using the Libri-Light60k data to convert the input data into general speech features.

[0085] In the specific implementation process of this application, the feature extractor includes two parts: a waveform encoder and a BERT encoder. The waveform encoder can include a seven-layer convolutional neural network with strides of (5, 2, 2, 2, 2, 2, 2) respectively, and convolutional kernel sizes of (10, 3, 3, 3, 3, 2, 2) respectively, and the number of channels is 512 for all. The BERT encoder can contain 24 Transformer layers, among which 16-head attention is used and the feature dimension is 1024. In the emotion recognition task, the output result of the 17th Transformer layer of this network can be used, but is not limited to, as the general speech feature for emotion recognition processing in subsequent steps, and no limitation is made here.

[0086] S204. Train the initial emotion representation adapter according to the first speech feature and the second speech feature to obtain a pre-trained emotion representation adapter.

[0087] In the specific implementation process of this application, the adopted emotion representation adapter can include, but is not limited to, two local adaptation layers, a 1x1 convolutional layer, and a local attribute contrast module, which are used to adapt the speech features output by the feature extractor to the speech emotion task.

[0088] Optionally, in another embodiment of this application, an implementation manner of step S204 is as Figure 3 shown, including:

[0089] S301. Construct a local attribute set according to the first speech feature and the second speech feature.

[0090] Among them, the local attribute set is obtained by training using a contrast loss function.

[0091] Continuing the above example, the structure of the local adaptation layer is similar to that of the Transformer layer, including a 16-head self-attention module and a feed-forward neural network layer, which are used to adjust the local features by combining context information. The 1*1 convolutional layer is used to adjust the dimension of the local features to , where is the number of frame-level features included in the sample, to avoid computational difficulties caused by too high dimensions. In the local attribute contrast module, a local attribute set is constructed, where

[0092] S302. The data in the first domain and the data in the second domain are respectively mapped to the local attribute set to obtain a first local feature map and a second local feature map.

[0093] Specifically, the method of training the local attribute set using the contrastive loss function can be as follows:

[0094] ; where is the contrastive loss, represents the attribute in the attribute set that is closest to the local feature . Without using any labels, the attribute set is trained using this loss function. And the attributes therein are used as classification targets to classify the local features. The classification score is used as the mapping of the local features. The mapping process is as follows:

[0095] ;

[0096] where represents the cosine similarity metric between the local feature and the th attribute . And is the softmax value of , which can be regarded as the probability that belongs to the target attribute . Here is the temperature hyperparameter, which is usually used to control the "smoothness" of the softmax output, and the default setting is .

[0097] S303. According to the first local feature map and the second local feature map, determine the local inter-domain distribution loss, and use the local inter-domain distribution loss to pre-train the initial sentiment representation adapter to obtain a pre-trained sentiment representation adapter.

[0098] In the specific implementation process of the application, it can be achieved by, but not limited to, calculating the between-class covariance of the data in the first domain and the data in the second domain to model the deviation between different domains:

[0099] ;

[0100] where represents different dimensions of the local features, and represent the local feature maps obtained from samples of different classes and after passing through the sentiment representation adapter. By calculating the Frobenius norm of the difference in between-class covariance between different domains, it is used as the local inter-domain distribution loss ) :

[0101] ;

[0102] Among them, and respectively represent the between-class covariance of domain A (the first domain) and domain B (the second domain). Using this loss function to train the local sentiment adapter enables the learned local attributes to have better cross-domain robustness.

[0103] S205. Train the pre-trained sentiment representation adapter and the initial sentiment allocator according to the data of the first domain and the data of the second domain to obtain the target sentiment representation adapter and the target sentiment allocator.

[0104] In the specific implementation process of this application, the sentiment allocator can be but is not limited to being composed of a pooling layer and a fully connected layer. However, considering the variability of sentiment in sentences, a global max pooling layer can be used to select the most representative local feature in the sample on a specific attribute as the score of the entire sample on this attribute, serving as the segment-level embedding. Finally, a fully connected layer is used to obtain the sentiment classification result.

[0105] In the specific implementation process of this application, a decoupled learning method can be adopted but is not limited to it. Use a specific speech sentiment classification task to fine-tune the model. During this period, different domain data are used to train the sentiment representation adapter and the sentiment classifier respectively, simulating the situation in the real classification task. This can not only enable the classifier to adapt to the sentiment local features with deviations but also make the features output by the sentiment representation adapter more robust, thereby enhancing the domain generalization ability of the entire system.

[0106] The following two loss functions can be used but are not limited to in this stage:

[0107] 1. Classification loss function , that is, calculate the cross-entropy loss function between the predicted sentiment label score and the labeled segment-level sentiment label. 2. Global domain distribution alignment loss , the calculation method of this loss function is the same as the local domain distribution alignment loss in the formula in step S303. The difference is that the input feature here is the global feature after max pooling. Then, an implementation manner of step S205, as Figure 4 shown, includes:

[0108] S401. Input the data of the first domain into the pre-trained sentiment representation adapter and the initial sentiment allocator, and calculate to obtain the first sentiment classification loss.

[0109] S402. Update the parameters in the pre-trained sentiment representation adapter according to the first sentiment classification loss to obtain the updated sentiment representation adapter.

[0110] S403. Input the data of the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculate the second emotion classification loss.

[0111] S404. Update the parameters in the initial emotion allocator according to the second emotion classification loss to obtain the target emotion allocator.

[0112] S405. Input the data of the first domain and the data of the second domain into the pre-trained emotion representation adapter, and calculate the global inter-domain distribution loss.

[0113] S406. Input the data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculate the third emotion classification loss.

[0114] S407. Update the parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss to obtain the target emotion representation adapter.

[0115] The following illustrates the specific training process of emotion classification decoupling learning in an algorithmic way:

[0116]

[0117] That is, first use the data of domain A to calculate the emotion classification loss , and at this time, fix the emotion classifier , and adjust the parameters of the emotion representation adapter :

[0118] ;

[0119] where is the learning rate of the emotion representation adapter. Then, in order to simulate the differences between speakers during training and testing, use and to perform emotion classification on domain B, calculate the emotion classification loss , fix the emotion representation adapter , and adjust the parameters of the emotion classifier :

[0120] ;

[0121] Since contains the distribution information of domain A, this step enables the classifier to learn the distribution transfer from domain A to domain B and improves the adaptability of the classifier when faced with differences in domain distributions. Finally, use and the emotion classifier trained with domain B transfer, and then perform emotion recognition on the data of domain A to calculate the classification loss , and calculate the global inter-domain distribution loss using data from both domains simultaneously. . Fix the sentiment classifier , and re-optimize to obtain sentiment representation adapter parameters that can adapt to different domain distribution differences. :

[0122] ;

[0123] By randomly selecting different domain combinations multiple times and repeating scenario training, we finally obtain the sentiment speech recognizer with domain robustness that we need.

[0124] Figure 5 This is the overall structure diagram of the training process of a speech emotion recognition model provided by an embodiment of this application. For a given multi-domain emotion dataset containing N domains, randomly select an attribute information in the multi-domain emotion dataset, and randomly extract two domain data with deviations as the first domain data and the second domain data. Use the first domain data and the second domain data as a scenario task to perform one domain adaptation training on the model. The pre-trained feature extractor can be, but is not limited to, the HuBERT-large network that has been unsupervised pre-trained with Libri-Light 60k data to convert the input waveform signal into general speech features. This network is divided into two parts: a waveform encoder and a BERT encoder. The sentiment representation adapter part includes two local adaptation layers, a 1x1 convolutional layer, and a local attribute contrast module, which are used to adapt the general speech features output by the pre-trained feature extractor to the speech emotion task. The sentiment classifier consists of a pooling layer and a fully connected layer. Considering the variability of emotions in sentences, a global maximum pooling layer is used to select the most representative local feature in the sample on a specific attribute as the score of the entire sample on this attribute, serving as the segment-level embedding. Finally, a fully connected layer is used to obtain the sentiment classification result.

[0125] To fully verify the effectiveness of the method proposed by the present invention, the applicant uses different speakers as different domains in the experiment and conducts experiments with different speakers in training and testing to test the generalization ability of the emotional speech features. The significant improvement in the experimental results proves that the method proposed by the invention successfully reduces the inter-domain distribution difference of emotional features, greatly improves the domain robustness of the emotion recognition system, and significantly improves the emotion classification effect in unknown situations. The specific experimental introduction is as follows:

[0126] (1) Experimental settings

[0127] Experiments were conducted on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset to evaluate the effectiveness of the proposed method. The IEMOCAP dataset contains approximately 12 hours of English audio-visual data. The database is divided into 5 parts, each part containing one male and one female actor. According to the recording scenarios, it can be further divided into improvised conversation or scripted conversation parts. Each utterance is annotated by multiple annotators with 8 emotion labels. To make a fair comparison with previous work, 4 emotion types (i.e., neutral, happy, angry, and sad) were selected in the experiment. A 5-fold cross-validation strategy was adopted to evaluate this dataset, with the utterances of 4 parts used for training in each fold, and the remaining part used for validation and testing.

[0128] All systems were implemented using the PyTorch deep learning framework. The SGD optimizer was adopted, and the mini-batch size was set to 64. The system was trained for 50 epochs, with the initial learning rate set to 0.01, which was reduced by a factor of 10 at the 21st, 31st, and 41st rounds respectively.

[0129] (2) Evaluation Metrics

[0130] For the IEMOCAP dataset, we used the following metrics to evaluate the results of four-class emotion recognition.

[0131] Weighted Accuracy (WA): A way of calculating accuracy that takes into account the number of samples in each class. The calculation formula is:

[0132] ;

[0133] where represents the number of samples in class , represents the accuracy of class , and N represents the total number of samples. This calculation method makes the contribution of the errors of each class to the overall accuracy correspond to its proportion in the dataset.

[0134] Unweighted Accuracy (UA): The average precision calculated separately for each emotion category, that is, the number of correctly predicted instances in a category divided by the total number of instances in that category, and then the accuracies of all categories are averaged. The calculation formula is:

[0135] ;

[0136] where represents the accuracy of class , represents the number of classes.

[0137] F1-Score: The harmonic mean of precision and recall, calculated as follows:

[0138] ;

[0139] where precision represents precision rate and recall represents recall rate.

[0140] (3) Experimental results

[0141] As shown in Table 1, it presents the effect of the model's sentiment classification results on the IEMOCAP dataset after using the meta-feature learning method. The metrics are WA, UA, and F1-score. The experimental results show that the proposed context learning method has further improved the sentiment recognition performance. When the number of prototypes in the prototype set is set to 128, the effect is the best. The weighted accuracy of classification has increased from 74.05% to 78.38%, and the F1-score has increased from 69.10% to 76.90%.

[0142] Table 1

[0143]

[0144] Table 2 shows the ablation experiment. When the number of attributes used is 128, it compares the contributions of different loss functions in this scheme to the experimental results and contrasts the improvement of the sentiment recognition effect in two stages. The method used in the first row of Table 2 is the experimental result when the scenario learning strategy is not used and the number of prototypes is 128. The second row represents the experimental effect when only the sentiment classification loss is used and the domain distribution alignment loss is not added. At this time, the three metrics of WA, UA, and F1 have increased to 77.10%, 74.26%, and 74.98% respectively, which are 2.03%, 1.77%, and 0.25% higher than the baseline respectively. This represents that the decoupled training method proposed in the present invention has successfully enabled the sentiment representation to obtain a certain domain generalization ability. The third row is the experimental result of the scenario training with the domain distribution alignment loss. Based on the previous experimental configuration, the WA has increased by 1.28% again, the UA metric has increased by 2.98%, and the F1 metric has increased by 1.92%. This proves that the domain distribution alignment loss has successfully reduced the distribution difference between different domains and further improved the robustness of the model.

[0145] Table 2

[0146]

[0147] The applicant also compared the present invention with some of the current best methods, as shown in Table 3. The results show that the proposed scenario learning-based robust speech emotion recognition method of the present invention has achieved significant improvements in performance compared to the listed systems. All three metrics are far ahead of other systems. Especially in the key evaluation metric of WA accuracy, the improvement obtained by our meta-feature training strategy is particularly obvious, being more than 4% higher than the recently proposed advanced methods.

[0148] Table 3

[0149]

[0150] As can be seen from the above solution, the present application provides a method for recognizing speech emotion, which uses a speech emotion recognition model integrating a feature extractor, a target emotion representation adapter, and a target emotion allocator to recognize the emotion of speech data and obtain an emotion classification result. During the training process, diverse scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, a local domain distribution alignment loss is introduced in the pre-training stage, and through fine analysis and reduction of the distribution deviation of inter-domain emotion features, cross-domain alignment and fusion of emotion features are achieved. In the fine-tuning stage, a decoupled learning method is used to prompt the emotion representation adapter and the classifier to gradually adapt to new emotion expression scenarios. The influence of inter-domain differences on emotion recognition performance is effectively reduced, and the robustness and generalization of emotion features are greatly improved.

[0151] Another embodiment of the present application provides a device for recognizing speech emotion, as Figure 6 shown, specifically including:

[0152] A receiving unit 601, configured to receive speech data.

[0153] An identifying unit 602, configured to input the speech data into the speech emotion recognition model and output an emotion classification result.

[0154] Wherein, the speech emotion recognition model includes a feature extractor, a target emotion representation adapter, and a target emotion allocator; the speech emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data of multiple domains; after receiving the speech data, the speech emotion recognition model inputs the speech data into the feature extractor to output speech features; inputs the speech features into the target emotion representation adapter to output a mapping of local features; and inputs the mapping of local features into the target emotion allocator to output an emotion classification result.

[0155] For the specific working process of the unit disclosed in the foregoing embodiments of the present application, reference may be made to the corresponding method embodiment content, such as Figure 1 as shown, which will not be elaborated here.

[0156] Optionally, in another embodiment of the present application, an implementation manner of the construction unit of the voice emotion recognition model includes:

[0157] A random unit for randomly selecting an attribute information from the multi-domain emotion dataset.

[0158] An extraction unit for extracting the data of the first domain and the data of the second domain for the attribute information.

[0159] A feature extraction unit for respectively inputting the data of the first domain and the data of the second domain into a feature extractor, and outputting a first voice feature and a second voice feature.

[0160] A first training unit for training an initial emotion representation adapter according to the first voice feature and the second voice feature to obtain a pre-trained emotion representation adapter.

[0161] A second training unit for training the pre-trained emotion representation adapter and the initial emotion allocator according to the data of the first domain and the data of the second domain to obtain a target emotion representation adapter and a target emotion allocator.

[0162] For the specific working process of the unit disclosed in the foregoing embodiments of the present application, reference may be made to the corresponding method embodiment content, such as Figure 2 as shown, which will not be elaborated here.

[0163] Optionally, in another embodiment of the present application, an implementation manner of the first training unit includes:

[0164] A local attribute construction unit for constructing a local attribute set according to the first voice feature and the second voice feature; wherein, the local attribute set is obtained by training using a contrast loss function.

[0165] A local feature mapping training unit for mapping the data of the first domain and the data of the second domain to the local attribute set respectively to obtain a first local feature mapping and a second local feature mapping.

[0166] A pre-training unit for determining a local inter-domain distribution loss according to the first local feature mapping and the second local feature mapping, and pre-training the initial emotion representation adapter using the local inter-domain distribution loss to obtain a pre-trained emotion representation adapter.

[0167] For the specific working process of the unit disclosed in the foregoing embodiments of the present application, reference may be made to the corresponding method embodiment content, such as Figure 3 as shown, which will not be elaborated here.

[0168] Optionally, in another embodiment of the present application, an implementation manner of the second training unit includes:

[0169] A first calculation unit, configured to input data of the first domain into a pre-trained emotion representation adapter and an initial emotion allocator, and calculate a first emotion classification loss.

[0170] A first update unit, configured to update parameters in the pre-trained emotion representation adapter according to the first emotion classification loss, and obtain an updated emotion representation adapter.

[0171] A second calculation unit, configured to input data of the second domain into the updated emotion representation adapter and the initial emotion allocator, and calculate a second emotion classification loss.

[0172] A second update unit, configured to update parameters in the initial emotion allocator according to the second emotion classification loss, and obtain a target emotion allocator.

[0173] A third calculation unit, configured to input data of the first domain and data of the second domain into the pre-trained emotion representation adapter, and calculate a global inter-domain distribution loss.

[0174] A fourth calculation unit, configured to input data of the first domain into the pre-trained emotion representation adapter and the target emotion allocator, and calculate a third emotion classification loss.

[0175] A third update unit, configured to update parameters in the pre-trained emotion representation adapter according to the third emotion classification loss and the global inter-domain distribution loss, and obtain a target emotion representation adapter.

[0176] For the specific working process of the unit disclosed in the above embodiment of the present application, reference may be made to the corresponding method embodiment content, as Figure 4 shown, and details are not described herein again.

[0177] Optionally, in another embodiment of the present application, the feature extractor includes a waveform encoder and a BERT encoder.

[0178] Optionally, in another embodiment of the present application, the initial emotion representation adapter includes two local adaptation layers, a 1×1 convolutional layer, and a local attribute contrast module.

[0179] Optionally, in another embodiment of the present application, the initial emotion allocator includes a global maximum pooling layer, a waveform encoder, and a fully connected layer.

[0180] As can be seen from the above solution, the present application provides a voice emotion recognition device, which recognizes the emotion of voice data through a voice emotion recognition model integrating a feature extractor, a target emotion representation adapter, and a target emotion allocator to obtain an emotion classification result. During the training process, diverse scenario tasks are constructed to simulate the domain distribution differences in actual complex scenarios, and a "pre-training + fine-tuning" strategy based on scenario learning is adopted to improve the domain generalization ability in actual complex scenarios. Specifically, a local domain distribution alignment loss is introduced in the pre-training stage, and through fine analysis and reduction of the distribution deviation of inter-domain emotion features, cross-domain alignment and fusion of emotion features are achieved. In the fine-tuning stage, a decoupled learning method is used to prompt the emotion representation adapter and the classifier to gradually adapt to new emotion expression scenarios. The influence of inter-domain differences on emotion recognition performance is effectively reduced, and the robustness and generalization of emotion features are greatly improved.

[0181] The functions described above in this article can be performed at least in part by one or more hardware logic components. For example, without limitation, the exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0182] Another embodiment of the present application provides an electronic device, such as Figure 7 shown, including:

[0183] One or more processors 701.

[0184] A storage device 702, on which one or more programs are stored.

[0185] When the one or more programs are executed by the one or more processors 701, the one or more processors 701 are caused to implement the voice emotion recognition method as described in the above embodiment.

[0186] Another embodiment of the present application provides a computer storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the voice emotion recognition method as described in the above embodiment is implemented.

[0187] In the context of the present application, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] It should be noted that the computer-readable medium described above in the present application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any combination of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. And in the present application, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0189] The above computer-readable medium may be included in the above electronic device; or it may exist separately without being assembled into the electronic device.

[0190] Another embodiment of the present application provides a computer program product which, when executed, is used to perform the above-mentioned method for recognizing speech emotion.

[0191] Specifically, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the above-mentioned functions defined in the method of the embodiments of the present application.

[0192] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the present application is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms of implementing the present application.

[0193] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present application. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0194] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, a technical solution formed by mutually replacing the above features with technical features having similar functions (but not limited to) applied in the present application.

Claims

1. A method for recognizing speech emotions, characterized in that, Including: Receiving voice data; Inputting the voice data into a voice emotion recognition model to output an emotion classification result; wherein, the voice emotion recognition model includes a feature extractor, a target emotion representation adapter, and a target emotion allocator; the voice emotion recognition model is obtained by training an initial emotion representation adapter and an initial emotion classifier with a multi-domain emotion dataset; the multi-domain emotion dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains; after receiving the voice data, the voice emotion recognition model inputs the voice data into the feature extractor to output voice features; inputs the voice features into the target emotion representation adapter to output a mapping of local features; and inputs the mapping of the local features into the target emotion allocator to output an emotion classification result. The method for constructing the voice emotion recognition model includes: randomly selecting an attribute information from the multi-domain emotion dataset; for the attribute information, extracting the data in the first domain and the data in the second domain; respectively inputting the data in the first domain and the data in the second domain into the feature extractor to output the first voice features and the second voice features; training the initial emotion representation adapter according to the first voice features and the second voice features to obtain a pre-trained emotion representation adapter; and training the pre-trained emotion representation adapter and the initial emotion allocator according to the data in the first domain and the data in the second domain to obtain a target emotion representation adapter and a target emotion allocator.

2. The method for recognizing speech emotion according to claim 1, wherein The training of the initial emotion representation adapter according to the first voice features and the second voice features to obtain a pre-trained emotion representation adapter includes: Constructing a local attribute set according to the first voice features and the second voice features; wherein, the local attribute set is obtained by training with a contrast loss function. The data in the first domain and the data in the second domain are respectively mapped with the local attribute set to obtain a first local feature mapping and a second local feature mapping. According to the first local feature mapping and the second local feature mapping, determining a local inter-domain distribution loss, and pre-training the initial emotion representation adapter with the local inter-domain distribution loss to obtain a pre-trained emotion representation adapter.

3. The method for recognizing speech emotion according to claim 1, characterized in that, The training of the pre-trained emotion representation adapter and the initial emotion allocator according to the data in the first domain and the data in the second domain to obtain a target emotion representation adapter and a target emotion allocator includes: Inputting the data in the first domain into the pre-trained emotion representation adapter and the initial emotion allocator to calculate a first emotion classification loss. Updating the parameters in the pre-trained emotion representation adapter according to the first emotion classification loss to obtain an updated emotion representation adapter. Inputting the data in the second domain into the updated emotion representation adapter and the initial emotion allocator to calculate a second emotion classification loss. Updating the parameters in the initial emotion allocator according to the second emotion classification loss to obtain a target emotion allocator. Inputting the data in the first domain and the data in the second domain into the pre-trained emotion representation adapter to calculate a global inter-domain distribution loss. Input the data of the first domain into the pre-trained sentiment representation adapter and the target sentiment allocator, and calculate the third sentiment classification loss. Update the parameters in the pre-trained sentiment representation adapter according to the third sentiment classification loss and the global inter-domain distribution loss to obtain the target sentiment representation adapter.

4. The method for recognizing speech emotion according to claim 1, characterized in that, The feature extractor includes a waveform encoder and a BERT encoder.

5. The method for recognizing speech emotion according to claim 1, characterized in that The initial sentiment representation adapter includes two local adaptation layers, a 1×1 convolutional layer, and a local attribute contrast module.

6. The method for recognizing speech emotion according to claim 1, characterized in that, The initial sentiment allocator includes a global maximum pooling layer, a waveform encoder, and a fully connected layer.

7. An apparatus for recognizing speech emotions, characterized in that, Comprises: A receiving unit for receiving voice data. An identifying unit for inputting the voice data into a voice sentiment recognition model and outputting a sentiment classification result. The voice sentiment recognition model includes a feature extractor, a target sentiment representation adapter, and a target sentiment allocator. The voice sentiment recognition model is obtained by training an initial sentiment representation adapter and an initial sentiment classifier with a multi-domain sentiment dataset. The multi-domain sentiment dataset includes multiple attribute information, and each attribute information corresponds to data in multiple domains. After receiving the voice data, the voice sentiment recognition model inputs the voice data into the feature extractor to output voice features, inputs the voice features into the target sentiment representation adapter to output a mapping of local features, and inputs the mapping of local features into the target sentiment allocator to output a sentiment classification result. The construction unit of the voice sentiment recognition model includes: a random unit, an extraction unit, a feature extraction unit, a first training unit, and a second training unit. The random unit is used to randomly select an attribute information from the multi-domain sentiment dataset. The extraction unit is used to extract the data of the first domain and the data of the second domain for the attribute information. The feature extraction unit is used to input the data of the first domain and the data of the second domain into the feature extractor respectively, and output the first voice feature and the second voice feature. The first training unit is used to train the initial sentiment representation adapter according to the first voice feature and the second voice feature to obtain the pre-trained sentiment representation adapter. The second training unit is used to train the pre-trained sentiment representation adapter and the initial sentiment allocator according to the data of the first domain and the data of the second domain to obtain the target sentiment representation adapter and the target sentiment allocator.

8. An electronic device, characterized in that, Comprises: One or more processors. A storage device on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement the voice sentiment recognition method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by a processor, the voice sentiment recognition method as described in any one of claims 1 to 6 is implemented.

Images