A speech emotion recognition method, system, electronic device and medium

Abstract

The application discloses a speech emotion recognition method and system, electronic equipment and medium, and relates to the field of audiovisual content emotion recognition. The method comprises the following steps: obtaining a speech signal to be recognized; using a speech emotion recognition model to recognize the speech signal to be recognized and predict an emotion category; wherein the speech emotion recognition model is obtained by training a speech emotion recognition initial model using a training data set; the training data set comprises a plurality of dialect speech signals labeled with geographical tags and emotion category tags; the speech emotion recognition initial model comprises a Chinese speech pre-training model chinese-hubert-base, a first full connection layer and a second full connection layer connected in sequence; and the emotion category is anger, expectation, disgust, fear, joy, sadness, surprise or trust. The application improves the accuracy of speech emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition in audiovisual content, and in particular to a method, system, electronic device, and medium for voice emotion recognition. Background Technology

[0002] Speech carries a wealth of sparse emotional information. Speech emotion recognition (SER) detects a speaker's emotional state from their speech signal and is generally considered a classification task, typically involving preprocessing, feature extraction, and classification. Early feature extraction relied on professional human experience and formulas to extract various acoustic features such as spectrograms and prosody. With the advancement of deep learning, the use of deep acoustic features extracted by neural networks has improved the recognition rate of SER tasks. However, traditional deep models based on recurrent neural networks or convolutional neural networks suffer from long-range gradient vanishing and information loss when converting long sequences to fixed-length vectors, limiting their ability to extract global speech emotion features. In recent years, end-to-end deep models have been used to implicitly extract features (e.g., using convolutional layers). Current mainstream research on SER tasks focuses on designing deep models that can extract more effective global speech emotion features to improve recognition rates, with typical model recognition rates around 70%. To further improve SER task recognition rates, factors such as gender differences and cultural differences that influence emotional expression and understanding must also be considered. The current problem with SER (Speech Emotion Recognition) is that regional differences in language expression and corresponding emotions mean that models that perform well in one language cannot be transferred to SER tasks in another language. China has a large population and many regional languages, and different dialects express and recognize different emotional prosody. This leads to dialect barriers in speech emotion recognition. For example, Southerners find Northern dialects rough, while Northerners find Southern dialects high-pitched. Therefore, considering dialect barriers and developing speech emotion recognition methods based on dialect differences is significant for improving the recognition rate of SER tasks.

[0003] In 2016, a multilingual approach to enhancing emotion recognition in speech was proposed, using language identifiers to detect language and selecting appropriate models for emotion recognition. In 2017, an approach was proposed to extract emotion features from speech acoustics and facial expression modalities, verifying the influence of cultural factors on emotion recognition and designing a multicultural joint training strategy based on a shared emotion space to mitigate the impact of cultural factors. In 2020, a method for recognition and emotion analysis targeting specific dialects was proposed, establishing a speech emotion database for the Yangzhou-Taizhou dialect and classifying its emotions. To improve the recognition rate of the SER task, there is limited research addressing dialect barriers. Existing studies have employed methods such as first identifying the language type and then selecting the corresponding model for emotion recognition, designing training strategies to mitigate cultural factors, or conducting speech emotion recognition research on a specific dialect. However, these methods have limitations, including the model's ability to recognize only trained languages ​​and its inability to recognize new languages, resulting in low generalization ability.

[0004] In summary, current speech emotion recognition models cannot accurately recognize speech in languages ​​other than those used in the training set for model training. Summary of the Invention

[0005] The purpose of this invention is to provide a voice emotion recognition method, system, electronic device, and medium to improve the accuracy of voice emotion recognition.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A voice emotion recognition method includes:

[0008] Acquire the speech signal to be recognized;

[0009] A speech emotion recognition model is used to identify the speech signal to be recognized and determine the emotion type. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags. The initial speech emotion recognition model includes a Chinese-Hubert-base model, a first fully connected layer, and a second fully connected layer connected in sequence. The emotion type is anger, expectation, disgust, fear, joy, sadness, surprise, or trust.

[0010] Optionally, the training process of the speech emotion recognition model specifically includes:

[0011] Build the training dataset;

[0012] The dialect speech signal with the labeled geographic tags and emotion category tags is input into the current speech emotion recognition initial model to obtain the predicted probability distribution of emotion categories;

[0013] Initialize the parameters of the current speech emotion recognition initial model;

[0014] Based on the cross-entropy loss function and the dialect-emotion regularization term, the loss function of the current initial speech emotion recognition model is determined. The parameters of the current initial speech emotion recognition model are iteratively updated using the stochastic gradient descent algorithm until the value of the loss function converges or the preset number of iterations is reached, at which point training stops. The dialect-emotion regularization term is determined based on the geographic label of the current dialect speech signal, the predicted probability distribution of the emotion type of the current dialect speech signal, the geographic label of another dialect speech signal corresponding to the current emotion type label, and the predicted probability distribution of the emotion type of another dialect speech signal corresponding to the current emotion type label.

[0015] The initial speech emotion recognition model corresponding to the minimum loss value of the loss function is used as the speech emotion recognition model.

[0016] Optionally, the dialect speech signal labeled with geographical tags and sentiment type tags is input into the current initial model for speech sentiment recognition to obtain the predicted probability distribution of sentiment type, specifically including:

[0017] Using the Chinese speech pre-training model chinese-hubert-base, the speech feature sequences of the dialect speech signals labeled with geographical tags and sentiment tags are extracted;

[0018] The first fully connected layer is used to transform the speech feature sequence into a single speech feature vector.

[0019] Based on the speech feature vector, the second fully connected layer is used to determine the probability distribution of emotion type prediction.

[0020] Optionally, the process of determining the dialect-emotion regularization term specifically includes:

[0021] Based on the geographic label of the current dialect speech signal and the geographic label of the other dialect speech signal corresponding to the current emotion category label, determine the positional difference between the current dialect speech signal and the other dialect speech signal corresponding to the current emotion category label.

[0022] The dialect-emotion regularization term is determined based on the position difference, the predicted probability distribution of the emotion type of the current dialect speech signal, and the predicted probability distribution of the emotion type of the other dialect speech signal corresponding to the current emotion type label.

[0023] Optionally, the loss function of the current initial speech emotion recognition model is determined based on the cross-entropy loss function and the dialect-emotion regularization term, specifically including:

[0024] Using the formula L = L e +λLg Determine the loss function for the current initial model of speech emotion recognition; where L e L is the cross-entropy loss function; g λ is the dialect-emotion regularization term; λ is the hyperparameter.

[0025] A voice emotion recognition system, comprising:

[0026] The signal acquisition module is used to acquire the speech signal to be recognized.

[0027] The recognition module is used to identify the speech signal to be recognized and determine the emotion type using a speech emotion recognition model. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags. The initial speech emotion recognition model includes a sequentially connected Chinese-Hubert-base model, a first fully connected layer, and a second fully connected layer. The emotion type is anger, expectation, disgust, fear, joy, sadness, surprise, or trust.

[0028] An electronic device includes: a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described voice emotion recognition method.

[0029] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech emotion recognition method.

[0030] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] The speech emotion recognition method, system, electronic device, and medium of this invention utilize a speech emotion recognition model to identify the acquired speech signal and determine its emotion type. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset is determined by labeling speech signals from different dialects with geographical and emotion type tags. This invention considers the impact of dialect barriers on speech emotion recognition and designs a speech emotion recognition model based on spatiotemporal and contextual association, addressing geospatial factors. Even when the input is speech in a language not trained by the model, the speech emotion recognition model can accurately classify the emotion. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart of the speech emotion recognition method provided by the present invention;

[0034] Figure 2 This is a flowchart of the speech emotion recognition model training process in this invention;

[0035] Figure 3 This is a schematic diagram illustrating the training and inference of the speech emotion recognition model in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The purpose of this invention is to provide a voice emotion recognition method, system, electronic device, and medium to improve the accuracy of voice emotion recognition.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Emotion processing is categorized into discrete and continuous states. Discrete states involve classifying emotions into several categories, such as an eight-category system of anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. Continuous states involve representing the degree of an emotion state using scores across three dimensions: emotional value, arousal, and dominance. This invention is not limited to either continuous or discrete emotion datasets; the final prediction result (whether it's an emotion category or an emotion dimension score) depends on how the task dataset is labeled. Continuous emotion labeling is typically more costly and uses fewer available datasets; therefore, most current research tasks utilize discrete emotion datasets. This invention uses discrete emotion datasets as an example for illustration.

[0040] Example 1

[0041] like Figure 1 As shown, the speech emotion recognition method provided by the present invention includes:

[0042] Step 101: Acquire the speech signal to be recognized.

[0043] Step 102: Train the initial speech emotion recognition model to obtain the speech emotion recognition model.

[0044] Step 103: Use a speech emotion recognition model to identify the speech signal to be identified and determine the emotion type; wherein, the speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset; the training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags; the initial speech emotion recognition model includes a Chinese-Hubert-base model, a first fully connected layer, and a second fully connected layer connected in sequence; the emotion type is anger, expectation, disgust, fear, joy, sadness, surprise, or trust.

[0045] As an optional implementation, the training process of the speech emotion recognition model is as follows: Figure 2 and Figure 3 As shown, the specific steps are as follows:

[0046] 1) Construct the training dataset.

[0047] In practical applications, the speech signals of existing dialect speech databases are labeled to obtain dialect speech emotion recognition datasets.

[0048] Each dialect speech signal is geographically labeled using the latitude and longitude information corresponding to the geographical center of the dialect's distribution area. Each speech signal contains a geographic label G (latitude n, longitude e) and a sentiment category label y.

[0049] The extracted speech signal samples from various dialects were proportioned in a consistent manner to obtain the speech dataset. The speech dataset was then divided into training, validation, and test datasets in a 6:2:2 ratio.

[0050] 2) Input the dialect speech signal with the labeled geographical tags and emotion type tags into the current speech emotion recognition initial model to obtain the predicted probability distribution of emotion type.

[0051] Specifically, 1. Using the Chinese-Hubert-base model, extract the speech feature sequence of the dialect speech signal that has been labeled with geographical tags and sentiment tags.

[0052] In practical applications, the input raw speech waveform (dialect speech signal labeled with geographical and sentiment tags) x is used to extract speech feature sequences z by a pre-trained Chinese-Hubert-base model.

[0053] Original speech waveform x∈R LIts length is L.

[0054] The Chinese-Hubert-Base model is represented as a feature extractor f. θ (). The output of the last hidden layer of the Chinese-Hubert-base model is taken as the speech feature sequence z, z∈R. L×d =f θ (x), where d is the hidden dimension, usually 768, and θ represents the parameters in f.

[0055] 2. Using the first fully connected layer, the speech feature sequence is converted into a single speech feature vector.

[0056] The speech feature sequence z is passed through the first fully connected layer g. ψ Transformed into speech feature vectors based on spatiotemporal and contextual associations.

[0057] First fully connected layer g ψ Given a speech vector sequence z∈R of length L. L×d Transform into a single vector ψ is the first fully connected layer g ψ The parameters.

[0058] 3. Based on the speech feature vector, the second fully connected layer is used to determine the probability distribution of emotion type prediction.

[0059] Second fully connected layer eigenvectors Mapping to the emotional space R C Obtain the original output Use the softmax activation function to... Convert to probability vector

[0060] eigenvectors Mapped to C represents the sentiment type, and the predicted sentiment type is obtained. It is the second fully connected layer The parameters.

[0061] The positional difference (pos) between u and v in two dialects of the same emotional category (u,v) =||(n,e) u -(n,e) v ||2, then the dialect-emotion regularization term is represented as μ is a hyperparameter.

[0062] Second fully connected layer The output of the above method, after undergoing dialect-sentiment regularization, is input into a softmax activation function, and the output is the probability distribution of predicted sentiment categories.

[0063] 3) Initialize the parameters of the current speech emotion recognition initial model.

[0064] If not, then determine the cross-entropy loss value of the predicted sentiment type probability distribution and the sentiment type label. Specifically:

[0065] Predicted probability distribution The cross-entropy loss function between the true sentiment label y and the actual sentiment label y is as follows:

[0066]

[0067] 4) Based on the cross-entropy loss function and the dialect-emotion regularization term, determine the loss function of the current initial speech emotion recognition model. Iteratively update the parameters of the current initial speech emotion recognition model using the stochastic gradient descent algorithm until the loss function converges or a preset number of iterations is reached, at which point training stops. The dialect-emotion regularization term is determined based on the geographic label of the current dialect speech signal, the predicted probability distribution of the emotion type of the current dialect speech signal, the geographic label of the other dialect speech signal corresponding to the current emotion type label, and the predicted probability distribution of the emotion type of the other dialect speech signal corresponding to the current emotion type label. The number of training iterations is usually adjusted based on experimental results. Training stops when the loss function no longer decreases significantly with increasing training iterations or when the performance metrics (loss, accuracy) of the validation set no longer improve significantly.

[0068] As an optional implementation, the loss function of the current initial speech emotion recognition model is determined based on the cross-entropy loss function and the dialect-emotion regularization term, specifically including:

[0069] Using the formula L = L e +λL g Determine the loss function for the current initial model of speech emotion recognition; where L e L is the cross-entropy loss function; g λ is the dialect-emotion regularization term; λ is the hyperparameter.

[0070] The total loss function is obtained by weighting and adding the cross-entropy loss function and the dialect-sentiment regularization term, and then the model parameters are initialized as constants based on experience.

[0071] According to the stochastic gradient descent algorithm, during training, the data is divided into several batches. In each iteration, a sample is randomly selected from one batch to calculate the model's loss value and the gradient of the model parameters. Then, the parameters of the speech emotion recognition model are iteratively updated based on the direction and magnitude of the gradient. This process is repeated until the loss function converges or the preset number of iterations is reached, at which point training stops.

[0072] The appropriate number of training iterations is determined by the change in the loss value in each iteration. Each iteration is the role of the SGD algorithm. After performing regularization and softmax operations in the second fully connected layer, the stochastic gradient descent algorithm SGD is used to continuously update the model parameters in order to find the minimum loss value of the model.

[0073] 5) Use the current initial model for speech emotion recognition corresponding to the minimum loss value of the loss function as the speech emotion recognition model.

[0074] The determination of the regularization term for the dialect-emotion relationship is as follows:

[0075] Based on the geographic label of the current dialect speech signal and the geographic label of the other dialect speech signal corresponding to the current emotion category label, determine the location difference between the current dialect speech signal and the other dialect speech signal corresponding to the current emotion category label.

[0076] The dialect-emotion regularization term is determined based on the position difference, the predicted probability distribution of the emotion type of the current dialect speech signal, and the predicted probability distribution of the emotion type of the other dialect speech signal corresponding to the current emotion type label.

[0077] The model parameters are updated using the stochastic gradient descent (SGD) algorithm to minimize the loss value calculated by the model's total loss function L.

[0078] minL=L e +λL g , where λ is a hyperparameter. The parameter constants for model initialization are shown in Table 1.

[0079] Table 1. Statistical table of parameter constants during model initialization

[0080]

[0081] In speech emotion classification tasks, the stability of speech signals is assumed to be such that a segment of speech corresponds to one emotion label. This invention considers the impact of dialect barriers on speech emotion recognition and proposes an end-to-end model based on spatiotemporal and contextual association. In the audio modality, a dialect-emotion classification task is proposed: considering that the contextual information of speech may contain associated emotions, the original speech waveform is directly used as input to preserve the original information to the maximum extent; speech features are obtained using a pre-trained model based on a 10,000+ hour Mandarin speech recognition corpus from Wenetspeech (chinese-hubert-base); based on the characteristic that geographical proximity leads to similar language emotional styles, a dialect-emotion regularization expression is designed. For the original speech signal to be recognized, the speech features of the geographically closest dialect are selected for further emotion classification, outputting the predicted emotion category.

[0082] The speech emotion recognition method of the present invention has the following advantages compared with the prior art:

[0083] 1. No need to train a model for dialect recognition, saving computational costs.

[0084] 2. There are no restrictions on the types of dialects to be identified. Even if the dialect of the speech signal to be identified is not in the training model, the model will select the regional speech features that are closest to it for emotion classification based on the speech features.

[0085] 3. It can be applied to sentiment analysis in various dialect speaking scenarios or in videos with sound but no human presence.

[0086] Example 2

[0087] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a voice emotion recognition system is provided below, including:

[0088] The signal acquisition module is used to acquire the speech signal to be recognized.

[0089] The recognition module is used to identify the speech signal to be recognized and determine the emotion type using a speech emotion recognition model. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags. The initial speech emotion recognition model includes a sequentially connected Chinese-Hubert-base model, a first fully connected layer, and a second fully connected layer. The emotion type is anger, expectation, disgust, fear, joy, sadness, surprise, or trust.

[0090] Example 3

[0091] The present invention provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the voice emotion recognition method of Embodiment 1.

[0092] Example 4

[0093] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the speech emotion recognition method of Embodiment 1.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0095] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A voice emotion recognition method, characterized in that, include: Acquire the speech signal to be recognized; A speech emotion recognition model is used to identify the speech signal to be recognized and determine the emotion type. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags. The initial speech emotion recognition model includes a Chinese speech pre-training model (chinese-hubert-base), a first fully connected layer, and a second fully connected layer connected in sequence. The emotion types are anger, anticipation, disgust, fear, joy, sadness, surprise, or trust. The training process of the speech emotion recognition model specifically includes: Build the training dataset; The dialect speech signal with the labeled geographic tags and emotion category tags is input into the current speech emotion recognition initial model to obtain the predicted probability distribution of emotion categories; Initialize the parameters of the current speech emotion recognition initial model; Based on the cross-entropy loss function and the dialect-emotion regularization term, the loss function of the current initial speech emotion recognition model is determined. The parameters of the current initial speech emotion recognition model are iteratively updated using the stochastic gradient descent algorithm until the value of the loss function converges or the preset number of iterations is reached, at which point training stops. The dialect-emotion regularization term is determined based on the geographic label of the current dialect speech signal, the predicted probability distribution of the emotion type of the current dialect speech signal, the geographic label of another dialect speech signal corresponding to the current emotion type label, and the predicted probability distribution of the emotion type of another dialect speech signal corresponding to the current emotion type label. The initial speech emotion recognition model corresponding to the minimum loss value of the loss function is used as the speech emotion recognition model.

2. The speech emotion recognition method according to claim 1, characterized in that, The dialect speech signal with the labeled geographic tags and sentiment type tags is input into the current initial model for speech sentiment recognition to obtain the sentiment type prediction probability distribution, specifically including: Using the Chinese speech pre-training model chinese-hubert-base, the speech feature sequences of the dialect speech signals labeled with geographical tags and sentiment tags are extracted; The first fully connected layer is used to transform the speech feature sequence into a single speech feature vector. Based on the speech feature vector, the second fully connected layer is used to determine the probability distribution of emotion type prediction.

3. The speech emotion recognition method according to claim 1, characterized in that, The process of determining the regularization term for the dialect-emotion equation specifically includes: Based on the geographic label of the current dialect speech signal and the geographic label of the other dialect speech signal corresponding to the current emotion category label, determine the positional difference between the current dialect speech signal and the other dialect speech signal corresponding to the current emotion category label. The dialect-emotion regularization term is determined based on the position difference, the predicted probability distribution of the emotion type of the current dialect speech signal, and the predicted probability distribution of the emotion type of the other dialect speech signal corresponding to the current emotion type label.

4. The speech emotion recognition method according to claim 1, characterized in that, Based on the cross-entropy loss function and the dialect-emotion regularization term, the loss function of the current initial speech emotion recognition model is determined, specifically including: Using formula Determine the loss function for the current initial model of speech emotion recognition; where, The cross-entropy loss function; For dialect-emotion regularization terms; This is a hyperparameter.

5. A voice emotion recognition system, characterized in that, The aforementioned voice emotion recognition system is used to implement the voice emotion recognition method according to any one of claims 1-4, and the voice emotion recognition system comprises: The signal acquisition module is used to acquire the speech signal to be recognized. The recognition module is used to identify the speech signal to be recognized and determine the emotion type using a speech emotion recognition model. The speech emotion recognition model is obtained by training an initial speech emotion recognition model using a training dataset. The training dataset includes multiple dialect speech signals labeled with geographical tags and emotion type tags. The initial speech emotion recognition model includes a sequentially connected Chinese-Hubert-base model, a first fully connected layer, and a second fully connected layer. The emotion type is anger, expectation, disgust, fear, joy, sadness, surprise, or trust.

6. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program, the processor running the computer program to cause the electronic device to perform the voice emotion recognition method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the speech emotion recognition method according to any one of claims 1-4.

Images