Methods, devices, computer equipment and media for training speech emotion recognition models

Abstract

This invention discloses a method for training a speech emotion recognition model. The method involves acquiring phoneme speech features, gender speech features, and emotion speech features; associating phoneme speech features with phoneme speech labels, gender speech features with gender speech labels, and emotion speech features with emotion coordinate labels; inputting these features into a preset recognition model containing initial parameters; obtaining the output feature vector of the preset recognition model; obtaining predicted phoneme labels based on the first feature vector; obtaining predicted gender labels and predicted emotion labels based on the output feature vector; determining a prediction loss value based on the predicted phoneme labels, predicted gender labels, predicted emotion labels, phoneme speech labels, gender speech labels, and emotion coordinate labels; iteratively updating the initial parameters in the preset recognition model until the prediction loss value reaches the convergence condition, and recording this as the speech emotion recognition model. This invention improves the accuracy of speech emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of recognition model technology, and in particular to a method, apparatus, computer equipment, and medium for training a speech emotion recognition model. Background Technology

[0002] With the development of science and technology, artificial intelligence technology is gradually being applied in various fields. For example, in intelligent human-computer interaction systems, the system needs to instantly identify the emotions expressed by users in their conversations in order to take appropriate measures in response to those emotions.

[0003] In existing technologies, speech data is first converted into text by machine recognition, and then text emotion recognition is performed. However, by using the emotional information reflected in the text information of the speech data, the emotional information in the speech data is lost, resulting in a low accuracy rate of emotion recognition. Summary of the Invention

[0004] This invention provides a method, apparatus, computer equipment, and medium for training a speech emotion recognition model to solve the problem of low accuracy in speech emotion recognition.

[0005] A method for training a speech emotion recognition model includes:

[0006] Obtain phoneme speech features, gender speech features, and emotional speech features; the phoneme speech features are associated with phoneme speech tags, the gender speech features are associated with gender speech tags, and the emotional speech features are associated with emotional coordinate tags;

[0007] The phoneme speech features, gender speech features, and emotion speech features are input into a preset recognition model containing initial parameters; the preset recognition model includes a convolutional pooling layer and multiple residual network layers;

[0008] Obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing the speech feature vector through the convolutional pooling layer; the speech feature vector includes the first feature vector output by the first residual network layer.

[0009] Based on the first residual network layer, the first feature vector is output to obtain the predicted phoneme label; based on the output feature vector, the predicted gender label and the predicted emotion label are obtained.

[0010] Based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label, and emotion coordinate label, determine the prediction loss value of the preset recognition model;

[0011] When the predicted loss value does not reach the preset convergence condition, the initial parameters in the preset recognition model are iteratively updated until the predicted loss value reaches the convergence condition. Then, the converged preset recognition model is recorded as a speech emotion recognition model.

[0012] A voice emotion recognition model training device, comprising:

[0013] The feature acquisition module is used to acquire phoneme speech features, gender speech features, and emotion speech features; the phoneme speech features are associated with phoneme speech tags, the gender speech features are associated with gender speech tags, and the emotion speech features are associated with emotion coordinate tags.

[0014] The input feature module is used to input the phoneme speech features, gender speech features, and emotion speech features into a preset recognition model containing initial parameters; the preset recognition model includes a convolutional pooling layer and multiple residual network layers.

[0015] The feature vector module is used to obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing of the speech feature vector through the convolutional pooling layer; the speech feature vector includes the first feature vector output by the first residual network layer.

[0016] The prediction label module is used to obtain predicted phoneme labels based on the first feature vector output by the first residual network layer; and to obtain predicted gender labels and predicted emotion labels based on the output feature vector.

[0017] The loss determination module is used to determine the predicted loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label and emotion coordinate label;

[0018] The model convergence module is used to iteratively update the initial parameters in the preset recognition model when the predicted loss value does not reach the preset convergence condition, until the predicted loss value reaches the convergence condition, and then record the converged preset recognition model as a speech emotion recognition model.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described speech emotion recognition model training method.

[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech emotion recognition model training method.

[0021] This invention provides a method, apparatus, computer device, and storage medium for training a speech emotion recognition model. The method improves the accuracy of feature recognition by setting multiple residual network layers in a preset recognition model. These residual network layers can infer and recognize different features (phoneme speech features, gender speech features, and emotional speech features), thereby improving the accuracy of emotion recognition. Furthermore, the preset recognition model can learn phoneme and gender features from the speech data during the emotion recognition process. Therefore, during the emotion recognition process using the trained speech emotion recognition model, the additional recognition of phoneme and gender features in the speech data can assist in emotion recognition, further improving the accuracy of emotion recognition. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the application environment of the speech emotion recognition model training method in one embodiment of the present invention;

[0024] Figure 2 This is a flowchart of a speech emotion recognition model training method in one embodiment of the present invention;

[0025] Figure 3 This is a flowchart of step S30 in the speech emotion recognition model training method in one embodiment of the present invention;

[0026] Figure 4 This is a flowchart of step S50 in the speech emotion recognition model training method in one embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of a speech emotion recognition model training device according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The speech emotion recognition model training method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment is shown. Specifically, this speech emotion recognition model training method is applied in a speech emotion recognition model training device, which includes, as shown in the example, a speech emotion recognition model training device. Figure 1 The client and server shown communicate over a network to address the low accuracy of emotion recognition in existing technologies. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, refers to the program that provides local services to the client, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0031] In one embodiment, such as Figure 2 As shown, a method for training a speech emotion recognition model is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0032] S10: Obtain phoneme speech features, gender speech features, and emotional speech features; the phoneme speech features are associated with phoneme speech labels, the gender speech features are associated with gender speech labels, and the emotional speech features are associated with emotional coordinate labels.

[0033] Understandably, phoneme speech features, gender speech features, and emotional speech features can be extracted from the same speech data to be recognized, or from different speech data to be recognized. For example, they can be speech data from the same intelligent human-computer interaction system, or speech data from multiple intelligent human-computer interaction systems. A phoneme is the smallest unit of speech, including 21 initials, 36 finals, 5 tones, and silence, for a total of 202 phonemes. Gender includes male and female. Emotion is the state a person experiences when engaging in a certain activity, that is, the outward expression of a person's inner world, such as joy, anger, sorrow, happiness, and sadness. Phoneme speech features represent the phonemes in the speech data to be recognized. Gender speech features represent the gender in the speech data to be recognized. Emotional speech features represent the emotions in the speech data to be recognized. Phoneme speech labels represent the initials, finals, tones, and silences in the phoneme speech features. Gender speech labels represent whether the gender speech feature corresponds to male or female. Emotion coordinate labels represent the emotions in the emotional speech features, such as joy, anger, sorrow, and happiness. Furthermore, phoneme speech features, gender speech features, and emotional speech features are obtained from the server.

[0034] S20: Input the phoneme speech features, gender speech features, and emotion speech features into a preset recognition model containing initial parameters; the preset recognition model includes a convolutional pooling layer and multiple residual network layers.

[0035] Understandably, the preset recognition model includes initial parameters, which are the parameters of each layer in the preset recognition model. The residual network layer is obtained by combining Res2Net, bidirectional RNN, and SE-Block. The number of residual network layers can be set according to the actual situation, such as 3 or 5. The convolutional pooling layer includes a second convolutional layer and an attention pooling layer. That is, it is used to fuse and pool the vectors output by multiple residual network layers. The second convolutional layer is obtained by combining Conv1D one-dimensional convolution, ReLU linear function, and one-layer normalization. The attention pooling layer is obtained by combining self-attention mechanism and one-layer normalization.

[0036] Specifically, after obtaining phoneme speech features, gender speech features, and emotional speech features, a preset recognition model is retrieved from the server. The phoneme speech features, gender speech features, and emotional speech features are then input into the preset recognition model, and the preset recognition model is trained using the phoneme speech features, gender speech features, and emotional speech features.

[0037] S30: Obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing of the speech feature vector through the attention layer; the speech feature vector includes the first feature vector output by the first residual network layer.

[0038] Understandably, the output feature vector is obtained by processing the speech feature vector through an attention pooling layer. The speech feature vector is obtained by convolving the vectors output by all residual network layers through the second convolutional layer. Further, the recognition model includes a first convolutional layer, multiple residual network layers, and convolutional pooling layers. The first residual network layer refers to the residual network layer adjacent to and following the first convolutional layer. The first feature vector is obtained by processing the vector output by the first convolutional layer through the first residual network layer.

[0039] Specifically, after inputting speech features into a preset recognition model, the phoneme speech features, gender speech features, and emotion speech features are convolved through the first convolutional layer of the preset recognition model to obtain convolutional feature vectors. These convolutional feature vectors are then processed sequentially through three residual network layers to obtain three residual feature vectors. The residual feature vectors output from each residual network layer are then convolved through the second convolutional layer of the preset recognition model to obtain speech feature vectors. Finally, the speech feature vectors are pooled through an attention pooling layer to obtain the output feature vector. The speech feature vector includes the first feature vector output from the first residual network layer.

[0040] S40: Based on the first residual network layer, output the first feature vector to obtain the predicted phoneme label; based on the output feature vector, obtain the predicted gender label and the predicted emotion label.

[0041] Understandably, the predicted phoneme label is obtained by the preset recognition model from the phonemes in the speech data to be recognized, such as initials, finals, or tones. The predicted gender label is obtained by the preset recognition model from the gender in the speech data to be recognized, such as male or female. The predicted emotion label is obtained by the preset recognition model from the emotion in the speech data to be recognized, such as joy, anger, or happiness. Following the preset recognition model is a backend recognition model, which is used to determine the predicted labels (such as predicted phoneme labels, predicted gender labels, and predicted emotion labels).

[0042] Specifically, after obtaining the output feature vector, the first feature vector output by the first residual network layer in the speech feature vector is subjected to one-dimensional convolution and probability calculation to obtain the predicted phoneme label. The output feature vector of the preset recognition model is input into the fully connected layer in the back-end recognition model. The fully connected layer in the back-end recognition model calculates the output feature vectors of all preset recognition models by multiplying them by the weight matrix to obtain the calculation result. The calculation result is normalized by a normalization layer in the back-end recognition model to obtain the predicted gender label and predicted emotion label.

[0043] S50: Determine the prediction loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label and emotion coordinate label;

[0044] Understandably, the prediction loss value represents the loss generated by the preset recognition model during the emotion recognition process of speech features. The first loss value characterizes the degree of difference between the phoneme speech label and the predicted phoneme label. The second loss value characterizes the degree of difference between the gender speech label and the predicted gender label. The first loss value characterizes the degree of difference between the emotion coordinate label and the predicted emotion label.

[0045] Specifically, after determining the predicted phoneme label, predicted gender label, and predicted emotion label, a first loss value is obtained by determining the loss value between the predicted phoneme label and the phoneme speech label associated with the phoneme speech feature. A second loss value is obtained by determining the loss value between the predicted gender label and the gender speech label associated with the gender speech feature. A third loss value is obtained by determining the loss value between the predicted emotion label and the emotion coordinate label associated with the emotion speech feature. Finally, based on the first, second, and third loss values, the predicted loss value is determined. The preset loss model can be a model built based on the cross-entropy loss function, or a model built based on the maximum likelihood function, etc.

[0046] S60: When the predicted loss value does not reach the preset convergence condition, iteratively update the initial parameters in the preset classification model until the predicted loss value reaches the convergence condition, and record the converged preset recognition model as a speech emotion recognition model.

[0047] Understandably, the convergence condition can be the condition that the predicted loss value is less than a set threshold, that is, when the predicted loss value is less than the set threshold, training stops; the convergence condition can also be the condition that the predicted loss value is very small after 5000 calculations and will not decrease further, that is, when the predicted loss value is very small after 5000 calculations and will not decrease further, training stops, and the preset recognition model after convergence is recorded as the speech emotion recognition model.

[0048] Furthermore, after determining the prediction loss value of the preset recognition model, if the prediction loss value does not reach the preset convergence condition, the initial parameters of the preset recognition model are adjusted according to the prediction loss value. Phoneme speech features, gender speech features, and emotion speech features are then re-inputted into the preset recognition model after adjusting the initial parameters. When the prediction loss values ​​of the phoneme speech features, gender speech features, and emotion speech features reach the preset convergence condition, other phoneme speech features, gender speech features, and emotion speech features in the preset speech training set are selected, and the above steps S20 to S50 are executed to obtain the prediction loss value corresponding to the preset recognition model with adjusted initial parameters. If the prediction loss value does not reach the preset convergence condition, the initial parameters of the preset recognition model are adjusted again according to the prediction loss value, so that the prediction loss value of the preset recognition model with adjusted initial parameters reaches the preset convergence condition.

[0049] Thus, after training the preset recognition model with all speech features, the output of the preset recognition model can continuously approach the accurate result, making the recognition accuracy higher and higher, until the prediction loss value of all speech features reaches the preset convergence condition, and the converged preset recognition model is recorded as a speech emotion recognition model.

[0050] In this embodiment of the invention, the method improves the accuracy of feature recognition by setting multiple residual network layers in a preset recognition model. These residual network layers can then perform reasoning and recognition on different features (phoneme speech features, gender speech features, and emotional speech features), thereby improving the accuracy of emotion recognition. Furthermore, the preset recognition model can learn phoneme and gender features from the speech data during the emotion recognition process. Therefore, during the emotion recognition process using the trained speech emotion recognition model, the additional recognition of phoneme and gender features in the speech data can assist in emotion recognition, further improving the accuracy of emotion recognition.

[0051] In one embodiment, before step S10, that is, before acquiring phoneme speech features, gender speech features, and emotional speech features, the following steps are included:

[0052] (1) Obtain a preset speech training set; the preset speech training set contains at least one speech data of a phoneme to be identified, speech data of a gender to be identified, and speech data of an emotion to be identified;

[0053] Understandably, the speech data to be recognized refers to the speech data for which emotion recognition is required. A pre-defined speech training set is then constructed by acquiring all the speech data to be recognized. The pre-defined speech training set contains speech data for at least one phoneme to be recognized, speech data for the gender to be recognized, and speech data for the emotion to be recognized.

[0054] Specifically, the data source for the speech data to be recognized varies depending on the application scenario. For example, in an intelligent customer service application scenario, the speech data to be recognized can be the speech data received by the system from users. In an intelligent conferencing scenario, the speech data to be recognized can also be the dialogue between different users. Specifically, the speech data for phoneme recognition can use speech data from the Chinese speech training set AiShell and the English speech training set Librispeech. The speech data for gender recognition can use speech data from the speech training sets AiShell and Voxceleb. The speech data for emotion recognition can use speech data from the Chinese speech training sets CASIA and ESD, and the English speech training sets RAVDESS and CREMA-D.

[0055] (2) Perform spectrum analysis on the speech data of the phonemes to be identified, the speech data of the gender to be identified, and the speech data of the emotions to be identified to obtain the phoneme spectrum diagram corresponding to the speech data of the phonemes to be identified, the gender spectrum diagram corresponding to the speech data of the gender to be identified, and the emotion spectrum diagram corresponding to the speech data of the emotions to be identified.

[0056] Understandably, a phoneme spectrogram is a spectrogram of the speech data containing the phonemes to be identified. A gender spectrogram is a spectrogram of the speech data containing the gender to be identified. An emotion spectrogram is a spectrogram of the speech data containing the emotion to be identified. A spectrogram represents the volume of various frequencies in an audio clip at a given moment, with the horizontal axis representing the frequency direction and the vertical axis representing the amplitude direction.

[0057] Specifically, after obtaining the preset speech training set, the speech data of the phonemes to be identified, the speech data of the gender to be identified, and the speech data of the emotions to be identified included in the preset speech training set are processed by framing. That is, the speech data of the phonemes to be identified, the speech data of the gender to be identified, and the speech data of the emotions to be identified are divided into multiple frame units by fixed time intervals (e.g., 25 milliseconds). In order to avoid excessive changes between adjacent frame units, an overlapping area is allowed between two adjacent frame units. Each frame unit is multiplied by a window function to make the left and right ends of each frame unit continuous, thus obtaining a continuous time window. Short-time Fourier transform is performed on all windowed frame units to obtain the spectrum corresponding to each frame unit, that is, to obtain the spectrum distributed in different time windows on the time axis.

[0058] (3) Extract features from the phoneme spectrogram, gender spectrogram and emotion spectrogram to obtain the phoneme speech features corresponding to the phoneme spectrogram, the gender speech features corresponding to the gender spectrogram and the emotion speech features corresponding to the emotion spectrogram.

[0059] Understandably, phoneme speech features are the features that can represent the spectrogram in a phoneme spectrogram, and these features can characterize whether it is an initial consonant, a final vowel, or a tone. Gender speech features are the features that can represent the spectrogram in a gender spectrogram, and these features can characterize whether the speaker is male or female. Emotional speech features are the features that can represent the spectrogram in an emotion spectrogram, and these features can characterize emotions such as joy or anger.

[0060] Specifically, after obtaining the spectrogram, the spectrum is processed using a Mel filter to obtain the Mel spectrograms corresponding to the phoneme spectrogram, gender spectrogram, and emotion spectrogram. This transforms the linear natural spectrum into a Mel spectrum that reflects human auditory characteristics. The logarithm of the Mel spectrum is obtained to acquire its logarithmic energy. This logarithmic energy is then inversely transformed using the discrete cosine transform (DCT), and the second to thirteenth coefficients after the DCT are taken as Mel frequency cepstral coefficients. These Mel frequency cepstral coefficients are then used as phoneme speech features. Similarly, the gender speech features corresponding to the gender spectrogram and the emotional speech features corresponding to the emotion spectrogram can be obtained.

[0061] This invention, through spectral analysis of the speech data to be recognized included in a preset speech training set, determines the phoneme spectrogram, gender spectrogram, and emotion spectrogram. By extracting features from the spectrograms, phoneme speech features, gender speech features, and emotion speech features are determined. This facilitates the training of the preset recognition model, thereby improving the efficiency of its training.

[0062] In one embodiment, such as Figure 3 As shown, in step S30, the convolutional pooling layer includes a second convolutional layer and an attention pooling layer, that is, obtaining the output feature vector of the preset recognition model includes:

[0063] S301: The phoneme speech features, gender speech features and emotion speech features are processed by convolution through the first convolutional layer to obtain the convolutional feature vector.

[0064] Understandably, the convolutional feature vector is obtained by processing speech features through the first convolutional layer.

[0065] Specifically, after inputting phoneme speech features, gender speech features, and emotional speech features into the preset recognition model, the phoneme speech features, gender speech features, and emotional speech features are convolved through the first convolutional layer of the preset recognition model. First, Conv1D is used to perform one-dimensional convolution processing on the phoneme speech features, gender speech features, and emotional speech features, that is, the number of frames in the convolutional feature vector is compressed into a one-dimensional feature vector. The one-dimensional feature vector is activated by the ReLU function (Rectified Linear Unit). After activation, the one-dimensional feature vector is input into the normalization layer in the first convolutional layer to normalize the one-dimensional feature vector, thus obtaining the convolutional feature vector.

[0066] S302: Input the convolutional feature vector into the residual network layer and obtain the residual feature vector output by each residual network layer.

[0067] Understandably, the residual feature vector is obtained by processing the convolutional feature vector through the residual network layer.

[0068] Specifically, the convolutional feature vector is input into the residual network layers, which process it through three residual network layers. First, the convolutional feature vector is compressed in dimension by using a (1*1) convolution matrix. Then, the weights for each channel are calculated using a (3*3) convolution matrix. Next, the original dimensions are restored using a (1*1) convolution matrix, and the SE-Block layer calculates the residual feature vector by combining the convolutional feature vector with the weights for each channel. Similarly, the three residual network layers process the convolutional feature vector separately to obtain three residual feature vectors.

[0069] S303: The residual feature vectors output by all residual network layers are convolved by the second convolutional layer to obtain the speech feature vector.

[0070] S304: The speech feature vector is processed by an attention pooling layer to obtain the output feature vector.

[0071] Understandably, the speech feature vector is obtained by convolving all the residual feature vectors through the second convolutional layer.

[0072] Specifically, the second convolutional layer performs convolution processing on the residual feature vectors output from all residual network layers, that is, it simultaneously performs feature fusion processing on multiple residual feature vectors. The three residual feature vectors are then concatenated to obtain the speech feature vector. An attention pooling layer is then used to pool the speech feature vectors, obtaining different weights for each frame. Then, a normalization layer within the attention pooling layer normalizes the different weights for each frame, obtaining the mean and variance of the speech feature vector. The speech feature vectors are then concatenated based on the mean and variance to obtain the output feature vector.

[0073] This invention employs a first convolutional layer to perform convolutional processing, thereby acquiring the convolutional feature vector. A residual network layer is then used to infer and recognize different features (phoneme speech features, gender speech features, and emotional speech features), improving the accuracy of feature recognition. A second convolutional layer further processes the residual feature vector, determining the speech feature vector. Finally, an attention pooling layer performs vector processing on the speech feature vector, determining the output feature vector and thus improving the accuracy of predictive label determination.

[0074] In one embodiment, in step S302, the residual feature vector output by the first residual network layer is the first feature vector; the residual feature vector output by the second residual network layer is the second feature vector; and the residual feature vector output by the third residual network layer is the third feature vector.

[0075] The step of inputting the convolutional feature vector into the residual network layer and obtaining the residual feature vector output by each residual network layer includes:

[0076] (1) The first feature vector is obtained by extracting features from the convolutional feature vector through the first residual network layer.

[0077] Understandably, the first feature vector is obtained by extracting features from the convolutional feature vector through the first residual network layer.

[0078] Specifically, after obtaining the convolutional feature vector, it is input into the first residual network layer. The first convolutional layer within this residual network performs convolution on the feature vector, compressing its dimensionality. Then, a recurrent neural network processes the compressed feature vector to enable long-term inference. Dilated convolution is then applied to the compressed feature vector, using a weight matrix determined by the set number of kernels and their distances, thus imbuing the compressed feature vector with more contextual information. The second convolutional layer within the first residual network then performs convolution on the dilated feature vector, restoring its dimensionality. Finally, the restored feature vector is processed by an SE-Block layer, multiplying it by the weight matrix to obtain the first feature vector.

[0079] (2) The first feature vector is extracted by the second residual network layer to obtain the second feature vector.

[0080] Understandably, the second feature vector is obtained by extracting features from the first feature vector through the second residual network layer.

[0081] Specifically, after obtaining the first feature vector, it is input into the second residual network layer. The first convolutional layer within this second residual network layer performs convolution on the first feature vector, compressing its dimension. Then, a recurrent neural network processes the compressed first feature vector to enable long-term inference. Dilated convolution is performed on the compressed first feature vector, using a weight matrix determined by the set number of convolution kernels and their distances, thus imbuing the compressed feature vector with more contextual information. The second convolutional layer within the second residual network layer then performs convolution on the dilated first feature vector, restoring its dimension. Finally, the restored first feature vector is processed by an SE-Block layer, multiplying it by the weight matrix to obtain the second feature vector.

[0082] (3) The second feature vector is extracted through the third residual network layer to obtain the third feature vector.

[0083] Understandably, the third feature vector is obtained by extracting features from the second feature vector through the third residual network layer.

[0084] Specifically, after obtaining the second feature vector, it is input into the third residual network layer. The first convolutional layer in this third residual network layer performs convolution on the second feature vector, compressing its dimension. Then, dilated convolution is applied to the compressed second feature vector, where a weight matrix is ​​obtained based on the set number of convolution kernels and the distance between them, giving the compressed second feature vector more contextual information. The second convolutional layer in the third residual network layer then performs convolution on the dilated second feature vector, restoring its dimension. Finally, the restored second feature vector is processed by the SE-Block layer, multiplying it by the weight matrix to obtain the third feature vector.

[0085] This invention uses three residual network layers to process convolutional feature vectors, thereby determining different residual feature vectors, improving the accuracy of subsequent feature recognition, and reducing the loss value of the preset recognition model.

[0086] In one embodiment, such as Figure 4 As shown, in step S50, namely, determining the prediction loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label, and emotion coordinate label, includes:

[0087] S501, determine the first loss value based on the predicted phoneme label and the phoneme speech label.

[0088] Specifically, after determining the predicted phoneme labels, the phoneme speech labels associated with the phoneme speech features are arranged according to the order of the phoneme speech data to be identified. Then, the predicted phoneme labels obtained through the preset recognition model and the phoneme speech labels associated with the phoneme speech features of the same sequence are used to calculate the loss through the CTC loss function. That is, according to the order of the phoneme speech data to be identified, the predicted phoneme label corresponding to the first phoneme speech data to be identified and the phoneme speech label corresponding to the first phoneme speech feature are used to calculate the loss through the CTC loss function, so as to determine the first loss value between the predicted phoneme label and the phoneme speech label.

[0089] S502, determine a second loss value based on the predicted gender label and the gender voice label.

[0090] Specifically, after determining the predicted gender label, the gender speech labels associated with the gender speech features are arranged according to the order of the gender speech data to be identified. Then, the predicted gender labels obtained through the preset recognition model and the gender speech labels associated with the gender speech features of the same sequence are calculated using the Focal Loss loss function. That is, according to the order of the gender speech data to be identified, the predicted gender label corresponding to the first gender speech data to be identified and the gender speech label corresponding to the first gender speech feature are calculated using the Focal Loss loss function to determine the second loss value between the predicted gender label and the gender speech label.

[0091] S503, determine the third loss value based on the predicted emotion label and the emotion coordinate label.

[0092] Specifically, after determining the predicted emotion label, the emotion coordinate labels associated with the emotion speech features are arranged according to the order of the emotion speech data to be identified. Then, the predicted emotion labels obtained through the preset recognition model and the emotion coordinate labels associated with the emotion speech features of the same sequence are used to calculate the loss using the CCC Loss function. That is, according to the order of the emotion speech data to be identified, the predicted emotion label corresponding to the first emotion speech data to be identified and the emotion coordinate label corresponding to the first emotion speech feature are used to calculate the loss using the CCC Loss function, thus determining the third loss value between the predicted emotion label and the emotion coordinate label. The CCC Loss function is a loss function constructed from the coefficients of Lin's Concordance Correlation Coefficient.

[0093] S504, obtain a preset weighting factor, and determine the prediction loss value of the preset recognition model based on the preset weighting factor, the first loss value, the second loss value, and the third loss value.

[0094] Understandably, the preset weighting factor is a coefficient used to characterize the differences between different loss values. The predicted loss value is the loss generated by the preset recognition model during the emotion recognition process of the speech data to be recognized.

[0095] Specifically, a preset weighting factor is obtained from the server. The preset weighting factor, the first loss value, the second loss value, and the third loss value are then input into a preset overall loss function. The loss value of the preset recognition model is calculated using the preset overall loss function to obtain the predicted loss value of the preset recognition model. The expression for the preset overall loss function is as follows: in, This is the third loss value. This is the second loss value. This is the first loss value. α and β are preset weighting factors.

[0096] The embodiments of the present invention determine the preset overall loss function by predicting phoneme labels and phoneme speech labels, predicting gender labels and gender speech labels, predicting emotion labels and emotion coordinate labels, and using preset weighting factors, thereby obtaining the predicted loss value and making the output results of the preset recognition model more accurate.

[0097] In one embodiment, after step S60, i.e., after recording the converged preset recognition model as a speech emotion recognition model, the process includes:

[0098] (1) Obtain target speech data, and perform emotion recognition on the target speech data through the speech emotion recognition model to obtain the emotion recognition coordinates corresponding to the target speech data.

[0099] Understandably, the target speech data is the speech data for which emotion recognition needs to be performed. For example, the target speech data can be the speech data corresponding to a question posed by a target user in an intelligent question-answering scenario. The emotion recognition coordinates are used to characterize the result of the speech emotion recognition model performing emotion recognition on the target speech data. The emotion recognition coordinates include an horizontal axis and a vertical axis.

[0100] Specifically, the target speech data for which emotion recognition is required is obtained, and the target speech data is input into the speech emotion recognition model. The speech emotion recognition model performs emotion recognition on the target speech data, that is, it performs emotion recognition on the phonemes, gender and emotions in the target speech data according to the speech emotion recognition model, and obtains the emotion recognition coordinates corresponding to the target speech data.

[0101] (2) Obtain a preset emotion compass; the preset emotion compass includes at least one emotion region; each emotion region includes multiple target emotions; each target emotion corresponds to a coordinate range.

[0102] (3) Match the emotion recognition coordinates with all the coordinate ranges.

[0103] (4) The target emotion corresponding to the coordinate range to which the emotion recognition coordinate belongs is determined as the emotion recognition result corresponding to the target speech data.

[0104] Understandably, the preset emotion compass is a compass used to represent the positions of different emotions. That is, the preset emotion compass set in this embodiment is equivalent to a coordinate system. This preset emotion compass includes multiple emotion regions. Each emotion region includes multiple target emotions. Each target emotion corresponds to a coordinate range. The coordinate range represents the position information of the target emotion within the preset emotion compass. That is, this coordinate range has an horizontal coordinate range and a vertical coordinate range. Thus, the horizontal and vertical coordinate ranges define the different position information of each target emotion within the preset emotion compass. The target emotion is a specific emotion categorized by its intensity, such as anger, rage, happiness, or trust. The emotion recognition result is the result used to represent the emotion in the target speech data.

[0105] Specifically, after determining the emotion recognition coordinates of the target speech data, a preset emotion compass can be retrieved from the server. Then, based on the x-coordinate and y-coordinate of the emotion recognition coordinates, the corresponding region is queried from the preset emotion compass. That is, firstly, the x-coordinate of the emotion recognition coordinates is matched with the x-coordinate range of all target emotions, and the coordinate range that matches the x-coordinate of the emotion recognition coordinates is selected. Next, the y-coordinate range of the selected matching coordinate range is matched with the y-coordinate of the emotion recognition coordinates, thus selecting the coordinate range that matches both the x-coordinate and y-coordinate of the target emotion coordinates. Finally, the target emotion corresponding to this selected coordinate range is determined as the emotion recognition result of the target speech data.

[0106] This invention utilizes a speech emotion recognition model to identify target data, thereby determining the coordinates for emotion recognition and facilitating accurate subsequent emotion assessment. By comparing the emotion recognition coordinates with a preset range in an emotion compass, and using the target emotion corresponding to that range as the emotion recognition result, precise identification of emotions in target speech data is achieved.

[0107] In one embodiment, prior to obtaining the preset emotion compass, the following steps are included:

[0108] (1) Obtain a preset emotion set and cluster all the target emotions in the preset emotion set to obtain a first number of emotion cluster groups; an emotion cluster group includes multiple target emotions.

[0109] Understandably, a pre-defined emotion set is constructed by acquiring all target emotions. Emotion clusters are sets of emotions with the same meaning, such as "anger" including anger, resentment, and annoyance in the "anger" emotion cluster, and "joy" including ecstasy, happiness, and happiness in the "joy" emotion cluster.

[0110] Specifically, before obtaining the preset emotion compass, a preset emotion set is acquired, which includes multiple target emotions. All target emotions are clustered using the k-means clustering algorithm. k cluster centers are randomly selected from the preset emotion set. The distance between each target emotion and each cluster center is calculated using Euclidean distance or cosine similarity. Each target emotion is assigned to the nearest cluster center, and this cluster center and the assigned target emotion represent an emotion cluster group. Each time a target emotion is assigned, the cluster center is recalculated based on the existing target emotions in the emotion cluster group. This process is repeated until a termination condition is met. The termination condition could be that no (or a minimum number) target emotions are reassigned to different clusters, no (or a minimum number) cluster centers change, or the sum of squared errors reaches a local minimum. In this way, the first number of emotion cluster groups are obtained.

[0111] (2) Divide the target emotion in each of the emotion clusters into intensity values ​​to obtain the emotion intensity values ​​corresponding to the target emotion.

[0112] Understandably, the emotion intensity value represents the strength of the target emotion. Target emotions within the same emotion cluster may have different intensities. For example, the "joy" emotion cluster may include target emotions such as ecstasy, happiness, or tranquility. Ecstasy has a higher emotion intensity value than happiness; that is, when expressing joy, ecstasy is more intense than happiness.

[0113] Specifically, after obtaining the emotion clusters, the intensity of the target emotions within each cluster is determined. This involves assessing the intensity of the emotions expressed by the target emotions within each cluster and dividing each cluster into N intensity levels. The distance between the target emotion and each intensity level is calculated using Euclidean distance or cosine similarity. The distances between the target emotion and all intensity levels are compared, and the target emotion is assigned to the closest intensity level. This process is repeated for each target emotion within each cluster, and the intensity level to which the target emotion belongs is defined as its emotion intensity value. In this way, the emotion intensity value corresponding to each target emotion is obtained.

[0114] (3) Obtain a preset emotion coordinate system and divide the preset emotion coordinate system into regions to obtain a second number of coordinate regions; the second number is equal to the first number.

[0115] (4) Set all the emotion clusters in the coordinate region to obtain the emotion region; one emotion cluster corresponds to one emotion region.

[0116] Understandably, the preset emotion coordinate system is a coordinate system used to represent different target emotions. The emotion region is the coordinate region corresponding to the emotion cluster group in the preset emotion coordinate system.

[0117] Specifically, the preset emotional coordinate system is divided into regions, such as four major regions based on the positive and negative values ​​of the horizontal and vertical axes. The preset emotional coordinate system can be further divided into regions based on emotional clusters, resulting in a second number of coordinate regions. This second number is equal to the first number. All the obtained emotional clusters are then placed within the coordinate regions according to preset rules, thus creating emotional regions; one emotional cluster corresponds to one emotional region. For example, when there are four emotional clusters, the joy cluster is placed in the region where both the horizontal and vertical axes are positive; the anger cluster is placed in the region where both the horizontal and vertical axes are negative; the sorrow cluster is placed in the region where the horizontal axis is negative and the vertical axis is positive; and the happiness cluster is placed in the region where the horizontal axis is negative and the vertical axis is positive.

[0118] (5) Divide each of the emotional regions into regions to obtain emotional sub-regions; each emotional sub-region corresponds to a coordinate range.

[0119] Understandably, the emotion sub-region is obtained by dividing the emotion region according to different target emotions.

[0120] Specifically, after obtaining the emotion regions, each emotion region is divided into different areas, such as dividing one emotion region into three regions, and these three regions are defined as emotion sub-regions. The range of the region corresponding to each emotion sub-region is defined as a coordinate range, that is, one emotion sub-region corresponds to one coordinate range. The coordinate range corresponding to an emotion region is greater than the coordinate range corresponding to one emotion sub-region, and equal to the sum of the coordinate ranges corresponding to multiple emotion sub-regions. This process of dividing all emotion regions is repeated to obtain the emotion sub-regions corresponding to all emotion regions.

[0121] (6) Set the target emotion in the emotion sub-region according to the emotion intensity value, and after all the target emotions are set in the emotion sub-region, record the preset emotion coordinate system as the preset emotion compass.

[0122] Specifically, after obtaining the emotion regions, the target emotions are set in the emotion sub-regions according to their intensity values. This can be done by setting the intensity values ​​from minimum to maximum, or by setting them from maximum to minimum. All target emotions are set sequentially, and after all target emotions are set in the emotion sub-regions, the preset emotion coordinate system is recorded as a preset emotion compass.

[0123] This invention achieves the acquisition of emotion clusters representing the same meaning by clustering target emotions. By dividing these emotion clusters by intensity, it obtains emotion intensity values, thereby enabling accurate judgment of emotions in speech data. By dividing a preset emotion coordinate system into regions and setting emotion clusters within these regions, it determines the emotion region and its sub-regions. Finally, by setting the target emotion within an emotion sub-region based on its intensity value, it establishes a preset emotion compass.

[0124] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] In one embodiment, a speech emotion recognition model training device is provided, which corresponds one-to-one with the speech emotion recognition model training method described in the above embodiments. For example... Figure 4 As shown, the voice emotion recognition model training device includes a feature acquisition module 11, an input feature module 12, a feature vector module 13, a prediction label module 14, a loss determination module 15, and a model convergence module 16.

[0126] Detailed descriptions of each functional module are as follows:

[0127] The feature acquisition module 11 is used to acquire phoneme speech features, gender speech features, and emotion speech features; the phoneme speech features are associated with phoneme speech tags, the gender speech features are associated with gender speech tags, and the emotion speech features are associated with emotion coordinate tags.

[0128] The input feature module 12 is used to input the phoneme speech features, gender speech features, and emotion speech features into a preset recognition model containing initial parameters; the preset recognition model includes a convolutional pooling layer and multiple residual network layers;

[0129] Feature vector module 13 is used to obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing of the speech feature vector through the convolutional pooling layer; the speech feature vector includes the first feature vector output by the first residual network layer;

[0130] The prediction label module 14 is used to obtain the predicted phoneme label based on the first feature vector output by the first residual network layer; and to obtain the predicted gender label and the predicted emotion label based on the output feature vector.

[0131] The loss determination module 15 is used to determine the predicted loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label and emotion coordinate label.

[0132] The model convergence module 16 is used to iteratively update the initial parameters in the preset recognition model when the predicted loss value does not reach the preset convergence condition, until the predicted loss value reaches the convergence condition, and then record the converged preset recognition model as a speech emotion recognition model.

[0133] In one embodiment, the feature acquisition module 11 includes:

[0134] A data acquisition unit is used to acquire a preset speech training set; the preset speech training set includes at least one speech data of a phoneme to be identified, speech data of a gender to be identified, and speech data of an emotion to be identified.

[0135] The spectrum analysis unit is used to perform spectrum analysis on the speech data of the phoneme to be identified, the speech data of the gender to be identified, and the speech data of the emotion to be identified, to obtain the phoneme spectrum diagram corresponding to the speech data of the phoneme to be identified, the gender spectrum diagram corresponding to the speech data of the gender to be identified, and the emotion spectrum diagram corresponding to the speech data of the emotion to be identified.

[0136] The feature extraction unit is used to extract features from the phoneme spectrogram, the gender spectrogram, and the emotion spectrogram to obtain the phoneme speech features corresponding to the phoneme spectrogram, the gender speech features corresponding to the gender spectrogram, and the emotion speech features corresponding to the emotion spectrogram.

[0137] In one embodiment, the feature vector module 13 includes:

[0138] A convolutional unit is used to perform convolution processing on the phoneme speech features, gender speech features and emotion speech features through the first convolutional layer to obtain a convolutional feature vector.

[0139] The residual unit is used to input the convolutional feature vector into the residual network layer and obtain the residual feature vector output by each residual network layer.

[0140] The speech feature unit is used to perform convolution processing on the residual feature vectors output by all residual network layers through the second convolutional layer to obtain the speech feature vector;

[0141] The pooling unit is used to perform vector processing on the speech feature vector through the attention pooling layer to obtain the output feature vector.

[0142] In one embodiment, the residual unit includes:

[0143] The first residual subunit is used to extract features from the convolutional feature vector through the first residual network layer to obtain the first feature vector;

[0144] The second residual subunit is used to extract features from the first feature vector through the second residual network layer to obtain the second feature vector.

[0145] The third residual subunit is used to extract features from the second feature vector through the third residual network layer to obtain the third feature vector.

[0146] In one embodiment, the loss determination module 15 includes:

[0147] The first loss unit is used to determine a first loss value based on the predicted phoneme label and the phoneme speech label;

[0148] The second loss unit is used to determine a second loss value based on the predicted gender label and the gender voice label;

[0149] The third loss unit is used to determine a third loss value based on the predicted emotion label and the emotion coordinate label;

[0150] The factor acquisition unit is used to acquire a preset weighting factor and determine the prediction loss value of the preset recognition model based on the preset weighting factor, the first loss value, the second loss value, and the third loss value.

[0151] In one embodiment, the model convergence module 16 includes:

[0152] An emotion recognition unit is used to acquire target speech data, perform emotion recognition on the target speech data through the speech emotion recognition model, and obtain the emotion recognition coordinates corresponding to the target speech data.

[0153] An emotion compass unit is used to acquire a preset emotion compass; the preset emotion compass includes at least one emotion region; one emotion region includes multiple target emotions; one target emotion corresponds to a coordinate range.

[0154] A coordinate matching unit is used to match the emotion recognition coordinates with all the coordinate ranges.

[0155] The result determination unit is used to determine the target emotion corresponding to the coordinate range to which the emotion recognition coordinates belong as the emotion recognition result corresponding to the target speech data.

[0156] In one embodiment, the emotion compass unit includes:

[0157] A clustering subunit is used to acquire a preset emotion set and cluster all the target emotions in the preset emotion set to obtain a first number of emotion cluster groups; one emotion cluster group includes multiple target emotions;

[0158] The intensity division subunit is used to divide the target emotion in each of the emotion clusters to obtain the emotion intensity value corresponding to the target emotion.

[0159] A region division subunit is used to obtain a preset emotion coordinate system and divide the preset emotion coordinate system into regions to obtain a second number of coordinate regions; the second number is equal to the first number.

[0160] A region determination subunit is used to set all the emotion clusters in the coordinate region to obtain an emotion region; one emotion cluster corresponds to one emotion region;

[0161] A sub-region division unit is used to divide each of the emotion regions into emotion sub-regions; each emotion sub-region corresponds to a coordinate range.

[0162] The recording unit is used to set the target emotion in the emotion sub-region according to the emotion intensity value, and after all the target emotions are set in the emotion sub-region, record the preset emotion coordinate system as the preset emotion compass.

[0163] Specific limitations regarding the training device for the speech emotion recognition model can be found in the limitations of the training method for the speech emotion recognition model mentioned above, and will not be repeated here. Each module in the aforementioned training device for the speech emotion recognition model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0164] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores the data used in the speech emotion recognition model training method described in the above embodiments. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech emotion recognition model training method.

[0165] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the speech emotion recognition model training method described in the above embodiment.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the speech emotion recognition model training method described in the above embodiment.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0169] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for training a speech emotion recognition model, characterized in that, include: Obtain phoneme speech features, gender speech features, and emotional speech features; the phoneme speech features are associated with phoneme speech tags, the gender speech features are associated with gender speech tags, and the emotional speech features are associated with emotional coordinate tags; The phoneme speech features, gender speech features, and emotion speech features are input into a preset recognition model containing initial parameters; the preset recognition model includes convolutional pooling layers and multiple residual network layers; wherein, the convolutional pooling layers include a second convolutional layer and an attention pooling layer; the residual network layers are obtained by combining Res2Net, bidirectional RNN, and SE-Block; Obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing the speech feature vector through the convolutional pooling layer; the speech feature vector includes the first feature vector output by the first residual network layer. Based on the first residual network layer, the first feature vector is output to obtain the predicted phoneme label; based on the output feature vector, the predicted gender label and the predicted emotion label are obtained. Based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label, and emotion coordinate label, determine the prediction loss value of the preset recognition model; When the predicted loss value does not reach the preset convergence condition, the initial parameters in the preset recognition model are iteratively updated until the predicted loss value reaches the convergence condition. Then, the converged preset recognition model is recorded as a speech emotion recognition model. The step of obtaining the output feature vector of the preset recognition model includes: The phoneme speech features, gender speech features, and emotion speech features are convolved by the first convolutional layer to obtain the convolutional feature vector. The convolutional feature vector is input into the residual network layer, and the residual feature vector output by each residual network layer is obtained. Specifically, the convolutional feature vector is compressed using a 1*1 convolutional matrix, the weight values ​​of each channel are calculated using a 3*3 convolutional matrix, and the original dimension is restored using a 1*1 convolutional matrix. The residual feature vector is obtained by calculating the convolutional feature vector and the weight values ​​of each channel using an SE-Block layer. The speech feature vector is obtained by convolving the residual feature vectors output by all residual network layers through the second convolutional layer. The speech feature vector is processed by an attention pooling layer to obtain the output feature vector.

2. The voice emotion recognition model training method as described in claim 1, characterized in that, Before acquiring phoneme speech features, gender speech features, and emotion speech features, the following steps are included: Obtain a preset speech training set; the preset speech training set contains at least one speech data of a phoneme to be identified, speech data of a gender to be identified, and speech data of an emotion to be identified. Spectral analysis is performed on the speech data of the phonemes to be identified, the speech data of the gender to be identified, and the speech data of the emotions to be identified to obtain the phoneme spectrogram corresponding to the speech data of the phonemes to be identified, the gender spectrogram corresponding to the speech data of the gender to be identified, and the emotion spectrogram corresponding to the speech data of the emotions to be identified. Feature extraction is performed on the phoneme spectrogram, gender spectrogram, and emotion spectrogram to obtain the phoneme speech features corresponding to the phoneme spectrogram, the gender speech features corresponding to the gender spectrogram, and the emotion speech features corresponding to the emotion spectrogram.

3. The speech emotion recognition model training method as described in claim 1, characterized in that, The residual feature vector output by the first residual network layer is the first feature vector; the residual feature vector output by the second residual network layer is the second feature vector; and the residual feature vector output by the third residual network layer is the third feature vector. The step of inputting the convolutional feature vector into the residual network layer and obtaining the residual feature vector output by each residual network layer includes: The first feature vector is obtained by extracting features from the convolutional feature vector through the first residual network layer. The second feature vector is obtained by extracting features from the first feature vector through the second residual network layer. The third feature vector is obtained by extracting features from the second feature vector through the third residual network layer.

4. The voice emotion recognition model training method as described in claim 1, characterized in that, The step of determining the prediction loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label, and emotion coordinate label includes: A first loss value is determined based on the predicted phoneme label and the phoneme speech label; A second loss value is determined based on the predicted gender label and the gender voice label; A third loss value is determined based on the predicted sentiment label and the sentiment coordinate label; Obtain a preset weighting factor, and determine the predicted loss value of the preset recognition model based on the preset weighting factor, the first loss value, the second loss value, and the third loss value.

5. The method for training a speech emotion recognition model as described in claim 1, characterized in that, After recording the converged preset recognition model as a speech emotion recognition model, the process includes: Acquire target speech data, and perform emotion recognition on the target speech data using the speech emotion recognition model to obtain the emotion recognition coordinates corresponding to the target speech data; A preset emotion compass is obtained; the preset emotion compass includes at least one emotion region; each emotion region includes multiple target emotions; each target emotion corresponds to a coordinate range; Match the emotion recognition coordinates with all of the coordinate ranges; The target emotion corresponding to the coordinate range to which the emotion recognition coordinates belong is determined as the emotion recognition result corresponding to the target speech data.

6. The speech emotion recognition model training method as described in claim 5, characterized in that, Before obtaining the preset emotion compass, the following steps are included: A preset emotion set is obtained, and all the target emotions in the preset emotion set are clustered to obtain a first number of emotion cluster groups; each emotion cluster group includes multiple target emotions. The intensity of the target emotion in each of the emotion clusters is divided to obtain the emotion intensity value corresponding to the target emotion. A preset emotion coordinate system is obtained, and the preset emotion coordinate system is divided into regions to obtain a second number of coordinate regions; the second number is equal to the first number. All the emotion clusters are set in the coordinate region to obtain the emotion region; one emotion cluster corresponds to one emotion region. Each of the emotional regions is divided into emotional sub-regions; each emotional sub-region corresponds to a coordinate range. The target emotion is set in the emotion sub-region according to the emotion intensity value, and after all the target emotions are set in the emotion sub-region, the preset emotion coordinate system is recorded as the preset emotion compass.

7. A training device for a speech emotion recognition model, characterized in that, include: The feature acquisition module is used to acquire phoneme speech features, gender speech features, and emotion speech features; the phoneme speech features are associated with phoneme speech tags, the gender speech features are associated with gender speech tags, and the emotion speech features are associated with emotion coordinate tags. The input feature module is used to input the phoneme speech features, gender speech features, and emotion speech features into a preset recognition model containing initial parameters; the preset recognition model includes convolutional pooling layers and multiple residual network layers; wherein, the convolutional pooling layer includes a second convolutional layer and an attention pooling layer; the residual network layer is obtained by combining Res2Net, bidirectional RNN, and SE-Block; The feature vector module is used to obtain the output feature vector of the preset recognition model; the output feature vector is obtained by vector processing of the speech feature vector through the convolutional pooling layer; the speech feature vector includes the first feature vector output by the first residual network layer. The prediction label module is used to obtain predicted phoneme labels based on the first feature vector output by the first residual network layer; and to obtain predicted gender labels and predicted emotion labels based on the output feature vector. The loss determination module is used to determine the predicted loss value of the preset recognition model based on the predicted phoneme label, predicted gender label, predicted emotion label, phoneme speech label, gender speech label and emotion coordinate label; The model convergence module is used to iteratively update the initial parameters in the preset recognition model when the predicted loss value does not reach the preset convergence condition, until the predicted loss value reaches the convergence condition, and then record the converged preset recognition model as a speech emotion recognition model. The feature vector module includes: A convolutional unit is used to perform convolution processing on the phoneme speech features, gender speech features and emotion speech features through the first convolutional layer to obtain a convolutional feature vector. The residual unit is used to input the convolutional feature vector into the residual network layer and obtain the residual feature vector output by each residual network layer; wherein, the convolutional feature vector is compressed by a 1*1 convolution matrix, the weight values ​​of each channel are calculated by a 3*3 convolution matrix, and the original dimension is restored by a 1*1 convolution matrix. The residual feature vector is obtained by calculating the convolutional feature vector and the weight values ​​of each channel through an SE-Block layer. The speech feature unit is used to perform convolution processing on the residual feature vectors output by all residual network layers through the second convolutional layer to obtain the speech feature vector; The pooling unit is used to perform vector processing on the speech feature vector through the attention pooling layer to obtain the output feature vector.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech emotion recognition model training method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech emotion recognition model training method as described in any one of claims 1 to 6.

Images