Speech emotion recognition model training method, emotion recognition method, device and equipment
By obtaining the audio waveform and speaker characteristics for decoupling, the problem of low accuracy of speech emotion recognition is solved, and a more accurate emotion recognition effect is achieved.
Patent Information
- Application Number
- CN202210153023.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-02-18
AI Technical Summary
The existing speech emotion recognition technology has the problem of low accuracy, mainly because different people have different emotions in the same period of speech and cultural differences, resulting in low accuracy in speech emotion recognition.
By obtaining the waveform features of the audio and speaker features, decoupling of emotional features is performed, and the emotional features obtained by decoupling is used for emotional recognition training, and the trained speech emotion recognition model is obtained to improve the accuracy of emotional recognition.
More accurate emotion recognition is achieved, purer emotional characteristics are obtained through the decoupling process, and the recognition accuracy of the speech emotion recognition model is improved.
Smart Images

Figure CN114495915B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, in particular to the fields of intelligent speech recognition and intelligent emotion recognition, and specifically to a speech emotion recognition model training method, emotion recognition method, device and equipment. Background Art
[0002] Speech is a crucial carrier of emotion in human communication. Speech recognition primarily focuses on what the speaker says. Emotion recognition focuses on the speaker's emotional state. People express themselves differently in different emotional states. For example, when happy, their tone of voice is more cheerful, while when upset or sad, their tone of voice is more subdued.
[0003] Deep learning technology has accelerated progress in detecting emotion from speech, but research in this area remains limited. The difficulty with speech emotion detection lies in the fact that the emotion expressed in a sentence can vary from person to person. Different people interpret the same speech differently, and cultural differences exist. Consequently, the accuracy of speech emotion recognition is low. Summary of the Invention
[0004] The present disclosure provides a speech emotion recognition model training method, an emotion recognition method, an apparatus, a device, and a storage medium.
[0005] According to one aspect of the present disclosure, a method for training a speech emotion recognition model is provided, comprising:
[0006] Obtaining a first feature and a second feature of the sample audio, wherein the first feature is used to characterize a feature related to a waveform of the sample audio, and the second feature is used to characterize a feature related to a speaker of the sample audio;
[0007] Decoupling emotion features using the first feature and the second feature;
[0008] The emotion features obtained by decoupling are used for emotion recognition training to obtain a trained speech emotion recognition model.
[0009] According to another aspect of the present disclosure, there is provided an emotion recognition method, comprising:
[0010] Obtaining a first feature and a second feature of the audio to be identified, wherein the first feature is used to characterize a feature related to the waveform of the audio to be identified, and the second feature is used to characterize a feature related to the speaker of the audio to be identified;
[0011] Inputting the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition, thereby obtaining a first recognition result;
[0012] Among them, the speech emotion recognition model is a speech emotion recognition model trained using the training method of the speech emotion recognition model of the embodiment of the present disclosure.
[0013] According to another aspect of the present disclosure, a speech emotion recognition model training device is provided, comprising:
[0014] an acquisition module, configured to acquire a first feature and a second feature of the sample audio, wherein the first feature is used to characterize a feature related to the waveform of the sample audio, and the second feature is used to characterize a feature related to the speaker of the sample audio;
[0015] a decoupling module, configured to decouple emotion features using the first feature and the second feature;
[0016] The training module is used to perform emotion recognition training using the emotion features obtained by decoupling to obtain a trained speech emotion recognition model.
[0017] According to another aspect of the present disclosure, there is provided an emotion recognition device, comprising:
[0018] an acquisition module, configured to acquire a first feature and a second feature of the audio to be recognized, wherein the first feature is used to characterize a feature related to the waveform of the audio to be recognized, and the second feature is used to characterize a feature related to the speaker of the audio to be recognized;
[0019] A first recognition module is configured to input the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition and obtain a first recognition result;
[0020] Among them, the speech emotion recognition model is a speech emotion recognition model trained using the speech emotion recognition device of the embodiment of the present disclosure.
[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0022] at least one processor; and
[0023] a memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.
[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.
[0026] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.
[0027] The present disclosure uses decoupled emotion features for emotion recognition training, and the trained speech emotion recognition model can perform emotion recognition more accurately.
[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.
[0030] Figure 1 1 is a flow chart of a method for training a speech emotion recognition model according to an embodiment of the present disclosure;
[0031] Figure 2 is a flowchart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure;
[0032] Figure 3 is a flowchart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure;
[0033] Figure 4 is a flowchart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure;
[0034] Figure 5 is a flowchart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure;
[0035] Figure 6 is a flowchart of an emotion recognition method according to an embodiment of the present disclosure;
[0036] Figure 7 is a flowchart of an emotion recognition method according to another embodiment of the present disclosure;
[0037] Figure 8 2 is a schematic structural diagram of a speech emotion recognition model training device according to an embodiment of the present disclosure;
[0038] Figure 9 is a structural diagram of a speech emotion recognition model training device according to another embodiment of the present disclosure;
[0039] Figure 10 is a structural diagram of a speech emotion recognition model training device according to another embodiment of the present disclosure;
[0040] Figure 11 is a structural diagram of a speech emotion recognition model training method according to another embodiment of the present disclosure;
[0041] Figure 12 is a structural diagram of an emotion recognition device according to an embodiment of the present disclosure;
[0042] Figure 13 is a structural diagram of an emotion recognition device according to another embodiment of the present disclosure;
[0043] Figure 14 is a schematic diagram of an example of a speech emotion recognition process;
[0044] Figure 15 is a schematic diagram of a speech emotion recognition framework according to an embodiment of the present disclosure;
[0045] Figure 16 is a schematic diagram of a speech emotion recognition process according to an embodiment of the present disclosure;
[0046] Figure 17 is a schematic block diagram of an example electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0047] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0048] Figure 1 : is a flow chart of a method for training a speech emotion recognition model according to an embodiment of the present disclosure, the method comprising:
[0049] S101, obtaining a first feature and a second feature of a sample audio, wherein the first feature is used to characterize a feature related to a waveform of the sample audio, and the second feature is used to characterize a feature related to a speaker of the sample audio;
[0050] S102, using the first feature and the second feature to perform emotion feature decoupling;
[0051] S103. Perform emotion recognition training using the emotion features obtained by decoupling to obtain a trained speech emotion recognition model.
[0052] In the disclosed embodiments, audio may reflect a variety of speaker emotions, including anger, happiness, neutrality, sadness, excitement, fear, and so on. Emotions may also be referred to as feelings, emotions, and so on. Speakers are generally humans, but may also include other types of speakers, such as AI robots. Audio may include multiple types of features. In the disclosed embodiments, the first feature may represent features related to the waveform of the audio itself, and may also be referred to as waveform features or sound wave features. The second feature may represent features related to the speaker in the audio, and may also be referred to as speaker features or speaker spectral features. In the disclosed embodiments, the first and second features of the sample audio can be decoupled from the emotion features, resulting in a purer emotion feature. Using the decoupled emotion features for emotion recognition training, the trained speech emotion recognition model can more accurately recognize emotions. Using the decoupled emotion features for emotion recognition training decouples emotion features, speaker features, and content features, resulting in purer emotion features. The trained speech emotion recognition model can then be used for recognition and classification, and combined with the text recognition results to determine the emotion type of the audio, resulting in a more accurate judgment result.
[0053] The speech emotion recognition model training method of the present application embodiment can be executed by a terminal, server, or other processing device in a single-machine, multi-machine, or cluster system. The terminal may include but is not limited to user equipment, mobile devices, personal digital assistants, handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. The server may include but is not limited to application servers, data servers, cloud servers, etc.
[0054] Figure 2 It is a flow chart of a speech emotion recognition model training method according to another embodiment of the present disclosure, and the method of this embodiment includes one or more features of the above-mentioned speech emotion recognition model training method embodiment. In one possible implementation, S101 of the method includes: obtaining a first feature of the sample audio. In the embodiment of the present disclosure, the audio may have a variety of attributes that represent the audio waveform, including but not limited to pitch, volume, speaking speed, stress, etc. For example, a multidimensional vector can be extracted from the audio, and the eigenvalues in the vector can represent the first feature of the audio. There is no restriction on the dimension of the vector, and it can be selected according to the actual scenario. For example, it can be 512 dimensions, 1024 dimensions or other values.
[0055] In a possible implementation, obtaining the first feature of the sample audio includes:
[0056] S201. Utilize the waveform to vector (Wav2vec) model to extract the first feature from the sample audio. For example, the Wav2vec model can be a pre-trained model. The Wav2vec model can include multiple layers of transformer layers (TransformerLayer), and the specific number of transformer layers can be set as needed. A large amount of unlabeled audio data is used to perform unsupervised training on the Wav2vec model, and after multiple rounds of iterative convergence, a hierarchical feature extractor can be obtained. The features of each layer can deeply characterize the different properties of the audio from different angles. The Wav2vec model can be used to extract as many features as possible from the waveform of the audio, providing a rich data foundation for training.
[0057] In one possible implementation, S201 extracts the first feature from the sample audio using a Wav2vec model, including: framing the sample audio to obtain a plurality of first audio frames; extracting at least one audio segment from the plurality of first audio frames and inputting the segment into the Wav2vec model to obtain the first feature, where the first feature includes a Wav2vec feature of the audio segment. The audio segment may include a first number of first audio frames.
[0058] For example, after the sample audio is divided into frames to obtain multiple first audio frames, an audio segment can be extracted from these first audio frames each time, and the audio segment can include multiple first audio frames. For example, if the entire audio is divided into 960 first audio frames, if each audio segment includes 96 first audio frames, 10 audio segments can be obtained. If each audio segment includes 48 first audio frames, 20 audio segments can be obtained. Each time training, one audio segment can be input into the model to be trained, and the trained model can be obtained through multiple iterations. For example, the first audio segment is input for the first time, and the second audio segment is input for the second time, until all audio segments are input or the convergence result of the model meets the requirements.
[0059] In addition, the sample audio may also include multiple ones, and for each sample audio, the process of framing, extracting audio segments, and iterative training may be repeated.
[0060] In the disclosed embodiment, training through audio segmentation can control the amount of data for each training, achieve the purpose of rapid convergence, and facilitate more effective use of samples.
[0061] In one possible implementation, S101 of the method includes: obtaining a second feature of the sample audio. In the disclosed embodiment, the attributes in the audio that can represent the characteristics of the speaker's voice include, but are not limited to, characteristics such as vocal cord vibration frequency, mouth size, lip thickness, tongue position, and nasopharyngeal thickness. For example, a multidimensional vector can be extracted from the audio, and the eigenvalues in the vector can represent the second feature of the audio. The dimension of the vector is not limited and can be selected according to the actual scenario, for example, it can be 128 dimensions, 256 dimensions, or other values.
[0062] In one possible implementation, obtaining a second feature of the sample audio includes: S202, extracting the second feature from the sample audio using a speaker classification model. For example, the speaker classification model can be referred to as a speaker feature extraction model, which may include an LSTM (Long-Short Term Memory) layer, a linear mapping layer (which may be simply referred to as a linear layer), and a fully connected layer. The input of the model may include features of each frame of audio, and the output may include predicted probabilities of each speaker. For example, the type of input audio features may include, but is not limited to, at least one of MFCC (Mel-Frequency Cepstral Coefficients), PLP (Perceptual Linear Predictive), or Fbank (FilterBank-based features). The output features may include speaker identity embeddings, etc. The speaker classification model may also be a pre-trained model. The speaker classification model can be used to extract features related to speaker roles, thereby facilitating differentiation between different speakers and providing personalized and accurate emotion recognition.
[0063] In a possible implementation, S202 extracts the second feature from the sample audio using a speaker classification model, including: framing the sample audio to obtain multiple second audio frames; and inputting the multiple second audio frames into the speaker classification model to obtain the second feature.
[0064] For example, the second feature may include a multidimensional vector. The vector may characterize the characteristics of the speaker. For example, a portion of the dimensions of the vector may represent the frequency of vocal cord vibration, a portion of the dimensions may represent the size of the oral cavity, a portion of the dimensions may represent the thickness of the lips, a portion of the dimensions may represent the position of the tongue, and a portion of the dimensions may represent the thickness of the nasopharyngeal cavity. The second feature may also be referred to as the speaker's spectral feature. The speaker classification model may calculate the probability of the speaker corresponding to each frame separately, and then obtain the second feature by performing multi-frame accumulation and averaging on the second feature. For example, the average value of the vector may be calculated every 50 frames. When calculating the probability of the speaker corresponding to each frame, a certain amount of context may also be included. For example, when calculating the first frame, the first and second frames are input. For another example, when calculating the second frame, the first, second, and third frames are input. For another example, when calculating the third frame, the third, fourth, and fifth frames are input. The specific number and method of context used may be set according to the actual application and are not limited in the embodiments of the present disclosure.
[0065] In the embodiment of the present disclosure, a speaker classification model is trained by using multiple audio frames in the audio, so as to obtain a model that outputs more accurate speaker features.
[0066] In the embodiment of the present disclosure, obtaining the first feature of the sample audio and the second feature of the sample audio can be performed separately. This embodiment does not limit the execution sequence of the two, and they can be performed in sequence or in parallel.
[0067] Figure 3 This is a flow chart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure. The method of this embodiment includes one or more features of the above-mentioned embodiment of the method for training a speech emotion recognition model. In one possible implementation, S102 utilizes the first feature and the second feature to perform emotion feature decoupling, further comprising:
[0068] S301: Input the first feature and the second feature into an encoder for encoding to achieve emotion feature decoupling. The encoder can extract emotion features based on the first feature and the second feature, thereby more accurately training emotion recognition.
[0069] In one possible embodiment, the encoder includes a weight averaging layer, a first connection layer, a first convolutional regularization layer, a BLSTM (bidirectional LSTM) layer, and a downsampling layer, and the first feature and the second feature are input into the encoder for encoding processing, including: inputting the first feature into the weight averaging layer; inputting the second feature into the first connection layer; splicing the output feature of the weight averaging layer and the second feature in the first connection layer to obtain a first spliced feature; inputting the first spliced feature into the first convolutional regularization layer, the BLSTM layer, and the downsampling layer in series for processing in sequence to obtain the output feature of the encoder, and the output feature of the encoder includes an emotional feature. Specifically, the first spliced feature can be input into the first convolutional regularization layer for processing, the output feature of the first convolutional regularization layer can be input into the BLSTM layer for processing, and the output feature of the BLSTM layer can be input into the downsampling layer for processing to obtain the output feature of the encoder.
[0070] For example, the input of the encoder may include features extracted from an audio clip (assuming it includes H frames) through a Wav2vec model, and the output may include emotional features. The feature dimensions of each frame in the audio clip are then vector-weighted using learnable coefficients, such as weighted averaging of the vectors of each frame in the multi-layer (e.g., 25-layer) transformer of the Wav2vec model. The coefficient of each vector can represent the importance of each feature, the initial value of the coefficient can be set in advance, and the value of the coefficient can be updated during training. After obtaining the weighted average structure result of each frame, it can be spliced with the second feature in the first connection layer, and then input into the convolution regularization layer, and then input into the BLSTM layer to obtain H 2*d dimensional vectors. The vector is then downsampled.
[0071] In the embodiment of the present disclosure, after the first feature is weighted averaged by the weight averaging layer of the encoder, it is spliced with the second feature in the first connection layer, and then processed by the first convolution regularization layer, the BLSTM layer and downsampling. The input features can be decoupled, the speaker features and text content features are omitted, and the emotional features are decomposed, so that the emotion recognition training can be performed more accurately.
[0072] Figure 4 It is a flow chart of a speech emotion recognition model training method according to another embodiment of the present disclosure. The method of this embodiment includes one or more features of the above-mentioned speech emotion recognition model training method embodiment.
[0073] In one possible implementation, the method further includes:
[0074] S401: Extract phoneme features from the sample audio using a phoneme alignment model.
[0075] In an embodiment of the present disclosure, a phoneme alignment model extracts phoneme features from an audio clip. The audio clip may be similar to the audio clip used by the above-mentioned Wav2vec model. For example, an audio clip including a first number of first audio frames is input into the Wav2vec model to extract the first feature and also input into the phoneme alignment model to extract the phoneme features. The phoneme alignment model may also be referred to as a phoneme encoder. The phoneme alignment model may be a pre-trained model. The phoneme alignment model may adopt network structures such as GMM (Gaussian Mixed Model)-HMM (Hidden Markov Model), DNN (Deep Neural Networks)-HMM, LSTM-CTC (Connectionist Temporal Classification), RNN-T (Transformer based on Recurrent Neural Network), etc. The phoneme alignment model can extract the phoneme features of the sample audio, which is beneficial for the subsequent reconstruction of the audio using the phoneme features and the second features, thereby improving the decoupling effect.
[0076] In the embodiment of the present disclosure, there is no restriction on the execution sequence of extracting the phoneme feature, the first feature, and the second feature from the sample audio, and they can be performed in sequence or in parallel.
[0077] In a possible implementation, S102 uses the first feature and the second feature to perform emotion feature decoupling, further comprising:
[0078] S402: input the output feature of the encoder, the second feature, and the phoneme feature into a decoder for decoding;
[0079] S403: Utilize the output features of the decoder to update the parameters of at least one of the encoder, the decoder, and the phoneme alignment model.
[0080] In the disclosed embodiments, the decoder processes the encoder's output features, the second features, and the phoneme features to reconstruct the audio and use the differences between the reconstructed audio and the original sample audio to update the encoder's parameters. Furthermore, the decoder and / or phoneme alignment model parameters can also be updated. In this way, the decoder and phoneme alignment model can be used to update model parameters, further improving the purity of the emotional features output by the encoder.
[0081] In one possible implementation, the decoder includes an upsampling layer, a second connection layer, an LSTM layer, a first linear layer, and a second convolutional regularization layer. S401 inputs the output feature of the encoder, the second feature, and the phoneme feature into the decoder for decoding processing, including:
[0082] Inputting the output features of the encoder into the upsampling layer for upsampling;
[0083] Inputting the second feature and the phoneme feature into the second connection layer;
[0084] Concatenating the upsampled feature, the second feature, and the phoneme feature at the second connection layer to obtain a second concatenated feature;
[0085] The second concatenated feature is input into the LSTM layer and the first linear layer connected in series for processing in sequence to obtain a first mel spectrum feature and a first error. Specifically, the second concatenated feature can be input into the LSTM layer for processing, and the output feature of the LSTM layer can be input into the first linear layer for processing to obtain a first mel spectrum feature and a first error.
[0086] The first mel spectrum feature is input into the second convolution regularization layer for convolution regularization processing to obtain a second mel spectrum feature and a second error.
[0087] For example, the input of the decoder may include the output features of the encoder, and the upsampling is repeated multiple times to restore the dimension of the vector, for example, upsampling to obtain H 2*d dimensional vectors. Then, the features obtained by upsampling are spliced with the second features and the output features of the phoneme alignment model and then input into the network module of the subsequent encoder. For example, H M-dimensional features F1 are obtained in the linear layer of the encoder, and the feature F1 can represent the sound spectrum of the audio clip (including the first number of first audio frames), such as the Mel spectrum. Then, the first error between the feature F1 and the true Mel spectrum is calculated, such as the minimum mean square error L r1 The feature F1 is input into the subsequent convolution layer to obtain a new Mel spectrum, and the second error between the feature F1 and the true Mel spectrum is calculated, such as the minimum mean square error L r2 .
[0088] In the embodiment of the present disclosure, the upsampling layer of the decoder can be used to restore the vector dimension of the encoding vector output feature, and spliced with the phoneme feature and the second feature in the connection layer, and then the mel spectrum of the audio clip can be reconstructed through the subsequent network module of the encoder, so as to update the model parameters according to the error between the reconstructed mel spectrum and the true mel spectrum of the sample audio, thereby further improving the purity of the emotional features output by the encoder.
[0089] In one possible implementation, S402 uses the output features of the decoder to update parameters of at least one of the encoder, the decoder, and the phoneme alignment model, including:
[0090] The first error and the second error are used to update the parameters of at least one of the encoder, the decoder, and the phoneme alignment model according to the stochastic gradient descent criterion. In the embodiment of the present disclosure, the model parameters can be updated using each training sample, i.e., the sample audio, based on the stochastic gradient descent criterion. The model parameters are updated once each time the stochastic gradient descent is executed, which is faster and can quickly and accurately achieve model convergence. In the embodiment of the present disclosure, the parameter updates of the encoder, decoder, and phoneme alignment model can be achieved through multiple iterations. The embodiment of the present disclosure does not limit the specific number of iterations. After the training target is reached, the training can be stopped.
[0091] Figure 5 This is a flow chart of a method for training a speech emotion recognition model according to another embodiment of the present disclosure. The method of this embodiment includes one or more features of the aforementioned embodiments of the method for training a speech emotion recognition model. In one possible implementation, S103 uses the decoupled emotion features to perform emotion recognition training, and further includes:
[0092] S501: Input the output features of the encoder into an emotion recognition classifier to perform emotion category recognition, and use the emotion category recognition result to update the parameters of the encoder and the emotion recognition classifier.
[0093] In the disclosed embodiment, S501 may be performed after S301. The emotion recognition classifier may also be referred to as an emotion recognition classification model, an emotion recognition classification network, etc. The output features of the encoder may include decoupled emotion features. After inputting the output features of the encoder into the emotion recognition classifier, an emotion classification result may be obtained through an activation function such as softmax. By using the emotion category recognition results of purer emotion features to update the parameters of the encoder and emotion recognition classifier, the accuracy of the speech emotion recognition results can be improved.
[0094] In one possible implementation, the emotion recognition classifier includes a second linear layer, a dropout layer, and a third linear layer. S501 inputs the output features of the encoder into the emotion recognition classifier to perform emotion category recognition, and uses the emotion category recognition result to update the parameters of the encoder and the emotion recognition classifier, including:
[0095] Inputting the output features of the encoder into the second linear layer, the dropout layer, and the third linear layer connected in series for processing in sequence to obtain an emotion recognition result; specifically, the output features of the encoder can be input into the second linear layer for processing, the output features of the second linear layer can be input into the dropout layer for processing, and the output features of the dropout layer can be input into the third linear layer for processing to obtain an emotion recognition result;
[0096] The loss is calculated using the cross entropy of the emotion recognition result;
[0097] The loss is used to update parameters of the encoder and / or the emotion recognition classifier.
[0098] In the disclosed embodiments, the emotion classification results may include the probability corresponding to each emotion. Assuming the audio consists of N segments, the probabilities of these N segments can be averaged to obtain the emotion classification result for the entire audio. Furthermore, a loss function can be constructed using cross-entropy, and the classifier parameters can be adjusted using this loss function to increase the probability of the target emotion and decrease the probabilities of other emotions.
[0099] In the disclosed embodiments, the emotion recognition classifier can be trained through multiple iterations. The disclosed embodiments do not limit the number of iterations, and training can be stopped after reaching the training target. By using the emotion recognition classifier to identify emotion categories based on purer emotional features and using cross-entropy to update the parameters of the encoder and emotion recognition classifier, a speech emotion recognition model with more accurate recognition results can be obtained.
[0100] Figure 6 FIG. 1 is a flow chart of an emotion recognition method according to an embodiment of the present disclosure, the method comprising:
[0101] S601: Acquire a first feature and a second feature of audio to be recognized, wherein the first feature is used to characterize a feature related to a waveform of the audio to be recognized, and the second feature is used to characterize a feature related to a speaker of the audio to be recognized;
[0102] S602: Input the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition, and obtain a first recognition result;
[0103] The speech emotion recognition model is a speech emotion recognition model trained using a speech emotion recognition model training method.
[0104] For example, the audio to be recognized may include test audio. A first feature related to the waveform and a second feature related to the speaker are extracted from the audio to be recognized. These first and second features are input into a speech emotion recognition model. The encoder of the speech emotion recognition model can be used to obtain decoupled emotion features. A classifier is then used to classify the emotion features, thereby accurately identifying the emotion category.
[0105] In the embodiment of the present disclosure, obtaining the first feature of the audio to be identified and obtaining the second feature of the audio to be identified can be performed separately. This embodiment does not limit the execution sequence of the two, and they can be performed in sequence or in parallel.
[0106] Figure 7 It is a flow chart of an emotion recognition method according to another embodiment of the present disclosure, and the method of this embodiment includes one or more features of the above-mentioned emotion recognition method embodiment. In one possible implementation, in S601, obtaining the first feature of the audio to be recognized includes: extracting the first feature from the audio to be recognized using a Wav2vec model. The specific structure and function of the Wav2vec model can be found in the relevant description of the Wav2vec model in the above-mentioned training method embodiment. The Wav2vec model can be used to extract as many features as possible from the audio waveform, providing a rich data basis for emotion recognition.
[0107] In one possible implementation, extracting the first feature from the audio to be recognized using a Wav2vec model includes: segmenting the audio to be recognized to obtain multiple first audio frames; extracting at least one audio segment from the multiple first audio frames and inputting the segment into the Wav2vec model to obtain the first feature, where the first feature includes the Wav2vec feature of the audio segment. The audio segment may include a first number of first audio frames. For example, after segmenting the audio to be recognized to obtain multiple first audio frames, an audio segment may be extracted from each of the first audio frames, each of which may include multiple first audio frames. Each audio segment is input into the Wav2vec model to obtain the Wav2vec feature of the segment. The Wav2vec feature is then input into the model's encoder and combined with speaker features for speech emotion recognition to obtain a speech emotion recognition result. The recognition results of the multiple audio segments are then averaged to obtain an emotion recognition result for the entire audio segment. The emotion recognition result may also be referred to as an emotion classification result. Using multiple audio segments for emotion recognition can combine the recognition results of each segment to achieve a more accurate recognition result.
[0108] In one possible implementation, in S601, obtaining the second feature of the audio to be recognized includes extracting the second feature from the audio to be recognized using a speaker classification model. The specific structure and function of the speaker classification model can be found in the description of the speaker classification model in the training method embodiment described above. The features extracted using the speaker classification model and the Wav2vec features can decouple emotion features, improve the purity of the emotion features, and thus obtain more accurate emotion recognition results.
[0109] In one possible implementation, extracting the second feature from the audio to be identified using a speaker classification model includes: framing the audio to be identified to obtain multiple second audio frames; and inputting the multiple second audio frames into the speaker classification model to obtain the second feature. For example, the second feature may include a multidimensional vector. This vector may characterize the characteristics of the speaker. The speaker classification model may calculate the probability of the speaker corresponding to each frame, then accumulate and average the multiple frames to obtain the second feature. The calculation of the speaker probability corresponding to each frame may also include a certain amount of context.
[0110] In the embodiment of the present disclosure, more accurate speaker features can be obtained by classifying multiple audio frames in the audio.
[0111] In one possible implementation, the method further includes:
[0112] S701, using a text emotion recognition model to perform text recognition on the text content corresponding to the audio to be recognized, to obtain a second recognition result;
[0113] S702: Perform weighted processing on the first recognition result and the second recognition result to obtain a third recognition result.
[0114] In the disclosed embodiments, the text emotion recognition model may also be referred to as a text emotion classification model, a text emotion classifier, etc. The text emotion recognition model may include: a CNN (Convolutional Neural Networks) layer, a mean pooling layer (Mean Pooling), a flattening layer (Flatten) and a linear layer, etc. The input of the text emotion classification model may include a feature vector corresponding to each word in the text content of the audio to be recognized. The feature vector may be obtained by pre-training the Transformer network. The loss function may be calculated using cross entropy and may be based on training of the entire text content. Combining the speech emotion recognition results with the text emotion recognition results can obtain more accurate emotion recognition results.
[0115] The emotion recognition method of the embodiment of the present application and the speech emotion recognition model training method of the embodiment of the present application can be executed by a terminal, server or other processing device in a single machine, multi-machine or cluster system. Among them, the terminal may include but is not limited to user equipment, mobile devices, personal digital processing, handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. The server may include but is not limited to application servers, data servers, cloud servers, etc. The devices that execute the speech emotion recognition model training method and the emotion recognition method can be the same device or different devices, and the specific selection can be based on the needs of the actual application scenario.
[0116] Figure 8 : is a structural diagram of a speech emotion recognition model training device according to an embodiment of the present disclosure, the device comprising:
[0117] An acquisition module 801 is configured to acquire a first feature and a second feature of a sample audio, wherein the first feature is used to characterize a feature related to a waveform of the sample audio, and the second feature is used to characterize a feature related to a speaker of the sample audio;
[0118] A decoupling module 802 is configured to decouple emotion features using the first feature and the second feature;
[0119] The training module 803 is used to perform emotion recognition training using the emotion features obtained by decoupling to obtain a trained speech emotion recognition model.
[0120] Figure 9 8 is a schematic diagram of the structure of a speech emotion recognition model training apparatus according to another embodiment of the present disclosure, which includes one or more features of the aforementioned speech emotion recognition model training apparatus embodiment. In one possible implementation, the acquisition module 801 is configured to extract the first feature from the sample audio using a waveform-to-vector (Wav2vec) model.
[0121] In a possible implementation, the acquisition module 801 includes:
[0122] A first framing submodule 9011 is configured to frame the sample audio to obtain a plurality of first audio frames;
[0123] The first feature extraction submodule 9012 is configured to extract at least one audio segment from the plurality of first audio frames and input the extracted audio segment into the Wav2vec model to obtain a first feature, wherein the first feature includes a Wav2vec feature of the audio segment. The audio segment may include a first number of first audio frames.
[0124] In a possible implementation, the acquisition module 801 is configured to extract the second feature from the sample audio using a speaker classification model.
[0125] In a possible implementation, the acquisition module 801 includes:
[0126] A second framing submodule 9021 is configured to frame the sample audio to obtain a plurality of second audio frames;
[0127] The second feature extraction submodule 9022 is configured to input the plurality of second audio frames into the speaker classification model to obtain the second feature.
[0128] Figure 10 is a schematic diagram of the structure of a speech emotion recognition model training device according to another embodiment of the present disclosure, which includes one or more features of the speech emotion recognition model training device embodiment described above. In one possible implementation, the decoupling module 802 includes:
[0129] The encoding submodule 1001 is configured to input the first feature and the second feature into an encoder for encoding processing to achieve emotional feature decoupling.
[0130] In one possible implementation, the encoder includes a weight averaging layer, a first connection layer, a first convolutional regularization layer, a BLSTM layer, and a downsampling layer. The encoding submodule 1001 is used to:
[0131] Inputting the first feature into the weighted average layer;
[0132] inputting the second feature into the first connection layer;
[0133] Concatenate the output feature of the weighted average layer and the second feature in the first connection layer to obtain a first concatenated feature;
[0134] The first concatenated features are input into the first convolutional regularization layer, the BLSTM layer, and the downsampling layer connected in series for processing in sequence to obtain output features of the encoder, which include emotion features. Specifically, the first concatenated features can be input into the first convolutional regularization layer for processing, the output features of the first convolutional regularization layer can be input into the BLSTM layer for processing, and the output features of the BLSTM layer can be input into the downsampling layer for processing to obtain the output features of the encoder.
[0135] Figure 11 It is a structural diagram of a speech emotion recognition model training device according to another embodiment of the present disclosure. The device of this embodiment includes one or more features of the above-mentioned speech emotion recognition model training device embodiment.
[0136] In one possible implementation, the device further includes:
[0137] The phoneme feature extraction module 804 is configured to extract phoneme features from the sample audio using a phoneme alignment model.
[0138] In a possible implementation, the decoupling module 802 includes:
[0139] A decoding submodule 1101 is configured to input the output feature of the encoder, the second feature, and the phoneme feature into a decoder for decoding;
[0140] The updating submodule 1102 is configured to update parameters of at least one of the encoder, the decoder, and the phoneme alignment model using the output features of the decoder.
[0141] In one possible implementation, the decoder includes an upsampling layer, a second connection layer, an LSTM layer, a first linear layer, and a second convolutional regularization layer, and the decoding submodule 1101 is used to:
[0142] Inputting the output features of the encoder into the upsampling layer of the decoder for upsampling;
[0143] Inputting the second feature and the phoneme feature into the second connection layer;
[0144] Concatenating the upsampled feature, the second feature, and the phoneme feature at the second connection layer to obtain a second concatenated feature;
[0145] The second concatenated feature is input into the LSTM layer and the first linear layer connected in series for processing in sequence to obtain a first mel spectrum feature and a first error. Specifically, the second concatenated feature can be input into the LSTM layer for processing, and the output feature of the LSTM layer can be input into the first linear layer for processing to obtain a first mel spectrum feature and a first error.
[0146] The first mel spectrum feature is input into the second convolution regularization layer for convolution regularization processing to obtain a second mel spectrum feature and a second error.
[0147] In a possible implementation, the updating submodule 1102 is used to:
[0148] The first error and the second error are used to update parameters of at least one of the encoder, the decoder, and the phoneme alignment model according to a stochastic gradient descent criterion.
[0149] In a possible implementation, the training module 803 is configured to input the output features of the encoder into an emotion recognition classifier for emotion category recognition, and update parameters of the encoder and the emotion recognition classifier using the emotion category recognition result.
[0150] In one possible implementation, the emotion recognition classifier includes a second linear layer, a dropout layer, and a third linear layer. The output features of the encoder are input into the emotion recognition classifier for emotion category recognition. The training module 803 includes:
[0151] The emotion recognition submodule 1103 is configured to input the output features of the encoder into the second linear layer, the dropout layer, and the third linear layer connected in series for processing in sequence to obtain an emotion recognition result. Specifically, the output features of the encoder may be input into the second linear layer for processing, the output features of the second linear layer may be input into the dropout layer for processing, and the output features of the dropout layer may be input into the third linear layer for processing to obtain an emotion recognition result.
[0152] A loss calculation submodule 1104 is configured to calculate the loss using the cross entropy of the emotion recognition result;
[0153] The parameter updating submodule 1105 is configured to update the parameters of the encoder and / or the emotion recognition classifier using the loss.
[0154] For the description of the specific functions and examples of each module and sub-module of the speech emotion recognition model training device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above-mentioned speech emotion recognition model training method embodiment, which will not be repeated here.
[0155] Figure 12 FIG. 1 is a schematic structural diagram of an emotion recognition device according to an embodiment of the present disclosure, the device comprising:
[0156] An acquisition module 1201 is configured to acquire a first feature and a second feature of the audio to be recognized, wherein the first feature is used to characterize a feature related to the waveform of the audio to be recognized, and the second feature is used to characterize a feature related to the speaker of the audio to be recognized;
[0157] A first recognition module 1202 is configured to input the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition and obtain a first recognition result;
[0158] The speech emotion recognition model is a speech emotion recognition model trained by a speech emotion recognition model training device.
[0159] Figure 13 FIG2 is a schematic diagram of the structure of an emotion recognition device according to another embodiment of the present disclosure, wherein the device of this embodiment includes one or more features of the above-mentioned emotion recognition device embodiment. In one possible implementation, the acquisition module 1201 is configured to extract the first feature from the audio to be recognized using a waveform-to-vector (Wav2vec) model.
[0160] In a possible implementation, the acquisition module 1201 includes:
[0161] A first framing submodule 1301 is configured to frame the audio to be recognized to obtain a plurality of first audio frames;
[0162] The first extraction submodule 1302 is configured to extract at least one audio segment from the plurality of first audio frames and input the extracted audio segment into the Wav2vec model to obtain the first feature, wherein the first feature includes the Wav2vec feature of the audio segment. The audio segment may include a first number of first audio frames.
[0163] In a possible implementation, the acquisition module 1201 is configured to extract the second feature from the audio to be recognized using a speaker classification model.
[0164] In a possible implementation, the acquisition module 1201 includes:
[0165] The second framing submodule 1303 is configured to frame the audio to be recognized to obtain a plurality of second audio frames;
[0166] The second extraction submodule 1304 is configured to input the plurality of second audio frames into the speaker classification model to obtain the second feature.
[0167] In one possible implementation, the device further includes:
[0168] The second recognition module 1203 is configured to perform text recognition on the text content corresponding to the audio to be recognized using a text emotion recognition model to obtain a second recognition result;
[0169] The processing module 1204 is configured to perform weighted processing on the first recognition result and the second recognition result to obtain a third recognition result.
[0170] For the description of specific functions and examples of each module and submodule of the emotion recognition device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above-mentioned emotion recognition method embodiment, which will not be repeated here.
[0171] In some application scenarios, such as Figure 14As shown, an example of a speech emotion recognition process includes: extracting front-end features of speech data. For example, extracting MFCC, PLP, Fbank, FFT (fast Fourier transform), etc. Then, using speech recognition models such as GMM, DNN, CNN, ResNet (Residual Neural Network), SincNet (optimized convolutional neural network) to extract emotion features from the front-end features. Then, the emotion features are input into the back-end classifier such as COS (cosine) classifier, SVM (Support Vector Machines) classifier, LDA (Linear Discriminant Analysis) classifier to obtain the recognition results. Generally, the accuracy of the recognition results can be improved by optimizing the front-end feature extraction, optimizing the emotion feature model, adding more training data, etc.
[0172] Optimize front-end feature extraction: Try different feature combinations, such as MFCC and PLP, or varying the dimension, such as from 40 to 80. Optimize emotion feature models: For example, use traditional HMM and SVM models for classification, or deep learning models such as DNN, CNN, and LSTM for classification and recognition. Furthermore, adding more training data can improve coverage.
[0173] Typically, speech emotion, from an audio perspective, is reflected in multiple dimensions, such as pitch, volume, speaking speed, and emphasis, and has no direct relationship to the speaker or the content. For example, someone, whether they are angry or not, may say anything.
[0174] From the perspective of decoupling, the embodiments of the present disclosure divide the audio features into emotion features, speaker features, content features, etc., which helps to accurately extract emotion features.
[0175] In addition, the text content of the audio may reflect certain emotions. For example, "I love this down jacket so much. It feels so comfortable." The text shows that the speaker is happy. The disclosed embodiment can achieve multi-dimensional integration and use text to assist in emotion recognition.
[0176] The present disclosure provides a multimodal (audio and text) emotion recognition method and system based on disentangled representation learning, which can recognize emotion types including but not limited to anger, happiness, neutrality, sadness, etc. Figure 15The system mainly includes: 1) audio feature decoupling module; 2) audio emotion classification module (Classifier); 3) text emotion classification module. Among them, the audio feature decoupling module may include an encoder (Encoder), a decoder (Decoder) and a phoneme encoder (Phone Encoder), etc., which model the features by disassembling the emotion features, speaker features and content features and synthesizing and reconstructing the original audio. In this way, the three features can be stripped more thoroughly to obtain pure emotion features. Then recognition and classification are performed, and the emotion type of the audio is jointly determined in combination with the information assistance of the text. See below Figure 15 This section introduces the functions of each module of the Speech Emotion Recognition Model (SER model) in detail:
[0177] 1: Audio feature decoupling module
[0178] a: Wav2vec feature extraction module
[0179] For example, Wav2vec features can be obtained from a pre-trained model. A pre-trained model can include multiple layers of transformer layers. For example, {Transformer Layer*25} represents a pre-trained model with 25 transformer layers. The specific number of transformer layers can be set as needed. Using a large amount of audio data, such as LibriVox (an open speech dataset), approximately 60,000 hours of unlabeled data, the model is trained unsupervised. After multiple rounds of iterative convergence, a hierarchical feature extractor is obtained. For each frame of audio (40ms duration, 20ms frame shift), a 25*1024-dimensional vector is obtained (25 layers in total, each with a 1024-dimensional output). The features of each layer can deeply characterize different properties of the audio from different perspectives. The pre-trained model can include features of various dimensions to extract the full range of features of the audio waveform. If 96 frames are extracted at a time, the Wav2vec feature dimensions can be expressed as [25, 1024, 96]. The above 1024 dimensions and 96 frames are merely examples and are not limiting, and other numbers are also possible.
[0180] b: Speaker Identity module
[0181] For example, the speaker feature extraction model may include multiple LSTM layers, linear mapping layers, and full connection (FC) layers. The output of the speaker feature extraction model includes the predicted probability of each speaker, and the input may include the features of each frame of audio. The type of input audio features may include, but is not limited to, at least one of MFCC (Mel-Frequency Cepstral Coefficients), PLP (Perceptual linear predictive), or Fbank (FilterBank, filter bank-based features). For example, the dimension of the audio feature may be 20, with a frame length of 25ms per frame and a frame shift of 10ms. The calculation may include a certain amount of context, such as two frames above and below (the previous frame and the next frame). For example, when calculating the third frame, the second, third, and fourth frames may be input, and the calculation result may include the probability that the third frame is the speech of a certain person. The training loss function may be CE (cross entropy). For example, the loss function can be used to achieve the desired goal: the probability that the audio is from Li Si is the highest, and the probability that it is from Zhang San or Wang Wu is low.
[0182] At the linear layer, the speaker's spectral features x are extracted as speaker features for the input audio. For example, speaker features may include d-vectors or x-vectors. These vectors can represent characteristics of the speaker, such as vocal cord vibration frequency, mouth size, lip thickness, tongue position, and nasopharyngeal diameter. The vector dimensions can be 128 or other dimensions. For example, a portion of the vector's dimensions can represent vocal cord vibration frequency, mouth size, lip thickness, tongue position, and nasopharyngeal diameter. The speaker's spectral features x can be obtained by accumulating multiple frames and then processing them through averaging or other methods. For example, the vector average can be calculated every 50 frames. For example, using d-vectors, the speech can be segmented into multiple segments, and the average of the features of each segment processed by the model can be taken. Using x-vectors, the speech can be segmented into multiple segments, and the features of each segment processed by the model can be aggregated in a certain way.
[0183] The speaker feature extraction module can be pre-trained, for example, using open-source data such as VoxCeleb1, VoxCeleb2, and Librispeech to obtain a stable model. In subsequent systems, the module can be used as a speaker identity embedding extraction module without updating its model parameters.
[0184] When the system extracts the speaker's features, it can select multiple audios of the speaker, for example more than 10, and average the vectors extracted from each audio as the final features of the speaker, namely, speaker identity embedding.
[0185] c: encoder
[0186] The encoder can include a weighted average layer, a concatenation layer, a convolutional normalization layer, a BLSTM layer, and a downsampling layer. For example, an exemplary encoder structure is {weighted average+Concat+{Conv1D+BatchNormalization}*3+BLSTM*2+Downsampler}, indicating that the encoder includes a weighted average layer (weighted average), a concatenation layer (Concat), three convolutional normalization layers (Conv1D+BatchNormalization), two BLSTM layers, and a downsampling layer (Downsampler).
[0187] For example, the encoder input may include 96 frames (a total of about 2 seconds of audio clips) of features extracted by the Wav2vec model. Each frame feature dimension may include 25*1024, which is first weighted by a learnable coefficient vector, as shown in the following formula:
[0188]
[0189] Among them, h avg is the weighted average of 25 vectors. Among them, α is the coefficient of each vector, which represents the importance of each feature. The initial value of the coefficient can be pre-set, such as 0.1, 0.3, etc. The value of the coefficient can be updated during training. For example, if there are 25 coefficients in the initialization, h i is each vector, i is the i-th frame. Get h avgAfter being concatenated with the speaker identity embedding (256 dimensions), the input is fed into a convolutional regularization layer and then into a BLSTM layer, resulting in 96 2*d-dimensional vectors (called bottleneck features). This bottleneck is then downsampled to obtain 96 / f 2*d-dimensional vectors, where d is 128 and f is 2, or d is 8 and f is 48. A smaller dimension results in a larger downsampling granularity. Downsampling decouples the input features, eliminating speaker and text content features and decomposing them into emotional features. Combined with subsequent decoder and phoneme encoder training, this decoupled, purer emotional feature can be obtained.
[0190] d: decoder
[0191] The decoder may include an upsampling layer, a connection layer, an LSTM layer, a linear layer, and a convolutional regularization layer, etc. For example, the structure of an exemplary decoder includes {Upsampler average+Concat+LSTM+Liner+{Conv1D+BatchNormalization}*5}, which means that the decoder includes an upsampling layer (Upsampler average), a connection layer (Concat), an LSTM layer, a linear layer (Liner), and 5 layers of convolutional regularization layers (Conv1D+BatchNormalization). The input of the decoder may include the output of the encoder (96 / f 2*d dimensional vectors), which are upsampled, i.e., repeated f times, to obtain 96 2*d dimensional vectors. Then, the features obtained by upsampling are spliced with the speaker feature vector and the output of the phoneme encoder and then input into the subsequent network module, and 96 80-dimensional features F1 are obtained in the linear layer, i.e., the Mel spectrum of the small segment (about 2s). The minimum mean square error L between the feature F1 and the true Mel spectrum is then calculated. r1 The feature F1 is input into the subsequent convolutional layer to obtain a new Mel spectrum, and the error L between the feature F1 and the true Mel spectrum is calculated. r2 . L r1 and L r2 The calculation can be done by referring to the following formula:
[0192]
[0193] Among them, L r Represents the error between the calculated value and the true value. x and y represent the calculated value and the true value respectively. M represents the number of dimensions (points) of the feature, and k represents the kth dimension.
[0194] e: Phoneme encoder
[0195] Through open source phoneme alignment models such as GMM-HMM, DNN-HMM, LSTM-CTC, RNN-T, etc., the phoneme content of the audio can be obtained, and then each phoneme can be mapped to the corresponding vector.
[0196] For example, referring to the framing method and audio segment extraction method described above, 96 frames of data (i.e., one audio segment) are run through the phoneme alignment model to obtain the phonemes for each frame. This is then mapped to a 128-dimensional vector and input into the phoneme encoder to produce 96 256-dimensional vectors.
[0197] The phoneme encoder can include convolutional normalization layers, BLSTM layers, and so on. An exemplary phoneme encoder structure includes {{Conv1D+BatchNormalization}*3+BLSTM*2}, indicating that the phoneme encoder includes three convolutional normalization layers and two BLSTM layers. The phoneme encoder can extract phoneme features from a phoneme sequence (PhoneSequence).
[0198] 2: Audio emotion classification module
[0199] The output of the encoder is connected to the emotion recognition classifier (Classifier (C)), and then the final emotion classification result (such as speech emotion recognition (SER) Emotion ClassesLogits) is obtained through an activation function (such as softmax). Combined with the emotion classification label (Emotion Class Label), the loss function is obtained. The loss function can be calculated using cross entropy. For example, the loss function can be shown in the following formula:
[0200]
[0201] Among them, z j represents the output of each node (the probability of each emotion), z c Indicates the target emotion (e.g. happy, 4 z j Adjusting the model parameters can increase the probability of the target emotion. j represents the number of each emotion, such as happy, sad, heartbroken, neutral, etc.
[0202] The classifier may include: a linear layer, a dropout layer, and a linear layer. An exemplary classifier structure includes {Linear*2+Dropout+Linear*4}, indicating that the classifier includes 2 linear layers, a dropout layer (Dropout), and 4 linear layers.
[0203] 3: Text Sentiment Classification Module
[0204] a: The text emotion classification model used in this module may include: CNN layer, mean pooling layer (MeanPooling), flattening layer (Flatten) and linear layer. The text emotion classification model can also be called a text emotion recognition model (TER model). The structure of an exemplary text emotion classification model includes {CNN*3+Mean Pooling+Flatten+Linear}, indicating that the text emotion classification model includes. The input of the text emotion classification model is the feature vector corresponding to each word, with a dimension of L. The feature vector can be obtained through pre-training of the Transformer network, and the output is the predicted emotion category. The loss function can be calculated using cross entropy, and can be based on training of the entire sentence, and does not need to be based on fragments (96 frames) as in audio emotion classification.
[0205] b: This module is equivalent to mapping the text to a high-dimensional space through a neural network, representing the content information represented by the text, the association information with its context, and the semantic information, and then integrating this information to extract the emotional information.
[0206] The results of speech emotion recognition and text emotion recognition can be score fused to obtain the multimodal emotion class probabilities corresponding to the audio.
[0207] The specific structures of the various models in the embodiments of the present disclosure are merely examples and not limitations. In actual applications, the structure of the model, such as the number of layers and the type of layers, can be flexibly modified according to specific needs.
[0208] In practical applications, this network architecture can include the following processes: pre-training; training; testing. Figure 16 As shown, each process is introduced below:
[0209] S1601: Pre-training
[0210] S1601a: Pretrained Wav2vec model: This is achieved by pretraining a model structure, such as {TransformerLayer*25}. This model is trained unsupervised using a large amount of audio data, such as LibriVox, which contains approximately 60,000 hours of unlabeled audio data. After multiple rounds of iterative convergence, a hierarchical feature extractor is obtained. The type of audio data and the 60,000 hours of unlabeled audio data are examples only, not limitations. The specific model can be flexibly selected based on actual application needs.
[0211] S1601b: Pre-train speaker feature extractor: Use open source data such as VoxCeleb1, VoxCeleb2, Librispeech, etc. to train a speaker classifier on a certain model structure such as {LSTM+Linear+FC} to obtain the speaker feature (speaker identity embedding) extractor in the system.
[0212] S1601c: Pre-trained phoneme alignment model: Using open source data such as Librispeech and Aishell, and training on models such as GMM-HMM, DNN-HMM, LSTM-CTC, and RNN-T, a model can be obtained to recognize the phoneme content of audio.
[0213] S1602: Training
[0214] S1602a: Using open-source data such as IEMOCAP, each audio clip is annotated with emotion and corresponding content. Each audio clip is framed (e.g., 40ms frame length, 20ms frame shift). 96 frames are extracted at a time, and Wav2vec features w are extracted. Furthermore, speaker features s are extracted for the entire audio clip, and corresponding phoneme features p are extracted from each of the 96 frames using a phoneme alignment model. Each 96-frame clip can be processed as a single audio segment.
[0215] S1602b: Input the Wav2vec feature w to the encoder and concatenate it with the feature s. Then, the emotion feature e is obtained by downsampling the concatenated feature. The feature e is input to the emotion classification network (such as the classifier mentioned above) to perform the emotion category recognition calculation, and the loss Le is obtained through cross entropy. In addition, the feature e is input to the decoder to reconstruct the spectral (Mel spectrum) feature together with the phoneme feature and speaker feature, and the minimum mean square error L is calculated respectively. r1 and L r2 In this way, the gradient is back-propagated according to the stochastic gradient descent criterion to update the network parameters, such as updating the parameters of at least one of the encoder, decoder, and phoneme alignment model.
[0216] S1602c: Repeat the above training steps, such as S1601 and S1602, for each segment, and iterate multiple rounds until convergence. In this way, a speech emotion classification model that decouples emotion, speaker, and content can be obtained.
[0217] S1602d: For each audio content, input it into the text sentiment classifier, train the model, calculate the cross entropy loss, and then reversely update the network. Repeat multiple iterations until convergence, and thus obtain a text sentiment classification model.
[0218] S1603: Testing
[0219] S1603a: Extract speaker features s from the test audio. Furthermore, the test audio can be divided into frames, and Wav2vec features are extracted every 96 frames. Then, the speaker features are combined and input into the encoder and audio emotion classifier to obtain the emotion recognition result R of the segment (e.g., the probability of four categories of emotions). Assuming that the audio has N 96 frames, the N calculated Rs are averaged to obtain the emotion classification result p for the entire audio. s .
[0220] S1603b: Input the text content of the audio into the text emotion classifier to obtain the corresponding emotion recognition result p t .
[0221] S1603c: Weight the emotion recognition results of speech and text to obtain the final recognition result p f , see the following formula:
[0222] p f =w1·p s +w2·p t
[0223] Among them, w1 and w2 can be empirical values, for example, they can be 0.6 and 0.9 respectively.
[0224] In the disclosed embodiment, by reconstructing the sound spectrum and using multi-task learning, the emotional features, speaker features, and content features in the audio are separated and decoupled, minimizing the correlation between them, which helps to improve the accuracy of the emotional features. In addition, the feature recognition results of the emotions in the text semantics are combined to further improve the overall accuracy of emotion recognition.
[0225] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0226] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0227] Figure 17A schematic block diagram of an example electronic device 1700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0228] like Figure 17 As shown, device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1702 or a computer program loaded from a storage unit 1708 into a random access memory (RAM) 1703. Various programs and data required for the operation of device 1700 can also be stored in RAM 1703. Computing unit 1701, ROM 1702, and RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to bus 1704.
[0229] Various components in device 1700 are connected to I / O interface 1705, including an input unit 1706, such as a keyboard, mouse, etc.; an output unit 1707, such as various types of displays, speakers, etc.; a storage unit 1708, such as a magnetic disk, optical disk, etc.; and a communication unit 1709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1709 allows device 1700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0230] The computing unit 1701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1701 performs the various methods and processes described above, such as the speech emotion recognition model training method or the emotion recognition method. For example, in some embodiments, the speech emotion recognition model training method or the emotion recognition method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1700 via the ROM 1702 and / or the communication unit 1709. When the computer program is loaded into the RAM 1703 and executed by the computing unit 1701, one or more steps of the speech emotion recognition model training method or the emotion recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured to execute the speech emotion recognition model training method or the emotion recognition method in any other appropriate manner (for example, by means of firmware).
[0231] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0232] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0233] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0235] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0236] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.
[0237] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.
[0238] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.
Claims
1. A speech emotion recognition model training method, comprising: Acquire a first feature and a second feature of the sample audio, wherein the first feature is used to characterize a feature related to a waveform of the sample audio, and the second feature is used to characterize a feature related to a speaker of the sample audio; Decoupling emotion features using the first feature and the second feature; Using the decoupled emotional features to perform emotion recognition training, a trained speech emotion recognition model is obtained; The method of using the first feature and the second feature to decouple the emotion feature comprises: inputting the first feature and the second feature into an encoder for encoding processing to achieve the emotion feature decoupling; The encoder includes a weight averaging layer, a first connection layer, a first convolution regularization layer, a bidirectional long short-term memory BLSTM layer and a downsampling layer. The first feature and the second feature are input into the encoder for encoding processing, including: inputting the first feature into the weight averaging layer; inputting the second feature into the first connection layer; splicing the output feature of the weight averaging layer and the second feature in the first connection layer to obtain a first spliced feature; inputting the first spliced feature into the first convolution regularization layer, the BLSTM layer and the downsampling layer connected in series for processing in sequence to obtain the output feature of the encoder, and the output feature of the encoder includes the decoupled emotion feature.
2. According to the method of claim 1, obtaining the first feature of the sample audio comprises: The first feature is extracted from the sample audio using a waveform-to-vector (Wav2vec) model.
3. The method according to claim 2, wherein: Extracting the first feature from the sample audio using a Wav2vec model includes: Divide the sample audio into frames to obtain a plurality of first audio frames; At least one audio segment is extracted from the multiple first audio frames and input into the Wav2vec model to obtain the first feature, where the first feature includes the Wav2vec feature of the audio segment.
4. According to the method of any one of claims 1 to 3, obtaining the second feature of the sample audio comprises: The second feature is extracted from the sample audio using a speaker classification model.
5. The method according to claim 4, wherein: Extracting the second feature from the sample audio using a speaker classification model includes: Divide the sample audio into frames to obtain a plurality of second audio frames; The multiple second audio frames are input into the speaker classification model to obtain the second feature.
6. The method according to any one of claims 1 to 3, wherein: The method further comprises: extracting phoneme features from the sample audio using a phoneme alignment model; Decoupling emotion features using the first feature and the second feature also includes: Inputting the output feature of the encoder, the second feature and the phoneme feature into a decoder for decoding; The output features of the decoder are used to update parameters of at least one of the encoder, the decoder and the phoneme alignment model.
7. The method according to claim 6, wherein: The decoder includes an upsampling layer, a second connection layer, a long short-term memory (LSTM) layer, a first linear layer, and a second convolution regularization layer. The output feature of the encoder, the second feature, and the phoneme feature are input into the decoder for decoding processing, including: Inputting the output features of the encoder into the upsampling layer for upsampling; Inputting the second feature and the phoneme feature into the second connection layer; Concatenating the upsampled feature, the second feature, and the phoneme feature at the second connection layer to obtain a second concatenated feature; Inputting the second concatenated feature into the LSTM layer and the first linear layer connected in series for processing in sequence to obtain a first Mel spectrum feature and a first error; The first Mel spectrum feature is input into the second convolution regularization layer for convolution regularization processing to obtain a second Mel spectrum feature and a second error.
8. The method according to claim 7, wherein: Updating parameters of at least one of the encoder, the decoder, and the phoneme alignment model using the output features of the decoder, comprising: The first error and the second error are used to update parameters of at least one of the encoder, the decoder and the phoneme alignment model according to a stochastic gradient descent criterion.
9. The method according to any one of claims 1 to 3, wherein: Emotion recognition training is performed using the decoupled emotion features, including: The output features of the encoder are input into an emotion recognition classifier to perform emotion category recognition, and the emotion category recognition result is used to update the parameters of the encoder and the emotion recognition classifier.
10. The method according to claim 9, wherein: The emotion recognition classifier includes a second linear layer, a discard layer and a third linear layer, and the output features of the encoder are input into the emotion recognition classifier for emotion category recognition, and the parameters of the encoder and the emotion recognition classifier are updated using the emotion category recognition result, including: Inputting the output features of the encoder into the second linear layer, the discard layer and the third linear layer connected in series for processing in sequence to obtain an emotion recognition result; Calculating the loss using the cross entropy of the emotion recognition result; The loss is used to update parameters of the encoder and / or the emotion recognition classifier.
11. An emotion recognition method, comprising: Acquire a first feature and a second feature of the audio to be recognized, wherein the first feature is used to characterize a feature related to a waveform of the audio to be recognized, and the second feature is used to characterize a feature related to a speaker of the audio to be recognized; Inputting the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition to obtain a first recognition result; Wherein, the speech emotion recognition model is a speech emotion recognition model trained using the training method described in any one of claims 1 to 10.
12. The method according to claim 11, wherein: Obtaining the first feature of the audio to be recognized includes: The first feature is extracted from the audio to be recognized using a Wav2vec model.
13. The method according to claim 12, wherein: Extracting the first feature from the audio to be recognized by using a Wav2vec model includes: Divide the audio to be recognized into frames to obtain a plurality of first audio frames; At least one audio segment is extracted from the multiple first audio frames and input into the Wav2vec model to obtain the first feature, where the first feature includes the Wav2vec feature of the audio segment.
14. The method according to any one of claims 11 to 13, wherein: Obtaining the second feature of the audio to be recognized includes: The second feature is extracted from the audio to be recognized using a speaker classification model.
15. The method according to claim 14, wherein: Extracting the second feature from the audio to be recognized by using a speaker classification model includes: Divide the audio to be recognized into frames to obtain a plurality of second audio frames; The multiple second audio frames are input into the speaker classification model to obtain the second feature.
16. The method according to any one of claims 11 to 13, wherein: The method further comprises: Performing text recognition on the text content corresponding to the audio to be recognized using the text emotion recognition model to obtain a second recognition result; The first recognition result and the second recognition result are weighted to obtain a third recognition result.
17. A speech emotion recognition model training device, comprising: An acquisition module, configured to acquire a first feature and a second feature of a sample audio, wherein the first feature is used to characterize a feature related to a waveform of the sample audio, and the second feature is used to characterize a feature related to a speaker of the sample audio; A decoupling module, used for decoupling emotion features by using the first feature and the second feature; A training module is used to perform emotion recognition training using the emotion features obtained by decoupling to obtain a trained speech emotion recognition model; Among them, the decoupling module includes: an encoding submodule, which is used to input the first feature and the second feature into an encoder for encoding processing to achieve emotional feature decoupling; the encoder includes a weight averaging layer, a first connection layer, a first convolutional regularization layer, a BLSTM layer and a downsampling layer, and the encoding submodule is also used to: input the first feature into the weight averaging layer; input the second feature into the first connection layer; splice the output feature of the weight averaging layer and the second feature in the first connection layer to obtain a first spliced feature; input the first spliced feature into the first convolutional regularization layer, the BLSTM layer and the downsampling layer connected in series for processing in sequence to obtain the output feature of the encoder, and the output feature of the encoder includes emotional features.
18. The device according to claim 17, wherein the acquisition module is used to extract the first feature from the sample audio using a Wav2vec model.
19. The device according to claim 18, wherein The acquisition module comprises: A first framing submodule, configured to frame the sample audio to obtain a plurality of first audio frames; The first feature extraction submodule is used to extract at least one audio segment from the multiple first audio frames and input it into the Wav2vec model to obtain the first feature, where the first feature includes the Wav2vec feature of the audio segment.
20. The apparatus according to any one of claims 17 to 19, wherein the acquisition module is used to extract the second feature from the sample audio using a speaker classification model.
21. The device according to claim 20, wherein: The acquisition module comprises: A second framing submodule, configured to frame the sample audio to obtain a plurality of second audio frames; The second feature extraction submodule is used to input the multiple second audio frames into the speaker classification model to obtain the second feature.
22. The device according to any one of claims 17 to 19, wherein: The device further comprises: a phoneme feature extraction module, configured to extract phoneme features from the sample audio using a phoneme alignment model; The decoupling module also includes: A decoding submodule, used for inputting the output feature of the encoder, the second feature and the phoneme feature into a decoder for decoding processing; An updating submodule is used to update parameters of at least one of the encoder, the decoder and the phoneme alignment model using the output features of the decoder.
23. The device according to claim 22, wherein: The decoder comprises an upsampling layer, a second connection layer, an LSTM layer, a first linear layer and a second convolution regularization layer, and the decoding submodule is used for: Inputting the output features of the encoder into the upsampling layer for upsampling; Inputting the second feature and the phoneme feature into the second connection layer; Concatenating the upsampled feature, the second feature, and the phoneme feature at the second connection layer to obtain a second concatenated feature; Inputting the second concatenated feature into the LSTM layer and the first linear layer connected in series for processing in sequence to obtain a first Mel spectrum feature and a first error; The first Mel spectrum feature is input into the second convolution regularization layer for convolution regularization processing to obtain a second Mel spectrum feature and a second error.
24. The device according to claim 23, wherein: The update submodule is used to: The first error and the second error are used to update parameters of at least one of the encoder, the decoder and the phoneme alignment model according to a stochastic gradient descent criterion.
25. The device according to any one of claims 17 to 19, wherein: The training module is used to: The output features of the encoder are input into an emotion recognition classifier to perform emotion category recognition, and the emotion category recognition result is used to update the parameters of the encoder and the emotion recognition classifier.
26. The device according to claim 25, wherein The emotion recognition classifier includes a second linear layer, a discard layer and a third linear layer. The output features of the encoder are input into the emotion recognition classifier for emotion category recognition. The training module includes: An emotion recognition submodule, used for inputting the output features of the encoder into the second linear layer, the discard layer and the third linear layer connected in series for processing in sequence to obtain an emotion recognition result; A loss calculation submodule, used to calculate the loss using the cross entropy of the emotion recognition result; A parameter updating submodule is used to update the parameters of the encoder and / or the emotion recognition classifier using the loss.
27. An emotion recognition device, comprising: An acquisition module, configured to acquire a first feature and a second feature of the audio to be recognized, wherein the first feature is used to characterize a feature related to a waveform of the audio to be recognized, and the second feature is used to characterize a feature related to a speaker of the audio to be recognized; A first recognition module, used for inputting the first feature and the second feature into a speech emotion recognition model to perform emotion category recognition and obtain a first recognition result; Wherein, the speech emotion recognition model is a speech emotion recognition model trained using the training device described in any one of claims 17 to 26.
28. The device according to claim 27, wherein The acquisition module is used to extract the first feature from the audio to be recognized by using a Wav2vec model.
29. The device according to claim 28, wherein The acquisition module comprises: A first framing submodule, configured to frame the audio to be recognized to obtain a plurality of first audio frames; The first extraction submodule is configured to extract at least one audio segment from the plurality of first audio frames and input the extracted audio segment into the Wav2vec model to obtain the first feature, wherein the first feature includes the Wav2vec feature of the audio segment.
30. The device according to any one of claims 27 to 29, wherein: The acquisition module is used to extract the second feature from the audio to be recognized by using a speaker classification model.
31. The device according to claim 30, wherein The acquisition module comprises: A second framing submodule, configured to frame the audio to be recognized to obtain a plurality of second audio frames; The second extraction submodule is used to input the multiple second audio frames into the speaker classification model to obtain the second feature.
32. The device according to any one of claims 27 to 29, wherein: The device also includes: A second recognition module is used to perform text recognition on the text content corresponding to the audio to be recognized using a text emotion recognition model to obtain a second recognition result; The processing module is used to perform weighted processing on the first recognition result and the second recognition result to obtain a third recognition result.
33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.
34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.
35. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 16.
Citation Information
Patent Citations
PCA-BP-based emotion recognition method and system
CN108053840A
Speaking face video generation method based on decoupling representation learning
CN111666831A
