Multimodal emotion recognition method based on audio and text and application

Abstract

The invention provides a multi-mode emotion recognition method based on audios and texts and application, belongs to the field of artificial intelligence calculation, and aims to solve the problem that an existing emotion recognition technology is low in accuracy in a complex environment. Pre-training is carried out by combining voice and text information, a voice emotion recognition model and a text emotion recognition model are generated, a spectrogram and MFCC features are extracted from audio, and high-order audio features are obtained. Then, text and audio features are fused through an attention network, and finally a time sequence decomposition causal convolution block is input for emotion classification recognition. According to the method, the robustness and generalization ability of recognition are improved through multi-level feature learning, the recognition accuracy is remarkably improved, and particularly, the method is excellent in performance in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an audio and text-based emotion information extraction method and a multimodal emotion recognition method and application. Background Art

[0002] With the continuous advancement of artificial intelligence and human-computer interaction technology, the application scope of emotion recognition technology continues to expand, covering many industries such as entertainment, transportation, healthcare, and retail. By identifying user emotions, these applications can provide personalized services and improve the naturalness and efficiency of human-computer interaction.

[0003] Among them, speech emotion recognition technology has gradually attracted widespread attention in the field of voice interaction. By identifying the user's emotional state, voice interaction can become more intelligent. However, traditional speech emotion recognition methods mainly rely on the acoustic characteristics of speech, ignoring the key role of text semantics in emotion recognition. In addition, existing technologies usually rely on standard, clean speech signals, overly rely on the audio characteristics of speech, and fail to fully consider the impact of environmental noise on emotional expression in actual scenarios, resulting in low emotion recognition accuracy in complex environments, making it difficult to meet the needs of practical applications.

[0004] In the current mainstream multimodal emotion recognition methods, when fusing information from different modalities, due to the lack of rigorous fusion method, too much noise is often introduced or important features are lost, and the characteristics of each modality cannot be fully reflected. Summary of the invention

[0005] In view of the deficiencies in the above-mentioned background technology, the present invention proposes an audio and text-based emotion information extraction method and a multimodal emotion recognition method and application to solve the above-mentioned technical problems.

[0006] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: collecting open source speech recognition data sets, and realizing audio enhancement by adding white noise, time domain random jitter and frequency domain random jitter; obtaining a speech emotion recognition network model and a text emotion recognition network model through pre-training of the finally enhanced speech audio; obtaining a time-frequency diagram of the enhanced speech audio through short-time Fourier transform (STFT), extracting the Mel frequency cepstral coefficient to obtain MFCC features, and inputting its time-frequency diagram and MFCC features into the audio high-order feature extraction subnetwork to obtain high-order audio features; inputting the enhanced speech audio into the pre-trained speech emotion recognition network model and the text emotion recognition model, extracting high-order text features and high-order emotion features respectively, and fusing them with high-order audio features through the attention network module; inputting the fused features into the temporal decomposition causal convolution block to generate high-order temporal causal composite features, and outputting them to the classification network layer through the fully connected layer to complete the classification and recognition of speech emotions.

[0007] In addition, the present invention also proposes a multimodal emotion recognition device based on audio and text. Its structure includes the following modules: a model pre-training module, which is responsible for collecting open source emotional speech data sets, enhancing audio, generating enhanced speech audio, and obtaining speech emotion recognition network models and text emotion recognition network models through pre-training; a high-order feature extraction module, which is used to extract time-frequency graphs and MFCC features from the enhanced speech audio, and input them into the audio high-order feature extraction sub-network to obtain high-order audio features; a feature fusion module, which inputs the enhanced speech audio into the pre-trained speech emotion recognition network model and text emotion recognition network model to obtain high-order text features and environmental features, and then fuses the two with high-order audio features; an emotion recognition module, which inputs the fused features into the temporal decomposition causal convolution block to generate high-order temporal causal composite features, which are then transmitted to the classification network layer through the fully connected layer to complete the classification and recognition of speech emotions.

[0008] The beneficial effects of the present invention are:

[0009] (1) The emotion recognition method proposed in this invention constructs multi-level features for task learning and combines the complementary training of speech and text information, so that the classification and recognition of speech emotions can adapt to complex environments and have high robustness. Compared with the traditional method of speech emotion recognition that relies on a single acoustic signal, this method greatly improves the recognition accuracy;

[0010] (2) For the collected open source emotional speech dataset, white noise is added to achieve the preset signal-to-noise ratio, and enhanced by random jitter in the time domain and jitter in the frequency domain to obtain enhanced emotional speech audio. In this process, although the audio content is not changed, the continuity of the original speech in the time domain is disturbed, which enhances the model's ability to learn time information and improves the model's robustness and generalization ability to time domain jitter.

[0011] (3) By focusing on the fusion of high-order text features, audio features, and high-order frequency domain features, the most significant features can be highlighted, thereby maximizing the effectiveness of multi-source feature fusion and achieving more accurate feature recognition results in the subsequent stages;

[0012] (4) The present invention injects emotional information into text encoding features through the emotional feature extraction layer, extracts audio features through the audio high-order feature extraction layer, and then uses the emotional fusion layer to fuse the text emotional information with the audio emotional features, thereby further improving the global feature expression of text and audio features. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall flow chart of the present invention;

[0014] Figure 2 A schematic diagram of a specific implementation network architecture of multimodal emotion recognition based on audio and text provided by the present invention;

[0015] Figure 3 A schematic diagram of the structure of a multimodal emotion recognition device based on audio and text provided by the present invention;

[0016] Figure 4 is a scene graph of a multimodal emotion recognition device according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] like Figure 1 As shown, the present invention proposes a multimodal emotion recognition method based on audio and text. The method comprises the following steps: first, an open source speech recognition data set is collected, and the audio is enhanced, specifically by adding white noise and introducing random jitter in the time domain and frequency domain respectively. The enhanced speech audio is pre-trained to obtain a speech emotion recognition network model and a text emotion recognition network model. Then, the enhanced audio is converted into a time-frequency graph by short-time Fourier transform (STFT), and the Mel frequency cepstral coefficient is extracted to obtain MFCC features, and these time-frequency graphs and MFCC features are input into the audio high-order feature extraction subnetwork to obtain high-order audio features. At the same time, the enhanced audio is respectively input into the pre-trained speech emotion recognition and text emotion recognition models, and high-order text features and emotion features are extracted therefrom. Then, the high-order audio features, text features and emotion features are fused through the attention network module, and the fused features enter the temporal decomposition causal convolution block to generate high-order temporal causal composite features, which are finally transmitted to the classification network layer through the fully connected layer to realize the classification and recognition of speech emotions.

[0019] In this embodiment, the training data set refers to the original data set without emotion labels, and the fine-tuning data set refers to the original data set with emotion labels. The original emotion samples include: the original audio sequence, the corresponding original text, and the corresponding emotion category label (for the fine-tuning data set).

[0020] The pre-training model includes: encoding layer, feature extraction layer, attention fusion layer and classification layer. The pre-training process of the speech emotion recognition model is as follows:

[0021] S101: Before the original text sequence in the pre-training dataset is input into the encoding layer, a masking strategy based on sentiment words is used to mask the text sequence to generate a masked text sequence in the pre-training stage.

[0022] S102: The encoding layer extracts features from the masked text sequence and the original audio sequence to obtain a text feature vector and an original audio feature vector in a pre-training phase.

[0023] S103: Extract the spectrogram and MFCC features from the original audio, and input them into the feature extraction layer to obtain the frequency domain weight features in the pre-training stage.

[0024] S104: Use the attention fusion layer to fuse the frequency domain weight features with the original audio feature vector to generate an enhanced audio feature vector.

[0025] S105: Input the enhanced audio feature vector and the text feature vector into the classification layer together to perform the emotion classification task and construct the intra-modal loss.

[0026] S106: Combine the inter-modality global contrast loss (GCL) and the intra-modality occlusion mask loss (eMLM) to calculate the loss function of the pre-training stage. Use the RMSprop optimizer to adjust the gradient until the pre-training loss function converges, completing the pre-training process.

Claims

1. A multimodal emotion recognition method based on audio and text, characterized in that: The method comprises the following steps: Collect open source speech recognition datasets for audio enhancement; The enhanced speech audio is obtained through pre-training to obtain a speech emotion recognition network model and a text emotion recognition network model; The enhanced speech audio is transformed by short-time Fourier transform (STFT) to obtain a time-frequency graph, and the Mel-frequency cepstral coefficients are extracted to obtain MFCC features. The time-frequency graph and MFCC features are then input into the audio high-order feature extraction subnetwork to obtain high-order audio features. The enhanced speech audio is input into the pre-trained speech emotion recognition network model and text emotion recognition model to extract high-order text features and high-order emotion features respectively, and then they are fused with high-order audio features through the attention network module; The fused features are input into the temporal decomposition causal convolution block to generate high-order temporal causal composite features, which are then output to the classification network layer through the fully connected layer to complete the classification and recognition of speech emotions.

2. The multimodal emotion recognition method based on audio and text as claimed in claim 1, characterized in that: This method achieves audio enhancement by adding white noise to achieve a preset signal-to-noise ratio. In addition, the enhancement process also includes time domain random jitter and frequency domain jitter, and finally the enhanced emotional speech audio is obtained.

3. The multimodal emotion recognition method based on audio and text as claimed in claim 1, characterized in that: The speech emotion recognition network model is composed of a wav2vec2 network, which is divided into two parts: the encoder network (consisting of 5 layers of convolution) encodes the original audio x into a latent space z, and the context network (consisting of 9 layers of convolution) converts z into a contextualized representation, and the final feature dimension is 512×the number of frames.

4. The multimodal emotion recognition method based on audio and text as claimed in claim 1, characterized in that: The steps for extracting the time-frequency graph and MFCC features of the audio include: First, the enhanced speech audio is divided into frames, and then a Hamming window is applied to each frame signal and Fourier transform is performed to obtain a spectrum representation. Then, the spectrum is filtered through a Mel filter bank and the logarithm is taken. Finally, MFCC features are generated through discrete cosine transform. The extracted spectrogram and MFCC features will be input into the audio high-order feature extraction sub-network composed of an 8-layer convolutional neural network Alex net and a bidirectional LSTM network (input dimension is 40, hidden layer dimension is 256, and number of layers is 2), respectively.

5. The multimodal emotion recognition method based on audio and text as claimed in claim 1, characterized in that: The text data extraction process generates text features, using preset natural language processing tools such as RoBERTa and BERT.

6. The multimodal emotion recognition method based on audio and text as claimed in claim 1, characterized in that: When the audio is input into the pre-trained speech emotion recognition network model and text emotion recognition model, high-order text features and high-order emotion features are extracted respectively and fused with high-order audio features through the attention network module, including: Input the enhanced audio into the pre-trained speech emotion recognition network model to extract high-order emotion features before the last layer; Input text data into the pre-trained text sentiment recognition model to extract text features before the last layer; Finally, the high-order spectrum features, high-order MFCC features, high-order text features and high-order sentiment features are fused to obtain the maximum attention fusion feature.

7. The method according to claim 6, characterized in that The fused features are input into the temporal decomposition causal convolution block to generate high-order temporal causal composite features, which are then output to the classification network layer through the fully connected layer to complete the classification and recognition of speech emotions. This process includes inputting the maximum attention fusion features into the multi-label classification model and identifying the target emotion to be extracted through the model. The multi-label classification model consists of a multi-layer perceptron containing a softmax fully connected layer and a ReLU layer.

8. The method according to claim 7, characterized in that During the training process of the multi-label classification model, the loss function of the pre-training stage is calculated through the global contrast loss (GCL) between modalities and the eMLM loss within modality. The gradient is adjusted using the RMSprop optimizer.

9. A multimodal emotion recognition device based on audio and text, characterized in that: include: The model pre-training module is responsible for collecting open source emotional speech data sets, enhancing the audio, generating enhanced speech audio, and obtaining the speech emotion recognition network model and text emotion recognition network model through pre-training. A high-order feature extraction module is used to extract the time-frequency graph and MFCC features from the enhanced speech audio, and input them into the audio high-order feature extraction sub-network to obtain high-order audio features; The feature fusion module inputs the enhanced speech audio into the pre-trained speech emotion recognition network model and text emotion recognition network model to obtain high-order text features and environmental features respectively, and then fuses these two with the high-order audio features; The emotion recognition module inputs the fused features into the temporal decomposition causal convolution block to generate high-order temporal causal composite features, which are then transmitted to the classification network layer through the fully connected layer to complete the classification and recognition of speech emotions.

Images