Speech emotion recognition method introducing text mode and related device
By combining Hubert and Roberta models with SenticNet, a graph neural network and modal interactive coding module are constructed, which solves the problem of unutilized text modality and poor feature fusion, and improves the accuracy of speech emotion recognition and emotional prediction effect.
Patent Information
- Application Number
- CN202410975728.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-26
Smart Images

Figure CN119028378A_ABST
Abstract
Description
Background Art
[0002] Among other communication methods such as text, body language, and facial expressions, humans use language as the most common way to communicate. It contains a lot of information, including the speaker's feelings. Detecting the speaker's emotions from his speech is very useful in various real-world applications. Voice is now used as a quick way to connect people and computers. Although there have been great advances in the field of speech recognition, there is still a big gap in the interaction between people and computers. The main reason is that computers cannot understand the user's feelings. If we want computers to be truly intelligent and interact with us naturally, we must give computers the ability to recognize, understand, and even have and express emotions. At the same time, speech emotion recognition can extract meaningful information. Speech is considered to carry explicit language content and implicit paralinguistic knowledge such as semantic information, speaker identity information, and emotional state. The true meaning of speech is contained in the words spoken, which can be detected by understanding the speaker's emotions.
[0003] Emotions play a vital role in real-life human interactions, and in the past few decades, emotion computing has become an important research field. Human emotions can be recognized through facial expressions, body postures, written information, intonation, etc. In human-computer interaction systems, people's language often contains multiple emotions, and accurately distinguishing their emotions plays a vital role in many applications. For example, speech emotion recognition (SER) can help treat patients and improve their emotional communication skills.
[0004] The different emotional characteristics of speech can be characterized by pitch, duration, energy and excitement sources, the shape of the vocal tract system, language information, and the speaker's emotional behavior. For example, when people chat with chatbots, understanding people's emotional states through the information in the conversation can help collect more user emotional information and personality characteristics, and provide more help for subsequent communication with customers, thereby improving the communication experience of both parties. In real life and production processes, there are many scenarios and needs to obtain emotional tendencies through conversations. At this time, speech emotion recognition algorithms provide such a way to extract these key information.
[0005] Speech emotion recognition aims to extract the conversation information between people and machines, integrate the semantic information, and use it to identify people's emotional tendencies and emotional information in the conversation. This field mainly studies how to better extract the sound features and semantic information in the conversation, and integrate or process the two separately, so as to perform accurate emotion recognition tasks. Speech emotion recognition is of great significance.
[0006] In speech recognition, the most commonly used speech feature is the Mel-Frequency Cepstral Coefficient (MFCC). It can display a lot of information in the speech signal and display the signal spectrum in a simple and clear form. People arrange a group of bandpass filters from low frequency to high frequency according to the size of the critical bandwidth from dense to sparse to filter the input signal. The signal energy output by each bandpass filter is used as the basic feature of the signal. After further processing, this feature can be used as the input feature of the speech. Since this feature does not depend on the nature of the signal, it does not make any assumptions or restrictions on the input signal, and also utilizes the research results of the auditory model.
[0007] The wav2vec model is a model of AR (AutoRegression) mechanism, so the wav2vec model in the pre-training stage is more like a language model with raw audio as input. As the amount of training data increases, the role of wav2vec is gradually weakening. In addition, as for the autonomous selection of modeling units for speech recognition, it is similar to the transition probability of HMM state spin and jump. It just uses a more expressive model to depict the nonlinear relationship. In general, the era of deep learning that autonomously selects representation features has also arrived in the field of speech recognition, opening a new door for speech recognition.
[0008] With the deepening of scientific research, great progress has been made in the field of speech emotion recognition, involving building a corpus, obtaining speech emotion features, and developing more accurate speaker emotion recognition models.
[0009] In terms of speech emotion corpora, corpora are mainly divided into three types: simulated, semi-simulated, and natural speech datasets. EMO-DB (German), DES (Danish), RAVDESS (English), TESS (English), and CREMA-D (English) are all standardized simulated datasets, which are recorded by trained speakers reading the same text with different emotions. However, because the speaker controls the pronunciation of emotions, it is easy to cause the speech emotion recognition model to overfit during the training process, and the model will have a large deviation in the prediction of speech emotions used for conversations in daily life. IEMOCAP (English) is a semi-simulated dataset, in which speakers or actors conduct dialogues according to scripts or improvise dialogues for different scenes and record them. The VAM (German) dataset is a natural audio-visual dataset based on the dialogues between the host and the guests in a German TV talk show. This dataset is close to conversations in daily life. The Chinese Institute of Automation has established a multimodal natural language database called CHEAVD. The rich database provides a solid data foundation for subsequent speech emotion recognition research.
[0010] In terms of emotion feature extraction, in early studies, the commonly used speech emotion features are to extract low-level features (LowLevelDescriptors, LLDs) from the original speech, and then perform statistical operations on the low-level features to obtain high-level features (HighLevelStatisticsFunctions, HSFs), and use classifiers for classification. With the continuous development of deep learning, people began to use deep learning networks to extract features and extract low-level features from speech signals, among which spectrograms are the most widely used. Then, the technology of extracting acoustic features from spectrograms using deep learning networks has been widely recognized.
[0011] In terms of emotion recognition networks, traditional classification algorithms were mainly used in the early days, including k-nearest neighbor, decision tree, naive Bayes classifier, Gaussian mixture model and hidden Markov model.
[0012] In 2022, Chu et al. proposed an audio-text pre-trained cross-modal transformer model built on the RoBERTa and HuBERT unimodal pre-trained models. For their cross-modal transformer model, they proposed a new staged training scheme, which includes initial pre-training, task-adaptive pre-training, and fine-tuning on downstream tasks.
[0013] In 2023, Hu et al. proposed a joint network consisting of a spectrum-based pre-trained model and designed an interactive attention module to effectively fuse the intermediate features of the two models. They also proposed a spectrum-based multi-stage SER model, including a feature aggregation unit (FAU) and a global local attention module (GLAM), and verified that both FAU and GLAM can improve the robustness of SER. They designed an interactive attention module (IAM) to effectively fuse the intermediate features of the two models. They used a multi-branch training strategy (MBTS) to effectively optimize the model so that the spectrum-based model, the pre-trained model, and the joint network can simultaneously output SER predictions during the training phase.
[0014] In 2023, Wang et al. recognized emotions from auditory and linguistic information. Using MPNet and HuBERT to extract text and audio features, they developed a Modal Interaction Transformer (MIT) that retrieves emotional information from concatenated text and audio features using a set of trainable emotion query tokens. Two parallel unimodal emotion feature refinement encoders were used, followed by a self-attention module. Finally, a Modal Interaction Transformer was proposed to integrate emotional features from both modalities through learnable emotion query tokens. In addition, they designed a random modality masking training strategy by randomly masking the audio or text embeddings during training to force the model to fully perceive the emotional information of each modality.
[0015] There are two difficulties in the speech emotion recognition task. First, the complementarity of multimodal information has not been well utilized. In addition to the rich information contained in the text form, the speaker's tone and intonation can indicate the intensity of the emotion. Second, previous studies have mainly focused on the interaction of multimodal, ignoring the explicit emotional semantic information conveyed by words in the text form. Such as "laugh", "love", "hate", etc. This explicit emotional information in the text can guide the multimodal fusion process of the model, thereby improving performance. In addition, there will be implicit connections between modalities.
[0016] In the face of these problems, existing methods have proposed a solution, which is to process the speech first and extract the semantic information to assist in emotion recognition.
[0017] For example, the method proposed by Feng et al. in the 2020 article "End-to-End Speech Emotion Recognition Combined with Acoustic-to-Word ASR Model" is a SER model combined with an acoustic-to-word ASR model. The model was enhanced using the self-attention mechanism, and then the ASR model was fine-tuned using a multi-task learning method. They used the hidden state of the decoder in the ASR model to replace the text data in the SER model. Therefore, their model is an ASR-SER model that only uses speech. In combination with the self-attention mechanism, the acoustic features and ASR features are processed separately. Through multi-task learning, the ASR model can be fine-tuned, and the SER model can obtain more accurate ASR features, thereby improving recognition performance.
[0018] However, there are some problems with the existing methods. First, the text modality is not fully processed to obtain more useful emotional information. At the same time, since the output of ASR is not processed, the extracted features will have a lot of noise that affects the effect of the model. In addition, the extracted text features and speech features are simply spliced, and there is no clever fusion method to obtain a better hidden state of the two for emotion classification. Summary of the invention
[0019] The technical problem to be solved by the present invention is to provide a speech emotion recognition method and related devices introducing text modality in response to the above-mentioned deficiencies in the prior art, so as to solve the technical problem that the rich emotional information in a single modality is not fully utilized, resulting in many implicit emotional features in a single modality not being extracted or learned.
[0020] The present invention adopts the following technical solutions:
[0021] A speech emotion recognition method introducing text modality comprises the following steps:
[0022] Preprocessing and feature extraction of input data of two or more modalities;
[0023] Then it is sent to the speech emotion recognition model that introduces the text modality to obtain the final emotion prediction result.
[0024] Preferably, the preprocessing and feature extraction are specifically performed as follows:
[0025] S101, using Hubert self-supervised speech representation learning model to process audio data;
[0026] S102, using the Roberta pre-training model to process text data;
[0027] S103, first use SenticNet to extract all words whose sentiment polarity is greater than a specified threshold, splice them into the original sentence, obtain a graph neural network and the spliced sentence, and use all the words to establish a graph neural network of words;
[0028] S104, divide the data, complete the division of the training set by the default division method in the official data set, standardize the two modes of each data, subtract the mean and divide by the variance;
[0029] S105. Obtain features through the pre-trained feature extractor, use the pickle module to save all features into the .pkl file, and finally build the dataloader to complete the setting of the data set and set the collection function.
[0030] Preferably, the speech emotion recognition model introducing text modality includes:
[0031] The ASR text extraction module recognizes all speech and saves it as an extended modality to participate in the task of emotion recognition;
[0032] Modal interaction encoding module, including text reference cross-modal encoder and speech reference cross-modal encoder. The text reference cross-modal encoder and speech reference cross-modal encoder are extended from the original transformer encoder to the bimodal scenario.
[0033] The text graph neural network information extraction module builds a sentiment knowledge graph for each sample’s text modality, and then applies the graph embedding calculation method to obtain the final sentiment knowledge enhanced embedding;
[0034] The splicing feature self-attention module dynamically adjusts the weight of each feature dimension of each modality and performs multimodal fusion to obtain the joint representation h';
[0035] The prediction module expands the joint representation h' into a 1-dimensional tensor, and then passes it through a fully connected layer and a Softmax activation function to obtain the predicted probability of the emotion.
[0036] Preferably, the modal interaction encoding module is specifically:
[0037] A multi-head self-attention mechanism is applied to the audio sequence of each block of the cross-modal text encoder, a residual connection and layer normalization are added to the self-attention output, and cross-modal attention is used to learn the interaction between speech and text modalities. At the same time, residual connections and layer normalization are added at the end, and a multi-head self-attention mechanism is applied to integrate the features obtained previously. Finally, the output of this module is obtained through a feedforward neural network, and the cross-modal text representation h is obtained from the last layer of text reference cross-modal encoder. t, for cross-modal text representation, the first token embedding is used, i.e. as a sentence representation; the final speech representation is calculated in a mirrored manner to obtain a cross-modal speech representation h a ; For cross-modal audio representation, all speech embeddings are used as audio representation.
[0038] Preferably, the modal interaction encoding module specifically includes:
[0039] QKV fusion of attention mechanism:
[0040] The text feature ht and the speech feature ha are symmetrically used as query Q, key K and value V input;
[0041] The cross-modal attention mechanism in the text citation cross-modal encoder uses the text feature ht as Q and the speech feature ha as K, V;
[0042] The fused features are repeatedly used as QKV for cross-modal attention. The attention mechanism calculates the dot product between the K vector and the Q vector. After the input passes through the modal interaction encoding module, the output feature map representing the fusion of text and sound modalities is obtained. The dimensions and lengths of the input of speech features and the input of text features are all set to the same.
[0043] Text graph neural network information extraction module:
[0044] First, use SenticNet to extract all the words whose sentiment polarity is greater than the specified threshold, and splice them into the original sentence to obtain a graph neural network and the spliced sentence; during initialization, load all information and data, and create a graph for each sample during the forward propagation process; for each graph, first assign embeddings to its nodes and edges and then use the GNN aggregation operation to update the graph; finally, use a linear layer, Dropout, and Sigmoid activation function to process the graph embedding to generate the final graph features.
[0045] Preferably, the splicing feature self-attention module is specifically:
[0046] Concatenate the four obtained feature representations into a multimodal representation; calculate the weighted featurization attention matrix W att ; Applied to enhance the contribution of key features to the multimodal fusion process; Finally, the connected representation is combined with W att Multiply them together to get the joint representation h'.
[0047] Preferably, the predicted probability P is as follows:
[0048] P = Softmax(fc(flatten(h')))
[0049] Among them, fc is the fully connected layer, and flatten is the dimensionality reduction function for multi-dimensional data.
[0050] In a second aspect, an embodiment of the present invention provides a speech emotion recognition system introducing a text modality, comprising:
[0051] A processing module performs preprocessing and feature extraction on input data of two or more modalities;
[0052] The recognition module is sent to the speech emotion recognition model that introduces text modality to obtain the final emotion prediction result.
[0053] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for speech emotion recognition introducing text modality when executing the computer program.
[0054] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for speech emotion recognition introducing text modality.
[0055] Compared with the prior art, the present invention has at least the following beneficial effects:
[0056] A speech emotion recognition method that introduces text modality. The purpose of preprocessing the text modality and speech modality is to improve the feature extraction capabilities of the text feature processor Roberta and the speech modality feature processor Hubert we use, and to obtain features with stronger representation capabilities. The purpose of feature extraction is to store the extracted features in a .pkl file, which can be more convenient for subsequent processing.
[0057] The purpose of introducing text modality into the speech emotion recognition model is to make full use of the text modality to obtain better model performance and better emotion prediction results.
[0058] Furthermore, through step S1, in order to preliminarily extract audio information and text information, all words greater than the specified threshold are extracted, and a graph neural network is constructed; it is possible to utilize text information at the word level to extract richer emotional features; during training, there is no need to focus too much on dividing the data set; at the same time, standardization is beneficial for eliminating the differences between features. After saving the .pkl file, the time for re-extracting features during each training is saved, and more focus is placed on the training of the model itself, reducing the time for data processing. Constructing dataloader and setting collection functions are to load .pkl data in an appropriate manner.
[0059] Furthermore, the ASR module obtains the text corresponding to the speech through the openai / whisper-large-v3 model. The main purpose is to expand the original speech mode to the text mode and obtain more information; the role of the modal interaction encoding module is to better learn the emotional information of the text mode and the speech mode through fusion, so that the model can extract better emotional representation; the text graph neural network information extraction module aims to enhance the fusion embedding by generating additional emotional knowledge representation, mainly to extract emotional information at the word level; the splicing feature self-attention module is used to obtain the importance of each feature channel, and then used to enhance the contribution of key features to the multimodal fusion process; the prediction module is the last module of the entire model, which is used to output the model's prediction probability for emotion.
[0060] Furthermore, the main function of the modal interaction encoding module is to enable the multimodal attention mechanism to better extract emotional feature information. Different multi-head attention layers have different functions. The first self-attention layer is mainly used to integrate its own information, the second modal interaction attention layer is used to extract emotional information through multimodal fusion, and the third self-attention layer is used to integrate emotional information after modal interaction. In order to reduce noise, h_t^0 is used as the sentence representation.
[0061] Furthermore, the attention mechanism is a very popular and effective way to extract information. In the modal interaction encoding module, the roles of voice and text are placed in an equal position. Therefore, the same method is used to process the two modalities. In the modal interaction attention layer, we use different modalities as Q, and the relative modalities as K and V. The main purpose is to effectively extract the emotional feature information of different modalities. The length and dimension of the input features are set to be the same in order to better process the features of different modalities. The text graph neural network information extraction module extraction module aims to enhance the fusion embedding by generating additional emotional knowledge representations, mainly to extract emotional information at the word level. Using SenticNet, the emotional expression level of the extracted emotional words can be controlled by controlling the threshold.
[0062] Furthermore, in order to balance the contribution of the features extracted by the graph neural network, the features extracted by the text modality and speech modality through the modal interaction encoding module, and the original speech modality features to the final classification, the concatenated feature self-attention module is used for processing, and the final joint representation h' is obtained.
[0063] Furthermore, after obtaining the final joint representation, it is flattened in dimension and mapped to the sentiment classification through a linear layer, and then processed using softmax to obtain the final sentiment probability distribution, so that the distribution of sentiment can be more intuitively felt.
[0064] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.
[0065] In summary, the present invention expands a single speech modality to a text modality; utilizes newly extracted text modalities at different scales; prevents and processes noise existing in the text modality; obtains emotional features with stronger representation capabilities; and achieves good emotional classification effects.
[0066] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 This is a structural diagram of the speech emotion recognition model of the present invention;
[0068] Figure 2 This is a structural diagram of the modal interaction encoding module of the present invention;
[0069] Figure 3 This is a structural diagram of the text graph neural network information extraction module of the present invention;
[0070] Figure 4 This is a structural diagram of the splicing feature self-attention module of the present invention;
[0071] Figure 5 A schematic diagram of a computer device provided by an embodiment of the present invention;
[0072] Figure 6 A block diagram of an electronic device provided according to an embodiment of the present invention;
[0073] Figure 7 Schematic diagram of text information obtained by ASR;
[0074] Figure 8 Schematic diagram of the process of fine-tuning and extracting features;
[0075] Fig. 9 Schematic diagram of the input required for obtaining graph neural networks to process text obtained by ASR. DETAILED DESCRIPTION
[0076] The present invention provides a speech emotion recognition method introducing text modality, adopting a fusion network of text modality and speech modality, the two modalities are respectively input into a fusion neural network block for interactive information, and the fusion neural network block of each layer includes three attention blocks, the first attention block is used to integrate all information of its own features, the second fusion attention block is used to fuse the information of the interactive two modalities, and the third attention block is used to integrate the fused information; in order to better utilize the explicit emotional semantic information of the text, this paper proposes an emotional word graph neural network to extract features from the entire ASR text for emotion classification; through the modality fusion network and using the graph neural network to extract features from the explicit emotional semantic information of the text, the utilization efficiency of the text modality is high, and the fusion operation can be performed on the multimodal representation, and the introduction of the text auxiliary network helps the main network to obtain more effective information and reduce the influence of noise and harmful information.
[0077] See also Figure 1 The present invention provides a speech emotion recognition method introducing text modality, comprising the following steps:
[0078] S1, data preprocessing;
[0079] The multimodal sentiment analysis task is mainly aimed at input data of two or more modes, such as text information and audio content. After preprocessing and feature extraction of these raw data, they are sent to the feature fusion module, and then the high-dimensional vector is obtained and input into the prediction layer to obtain the final sentiment prediction result.
[0080] Specifically speaking of the data set, given a sentence, it has corresponding text content. The task to be done is to predict its emotional tendency based on the audio and the corresponding text. The prediction task is essentially a classification task.
[0081] IEMOCAP (The Interactive Emotional Dyadic Motion Capture) is an emotion classification dialogue dataset that contains 12 hours of multimodal emotion classification data. It is obtained by recording improvisations of 10 actors and actresses based on a script. Each scene contains two speakers. The IEMOCAP dataset is used for four-category emotion recognition.
[0082] The modal data in the dataset has been pre-processed before being sent to the fusion layer. The specific processing method is as follows:
[0083] S101. Audio data processing
[0084] Hubert is a self-supervised speech representation learning model for speech recognition, generation, and compression published by Facebook AI in 2021.
[0085] It uses a clustering-based pre-training method to learn high-quality speech features on a large amount of unlabeled speech data. The features extracted by Hubert can improve the accuracy of the recognition model. In order to improve Hubert's ability to extract speech information, Hubert is first fine-tuned using training data so that Hubert can better adapt to the classification task of the current dataset and obtain low-level features LLDs (low level descriptors) that are more suitable for downstream tasks.
[0086] The fine-tuning process usually adopts supervised learning methods, using labeled data as training data. The labeled data can be labels for speech recognition, text generated by speech, or target bitrate for speech compression. During the fine-tuning process, Hubert's parameters are updated based on the training data, allowing Hubert to better predict labels or generate text or compress speech.
[0087] The fine-tuned Hubert can extract richer speech information, thereby improving the performance of tasks such as speech recognition, generation, and compression.
[0088] The fine-tuned Hubert improves the recognition of speech features, the expressiveness of speech features, and the compression rate of speech features. Hubert's fine-tuning method is an effective method to improve the ability to extract speech features. It has achieved remarkable results in multiple speech processing tasks. Finally, an audio feature with a dimension of 768 and a length of 444 is extracted.
[0089] S102. Text data processing
[0090] As a powerful pre-training model, Roberta has significant advantages in extracting text features. It can learn the vocabulary and grammar knowledge of the text, as well as the semantic and logical relationships, thus providing a solid foundation for subsequent text analysis and understanding.
[0091] First, Roberta has the ability to learn the lexical features of text. Through pre-training, Roberta can learn the meaning and positional relationship of each word. This means that it can understand the similarities and differences between words, so as to better capture the semantic information in the text. This learning of lexical features enables Roberta to better understand and express concepts and entities in the text.
[0092] Secondly, Roberta can also learn the grammatical features of the text. Grammatical features refer to the relationship between sentence structures and components in the text. Through pre-training, Roberta can learn the rules and patterns between different sentence structures. This enables Roberta to better parse and analyze complex text structures, thereby extracting richer grammatical information.
[0093] In summary, Roberta has significant advantages in extracting text features; it can learn the vocabulary and grammar of the text, as well as the semantic and logical relations. These advantages make Roberta a powerful tool that can be used for various text processing tasks.
[0094] Before extracting features, Roberta is fine-tuned to make it better adapt to the current data set. The feature extraction capability on this data set will be further enhanced, and the emotional information contained in the text can be better extracted. Finally, an audio feature with a dimension of 768 and a length of 444 is extracted.
[0095] S103, using all the words to build a graph neural network of words. To ensure that explicit emotional words have a larger proportion, first use SenticNet to extract all words with emotional polarity greater than a specified threshold, and splice them into the original sentence to increase the proportion of emotional words. Finally, a graph neural network and a spliced sentence are obtained;
[0096] S104. Perform data division. The training set can be divided by the default division method in the official data set. Similarly, the development set and test set can also be divided. Then, the two modes of each data are standardized, the mean is subtracted and then divided by the variance to facilitate subsequent processing.
[0097] At the same time, for all data, features are obtained through pre-trained feature extractors, and all features are saved into .pkl files using the pickle module, which facilitates subsequent processing. Finally, dataloader is built to complete the data set setting and set the collection function.
[0098] S2. Construct a speech emotion recognition model that introduces text modality;
[0099] The goal of the task is to determine the emotional tendency of a given audio data. The overall architecture of the speech emotion recognition model that introduces text modality includes: ASR text extraction module, modality interaction encoding module, text graph neural network information extraction module, splicing feature self-attention module and prediction module.
[0100] ASR text extraction module
[0101] One of the shortcomings of current speech emotion recognition is that it cannot effectively utilize the speech information contained in the speech. In order to extract the semantic information of the speech, it is necessary to first perform speech recognition (ASR) on each segment of speech. The speech recognition model used in this invention is openai / whisper-large-v3, the purpose of which is to recognize and save all the speech, and participate in the task of emotion recognition as an extended modality. After being processed by the ASR module, the text information corresponding to the speech will be obtained, which is ready for the subsequent extraction of text explicit emotional words and text feature extraction.
[0102] Modal Interaction Encoding Module
[0103] It has a symmetrical structure, consisting of a text reference cross-modal encoder and a speech reference cross-modal encoder. Both cross-modal encoders are extended from the original transformer encoder to the bimodal scenario. Here, the text reference cross-modal encoder is taken as an example.
[0104] We first apply a multi-head self-attention mechanism to the audio sequence of each block of the cross-modal text encoder.
[0105]
[0106] in, is the output of the l-th layer cross-modal text encoder block. Q, K, V represent the query, key, and value in the multi-head attention mechanism.
[0107] Similar to previous work on transformer encoders, a residual connection and layer normalization (LN) are added to the self-attention output.
[0108]
[0109] Cross-modal attention is then used to learn the interaction between speech and text modalities, where It represents the output of the l-layer speech reference cross-modal encoder, with residual connections and layer normalization added at the end.
[0110]
[0111]
[0112] Then a multi-head self-attention mechanism is applied to integrate the previously obtained features, and finally the output of the module is obtained through a feedforward neural network.
[0113]
[0114]
[0115]
[0116] The entire module can be repeated n times; finally, the cross-modal text representation h can be obtained from the last layer of text reference cross-modal encoder t ∈R N×d ; For cross-modal text representation, the first token embedding is used, i.e. As a sentence.
[0117] Due to the symmetrical architecture of the modal interaction encoding module, the final speech representation is calculated in a mirrored manner, resulting in a cross-modal speech representation h a ∈R N×d ; For cross-modal audio representation, all speech embeddings are used as audio representation.
[0118] The modal interaction encoding module specifically includes:
[0119] (1) QKV fusion of attention mechanism
[0120] One of the key features of Transformer is the multi-head attention mechanism, which enables it to focus on multiple dimensions of input data at the same time, thereby capturing its inherent complex relationships. This mechanism gives Transformer excellent performance in a variety of tasks such as machine translation and sentiment analysis. With the help of multi-head attention, the features of different modalities can be integrated to achieve effective modality integration.
[0121] In order to effectively learn the information within the two modalities, the text feature ht and the voice feature ha are symmetrically used as query (Q), key (K), and value (V) inputs. The cross-modal attention mechanism in the text reference cross-modal encoder uses the text feature ht as Q and the voice feature ha as K, V. Because the modal interaction encoding module is symmetrical, the cross-modal attention mechanism in the voice reference cross-modal encoder uses the voice feature ha as Q and the voice feature ht as K, V. This helps the model learn the mapping relationship between features and labels more comprehensively. The fused features are repeatedly used as QKV for cross-modal attention. The attention mechanism calculates the dot product between the K vector and the Q vector and assigns weights to different features. In this way, the model can pay more attention to the intra-modal correlation between different features, thereby more effectively and robustly fusing emotional information. After the input passes through the modal interaction encoding module, an output feature map representing the fusion of text and sound modalities is obtained. As shown Figure 2 shown.
[0122] It is worth noting that in order to ensure the compatibility between the input features, the dimensions and lengths of the speech feature input and the text feature input are all set to be the same, that is, h t ,h a ∈R 444×768 .
[0123] (2) Text graph neural network information extraction module
[0124] In order to make better use of the sentiment-dominant words in the text modality, we first use SenticNet to extract all words with sentiment polarity greater than the specified threshold and concatenate them into the original sentence. This operation allows the graph neural network to pay more attention to sentiment-dominant words and increase the proportion of sentiment words. In the end, we get a graph neural network and a concatenated sentence.
[0125] At initialization, all necessary information and data are loaded, and during the forward propagation process, a graph is created for each sample (sentence). For each graph, embeddings are first assigned to its nodes and edges, and then the graph is updated using the aggregation operation of GNN; finally, the graph embedding is processed using a linear layer, Dropout, and Sigmoid activation function to generate the final graph features.
[0126] This model is particularly suitable for processing text data and is able to leverage information in the graph structure to generate meaningful text representations.
[0127] The Text Graph Neural Network Information Extraction module aims to enhance the fusion embedding by generating additional sentiment knowledge representation. It is based on the external SenticNet knowledge base and network embedding method. Specifically,
[0128] First, a sentiment knowledge graph is constructed for each sample’s text modality, and then the graph embedding calculation method is applied to obtain the final sentiment knowledge enhanced embedding.
[0129] An emotion knowledge graph is represented as G = (V, E, H), where V and E are the sets of emotion nodes and edges respectively, and H represents the feature set of each emotion node; ij =(v i ,v j )∈E represents the node v in V i and v j In order to better express the relationship between emotion nodes, N(v) is also introduced to represent the neighborhood of node v, where N(v) = {u∈V|e vu ∈E}; the adjacency matrix A is an n×n matrix, where n is the number of nodes.
[0130] Graph construction. Since not all words from the text modality are consistent with SenticNet, the original text is filtered to keep only words with letters and numbers. In addition, it is worth noting that only a few words in each sentence appear in SenticNet. Therefore, there are two types of nodes in the graph, V1 is the word in the original text, and V2 is the external sensitivity and sentiment knowledge of the word from SenticNet, V1∪V2=V. The corresponding sentiment word embedding is set as the node feature H, where the hi of the node vi is extracted by the unsimplified pre-trained model.
[0131] To construct the edges of the sentiment knowledge graph, assume that there is an edge eij between nodes vi and vj, where vi∈V1 and vj∈V2. In order to maximize the use of external sentiment knowledge, the interactions of internal nodes in V1 and V2 are ignored. Therefore, the adjacency matrix A is defined as follows:
[0132]
[0133] Among them, Wij is the weight of eij, which is proportional to the number of times vi and vj appear together. The frequency of each node vi is further obtained.
[0134]
[0135] Finally, the positive pointwise mutual information (PPMI) matrix is calculated to obtain the co-occurrence information of sentiment nodes:
[0136]
[0137] Among them, P(v i ,v j ) is the frequency of nodes vi and vj appearing simultaneously in the sampled random walk. In our model, the PPMI matrix is used to initialize the weights of the edges. Using the above node and edge initialization, the sentiment knowledge graph G is constructed.
[0138] Since the graph-level sentiment knowledge representation depends on each node, the sentiment feature of each node is first updated. For vi in G, a binary multiplication operation is then performed between the node feature and its edge weight to obtain information from neighboring nodes N(vi):
[0139]
[0140] Among them, b i It is the node sentiment feature after message delivery and aggregation.
[0141] Then calculate the node v i The final sentiment features are as follows:
[0142] h i =αhi +(1-α)h′ i
[0143] Among them, α is a learnable parameter with a default value of 0, and α is used to dynamically adjust the weight to prevent the degradation of node sentiment characteristics.
[0144] After aggregating the node emotion features, the embedded graph-level emotion knowledge h is finally generated. k :
[0145] h k =F(h1,h2,...,h i ,...,h n )
[0146] Where F is the readout function of G, and the global average operation of each node is adopted here.
[0147] Splicing feature self-attention module
[0148] It is designed to dynamically adjust the weight of each feature dimension of each modality and perform multimodal fusion. The functions of this module are as follows:
[0149] First, concatenate the four feature representations obtained above into a multimodal representation:
[0150]
[0151] Then, the weighted featurized attention matrix W is calculated att :
[0152] W att =W2(ReLU(W1*h))
[0153] Among them, W1 and W2 are learnable parameters, W1,W2∈R 4×d .
[0154] In addition, the ReLU activation function ensures that W att Each value in ranges from 0 to 1. In this way, the importance of each feature channel is obtained.
[0155] It is then applied to enhance the contribution of key features to the multimodal fusion process;
[0156] Finally, the connected representation is combined with W att Multiply them together to get the joint representation h':
[0157] h'=W att *h
[0158] Among them, h' is the final output of this module.
[0159] Prediction Module
[0160] After obtaining h', it is first expanded into a 1-dimensional tensor, and then passed through a fully connected layer and a Softmax activation function to obtain the predicted probability of the emotion.
[0161] P = Softmax(fc(flatten(h')))
[0162] Among them, P is the final output of the model, that is, the predicted probability; fc is the fully connected layer, and flatten is the dimensionality reduction function of multidimensional data.
[0163] S3. Input the data preprocessed in step S1 into the speech emotion recognition model introduced into the text modality constructed in step S2 to realize speech emotion recognition.
[0164] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".
[0165] In yet another embodiment of the present invention, a speech emotion recognition system introducing text modality is provided, which can be used to implement the above-mentioned speech emotion recognition method introducing text modality. Specifically, the speech emotion recognition system introducing text modality includes a processing module and a recognition module.
[0166] Wherein, the processing module performs preprocessing and feature extraction on the input data of two or more modalities;
[0167] The recognition module is sent to the speech emotion recognition model that introduces text modality to obtain the final emotion prediction result.
[0168] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to introduce the operation of the speech emotion recognition method of the text modality, including:
[0169] The input data of two or more modes are preprocessed and feature extracted; then they are sent to the speech emotion recognition model that introduces the text modality to obtain the final emotion prediction result.
[0170] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0171] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0172] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0173] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for speech emotion recognition with text modality introduced in the above embodiment; the processor may load and execute the following steps:
[0174] The input data of two or more modes are preprocessed and feature extracted; then they are sent to the speech emotion recognition model that introduces the text modality to obtain the final emotion prediction result.
[0175] See also Figure 5 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the method for calculating the composition of fluid in the reservoir transformation wellbore in the embodiment is realized. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the calculation system for the composition of fluid in the reservoir transformation wellbore in the embodiment are realized. To avoid repetition, it is not described one by one here.
[0176] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 5 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.
[0177] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0178] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 60.
[0179] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.
[0180] Any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0181] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0182] See also Figure 6 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0183] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .
[0184] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0185] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0186] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0187] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.
[0188] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0189] Experiments were conducted on a benchmark multimodal dataset for ER tasks: the IEMOCAP dataset. IEMOCAP has 9 emotion types: excitement, happiness, anger, sadness, frustration, neutral, fear, surprise, and other states. The training set, development set, and test set of IEMOCAP contain 4425, 552, and 554 sets of samples, respectively. To comprehensively analyze the performance of our method, we used weighted accuracy, unweighted accuracy, Micro-F1, and weighted F1 as evaluation metrics, and considered imbalanced label classification.
[0190] First, without any fusion or other processing measures, experiments were conducted on speech emotion classification using Hubert, and on text data obtained by ASR using Roberta. Then experiments were conducted using the entire model of the present invention to compare the results. In addition, multiple ablation studies were conducted to examine the role and impact of the modal interaction encoding module; the text graph neural network information extraction module; and the splicing feature self-attention module.
[0191] (I) Experimental results and analysis
[0192] Table 1 lists the comparison between the present model and the use of Hubert alone for speech sentiment classification and Roberta for sentiment classification of text data, demonstrating the effectiveness of the present model.
[0193] Table 1
[0194]
[0195] Table 2 lists the experimental results of four classifications of the IEMOCAP dataset. On the same dataset, the performance of the proposed model is compared with various baseline models for unimodal, bimodal and multimodal emotion recognition.
[0196] Table 2
[0197]
[0198]
[0199] As can be seen from Table 1, the model proposed in the present invention has a 0.8% improvement in accuracy compared to the single Huber sentiment classification; and has a 12.8% improvement in accuracy compared to the single ASR extracted text for Roberta classification.
[0200] The WAcc (Weighted-Accuracy) or weighted accuracy of the model proposed in the present invention can reach 77.1%, and the UAcc (Unweighted-Accuracy) or unweighted accuracy can reach 76.5%.
[0201] On the IEMOCAP dataset, by comparing with the existing models, it is found that the model of the present invention is superior to the existing models in both WAcc and UAcc. Compared with the models ASR-SER, IEmoNet, CMT, Fengetc.'snet that also use the ASR method for modal expansion, the unweighted accuracy rate can be improved by up to 13.1% and down to 0.1% on the IEMOCAP dataset; the weighted accuracy rate can be improved by up to 5.5% and down to 0.7%.
[0202] (II) Ablation experiment
[0203] Ablation experiments were conducted in IEMOCAP to test the effects of the modal interaction encoding module Cross_Encoder, the text graph neural network information extraction module textGNN, and the splicing feature self-attention module FAFW on the entire model.
[0204] The textGNN has a great influence on the model, which shows that the emotional explicit words in the text modality have a great effect on the sentiment classification ability of the model, and the WAcc effect is reduced by 1.47% compared with the whole model. Similarly, the designed modality interaction encoding module Cross_Encoder also has a 1.47% WAcc effect reduction on the whole model, which shows that the module designed by the present invention can well extract the relevant features in the modality, which is of great help to the task of sentiment recognition; the FAFW module also has an effect on the model, which can make the final features focus more on the information related to sentiment and reduce the influence of noise, and the WAcc effect is reduced by 0.9% compared with the whole model; the results are shown in Table 3.
[0205] Table 3
[0206]
[0207] The reason why the present invention has a good performance is mainly because we have expanded the text modality and used the modal interaction encoding module to effectively extract the relevant information between the text modality and the speech modality, and made good use of the emotional features provided by the emotionally explicit words of the extended text modality. In addition, the attention weighting of all the obtained features also plays a very important role.
[0208] The running results during the experiment are shown as follows:
[0209] (1) The text information obtained by ASR is as follows Figure 7 As shown,
[0210] (2) Fine-tuning and feature extraction process: Use the training set data to fine-tune the feature extractor (Roberta, Hubert), then extract the features of the text mode and speech mode, and generate a .pkl file for saving. Take the validation set as an example, Figure 8 shown.
[0211] (3) Process the text obtained by ASR to obtain the input required by the graph neural network. From top to bottom, the files are word embedding file, edge weight file, graph adjacency matrix, dictionary encoded array storage, all text information, emotional explicit words, vocabulary .npy file, vocabulary .txt file, such as Fig. 9 shown.
[0212] The parameter values of the model are as follows:
[0213] Learning rate: 1e -5 ; Optimizer: Adam optimizer; Batchsize: 8; Nlayers: 8; num_heads: 8; Dropout: 0.5; Epochs: 64.
[0214] In summary, the speech emotion recognition method and related devices introducing text modality of the present invention are compared with the speech emotion classification using Hubert alone and the emotion classification of text data using Roberta, which demonstrates the effectiveness of the model of the present invention; on the same data set IEMOCAP, the performance of the model of the present invention is compared with various baseline models of single-modal, bimodal and multimodal emotion recognition; by comparing with the existing models, the model of the present invention is superior to the existing models in both WAcc and UAcc, and compared with the models ASR-SER, IEmoNet, CMT, Feng etc.'s net that also use the ASR method for modal expansion, the unweighted accuracy rate can be improved by up to 13.1% and the minimum by 0.1% on the IEMOCAP data set; the weighted accuracy rate can be improved by up to 5.5% and the minimum by 0.7%.
[0215] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0216] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0217] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0218] The above contents are only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.
Claims
1. A speech emotion recognition method introducing text modality, characterized in that: The following steps are involved: Preprocessing and feature extraction of input data of two or more modalities; Then it is sent to the speech emotion recognition model that introduces the text modality to obtain the final emotion prediction result.
2. The speech emotion recognition method using text modality according to claim 1, characterized in that: The specific preprocessing and feature extraction are as follows: S101, using Hubert self-supervised speech representation learning model to process audio data; S102, using the Roberta pre-training model to process text data; S103, first use SenticNet to extract all words whose sentiment polarity is greater than a specified threshold, splice them into the original sentence, obtain a graph neural network and the spliced sentence, and use all the words to establish a graph neural network of words; S104, divide the data, complete the division of the training set by the default division method in the official data set, standardize the two modes of each data, subtract the mean and divide by the variance; S105. Obtain features through the pre-trained feature extractor, use the pickle module to save all features into the .pkl file, and finally build the dataloader to complete the setting of the data set and set the collection function.
3. The speech emotion recognition method introducing text modality according to claim 1 is characterized in that: The speech emotion recognition model that introduces text modality includes: The ASR text extraction module recognizes all speech and saves it as an extended modality to participate in the task of emotion recognition; Modal interaction encoding module, including text reference cross-modal encoder and speech reference cross-modal encoder. The text reference cross-modal encoder and speech reference cross-modal encoder are extended from the original transformer encoder to the bimodal scenario. The text graph neural network information extraction module builds a sentiment knowledge graph for each sample’s text modality, and then applies the graph embedding calculation method to obtain the final sentiment knowledge enhanced embedding; The splicing feature self-attention module dynamically adjusts the weight of each feature dimension of each modality and performs multimodal fusion to obtain the joint representation h'; The prediction module expands the joint representation h' into a 1-dimensional tensor, and then passes it through a fully connected layer and a Softmax activation function to obtain the predicted probability of the emotion.
4. The method for speech emotion recognition using text modality according to claim 3, characterized in that: The modal interaction encoding module is specifically: A multi-head self-attention mechanism is applied to the audio sequence of each block of the cross-modal text encoder, a residual connection and layer normalization are added to the self-attention output, and cross-modal attention is used to learn the interaction between speech and text modalities. At the same time, residual connections and layer normalization are added at the end, and a multi-head self-attention mechanism is applied to integrate the features obtained previously. Finally, the output of this module is obtained through a feedforward neural network, and the cross-modal text representation h is obtained from the last layer of text reference cross-modal encoder. t , for cross-modal text representation, the first token embedding is used, i.e. as a sentence representation; the final speech representation is calculated in a mirrored manner to obtain a cross-modal speech representation h a ; For cross-modal audio representation, all speech embeddings are used as audio representation.
5. The method for speech emotion recognition using text modality according to claim 3, characterized in that: The modal interaction encoding module specifically includes: QKV Fusion of Attention Mechanism The text feature ht and the speech feature ha are symmetrically used as query Q, key K and value V input; The cross-modal attention mechanism in the text citation cross-modal encoder uses the text feature ht as Q and the speech feature ha as K, V; The fused features are repeatedly used as QKV for cross-modal attention. The attention mechanism calculates the dot product between the K vector and the Q vector. After the input passes through the modal interaction encoding module, the output feature map representing the fusion of text and sound modalities is obtained. The dimensions and lengths of the input of speech features and the input of text features are all set to the same. Text graph neural network information extraction module: First, use SenticNet to extract all the words whose sentiment polarity is greater than the specified threshold, and splice them into the original sentence to obtain a graph neural network and the spliced sentence; during initialization, load all information and data, and create a graph for each sample during the forward propagation process; for each graph, first assign embeddings to its nodes and edges and then use the GNN aggregation operation to update the graph; finally, use a linear layer, Dropout, and Sigmoid activation function to process the graph embedding to generate the final graph features.
6. The method for speech emotion recognition using text modality according to claim 3, characterized in that: The splicing feature self-attention module is specifically: Concatenate the four obtained feature representations into a multimodal representation; calculate the weighted featurization attention matrix W att ; Applied to enhance the contribution of key features to the multimodal fusion process; Finally, the connected representation is combined with W att Multiply them together to get the joint representation h'.
7. The method for speech emotion recognition using text modality according to claim 3, characterized in that: The predicted probability P is as follows: P = Softmax(fc(flatten(h'))) Among them, fc is the fully connected layer, and flatten is the dimensionality reduction function for multi-dimensional data.
8. A speech emotion recognition system introducing text modality, characterized in that: include: A processing module performs preprocessing and feature extraction on input data of two or more modalities; The recognition module is sent to the speech emotion recognition model that introduces text modality to obtain the final emotion prediction result.
9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 7.
10. A computing device, characterized in that: include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 7.
Citation Information
Cited By
Speech recognition method and device, electronic equipment and computer readable storage medium
CN121393430A
