A speech emotion classification method based on context information enhancement and cross attention

Abstract

The present invention proposes a speech emotion classification method based on context information enhancement and cross attention, including obtaining a speech data set including a speech signal, preprocessing the speech data set to obtain a text data set including text data; inputting the speech data set into an audio modality encoder to extract context representation to obtain speech features; inputting the text data set into a BERT pre-training model to extract text features to obtain text features; inputting speech features and text features into a cross-modal fusion module, integrating speech features into text features when learning audio modality features to obtain speech multimodal fusion features; integrating text features into speech features when learning text modality features to obtain text multimodal fusion features; inputting speech multimodal fusion features and text multimodal fusion features into a decision layer, performing average pooling, connection and classification to obtain classification results. The present invention can make the classification results of speech emotions more accurate.

Description

Technical Field

[0001] The present invention relates to the field of speech emotion recognition, and in particular to a speech emotion classification method based on context information enhancement and cross attention.

[0002] Background

[0003] Speech Emotion Recognition (SER) technology focuses on identifying and understanding emotional states from speech signals. This technology is widely used in human-computer interaction, intelligent customer service and other fields to enhance the machine's perception and response to human emotions. Speech emotion recognition methods are mainly divided into two steps: speech feature extraction and model recognition. In the speech feature extraction stage, the common practice is to extract acoustic features such as spectrograms, Mel frequency cepstral coefficients MFCC, pitch and its harmonics, or jitter from speech signals, and then use traditional classifiers such as GMM, HMM, SVM to classify them. However, these acoustic features are difficult to accurately capture complex speech emotions, so the effect is limited.

[0004] With the development of deep learning, speech emotion recognition methods based on deep learning have become increasingly mainstream. For example, Yang et al. proposed a method to extract complementary information using waveforms and spectrograms, and used stacked BLSTM layers for arousal and valence classification, which improved the speech emotion recognition effect of a single feature. However, these descriptive features are difficult to fully capture emotional information, resulting in limited classification effects. In recent years, by constructing a deep learning model that integrates multimodal data, it is possible to make full use of effective information from different sources, and use technologies such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to simultaneously encode and extract features of speech and text data. Existing studies have shown that compared with traditional single-modal methods, dual-modal emotion recognition methods have achieved significant improvements in accuracy and robustness. For example, Yoon et al. proposed using dual recurrent neural networks (RNNs) to encode information in audio and text sequences, and then combined the information from these sources to predict emotion categories. The model uses both text data and audio signals to better understand speech data. In addition, the use of self-attention mechanisms and cross-modal attention can fuse the attention weights of speech and text information to obtain more accurate emotion recognition results. For example, Xu et al. used an attention network to learn the alignment between speech and text, aiming to produce more accurate multimodal feature representations, and then used a Bi-LSTM network to learn emotions. Zou et al. used CNN, BiLSTM, and wav2vec2.0 to extract multi-level acoustic information, including MFCC, spectrogram, and embedded high-level acoustic information. These extracted features were then used as multimodal inputs and fused through the proposed joint attention mechanism.

[0005] Although significant progress has been made in the field of speech emotion recognition in recent years, existing technologies still have limitations in making full use of contextual information. Existing methods often fail to effectively identify and utilize emotional information at key moments when dealing with emotional accumulation and changes in longer conversations. For example, although the emotion recognition method based on pre-trained models mentioned by Wu et al. has made some progress in extracting speech features and contextual representation of text, such models rely on the speech and script after the current speech. In real application scenarios, the information after the current speech is unknown information, so it is not suitable for real application scenarios. In addition, the multi-scale SER parallel network based on the connection attention mechanism proposed by Chen et al., which combines fine-grained frame-level manual features and coarse-grained speech-level deep features, performs well in capturing speech signal features, but its sensitivity to complex conversation situations and context evolution still needs to be improved. Existing technologies have limitations in the cross-attention mechanism. The cross-attention mechanism shows potential when integrating multimodal information such as audio and text, but existing models such as MCSAN proposed by Sun et al. and the joint attention network proposed by Zou et al. still face some challenges. Although these models can handle the interaction between audio and text, they are not ideal when dealing with the ever-changing emotional dynamics and multi-faceted interactions in conversations. In addition, although the research of Xu et al. has made progress in aligning audio and text signals, it is still limited in capturing fine-grained emotional changes and complex interaction patterns in conversations. In addition, current technology also faces the challenge of accurately identifying emotions in complex conversation environments. Emotional expressions in conversations are often affected by multiple factors, such as the speaker's personality, the environmental background of the conversation, and the interactive relationship between speakers. Existing technologies still have a lot of room for improvement in capturing the impact of these complex factors on emotional expression. In view of the above-mentioned problems, it is necessary to study and design a new speech emotion classification method based on contextual information enhancement and cross-attention to overcome the problems existing in existing speech emotion classification methods. Summary of the invention

[0006] In order to solve the problems that existing speech emotion classification methods are difficult to accurately capture complex speech emotions, context information is not fully utilized, and the cross-attention mechanism has limitations, resulting in low classification accuracy, the present invention proposes a speech emotion classification method based on context information enhancement and cross-attention.

[0007] The present invention provides a speech emotion classification method based on context information enhancement and cross attention, comprising the following steps:

[0008] S1. Acquire a speech data set including a speech signal, preprocess the speech data set to obtain a text data set including text data;

[0009] S2. Input the speech data set obtained in step S1 into the audio modality encoder to extract context representation and obtain speech features; input the text data set into the BERT pre-trained model to extract text features and obtain text features;

[0010] S3. Input the speech features and text features obtained in step S2 into a cross-modal fusion module, wherein the speech features are integrated into the text features during audio modal feature learning to obtain speech multimodal fusion features; and the text features are integrated into the speech features during text modal feature learning to obtain text multimodal fusion features;

[0011] S4. Input the speech multimodal fusion features and text multimodal fusion features obtained in step S3 into the decision layer, perform average pooling, connection and classification, and obtain a classification result.

[0012] According to a method for speech emotion classification based on context information enhancement and cross attention in some embodiments of the present invention, in step S1, preprocessing the speech data set includes generating text data through a Wave2Vect2 model, as shown in formula (1):

[0013]

[0014] Among them, x t represents text data, X represents speech signal, ASR represents speech text recognition algorithm, X n Represents the starting conversation in the speech signal, n∈(0,1,2,3,4…).

[0015] According to a method for speech emotion classification based on context information enhancement and cross attention in some embodiments of the present invention, in step S2, the audio modality encoder includes a feature encoder module, a Transformer context representation module and a quantization module;

[0016] The feature encoder module is used to process the input speech signal into low-level features;

[0017] The Transformer context representation module is used to map the input speech signal to a feature space that better represents data features;

[0018] The quantization module is used to discretize low-level features into a trainable codebook.

[0019] According to a speech emotion classification method based on contextual information enhancement and cross-attention in some embodiments of the present invention, in step S2, the audio modality encoder is a Wav2vec2.0 model.

[0020] According to a method for speech emotion classification based on contextual information enhancement and cross attention in some embodiments of the present invention, in step S2, the BERT pre-training model extracts text features from the text data in the text dataset through bert-base-uncased, as shown in formula (2):

[0021] hidden TE =BERT(x t ) (2)

[0022] Among them, hidden TE represents text features, x t Represents text data, and BERT stands for bert-base-uncased.

[0023] According to a method for speech emotion classification based on context information enhancement and cross-attention in some embodiments of the present invention, in step S3, the cross-modal fusion module includes a cross-attention layer and a modal feature alignment fusion layer;

[0024] The cross attention layer is used for audio modality feature learning. The cross attention layer performs deep semantic extraction on the speech features through multi-head cross attention between modalities, as shown in formula (3):

[0025] mht A =Concat(head A1 ,head A2 ,head A3 ,...,head A8 )W A O (3)

[0026] Among them, mht A It represents the deep semantic extraction result of speech features. Concat represents multi-head cross attention between modalities. Ai represents the attention extraction of the i-th head of speech features, i∈(1,2,…8), W A O represents the speech output transformation matrix,

[0027] Attention extraction of the i-th head of speech features Ai As shown in formula (4):

[0028]

[0029] Among them, Attention A represents the audio modality cross attention, Q A The query matrix representing the speech features, represents the query transformation matrix of the i-th head of speech features, K A The key matrix representing the speech features, represents the key transformation matrix of the i-th head of speech features, V A A matrix of values ​​representing speech features, Represents the value transformation matrix of the i-th head of speech features,

[0030] Audio modality cross attention A The calculation formula is shown in (5):

[0031]

[0032] Among them, K TE The key matrix representing the text features, V TE represents the value matrix of text features, T represents the transpose operation of the matrix, represents the scaling factor,

[0033] The cross attention layer is also used for text modality feature learning. The cross attention layer performs deep semantic extraction on the text features through multi-head cross attention between modalities, as shown in formula (6):

[0034] mht TE =Concat(head TE1 ,head TE2 ,head TE3 ,...,head TE8 )W TE O (6)

[0035] Among them, mht TE Indicates the deep semantic extraction results of text features, head TEi represents the attention extraction of the i-th head of text features, i∈(1,2,…8), W TE O represents the text output transformation matrix,

[0036] Text feature attention extraction head of the i-th head TEi As shown in formula (7):

[0037]

[0038] Among them, Attention TE represents the text modality cross attention, Q TE The query matrix representing the text features, represents the query transformation matrix of the i-th head of the text feature, K TE The key matrix representing the text features, Represents the key transformation matrix of the i-th head of the text feature, V TE A matrix of values ​​representing text features, Represents the value transformation matrix of the i-th head of the text feature,

[0039] Text modality cross attention TE The calculation formula is shown in (8):

[0040]

[0041] The modal feature alignment fusion layer is used to extract the deep semantics of the speech feature mht A Combined with the speech features, the speech multimodal fusion features are obtained, as shown in formula (9):

[0042] CrossAtt A =mht A +hidden A (9)

[0043] Among them, CrossAtt A Indicates the multimodal fusion features of speech, hidden A Represents the voice features,

[0044] The modal feature alignment fusion layer is also used to extract the deep semantics of the text features mht TE Combined with the text features, we get the text multimodal fusion features, as shown in formula (10):

[0045] CrossAtt TE =mht TE +hidden TE (10)

[0046] Among them, CrossAtt TE Indicates the multimodal fusion features of text, hidden TE Represents text features.

[0047] According to some embodiments of the present invention, a speech emotion classification method based on contextual information enhancement and cross-attention is provided. In step S3, the cross-attention layer performs deep semantic extraction on the speech features through 8-head inter-modal multi-head cross-attention. The cross-attention layer performs deep semantic extraction on the speech features through 8-head inter-modal multi-head cross-attention.

[0048] According to a method for speech emotion classification based on contextual information enhancement and cross attention according to some embodiments of the present invention, in step S4, the speech multimodal fusion features are average-pooled at the decision layer to obtain an audio feature vector, as shown in formula (11):

[0049]

[0050] Among them, x' W represents the audio feature vector, x W represents the speech segment in the speech dataset, x' W-1 The audio feature vector generated for the last speech segment,

[0051] The text multimodal fusion features are averaged and pooled at the decision layer to obtain a text feature vector, as shown in formula (12):

[0052]

[0053] Among them, x TE ' W represents the text feature vector, x TEW Represents a text segment in a text dataset, x TE ' W-1 The text feature vector generated for the last speech segment,

[0054] The audio feature vector x' w and the text feature vector x TE ' W The connected feature vector is obtained by connecting them, and the connected feature vector is classified into multiple categories using a linear classifier to obtain a classification result, as shown in formula (13):

[0055]

[0056] in, represents the classification result, w represents the weight to be learned, and b represents the bias to be learned.

[0057] According to a speech emotion classification method based on context information enhancement and cross-attention according to some embodiments of the present invention, in step S4, the loss function is a category cross entropy loss function.

[0058] According to a method for speech emotion classification based on contextual information enhancement and cross-attention in some embodiments of the present invention, in step S4, the classification result is one of happiness, neutrality, sadness or anger.

[0059] The present invention proposes a speech emotion classification method based on contextual information enhancement and cross-attention, which can make full use of contextual information, enhance contextual semantic information based on contextual information, and better understand the cumulative emotions of speech signals by utilizing the contextual information enhancement of text data and audio data. In addition, the method can obtain the alignment between multimodal signals and the weights of different feature mappings through the cross-attention mechanism of contextual information enhancement, and can pay more attention to key information, thereby making the classification results of speech emotions more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of a speech emotion classification method based on context information enhancement and cross-attention of the present invention. DETAILED DESCRIPTION

[0061] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0062] Example 1

[0063] A speech emotion classification method based on context information enhancement and cross attention, such as Figure 1 As shown, the following steps are included:

[0064] S1. Acquire a speech data set including a speech signal, preprocess the speech data set to obtain a text data set including text data;

[0065] Specifically, preprocessing the speech data set includes generating text data through the Wave2Vect2 model, as shown in formula (1):

[0066]

[0067] Among them, x t represents text data, X represents speech signal, ASR represents speech text recognition algorithm, X n Indicates the starting session in the speech signal, n∈(0,1,2,3,4…). In this embodiment, the Wave2Vect2 model version used in the preprocessing process may be wav2vec2-base-960h;

[0068] S2. Input the speech data set obtained in step S1 into the audio modality encoder to extract context representation and obtain speech features; input the text data set into the BERT pre-trained model to extract text features and obtain text features;

[0069] Specifically, the audio modality encoder includes a feature encoder module, a Transformer context representation module, and a quantization module;

[0070] Among them, the feature encoder module is used to process the input speech signal into low-level features; the Transformer context representation module is used to map the input speech signal to a feature space that can better represent the data features; the quantization module is used to discretize the low-level features into a trainable codebook. During the training process of the feature encoder module, some low-level features are masked by the Transformer context representation module in order to identify the numerical vector of the masked features based on their context. In addition, the feature encoder module of this embodiment may include 12 Transformer context representation modules.

[0071] The audio modality encoder can be a Wav2vec2.0 model, which can be applied to various task objectives. In the process of training the Wav2vec2.0 model, this embodiment performs pre-training fine-tuning to improve Wav2vec2.0 to adapt to the speech emotion recognition task of this embodiment. The original audio signal in the speech data set is directly sent to the wav2vec2.0 processor to generate the wav2vec2.0 model pre-training representation. The original audio signal used is sampled at 16kHz, and each audio session is divided into several speech segments. When the length of a speech segment is less than 7.7 seconds, padding is used to keep the same length.

[0072] In step S2, the BERT pre-trained model extracts text features from the text data in the text dataset through bert-base-uncased, as shown in formula (2):

[0073] hidden TE =BERT(x t ) (2)

[0074] Among them, hidden TE represents text features, x t Represents text data, and BERT stands for bert-base-uncased.

[0075] S3. Input the speech features and text features obtained in step S2 into the cross-modal fusion module, wherein the speech features are integrated into the text features when learning the audio modal features to obtain speech multimodal fusion features; and the text features are integrated into the speech features when learning the text modal features to obtain text multimodal fusion features;

[0076] In step S3, the cross-modal fusion module includes a cross-attention layer and a modality feature alignment fusion layer;

[0077] The cross-attention layer is used for audio modality feature learning. The cross-attention layer performs deep semantic extraction of speech features through multi-head cross-attention between modalities, as shown in formula (3):

[0078] mht A =Concat(head A1 ,head A2 ,head A3 ,...,head A8 )W A O (3)

[0079] Among them, mht A It represents the deep semantic extraction result of speech features. Concat represents multi-head cross attention between modalities. Ai represents the attention extraction of the i-th head of speech features, i∈(1,2,…8), W A O represents the speech output transformation matrix,

[0080] Attention extraction of the i-th head of speech features Ai As shown in formula (4):

[0081]

[0082] Among them, Attention A represents the audio modality cross attention, Q A The query matrix representing the speech features, represents the query transformation matrix of the i-th head of speech features, K A The key matrix representing the speech features, represents the key transformation matrix of the i-th head of speech features, V A A matrix of values ​​representing speech features, Represents the value transformation matrix of the i-th head of speech features,

[0083] Audio modality cross attention A The calculation formula is shown in (5):

[0084]

[0085] Among them, K TE The key matrix representing the text features, V TE represents the value matrix of text features, T represents the transpose operation of the matrix, represents the scaling factor,

[0086] The cross-attention layer is also used for text modality feature learning. The cross-attention layer performs deep semantic extraction on text features through multi-head cross-attention between modalities, as shown in formula (6):

[0087] mht TE=Concat(head TE1 ,head TE2 ,head TE3 ,...,head TE8 )W TE O (6)

[0088] Among them, mht TE Indicates the deep semantic extraction results of text features, head TEi represents the attention extraction of the i-th head of text features, i∈(1,2,…8), W TE O represents the text output transformation matrix,

[0089] Text feature attention extraction head of the i-th head TEi As shown in formula (7):

[0090]

[0091] Among them, Attention TE represents the text modality cross attention, Q TE The query matrix representing the text features, represents the query transformation matrix of the i-th head of the text feature, K TE The key matrix representing the text features, Represents the key transformation matrix of the i-th head of the text feature, V TE A matrix of values ​​representing text features, Represents the value transformation matrix of the i-th head of the text feature,

[0092] Text modality cross attention TE The calculation formula is shown in (8):

[0093]

[0094] The modal feature alignment fusion layer is used to extract the deep semantics of speech features into mht A Combined with the speech features, the speech multimodal fusion features are obtained, as shown in formula (9):

[0095] CrossAtt A =mht A +hidden A (9)

[0096] Among them, CrossAtt A Indicates the multimodal fusion features of speech, hidden A Represents the voice features,

[0097] The modal feature alignment fusion layer is also used to extract the deep semantics of text features into mht TE Combined with the text features, we get the text multimodal fusion features, as shown in formula (10):

[0098] CrossAtt TE =mht TE +hidden TE (10)

[0099] Among them, CrossAtt TE Indicates the multimodal fusion features of text, hidden TE Represents text features.

[0100] In step S3, the cross-attention layer performs deep semantic extraction on speech features through 8-head inter-modal multi-head cross-attention. The cross-attention layer performs deep semantic extraction on speech features through 8-head inter-modal multi-head cross-attention.

[0101] When processing the converted text data and speech signals through the cross-modal fusion module, it is possible to focus on the key features of the speech signal and the text data at the same time. By mapping the audio features and text data to another high-dimensional representation specifically used for fusion, the original data of the audio features and text features and the key emotional information between them can be identified, and the understanding of this information can be enhanced to reduce information loss.

[0102] S4. Input the speech multimodal fusion features and text multimodal fusion features obtained in step S3 into the decision layer, perform average pooling, connection and classification, and obtain the classification result.

[0103] In step S4, the speech multimodal fusion features are averaged and pooled at the decision layer to obtain the audio feature vector, as shown in formula (11):

[0104]

[0105] Among them, x' W represents the audio feature vector, x W represents the speech segment in the speech dataset, x' W-1 The audio feature vector generated for the last speech segment is output as the audio feature vector. The decision-making method enhanced by speech segments can effectively obtain the ability to capture segment-level information useful for emotion recognition, which is helpful for speech emotion recognition and cross-modal attention alignment.

[0106] The text multimodal fusion features are averaged and pooled at the decision layer to obtain the text feature vector, as shown in formula (12):

[0107]

[0108] Among them, x TE ' W represents the text feature vector, x TEW Represents a text segment in a text dataset, x TE ' W-1 The text feature vector generated for the last speech segment takes the representation output of the last speech segment as the text feature vector. The decision-making method enhanced by the text segment can effectively obtain the ability to capture segment-level information useful for emotion recognition, which is helpful for speech emotion recognition and cross-modal attention alignment.

[0109] The audio feature vector x' w and the text feature vector x TE ' W Connect them to get the connection feature vector, use the linear classifier to classify the connection feature vector into multiple categories, and use the category cross entropy loss function as the loss function to get the classification result, as shown in formula (13):

[0110]

[0111] in, represents the classification result, w represents the weight to be learned, and b represents the bias to be learned.

[0112] In step S4, the classification result is one of happiness, neutrality, sadness or anger.

[0113] Example 2

[0114] A speech emotion classification method based on contextual information enhancement and cross-attention in this embodiment verifies the effectiveness of the method of the present invention on the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset, a public dataset for speech emotion analysis. The IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset is collected by the Sail Laboratory of the University of Southern California and contains information in four modes: speech, vision, text, and gestures. The dataset covers approximately 12 hours of audio and video files recorded by five actors and five actresses. The files are divided into Session 1 to Session 5, a total of 5 groups. Participants perform dialogues in English, and the performances are divided into improvisations and fixed script scene performances. The average duration of dialogues in the IEMOCAP dataset is 4.5 seconds, and the average number of words is 11.4. The speech information and text data in the IEMOCAP dataset are selected as the experimental dataset, and the four common emotion tags in the IEMOCAP dataset are identified, including anger, sadness, happiness, and neutrality, totaling 5531. The distribution of IEMOCAP dataset samples in the experiment of this embodiment is shown in Table 1.

[0115] Table 1. Sample distribution of IEMOCAP experimental dataset

[0116]

[0117] The speech signal used in this embodiment is sampled at 16kHz, and each audio session is divided into several speech segments of 7.7 seconds in length. When a speech segment is less than 7.7 seconds, a padding operation with 0 will be applied to the speech segment to keep the same length. The final prediction result of each audio session utterance will be determined by all the segmented segments of the utterance. The method of this embodiment is implemented in PyTorch, and the optimizer of the model adopts AdamW with a learning rate of 1e-5. The batch size of the training is 36, and the early stopping cycle of the training is set to 20 epochs. The entire method is pre-trained and trained on the Nvidia V100 32G GPU. In order to make a fair comparison with the previous method, the method proposed in this embodiment splits the data set based on the principle of session independence for five-fold cross-validation, that is, when one session is used as a test set, the other four sessions are used as training sets.

[0118] According to the general evaluation index of multi-category speech emotion classification in the literature, the evaluation index of the speech emotion classification method in this embodiment adopts weighted accuracy (WA) and unweighted accuracy (UA). The weighted accuracy WA assumes that each sample has an equal weight, and the weighted average of the prediction results of each category is obtained. The calculation is shown in formula (14):

[0119]

[0120] Among them, TP represents the number of samples correctly classified as the i-th category of emotion, FP represents the number of samples incorrectly classified as the i-th category, and N represents the number of samples.

[0121] The unweighted accuracy UA is the average accuracy of the prediction results of all categories, focusing on the comprehensive average recognition performance of various types of emotions, as shown in formula (15):

[0122]

[0123] Where Acci represents the classification accuracy of each emotion category, as shown in formula (16):

[0124]

[0125] This embodiment selects six advanced speech emotion classification methods and compares them with the method of the present invention based on the five-fold cross-validation experimental results of Session-independent dataset segmentation on the IEMOCAP public dataset. The six advanced speech emotion classification methods include:

[0126] CMAspec: A convolutional neural network based on cross-modal attention and speech spectrogram, proposed in 2021.

[0127] CMARawwaveform: A convolutional neural network based on cross-modal attention and raw waveforms, proposed in 2021.

[0128] TSIN: A multimodal emotion recognition model based on temporal and semantic consistency proposed in 2021.

[0129] TSB&TAB ASR text: a multimodal sentiment recognition model proposed in 2021 that integrates time-synchronous and time-asynchronous representations. The text modality uses the text generated by speech recognition as input, and the text context window is [-3,+3].

[0130] TSB&TAB original text: a multimodal sentiment recognition model proposed in 2021 that integrates time-synchronous and time-asynchronous representations. The text modality uses the original text in the dataset as input, and the text context window is [-3,+3].

[0131] Coattention&Fusion: A multimodal co-attention and multi-level fusion emotion recognition model proposed in 2022.

[0132] AuxiliaryTasks: A multimodal sentiment recognition model based on randomly recombined text and audio inputs, proposed in 2022.

[0133] The experimental results are shown in Table 3, where “--” indicates that only UA results are provided in the literature, and WA results are not provided.

[0134] Table 2 Experimental results of comparison with existing models

[0135]

[0136]

[0137] For the sake of fairness, the control group using speech and original text as input is first compared. As can be seen from Table 2, compared with the convolutional neural network CMAspec machine improvement method based on cross-modal attention and speech spectrogram using original text, and the convolutional neural network CMARawwaveform based on cross-modal attention and original waveform, the speech emotion recognition accuracy of the method in this embodiment is significantly improved, and the unweighted accuracy UA is increased by 10.26% and 9.66% respectively. Compared with the multimodal emotion recognition model TSIN based on time and semantic consistency, the weighted accuracy WA and unweighted accuracy UA of speech emotion recognition of the method in this embodiment are increased by 6.72% and 5.86% respectively. Compared with the Coattention&Fusion and AuxiliaryTasks models that use speech and original text as input, the unweighted accuracy UA of the method in this embodiment is significantly improved, by 6.19% and 2.79% respectively. Compared with the TSB&TAB model that uses speech and original text as input, when TSB&TAB uses more text information [-3, +3] as input, the unweighted accuracy UA of the method in this embodiment is 1.28% higher.

[0138] Then, the models using speech and ASR text as input are compared. The TSB&TAB ASR text model adopts a text context window of [-3,+3] and uses more text input information than the method in this embodiment. However, the speech emotion recognition weighted accuracy WA and unweighted accuracy UA of the ASR model proposed in this embodiment are higher than those of the TSB&TAB ASR text model, and the advantages are more significant. The weighted accuracy WA and unweighted accuracy UA are 5.42% and 5.41% higher than those of the TSB&TAB ASR text model, respectively.

[0139] In order to explore the role and influence of speech multimodal fusion features and text multimodal fusion features on speech emotion classification, this embodiment conducts an ablation experiment for comparison, as shown in Table 3.

[0140] Table 3 Modal ablation experimental results

[0141]

[0142] It can be seen from Table 3 that when the speech multimodal fusion features and the ASR text multimodal fusion features are used simultaneously, that is, the speech modal enhancement information and the automatic recognition ASR text enhancement information are used simultaneously, the weighted accuracy WA and the unweighted accuracy UA are 9.96% and 10.3% higher than those of the speech multimodal fusion features, that is, the speech enhancement information, respectively, and the weighted accuracy WA and the unweighted accuracy UA are 6.29% and 6.01% higher than those of the ASR text multimodal fusion features, that is, the ASR text enhancement information, respectively. This shows that both the speech multimodal fusion features and the ASR text multimodal fusion features have made significant contributions to the improvement of the accuracy of the emotion recognition task. When the speech multimodal fusion features and the text multimodal fusion features of the original text are used at the same time, that is, the speech modality enhancement information and the original text enhancement information are used at the same time, the weighted precision WA and the unweighted precision UA ​​are 15.22% and 15.49% higher than those of using the speech multimodal fusion features, that is, the speech enhancement information alone, respectively, and the weighted precision WA and the unweighted precision UA ​​are 3.84% and 3.03% higher than those of using the text multimodal fusion features of the original text, that is, the original text enhancement information alone. This once again shows that both the speech multimodal fusion features and the text multimodal fusion features of the original text have made significant contributions to the improvement of the accuracy of the emotion recognition task, and also proves that the original text has better emotion recognition ability than the ASR text. The reason is that the accuracy of automatic recognition of ASR text cannot reach the accuracy of the original text.

[0143] In order to explore the effect of context information enhancement on the emotion recognition task, the results of the context information enhancement ablation experiment are compared in this embodiment, as shown in Table 4.

[0144] Table 4 Ablation experimental results of context information enhancement

[0145]

[0146] As can be seen from Table 4, when the speech multimodal fusion features and the original text multimodal fusion features are used simultaneously, that is, the speech modality enhancement information and the original text enhancement information are used simultaneously, the weighted accuracy WA and the unweighted accuracy UA are 4.83% and 4.1% higher than those of the original text. When the speech enhancement information is eliminated again and only the original speech and original text information are used, the weighted accuracy WA and the unweighted accuracy UA drop by 2.23% and 2.51% respectively, indicating that the speech multimodal fusion features and the original text multimodal fusion features, that is, the speech modality enhancement information and the text enhancement information, have made significant contributions to the improvement of the accuracy of the emotion recognition task.

[0147] In order to further explore the effect of contextual information enhancement on the emotion recognition task, in this embodiment, an experiment is conducted to compare the effect of the window length of text enhancement information on emotion recognition. In the experiment, Session 1-4 of the IEMOCAP dataset is used as the training set, and Session 5 is used as the test set. The experimental results are shown in Table 5.

[0148] Table 5 Experimental results on the impact of text enhancement information window length on emotion recognition

[0149]

[0150] It can be seen from Table 5 that when the window length of the original text enhancement information increases from 0 to -3, the weighted accuracy WA and unweighted accuracy UA of the speech emotion recognition experimental results gradually increase, indicating that the longer the window length of the text enhancement information, the more significant the positive impact on speech emotion classification.

[0151] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A speech emotion classification method based on context information enhancement and cross attention, characterized in that: The steps include: S1. Obtaining a speech data set including a speech signal, preprocessing the speech data set to obtain a text data set including text data; S2. The speech data set obtained in step S1 is input into an audio modality encoder for context representation extraction to obtain speech features; Input the text data set into the BERT pre-training model to extract text features and obtain text features; S3. The speech features and text features obtained in step S2 are input into a cross-modal fusion module, wherein the speech features are integrated into the text features when learning audio modal features to obtain speech multimodal fusion features; and the text features are integrated into the speech features when learning text modal features to obtain text multimodal fusion features; S4. The speech multimodal fusion features and text multimodal fusion features obtained in step S3 are input into the decision layer, and average pooling, connection and classification are performed to obtain a classification result; In step S3, the cross-modal fusion module includes a cross-attention layer and a modality feature alignment fusion layer; The cross attention layer is used for audio modality feature learning. The cross attention layer performs deep semantic extraction on the speech features through multi-head cross attention between modalities, as shown in formula (1): （1） in, Represents the deep semantic extraction results of speech features, represents multi-head cross attention between modalities, represents the attention extraction of the i-th head of speech features, , represents the speech output transformation matrix, The cross attention layer is also used for text modality feature learning. The cross attention layer performs deep semantic extraction on the text features through multi-head cross attention between modalities, as shown in formula (2): （2） in, Represents the deep semantic extraction results of text features, represents the attention extraction of the i-th head of text features, , represents the text output transformation matrix, The modal feature alignment fusion layer is used to extract the deep semantics of the speech features. Combined with the speech features, we get the speech multimodal fusion features, as shown in formula (3): （3） in, Represents the multimodal fusion features of speech, Represents the voice features, The modal feature alignment fusion layer is also used to extract the deep semantics of the text features. Combined with the text features, we get the text multimodal fusion features, as shown in formula (4): （4） in, Represents the multimodal fusion features of text, Represents text features.

2. A speech emotion classification method based on context information enhancement and cross attention according to claim 1, characterized in that: In the step S1, preprocessing the speech data set includes generating text data through the Wave2Vect2 model, as shown in formula (5): （5） Among them, x t represents text data, X represents speech signal, ASR represents speech text recognition algorithm, X n Indicates the start of a conversation in a speech signal, .

3. A speech emotion classification method based on context information enhancement and cross attention according to claim 1, characterized in that: In step S2, the audio modality encoder includes a feature encoder module, a Transformer context representation module and a quantization module; The feature encoder module is used to process the input speech signal into low-level features; The Transformer context representation module is used to map the input speech signal to a feature space that better represents data features; The quantization module is used to discretize low-level features into a trainable codebook.

4. A method for speech emotion classification based on context information enhancement and cross attention according to claim 3, characterized in that: In step S2, the audio modal encoder is a Wav2vec2.0 model.

5. The method for speech emotion classification based on context information enhancement and cross attention according to claim 1, characterized in that: In step S2, the BERT pre-trained model extracts text features from the text data in the text dataset through bert-base-uncased, as shown in formula (6): （6） in, represents text features, x t Represents text data, and BERT stands for bert-base-uncased.

6. A method for speech emotion classification based on context information enhancement and cross attention according to claim 5, characterized in that: Attention extraction of the i-th head of speech features As shown in formula (7): （7） in, represents audio modality cross attention, The query matrix representing the speech features, represents the query transformation matrix of the i-th head of speech features, The key matrix representing the speech features, represents the key transformation matrix of the i-th head of speech features, A matrix of values ​​representing speech features, Represents the value transformation matrix of the i-th head of speech features, Audio modality cross attention The calculation formula is shown in (8): （8） in, The key matrix representing the text features, represents the value matrix of text features, T represents the transpose operation of the matrix, represents the scaling factor, Attention extraction of the i-th head of text features As shown in formula (9): （9） in, represents text modality cross attention, The query matrix representing the text features, represents the query transformation matrix of the i-th head of the text feature, The key matrix representing the text features, represents the key transformation matrix of the i-th head of the text feature, A matrix of values ​​representing text features, Represents the value transformation matrix of the i-th head of the text feature, Text modality cross attention The calculation formula is shown in (10): （10）。 7. The method for speech emotion classification based on context information enhancement and cross attention according to claim 6, characterized in that: In the step S3, the cross-attention layer performs deep semantic extraction on the speech features through 8-head inter-modal multi-head cross-attention. The cross-attention layer performs deep semantic extraction on the speech features through 8-head inter-modal multi-head cross-attention.

8. The method for speech emotion classification based on context information enhancement and cross attention according to claim 6, characterized in that: In step S4, the speech multimodal fusion features are averaged and pooled at the decision layer to obtain an audio feature vector, as shown in formula (11): （11） in, represents the audio feature vector, represents a speech segment in the speech dataset, The audio feature vector generated for the last speech segment, The text multimodal fusion features are averaged and pooled at the decision layer to obtain a text feature vector, as shown in formula (12): （12） in, represents the text feature vector, Represents a text segment in a text dataset, The text feature vector generated for the last speech segment, The audio feature vector and text feature vector The connected feature vector is obtained by connecting them, and the connected feature vector is classified into multiple categories using a linear classifier to obtain a classification result, as shown in formula (13): （13） in, represents the classification result, w represents the weight to be learned, and b represents the bias to be learned.

9. The method for speech emotion classification based on context information enhancement and cross attention according to claim 8, characterized in that: In step S4, the loss function is a category cross entropy loss function.

10. The method for speech emotion classification based on context information enhancement and cross attention according to claim 1, characterized in that: In step S4, the classification result is one of happiness, neutrality, sadness or anger.

Images