A multimodal speech emotion recognition method and system based on a dual cross-modal attention mechanism

Abstract

This invention provides a multimodal speech emotion recognition method and system based on a dual cross-modal attention mechanism, belonging to the fields of emotion computing and speech recognition. Addressing the shortcomings of existing methods in handling modal heterogeneity and insufficient cross-modal interaction modeling, this invention acquires paired audio and text data; utilizes a pre-trained BERT model combined with an NRC emotion dictionary and a BiLSTM network to extract text features, and extracts dual-scale MFCC and Mel spectrogram features, which are then encoded using a convolutional neural network to obtain speech features; constructs a dual cross-modal attention fusion model including CDAM and CMAM network branches. CDAM performs bidirectional cross-attention calculation to extract modal difference signals, while CMAM performs matching attention calculation on the difference enhancement features and the original modal features to obtain semantic alignment features; the aligned features are concatenated and fused before being input into a classifier to output the emotion recognition result. This invention improves the ability to model intermodal interactions, enhances the effect of modal heterogeneity handling, and improves the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, and more specifically, to a multimodal speech emotion recognition method and system based on a dual cross-modal attention mechanism. It belongs to the interdisciplinary field of multimodal affective computing, speech signal processing, natural language processing, and cross-modal feature fusion technology, and is applicable to emotion recognition tasks based on joint modeling of speech and text. Background Technology

[0002] Speech emotion recognition is an important research direction in the field of affective computing, aiming to infer the speaker's emotional state by analyzing acoustic features in speech signals (such as fundamental frequency, short-time energy, and Mel-frequency cepstral coefficients). Traditional methods mainly rely on a single speech modality, combined with classifiers such as Support Vector Machines (SVM) and Nearest Neighbors (KNN) to achieve emotion classification. However, the emotional information representation capability of a single modality is limited, making it difficult to distinguish between semantically similar but emotionally different sentences, thus restricting recognition accuracy and robustness.

[0003] To improve recognition performance, multimodal speech emotion recognition has gradually become a research hotspot. Text modality provides semantic information, while speech modality supplements non-linguistic features such as intonation and rhythm, making them complementary. While existing methods have improved recognition performance to some extent, the following problems remain: First, text and speech modalities differ significantly in feature distribution and time scale, making simple feature concatenation insufficient to fully integrate multimodal information; second, existing methods often focus on similarity alignment between modalities while neglecting the effective utilization of differences; furthermore, some methods fail to incorporate prior emotion knowledge during feature extraction or fail to jointly model multi-scale speech features; finally, the robustness of the model needs improvement in scenarios with long sequence dependencies and noise interference. Therefore, a method that can fully exploit complementary information between modalities and enhance contextual modeling capabilities is urgently needed to optimize the performance of multimodal speech emotion recognition. Summary of the Invention

[0004] To address the technical problems of insufficient modal heterogeneity handling, inadequate cross-modal interaction modeling, incomplete text and speech feature extraction, and low emotion recognition accuracy in existing multimodal speech emotion recognition methods, this invention provides a multimodal speech emotion recognition method and system based on a dual cross-modal attention mechanism. This invention primarily utilizes a dual cross-modal attention fusion model comprising a cross-modal differential attention (CDAM) network branch and a cross-modal matched attention (CMAM) network branch. It sequentially performs bidirectional cross-attention calculations on text features and speech features to extract modal difference signals, and then performs matched attention calculations on the modal difference signals and the original modal features to achieve semantic alignment. Simultaneously, it combines a text feature extraction scheme based on a pre-trained BERT model combined with an NRC sentiment dictionary and a BiLSTM network, and a speech feature extraction scheme based on dual-scale MFCC features, Mel spectrogram features, and a multi-branch convolutional network. This enhances the ability to mine intermodal difference and complementary information, improves the modal heterogeneity handling effect, and increases the accuracy and robustness of emotion recognition.

[0005] The technical means employed in this invention are as follows:

[0006] A multimodal speech emotion recognition method based on a dual cross-modal attention mechanism includes: S1. Acquire voice data, including audio data and text data corresponding to the audio data; S2. Extract text features from the text data to obtain text features; S3. Extract speech features from the audio data to obtain speech features; S4. Construct a dual cross-modal attention fusion model, including a cross-modal differential attention (CDAM) network branch and a cross-modal matching attention (CMAM) network branch. The CDAM network branch is used to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. The CMAM network branch is used to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. S5. The aligned text features and speech features are concatenated and fused to obtain fused features. The fused features are then input into the classifier to output the emotion recognition result.

[0007] Further, step S1 includes: S11. Obtain the audio data of the sentence to be recognized. The audio data is a digital audio signal containing the speaker's voice information. S12. Obtain text data corresponding to the audio data. The text data is speech recognition transcribed text or manually annotated text of the audio data, used to provide text information that corresponds to the semantics of the audio data. S13. Pair the audio data with the text data to create a voice data sample, wherein each sample contains a pair of paired audio data and text data.

[0008] Further, step S2 includes: S21. Let the input text sequence be... ,in, Indicates the first 1 term, Indicates the sequence length; S22. Use a pre-trained BERT model to map the input text sequence into context-sensitive initial word vectors. , means as follows:

[0009] in, , Indicates the embedding dimension; S23. To enhance the emotional information in the text, an NRC sentiment dictionary is introduced, which is used for each word in the text sequence. Obtain the corresponding sentiment weight vector from the NRC sentiment dictionary. This forms the sentiment weight matrix, as shown in the following formula:

[0010] S24. Adjust the sentiment weight matrix. With the initial word vector Perform element-wise multiplication to obtain the emotionally enhanced text. The formula is as follows:

[0011] in, Represents element-wise multiplication; S25. Text with enhanced emotion Inputting the BiLSTM network for bidirectional context modeling, let the output of the forward LSTM be... The output of the LSTM is... Then the first The text features at each time point are represented as follows:

[0012] This yields text features. , .

[0013] Further, step S3 includes: S31. Preprocess the input speech signal, including sampling rate unification, pre-emphasis, framing, windowing, and silence removal; S32. Extract time-scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain time-scale MFCC features. , ; S33. Extract frequency scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain frequency-scale MFCC features. , ; S34. Encode the time-scale MFCC features Frequency-scale MFCC characteristics By splicing the features together, the MFCC fusion feature is obtained. , ; S35. Let the output of the standard convolution branch be... The output of the Sigmoid-gated convolution branch is Then, gated fusion is performed to extract Mel spectrum features. The formula is as follows:

[0014] in, This represents the Sigmoid activation function; S36. Fuse the MFCC features and the Mel spectrum features The features are concatenated and mapped to the same dimensional space as the text features through a unified convolutional layer to obtain the speech features. , .

[0015] Further, step S4 includes: S41. In the cross-modal differential attention network branch (CDAM), using text features... and speech features Using one modality feature as the query and the other modality feature as the key and value, a linear mapping is performed on the text features and speech features respectively to obtain the query, key, and value, as shown in the following formula:

[0016]

[0017]

[0018] in, , , It is a trainable parameter matrix; S42. Based on the obtained query, key, and value, perform scaled dot product attention calculation to obtain the text-to-speech cross-attention and the speech-to-text cross-attention, as shown in the following formula:

[0019]

[0020] in, The dimension of the key vector; S43. Based on the cross-attention from text to speech and the cross-attention from speech to text, calculate the difference between the cross-attention and the corresponding original value matrix to obtain the modal difference signal from text to speech and the modal difference signal from speech to text. These signals are used to characterize the difference information between the speech modality and the text modality from the perspective of the text modality. The formula is as follows:

[0021]

[0022] S44. Based on the modal difference signals from text to speech and speech to text, respectively, they are fused with the original query, and then processed by residual connection, layer normalization, and multilayer perceptron to obtain the text-to-speech CDAM output. CDAM output in the speech-to-text direction The formula is as follows:

[0023]

[0024] S45. In the cross-modal matching attention (CMAM) network branch, the CDAM output in the text-to-speech direction... CDAM output in the speech-to-text direction As a query, attention is calculated using the original features of the corresponding modality as the key and value, as shown in the following formula:

[0025]

[0026] in, , , , ; S46. After residual connection, layer normalization, and multilayer perceptron processing, the aligned text features are obtained. and speech features The formula is as follows:

[0027]

[0028] The Cross-Modal Matching Attention (CMAM) network branch is used to realign the features after cross-modal interaction with the original semantics of the modality itself.

[0029] Further, step S5 includes: S51. Text features after semantic alignment and speech features By stitching along the channel dimension, the final fused feature is obtained. The formula is as follows:

[0030] S52, Final fusion features The input is a fully connected layer classifier. The features are then mapped to the probability distributions of four emotions: happiness, anger, sadness, and neutrality through the Softmax activation function. The emotion category corresponding to the maximum probability is taken as the final emotion recognition result.

[0031] This invention also provides a multimodal speech emotion recognition system based on the aforementioned multimodal speech emotion recognition method based on a dual cross-modal attention mechanism, comprising: a data acquisition module, a text feature extraction module, a speech feature extraction module, a dual cross-modal attention fusion module, and a classification and recognition module, wherein: The data acquisition module is used to acquire voice data, including audio data and text data corresponding to the audio data; The text feature extraction module is used to extract text features from text data to obtain text features; The speech feature extraction module is used to extract speech features from audio data to obtain speech features; The dual cross-modal attention fusion module is used to construct a dual cross-modal attention fusion model, including a cross-modal differential attention (CDAM) network branch and a cross-modal matching attention (CMAM) network branch. The CDAM network branch is used to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. The CMAM network branch is used to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. The classification and recognition module is used to concatenate and fuse the aligned text features and speech features to obtain fused features, and input the fused features into the classifier to output the emotion recognition result.

[0032] Compared with the prior art, the present invention has the following advantages: 1. The multimodal speech emotion recognition method based on dual cross-modal attention mechanism provided by this invention extracts text features by combining a pre-trained BERT model with an NRC emotion dictionary and a BiLSTM network, and extracts speech features by fusing dual-scale MFCC features and Mel spectrogram features through a multi-branch convolutional network. This achieves the effective introduction of prior knowledge of text emotion and the comprehensive capture of multi-level acoustic emotion information of speech, thereby improving the integrity and accuracy of emotion representation.

[0033] 2. The multimodal speech emotion recognition method based on dual cross-modal attention mechanism provided by this invention constructs a dual cross-modal attention fusion model. It performs bidirectional cross-attention calculation on text features and speech features through cross-modal differential attention (CDAM) network branches to explicitly extract modal difference signals, and combines cross-modal matched attention (CMAM) network branches to perform matched attention calculation on the modal difference signals and the original modal features. This achieves effective mitigation of modal heterogeneity and deep semantic interaction between text and speech, ensuring that the fused features have both cross-modal complementary information and original semantic consistency.

[0034] 3. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism provided in this invention, through visual analysis of the attention weights of CDAM and CMAM, enables the model to automatically focus on emotional keywords in the text and energy-transitional segments in the speech. The attention distribution has a clear emotional orientation, improving the interpretability of the model. The model adopts a dual-path feature extraction and modular attention fusion architecture, which is easy to extend and modify, and can be generalized to different multimodal emotion recognition scenarios.

[0035] In summary, the technical solution of this invention solves the problems of insufficient modal heterogeneity processing, inadequate cross-modal interaction modeling, incomplete text and speech feature extraction, and low accuracy and insufficient robustness of emotion recognition in the prior art.

[0036] Based on the above reasons, this invention can be widely applied in fields such as multimodal emotion recognition. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1This is an overall flowchart of the multimodal speech emotion recognition method of the present invention.

[0039] Figure 2 This is a diagram showing the overall structure of the multimodal speech emotion recognition model of the present invention. Figure 3 This is a branch structure diagram of the cross-modal differential attention (CDAM) network of the present invention; Figure 4 This is a branch structure diagram of the cross-modal matching attention (CMAM) network provided in an embodiment of the present invention; Figure 5 A performance comparison chart of the model of the present invention and the baseline model on the IEMOCAP dataset provided for embodiments of the present invention; Figure 6 A performance comparison chart of the model of the present invention and the baseline model on the MELD dataset is provided for embodiments of the present invention. Figure 7 The figure shows the ablation experiment results provided in the embodiments of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or device.

[0042] To avoid ambiguity, some terms used in this specification are further explained below.

[0043] Text features refer to the continuous vector representation obtained by encoding text data. The representation may contain semantic information or emotional information introduced by external emotional priors.

[0044] Speech features refer to the continuous vector representation obtained by acoustic feature extraction and convolutional coding of audio data, and the representation includes at least one or more of MFCC features and Mel spectrogram features.

[0045] Cross-modal differential attention (CDAM) network branches refer to obtaining a weighted representation of one modality on another through cross-attention, and then subtracting it from the original value matrix to extract the difference information.

[0046] The Cross-Modal Matching Attention (CMAM) network branch refers to using the differentially enhanced modal features as queries and the original modal features as keys and values ​​to perform attention calculations, thereby completing intra-modal realignment and semantic enhancement.

[0047] Sentiment enhancement refers to weighting sentiment-related terms in a text using an external sentiment dictionary or equivalent prior knowledge, thereby increasing their discriminative power in the final representation.

[0048] The terms used above are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention.

[0049] like Figure 1 As shown, this invention provides a multimodal speech emotion recognition method based on a dual cross-modal attention mechanism, comprising: S1. Acquire voice data, including audio data and text data corresponding to the audio data; S2. Extract text features from the text data to obtain text features; S3. Extract speech features from the audio data to obtain speech features; S4. Construct a dual cross-modal attention fusion model, such as Figure 2 As shown, it includes a cross-modal differential attention (CDAM) network branch and a cross-modal matching attention (CMAM) network branch. The CDAM network branch is used to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. The CMAM network branch is used to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. S5. The aligned text features and speech features are concatenated and fused to obtain fused features. The fused features are then input into the classifier to output the emotion recognition result.

[0050] In a specific implementation, as a preferred embodiment of the present invention, step S1 includes: S11. Obtain the audio data of the sentence to be recognized. The audio data is a digital audio signal containing the speaker's voice information. S12. Obtain text data corresponding to the audio data. The text data is speech recognition transcribed text or manually annotated text of the audio data, used to provide text information that corresponds to the semantics of the audio data. S13. Pair the audio data with the text data to create a voice data sample, wherein each sample contains a pair of paired audio data and text data.

[0051] In a specific implementation, as a preferred embodiment of the present invention, step S2 includes: S21. Let the input text sequence be... ,in, Indicates the first 1 term, Indicates the sequence length; S22. Use a pre-trained BERT model to map the input text sequence into context-sensitive initial word vectors. , means as follows:

[0052] in, , Indicates the embedding dimension; S23. To enhance the emotional information in the text, an NRC sentiment dictionary is introduced, which is used for each word in the text sequence. Obtain the corresponding sentiment weight vector from the NRC sentiment dictionary. This forms the sentiment weight matrix, as shown in the following formula:

[0053] S24. Adjust the sentiment weight matrix. With the initial word vector Perform element-wise multiplication to obtain the emotionally enhanced text. The formula is as follows:

[0054] in, This represents element-wise multiplication; this operation can highlight emotion-related words in text representations.

[0055] S25. Text with enhanced emotion Inputting the BiLSTM network for bidirectional context modeling, let the output of the forward LSTM be... The output of the LSTM is... Then the first The text features at each time point are represented as follows:

[0056] This yields text features. , In this embodiment, the hidden state dimension of the BiLSTM network in each direction is set to 128, the concatenated text feature dimension is 256, and dropout can be set to 0.2.

[0057] In a specific implementation, as a preferred embodiment of the present invention, step S3 includes: S31. Preprocess the input speech signal, including sampling rate unification, pre-emphasis, framing, windowing, and silence removal; S32. Extract time-scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain time-scale MFCC features. , In this embodiment, the number of channels in the one-dimensional convolution branch is 128. S33. Extract frequency scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain frequency-scale MFCC features. , In this embodiment, the number of channels in the one-dimensional convolution branch is 128. S34. Encode the time-scale MFCC features Frequency-scale MFCC characteristics By splicing the features together, the MFCC fusion feature is obtained. , ; S35. Let the output of the standard convolution branch be... The output of the Sigmoid-gated convolution branch is Then, gated fusion is performed to extract Mel spectrum features. The formula is as follows:

[0058] in, This represents the Sigmoid activation function; in this embodiment, the Mel spectrum can be extracted using an 80-dimensional Mel filter with a frequency range of 0–8 kHz.

[0059] S36. Fuse the MFCC features and the Mel spectrum features The features are concatenated and mapped to the same dimensional space as the text features through a unified convolutional layer to obtain the speech features. , In this embodiment, speech features The dimension is 256.

[0060] In a specific implementation, as a preferred embodiment of the present invention, step S4 includes: S41, such as Figure 3As shown, in the Cross-Modal Differential Attention Network (CDAM) branch, text features are used... and speech features Using one modality feature as the query and the other modality feature as the key and value, a linear mapping is performed on the text features and speech features respectively to obtain the query, key, and value, as shown in the following formula:

[0061]

[0062]

[0063] in, , , It is a trainable parameter matrix; S42. Based on the obtained query, key, and value, perform scaled dot product attention calculation to obtain the text-to-speech cross-attention and the speech-to-text cross-attention, as shown in the following formula:

[0064]

[0065] in, The dimension of the key vector; S43. Based on the cross-attention from text to speech and the cross-attention from speech to text, calculate the difference between the cross-attention and the corresponding original value matrix to obtain the modal difference signal from text to speech and the modal difference signal from speech to text. These signals are used to characterize the difference information between the speech modality and the text modality from the perspective of the text modality. The formula is as follows:

[0066]

[0067] S44. Based on the modal difference signals from text to speech and speech to text, respectively, they are fused with the original query, and then processed by residual connection, layer normalization, and multilayer perceptron to obtain the text-to-speech CDAM output. CDAM output in the speech-to-text direction The formula is as follows:

[0068]

[0069] S45, such as Figure 4 As shown, in the Cross-Modal Matching Attention (CMAM) network branch, the CDAM output is in the text-to-speech direction. CDAM output in the speech-to-text direction As a query, attention is calculated using the original features of the corresponding modality as the key and value, as shown in the following formula:

[0070]

[0071] in, , , , ; S46. After residual connection, layer normalization, and multilayer perceptron processing, the aligned text features are obtained. and speech features The formula is as follows:

[0072]

[0073] The Cross-Modal Matching Attention (CMAM) network branch is used to realign the features after cross-modal interaction with the original semantics of the modality itself.

[0074] In a specific implementation, as a preferred embodiment of the present invention, step S5 includes: S51. Text features after semantic alignment and speech features By stitching along the channel dimension, the final fused feature is obtained. The formula is as follows:

[0075] S52, Final fusion features The input is a fully connected layer classifier. The features are then mapped to the probability distributions of four emotions: happiness, anger, sadness, and neutrality through the Softmax activation function. The emotion category corresponding to the maximum probability is taken as the final emotion recognition result.

[0076] This invention also provides a multimodal speech emotion recognition system based on the aforementioned multimodal speech emotion recognition method based on a dual cross-modal attention mechanism, comprising: a data acquisition module, a text feature extraction module, a speech feature extraction module, a dual cross-modal attention fusion module, and a classification and recognition module, wherein: The data acquisition module is used to acquire voice data, including audio data and text data corresponding to the audio data; The text feature extraction module is used to extract text features from text data to obtain text features; The speech feature extraction module is used to extract speech features from audio data to obtain speech features; The dual cross-modal attention fusion module is used to construct a dual cross-modal attention fusion model, including a cross-modal differential attention (CDAM) network branch and a cross-modal matching attention (CMAM) network branch. The CDAM network branch is used to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. The CMAM network branch is used to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. The classification and recognition module is used to concatenate and fuse the aligned text features and speech features to obtain fused features, and input the fused features into the classifier to output the emotion recognition result.

[0077] The embodiments of the present invention are described simply because they correspond to those in the embodiments above. For any similarities, please refer to the descriptions in the embodiments above, which will not be elaborated here.

[0078] Example This embodiment uses the IEMOCAP and MELD multimodal emotion recognition datasets to validate the method of the present invention. The IEMOCAP dataset contains approximately 12 hours of dialogue from 10 professional actors, totaling 10,039 sentences; the MELD dataset comes from the TV series "Friends," containing 1,433 dialogues and 13,708 sentences. The IEMOCAP dataset is divided into training, validation, and test sets in an 8:1:1 ratio; the MELD dataset uses the official partitioning method. This embodiment implements the multimodal speech emotion recognition method based on a dual cross-modal attention mechanism based on the IEMOCAP and MELD datasets, and the specific steps are as follows: Text preprocessing employs natural word segmentation, stop word removal, punctuation and number removal, and standardizes the text to lowercase. Speech preprocessing includes pre-emphasis, 16kHz resampling, 25ms framing, 10ms frame shift, Hamming windowing, and silence removal.

[0079] The text is encoded using a pre-trained BERT-base model, and sentiment weight vectors are generated by combining the NRC sentiment dictionary to perform element-level sentiment enhancement on the initial word vectors. Then, a single-layer BiLSTM is used for context modeling, with a hidden state dimension of 128 and an output dimension of 256 for each direction.

[0080] On the speech side, dual-scale MFCC features and Mel spectrogram features are jointly extracted. Temporal-scale MFCC and frequency-scale MFCC are extracted via... and Encoding is performed using a one-dimensional convolutional kernel with 128 channels; the Mel spectrum uses an 80-dimensional Mel filter with a frequency range set to 0. 8kHz, via After dual-branch convolutional coding and gating fusion, it is then mapped to 256-dimensional speech features through a unified convolutional layer.

[0081] The attention space dimension in CDAM and CMAM can be set to 64. CDAM uses bidirectional cross-attention to obtain differential enhancement features in both text-to-speech and speech-to-text directions. CMAM uses the above enhancement features as queries and aligns them with their respective original modal features as keys and values. Finally, the two aligned features are concatenated and input into the fully connected layer classifier.

[0082] like Figure 5 As shown, experimental results on the IEMOCAP dataset demonstrate that the unweighted accuracy (UA) of the multimodal model is 75.24%, the weighted accuracy (WA) is 74.37%, and the overall accuracy (Acc) is 74.38%. On the MELD dataset, the unweighted accuracy (UA) of the model is 72.46%, the weighted accuracy (WA) is 71.19%, and the overall accuracy (Acc) is 72.62%.

[0083] like Figure 6 As shown, ablation experiments were performed on the IEMOCAP dataset. The difference signals... When the original value matrix was used as a substitute, UA decreased to 71.27% and WA decreased to 69.72%; when the CDAM network branch was removed, UA was 71.13% and WA was 69.80%; when the CMAM network branch was removed, UA was 71.88% and WA was 70.53%; the UA and WA of the complete model were 75.24% and 74.37%, respectively. These results indicate that the difference signal, CDAM network branch, and CMAM network branch all have a positive effect on improving the performance of the proposed solution.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal speech emotion recognition method based on a dual cross-modal attention mechanism, characterized in that, include: S1. Acquire voice data, including audio data and text data corresponding to the audio data; S2. Extract text features from the text data to obtain text features; S3. Extract speech features from the audio data to obtain speech features; S4. Construct a dual cross-modal attention fusion model, including a cross-modal differential attention network branch and a cross-modal matching attention network branch. Use the cross-modal differential attention network branch to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. Use the cross-modal matching attention network branch to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. S5. The aligned text features and speech features are concatenated and fused to obtain fused features. The fused features are then input into the classifier to output the emotion recognition result.

2. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism according to claim 1, characterized in that, Step S1 includes: S11. Obtain the audio data of the sentence to be recognized. The audio data is a digital audio signal containing the speaker's voice information. S12. Obtain text data corresponding to the audio data. The text data is speech recognition transcribed text or manually annotated text of the audio data, used to provide text information that corresponds to the semantics of the audio data. S13. Pair the audio data with the text data to create a voice data sample, wherein each sample contains a pair of paired audio data and text data.

3. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism according to claim 1, characterized in that, Step S2 includes: S21. Let the input text sequence be... ,in, Indicates the first 1 term, Indicates the sequence length; S22. Use a pre-trained BERT model to map the input text sequence into context-sensitive initial word vectors. , means as follows: in, , Indicates the embedding dimension; S23. To enhance the emotional information in the text, an NRC sentiment dictionary is introduced, which is used for each word in the text sequence. Obtain the corresponding sentiment weight vector from the NRC sentiment dictionary. This forms the sentiment weight matrix, as shown in the following formula: S24. Adjust the sentiment weight matrix. With the initial word vector Perform element-wise multiplication to obtain the emotionally enhanced text. The formula is as follows: in, Represents element-wise multiplication; S25. Text with enhanced emotion Inputting the BiLSTM network for bidirectional context modeling, let the output of the forward LSTM be... The output of the LSTM is... Then the first The text features at each time point are represented as follows: This yields text features. , .

4. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism according to claim 1, characterized in that, Step S3 includes: S31. Preprocess the input speech signal, including sampling rate unification, pre-emphasis, framing, windowing, and silence removal; S32. Extract time-scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain time-scale MFCC features. , ; S33. Extract frequency scale features from the preprocessed speech signal. And through convolution kernels for Encoding is performed using a one-dimensional convolutional branch to obtain frequency-scale MFCC features. , ; S34. Encode the time-scale MFCC features Frequency-scale MFCC characteristics By splicing the features together, the MFCC fusion feature is obtained. , ; S35. Let the output of the standard convolution branch be... The output of the Sigmoid-gated convolution branch is Then, gated fusion is performed to extract Mel spectrum features. The formula is as follows: in, This represents the Sigmoid activation function; S36. Fuse the MFCC features and the Mel spectrum features The features are concatenated and mapped to the same dimensional space as the text features through a unified convolutional layer to obtain the speech features. , .

5. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism according to claim 1, characterized in that, Step S4 includes: S41. In the cross-modal differential attention network branch, using text features and speech features Using one modality feature as the query and the other modality feature as the key and value, a linear mapping is performed on the text features and speech features respectively to obtain the query, key, and value, as shown in the following formula: in, , , It is a trainable parameter matrix; S42. Based on the obtained query, key, and value, perform scaled dot product attention calculation to obtain the text-to-speech cross-attention and the speech-to-text cross-attention, as shown in the following formula: in, The dimension of the key vector; S43. Based on the cross-attention from text to speech and the cross-attention from speech to text, calculate the difference between the cross-attention and the corresponding original value matrix to obtain the modal difference signal from text to speech and the modal difference signal from speech to text. These signals are used to characterize the difference information between the speech modality and the text modality from the perspective of the text modality. The formula is as follows: S44. Based on the modal difference signals from text to speech and speech to text, the original query is fused with the original query, and then processed by residual connection, layer normalization, and multilayer perceptron to obtain the text-to-speech CDAM output. CDAM output in the speech-to-text direction The formula is as follows: S45. In the cross-modal matching attention network branch, the CDAM output in the text-to-speech direction... CDAM output in the speech-to-text direction As a query, attention is calculated using the original features of the corresponding modality as the key and value, as shown in the following formula: in, , , , ; S46. After residual connection, layer normalization, and multilayer perceptron processing, the aligned text features are obtained. and speech features The formula is as follows: The cross-modal matching attention network branch is used to realign the features after cross-modal interaction with the original semantics of the modality itself.

6. The multimodal speech emotion recognition method based on a dual cross-modal attention mechanism according to claim 1, characterized in that, Step S5 includes: S51. Text features after semantic alignment and speech features By stitching along the channel dimension, the final fused feature is obtained. The formula is as follows: S52, Final fusion features The input is a fully connected layer classifier. The features are then mapped to the probability distributions of four emotions: happiness, anger, sadness, and neutrality through the Softmax activation function. The emotion category corresponding to the maximum probability is taken as the final emotion recognition result.

7. A multimodal speech emotion recognition system based on a dual cross-modal attention mechanism, implemented using the multimodal speech emotion recognition method based on any one of claims 1-6, characterized in that, include: The module includes a data acquisition module, a text feature extraction module, a speech feature extraction module, a dual cross-modal attention fusion module, and a classification and recognition module, among which: The data acquisition module is used to acquire voice data, including audio data and text data corresponding to the audio data; The text feature extraction module is used to extract text features from text data to obtain text features; The speech feature extraction module is used to extract speech features from audio data to obtain speech features; The dual cross-modal attention fusion module is used to construct a dual cross-modal attention fusion model, including a cross-modal differential attention network branch and a cross-modal matching attention network branch. The cross-modal differential attention network branch is used to perform bidirectional cross-attention calculation on text features and speech features and extract modal difference signals. The cross-modal matching attention network branch is used to perform matching attention calculation on modal difference signals and original modal features to obtain aligned text features and speech features. The classification and recognition module is used to concatenate and fuse the aligned text features and speech features to obtain fused features, and input the fused features into the classifier to output the emotion recognition result.

Images