A mamba-based multi-modal dialogue emotion recognition method and system

Abstract

The application provides a Mamba-based multi-modal dialogue emotion recognition method and system, and belongs to the field of dialogue emotion recognition; solves the problems of limited memory capacity, insufficient use of information, and noisy accumulation in long sequence fusion of the current multi-modal emotion recognition method; includes the following steps: different preprocessing methods are used to process dialogue data under different modes to obtain corresponding feature encodings; different feature encodings are convolved and information embedded; external attention is used to capture semantic information inside each mode; a cross-fusion mechanism is used to realize interaction between different modes; Kalman filtering is used to filter noise in each mode and establish a connection between modes; grouping pooling and cross-attention are used to fuse the semantic information of different modes, and then the fusion result is fused with the result of Kalman filtering to obtain a fusion result; the fusion result is mapped to a predefined emotion category; and the application is applied to multi-modal dialogue emotion recognition.

Description

Technical Field

[0001] This application relates to the field of multimodal dialogue emotion technology in natural language processing, and in particular to a multimodal dialogue emotion recognition method and system based on Mamba. Background Technology

[0002] Emotion Recognition in Conversation (ERC) is gaining increasing importance and widespread attention in the fields of artificial intelligence and natural language processing. For example, emotion computing technology can enhance emotional communication between teachers and students and improve the interactivity of teaching contexts in education and training; assist in disease diagnosis and doctor-patient communication in the life and health field; improve the accuracy of customer service empathy and financial risk identification in business services; improve the performance of humanoid robots in industrial design; and be used for security monitoring and public credibility assessment in social governance.

[0003] Multimodal dialogue emotion recognition refers to the process of collecting video, audio, and text data from a speaker within a given dialogue and classifying the emotion of each utterance. The input to this task is a continuous dialogue, and the output is the emotion of all utterances in that dialogue. Unlike emotion recognition for a single sentence, emotion recognition in dialogue requires comprehensive consideration of background, context, and speaker information; therefore, integrating information from text, audio, and visual modalities is crucial.

[0004] Recently, deep learning has been widely used in multimodal sentiment analysis of dialogues. The main methods adopted by researchers are: (1) using recurrent neural networks for recognition, such as long short-term memory networks or gated recurrent unit neural networks. This network can process sequential data and combine it with historical data to generate long-term memory. It can also handle inputs of different lengths. When processing long sequences, it solves the problems of gradient vanishing and gradient explosion. At the same time, it introduces a gating mechanism to adjust the degree of information transmission in the sequence, thereby achieving better capture of long-distance dependencies; (2) using graph convolutional neural networks to record a series of information fragments in nodes and using the edges between nodes to capture complex relationships and dependencies. This method constructs the data into a graph structure, capturing the connections between each node that traditional models cannot obtain. However, modeling intermodal relationships is difficult and can easily introduce redundant information, thereby reducing the effect of multimodal feature fusion in emotion recognition; (3) Using external knowledge bases or large language models to enhance the richness of information, enhance context awareness, and improve the generalization ability of the model is crucial for building efficient artificial intelligence systems. The intervention of external knowledge can effectively reduce the ambiguity and uncertainty that a single model may bring, enabling the model to acquire information from a wider range of knowledge sources, thereby enhancing its robustness and stability; (4) In addition, the Transformer model, through its self-attention mechanism, can more accurately capture long-distance dependencies and performs well in processing long sequence data and tasks involving long-distance semantic relationships. By aggregating the data of the entire sequence, it deeply learns the intrinsic connections between data, further improving the performance and accuracy of the model.

[0005] While many works have successfully extracted semantic information about emotions, they still have some limitations. Compared to methods that utilize Mamba to capture semantic information, recurrent neural networks have limited memory capacity for long sequences, while Transformers and large language models require significant computational resources. Large language models, in particular, not only consume enormous resources but also pose a risk of privacy breaches and perform poorly in terms of debugging and interpretability. However, simply using state transition equations ignores the connections between modalities, and long sequence modeling can lead to noise accumulation. Summary of the Invention

[0006] To address the problems existing in current multimodal emotion recognition methods, such as limited memory capacity, underutilization of information, and noise accumulation during long sequence fusion, this application proposes a multimodal dialogue emotion recognition method and system based on Mamba. It adopts a dual fusion mechanism of cross-fusion and Kalman filtering fusion; it also utilizes Kalman filtering technology to unify the differences between different modalities, and captures global attention through segmented pooling technology to achieve accurate identification and analysis of emotions.

[0007] The technical solution adopted in this application is as follows:

[0008] The advantages of this application over existing technologies are as follows: A multimodal dialogue emotion recognition method based on Mamba, comprising the following steps:

[0009] Step 1: Process dialogue data in different modalities using different preprocessing methods to obtain the corresponding feature codes;

[0010] Step 2: Perform convolution and information embedding on different feature codes;

[0011] Step 3: Use external attention to capture the semantic information within each modality;

[0012] Step 4: Use a cross-fusion mechanism to achieve interaction between different modalities and obtain a more comprehensive modal representation;

[0013] Step 5: Use Kalman filtering to filter noise in each mode and establish the relationship between modes;

[0014] Step 6: Use group pooling and multi-layer attention to fuse the semantic information of different modalities, and then fuse it with the result of Kalman filtering to obtain the fused result;

[0015] Step 7: Construct a linear mapping layer to map the fusion result to a predefined emotion category. At this point, a multimodal dialogue emotion recognition model based on Mamba is obtained.

[0016] Step 8: Construct the corresponding loss function to train the model.

[0017] The dialogue data in step 1 includes audio modality data, text modality data, and visual modality data. The preprocessing methods for the three are as follows:

[0018] Step 1.1: Use the openSMILE model to process the audio modal data to obtain audio features;

[0019] Step 1.2: Use the RoBERTa model to process the text modal data to obtain text features;

[0020] Step 1.3: Use the DenseNet model to process the visual modal data to obtain visual features.

[0021] Furthermore, step 2 specifically includes:

[0022] Step 2.1: Perform one-dimensional convolution on the feature encodings of each modality to unify the feature dimension of the hidden layer and obtain a consistent data structure;

[0023] Step 2.2: Embed speaker information and location information into the model so that the model can distinguish between different speakers and their sequential location relationships.

[0024] Furthermore, step 4 specifically includes:

[0025] Step 4.1: Use the Mamba model to construct a state-space model to process the input tensor and construct a shared matrix between modalities. The text modal data is passed to the speech modal data and video modal data respectively, and the input values ​​are calculated in the forward and backward directions respectively.

[0026] Step 4.2: Perform multi-cross fusion on the forward and reverse outputs respectively. Use the text and visual outputs from the forward Mamba as queries and keys respectively, calculate cross attention, and concatenate the result with the original text as the forward text-visual attention. Then, use the forward text-visual attention and the audio output from the forward Mamba as queries and keys, calculate cross attention, and concatenate the result with the forward text-visual attention as the forward text-visual-audio attention. Similarly, calculate the reverse text-visual-audio attention, and concatenate the forward text-visual-audio attention and the reverse text-visual-audio attention to obtain the fused text-visual-audio attention.

[0027] Furthermore, step 5 specifically includes:

[0028] Step 5.1: Calculate the prior estimates of the predicted state vector and the predicted observation vector for the state transition equation of the single-mode feature, respectively;

[0029] Step 5.2: Calculate the predicted state covariance and predicted observation covariance for each single-modal feature;

[0030] Step 5.3: Calculate the Kalman filter gain matrix;

[0031] Step 5.4: Calculate the posterior estimates of the single-modal features and update the posterior error covariance.

[0032] Furthermore, step 6 specifically includes:

[0033] Step 6.1: Use group pooling to represent single-modal abstract features;

[0034] Step 6.2: Use cross attention to fuse the abstract features of different modalities, and after passing them through global attention, fuse them with the results of Kalman filtering.

[0035] Furthermore, the expression for the loss function L in step 8 is as follows:

[0036]

[0037] In the formula: and These are the i-th positive and negative true samples, respectively. and Let be the i-th positive sample and the i-th negative sample, respectively; h be the number of categories; Cos be the cosine similarity function; η be the proportionality coefficient; and n be the proportion of the current category.

[0038] A Mamba-based multimodal dialogue emotion recognition system includes:

[0039] Preprocessing module: Used to process dialogue data of different modalities and embed speaker information and location information;

[0040] The multi-modal fusion module includes an external attention module, a Mamba module, and a Kalman filter. The external attention module is used to capture speech information within different modalities, the Mamba module is used to perform forward and reverse multi-modal fusion of semantic information from different modalities, and the Kalman filter is used to filter noise from different modalities.

[0041] Single-modal abstract feature representation module: used to achieve single-modal abstract feature representation through group pooling;

[0042] Multi-layer attention module: used to fuse abstract features of different modalities through cross-attention, and fuse them with the output of the Kalman filter to obtain the fused result;

[0043] Fully connected module: used to map the fusion results to predefined sentiment categories.

[0044] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a Mamba-based multimodal dialogue emotion recognition method.

[0045] An electronic device includes a processor and a memory, wherein the memory is used to store computer programs;

[0046] When the processor executes the program stored in memory, it implements the steps of a Mamba-based multimodal dialogue emotion recognition method.

[0047] The advantages of this application over the prior art are as follows:

[0048] 1. This application proposes a modeling method that is not based on quadratic time complexity, which can efficiently capture multimodal emotional semantic information in dialogue with linear time complexity and achieve full fusion between modalities.

[0049] 2. The multi-cross fusion mechanism designed in this application can capture global attention more efficiently to obtain more complete semantic information, and the group pooling and multi-layer global attention can supplement the missing global semantic information in the fusion process.

[0050] 3. This application designs a method to enhance intermodal data fusion, introducing a Kalman filter to unify different modalities. By fitting speech and image modalities to text modalities and calculating the optimal state using the covariance matrix, we are able to effectively integrate the optimal semantic information from the three modalities. Attached Figure Description

[0051] The following description, in conjunction with the accompanying drawings, further illustrates this application:

[0052] Figure 1 The network structure diagram of the model provided in the embodiments of this application. Detailed Implementation

[0053] like Figure 1 As shown, this application provides a multimodal emotion dialogue recognition method based on Mamba. First, external attention is used to capture the semantic information within each modality. Then, a multi-cross fusion mechanism is used to obtain the correlation information between different modalities. Subsequently, Kalman filtering is used to filter noise in each modality and establish a closer connection between modalities, so that the information of each modality can complement and calibrate each other. Then, segmented pooling is used to pool the preprocessed data, and multiple sets of upgraded cross attention are used to establish global semantic information. Finally, a fully connected layer is used to obtain the final recognition result.

[0054] The specific implementation steps of this application are as follows:

[0055] Step 1: Use different preprocessing methods to obtain feature encodings for different modalities;

[0056] First, we can define the dataset C as consisting of M groups of dialogues C = {c1, c2, ..., c...} M The i-th group of dialogues is composed of} i There are N sentences c i ={u1,u2,...,u N}, sentence j j It consists of K speakers P = {p1, p2, ..., p...} K The k-th speaker p in} k What is spoken, that is, each group of dialogues c i It consists of several {u j ,p k Composed of}

[0057] Secondly, after independently preprocessing these three modalities, one-dimensional convolution is used to achieve dimension alignment, ensuring that the features of different modalities can be effectively fused and interacted in subsequent processing.

[0058]

[0059] In the formula: It is a text representation obtained through RoBERTa. The speech representation was obtained through openSMILE. Video representations obtained through DenseNet, C {t} It is a text dataset, C {a} It is a speech dataset, C {v} It is a visual dataset.

[0060] Step 2: Convolution and information embedding for different feature codes: Speaker information and location information are embedded into the model to ensure that the model can distinguish between different speakers and their sequential positional relationships. This process can be represented by the following formula:

[0061]

[0062] In the formula: Conv1D is a one-dimensional linear convolution, σ speaker It is an embedded representation of the speaker's information, σ position It is an embedded representation of location information. It is a text embedding representation obtained by embedding speaker information and location information after one-dimensional linear convolution. It is a speech embedding representation obtained by embedding speaker information and location information after one-dimensional linear convolution. It is a video embedding representation obtained by embedding speaker information and location information after one-dimensional linear convolution.

[0063] Step 3: Use an external attention mechanism to capture the semantic information within each modality. The specific implementation process is as follows:

[0064]

[0065] In the formula: For each modality, the similarity matrix W Q W is a learnable query weight matrix. K W is a learnable demand weight matrix. V The input weight matrix is ​​learnable, and the Linear layer represents a linear layer. SiLU, Softmax activation functions, and the normal mean function are used to modify the connections between different linear layers. For semantic information within a single modality, Reverse the last dimension to obtain

[0066] Step 4: Use a cross-fusion mechanism to achieve interaction between different modalities and obtain a more comprehensive modal representation, specifically including:

[0067] Step 4.1: Employ a cross-fusion mechanism to construct interactions between different modalities:

[0068] A state-space model is constructed based on the Mamba model. When processing input tensors, the matrix is... As a shared matrix between modalities, it is passed from the text modality to the speech modality and the video modality respectively. During the computation, the input values ​​are calculated both forwards and backwards, and the discretized state transition equations are as follows:

[0069]

[0070] in:

[0071]

[0072] In the formula: and The hyperparameters are for training, and their meanings are the hidden states of a single mode in time intervals τ and τ-1, respectively. Let be the input of the single-mode mode at time τ, where For the output of a single mode at time τ, ΔA and It is the scale for single-mode discretization within time τ, where Δ = H τ+1 -H τ A, and All are learnable parameters; the forward and inverse inputs share ΔA, and exp is an exponential function. and These are the values ​​of a single mode after discretization, calculated as follows: Similarly, given the result of a positive input, it can be calculated... This is the result of reverse input.

[0073] Step 4.2: Result of the positive input The result of reverse input The outputs are cross-fused respectively:

[0074]

[0075] In the formula: This is the text output by Mamba in the forward direction. For the visual output of the positive Mamba, calculate the cross attention and compare the result with... Connect. For obtaining positive text-visual attention; For the audio output from the positive Mamba input, calculate the cross-attention and then compare the result with... Connect. To obtain positive text-visual-audio attention; This is the text output by reverse Mamba. For the vision output from the reverse Mamba, calculate the cross-attention and compare the result with... Connect. For obtaining reverse text-visual attention; For the audio output from the reverse Mamba, calculate cross-attention and compare the result with... Connect. For the calculated inverse text-visual-audio attention, O is... The combined text-visual-audio attention mechanism, obtained by adding these two methods, uses Softmax as the activation function and Concat as the connection function. Scaling factor for transpose, for The transpose of .

[0076] Step 5: Use Kalman filtering to filter noise in each mode. Utilize the consistency of noise distribution characteristics across the three modes and share noise matrix parameters to reduce noise and establish closer connections within the modes. The specific implementation steps are as follows:

[0077] Step 5.1: Decompose O into three modes O m The prior estimates of the predicted state vector and predicted observation vector of the state transition equation are calculated using the following formulas:

[0078]

[0079] P(ω)~(0,Q)

[0080] P(υ)~(0,R)m∈{a,v,t}

[0081] In the formula: It is the single-modal input at time k. It is the hidden state of the k-time model given that the single-modality model is at time k-1. It is the posterior estimate of the single-mode at the (k-1)th time. It is the observation value of the k-time model given that the single mode is at time k-1, ω k-1 and υ kω and υ are the estimation errors of the predicted state vector and predicted observation vector at time k, respectively. The error variables ω and υ follow normal distributions Q and R, respectively. and All of these are learnable hyperparameters, and are defined as follows:

[0082]

[0083] in (·) T W is the matrix transpose. F W G and W E All of these are learnable hyperparameters derived from splicing multimodal data.

[0084] Step 5.2: Calculate the predicted state covariance and the predicted observation covariance, as follows:

[0085]

[0086] In the formula: Given the (k-1)th time, this is the prior estimate of the k-time model. Z0 is the covariance estimated a priori, and Z0 is the identity matrix I. W is the posterior estimate of the model at time k-1. F Z is the state transition matrix. k-1 The posterior estimated covariance of the model at time k-1:

[0087] Step 5.3: Calculate the Kalman filter gain matrix, as follows:

[0088]

[0089] Where: K k It is the Kalman filter gain matrix.

[0090] Step 5.4: Update the posterior estimate and posterior error covariance, as follows:

[0091]

[0092] In the formula: I is the identity matrix.

[0093] Step 6: Use group pooling and multi-layer attention to build semantic information. The specific implementation steps are as follows:

[0094] Step 6.1: Use group pooling to achieve single-modal abstract feature representation, as detailed below;

[0095]

[0096] Where: G pooling For group pooling functions, This is the result after grouping and pooling.

[0097] Step 6.2: Use cross-attention to fuse different modalities, as follows:

[0098]

[0099] In the formula: For the grouped pooling after dimensionality increase, CrossAttenion is used to combine D with the result of Kalman filtering. By fusion

[0100] Step 7: Construct a linear mapping layer to map the above fusion results to predefined emotion categories, as follows:

[0101]

[0102] Step 8: Construct the corresponding loss function. First, assume the number of data points in the current category is *a*, the total data volume is *S*, and the proportion of the current category is *[a]*. To bring the data volume of the current category to the same order of magnitude as other categories, you can multiply the current category by a factor. Right now To achieve this, the expression for the loss function L is as follows:

[0103]

[0104] In the formula: and These are the i-th positive and negative true samples, respectively. and Let be the i-th positive sample and the i-th negative sample, respectively; h be the number of categories; Cos be the cosine similarity function; η be the proportionality coefficient; and n be the proportion of the current category.

[0105] The Mamba-based multimodal emotion dialogue recognition model provided in this application has the following improvements:

[0106] (1) Modeling methods that are not based on quadratic time complexity, but rather methods that can efficiently capture multimodal emotional semantic information in dialogue with linear time complexity, such as using Mamba as a model for extracting semantic information.

[0107] (2) At the global level, global attention can be captured more efficiently by using cross-fusion mechanism and Kalman filter fusion strategy to obtain more complete semantic information;

[0108] (3) At different modal levels, data fusion between modalities is enhanced by introducing a Kalman filter to unify the Mamba output of different modalities. By fitting the speech and image modalities to the text modalities and using the covariance matrix to calculate the optimal state, the optimal semantic information in the three modalities can be effectively integrated.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for Mamba-based multi-modal dialogue emotion recognition, characterized in that: The method comprises the following steps: Step 1: different pre-processing methods are used to process the dialogue data under different modalities to obtain corresponding feature encodings; Step 2: different feature encodings are subjected to convolution and information embedding; Step 3: external attention is used to capture the semantic information within each modality; Step 4: a cross-fusion mechanism is used to realize the interaction between different modalities to obtain more comprehensive modal representations; Step 4 specifically comprises: Step 4.1: a state space model is constructed using a Mamba model to process the input tensor, and a shared matrix between modalities is constructed, with the text modality data being transmitted to the speech modality data and the video modality data respectively, and the input values being calculated forward and reverse respectively; Step 4.2: the forward and reverse outputs are subjected to multi-cross fusion, with the text and vision outputs of the forward Mamba being taken as the query and key value respectively, the cross attention is calculated, the result is connected with the original text as the forward text-vision attention; then the forward text-vision attention and the audio output of the forward Mamba are taken as the query and key value, the cross attention is calculated, the result is connected with the forward text-vision attention as the forward text-vision-audio attention; similarly, the reverse text-vision-audio attention is calculated, and the forward text-vision-audio attention and the reverse text-vision-audio attention are connected to obtain the fused text-vision-audio attention; Step 5: Kalman filtering is used to filter the noise in each modality and establish the relationship between the modalities; Step 6: group pooling and multi-layer attention are used to fuse the semantic information of different modalities and then fuse the results of Kalman filtering to obtain the fusion results; Step 7: a linear mapping layer is constructed to map the fusion results to the predefined emotion categories, thus obtaining a multi-modal dialogue emotion recognition model based on Mamba; Step 8: a corresponding loss function is constructed to realize model training.

2. The method of claim 1, wherein the Mamba-based multimodal dialogue emotion recognition method is characterized by: The dialogue data of different modalities in step 1 includes audio modality data, text modality data and visual modality data, and the pre-processing methods of the three are respectively: Step 1.1: the audio modality data is processed using an openSMILE model to obtain audio features; Step 1.2: the text modality data is processed using a RoBERTa model to obtain text features; Step 1.3: the visual modality data is processed using a DenseNet model to obtain visual features.

3. The method of claim 1, wherein the Mamba-based multimodal dialogue emotion recognition method is characterized by: Step 2 specifically comprises: Step 2.1: one-dimensional convolution is performed on the feature encodings of each modality to unify the hidden layer feature dimensions and obtain consistent data structures; Step 2.2: speaker information and location information are embedded into the model, so that the model can distinguish different speakers and the order of the positions.

4. The method of claim 1, wherein the Mamba-based multimodal dialogue emotion recognition method is characterized by: Step 5 specifically comprises: Step 5.1: the prior estimates of the predicted state vectors and the predicted observation vectors of the state transition equation of the single modality features are calculated respectively; Step 5.2: the predicted state covariance and the predicted observation covariance of the single modality features are calculated respectively; Step 5.3: the Kalman filtering gain matrix is calculated; Step 5.4: Calculate the posterior estimation of the single modal feature respectively and update the posterior error covariance.

5. The method of claim 1, wherein: Step 6 specifically comprises: Step 6.1: Single modal abstract feature representation is achieved by using group pooling; Step 6.2: Different modal abstract features are fused by using cross attention, and after global attention, the result is fused with the result of Kalman filter.

6. The Mamba-based multi-modal dialogue emotion recognition method of claim 1, wherein: The expression of the loss function L in step 8 is as follows: ； In the formula: and are the i-th positive and negative real samples, respectively, and are the i-th positive and negative samples, respectively, h is the number of categories, and Cos is a cosine similarity function, is a proportionality coefficient, and n is the proportion of the current category.

7. A Mamba based multi-modal dialogue emotion recognition system characterized in that: Comprise: A preprocessing module for processing dialog data of different modalities and embedding speaker information and location information; A multi-cross fusion module including an external attention module, a Mamba module and a Kalman filter, the external attention module is used to capture the voice information inside different modalities respectively; The Mamba module is used for forward and reverse multi-cross fusion of semantic information of different modalities, specifically, a state space model is constructed using the Mamba model to process the input tensor, and a shared matrix between modalities is constructed, the input value is calculated respectively by forward and reverse; The forward and reverse outputs are fused by multi-cross respectively, the text and vision output by the forward Mamba are taken as the query and key value respectively, the cross attention is calculated, the result is connected with the original text as the forward text-vision attention; Then the forward text-vision attention and the audio output by the forward Mamba are taken as the query and key value respectively, the cross attention is calculated, the result is connected with the forward text-vision attention as the forward text-vision-audio attention; Similarly, the reverse text-vision-audio attention is calculated, and the forward text-vision-audio attention and the reverse text-vision-audio attention are connected to obtain the fused text-vision-audio attention; The Kalman filter is used to filter the noise in different modalities; A single modal abstract feature representation module is used to achieve single modal abstract feature representation by group pooling; A multi-layer attention module is used to fuse different modal abstract features by cross attention, and fuse the output of the Kalman filter to obtain a fusion result; 8. A computer-readable storage medium, characterized in that: A fully connected module is used to map the fusion result to a predefined emotion category.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the Mamba-based multi-modal dialog emotion recognition method according to any one of claims 1-6. Comprise a processor and a memory, wherein the memory is used to store a computer program; The processor is used to execute the program stored on the memory to realize the steps of the Mamba-based multi-modal dialog emotion recognition method according to any one of claims 1-6.

Images