Multi-modal dialogue emotion recognition method and system based on Mama

Abstract

The invention provides a Mama-based multi-modal dialogue emotion recognition method and system, and belongs to the field of dialogue emotion recognition. The problems that an existing multi-mode emotion recognition method is limited in memory ability, information is not fully utilized, and noise accumulation exists in long sequence fusion are solved. Comprising the following steps: processing dialogue data in different modes by adopting different preprocessing modes to obtain corresponding feature codes; performing convolution and information embedding on different feature codes; external attention is adopted to capture semantic information in each modal; interaction among different modes is realized by using a cross fusion mechanism; filtering noise in each mode by adopting Kalman filtering and establishing a relation between the modes; fusing the semantic information of different modals by using grouping pooling and cross attention, and then fusing the fused semantic information with a Kalman filtering result to obtain a fusion result; mapping the fusion result into a predefined emotion category; the method and the device are applied to multi-mode dialogue emotion recognition.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal dialogue emotion in natural language processing, and in particular to a multimodal dialogue emotion recognition method and system based on Mamba. Background Art

[0002] The importance of Emotion Recognition in Conversation (ERC) in the fields of artificial intelligence and natural language processing is becoming increasingly prominent and has received widespread attention. For example, emotion computing technology can enhance the emotional communication between teachers and students in education and training, and improve the interactivity of contextualized teaching; in the field of life and health, it can assist in disease diagnosis and doctor-patient communication; in commercial services, it can improve customer service empathy and the accuracy of financial risk identification; in industrial design, it can improve the performance of humanoid robots; and in social governance, it can be used for security monitoring and assessing public credibility.

[0003] Multimodal conversation emotion recognition refers to the comprehensive collection of video, audio and text materials of the speaker in a given conversation, and the emotion classification of each utterance. The input of this task is a continuous conversation, and the output is the emotion of all utterances in this conversation. Unlike emotion recognition of a single sentence, emotion recognition in a conversation needs to comprehensively consider the background, context, speaker and other information in the conversation, so it is crucial to integrate the information of the three modalities of text, audio and vision.

[0004] Recently, deep learning has been widely used in multimodal sentiment analysis of conversations. The main methods adopted by researchers are: (1) using recurrent neural networks for recognition, such as long short-term memory networks or gated recurrent unit neural networks, which can process sequential data and combine historical data to generate long-term memory. They can also process inputs of different lengths, solve the gradient vanishing and gradient exploding problems when processing long sequences, and introduce gating mechanisms to adjust the degree of information transmission in the sequence, so as to better capture long-distance dependencies; (2) using graph convolutional neural networks to record a series of information fragments in nodes, and use the relationship edges between nodes to capture complex relationships and dependencies. This method constructs the data into a graph structure, capturing the connections between each node that traditional models cannot obtain. However, it is difficult to model the relationship between modalities, and it is easy to introduce redundant information, thereby reducing the effect of multimodal feature fusion in emotion recognition; (3) Using external knowledge bases or large language models to improve the richness of information, enhance context perception, and improve the generalization ability of the model is crucial to building an efficient artificial intelligence system. The intervention of external knowledge can effectively reduce the ambiguity and uncertainty that a single model may bring, enabling the model to obtain information from a wider range of knowledge sources, thereby enhancing its robustness and stability; (4) In addition, the Transformer model can more accurately capture long-distance dependencies through its self-attention mechanism, and performs well in processing long sequence data and tasks involving long-distance semantic relationships. It aggregates the data of the entire sequence and deeply learns the intrinsic connections between the data, further improving the performance and accuracy of the model.

[0005] Although many works have successfully extracted emotional semantic information, they still have some limitations. Compared with the method of using Mamba to capture semantic information, the memory capacity of long sequences in recurrent neural networks is limited, while Transformer and large language models require a lot of computing resources, especially large language models, which not only consume huge resources, but also have the risk of privacy leakage, and poor performance in debugging and interpretability. However, the simple use of state transfer equations ignores the connection between modes, and there will be problems with noise accumulation when modeling long sequences. Summary of the invention

[0006] In order to solve the problems existing in current multimodal emotion recognition methods, such as limited memory capacity, underutilized information, and noise accumulation in long sequence fusion, this application proposes a multimodal dialogue emotion recognition method and system based on Mamba, which adopts a dual fusion mechanism of cross fusion mechanism and Kalman filter fusion; it also uses Kalman filtering technology to unify the differences between different modalities, and captures global attention through segmented pooling technology to achieve accurate recognition and analysis of emotions.

[0007] The technical solution adopted in this application is:

[0008] The beneficial effect of the present application compared with the prior art is: a multimodal conversation emotion recognition method based on Mamba, comprising the following steps:

[0009] Step 1: Use different preprocessing methods to process the conversation data under different modes to obtain the corresponding feature coding;

[0010] Step 2: Convolution and information embedding of different feature codes;

[0011] Step 3: Use external attention to capture the semantic information inside each modality;

[0012] Step 4: Use the cross-fusion mechanism to achieve interaction between different modalities and obtain a more comprehensive modality representation;

[0013] Step 5: Use Kalman filtering to filter the noise in each mode and establish the connection between modes;

[0014] Step 6: Use group pooling and multi-layer attention to fuse the semantic information of different modalities and then fuse it with the result of Kalman filtering to obtain the fusion result;

[0015] Step 7: Build a linear mapping layer to map the fusion results to predefined emotion categories. So far, a multimodal conversation emotion recognition model based on Mamba is obtained.

[0016] Step 8: Construct the corresponding loss function to implement model training.

[0017] The conversation data of different modes in step 1 include audio mode data, text mode data and visual mode data, and the preprocessing methods of the three are as follows:

[0018] Step 1.1: Use the openSMILE model to process the audio modal data to obtain audio features;

[0019] Step 1.2: Use the RoBERTa model to process the text modality data to obtain text features;

[0020] Step 1.3: Use the DenseNet model to process the visual modality data to obtain visual features.

[0021] Furthermore, step 2 specifically includes:

[0022] Step 2.1: Perform one-dimensional convolution on the feature encoding of each modality, unify the hidden layer feature dimension, and obtain a consistent data structure;

[0023] Step 2.2: Embed the speaker information and position information into the model so that the model can distinguish different speakers and their positional relationships.

[0024] Furthermore, step 4 specifically includes:

[0025] Step 4.1: Use the Mamba model to build a state space model to process the input tensor and construct a shared matrix between modalities. The text modality data is passed to the voice modality data and the video modality data respectively, and the input values ​​are calculated forward and reversely respectively.

[0026] Step 4.2: Perform multiple cross-fusion on the forward and reverse outputs respectively, take the text and vision of the forward Mamba output as the query and key value respectively, calculate the cross attention, and connect the result with the original text as the forward text-visual attention; then take the forward text-visual attention and the audio of the forward Mamba output as the query and key value, calculate the cross attention, and connect the result with the forward text-visual attention as the forward text-visual-audio attention; similarly calculate the reverse text-visual-audio attention, and connect the forward text-visual-audio attention and the reverse text-visual-audio attention to obtain the fused text-visual-audio attention.

[0027] Furthermore, step 5 specifically includes:

[0028] Step 5.1: Calculate the prior estimates of the predicted state vector and the predicted observation vector of the state transfer equation of the single modal feature respectively;

[0029] Step 5.2: Calculate the predicted state covariance and predicted observation covariance of the unimodal features respectively;

[0030] Step 5.3: Calculate the Kalman filter gain matrix;

[0031] Step 5.4: Calculate the posterior estimates of the unimodal features separately and update the posterior error covariance.

[0032] Furthermore, step 6 specifically includes:

[0033] Step 6.1: Use group pooling to achieve single-modal abstract feature representation;

[0034] Step 6.2: Use cross attention to fuse the abstract features of different modalities and after global attention, fuse them with the results of Kalman filtering.

[0035] Furthermore, the expression of the loss function L in step 8 is as follows:

[0036]

[0037] Where: and are the i-th positive true sample and negative true sample respectively, and are the i-th positive sample and negative sample respectively, h is the number of categories, Cos is the cosine similarity function, η is the proportional coefficient, and n is the proportion of the current category.

[0038] A multimodal conversation emotion recognition system based on Mamba, including:

[0039] Preprocessing module: used to process conversation data of different modes and embed speaker information and location information;

[0040] Multi-cross fusion module: including external attention module, Mamba module and Kalman filter. The external attention module is used to capture the speech information in different modalities respectively. The Mamba module is used to perform forward and reverse multi-cross fusion of semantic information of different modalities. The Kalman filter is used to filter the noise in different modalities.

[0041] Single-modal abstract feature representation module: used to achieve single-modal abstract feature representation through grouping pooling;

[0042] Multi-layer attention module: used to fuse the abstract features of different modalities through cross attention and fuse them with the output of the Kalman filter to obtain the fusion result;

[0043] Fully connected module: used to map the fusion results into predefined emotion categories.

[0044] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a multimodal conversation emotion recognition method based on Mamba are implemented.

[0045] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program;

[0046] When the processor is used to execute the program stored in the memory, the steps of the multimodal dialogue emotion recognition method based on Mamba are implemented.

[0047] The beneficial effects of this application compared to the prior art are:

[0048] 1. This application proposes a modeling method that is not based on quadratic time complexity, which can efficiently capture multimodal emotional semantic information in conversations with linear time complexity and achieve full integration between modalities.

[0049] 2. The multi-cross fusion mechanism designed in this application can capture global attention more efficiently to obtain more complete semantic information, and the group pooling and multi-layer global attention can supplement the global semantic information missing in the fusion process.

[0050] 3. This application designs a method to enhance inter-modal data fusion and introduces a Kalman filter to unify different modalities. By fitting the speech and image modalities to the text modality and using the covariance matrix to calculate the optimal state, we can effectively integrate the optimal semantic information in the three modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present application is further described below with reference to the accompanying drawings:

[0052] Figure 1 A diagram of the model network structure provided for an embodiment of the present application. DETAILED DESCRIPTION

[0053] like Figure 1 As shown, the present application provides a multimodal emotional dialogue recognition method based on Mamba, which first uses external attention to capture the semantic information within each modality, and then uses a multi-cross fusion mechanism to obtain the correlation information between different modalities, and then uses Kalman filter fusion to filter the noise in each modality and establish a closer connection between the modalities, so that the information of each modality can complement and calibrate each other, and then uses segmented pooling to pool the preprocessed data, and uses multiple groups of cross-attention after dimensionality increase to establish global semantic information, and finally uses a fully connected layer to obtain the final recognition result.

[0054] The specific implementation steps of this application are as follows:

[0055] Step 1: Use different preprocessing methods for different modalities to obtain feature encoding;

[0056] First, we can define a dataset C consisting of M groups of conversations C = {c 1 ,c 2 ,...,c M}, the i-th group of dialogue c i There are N sentences c i = {u 1 ,u 2 ,...,u N}, the jth sentence u j There are K speakers P = {p 1 ,p 2 ,...,p K The kth speaker p in k Spoken, that is, each group of dialogue c i It is composed of several {u j ,pk} is composed of.

[0057] Secondly, after preprocessing the three modalities independently, one-dimensional convolution is further used to achieve dimensional alignment to ensure that the features of different modalities can be effectively integrated and interacted in subsequent processing, namely:

[0058]

[0059] Where: is the text representation obtained by RoBERTa, is the speech representation obtained through openSMILE, is the video representation obtained by DenseNet, C {t} is a dataset text dataset, C {a} is a speech dataset, C {v} It is a visual dataset.

[0060] Step 2: Convolve and embed different feature codes: embed speaker information and position information into the model to ensure that the model can distinguish different speakers and their positional relationships. The process can be expressed as follows:

[0061]

[0062] Where: Conv1D is a one-dimensional linear convolution, σ speaker is the embedded representation of the speaker information, σ position is the embedded representation of the position information, It is a text embedding representation obtained by embedding speaker information and position information through one-dimensional linear convolution. It is a speech embedding representation obtained by embedding speaker information and position information through one-dimensional linear convolution. It is a video embedding representation obtained by embedding speaker information and location information through one-dimensional linear convolution.

[0063] Step 3: Use the external attention mechanism to capture the semantic information within each modality. The specific implementation process is as follows:

[0064]

[0065] Where: is the similarity matrix of each modality, W Q is the learnable query weight matrix, W K is the learnable demand weight matrix, W V is a learnable input weight matrix, Linear is a linear layer, SiLU, Softmax activation function, and norm mean function are used to change the connection between different linear layers. is the semantic information within the single modality, Reversing the last dimension of

[0066] Step 4: Use the cross-fusion mechanism to achieve interaction between different modalities and obtain a more comprehensive modal representation, including:

[0067] Step 4.1: Use cross-fusion mechanism to build interaction between different modalities:

[0068] The state space model is constructed based on the Mamba model. When processing the input tensor, the matrix As a shared matrix between modalities, it is passed from the text modality to the voice modality and the video modality respectively. During the calculation process, the input values ​​are calculated forward and backward respectively, and the discretized state transfer equation is as follows:

[0069]

[0070] in:

[0071]

[0072] Where: and are the hyperparameters available for training, which respectively mean the hidden state of a single modality within τ time and τ-1 time, is the τ-time unimodal input, where is the output of the single mode at time τ, ΔA and is the scale of the unimodal discretization in time τ, where Δ = H τ+1 -H τ , A. and are all learnable parameters. The forward and reverse inputs share ΔA. exp is an exponential function. and They are the discretized values ​​of the single mode, and the calculation results are As a result of the positive input, we can also calculate is the result of reverse input.

[0073] Step 4.2: The result of forward input And the result of reverse input The outputs are cross-fused separately:

[0074]

[0075] Where: is the text that is output to Mamba. For the visual output of Mamba, calculate the cross attention and compare the result with Make a connection, is the positive text-visual attention obtained; For the audio output of Mamba, calculate the cross attention and compare the result with Make a connection, is the obtained positive text-visual-audio attention; The text output by reverse Mamba, For the reverse Mamba output vision, calculate the cross attention and compare the result with Make a connection, To obtain the reverse text-visual attention; For the reverse Mamba output audio, calculate the cross attention and compare the result with Make a connection, is the calculated inverse text-visual-audio attention, O is The fused text-visual-audio attention is obtained by adding the two together, Softmax is the activation function, and Concat is the connection function. is the scaling factor, for The transpose of for The transpose of .

[0076] Step 5: Use Kalman filtering to filter the noise in each mode, use the consistency of the distribution characteristics of the noise in the three modes, share the noise matrix parameters, thereby reducing the noise and establishing a closer connection within the mode. The specific implementation steps are as follows:

[0077] Step 5.1: Decompose O into three modal O m , calculate the prior estimate of the predicted state vector and predicted observation vector of the state transfer equation, that is, its formula is as follows:

[0078]

[0079] P(ω)~(0,Q)

[0080] P(υ)~(0,R)m∈{a,v,t}

[0081] Where: is the unimodal input at time k, is the hidden state of the k-time model under the premise of unimodality at time k-1, is the posterior estimate of the single mode at k-1th time, is the observation value of the k-time model under the premise of the unimodal at time k-1, ω k-1 andk are the estimated errors of the predicted state vector and the predicted observation vector at time k, respectively. The error variables ω and υ obey the normal distribution Q and R, respectively. and are all learnable hyperparameters and are defined as follows:

[0082]

[0083] in (·) T is the matrix transpose, W F , W G and W E These are all learnable hyperparameters after concatenating multiple modalities.

[0084] Step 5.2: Calculate the predicted state covariance and predicted observation covariance as follows:

[0085]

[0086] Where: is the prior estimate of the k-time model under the premise of the k-1th time, is the a priori estimated covariance, Z 0 is the identity matrix I, is the posterior estimate of the model at the k-1th moment, W F is the state transfer matrix, Z k-1 is the posterior estimated covariance of the model at time k-1:

[0087] Step 5.3: Calculate the Kalman filter gain matrix as follows:

[0088]

[0089] Where: K k is the Kalman filter gain matrix.

[0090] Step 5.4: Update the posterior estimate and posterior error covariance as follows:

[0091]

[0092] Where: I is the unit matrix.

[0093] Step 6: Use group pooling and multi-layer attention to build semantic information. The specific implementation steps are as follows:

[0094] Step 6.1: Use group pooling to achieve single-modal abstract feature representation, as follows;

[0095]

[0096] Where: Gpooling is the group pooling function, is the result after group pooling.

[0097] Step 6.2: Use cross attention to fuse different modalities as follows:

[0098]

[0099] Where: is the group pooling after dimensionality increase, CrossAttenion is the cross attention, and D is combined with the result of Kalman filtering. Fusion

[0100] Step 7: Construct a linear mapping layer to map the above fusion results to predefined emotion categories as follows:

[0101]

[0102] Step 8: Construct the corresponding loss function. First, assume that the number of current category data is a, the total data volume is S, and the current category accounts for In order to adjust the data volume of the current category to the same order of magnitude as that of other categories, we can multiply the current category by Right now To achieve this, the expression of the loss function L is as follows:

[0103]

[0104] Where: and are the i-th positive true sample and negative true sample respectively, and are the i-th positive sample and negative sample respectively, h is the number of categories, Cos is the cosine similarity function, η is the proportional coefficient, and n is the proportion of the current category.

[0105] The multimodal emotional dialogue recognition model based on Mamba provided in this application has the following improvements:

[0106] (1) Modeling methods that are not based on quadratic time complexity, but can efficiently capture multimodal emotional semantic information in conversations with linear time complexity, such as using Mamba as a model for extracting semantic information;

[0107] (2) At the global level, it can capture global attention more efficiently, using the cross-fusion mechanism and Kalman filter fusion strategy to obtain more complete semantic information;

[0108] (3) At the level of different modalities, we strengthen inter-modal data fusion and introduce a Kalman filter to unify the Mamba outputs of different modalities. By fitting the speech and image modalities to the text modality and using the covariance matrix to calculate the optimal state, we can effectively integrate the optimal semantic information of the three modalities.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal conversation emotion recognition method based on Mamba, characterized by: The following steps are involved: Step 1: Use different preprocessing methods to process the conversation data under different modes to obtain the corresponding feature coding; Step 2: Convolution and information embedding of different feature codes; Step 3: Use external attention to capture the semantic information inside each modality; Step 4: Use the cross-fusion mechanism to achieve interaction between different modalities and obtain a more comprehensive modality representation; Step 5: Use Kalman filtering to filter the noise in each mode and establish the connection between modes; Step 6: Use group pooling and multi-layer attention to fuse the semantic information of different modalities and then fuse it with the result of Kalman filtering to obtain the fusion result; Step 7: Build a linear mapping layer to map the fusion results to predefined emotion categories. So far, a multimodal conversation emotion recognition model based on Mamba is obtained. Step 8: Construct the corresponding loss function to implement model training.

2. The method for multimodal conversation emotion recognition based on Mamba according to claim 1, characterized in that: The conversation data of different modes in step 1 include audio mode data, text mode data and visual mode data, and the preprocessing methods of the three are as follows: Step 1.1: Use the openSMILE model to process the audio modal data to obtain audio features; Step 1.2: Use the RoBERTa model to process the text modality data to obtain text features; Step 1.3: Use the DenseNet model to process the visual modality data to obtain visual features.

3. The multimodal conversation emotion recognition method based on Mamba according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Perform one-dimensional convolution on the feature encoding of each modality, unify the hidden layer feature dimension, and obtain a consistent data structure; Step 2.2: Embed the speaker information and position information into the model so that the model can distinguish different speakers and their positional relationships.

4. The method for multimodal conversation emotion recognition based on Mamba according to claim 1, characterized in that: Step 4 specifically includes: Step 4.1: Use the Mamba model to build a state space model to process the input tensor and construct a shared matrix between modalities. The text modality data is passed to the voice modality data and the video modality data respectively, and the input values ​​are calculated forward and reversely respectively. Step 4.2: Perform multiple cross-fusion on the forward and reverse outputs respectively, take the text and vision of the forward Mamba output as the query and key value respectively, calculate the cross attention, and connect the result with the original text as the forward text-visual attention; then take the forward text-visual attention and the audio of the forward Mamba output as the query and key value, calculate the cross attention, and connect the result with the forward text-visual attention as the forward text-visual-audio attention; similarly calculate the reverse text-visual-audio attention, and connect the forward text-visual-audio attention and the reverse text-visual-audio attention to obtain the fused text-visual-audio attention.

5. The method for multimodal conversation emotion recognition based on Mamba according to claim 1, characterized in that: Step 5 specifically includes: Step 5.1: Calculate the prior estimates of the predicted state vector and the predicted observation vector of the state transfer equation of the unimodal feature respectively; Step 5.2: Calculate the predicted state covariance and the predicted observation covariance of the unimodal feature respectively; Step 5.3: Calculate the Kalman filter gain matrix; Step 5.4: Calculate the posterior estimates of the unimodal features separately and update the posterior error covariance.

6. The method for multimodal conversation emotion recognition based on Mamba according to claim 1, characterized in that: Step 6 specifically includes: Step 6.1: Use group pooling to achieve single-modal abstract feature representation; Step 6.2: Use cross attention to fuse the abstract features of different modalities and after global attention, fuse them with the results of Kalman filtering.

7. The method for multimodal conversation emotion recognition based on Mamba according to claim 1, characterized in that: The expression of the loss function L in step 8 is as follows: Where: and are the i-th positive true sample and negative true sample respectively, and are the i-th positive sample and negative sample respectively, h is the number of categories, Cos is the cosine similarity function, η is the proportional coefficient, and n is the proportion of the current category.

8. A multimodal conversation emotion recognition system based on Mamba, characterized by: include: Preprocessing module: used to process conversation data of different modes and embed speaker information and location information; Multi-cross fusion module: including external attention module, Mamba module and Kalman filter. The external attention module is used to capture the speech information in different modalities respectively. The Mamba module is used to perform forward and reverse multi-cross fusion of semantic information of different modalities. The Kalman filter is used to filter the noise in different modalities. Single-modal abstract feature representation module: used to achieve single-modal abstract feature representation through grouping pooling; Multi-layer attention module: used to fuse the abstract features of different modalities through cross attention and fuse them with the output of the Kalman filter to obtain the fusion result; Fully connected module: used to map the fusion results into predefined emotion categories.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the multimodal conversation emotion recognition method based on Mamba are implemented as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory is used to store a computer program; When the processor is used to execute the program stored in the memory, the steps of the multimodal conversation emotion recognition method based on Mamba as described in any one of claims 1-7 are implemented.

Images