Dialogue emotion recognition method and device based on multi-modal Mama network and medium

Abstract

The invention discloses a dialogue emotion recognition method and device based on a multi-mode Mama network and a medium, and can be widely applied to the technical field of emotion recognition. The dialogue information of a plurality of pieces of utterance data including multiple modes is input into the multi-mode Mama network, the features of the multi-mode utterance data are extracted from the dialogue information through the encoder in the multi-mode Mama network to serve as the first features, all the first features are aligned through the data alignment module to obtain the second features, and the second features are aligned through the data alignment module to obtain the multi-mode utterance data. According to the emotion recognition and classification method and device, the multi-modal data are extracted according to the first features to realize the consistency of the multi-modal data, then the context information in the dialogue information is extracted through the multi-modal fusion module according to the second features, and the emotion category label of each piece of speech data is determined through the emotion classifier according to the context information, so that the accuracy of emotion recognition and classification can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to a conversation emotion recognition method, device and medium based on a multimodal Mamba network. Background Art

[0002] In the related technology, the purpose of Emotion Recognition in Conversations (ERC) is to accurately identify the emotions expressed by different utterances in a conversation. Emotion plays a vital role in interpersonal communication. Therefore, developing a reliable ERC system is of great significance in applications such as social media analysis, customer service, and mental health monitoring. Existing ERC methods mainly rely on single-modal cues for emotion recognition, which cannot capture the complex changes of emotions in conversations. Although existing multimodal ERC methods can integrate information from multiple modalities, their emotion recognition classification accuracy is not high.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a conversation emotion recognition method, device and medium based on a multimodal Mamba network, which can effectively improve the accuracy of emotion recognition classification.

[0005] To achieve the above object, an embodiment of the present application proposes a method for conversation emotion recognition based on a multimodal Mamba network, the method comprising the following steps:

[0006] Acquiring dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes;

[0007] Inputting the dialogue information into a multimodal Mamba network to obtain a sentiment category label for each of the speech data;

[0008] Wherein, the multimodal Mamba network includes an encoder, a data alignment module, a multimodal fusion module and a sentiment classifier;

[0009] The encoder is used to extract a feature of the multimodal speech data from the dialogue information as a first feature;

[0010] The data alignment module is used to align all the first features to obtain a second feature;

[0011] The multimodal fusion module is used to extract context information in the dialogue information according to the second feature;

[0012] The sentiment classifier is used to determine the sentiment category label of each of the utterance data according to the context information.

[0013] In some embodiments, the encoder includes a text encoder, an audio encoder, and a video encoder;

[0014] The text encoder is used to extract a first emotion feature of the text data in the dialogue information;

[0015] The audio encoder is used to extract a second emotion feature of the audio data in the dialogue information;

[0016] The video encoder is used to extract the third emotion feature of the video data in the dialogue information.

[0017] In some embodiments, the aligning of all the first features comprises:

[0018] Mapping all of the first emotion features, the second emotion features, and the third emotion features to a normalized low-dimensional network to obtain a corresponding first similarity function, a second similarity function, and a third similarity function;

[0019] Calculating a similarity score of the momentum encoder according to the first similarity function, the second similarity function and the third similarity function;

[0020] All the first features are aligned according to the similarity scores.

[0021] In some embodiments, the aligning all the first features according to the similarity scores includes:

[0022] Calculating a preset loss value based on all the similarity scores, wherein the preset loss value is used to maximize the similarity scores of the matching pairs and minimize the similarity scores of the non-matching pairs;

[0023] All the first features are aligned according to the preset loss value.

[0024] In some embodiments, the calculation formula for calculating the preset loss value based on all the similarity scores is as follows:

[0025]

[0026] In the formula, represents the preset loss value; the numerator represents the exponent of the text data, the audio data or the video data; the denominator represents the exponent sum of all possible similarity scores corresponding to the text data, the audio data or the video data.

[0027] In some embodiments, extracting context information from the conversation information according to the second feature includes:

[0028] Input the second feature into the cross attention layer to obtain a fused feature;

[0029] Input the fused features into a fully connected layer to obtain standardized features;

[0030] All the standardized features are input into the Mamba network to obtain an output sequence, where the output sequence is used to describe the context information of the conversation information.

[0031] In some embodiments, determining the emotion category label of each of the speech data according to the context information includes:

[0032] Predicting a label probability of each of the utterance data according to the context information;

[0033] The label corresponding to the maximum label probability is used as the emotion category label corresponding to the speech data.

[0034] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a conversation emotion recognition device based on a multimodal Mamba network, the device comprising:

[0035] The first module is used to obtain dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes;

[0036] The second module is used to input the dialogue information into the multimodal Mamba network to obtain the emotion category label of each speech data;

[0037] Wherein, the multimodal Mamba network includes an encoder, a data alignment module, a multimodal fusion module and a sentiment classifier;

[0038] The encoder is used to extract a feature of the multimodal speech data from the dialogue information as a first feature;

[0039] The data alignment module is used to align all the first features to obtain a second feature;

[0040] The multimodal fusion module is used to extract context information in the dialogue information according to the second feature;

[0041] The sentiment classifier is used to determine the sentiment category label of each of the utterance data according to the context information.

[0042] To achieve the above objective, another aspect of the present application provides a computer device, including:

[0043] at least one processor;

[0044] at least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a method, device and medium for conversation emotion recognition based on a multimodal Mamba network. The scheme obtains conversation information including a plurality of speech data of multiple modalities and inputs it into the multimodal Mamba network, so as to extract the features of the multimodal speech data from the conversation information through an encoder in the multimodal Mamba network as the first feature, aligns all the first features through a data alignment module to obtain a second feature, so as to achieve consistency of the multimodal data, and then extracts context information from the conversation information according to the second feature through a multimodal fusion module, and then determines the emotion category label of each speech data according to the context information through an emotion classifier, thereby effectively improving the accuracy of emotion recognition classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a conversation emotion recognition method based on a multimodal Mamba network provided in an embodiment of the present application;

[0049] Figure 2 It is a schematic diagram of the architecture of a multimodal Mamba network provided in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of the architecture of a data alignment module provided in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the architecture of the Mamba network in the multimodal fusion module provided in the embodiment of the present application;

[0052] Figure 5 It is a structural diagram of a conversation emotion recognition device based on a multimodal Mamba network provided in an embodiment of the present application;

[0053] Figure 6 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0057] MEMO-Mamba: A novel conversational emotion recognition framework based on multimodal Mamba networks.

[0058] MEC-Loss (Multimodal Emotion Contrastive Loss): Multimodal emotion contrast loss, used to align single modal features to ensure the consistency of multimodal information.

[0059] RoBERTa: A Transformer-based language model for sequence modeling tasks.

[0060] WavLM: An audio codec that excels in noisy and overlapping speech conditions.

[0061] VisExtNet: A video encoder that efficiently captures facial expressions without encoding redundant visual contextual information.

[0062] ALBEF: A framework that proposes a method to align and fuse multimodal information.

[0063] MoCo (Momentum Contrast for Unsupervised Visual Representation Learning): Momentum Contrast Method for Unsupervised Visual Representation Learning.

[0064] Cross-Attention: Cross-attention mechanism that calculates attention by query, key, and value transformations.

[0065] WF1 (Weighted F1 Score): Weighted F1 score, used as an evaluation indicator of emotion recognition performance.

[0066] IEMOCAP: An emotion recognition dataset containing 12 hours of two-person conversation videos.

[0067] MELD: A multi-party dataset from the TV series Friends, containing conversation data of various emotion categories.

[0068] MultiEMO: An attention-based relevance-aware multimodal fusion framework for conversational emotion recognition.

[0069] Mamba Network: A network architecture for processing multimodal data that can dynamically integrate broader contextual information.

[0070] In related technologies, the goal of Emotion Recognition in Conversations (ERC) is to accurately identify the emotions expressed by different utterances in a conversation. Emotions play a vital role in interpersonal communication, so developing a reliable ERC system is of great significance in applications such as social media analysis, customer service, and mental health monitoring.

[0071] Existing ERC methods mainly rely on unimodal clues such as text, audio or visual data for sentiment classification. Although these methods have achieved good results to a certain extent, they often fail to fully capture the complex changes in sentiment in conversations. To address this limitation, multimodal ERC methods have begun to integrate information from multiple modalities. However, these multimodal ERC methods still face two major challenges in practical applications: one is the consistency problem between different modalities, and the other is the difficulty of capturing contextual information during multimodal feature fusion. For example, the ALBEF framework proposes a method to align and fuse multimodal information, and pre-trains the model through contrastive learning to obtain robust representations in downstream tasks. However, these methods still have difficulties in capturing complex interactions between modalities. In addition, the CoMPM method introduces a Transformer-based contextual embedding module and a pre-trained memory module. Although these methods improve context understanding to a certain extent, they still suffer from the problem of loss of sentiment clues during multimodal information fusion.

[0072] It can be seen that the existing methods have the following problems:

[0073] (1) Multimodal consistency problem: Existing methods have difficulty achieving cross-modal consistency due to the significant differences in representation and dynamics between different modalities. For example, the way text expresses emotions is very different from audio or visual cues. This difference makes it difficult to create a unified multimodal representation, which in turn affects the accuracy of sentiment classification.

[0074] (2) Difficulty in capturing contextual information: Contextual information is crucial for accurate sentiment classification, especially in conversations, where the order of discourse and the interaction between speakers are crucial for understanding the emotional tone. Existing methods often have difficulty effectively integrating this contextual information during the multimodal feature fusion process, resulting in the loss of important sentiment clues, which in turn affects the accuracy of classification.

[0075] In view of this, an embodiment of the present application provides a conversation emotion recognition method, device and medium based on a multimodal Mamba network. The present application obtains conversation information including a plurality of speech data in multiple modes and inputs it into the multimodal Mamba network, so as to extract the features of the multimodal speech data from the conversation information through an encoder in the multimodal Mamba network as the first feature, aligns all the first features through a data alignment module to obtain a second feature, so as to achieve consistency of the multimodal data, and then extracts context information in the conversation information according to the second feature through a multimodal fusion module, and then determines the emotion category label of each speech data according to the context information through an emotion classifier, thereby effectively improving the accuracy of emotion recognition classification.

[0076] The method for conversation emotion recognition based on a multimodal Mamba network provided in the embodiment of the present application relates to the field of emotion recognition technology. The method for conversation emotion recognition based on a multimodal Mamba network provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a method for conversation emotion recognition based on a multimodal Mamba network, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] Figure 1 is an optional flowchart of a conversation emotion recognition method based on a multimodal Mamba network provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S120:

[0079] Step S110: Acquire dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes;

[0080] Step S120: input the dialogue information into the multimodal Mamba network to obtain the emotion category label of each speech data.

[0081] In the embodiments of the present application, Figure 2 As shown in FIG. 1 , the multimodal Mamba network includes an encoder, a data alignment module (TAV-Alignment), a multimodal fusion module (Multimodal Fusion) and a sentiment classifier (Classifier).

[0082] In an embodiment of the present application, the encoder is used to extract features of multimodal speech data from the conversation information as the first feature. It can be understood that the conversation information includes data in three modes: text (T), audio (A) and video (V). Each conversation contains n speech data. For each speech data, this embodiment extracts features from text, audio and video as the first feature respectively. Since Transformer-based models (such as RoBERTa) perform well in sequence modeling and can be fine-tuned for ERC tasks, this embodiment will use the RoBERTa model as a text encoder to obtain word vectors with rich semantic information. Text content {T1, T2, …, T n} is converted into a 256-dimensional feature by RoBERTa as the first sentiment feature, denoted as {t1,t2,…,t n}.

[0083] Although existing methods have achieved impressive results in audio encoders, they often overlook robust performance under noisy and overlapping speech conditions. Therefore, this embodiment adopts WavLM as an audio encoder in the emotion recognition framework. WavLM excels in handling these conditions by learning a general speech representation from a large amount of unlabeled speech data and combining masked speech prediction and denoising pre-training. The audio content {A1, A2, …, A n} is extracted by WavLM as a 1024-dimensional feature as the second sentiment feature, denoted as {a1, a2, …, a n}.

[0084] To deal with the problem of redundant scene-related information, VisExtNet effectively captures facial expressions without encoding unnecessary visual context. Therefore, this embodiment uses VisExtNet as a video encoder in the emotion recognition framework. Video content {V1, V2, …, V n} is processed by VisExtNet to obtain a 1000-dimensional feature as the third sentiment feature, denoted as {v1,v2,…,v n}.

[0085] In the embodiment of the present application, after extracting the first feature from the text, audio and video, the embodiment uses a data alignment module to align all the first features to obtain the second feature. It can be understood that, if Figure 3 As shown, this embodiment maps all first emotion features, second emotion features and third emotion features to a normalized low-dimensional network to obtain the corresponding first similarity function, second similarity function and third similarity function, and then calculates the similarity score of the momentum encoder according to the first similarity function, the second similarity function and the third similarity function, and then aligns all first features according to the similarity score. Specifically, the processing process of this embodiment involves learning the similarity function between the [CLS] embeddings of text features, audio features and video features, and defines the following formula:

[0086]

[0087] In the formula, g t , g v , g a is a linear transformation that maps the [CLS] embeddings of text, video, and audio to normalized low-dimensional (256-dimensional) representations, respectively; a cls ,t cls and v clsdenote the [CLS] embeddings extracted from audio, text, and video, respectively; s(T,A), s(T,V), and s(A,V) denote the similarity functions () between text-audio, text-video, and audio-video features, respectively; T denotes the transpose of the matrix.

[0088] In addition, this embodiment follows the MoCo method and maintains three similarity function corresponding queues to store the most recent M text, audio and video representations, which come from the momentum unimodal encoder, respectively denoted as g′t(t′cls), g′a(a′cls) and g′v(v′cls). The calculation of the similarity score involving the momentum encoder is defined as follows:

[0089]

[0090] Then, a preset loss value is calculated based on all similarity scores, and all first features are aligned according to the preset loss value. Specifically, the preset loss value (MEC-Loss) is used to maximize the similarity score of matching pairs and minimize the similarity score of non-matching pairs to ensure effective alignment of text, audio, and video embeddings in a low-dimensional space. It can be understood that the calculation formula of the preset loss value is as follows:

[0091]

[0092] In the formula, Represents a preset loss value; the numerator represents the exponent of text data, audio data or video data; the denominator represents the exponential sum of the similarity scores corresponding to all possible text data, audio data or video data.

[0093] In an embodiment of the present application, the multimodal fusion module is used to extract context information in the conversation information based on the second feature. Specifically, the second feature can be input into the cross attention layer to obtain a fused feature, the fused feature can be input into the fully connected layer to obtain a standardized feature, and then all the standardized features can be input into the Mamba network to obtain an output sequence, wherein the output sequence is used to describe the context information of the conversation information. It can be understood that the cross attention layer of this embodiment includes 6 layers, and the formula of the cross attention is defined as follows:

[0094] Q=W Q [t cls ; a cls ;v cls ]

[0095] K=W K [t cls ; a cls ;v cls ];

[0096] V=W V [tcls ; a cls ;v cls ]

[0097] Among them, W Q , W K and W V is the weight matrix of query, key and value transformation, Q, K and V describe how to generate queries, keys and values ​​for cross attention mechanism from raw text, audio and video features.

[0098] The formula for calculating attention in the cross-attention mechanism is: This formula describes how to use these generated queries, keys, and values ​​to perform attention calculations to obtain the final fused features. The fused features are combined as follows: f = W F [t cls ; a cls ;v cls ]W F is the weight matrix of the combined features.

[0099] After obtaining the fused features, the fused features are input into the fully connected layer (FC) to generate the input {Z1, Z2, …, Z n}, in order to standardize the input dimension and enhance the model's ability to learn complex relationships in multimodal data, the features are transformed into a dimension suitable for the Mamba network: Z i =FC(f i ). The output standardized features are concatenated into a sequence x = [Z1, Z2, …, Z n ] and then input to the Mamba network. Specifically, the connected sequence x will go through Figure 4 Therefore, the process of the Mamba network model can be summarized as follows:

[0100] F=σ(f Ema (Z; G, J, M));

[0101] In the formula, Z is the eigenvector [Z1, Z2, …, Z n ]; F is the feature vector [F1, F2, …, F n ] composed of an output sequence; f Ema Represents the processing function of the Mamba network; and is the parameter of the state space model (SSM). σ represents the activation function σ transformation of the feature Z processed by the fEma function to obtain the final fusion feature F. This process aims to introduce nonlinearity and enhance the model's ability to learn and represent complex multimodal data relationships.

[0102] It can be seen that the above formula summarizes how the Mamba network uses the state-space model parameters G, J, M to process the input sequence and generate the corresponding output sequence.

[0103] In the embodiment of the present application, after obtaining the context information, the Softmax layer is used in the sentiment classifier to calculate the probability distribution on the sentiment category set, and the sentiment category with the maximum label probability is selected as the prediction result of the i-th utterance The calculation formula is as follows:

[0104]

[0105] In the formula, is the predicted sentiment label of the ith utterance. Softmax is the Softmax function. W and b are the weight and bias of the Softmax layer, respectively. F i is the i-th feature in the output sequence F.

[0106] In some embodiments, the method implemented in the present application and the existing method are tested on the IEMOCAP and MELD datasets, and the experimental results are obtained as shown in Table 1 (IEMOCAP dataset test results) and Table 2 (MELD dataset test results):

[0107] Table 1

[0108]

[0109]

[0110] Table 2

[0111]

[0112] As can be seen from Tables 1 and 2, in the IEMOCAP dataset, the MEMO-Mamba provided in this embodiment shows significant advantages in multiple emotion categories, especially in the neutral category, achieving the highest WF1 score of 74.88%. The model also performs well in the sadness and frustration categories, with WF1 scores of 85.26% and 70.59%, respectively. In the MELD dataset, the MEMO-Mamba provided in this embodiment performs well in the neutral, joyful and angry emotion categories, with WF1 scores of 82.13%, 64.52% and 56.28%, respectively. And it also performs well in the surprise and sadness categories, with WF1 scores of 55.64% and 38.21%, respectively. It can be seen that the network model provided in the embodiment of the present application can more effectively capture subtle expressions and changes in emotional intonation, thereby improving the accuracy of emotion recognition classification.

[0113] In this example, it can be seen from Tables 1 and 2 that when the Mamba network component is removed, the WF1 scores of various emotion categories decrease significantly. Specifically, in the IEMOCAP dataset, the WF1 scores of the happy, neutral, and excited categories are significantly reduced compared to the complete MEMO-Mamba model. This shows that the Mamba network component plays a key role in accurately capturing these positive emotions.

[0114] Similarly, removing the MEC-Loss component also affects the performance of the model, especially in the anger category, where the WF1 score drops significantly. This shows that the MEC-Loss component is essential for accurately classifying stronger emotions such as anger. In the MELD dataset, the absence of the Mamba network component leads to lower WF1 scores for the surprise, sadness, and anger categories, indicating the importance of this component in processing diverse emotional expressions. Removing the MEC-Loss component leads to lower scores for the surprise, fear, and sadness categories, emphasizing its importance in these negative emotions.

[0115] In summary, this application combines MEC-Loss to improve the feature similarity between text, audio, and video encoders, and uses the Mamba network to dynamically integrate a wider range of contextual information. MEMO-Mamba solves the key challenges faced by existing ERC models. Comprehensive experiments on the MELD and IEMOCAP benchmark datasets show that MEMO-Mamba not only outperforms the current state-of-the-art methods, but also significantly improves the accuracy of emotion recognition when information is distributed across multiple modalities and conversation turns. MEC-Loss helps to better align features extracted from text, audio, and video, thereby achieving more effective multimodal fusion. The Mamba network, with its ability to capture contextual dependencies, further enhances the system's ability to understand the emotional dynamics in conversations.

[0116] Reference Figure 5 The embodiment of the present application provides a conversation emotion recognition device based on a multimodal Mamba network, the device comprising:

[0117] The first module 610 is used to obtain dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes;

[0118] The second module 620 is used to input the dialogue information into the multimodal Mamba network to obtain the emotion category label of each speech data;

[0119] Among them, the multimodal Mamba network includes an encoder, a data alignment module, a multimodal fusion module and a sentiment classifier;

[0120] The encoder is used to extract features of the multimodal speech data from the dialogue information as the first feature;

[0121] The data alignment module is used to align all the first features to obtain the second features;

[0122] The multimodal fusion module is used to extract context information in the dialogue information according to the second feature;

[0123] The sentiment classifier is used to determine the sentiment category label of each utterance data according to context information.

[0124] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned conversation emotion recognition method based on the multimodal Mamba network when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0126] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0127] See also Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0128] The processor 710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0129] The memory 720 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 720 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 720, and the processor 710 calls and executes the methods of the embodiments of this application;

[0130] Input / output interface 730, used to implement information input and output;

[0131] Communication interface 740, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0132] bus 750 , which transmits information between the various components of the device (e.g., processor 710 , memory 720 , input / output interface 730 , and communication interface 740 );

[0133] The processor 710 , the memory 720 , the input / output interface 730 , and the communication interface 740 are connected to each other in communication within the device via the bus 750 .

[0134] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned conversation emotion recognition method based on the multimodal Mamba network.

[0135] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0138] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0139] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0141] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0142] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0147] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A conversation emotion recognition method based on a multimodal Mamba network, characterized in that: The method comprises the following steps: Acquiring dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes; Inputting the dialogue information into a multimodal Mamba network to obtain a sentiment category label for each of the speech data; Wherein, the multimodal Mamba network includes an encoder, a data alignment module, a multimodal fusion module and a sentiment classifier; The encoder is used to extract a feature of the multimodal speech data from the dialogue information as a first feature; The data alignment module is used to align all the first features to obtain a second feature; The multimodal fusion module is used to extract context information in the dialogue information according to the second feature; The sentiment classifier is used to determine the sentiment category label of each of the utterance data according to the context information.

2. The method according to claim 1, characterized in that: The encoders include a text encoder, an audio encoder and a video encoder; The text encoder is used to extract a first emotion feature of the text data in the dialogue information; The audio encoder is used to extract a second emotion feature of the audio data in the dialogue information; The video encoder is used to extract the third emotion feature of the video data in the dialogue information.

3. The method according to claim 2, characterized in that The aligning process of all the first features comprises: Mapping all of the first emotion features, the second emotion features, and the third emotion features to a normalized low-dimensional network to obtain a corresponding first similarity function, a second similarity function, and a third similarity function; Calculating a similarity score of the momentum encoder according to the first similarity function, the second similarity function and the third similarity function; All the first features are aligned according to the similarity scores.

4. The method according to claim 3, characterized in that: The aligning process of all the first features according to the similarity scores includes: Calculating a preset loss value based on all the similarity scores, wherein the preset loss value is used to maximize the similarity scores of the matching pairs and minimize the similarity scores of the non-matching pairs; All the first features are aligned according to the preset loss value.

5. The method according to claim 4, characterized in that The calculation formula for calculating the preset loss value based on all the similarity scores is as follows: In the formula, represents the preset loss value; the numerator represents the exponent of the text data, the audio data or the video data; the denominator represents the exponent sum of all possible similarity scores corresponding to the text data, the audio data or the video data.

6. The method according to claim 1, characterized in that The extracting the context information in the dialogue information according to the second feature includes: Input the second feature into the cross attention layer to obtain a fused feature; Input the fused features into a fully connected layer to obtain standardized features; All the standardized features are input into the Mamba network to obtain an output sequence, where the output sequence is used to describe the context information of the conversation information.

7. The method according to claim 1, characterized in that The determining the emotion category label of each of the speech data according to the context information comprises: Predicting a label probability of each of the utterance data according to the context information; The label corresponding to the maximum label probability is used as the emotion category label corresponding to the speech data.

8. A conversation emotion recognition device based on a multimodal Mamba network, characterized in that: The device comprises: The first module is used to obtain dialogue information, wherein the dialogue information includes a plurality of speech data in multiple modes; The second module is used to input the dialogue information into the multimodal Mamba network to obtain the emotion category label of each speech data; Wherein, the multimodal Mamba network includes an encoder, a data alignment module, a multimodal fusion module and a sentiment classifier; The encoder is used to extract a feature of the multimodal speech data from the dialogue information as a first feature; The data alignment module is used to align all the first features to obtain a second feature; The multimodal fusion module is used to extract context information in the dialogue information according to the second feature; The sentiment classifier is used to determine the sentiment category label of each of the utterance data according to the context information.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images