Dialogue emotion recognition method and device based on multi-modal large model

Abstract

The invention discloses a dialogue emotion recognition method and device based on a multi-modal large model, and relates to the field of emotion recognition, and the method comprises the steps: obtaining all statement sets in a dialogue in a current scene; each statement comprises an audio mode, a video mode and a text mode; constructing a dialogue emotion recognition model; the dialogue emotion recognition model comprises a feature extraction layer, a bidirectional gating unit, a multi-modal large model, a BERT language model, a modal information complementation module, a semantic graph-based multilayer residual image convolutional network and a full connection layer; and according to all statement sets in the dialogue, adopting the trained dialogue emotion recognition model to obtain an emotion recognition result. The dialogue emotion recognition method and device can improve the accuracy and robustness of dialogue emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition, and in particular to a method and device for dialogue emotion recognition based on a multimodal large model. Background Art

[0002] In recent years, the development of artificial intelligence (AI) technology has turned many seemingly sci-fi concepts into reality. For example, more and more household robots have the ability to provide comfort upon user request. The time to process such a request depends on the intelligence level of the machine and the efficiency of human-computer interaction. Therefore, efficiency has become a key indicator for improving machine intelligence. Making machines more humane, that is, enabling them to quickly adjust their behavior when users' emotions change, such as providing comfort when users are depressed, and timely reminding distracted learners in smart classrooms, is particularly important in the field of intelligent robots. In addition, accurately capturing user emotional changes can enhance human-computer interaction, thereby providing users with higher-level artificial intelligence services. Accurately identifying users' emotional states is not only a necessary condition for improving machine functions, but also an important research focus in the field of artificial intelligence. Therefore, emotion recognition tasks play a key role in many fields and have received widespread attention from scholars.

[0003] Conversational emotion recognition (ERC), also known as conversational emotion detection, is the task of discerning the emotional state of a speaker by analyzing the signals conveyed by the speaker in the context of the conversation (e.g., text, audio, or facial expressions). ERC has significant potential in multiple areas, including: (i) Customer service: ERC facilitates user interaction with customer service by identifying and responding to the user's emotional state, thereby improving overall satisfaction. (ii) Social media analysis: In the field of social media, ERC facilitates the capture and analysis of user emotions, allowing companies to collect user satisfaction with their products or services. (iii) Educational technology: ERC can detect the changes in the emotions of each student in the classroom in a timely manner and provide feedback to teachers, providing a valuable emotional basis for adjusting teaching methods. (iv) Healthcare: In the healthcare field, it can also assess the emotional health of patients in telemedicine consultations, providing insights for medical professionals for further testing. (v) Human-computer interaction: ERC makes human-computer interaction more intuitive and responsive, enabling virtual assistants and chatbots to effectively recognize and adapt to users' emotions. Overall, the ability to recognize emotions in conversations can help AI achieve greater empathy and more effective communication across a wide range of corpus contexts, and this ability is particularly important in the field of educational technology.

[0004] In the field of educational technology, ERC technology is widely used to improve teaching effectiveness and student experience. It can be roughly divided into two categories. One is sentiment analysis based on natural language processing (NLP), which identifies students' emotional state by analyzing their words and tone in class. This enables educators to better understand students' emotional feedback, adjust teaching strategies in a timely manner, and provide personalized learning support. The other is emotion recognition based on audio and video, which identifies students' emotional changes during the learning process by analyzing their voice features and expression changes. This technology can be applied in distance learning, and learners' emotions can be captured in real time through speech recognition technology, providing educators with a basis for adjusting teaching methods. However, the above technologies have some natural disadvantages. First, both of the above methods rely on time series modeling. When the conversation is too long, time series modeling cannot obtain good feature representation due to long-range dependency problems, resulting in inaccurate emotion recognition. Second, with the widespread application of multimodal data, multimodal dialogue emotion recognition has become a research hotspot. Traditional emotion recognition focuses on a single modality, but in actual conversations, people communicate information through text, voice, video and other methods. Multimodal conversation emotion recognition aims to use multimodal data to provide richer and more comprehensive emotion representation. In conversations, emotions are often expressed in multiple ways and are affected by the context and the relationship between modalities. By integrating multimodal data, we can understand the conversation context more comprehensively and improve the accuracy of emotion understanding. Traditional multimodal emotion recognition fuses data from different modalities through different feature fusion methods, ignoring the natural shared and unique information within the modalities. There may be some correlation between these information, which will affect the judgment of emotion. Secondly, most traditional graph-based methods are only an extension of the time series modeling method. Simply constructing multiple modal information into a graph cannot fully explore the key factors that affect emotion transformation.

[0005] The three modal data of text, voice and video of learners all imply the emotional state of the learner in the current conversation. How to use the data of the three modalities for high-quality representation learning directly affects the accuracy of the final emotional judgment. The two major challenges that need to be addressed are the understanding of multimodal fine-grained emotional semantic information and the modeling of long-term complex learner conversation contexts. Therefore, compared with the traditional temporal modeling method, the fusion of multimodal large language models (MMLMs) and graph representation learning for dialogue modeling has a natural advantage. Specifically, the multimodal large model integrates key components such as feature encoders of each modality, attention feature fusion mechanism, large language model and external subtask model. Under the premise of cross-modal semantic alignment such as vision-language and voice-language, it can accurately process complex cross-modal instructions and has high-level fine-grained multimodal semantic understanding capabilities. This capability can serve many cross-modal downstream tasks, such as visual understanding and question answering, pixel-level image or video editing, and voice human-computer interaction. By introducing a multimodal large model, we can use its fine-grained understanding of the potential emotional clues in the learner's multimodal dialogue data, obtain emotional semantic information directly related to the judgment of emotional state, and achieve multimodal fine-grained emotional clue semantic information gain in the model reasoning process, thereby effectively improving the representation quality of dialogue data in complex teaching situations. Graph representation learning performs well in dealing with complex nonlinear relationships and structured data. Many practical problems can be better represented by graph-based modeling, such as social networks, molecular structures in bioinformatics, and knowledge graphs. Traditional representation learning may not be able to effectively capture these complex relationships. At the same time, graph representation learning can consider the contextual information between nodes and their neighboring nodes. This makes the learned representation more reflective of the relationship between entities or events and their surroundings, which helps to express the context more accurately; in addition, graph representation learning methods are usually more general and can adapt to a variety of different tasks. This flexibility makes graph models more applicable when dealing with problems in various fields. Therefore, graph representation learning has always been one of the popular machine learning techniques, and a large number of researchers have conducted research in this area. In a dialogue graph, an object is usually regarded as a node, and the edges between nodes are used to represent the relationship between objects. Specifically, the three modalities of a single sentence in a conversation are regarded as three different nodes, and different types of edges are formed based on different semantic backgrounds. For multimodal data, graph representation learning converts the features of each modality into the relationship between edges and nodes through the above different types of edges, thereby taking the three modalities into consideration.

[0006] At present, multimodal large language models and algorithms based on graph representation learning have made great progress. However, common modal information interaction methods are mainly divided into early fusion, late fusion, and cross-learning fusion. The early fusion method first extracts feature expressions from each independent modality, and mixes each feature by multiplying and adding elements at corresponding positions, which alleviates the problem of inconsistent original data representation in different modalities. Late fusion uses different models for training different modalities, and then outputs the learning results of multiple models through maximum combination, mean combination or ensemble learning. This method solves the asynchrony of modal data processing and improves the scalability of modalities, but ignores the mutual correlation between modalities and lacks the representation of shared information. Cross-learning fusion strengthens the transmission of shared information between modalities by mapping feature representations of different modalities to a common semantic representation space, but cannot distinguish redundant information in shared information. Deep fusion of multimodal features is one of the key links of ERC technology. The design of this part is directly related to the inference accuracy of the final emotional state category. To this end, traditional multimodal emotion recognition methods based on deep neural networks often align multimodal feature semantics through deep representation and cross-modal shared semantic space learning, in order to effectively fuse cross-modal features and infer potential emotion categories based on cross-modal information consistency learning and information complementarity. However, such methods generally use loss function design and simple single-stage feature fusion (early, late fusion, etc.) to achieve cross-modal feature semantic alignment and complementary utilization, which easily leads to loss and misunderstanding of the emotional semantic information (modality-specific semantics) contained in each modality; in addition, the existing ERC methods still have the problem of insufficient perception granularity of emotional semantic information in each modality. The above two limitations make it difficult for existing ERC methods to accurately and robustly identify learners' emotional states in complex teaching and cognitive interaction situations.

[0007] Although the traditional graph neural network GCN has been very successful, and variant convolutional networks based on GCN, such as GCN2 and GraphSage, have also improved in algorithm complexity, these models are generally insufficient. For example, GCN has the problem of over-smoothing, and when the network layers are stacked, the model performance is significantly reduced. This phenomenon limits the development of traditional graph convolutional networks and cannot maximize the use of raw data. At the same time, in many large-scale conversation scenarios, these traditional network models cannot distinguish which information has a long-term impact on the emotions of the interlocutor, that is, the long-term topic factors in the conversation, and which information is the cause of the interlocutor's emotional mutation, that is, the short-term sensitive information in the conversation.

[0008] But overall, the accuracy and robustness of the task of conversation emotion recognition for learners using existing technologies still needs to be improved.

[0009] Therefore, based on the above problems, there is an urgent need to provide a new conversation emotion recognition method to improve the accuracy and robustness of conversation emotion recognition. Summary of the invention

[0010] The purpose of this application is to provide a method and device for conversation emotion recognition based on a multimodal large model, which can improve the accuracy and robustness of conversation emotion recognition.

[0011] To achieve the above objectives, this application provides the following solutions: First invention The present application provides a method for identifying emotion in a conversation based on a multimodal large model, and the method for identifying emotion in a conversation based on a multimodal large model comprises: Get all the sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; Construct a conversation emotion recognition model; the conversation emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementary module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementary module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality according to the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities according to the initial feature vector representations corresponding to different modalities; and according to The attention weight score matrix between different modalities uses attention operation to obtain modal shared information features and modal unique information features, and then uses differential regularization operator loss to ensure the information discrimination of modal shared information features and modal unique information features; the multi-layer residual graph convolution network based on semantic graph is used to splice modal shared information features and modal unique information features to obtain modal completion features; and the modal completion features are used as nodes to construct a semantic relationship graph; and then the multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and the long-term factors and emotional transient factors that affect the learner's emotional transformation are extracted based on the multi-layer residual graph convolution network; the long-term factor is the global and long-term dialogue information of the semantic graph node features obtained by shallow one-dimensional convolution in the long-term channel; the emotional transient factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of the short-term channel; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and emotional transient factors that affect the learner's emotional transformation; According to the set of all sentences in the conversation, the emotion recognition result is obtained by using the trained conversation emotion recognition model.

[0012] Optionally, the feature extraction layer specifically includes: a multi-layer perceptron, a long short-term memory network, and a text-based multi-layer perceptron; Use a multi-layer perceptron to extract the corresponding initial features of the original unprocessed features of the audio modality; The long short-term memory network is used to extract the corresponding initial features of the original unprocessed features of the video modality; A text-based multi-layer perceptron is used to extract the corresponding initial features of the original unprocessed features of the text modality.

[0013] Alternatively, using the formula Determine the final characteristics of each mode ; in, () is the concatenation function, is the initial feature of each mode, () is the encoding function of the BERT language model, is the fine-grained emotional cue feature of each modality, parameter , For modal video, For audio mode, For text mode.

[0014] Optionally, the modal information complementation module is used to obtain an initial feature vector representation of complementary learning of the corresponding modality according to the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities according to the initial feature vector representations corresponding to different modalities; and according to the attention weight score matrix between different modalities, an attention operation is used to obtain modal shared information features and modal unique information features, and then a differential regularization operator loss is used to ensure the information discrimination of the modal shared information features and the modal unique information features, specifically including: Using the formula Determining initial feature representation for complementary learning ; Using the formula Determine m The mode and n The attention weight score matrix between modalities ; Using the formula Determine text mode l The modality sharing information feature ; Using the formula Determine the unique information characteristics of the modality; Using the formula Ensure modality sharing of information features across all modalities Unique information features of the modality The degree of distinction between in, To embed the conversation person information, , For speaker groups In the The hidden state of the layer, , For all speaker groups A collection of words spoken, For speaker groups In the -1 layer of hidden state, is the original input statement, () is a mathematical function, represents the mapping matrix, is the cross-modal semantic information, , represents the identity matrix, is a hyperparameter that controls the degree of preservation of the original modal information. is the attention weight score matrix, the superscript T is the transpose, is the value matrix, () indicates vectorized operation, A function to obtain modality-specific information. is the difference regularization operator loss, F represents the Frobenius norm, is the query matrix of m modes, is the bond matrix of n modes, is a scale factor used to control the tightness of attention calculation.

[0015] Optionally, the semantic relationship graph uses the formula Determine edge weights ; in, ()Cosine similarity calculation function.

[0016] Alternatively, using the formula Determine the loss function for training the conversation emotion recognition model ; in, is the total classification loss, The preliminary classification loss calculated for the modality complementary feature information and the true label, and To balance the hyperparameters of the two losses, is the set of modal complementary features, is the true label set.

[0017] In a second aspect, the present application provides a conversation emotion recognition device based on a multimodal large model, and the conversation emotion recognition device based on the multimodal large model includes: The data acquisition module is used to obtain the set of all sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; A model building module is used to build a conversation emotion recognition model; the conversation emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementation module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementation module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality based on the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities based on the initial feature vector representations corresponding to different modalities ; and according to the attention weight score matrix between different modalities, the attention operation is used to obtain the modal shared information features and the modal unique information features, and then the differential regularization operator loss is used to ensure the information discrimination of the modal shared information features and the modal unique information features; the multi-layer residual graph convolution network based on the semantic graph is used to splice the modal shared information features and the modal unique information features to obtain the modal completion features; and the modal completion features are used as nodes to construct a semantic relationship graph; and then the multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and according to the multi-layer residual graph convolution network, the long-term factors and emotional transient factors that affect the learner's emotional transformation are extracted; the long-term factor is the global and long-term dialogue information of the semantic graph node features obtained by using shallow one-dimensional convolution in the long-term channel; the emotional transient factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of the short-term channel; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and emotional transient factors that affect the learner's emotional transformation; The recognition result output module is used to obtain the emotion recognition result based on the set of all sentences in the conversation using the trained conversation emotion recognition model.

[0018] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for dialogue emotion recognition based on a multimodal large model.

[0019] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method and device for dialogue emotion recognition based on a multimodal large model. The constructed dialogue emotion recognition model includes: feature extraction layer, bidirectional gating unit, multimodal large model, BERT language model, modal information complementation module, multi-layer residual graph convolution network based on semantic graph and fully connected layer; wherein, the modal information complementation module (AMC module) calculates the attention weight scores of different modalities to extract features between target modalities, which guarantees the information sharing between modalities to the greatest extent. At the same time, unlike the cross-learning fusion in the prior art, the proposed modal information complementation module can independently extract modal sharing and independent information, so as to facilitate the attribution of emotional information judgment from multiple perspectives and improve the robustness of the algorithm. The modal information complementation module performs cross-modal information extraction, and can take the three modalities of text, video and audio as the target modality (in parallel), and extract valuable information between them and the other two complementary modalities. A large multimodal model is introduced, and its ability to understand fine-grained emotional clues in each modal information is used to share and map the fine-grained emotional information implicit in different modalities to the text semantic space. With the help of this multimodal fine-grained emotional clue semantic information gain, the accuracy and robustness of learner conversation emotion recognition in complex teaching scenarios of the model are improved. In order to capture this special information to the greatest extent, while reducing the interference of redundant information, the final features are more reliable and richer. This application proposes a multi-layer residual graph convolutional network, which not only captures the emotionally sensitive information of the conversation, but also solves the problem that the traditional graph neural network (GCN) cannot stack multiple layers, improves the effect of model learning, and also provides a new tool for the efficient extraction of multimodal conversation sensitive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is a flow chart of a method for identifying conversation emotions based on a multimodal large model in one embodiment of the present application; Figure 2 A schematic diagram of a multimodal learner dialogue scenario; Figure 3 This is a schematic diagram of the dialogue emotion recognition model structure; Figure 4 This is a schematic diagram of a sample cross-modal instruction in video mode; Figure 5 This is a schematic diagram of a sample cross-modal instruction in audio mode; Figure 6 This is a sample diagram of instructions in text mode; Figure 7 Schematic diagram of a multi-layer residual graph convolutional network based on semantic graph. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] In an exemplary embodiment, Figure 1 As shown, a method for dialogue emotion recognition based on a multimodal large model is provided, and the method includes the following S101 to S103. Among them: S101, obtaining a set of all sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; like Figure 2 As shown, define a dialogue in a scene , ,in, It represents the number of sentences in this conversation. Respectively represent The original input of the three modalities (audio, video, text) corresponding to the sentence. Define the set of speakers in this conversation ,in, is the total number of speakers. In words, The mapping relationship between statements and speakers can be defined as For all speaker groups A collection of words spoken It can be defined as , the goal of this application is to build a large semantic relationship graph The proposed multi-layer residual graph convolutional network model is used to learn and derive the current speech on the graph. Corresponding emotional labels .

[0025] S102, constructing a dialogue emotion recognition model; Figure 3As shown, the dialogue emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementary module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementary module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality according to the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities according to the initial feature representations corresponding to different modalities; and according to the attention weight scores between different modalities Matrix, using attention operation to obtain modal shared information features and modal unique information features, and then using differential regularization operator loss to ensure the information discrimination of modal shared information features and modal unique information features, avoiding model collapse caused by the lack of discrimination of the two information; a multi-layer residual graph convolution network based on semantic graph is used to splice modal shared information features and modal unique information features to obtain modal completion features; and a semantic relationship graph is constructed with modal completion features as nodes; and then a multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and based on the multi-layer residual graph convolution network, long-term factors and transient emotional factors that affect learners' emotional changes are extracted; the long-term factor is the global and long-term dialogue information of semantic graph node features obtained by shallow one-dimensional convolution in long-term channels; the transient emotional factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of short-term channels; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and transient emotional factors that affect learners' emotional changes; The feature extraction layer specifically includes: a multi-layer perceptron, a long short-term memory network, and a text-based multi-layer perceptron; For audio modality, the original feature input is processed through a multi-layer perceptron (MLP) For feature extraction, the calculation is as follows: ; For the video modality, due to its strong contextual relevance, a long short-term memory network (LSTM) is used to train the original video features. Perform feature extraction, expressed as: ; Specifically, for audio and video modalities, feature extraction is expressed as: ; For the original text modality features, a text-based multi-layer perceptron (TMLP) is used for feature extraction, which is expressed as: ; in, Represents the output after the first feature encoding, that is, the initial input sentence; , Represents learnable parameters. At the same time, in order to simulate the scene in real conversation, a bidirectional gating unit is used here to extract the information embedding of the dialogue person , thereby improving the accuracy of emotion recognition, expressed as: ; in, For speaker groups In the The hidden state of the layer, , For all speaker groups A collection of words spoken, For speaker groups In the -1 layer of hidden state, is the original input statement; Specifically, when the human brain analyzes multimodal learner data such as video, audio, and text and makes judgments on the target emotional state, there is an implicit mining process for multidimensional fine-grained emotional clues in each modality. This process provides the human brain with rich granular valuable information for comprehensive analysis and complementary use of multimodal emotional semantics. Therefore, a multimodal large model is introduced in the ERC task, using its powerful cross-modal semantic understanding ability to fully and effectively mine the fine-grained emotional clue information in the learner data of each modality, so as to improve the sensitivity and accuracy of the ERC model in perceiving the learner's emotional state.

[0026] The multimodal large model "iFlytek Spark Large Model" developed by iFlytek is used as the backbone network of the multimodal fine-grained emotional clue mining module. It takes the three-modal semantic-rich data of video, audio, and text as input, and outputs the semantic text information of emotional clues at each modal granularity level. This process explicitly completes the shared mapping of the fine-grained emotional clue information of the three modalities of video, audio, and text to the text feature semantic space, reducing the natural semantic gap between the emotional information of each modality.

[0027] Cross-modal instructions are provided for semantic mining of fine-grained emotional clues in three modal data: video, audio, and text. Valuable text information of emotional clues in each modality is obtained in JSON format. Among them, cross-modal instructions all have emotional state perception as an implicit goal, and output of fine-grained emotional clues in three modalities as an explicit goal.

[0028] Specifically, for the video modality, cross-modal instructions are used to guide the multimodal large model to perceive, understand, and output fine-grained visual emotional cues in the video that are related to the learner's emotional state. , including the learner's facial expression-related muscle changes in the video modality (muscle changes in the corners of the eyes, eyebrows, etc.), body movements, etc., instruction design and examples are as follows: Figure 4 shown.

[0029] For the audio modality, cross-modal instructions are mainly aimed at obtaining fine-grained auditory emotional cues such as the tone, intonation, and speaking speed of the audio that are explicitly related to the learner's emotional state. For the purpose, instruction design and sample Figure 5 shown.

[0030] For text modality, the focus is on obtaining fine-grained language emotional clues such as contextual emotional semantics, sentence structure, and special punctuation in text sequences that reflect the learner's emotional state through a multimodal large model. , instruction design and sample examples Figure 6 shown.

[0031] Acquire fine-grained sentiment clue semantic text information in three modes: video, audio and text sentences of learners' conversations , and Finally, the BERT language model, which is pre-trained on a large-scale Chinese corpus and fine-tuned on a sentiment information corpus, is used to encode the fine-grained sentiment clue semantic text to obtain the fine-grained sentiment clue semantic embedding, and they are concatenated with the initial features of the corresponding modalities to obtain the final features of the three modalities of video, audio, and text before the complementary learning of modal information. express: ; in, () is the concatenation function, is the initial feature of each mode, () is the encoding function of the BERT language model, is the fine-grained emotional cue feature of each modality, parameter , For modal video, For audio mode, For text mode.

[0032] Since unimodal data rarely provide enough information for accurate assessment, a mature and reliable emotion recognition model should be able to exploit the commonalities and complementarities of multimodal data. Generally speaking, the commonalities between multimodal data are considered to represent the consistency of conversation or emotion polarity. In contrast, complementary information means specific emotions or emotion transitions. Intuitively, combining the commonalities and unique information between modalities can improve the accuracy of emotion recognition. Therefore, the initial feature representation for complementary learning is obtained by embedding the conversation person and concatenating it with the initial features. , expressed as: ; Then for all sentence features , there is a mapping relationship ,in, It is used to extract modal shared information from sentence features, and It is used to extract modality-specific information from sentence features, expressed as: ; in ( ), ( ) represents the modal shared information and modal unique information corresponding to each modality. Based on this, for a given statement The modal complementarity representation It can be expressed as: ; Considering the feature embeddings of the three modalities as word vectors, the modality-shared and modality-unique semantic information can be defined as cross-modal attention. Taking the text modality as an example, for a given text feature , the query matrix can be defined as , similarly, the bond matrix is , the value matrix is ,in, and is the transformation matrix, then use the formula Determine m The mode and n The attention weight score matrix between modalities ;based on , then the cross-modal semantic information extraction can be expressed as: ; in, represents the identity matrix, =1, which is a hyperparameter to control the degree of preservation of the original modal information. The unit matrix is ​​added to prevent the gradient from dissipating. Therefore, for the text modality, its modal shared information feature It can be expressed as: ; in, represents the mapping matrix, () indicates vectorized operation, then the mapping relationship It can be expressed as: ; in, is the function definer, The mth row of represents the degree of association of the mth modality with other modalities. The mth modality is regarded as the recognition target, and the unique semantic information in it, including the sudden emotional changes in the discourse, can be reflected by the difference in the attention paid by the mth modality to each other modality. Therefore, the attention matrix of each discourse in the conversation can be regarded as a representation in the modality-unique embedding subspace, which is used to represent the difference in unique information between modalities. Then the mapping relationship is It can be expressed as: ; In order to improve the difference between modal shared information and modal unique information and reduce the repeated extraction of redundant information, the differential regularization operator is introduced, which is expressed as: ; in, is the unique information feature of the modality, the superscript T is the transposition, is the value matrix, () indicates vectorized operation, is the difference regularization operator loss, F represents the Frobenius norm, is the query matrix of m modes, is the bond matrix of n modes, is a scale factor used to control the tightness of attention calculation.

[0033] Graph models are naturally suitable for handling heterogeneous relationships. They can more flexibly model the relationships between different types of edges and nodes. At the same time, they can better capture the changing process of emotions through the dynamic evolution of graphs at different time points or contexts. Considering that in many emotion recognition scenarios, different modalities may influence each other, in order to capture this interaction between modalities, a semantic graph based on modal completion features is constructed to improve the recognition accuracy of the model. Specifically, in a conversation, each sentence It can be represented as three types of nodes , representing audio nodes, video nodes and text nodes respectively, , then for N sentences, there are nodes, ,In order to establish the connection between discourse and different modalities, whether there is an edge connection between two nodes depends on the following rules: 1: Any two nodes of the same modality in the same discourse are connected. 2: Each node remains connected to nodes from different modalities but corresponding to the same discourse. 3: Any discourse Should be associated with its three modal nodes and In order to distinguish the importance of different adjacent nodes, different weights are assigned to the edges according to their importance, that is, a larger weight is assigned to nodes that are more important than other nodes. It can be expressed as: ; in, ()Cosine similarity calculation function.

[0034] In obtaining the semantic graph Then, in order to obtain the statement The precise feature expression of The features are initialized as Specifically, since the relationship between discourses in a conversation is changing, it is necessary to retain the historical interaction information of the interlocutors, while retaining the current interaction relationship information also plays a vital role in the ERC task. In order to balance the interaction parts that should be retained or discarded, this application sets up a short-term storage unit To preserve the current interaction status or emotional mutation information. Initialized to 0, indicating that there is no interaction that needs to be retained. At the same time, a long-term memory unit Used to save historical interaction information or conversation topic information, such as Figure 7 shown.

[0035] like Figure 7 As shown in , an improved graph convolution operation is selected to aggregate complementary semantics in a unique subspace. Here, the input semantic graph is regarded as a sequence, and the input graph flow can be expressed as ,in, Representing the semantic graphs of different layers, the message passing process can be expressed as: ; in, represents the regularized graph Laplacian matrix, is the regularized diagonal matrix, is the regularized adjacency matrix, and is a hyperparameter, , is also a hyperparameter, is the weight matrix, is the identity matrix, For the The layer semantic graph node feature matrix adopts an improved graph convolutional network instead of a common graph convolutional network, which provides strict control over the filter and reduces the computational complexity by avoiding the explicit use of graph Fourier basis. , For the Layer semantic graph node features, For the A short-term storage unit, which means embedding the current interaction information into the graph matrix At the same time, in order to accurately retain and clear the interactive information, there are: ; in, represents a message passing gate, , are learnable parameters, Represents the sigmoid function, for the parameter , then there is, Controls the information to be written to the memory cell, Controls the redundant interaction information to be erased from the memory cell, Control the feature representation of the final output. At the same time, the hidden state of the long-term memory unit is updated It can be expressed as: ; in, represents the activation function, , represents the learnable parameters, then the update of the long-term memory unit It can be expressed as: ; in, Represents the dot product operation. This memory unit stores the information that should be forgotten from the previous layer, and also stores the updated information from the current layer. The above structure is used to enrich the final feature representation. The calculation process can be expressed as: ; in, MREGCN That is the proposed multi-layer residual graph convolutional network, The first The set of semantic graph nodes and edges of the first layer Layer node feature representation, Features are initialized as modal complementary features ; In passing After the layer encoding, the final output feature matrix is ​​obtained In addition, in order to preserve the original interaction information, the final feature It is expressed as: ; in, Represents a splicing operation, Represents the learnable parameters, and the final Taking into account information from six different levels ( ), used for the final sentiment classification task, the prediction process can be expressed as: ; ; ; in, For the statement The characteristic vector of represents the feature representation of the middle hidden layer of the fully connected layer during prediction, is the output of the model, indicating that for the sentence The probability distribution of emotion, For the statement The final sentiment prediction result (take The maximum probability corresponds to the sentiment category under the index value). and represents the activation function and normalization function, , , , The learnable parameters of the fully connected layer used in the prediction process, the classification prediction result loss can be expressed as: ; in, represents all dialogue samples, Represents all sentences in the conversation, For the sample The category label of the i-th sentence in , represents the predicted label, and Represent the learnable parameters and regularization coefficients respectively. Based on this, the loss function of the prediction results of the training model is finally used for: ; in, is the total classification loss, The preliminary classification loss calculated for the modality complementary feature information and the true label, is the difference regularization operator loss, and To balance the hyperparameters of the two losses, is the set of modal complementary features, is the true label set.

[0036] S103, based on a set of all sentences in the conversation, using a trained conversation emotion recognition model to obtain an emotion recognition result.

[0037] Based on the same inventive concept, the embodiment of the present application also provides a conversation emotion recognition device for implementing the above-mentioned conversation emotion recognition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more conversation emotion recognition device embodiments provided below can refer to the limitations of the conversation emotion recognition method above, and will not be repeated here.

[0038] In an exemplary embodiment, a conversation emotion recognition device based on a multimodal large model is provided, comprising: The data acquisition module is used to obtain the set of all sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; A model building module is used to build a conversation emotion recognition model; the conversation emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementation module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementation module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality based on the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities based on the initial feature vector representations corresponding to different modalities ; and according to the attention weight score matrix between different modalities, the attention operation is used to obtain the modal shared information features and the modal unique information features, and then the differential regularization operator loss is used to ensure the information discrimination of the modal shared information features and the modal unique information features; the multi-layer residual graph convolution network based on the semantic graph is used to splice the modal shared information features and the modal unique information features to obtain the modal completion features; and the modal completion features are used as nodes to construct a semantic relationship graph; and then the multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and according to the multi-layer residual graph convolution network, the long-term factors and emotional transient factors that affect the learner's emotional transformation are extracted; the long-term factor is the global and long-term dialogue information of the semantic graph node features obtained by using shallow one-dimensional convolution in the long-term channel; the emotional transient factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of the short-term channel; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and emotional transient factors that affect the learner's emotional transformation; The recognition result output module is used to obtain the emotion recognition result based on the set of all sentences in the conversation using the trained conversation emotion recognition model.

[0039] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for dialogue emotion recognition based on a multimodal large model is implemented.

[0040] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0041] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0043] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0044] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0045] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0046] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0047] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A conversation emotion recognition method based on a multimodal large model, characterized in that: The conversation emotion recognition method based on the multimodal large model includes: Get all the sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; Construct a conversation emotion recognition model; the conversation emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementary module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementary module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality according to the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities according to the initial feature vector representations corresponding to different modalities; and according to The attention weight score matrix between different modalities uses attention operation to obtain modal shared information features and modal unique information features, and then uses differential regularization operator loss to ensure the information discrimination of modal shared information features and modal unique information features; the multi-layer residual graph convolution network based on semantic graph is used to splice modal shared information features and modal unique information features to obtain modal completion features; and the modal completion features are used as nodes to construct a semantic relationship graph; and then the multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and the long-term factors and emotional transient factors that affect the learner's emotional transformation are extracted based on the multi-layer residual graph convolution network; the long-term factor is the global and long-term dialogue information of the semantic graph node features obtained by shallow one-dimensional convolution in the long-term channel; the emotional transient factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of the short-term channel; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and emotional transient factors that affect the learner's emotional transformation; According to the set of all sentences in the conversation, the emotion recognition result is obtained by using the trained conversation emotion recognition model.

2. The method for dialogue emotion recognition based on a multimodal large model according to claim 1 is characterized in that: The feature extraction layer specifically includes: a multi-layer perceptron, a long short-term memory network, and a text-based multi-layer perceptron; Use a multi-layer perceptron to extract the corresponding initial features of the original unprocessed features of the audio modality; The long short-term memory network is used to extract the corresponding initial features of the original unprocessed features of the video modality; A text-based multi-layer perceptron is used to extract the corresponding initial features of the original unprocessed features of the text modality.

3. The method for dialogue emotion recognition based on a multimodal large model according to claim 1 is characterized in that: Using the formula Determine the final characteristics of each mode ; in, () is the concatenation function, is the initial feature of each mode, () is the encoding function of the BERT language model, is the fine-grained emotional cue feature of each modality, parameter , For modal video, For audio mode, For text mode.

4. The method for dialogue emotion recognition based on a multimodal large model according to claim 3 is characterized in that: The modal information complementation module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality according to the final features of each modality and the information embedding of the interlocutor; the complementary learning determines the attention weight score matrix between different modalities according to the initial feature vector representation corresponding to different modalities; and according to the attention weight score matrix between different modalities, the attention operation is used to obtain the modal shared information features and the modal unique information features, and then the differential regularization operator loss is used to ensure the information discrimination of the modal shared information features and the modal unique information features, specifically including: Using the formula Determining initial feature representations for complementary learning ; Using the formula Determine m The mode and n The attention weight score matrix between modalities ; Using the formula Determine text mode l The modality sharing information feature ; Using the formula Determine the unique information characteristics of the modality; Using the formula Ensure modality sharing of information features across all modalities Unique information features of the modality The degree of distinction between in, To embed the conversation person information, , For speaker groups In the The hidden state of the layer, , For all speaker groups A collection of words spoken, For speaker groups In the -1 layer of hidden state, is the original input statement, () is a mathematical function, represents the mapping matrix, is the cross-modal semantic information, , represents the identity matrix, is a hyperparameter that controls the degree of preservation of the original modal information. is the attention weight score matrix, the superscript T is the transpose, is the value matrix, () indicates vectorized operation, A function to obtain modality-specific information. is the difference regularization operator loss, F represents the Frobenius norm, is the query matrix of m modes, is the bond matrix of n modes, is a scale factor used to control the tightness of attention calculation.

5. The method for dialogue emotion recognition based on a multimodal large model according to claim 4 is characterized in that: Using formula in semantic relationship graph Determine edge weights ; in, ()Cosine similarity calculation function.

6. The method for dialogue emotion recognition based on a multimodal large model according to claim 5 is characterized in that: Using the formula Determine the loss function for training the conversation emotion recognition model ; in, is the total classification loss, The preliminary classification loss calculated for the modality complementary feature information and the true label, and To balance the hyperparameters of the two losses, is the set of modal complementary features, is the true label set.

7. A conversation emotion recognition device based on a multimodal large model, characterized in that: The conversation emotion recognition device based on the multimodal large model includes: The data acquisition module is used to obtain the set of all sentences in the conversation in the current scene; each sentence includes three modes: audio, video and text; A model building module is used to build a conversation emotion recognition model; the conversation emotion recognition model includes: a feature extraction layer, a bidirectional gating unit, a multimodal large model, a BERT language model, a modal information complementation module, a multi-layer residual graph convolutional network based on a semantic graph, and a fully connected layer; the feature extraction layer is used to extract the initial features of each modality respectively; the bidirectional gating unit is used to extract the interlocutor information embedding; the multimodal large model is used to extract the fine-grained emotional clue features of each modality; and the BERT language model is fine-tuned; the fine-tuned BERT language model is used to encode the fine-grained emotional clue features of each modality, and the encoded features are spliced ​​with the initial features of each modality using the multimodal large model to obtain the final features of the corresponding modality; the modal information complementation module is used to obtain the initial feature vector representation of the complementary learning of the corresponding modality based on the final features of each modality and the interlocutor information embedding; the complementary learning determines the attention weight score matrix between different modalities based on the initial feature vector representations corresponding to different modalities ; and according to the attention weight score matrix between different modalities, the attention operation is used to obtain the modal shared information features and the modal unique information features, and then the differential regularization operator loss is used to ensure the information discrimination of the modal shared information features and the modal unique information features; the multi-layer residual graph convolution network based on the semantic graph is used to splice the modal shared information features and the modal unique information features to obtain the modal completion features; and the modal completion features are used as nodes to construct a semantic relationship graph; and then the multi-layer residual graph convolution network is used to perform graph representation learning on the semantic relationship graph to obtain a multi-layer residual graph convolution network; and according to the multi-layer residual graph convolution network, the long-term factors and emotional transient factors that affect the learner's emotional transformation are extracted; the long-term factor is the global and long-term dialogue information of the semantic graph node features obtained by using shallow one-dimensional convolution in the long-term channel; the emotional transient factor is the local and short-term dialogue information after deep one-dimensional convolution and nonlinear mapping of the short-term channel; the fully connected layer is used to determine the emotion recognition result according to the long-term factors and emotional transient factors that affect the learner's emotional transformation; The recognition result output module is used to obtain the emotion recognition result based on the set of all sentences in the conversation using the trained conversation emotion recognition model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the conversation emotion recognition method based on a multimodal large model as described in any one of claims 1 to 6.

Images