Multi-modal dialogue emotion recognition method and device

Abstract

The invention discloses a multi-modal dialogue emotion recognition method and device, and relates to the technical field of multi-modal emotion recognition. More accurate emotion prediction is realized through the steps of extracting multi-modal features, inferring an implicit relationship, constructing a mixed relationship graph, performing multi-modal semantic alignment and the like. The method comprises the steps of performing single feature vector extraction and fusion on an input multi-modal dialogue sequence needing emotion recognition to obtain an initial multi-modal feature vector, determining an implicit association relationship between sentences included in each modal according to a constructed similarity function, and performing emotion recognition on the multi-modal dialogue sequence according to the implicit association relationship. And mapping the representation subjected to weighted fusion with the visual and audio modal information to a word list through a linear network to obtain a predicted word generation probability of the target sentence, and determining a difference loss value between the predicted word probability and an actual word probability based on a cross entropy loss function to obtain a loss function.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal emotion recognition, and more specifically to a multimodal conversation emotion recognition method and device. Background Art

[0002] With the prevalence of social media and instant messaging tools, public emotional expression and dissemination have become multimodal and real-time. Negative emotions that go unrecognized in conversations can trigger individual psychological crises or conflicts. Accurately identifying genuine emotions in conversations is crucial for social and psychological well-being. Therefore, integrating text, voice, and emoji information to identify genuine emotional connections across modalities and track the emotional transmission process will effectively reduce emotional misjudgments and support the creation of a healthy online environment.

[0003] Multimodal conversational emotion recognition aims to accurately capture the emotional state expressed by conversational participants in each utterance by integrating information from multiple modalities, including text, speech, and vision. The complexity of this problem lies in the need to not only understand the semantic and emotional cues within a single modality but also to uncover the interrelationships and influences between different modalities, thereby achieving comprehensive and accurate emotion recognition. In multimodal conversational emotion recognition, relationships between utterances, within modalities, and between modalities play a crucial role. Inter-utterance relationships reflect the coherence of the conversation and the dynamics of emotion. For example, a speaker's emotion may be influenced by the content of previous conversations, and this influence can be manifested in various ways, such as text semantics, voice intonation, or facial expressions. Intra-modal relationships reflect the inherent connections between different utterances within the same modality. Inter-modal relationships reveal the complementarity and interactivity of information from different modalities. For example, intonation changes in speech may correspond to positive or negative words in text, and facial expressions in vision may reflect the intensity of emotion in speech. These inter-modal relationships are crucial for accurate emotion recognition. However, because data from different modalities have different feature spaces and expressions, direct fusion often leads to information loss or increased noise. Therefore, how to effectively align and fuse information from different modalities so that they can work together is the key to improving the accuracy of emotion recognition.

[0004] Current multimodal conversational emotion recognition algorithms primarily focus on Transformer-based attention mechanisms and graph-based interaction modeling. These algorithms still face significant technical bottlenecks, stemming from systemic flaws in how existing models model conversation structure and multimodal interaction relationships. First, traditional methods overly rely on explicit rules to construct conversation graph structures. While these methods can capture shallow interaction relationships, they overlook the underlying semantic connections within a single modality. Second, cross-modal interaction modeling tends to be oversimplified. Most methods employ a unified connection model to handle relationships between modalities, ignoring the inherent differences in modal characteristics. Furthermore, existing methods fail to effectively leverage the emotional characteristics of conversation histories. Emotion labels are often simplified into discrete symbolic inputs, lacking semantic representation and difficulty conveying the continuity of emotional evolution. This limits the model's ability to capture contextual emotional dependencies.

[0005] The difficulty in addressing these shortcomings lies in the following: First, the dynamic nature and complexity of implicit relationships make traditional rules or static graph structures difficult to adapt. Implicit associations within a single modality must be dynamically inferred based on semantic similarity, while cross-modal relationships must account for modal heterogeneity, placing higher demands on the model's adaptive modeling capabilities. Second, the continuous expression of emotional semantics and cross-modal alignment require refined strategy design. The semantic transformation of historical emotion labels must not only preserve emotion category information but also encode their contextual relevance. Cross-modal comparative learning, on the other hand, must balance commonalities and differences between modalities to avoid over-alignment that overwhelms modal features. Third, there is a conflict between the efficiency and computational complexity of fusion for multimodal heterogeneous data. Dynamically constructing a hybrid relationship graph requires considering multi-level interactions within and across modalities. Without an efficient graph update mechanism, this can lead to a surge in computing resources and a decrease in model generalization. These challenges collectively limit the ability of existing methods to deeply analyze emotion in complex conversational scenarios. Summary of the Invention

[0006] An embodiment of the present invention provides a method and device for multimodal conversation emotion recognition, which achieves more accurate emotion prediction by extracting multimodal features, inferring implicit relationships, constructing a hybrid relationship graph, and multimodal semantic alignment.

[0007] An embodiment of the present invention provides a multimodal conversation emotion recognition method, comprising:

[0008] Single feature vectors are extracted and fused from the input multimodal dialogue sequence for emotion recognition to obtain an initial multimodal feature vector. This initial multimodal feature vector is semantically aligned based on historical sentiment text features and contrastive learning to obtain the initial contrastive learning loss for each sentence in each modality.

[0009] According to the constructed similarity function, the implicit association relationship between the sentences included in each modality is determined, and according to the implicit association relationship and the explicit relationship between the sentences included in each modality, a mixed relationship graph represented by nodes is obtained; the attention weights between the nodes are determined according to the mixed relationship graph, and the nodes are updated according to the attention weights to obtain updated nodes; the updated multimodal feature vectors are semantically aligned according to contrastive learning to obtain updated contrastive learning losses for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vectors of each sentence, and the updated nodes represent the updated multimodal feature vectors of each sentence;

[0010] Feature fusion is performed on the updated multimodal feature vector to obtain the aggregated multimodal feature vector of each sentence in the conversation history. The representation after weighted fusion with visual and audio modal information is determined based on the multi-head attention mechanism. The representation after weighted fusion with visual and audio modal information is mapped to the vocabulary through a linear network to obtain the predicted word generation probability of the target sentence. The difference loss value between the predicted word probability and the actual word probability is determined based on the cross-entropy loss function to obtain the loss function.

[0011] An embodiment of the present invention provides a multimodal conversation emotion recognition device, comprising:

[0012] A unit is obtained for extracting and fusing single feature vectors from the input multimodal dialogue sequence requiring emotion recognition to obtain an initial multimodal feature vector. This initial multimodal feature vector is semantically aligned based on historical sentiment text features and contrastive learning to obtain the initial contrastive learning loss for each sentence in each modality.

[0013] An updating unit is configured to determine, based on the constructed similarity function, an implicit association relationship between the sentences included in each modality, and obtain a mixed relationship graph represented by nodes based on the implicit association relationship and the explicit relationship between the sentences included in each modality; determine the attention weights between the nodes based on the mixed relationship graph, and update the nodes based on the attention weights to obtain updated nodes; and semantically align the updated multimodal feature vectors based on contrastive learning to obtain an updated contrastive learning loss for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vector of each sentence, and the update nodes represent the updated multimodal feature vector of each sentence;

[0014] A determination unit is used to perform feature fusion on the updated multimodal feature vector to obtain the aggregated multimodal feature vector of each sentence in the conversation history, determine the representation after weighted fusion with visual and audio modal information based on the multi-head attention mechanism, map the representation after weighted fusion with visual and audio modal information to the vocabulary through a linear network, obtain the predicted word generation probability of the target sentence, determine the difference loss value between the predicted word probability and the actual word probability based on the cross-entropy loss function, and obtain the loss function.

[0015] An embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any one of the above-mentioned multimodal dialogue emotion recognition methods.

[0016] An embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein. When the computer program is executed by a processor, the processor executes any one of the multimodal dialogue emotion recognition methods described above.

[0017] The embodiment of the present invention provides a multimodal conversation emotion recognition method and device. By integrating text, audio and visual modal information, the method can capture richer emotional clues and improve the accuracy of emotion recognition compared with a single modality. It effectively utilizes the emotional information contained in historical conversations, structures and encodes emotional tags, integrates historical emotions into model training, uses contrastive learning for semantic alignment multiple times, and uses historical emotions to align context and emotional expression, thereby enhancing the emotional semantic consistency between different modal features, deepening the model's attention to emotional clues, improving subsequent emotion predictions, and fully tapping the value of emotional features in the conversation history. value; enhance the mining of relationships within and between modalities, consider the adjacent relationships between sentences, the same speaker and potential implicit relationships, infer the implicit relationships within the modality by calculating the correlation between nodes, supplement the lack of explicit relationships, and construct a more complete and reasonable dialogue graph structure, which promotes the fusion of multimodal information and enables the model to better understand the complex emotional associations between modalities; dynamic fusion is flexible, and the multimodal dynamic fusion mechanism in the decoding stage uses operations such as multi-head attention to reasonably allocate weights, adaptively utilize information from each modality, improve the emotional prediction effect in different scenarios, and ultimately generate and output intuitive emotional prediction results, which is convenient for practical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a multimodal conversation emotion recognition method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of a multimodal conversation emotion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Figure 1 A flowchart of a multimodal conversation emotion recognition method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps:

[0023] Step 101: Extract and fuse single feature vectors from an input multimodal conversation sequence requiring emotion recognition to obtain an initial multimodal feature vector. Semantically align the initial multimodal feature vector based on historical sentiment text features and contrastive learning to obtain an initial contrastive learning loss for each sentence in each modality.

[0024] Step 102: Determine the implicit association relationship between the sentences included in each modality based on the constructed similarity function, and obtain a mixed relationship graph represented by nodes based on the implicit association relationship and the explicit relationship between the sentences included in each modality; determine the attention weights between the nodes based on the mixed relationship graph, update the nodes based on the attention weights to obtain updated nodes; perform semantic alignment on the updated multimodal feature vectors based on contrastive learning to obtain an updated contrastive learning loss for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vector of each sentence, and the updated nodes represent the updated multimodal feature vector of each sentence;

[0025] Step 103: Perform feature fusion on the updated multimodal feature vector to obtain an aggregated multimodal feature vector for each sentence in the conversation history. Determine a representation after weighted fusion with the visual and audio modal information based on a multi-head attention mechanism. Map the representation after weighted fusion with the visual and audio modal information to a vocabulary through a linear network to obtain the predicted word generation probability of the target sentence. Determine the difference loss value between the predicted word probability and the actual word probability based on a cross-entropy loss function to obtain a loss function.

[0026] The method provided by the embodiment of the present invention is described below in conjunction with specific embodiments. It should be noted that the execution subject of the method is a central processing unit.

[0027] In step 101, a single feature vector including text, speech, and vision is extracted from a multimodal dialogue sequence for emotion recognition according to input needs, and then early feature fusion is performed on the single feature vector to obtain an initial multimodal feature vector.

[0028] It should be noted that in actual applications, a multimodal dialogue sequence may include multiple rounds of dialogue, and each round of dialogue may include multiple sentences. Therefore, when extracting single feature vectors, it is necessary to obtain the single text feature vector, single visual feature vector and single audio feature vector of each sentence separately, that is, the single feature vector includes a single text feature vector, a single visual feature vector and a single audio feature vector.

[0029] Specifically, the pre-trained T5 encoder is used to extract single text feature vectors. represents the single text feature vector of the i-th sentence in the I-th round of dialogue; for example, the single text feature vector of the first sentence in the first round of dialogue (I=1) is The single text feature vector of the first sentence in the second round of dialogue (I=2) is The single text feature vector of the first sentence in the third round of dialogue (I=3) is

[0030] Furthermore, the deep neural network DenseNet is used to extract single visual feature vectors. represents the single visual feature vector of the i-th sentence in the I-th round of dialogue; for example, the single visual feature vector of the first sentence in the first round of dialogue (I=1) is The single visual feature vector of the first sentence in the second round of dialogue (I=2) is The single visual feature vector of the first sentence in the third round of dialogue (I=3) is

[0031] Use OpenSmile tool to extract single audio feature vector, represents the audio features of the i-th sentence in the I-th round of dialogue; for example, the single audio feature vector of the first sentence in the first round of dialogue (I=1) is The single audio feature vector of the first sentence in the second round of dialogue (I=2) is The single audio feature vector of the first sentence in the third round of dialogue (I=3) is

[0032] Furthermore, the early fusion in the embodiment of the present invention is also based on sentences, and early fusion is performed on the single text feature vector, single visual feature vector and single audio feature vector included in each sentence, and then the initial multimodal feature vector corresponding to each sentence is obtained in turn. The initial multimodal feature vector here includes the initial text modal feature vector, the initial visual modal feature vector and the initial audio modal feature vector.

[0033] Specifically, for the text feature fusion of the i-th sentence in the i-th round of dialogue, the single visual feature vector and the single audio feature vector are first concatenated, and then the fused initial multimodal feature vector is calculated through the cross-attention mechanism. The initial multimodal feature vectors are the initial text modality feature vector, the initial visual modality feature vector, and the initial audio modality feature vector. The specific formula is:

[0034]

[0035]

[0036] in, represents the concatenation of the single visual feature vector and the single audio feature vector of the i-th sentence in the i-th round of dialogue, represents the single visual feature vector of the i-th sentence in the i-th round of dialogue, represents the single audio feature vector of the i-th sentence in the i-th round of dialogue, || represents vector concatenation, represents the initial text modality feature vector of the i-th sentence in the i-th round of dialogue, Represents the single text feature vector of the i-th sentence in the i-th round of dialogue, represents the initial visual modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial audio modal feature vector of the i-th sentence in the I-th round of dialogue, I represents the dialogue round, i represents the sentence sequence in each round of dialogue, and CS-Attention is the abbreviation of Cross-SemanticAttention, which means cross-semantic attention mechanism in Chinese.

[0037] It should be noted that, in the embodiments of the present invention, a single feature vector refers to a feature vector extracted from a single modality of data (such as an image, text, audio, etc.), and the "single" only describes the feature information of the modality of data. The fused initial multimodal feature vector is a feature vector obtained by fusing single feature vectors from different modalities (such as through a certain fusion algorithm or model). The initial multimodal feature vector integrates the information of multiple single feature vectors and can more comprehensively represent the data. It is usually used in multimodal tasks to fully utilize the complementary information between different modalities to improve model performance.

[0038] Furthermore, the initial multimodal feature vectors are semantically aligned based on historical sentiment text features and contrastive learning to obtain the initial contrastive learning loss of each sentence included in each modality.

[0039] In this embodiment of the present invention, the initial contrastive learning loss is determined by the following formula:

[0040]

[0041] Among them, L PSA represents the initial contrastive learning loss of the multimodal dialogue sequence, n represents the number of dialogue rounds, I represents the dialogue round, i represents the sentence number in each dialogue round, j represents the sentence number in each dialogue round, represents the η modal feature vector of the ith sentence in the I-th round of dialogue, η∈{t,a,v}, t represents text, a represents vision, v represents audio, Represents the feature vector corresponding to the historical sentiment label of the i-th sentence in the i-th round of dialogue, represents the single visual feature vector of the jth sentence in the I-th round of dialogue, sim(·) represents the cosine similarity between two vectors, which is used to measure the similarity between vectors, and τ represents the temperature parameter.

[0042] It should be noted that when determining the initial contrastive learning loss for a multimodal dialogue sequence, the contribution of each sentence in each modality to the initial contrastive learning loss is calculated separately for each modality. For example, if the input multimodal dialogue sequence includes two rounds of dialogue, each containing two sentences, and a single feature vector and an initial multimodal feature vector have been obtained, the following describes how to determine the initial contrastive learning loss for the multimodal dialogue sequence:

[0043] Step 101-1, in the first round of dialogue (I=1), calculate each sentence pair L under the modality PSA Contributions:

[0044] If the current mode is text (η=t): for sentence 1 (i=1), calculate and Assumptions and 0.3, substituting into formula (3) can be used to calculate L PSA Contributions:

[0045]

[0046] In the formula, Indicates the i-th sentence in the I-th round of dialogue in the text mode to L PSA Contribution, Represents the feature vector corresponding to the historical sentiment label of the first sentence in the first round of dialogue.

[0047] For sentence 2 (i=2), calculate and Assumptions and Substituting into the formula we can get the contribution of this term:

[0048]

[0049] If the current modality is vision (η=a): for sentence 1 (i=1), calculate and Assumptions and Substituting into the formula we can get the contribution of this term:

[0050]

[0051] For sentence 2 (i=2), calculate and Assumptions and Substituting into the formula we can get the contribution of this term:

[0052]

[0053] If the current modality is visual (η=v): for sentence 1 (i=1), calculate and Assumptions and 0.1, and the contribution of this term can be obtained by substituting it into the formula:

[0054]

[0055] For sentence 2 (i=2), calculate and Assumptions and Substituting into the formula we can get the contribution of this term:

[0056]

[0057] Step 101-2, in the second round of dialogue (I=2), calculate the L of each sentence pair under the modality PSA Contributions:

[0058] If the current mode is text (η=t): for sentence 1 (i=1), we get For sentence 2 (i=2), we get

[0059] If the current modality is vision (η=a): for sentence 1 (i=1), we get For sentence 2 (i=2), we get

[0060] If the current mode is audio (η=v): for sentence 1 (i=1), we get For sentence 2 (i=2), we get

[0061] Step 101-3, summarize and obtain the initial contrastive learning loss L of the multimodal dialogue sequence PSA : When I=1, the contribution value of each sentence included in each mode is added: (-0.4)+(-0.2)+(-0.6)+

[0062] (-0.4)+(-0.8)+(-0.2)=-2.6; when I=2, the contribution value of each sentence in each modality is added: 0.4+0.2+0.6+0.4+0.8+0.6=3, then the initial contrastive learning loss of the multimodal dialogue sequence is: L PSA =-2.6+3=0.4.

[0063] In step 102, based on the constructed similarity function, the implicit association relationship between the sentences in each modality is determined, which specifically includes the following:

[0064] 1) Based on the semantic similarity of the initial text modality feature vectors of different sentences, the semantic similarity of the initial visual modality feature vectors of different sentences, and the semantic similarity of the initial audio modality feature vectors of different sentences, a similarity function and similarity score between the initial text modality feature vectors, a similarity function and similarity score between the initial visual modality feature vectors, and a similarity function and similarity score between the initial audio modality feature vectors are determined based on a single-layer feedforward neural network;

[0065] Based on the initial multimodal feature vectors under modality and the weight matrices corresponding to the initial multimodal feature vectors, a similarity function and similarity score between the initial multimodal feature vectors are determined based on a single-layer feedforward neural network. Since the initial multimodal feature vectors include the initial text modality feature vectors, the initial visual modality feature vectors, and the initial audio modality feature vectors, the similarity function and similarity score between the initial text modality feature vectors, the similarity function and similarity score between the initial visual modality feature vectors, and the similarity function and similarity score between the initial audio modality feature vectors can be obtained using the following formulas, respectively.

[0066]

[0067] in, represents the similarity score between the initial text modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial visual modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial audio modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the transposition of the weighted sentence corresponding to the initial text modality feature vector, represents the transpose of the weight matrix corresponding to the initial visual modality eigenvector, represents the transpose of the weighted sentence corresponding to the initial audio modal feature vector, represents the initial text modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial text modality feature vector of the jth sentence in the I-th round of dialogue, represents the initial visual modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial visual modality feature vector of the jth sentence in the I-th round of dialogue, represents the initial audio modal feature vector of the i-th sentence in the i-th round of dialogue, represents the initial audio modal feature vector of the jth sentence in the I-th round of dialogue, mask represents the mask matrix, which is used to prevent reverse dependency, and || represents vector concatenation.

[0068] 2) Determine the relative similarity score of the initial text modality feature vector using the following formula (5-1) based on the similarity score of the initial text modality feature vector. In practical applications, if the relative similarity score of the initial text modality feature vector satisfies the text judgment condition shown in the following formula (6-1), the turn number and the sequence number of different sentences included in the round of dialogue are added to the implicit relationship set of the text modality. Formulas (5-1) and (6-1) are as follows:

[0069]

[0070] 3) Similarly, the relative similarity score of the initial visual modality feature vector is determined by the following formula (5-2) based on the similarity score of the initial visual modality feature vector. If the relative similarity score of the initial visual modality feature vector satisfies the visual judgment condition shown in the following formula (6-2), the turn number and the sequence number of different sentences included in the round of dialogue are added to the implicit relationship set of the visual modality. Formula (5-2) and Formula (6-2) are as follows:

[0071]

[0072] 4) Similarly, the relative similarity score of the initial audio modal feature vector is determined by the following formula (5-3) based on the similarity score of the initial audio modal feature vector. If the relative similarity score of the initial audio modal feature vector satisfies the audio judgment condition shown in formula (6-3), the turn number and the sequence number of different sentences included in the round of dialogue are added to the implicit relationship set of the audio modality. Formula (5-3) and formula (6-3) are as follows:

[0073]

[0074] in, represents the relative similarity score of the initial text modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the relative similarity score of the initial visual modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the relative similarity score of the initial audio modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial text modality feature vectors of the kth sentence and the jth sentence in the I-th round of dialogue, represents the similarity score between the initial visual modality feature vectors of the kth sentence and the jth sentence in the I-th round of dialogue, represents the similarity score between the initial audio modality feature vectors of the k-th sentence and the j-th sentence in the I-th round of dialogue, and m represents the total number of sentences in each round of dialogue.

[0075] For example, if the input multimodal dialogue sequence includes two rounds of dialogue, each containing two sentences, and a single feature vector, initial multimodal feature vectors, and initial contrastive learning loss for the multimodal dialogue sequence have been obtained, the following describes how to determine the similarity score and relative similarity score between the initial multimodal feature vectors.

[0076] Step 102-1, calculate the similarity score between the initial multimodal feature vectors in the first round (I=1):

[0077] If the current modality is text (η=t), the similarity score of the initial text modality feature vector between sentence i and sentence j is calculated by formula (4-1) When i=1, j=2, formula (4-1) can be:

[0078] Here we assume that

[0079] When the current modality is visual (η=a), the similarity score of the initial visual modality feature vector between sentence i and sentence j is calculated as When i=1, j=2, formula (4-2) can be:

[0080] Here we assume that

[0081] When the current modality is audio (η=v), the similarity score of the initial audio modality feature vector between sentence i and sentence j is calculated as When i=1, j=2, formula (4-3) can be:

[0082] Here we assume that

[0083] Step 102-2, calculate the implicit relationship between each modality in the first round of dialogue (I=1):

[0084] When the current modality is text, the relative similarity score of the initial text modality feature vector is determined by formula (5-1) When i=1, j=2, formula (5-1) can be:

[0085] here The calculation method is the same as step 102-1. Assuming its value is 0.3, then According to the text judgment conditions, because Therefore, (I,i,j)=(1,1,2) can be added to the implicit relation set IC of the text modality t , that is, IC t ={(1,1,2)}.

[0086] When the current modality is vision, the relative similarity score of the initial visual modality feature vector is determined by formula (5-2) When i=1, j=2, formula (5-2) can be:

[0087] Assumptions The value is 0.2, then: because Therefore, (I,i,j)=(1,1,2) can be added to the implicit relation set IC of the visual modality a , that is, IC a ={(1,1,2)}.

[0088] When the current modality is audio, the relative similarity score of the initial audio modality feature vector is determined by formula (5-3) When i=1, j=2, formula (5-3) can be:

[0089] Assumptions The value is 0.3, then: because Therefore, (I,i,j)=(1,1,2) can be added to the implicit relation set IC of audio modality v , that is, IC v ={(1,1,2)}.

[0090] Step 102-3, calculate the similarity score between the multimodal feature vectors in the second round of dialogue (I=2): The method here can refer to step 102-1. Assume that the similarity score of the initial text modal feature vector between sentences 1 and 2 in the second round is: The similarity score of the initial visual modality feature vector between sentences 1 and 2 in round 2 is: The similarity score of the initial audio modality feature vector between sentences 1 and 2 in round 2 is

[0091] Further, referring to step 102-2, assuming that the relative similarity scores of the initial text modality feature vectors between sentence 1 and sentence 2 in the second round are obtained,

[0092] The relative similarity score of the initial visual modality feature vectors between sentences 1 and 2 in round 2 is The relative similarity score of the initial audio modality feature vectors between sentences 1 and 2 in round 2 is And all meet the text judgment condition, visual judgment condition and audio judgment condition, then (I,i,j)=(2,1,2) is added to the implicit relationship set IC of the text modalityt , that is, IC t ={(1,1,2),(2,1,2)}; add (I,i,j)=(2,1,2) to the implicit relation set IC of the visual modality a , that is, IC a ={(1,1,2),(2,1,2)}; add (I,i,j)=(2,1,2) to the implicit relation set IC of the audio modality v , that is, IC v ={(1,1,2),(2,1,2)}.

[0093] Based on the above content, it can be determined that in the first round of conversation, the similarity scores between the initial text modal feature vectors, the similarity scores between the initial visual modal feature vectors, and the similarity scores between the initial audio modal feature vectors are 0.6, 0.4, and 0.5, respectively, and the implicit relationship sets are ICt = (1, 1, 2), ICa = (1, 1, 2), and ICv = (1, 1, 2); in the second round of conversation, the similarity scores between the initial text modal feature vectors, the similarity scores between the initial visual modal feature vectors, and the similarity scores between the initial audio modal feature vectors are 0.55, 0.35, and 0.45, respectively, and the implicit relationship sets are ICt = (1, 1, 2), (2, 1, 2), ICa = (1, 1, 2), (2, 1, 2), and ICv = (1, 1, 2), (2, 1, 2).

[0094] In step 102, after determining the implicit association relationship between the sentences included in each modality, a mixed relationship graph G = (V, E) represented by nodes can be obtained based on the implicit relationship association and the explicit relationship between the sentences included in each modality, where the node is the initial multimodal feature vector of each sentence.

[0095] Specifically, V represents a node set, which contains the initial multimodal feature vectors of each sentence in each round of dialogue. The initial multimodal feature vectors here include the initial text modality feature vector, the initial visual modality feature vector and the initial audio modality feature vector. For example, for the first round of dialogue (I=1), the initial text modality feature vector of sentence 1 is Initial visual modality feature vector

[0096] Initial audio modal eigenvector And the initial text modality feature vector of sentence 2 Initial visual modality feature vector Initial audio modal eigenvector are all nodes in V.

[0097] E represents an edge set. If sentences 1 and 2 are adjacent and spoken by the same person in the first round of conversation (I = 1), then an edge connects the corresponding nodes. If, based on the implicit association, there is an implicit relationship between (1, 1, 2) in the text modality, then the corresponding initial text modality feature vectors are connected.

[0098] Furthermore, the attention weights between nodes (initial multimodal feature vectors) are determined according to the hybrid relationship graph, and each node is updated according to the attention weights to obtain an updated node (updated multimodal feature vector). Specifically, the node update can be performed using the following formula:

[0099]

[0100] in, represents the updated η modal feature vector of the i-th sentence in the l-th layer graph attention network in the i-th round of dialogue, represents the attention weight of the updated η modal feature vector of the i-th sentence and the j-th sentence in the l-th layer graph attention network in the I-th round of dialogue, represents the learnable weight matrix for updating the η modal feature vector in the l-th layer graph attention network, represents the updated η modal feature vector of the i-th sentence in the l-1 layer graph attention network in the I-th round of dialogue, N I,i Represents the set of neighbor nodes of the i-th sentence in the i-th round of dialogue.

[0101] It should be noted that It reflects the attention degree of the i-th sentence to its neighbor node j. The attention weights of all neighbor nodes j representing the i-th sentence are summed. During the calculation process, the updated η modal feature vector in the l-1th layer graph attention network is used for calculation, so it is calculated based on the updated η modal feature vector before the update (l-1th).

[0102] In the embodiment of the present invention, since the initial multimodal feature vector includes the initial text modal feature vector, the initial visual modal feature vector and the initial audio modal feature vector, updating the multimodal feature vector also includes updating the text modal feature vector, updating the visual modal feature vector and updating the audio modal feature vector, so formula (7) can be decomposed in sequence as follows:

[0103]

[0104] in, Indicates that the i-th sentence in the I-th round of dialogue updates the text modality feature vector in the l-th layer graph attention network, Indicates that the i-th sentence in the I-th round of dialogue updates the visual modality feature vector in the l-th layer graph attention network, Indicates that the i-th sentence in the I-th round of dialogue updates the audio modality feature vector in the l-th layer of the graph attention network, N I,i Represents the set of neighbor nodes of the i-th sentence in the i-th round of dialogue; represents the attention weights of updating the text modality feature vector in the l-layer graph attention network for the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the attention weights of updating the visual modality feature vector in the l-layer graph attention network for the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the attention weights of updating the audio modality feature vector in the l-layer graph attention network for the i-th sentence and the j-th sentence in the I-th round of dialogue; represents the learnable weight matrix for updating the text modality feature vector in the l-th layer graph attention network, represents the learnable weight matrix for updating the visual modality feature vector in the l-th layer graph attention network, represents the learnable weight matrix for updating the audio modality feature vector in the l-th layer graph attention network; σ(·) represents the sigmoid function.

[0105] For example, assume that a 2-layer (l=1,2) graph attention network is updated.

[0106] Layer 1 (l=1): Sentence 1 (I=

[0107] 1, i=1) as an example: determine the neighbor node set N 1,1 , assuming its neighbor nodes are Calculating attention weights The calculation of attention weights can usually be implemented through a learnable function, for example: in, represents the attention weight matrix of the first layer text modality, It means concatenating two vectors. represents the updated text modality feature vector of the first sentence in the first round of dialogue in the 0th layer of the graph attention network (it can also be called the initial text modality feature vector of the first sentence in the first round of dialogue), represents the updated text modality feature vector of the second sentence in the first round of dialogue in the 0th layer graph attention network, assuming is the learnable weight matrix of the first-layer text modality, then the updated text modality feature vector of the first sentence in the first round of dialogue in the first-layer graph attention network is expressed as:

[0108] Similarly, we can calculate the updates of other sentences and modalities in the first round of dialogue and all sentences and modalities in the second round of dialogue at the first layer.

[0109] The second layer (l=2) is composed of sentence 1 (I=

[0110] 1, i=1) as an example, the updated text modality after the first layer update is expressed as and

[0111] Recalculate attention weights Assumptions is the learnable weight matrix of the second-layer text modality, then the updated text modality feature vector of the first sentence in the first round of dialogue in the second-layer graph attention network is expressed as:

[0112] In this way, multi-layer graph attention network updates are performed on all rounds, all sentences, and all modalities, thereby enhancing feature fusion within the modality.

[0113] In this embodiment of the present invention, the initial multimodal feature vector has been updated to produce an updated multimodal feature vector. Compared to the initial multimodal feature vector, the updated multimodal feature vector incorporates more conversational context and intra-modal information. Subsequent tasks will be performed based on these updated multimodal feature vectors.

[0114] Furthermore, the updated multimodal feature vectors are semantically aligned based on contrastive learning to obtain the updated contrastive learning loss for each sentence in each modality, which is determined by the following formula:

[0115]

[0116] Among them, L MSA represents the updated contrastive learning loss of each sentence, n represents the number of dialogue rounds, I represents the dialogue round, i represents the sequence number of sentences in each dialogue round, j represents the sequence number of sentences in each dialogue round, represents the updated η modal feature vector of the i-th sentence in the i-th round of dialogue at the l-th layer, represents the updated ξ modal feature vector of the ith sentence in the ith round of dialogue at layer l, η∈{t,a,v},ξ∈{t,a,v}, t represents text, a represents vision, v represents audio, sim(·) represents the cosine similarity between two vectors, which is used to measure the similarity between vectors, τ represents the temperature parameter, and l represents the number of layers included in the graph attention network.

[0117] In embodiments of the present invention, a multimodal conversation sequence can include multiple rounds of dialogue, each of which may consist of a single sentence or multiple sentences. Therefore, when dealing with multiple sentences, the sentences can be arranged in a natural chronological order. The method provided in embodiments of the present invention aims to predict the emotion of a target sentence in a multimodal conversation sequence. Therefore, all sentences preceding the target sentence in the conversation can be collectively referred to as the conversation history and are required as input to predict the emotion of the target sentence.

[0118] In the following, we assume that the target sentence is the gth sentence in the conversation, then the sentences from 1 to g-1 are the conversation history.

[0119] In step 103, the updated text modality feature vectors of each sentence in the conversation history are first aggregated to obtain an aggregated updated text modality feature vector; the updated audio modality feature vectors of each sentence in the conversation history are aggregated to obtain an aggregated updated audio modality feature vector; and the updated visual modality feature vectors of each sentence in the conversation history are aggregated to obtain an aggregated updated visual modality feature vector. This is specifically expressed by the following formula:

[0120]

[0121]

[0122] in, represents the aggregated updated text modality feature vector, represents the aggregated updated visual modality feature vector, represents the aggregated updated audio modal feature vector, represents the updated text modality feature vector of the first sentence in the dialogue history at level l, represents the updated visual modality feature vector of the first sentence in the dialogue history at layer l, represents the updated audio modality feature vector of the first sentence in the conversation history at layer l, represents the updated text modality feature vector of the g-1th sentence in the conversation history at level l, represents the updated visual modality feature vector of the g-1th sentence in the dialogue history at layer l, represents the updated audio modality feature vector of the g-1th sentence in the conversation history at layer l, and || represents vector concatenation.

[0123] Furthermore, the aggregated updated visual modality feature vector and the aggregated updated audio modality feature vector are merged to obtain the aggregated updated visual and audio modality feature vector shown below:

[0124]

[0125] in, represents the aggregation update of visual and audio modality feature vectors, and || represents vector concatenation.

[0126] For example, if a multimodal dialogue sequence includes three rounds of dialogue, the first round of dialogue has 2 sentences, the second round of dialogue has 1 sentence, and the third round of dialogue has 3 sentences, if the target sentence is sentence 3 in the third round of dialogue, that is, g = 6, it is the 6th sentence from the perspective of the entire dialogue, then in chronological order, the dialogue history is the first 5 sentences, that is, 2 sentences in the first round of dialogue, 1 sentence in the second round of dialogue, and the first 2 sentences in the third round of dialogue. Then, according to the above formulas (9-1), (9-2) and (9-3), the updated text modal feature vectors, updated audio modal feature vectors, and updated visual modal feature vectors of these 5 sentences are aggregated respectively to obtain the aggregated updated text modal feature vector in the dialogue history Aggregate and update the visual modality feature vector And aggregate to update the audio modal feature vector

[0127] Then, the aggregated updated visual modality feature vector and the aggregated updated audio modality feature vector are merged to obtain the aggregated updated visual and audio modality feature vector.

[0128] Furthermore, based on the multi-head attention mechanism, the aggregated updated text modality feature vector in the conversation history is weighted and integrated into the decoder's input feature vector to facilitate the prediction of subsequent words. The formula is as follows:

[0129]

[0130] Among them, H DC Denotes the feature vector of the decoder input, H WDt It represents the representation after weighted fusion with the dialogue history text modality, and MultiHead-Attention represents the multi-head attention mechanism.

[0131] Furthermore, based on the multi-head attention mechanism, the updated visual and audio modality feature vectors aggregated in the conversation history are selectively incorporated into the decoder input feature vector, thereby obtaining a representation that is weighted and fused with the visual and audio modality information. Here, the decoder input feature vector is a representation that is weighted and fused with the text modality of the conversation history, as shown in the following formula:

[0132]

[0133] Among them, H WDtav Representation is a weighted fusion of visual and audio modality information.

[0134] Furthermore, the representation after weighted fusion with visual and audio modal information is mapped to the vocabulary through a linear network. The probability of generating the predicted word in the kth step is obtained by the following formula. The word with the highest probability is added to the text to be predicted to prepare for the generation of sentiment prediction text.

[0135] P voc (y k )=Softmax(W s H WDtav,k ) (11)

[0136] Among them, P voc (y k ) represents the probability distribution of the k-th step predicted word in the vocabulary, H WDtav,k represents the representation after the k-th step weighted fusion with visual and audio modality information, W s Represents a trainable parameter whose dimension is the vocabulary size (assuming it is V) multiplied by H WDtav,k Dimensions. By WDtav,k Perform matrix multiplication and then pass it through the Softmax function to convert the output into a probability distribution over the vocabulary, so that the sum of the probabilities of all words is 1. For example, if the vocabulary contains words such as "happy", "sad", and "calm", the generation probability value corresponding to each word is calculated.

[0137] For example, for formula (11), assume that the vocabulary size V = 5, that is, there are 5 words in the vocabulary: "excited", "sad", "surprised", "angry", and "happy". WDtav,k The dimension is 10, W s Is a 5×10 trainable parameter matrix. After matrix multiplication and Softmax function processing, we get P voc (y k ) is a 5-dimensional vector, such as P voc (y k )=

[0138] [0.2, 0.1, 0.3, 0.25, 0.15], which means that the probability of generating the word "excited" in the vocabulary for the kth step is 0.2, the probability of generating "sad" is 0.1, and so on. The word with the highest probability is "surprised" (with a probability of 0.3), so "surprised" is added to the text to be predicted to prepare for generating the emotion prediction text.

[0139] Furthermore, after determining the predicted word generation probability of each sentence, if the actual word probability is known, the cross entropy loss function is used to determine the difference loss value between the predicted word probability and the actual word probability through the following formula:

[0140]

[0141] Among them, L s Represents the difference loss value between the predicted word probability and the actual word probability, P voc (y k ) The kth step predicts the probability distribution of words in the vocabulary.

[0142] For example, suppose that the predicted word generation probability P of a sentence has been determined voc (y k ), for example, for a certain step k, P voc (y k )=[0.1,0.6,0.2,0.1](assuming the vocabulary has 4 words), and the probability of the actual word is known, assuming the probability distribution corresponding to the actual word is [0,1,0,0](indicating that the actual word is the second word). According to formula (12), here k represents only one step. So L s =-log(0.6)≈0.51, that is, the difference loss value between the predicted word probability and the actual word probability is about 0.51.

[0143] It should be noted that, in the embodiment of the present invention, a weighted loss function is used to measure the difference between the prediction method and the actual situation, and the weighted loss is expressed as:

[0144] L=L s +λ(L PSD +L MSA ) (13)

[0145] Here, λ represents the loss weight (which can be determined to be 0.001 in the embodiment of the present invention).

[0146] Finally, by identifying specific placeholders pre-set in the text, information related to sentiment prediction is accurately extracted from the generated text, and the sentiment prediction result is output.

[0147] In summary, the embodiments of the present invention provide a multimodal conversation emotion recognition method and device. By integrating text, audio and visual modal information, the method can capture richer emotional clues and improve the accuracy of emotion recognition compared with a single modality. It effectively utilizes the emotional information contained in historical conversations, structures and encodes emotional tags, integrates historical emotions into model training, uses contrastive learning for semantic alignment multiple times, uses historical emotions to align context and emotional expression, enhances the emotional semantic consistency between different modal features, deepens the model's attention to emotional clues, improves subsequent emotion prediction, and fully mines the emotional features in the conversation history. It enhances the mining of intra-modal and inter-modal relationships, considers the adjacent relationships between sentences, the same speaker, and potential implicit relationships, and infers implicit relationships within the modality by calculating the correlation between nodes, supplements the lack of explicit relationships, and constructs a more complete and reasonable dialogue graph structure, which promotes the fusion of multimodal information and enables the model to better understand the complex emotional associations between modalities; dynamic fusion is flexible, and the multimodal dynamic fusion mechanism in the decoding stage uses operations such as multi-head attention to reasonably allocate weights, adaptively utilize information from each modality, improve the emotional prediction effect in different scenarios, and ultimately generate and output intuitive emotional prediction results, which is convenient for practical application.

[0148] Based on the same inventive concept, an embodiment of the present invention provides a multimodal conversation emotion recognition device. Since the principle of solving the technical problem of the device is similar to that of a multimodal conversation emotion recognition method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0149] like Figure 2 As shown, the device includes an obtaining unit 201, an updating unit 202 and a determining unit 203.

[0150] An obtaining unit 201 is configured to extract and fuse single feature vectors from an input multimodal conversation sequence requiring emotion recognition to obtain an initial multimodal feature vector, perform semantic alignment on the initial multimodal feature vector based on historical emotion text features and contrastive learning, and obtain an initial contrastive learning loss for each sentence included in each modality;

[0151] An updating unit 202 is configured to determine, based on the constructed similarity function, an implicit association relationship between the sentences included in each modality, and obtain a mixed relationship graph represented by nodes based on the implicit association relationship and the explicit relationship between the sentences included in each modality; determine attention weights between the nodes based on the mixed relationship graph, and update the nodes based on the attention weights to obtain updated nodes; and semantically align the updated multimodal feature vectors based on contrastive learning to obtain an updated contrastive learning loss for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vector of each sentence, and the updated nodes represent the updated multimodal feature vector of each sentence.

[0152] Determination unit 203 is used to perform feature fusion on the updated multimodal feature vector to obtain an aggregated multimodal feature vector for each sentence in the conversation history, determine a representation after weighted fusion with visual and audio modal information based on a multi-head attention mechanism, map the representation after weighted fusion with visual and audio modal information to a vocabulary through a linear network, obtain a predicted word generation probability of the target sentence, determine the difference loss value between the predicted word probability and the actual word probability based on a cross-entropy loss function, and obtain a loss function.

[0153] It should be understood that the units included in the multimodal conversational emotion recognition device described above are merely logical divisions based on the functions implemented by the device. In actual applications, these units can be combined or separated. Furthermore, the functions implemented by the multimodal conversational emotion recognition device provided in this embodiment correspond one-to-one with the multimodal conversational emotion recognition method provided in the above embodiment. A more detailed description of the processing flow implemented by this device has been provided in detail in the first embodiment of the method described above and will not be repeated here.

[0154] Another embodiment of the present invention also provides a computer device, which includes: a processor and a scene database; the scene database is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes each step of a multimodal dialogue emotion recognition method shown in the above method embodiment.

[0155] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer device, the computer device executes each step of a multimodal dialogue emotion recognition method shown in the above method embodiment.

[0156] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multimodal dialogue emotion recognition method, characterized in that: include: Single feature vectors are extracted and fused from the input multimodal dialogue sequence for emotion recognition to obtain an initial multimodal feature vector. This initial multimodal feature vector is semantically aligned based on historical sentiment text features and contrastive learning to obtain the initial contrastive learning loss for each sentence in each modality. According to the constructed similarity function, the implicit association relationship between the sentences included in each modality is determined, and according to the implicit association relationship and the explicit relationship between the sentences included in each modality, a mixed relationship graph represented by nodes is obtained; the attention weights between the nodes are determined according to the mixed relationship graph, and the nodes are updated according to the attention weights to obtain updated nodes; the updated multimodal feature vectors are semantically aligned according to contrastive learning to obtain updated contrastive learning losses for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vectors of each sentence, and the updated nodes represent the updated multimodal feature vectors of each sentence; Feature fusion is performed on the updated multimodal feature vector to obtain the aggregated multimodal feature vector of each sentence in the conversation history. The representation after weighted fusion with visual and audio modal information is determined based on the multi-head attention mechanism. The representation after weighted fusion with visual and audio modal information is mapped to the vocabulary through a linear network to obtain the predicted word generation probability of the target sentence. The difference loss value between the predicted word probability and the actual word probability is determined based on the cross-entropy loss function to obtain the loss function.

2. The method according to claim 1, wherein Single feature vectors are extracted and fused for the input multimodal dialogue sequence that needs to be subjected to emotion recognition to obtain the initial multimodal feature vector, which specifically includes: The multimodal conversation sequence includes multiple rounds of conversation, each round of conversation includes text information, audio information, and visual information, single feature extraction is performed on the text information, audio information, and visual information included in each round of conversation to obtain a single text feature vector, a single audio feature vector, and a single visual feature vector, respectively, and the single text feature vector, the single audio feature vector, and the single visual feature vector are fused based on a cross-attention mechanism to obtain an initial text modality feature vector, an initial visual modality feature vector, and an initial audio modality feature vector, respectively; The initial text modality feature vector, the initial visual modality feature vector, and the initial audio modality feature vector are represented as follows: in, Represents the single text feature vector of the i-th sentence in the i-th round of dialogue, represents the single visual feature vector of the i-th sentence in the i-th round of dialogue, represents the single audio feature vector of the i-th sentence in the i-th round of dialogue, represents the initial text modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial visual modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial audio modal feature vector of the i-th sentence in the i-th round of dialogue, represents the concatenation of the single visual feature vector and the single audio feature vector of the i-th sentence in the I-th round of dialogue, I represents the dialogue round, i represents the sentence number in each round of dialogue, and || represents vector concatenation.

3. The method according to claim 1, wherein The initial contrastive learning loss of each sentence included in each modality is determined by the following formula: The updated contrastive learning loss of each sentence included in each modality is determined by the following formula: The difference loss between the predicted word probabilities and the actual word probabilities is determined by the following formula: Among them, L PSA represents the initial contrastive learning loss of the multimodal dialogue sequence, n represents the number of dialogue rounds, I represents the dialogue round, i represents the sentence number in each dialogue round, j represents the sentence number in each dialogue round, represents the initial η modal feature vector of the ith sentence in the I-th round of dialogue, η∈{t,a,v}, t represents text, a represents vision, v represents audio, Represents the feature vector corresponding to the historical sentiment label of the i-th sentence in the i-th round of dialogue, represents the single visual feature vector of the jth sentence in the Ith round of dialogue, sim(·) represents the cosine similarity between two vectors, which is used to measure the similarity between vectors, and τ represents the temperature parameter; L MSA represents the updated contrastive learning loss of each sentence, represents the updated η modal feature vector of the i-th sentence in the i-th round of dialogue at the l-th layer, represents the updated modal feature vector ξ∈{t,a,v} of the i-th sentence in the i-th round of dialogue at the l-th layer; L s Represents the difference loss value between the predicted word probability and the actual word probability, P voc (y k ) The kth step predicts the probability distribution of the word in the vocabulary, l represents the number of layers of the graph attention network, y k Represents the predicted word at step k.

4. The method according to claim 1, wherein Based on the constructed similarity function, the implicit association relationship between sentences in each modality is determined, including: According to the semantic similarity of the initial text modality feature vectors of different sentences, the semantic similarity of the initial visual modality feature vectors of different sentences, and the semantic similarity of the initial audio modality feature vectors of different sentences, a similarity function and similarity score between the initial text modality feature vectors, a similarity function and similarity score between the initial visual modality feature vectors, and a similarity function and similarity score between the initial audio modality feature vectors are determined based on a single-layer feedforward neural network; Determining a relative similarity score of the initial text modality feature vector based on the similarity score of the initial text modality feature vector; if the relative similarity score of the initial text modality feature vector satisfies a judgment condition, adding the turn number and the sequence number of different sentences included in the conversation to the implicit relationship set of the text modality; Determining a relative similarity score of the initial visual modality feature vector based on the similarity score of the initial visual modality feature vector; if the relative similarity score of the initial visual modality feature vector satisfies a judgment condition, adding the turn number and the sequence number of different sentences included in the conversation to the implicit relationship set of the visual modality; The relative similarity score of the initial audio modal feature vector is determined based on the similarity score of the initial audio modal feature vector. If the relative similarity score of the initial audio modal feature vector meets the judgment condition, the sequence numbers of the turns and different sentences included in the round of dialogue are added to the implicit relationship set of the audio modality.

5. The method according to claim 4, wherein The similarity function between the initial text modality feature vectors, the similarity function between the initial visual modality feature vectors, and the similarity function between the initial audio modality feature vectors are expressed as follows: The relative similarity scores of the initial text modality feature vectors, the relative similarity scores of the initial visual modality feature vectors, and the relative similarity scores of the initial audio modality feature vectors are respectively determined by the following formulas: in, represents the similarity score between the initial text modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial visual modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial audio modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the transposition of the weighted sentence corresponding to the initial text modality feature vector, represents the transpose of the weight matrix corresponding to the initial visual modality eigenvector, represents the transpose of the weighted sentence corresponding to the initial audio modal feature vector, represents the initial text modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial text modality feature vector of the jth sentence in the I-th round of dialogue, represents the initial visual modality feature vector of the i-th sentence in the i-th round of dialogue, represents the initial visual modality feature vector of the jth sentence in the I-th round of dialogue, represents the initial audio modal feature vector of the i-th sentence in the i-th round of dialogue, represents the initial audio modal feature vector of the jth sentence in the I-th round of dialogue, represents the relative similarity score of the initial text modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the relative similarity score of the initial visual modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the relative similarity score of the initial audio modality feature vectors of the i-th sentence and the j-th sentence in the I-th round of dialogue, represents the similarity score between the initial text modality feature vectors of the kth sentence and the jth sentence in the I-th round of dialogue, represents the similarity score between the initial visual modality feature vectors of the kth sentence and the jth sentence in the I-th round of dialogue, represents the similarity score between the initial audio modality feature vectors of the k-th sentence and the j-th sentence in the I-th round of dialogue, m represents the total number of sentences in each round of dialogue, || represents vector concatenation, and mask represents the mask matrix.

6. The method according to claim 1, wherein Determine the attention weights between nodes according to the hybrid relationship graph, and update the nodes according to the attention weights to obtain updated nodes, specifically including: The nodes included in the hybrid relationship graph represent the initial multimodal feature vectors of each sentence, and the update nodes are determined by the following formula: in, represents the updated η modal feature vector of the i-th sentence in the l-th layer graph attention network in the i-th round of dialogue, represents the attention weight of the updated η modal feature vector of the i-th sentence and the j-th sentence in the l-th layer graph attention network in the I-th round of dialogue, represents the learnable weight matrix for updating the η modal feature vector in the l-th layer graph attention network, σ(·) represents the Sigmoid function, and N I,i represents the set of neighbor nodes of the i-th sentence in the i-th round of dialogue, represents the updated η-modal feature vector of the i-th sentence in the l-1-th layer graph attention network in the i-th round of dialogue.

7. The method according to claim 1, wherein The aggregated multimodal feature vector of each sentence in the conversation history is represented as follows: The representation after weighted fusion with visual and audio modality information is expressed as follows: The representation after weighted fusion with text modality information is as follows: The predicted word generation probability of the target sentence is expressed as follows: P voc (y k )=Softmax(W s H WDtav,k ) in, represents the aggregated updated text modality feature vector, represents the aggregated updated visual modality feature vector, represents the aggregated updated audio modal feature vector, represents the updated text modality feature vector of the first sentence in the dialogue history at level l, represents the updated visual modality feature vector of the first sentence in the dialogue history at layer l, represents the updated audio modality feature vector of the first sentence in the conversation history at layer l, represents the updated text modality feature vector of the g-1th sentence in the dialogue history at level l, represents the updated visual modality feature vector of the g-1th sentence in the dialogue history at layer l, represents the updated audio modality feature vector of the g-1th sentence in the conversation history at layer l, H WDtav Represents the weighted fusion of visual and audio modality information, H WDt Representation after weighted fusion of the modality of the dialogue history text, represents the aggregated updated visual and audio modality feature vectors, P voc (y k ) represents the probability distribution of the k-th step predicted word in the vocabulary, W s represents trainable parameters, || represents vector concatenation, H WDtav,k represents the representation after weighted fusion of the k-th step with visual and audio modality information, H DC A feature vector representing the decoder input.

8. A multimodal conversation emotion recognition device, characterized in that: include: A unit is obtained for extracting and fusing single feature vectors from the input multimodal dialogue sequence requiring emotion recognition to obtain an initial multimodal feature vector. This initial multimodal feature vector is semantically aligned based on historical sentiment text features and contrastive learning to obtain the initial contrastive learning loss for each sentence in each modality. An updating unit is configured to determine, based on the constructed similarity function, an implicit association relationship between the sentences included in each modality, and obtain a mixed relationship graph represented by nodes based on the implicit association relationship and the explicit relationship between the sentences included in each modality; determine the attention weights between the nodes based on the mixed relationship graph, and update the nodes based on the attention weights to obtain updated nodes; and semantically align the updated multimodal feature vectors based on contrastive learning to obtain an updated contrastive learning loss for each sentence included in each modality; wherein the nodes represent the initial multimodal feature vector of each sentence, and the update nodes represent the updated multimodal feature vector of each sentence; A determination unit is used to perform feature fusion on the updated multimodal feature vector to obtain the aggregated multimodal feature vector of each sentence in the conversation history, determine the representation after weighted fusion with visual and audio modal information based on the multi-head attention mechanism, map the representation after weighted fusion with visual and audio modal information to the vocabulary through a linear network, obtain the predicted word generation probability of the target sentence, determine the difference loss value between the predicted word probability and the actual word probability based on the cross-entropy loss function, and obtain the loss function.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the multimodal dialogue emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the processor executes the multimodal dialogue emotion recognition method according to any one of claims 1 to 7.

Images