Dialogue-oriented image-text multi-modal sentiment understanding and empathetic reply generation method
By fusing textual and facial image information through a cross-modal attention mechanism, a multimodal emotion modeling method is constructed, which solves the problem of emotion interpretation bias in dialogue systems based on facial images in social network environments and achieves high-quality empathetic response generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CAPITAL NORMAL UNIVERSITY
- Filing Date
- 2026-02-28
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173637A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for dialogue-oriented multimodal image and text emotion understanding and empathetic response generation. Background Technology
[0002] With the widespread application of dialogue systems in key scenarios such as online customer service, psychological support, and social platforms, expectations for their capabilities have expanded from accurate information delivery to higher levels of emotional perception and empathetic interaction. However, current research on empathetic dialogue still primarily relies on the text modality, limited by semantic ambiguity and insufficient emotional expression. Its ability to model implicit expressions, weak emotional signals, and fine-grained emotional states in real-world social contexts is limited, making it difficult to support accurate emotional understanding and high-quality emotional interaction experiences in real-world scenarios. In fact, human emotional expression has a natural multimodal nature; the transmission of emotions depends on the synergistic effect of multiple signals, including language content, tone of voice, facial expressions, gestures, and visual symbols. In comparison, pure text can only carry partial semantic and emotional information, making it difficult to fully depict the intensity and subtle states of emotions.
[0003] In today's social media-dominated online interactive environment, the frequency of use of visual emotional symbols has increased significantly, with emoticons becoming one of the most important and culturally specific carriers of emotion. Especially when conveying subtle social emotions such as guilt, embarrassment, and shame—emotions that are difficult to describe precisely in words—emoticons, with their exaggerated, intuitive, and highly recognizable visual impact, provide a powerful emotional anchor, becoming a key clue to understanding a user's true emotional intent. Because existing methods struggle to fully utilize the emotional information carried by emoticons, emotional misunderstandings are prone to occur in dialogues, leading to mismatched emotional responses. Therefore, how to effectively integrate emoticons with emotional cues in the textual context within a dialogue setting to achieve accurate understanding of subtle emotions and generate empathetic responses is a pressing problem, and also the problem this method aims to solve.
[0004] Existing dialogue emotion recognition and empathic response generation technologies typically employ mechanisms such as context modeling, speaker state tracking, emotion memory, and external knowledge enhancement. However, they remain primarily text-based, failing to adequately capture the increasingly prevalent multimodal emotion expressions in real-world social conversations. While some multimodal emotion recognition research has incorporated multimodal information, its applications are mostly concentrated in face-to-face or video interactions, focusing on traditional modalities like voice and video, with limited systematic modeling of facial expressions widely used in social dialogues. These limitations result in insufficient modeling of the emoji modality in dialogues, consequently impacting fine-grained emotion understanding and empathic response generation.
[0005] For the problems of emotion recognition and empathy generation in dialogue, existing technologies are mainly divided into plain text-based generation methods and traditional multimodal recognition methods.
[0006] In the field of pure text empathy generation, the EmpDG model proposed by Li et al. in "EmpDG: Multi-resolution interactive empathetic dialogue generation" captures subtle emotional changes at the discourse and lexical levels through adversarial learning mechanisms, making it a classic approach. More recently, the IAMM model proposed by Yang et al. in "An iterative-associative memory model for empathetic response generation" designs an iterative associative memory mechanism, dynamically accumulating emotional states across multiple turns of dialogue, thus improving the model's ability to understand context. However, these methods are all limited to a single text modality and cannot utilize visual information for deeper emotional modeling.
[0007] In the field of multimodal emotion computing, existing research mainly focuses on integrating text, speech, and facial video features. For example, DialogueGCN, proposed by Ghosal et al. in "DialogueGCN: A graph convolutional neural network for emotion recognition in conversation," utilizes graph neural networks to capture dialogue dependencies; while InstructERC, proposed by Lei et al. in "InstructERC: Reforming emotion recognition in conversation with multi-task retrieval-augmented large language models," uses large model instructions for fine-tuning to improve emotion inference capabilities. However, it is worth noting that the above methods are mainly aimed at speech and video stream data in face-to-face communication, lacking targeted modeling for graph-text dialogue data in social network environments. Summary of the Invention
[0008] The main objective of this invention is to provide a dialogue-oriented, multimodal image-text emotion understanding and empathic response generation method.
[0009] Another objective of this invention is to propose a dialogue-oriented, multimodal image-text emotion understanding and empathic response generation device.
[0010] The third objective of this invention is to provide a computer device.
[0011] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.
[0012] To achieve the above objectives, a first aspect of the present invention proposes a dialogue-oriented graph-text multimodal emotion understanding and empathic response generation method, comprising:
[0013] S1 processes the dialogue text and contextual information through a text encoder to extract sentence semantic features, contextual semantic features, and global contextual semantic features; it extracts deep visual embeddings of facial expression images through a pre-trained visual encoder, maps them to the text semantic space through a learnable linear projection layer, and obtains aligned facial expression image features. S2 combines the semantic features of the statement with the facial image features, contextual semantic features, global contextual semantic features, and the previous round of global emotional memory sequence. Through a cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation. S3 involves sequentially splicing the four types of association representations generated in S2 to form a multimodal emotion association set, which is then added to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues. S4. Based on the memory units in the updated global emotional memory, the importance score is calculated, and Top-k filtering and weighted aggregation are used to obtain the key multimodal association representation. S5, based on global contextual semantic features and key multimodal association representations, generates semantic decoding state and association decoding state in parallel through dual-path decoders, dynamically fuses the two decoding states through an adaptive gating mechanism, and generates empathic responses that conform to multimodal emotional cues based on the fused features.
[0014] In one embodiment of the present invention, the step of combining the semantic features of the statement with facial expression image features, contextual semantic features, global contextual semantic features, and the previous round of global emotional memory sequence, and generating image-text association representations, text-context association representations, text-context association representations, and text-historical memory association representations through a cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, includes: Based on the semantic features of the sentences, the interaction space is mapped through a learnable projection matrix, and the attention score matrix is calculated with the corresponding combined modal features. The Top-k filtering strategy is used to select high response indices and extract key features under that modality. Using the obtained key features as anchors, attention scores are calculated in reverse with the semantic features of the sentences. A Top-k filtering strategy is used to lock in strongly related information in the text to obtain related text features. The key features and associated text features are concatenated along the feature dimension and then linearly mapped to generate the associated representation of the corresponding category.
[0015] In one embodiment of the present invention, the step of sequentially concatenating the four types of association representations generated in S2 to form a multimodal emotion association set, and adding it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues includes: Text-context association, text-context association, text-historical memory association, and image-text association are concatenated along the sequence length dimension to form a multimodal emotion association set, preserving the fine-grained features of each association representation; The global emotion memory sequence is iteratively updated by splicing operations. The multimodal emotion association set of the current round is directly appended to the end of the global emotion memory sequence of the previous round, forming an emotion memory that grows dynamically with each round of dialogue.
[0016] In one embodiment of the present invention, it further includes: Based on the updated global emotion memory sequence, four independent emotion prediction branches are constructed by combining global contextual semantic features, situational semantic features, and facial expression image features. The emotion probabilities output by each branch are multiplied together to obtain the final emotion probability. Based on the final emotion probability and the true label, the multimodal emotion classification loss is calculated using the log-likelihood loss function to assist in parameter optimization.
[0017] In one embodiment of the present invention, the step of generating semantic decoding states and relational decoding states in parallel through a dual-path decoder based on global contextual semantic features and key multimodal association representations, dynamically fusing the two decoding states through an adaptive gating mechanism, and generating an empathetic response that conforms to multimodal emotional cues based on the fused features includes: Based on the global context semantic features, a semantic decoding state is generated through the Transformer decoder to construct the semantic skeleton of the response; Based on the key multimodal association representation, it is mapped to semantic latent state through Transformer decoder to generate association relationship decoding state and supplement emotional expression information; An adaptive gating unit with a Sigmoid activation function is introduced to concatenate features of the semantic decoding state and the relational decoding state, calculate the adjustment coefficient, and then use the adjustment coefficient to perform weighted fusion of the two decoding states to obtain the final decoding state. The final decoded state is mapped to the vocabulary space, and the word probability distribution is obtained through the Softmax function. Based on this distribution, an empathetic response that conforms to multimodal emotional cues is generated.
[0018] In one embodiment of the present invention, the empathic response generation process employs a multi-task joint optimization mechanism, including: The response generation loss is calculated using the negative log-likelihood loss function for the given word probability distribution. The multimodal emotion classification loss and response generation loss are weighted and summed to obtain a joint loss function, and the parameters are optimized based on the joint loss function.
[0019] To achieve the above objectives, a second aspect of the present invention provides a dialogue-oriented image-text multimodal emotion understanding and empathic response generation device, comprising: The multimodal feature encoding module is used to process dialogue text and contextual information through a text encoder, extracting sentence semantic features, contextual semantic features and global contextual semantic features; and to extract deep visual embeddings of facial expression images through a pre-trained visual encoder, which are then mapped to the text semantic space through a learnable linear projection layer to obtain aligned facial expression image features. The cross-modal emotion association representation generation module is used to combine the semantic features of sentences with the facial expression image features, contextual semantic features, global contextual semantic features, and the previous round of global emotion memory sequence. Through the cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation. The global emotion memory iterative update module is used to concatenate the four types of association representations generated in S2 to form a multimodal emotion association set, and add it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues. The key multimodal association representation filtering module is used to calculate the importance score based on the memory units in the updated global emotion memory, and uses Top-k filtering and weighted aggregation to obtain key multimodal association representations; The dual-path empathic response generation module is used to generate semantic decoding state and relational decoding state in parallel through dual-path decoders based on global context semantic features and key multimodal association representations. The two decoding states are dynamically fused through an adaptive gating mechanism, and an empathic response that conforms to multimodal emotional cues is generated based on the fused features.
[0020] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing a dialogue-oriented graph-text multimodal emotion understanding and empathic response generation method as described in the first aspect embodiment.
[0021] To achieve the above objectives, the fourth aspect of this application proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a dialogue-oriented graph-text multimodal emotion understanding and empathic response generation method as described in the first aspect embodiment.
[0022] The embodiments of the present invention have the following beneficial effects: This invention introduces facial expressions, a frequently used visual emotion modality in social scenarios, and constructs a multimodal emotion modeling method that integrates text, context, and facial expressions. Through a cross-modal bidirectional attention interaction mechanism, it captures deep correlations between data from different modalities and writes the fused multi-source emotion information into an iterative emotion memory, thereby achieving continuous accumulation and correlation modeling of emotion information in multi-turn dialogues. By training with a joint multi-task loss function, it improves the accuracy of fine-grained emotion recognition while ensuring the quality of response generation. Finally, the emotion association information extracted from the emotion memory is used as an explicit condition to guide the generation of empathic responses through a dual-pathway decoding structure, thereby improving the emotional consistency and expressive quality of empathic responses. Attached Figure Description
[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a dialogue-oriented, multimodal image-text emotion understanding and empathic response generation method provided in an embodiment of the present invention; Figure 2 This is an overall framework diagram of the dialogue-oriented image-text multimodal emotion understanding and empathic response generation method provided in the embodiments of the present invention; Figure 3 A flowchart illustrating the cross-modal second-order interactive attention mechanism provided in this embodiment of the invention; Figure 4 This is a multimodal iterative correlation structure diagram provided in an embodiment of the present invention; Figure 5 This is a structural diagram of a dialogue-oriented, multimodal image-text emotion understanding and empathic response generation device provided in an embodiment of the present invention. Detailed Implementation
[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0026] In the task of multimodal emotion understanding and empathic response generation for dialogue, this embodiment receives dialogue text, contextual information and facial expression images input by the user, performs multimodal feature encoding and alignment on the input data, accumulates emotional cues using a multimodal iterative associative memory mechanism, and finally generates an empathic response through a dual-path decoder.
[0027] To address the problem that existing technologies often treat historical dialogues as a simple, flat sequence, ignoring the connections between sentences and making it difficult to capture deep emotional intentions, this embodiment employs an iterative processing mechanism. Specifically, instead of treating dialogues as a simple stack of isolated sentences, it processes each sentence in the dialogue iteratively through a cross-modal interaction mechanism involving text and images.
[0028] The following describes, with reference to the accompanying drawings, a method for dialogue-oriented multimodal emotion understanding and empathic response generation based on text and images according to an embodiment of the present invention.
[0029] Example 1 This embodiment provides a dialogue-oriented, multimodal image-text sentiment understanding and empathetic response generation method. For example... Figure 1 and Figure 2 As shown, the method includes the following steps: S1 processes the dialogue text and contextual information through a text encoder to extract sentence semantic features, contextual semantic features, and global contextual semantic features; it extracts deep visual embeddings of facial expression images through a pre-trained visual encoder, maps them to the text semantic space through a learnable linear projection layer, and obtains aligned facial expression image features.
[0030] In an embodiment of the present invention, a dialogue sequence comprising text input, facial expression input, and contextual information is assumed to be... .in Input contextual text relevant to the dialogue. For the first The text and images in the dialogue. This step aims to extract various modal features from the dialogue and map them to a unified semantic space, providing a basic data representation for subsequent iterative association. The specific process is as follows: First, text and context encoding are performed. This embodiment utilizes a Transformer-based text encoder to process the dialogue sequence and contextual information. The dialogue statements are then encoded... The context S is segmented to obtain a word sequence, which is then mapped to a vector space through a word embedding layer. Simultaneously, position vectors and role state vectors are superimposed on the embedded representation of the dialogue sentences to explicitly distinguish the roles of the speaker and the listener. Subsequently, word-level semantic features of the sentences are extracted through multiple Transformer encoding layers. and contextual semantic features To capture long-range semantic dependencies in multi-turn dialogues, the lexical-level features of each turn's statements in the historical dialogue are concatenated and used as input to the context-level encoder to obtain global contextual semantic features. .
[0031] Next, visual unit encoding is performed. This embodiment utilizes a pre-trained visual encoder to encode the first... The facial expression images sent by users are encoded, representing the images as feature sequences of multiple local regions. Then, a learnable linear projection layer maps these local region features to a feature space aligned with the text semantics, resulting in the facial expression image feature sequence. .
[0032] S2 combines the semantic features of the statement with the facial image features, contextual semantic features, global contextual semantic features, and the previous round of global emotional memory sequence. Through a cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation.
[0033] Specifically, this step constructs a cross-modal second-order interaction mechanism, aiming to deeply capture the implicit semantic relationships between multimodal data and filter out key features with high confidence. Let the... The semantic representation of the current statement in the wheel is as follows (Output from S1 Chinese text encoder), the aligned facial expression image features are: (Output from the pre-trained visual encoder in S1).
[0034] like Figure 3 As shown, the specific calculation process includes the following three sub-steps: S21. Based on the semantic features of the statement, the interaction space is mapped through a learnable projection matrix, and the attention score matrix is calculated with the corresponding combined modal features. A Top-k filtering strategy is used to select high-response indices and extract key features under that modality.
[0035] This step performs first-order interaction (visual key feature localization), projecting text features into query vectors and image features into key vectors, and calculating the attention score matrix. Subsequently, the score matrix is aggregated along the text lexical dimension to obtain a comprehensive relevance score for each image region, and Top-k filtering is performed to locate key image regions, obtaining visual anchor features.
[0036] First, the features are mapped to the interaction space using a learnable projection matrix:
[0037] in, These are the projection matrices of the first-order query and the key, respectively; These are the mapped query vector and key vector, respectively.
[0038] Next, the attention score matrix is calculated, and the top-k high-response indices are selected:
[0039]
[0040]
[0041] in, This is the scaling factor; This is a first-order attention score matrix; This indicates that maximum aggregation is performed along the text word dimension to obtain the comprehensive relevance score vector for each image region. ; This indicates an index selection operation, which returns the largest value. The index position of each element; Preset Select quantity. This is the set of indexes for the selected image regions; This indicates the operation of extracting features based on an index; These are the key visual features selected in the final screening.
[0042] S22. Using the obtained key features as anchors, the attention score is calculated in reverse with the semantic features of the sentence. The Top-k filtering strategy is used to lock in the strongly related information in the text to obtain the related text features.
[0043] Specifically, this step uses the visual anchor features obtained from the first-order screening as the query vector, and after independent projection, recalculates the association score with the text features to achieve secondary realignment and rescoring under visual evidence constraints. Furthermore, the Top-k screening strategy is used to lock in the strongly associated information in the text to obtain the associated text features.
[0044] This step performs a second-order interaction (visually guided text association calculation), using the selected visual features as anchors to retrieve key information from the text. The query uses the visual key features as the query and the current statement as the key; the expression is:
[0045] Furthermore, the reverse attention score matrix is calculated, and the score matrix is aggregated along the visual anchor dimension to obtain the comprehensive relevance score for each text word. Top-k filtering is then performed to locate text words strongly associated with the visual anchor, yielding the associated text features. The calculation formula is as follows:
[0046]
[0047]
[0048] in, For the selected set of related word indexes, The final selected text features are those that have a strong semantic relationship with the visual features. Preset Select quantity.
[0049] S23, the above key image features and associated text features are concatenated along the feature dimension, and a corresponding category association representation is generated through linear mapping.
[0050] This step performs modality-to-modality feature fusion, combining the aforementioned filtered related text features. With key visual features Pooling is performed separately, and the features are concatenated along the feature dimension, and then the first feature is generated through a linear mapping. Graphical and textual representation of wheels :
[0051] This correlation characterization As an explicit image-text multimodal emotional cue, it effectively integrates visual and textual information, serving as a foundational feature for subsequent global emotional memory sequence updates.
[0052] Furthermore, based on the same second-order interaction mechanism, this embodiment further calculates deep associations between text and context, text and historical memory to capture complete contextual cues, specifically including: Text-context association representation : With the current statement Contextual semantic features As input, the consistency between dialogue content and context is captured through second-order interaction steps; Text-contextual representation : With the current statement With global context semantic features As input, the association information between the current statement and the global context of the dialogue is captured through second-order interaction steps; Text-historical emotional memory association representation : With the current statement Compared with the accumulated global emotional memory sequence from the previous round As input, through a second-order interactive step, key clues strongly related to the current statement are retrieved and aggregated from historical emotional memories.
[0053] Ultimately, the correlation representation of the above four dimensions ( These together constitute the multimodal emotion association set for the current round, which is used for updating the global emotion memory sequence in the future.
[0054] S3 concatenates the four types of association representations generated in S2 to form a multimodal emotion association set, and adds it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues.
[0055] This step aims to capture the multimodal cues of the current round ( The selected relevant features are stored in the memory module and dynamically updated. The selected relevance features are fused with the current semantics and appended to the end of the global memory sequence. This mechanism allows key cues to be dynamically accumulated throughout the conversation, forming a dynamically growing global emotional memory. Figure 4 The diagram shown illustrates its multimodal iterative memory structure.
[0056] Specifically, step S3 includes: S31, the text-context association representation, text-historical memory association representation, and image-text association representation are spliced together along the sequence length dimension to form a multimodal emotion association set, while retaining the fine-grained features of each association representation.
[0057] In this embodiment, multimodal fusion is performed using sequence splicing to preserve the independence and fine-grained features of each modal cue. The modal association cue obtained in this round of calculation is spliced along the sequence length dimension to generate the first... The multimodal emotion association set of the wheel :
[0058] By introducing image modality association This approach overcomes the shortcomings of traditional methods that rely solely on text and context, and enhances the ability to perceive fine-grained emotions.
[0059] S32 uses a splicing operation to iteratively update the global emotion memory sequence, directly appending the multimodal emotion association set of the current round to the end of the global emotion memory sequence of the previous round, forming an emotion memory that dynamically grows with each round of dialogue.
[0060] In this embodiment, the association set of the current round is... Added to the global emotional memory sequence of the previous round At the end, update the global emotional memory sequence. :
[0061] Through this iterative appending mechanism, the global emotional memory sequence... The length increases with each round of dialogue, explicitly storing and accumulating all visual and textual emotion-related cues from the first round to the current round.
[0062] In the emotion recognition stage, based on the complete multimodal memory sequence accumulated at the current moment, the emotional state of the current dialogue is predicted and calibrated by integrating multi-view information. The feature extraction capability of the memory module is optimized in reverse through auxiliary supervision signals. The specific steps include: (1) Based on the updated global emotion memory sequence, four independent emotion prediction branches are constructed by combining global contextual semantic features, situational semantic features and facial expression image features. The emotion probabilities output by each branch are multiplied together to obtain the final emotion probability.
[0063] Specifically, in order to correct for possible comprehension biases caused by a single modality, global contextual semantic features are used respectively. Contextual semantic features Facial expression image features and global emotional memory features obtained from global emotional memory sequence encoding. Construct separate sentiment prediction branches for each. Let... As the identifier for each branch, its sentiment prediction distribution can be represented as:
[0064] in, This represents the feature vector of the corresponding view. For each branch, an independent classification network, This refers to the softmax function.
[0065] (2) Based on the final emotion probability and the true label, the multimodal emotion classification loss is calculated using the log-likelihood loss function to assist in optimizing the parameters.
[0066] Specifically, the emotion probabilities output from each branch are multiplied to obtain the final emotion probability, which is then compared with the true label. The parameters are optimized using the log-likelihood loss function:
[0067]
[0068] S4 calculates importance scores based on memory units in the updated global emotional memory, uses Top-k filtering and weighted aggregation to obtain key multimodal association representations.
[0069] In this embodiment, global emotion memory It stores a large amount of multimodal emotion-related information. To prioritize the use of more informative memory cues during response generation, memory filtering is performed. Specifically, for memories... Each memory unit is assigned an importance score, and a Top-k strategy is used to select the highest-scoring unit. The selected memory units are then weighted using their corresponding scores to obtain the filtered key memory representations. And aggregate them to form key multimodal correlation characterizations. :
[0070] S5, based on global contextual semantic features and key multimodal association representations, generates semantic decoding state and association decoding state in parallel through dual-path decoders, dynamically fuses the two decoding states through an adaptive gating mechanism, and generates empathic responses that conform to multimodal emotional cues based on the fused features.
[0071] This step aims to generate responses that combine semantic coherence and emotional resonance based on the global emotion memory sequence constructed using S3. Two decoding paths run in parallel: one focuses on the semantic structure of the dialogue, and the other focuses on processing multimodal associative features retrieved from memory. The two paths are dynamically fused through an adaptive gating unit to achieve fine-grained empathic expression.
[0072] Specifically, step S5 includes: S51, based on the global context semantic features, generate semantic decoding state through the Transformer decoder to construct the semantic skeleton of the response.
[0073] Specifically, the semantic decoding pathway takes global contextual semantic features as input and uses the Transformer decoder to generate the basic semantic skeleton of the response. Let the... The decoding state of the step is :
[0074] in, This is an embedded representation of the generated response words; this path is used to ensure that the response conforms to grammatical rules and contextual logic.
[0075] S52, based on the key multimodal association representation, the semantic latent state is mapped through the Transformer decoder to generate the association relationship decoding state and supplement the emotional expression information.
[0076] Specifically, the association decoding pathway uses the retrieved key association representations As input, a decoder maps multimodal emotion features to corresponding semantic latent states. Let the first... The decoding state of the step is :
[0077] This pathway maps multimodal emotional features to semantic latent states, making up for the shortcomings of the pure text pathway in emotional expression.
[0078] S53, an adaptive gating unit with a Sigmoid activation function is introduced to perform feature concatenation on the semantic decoding state and the relational decoding state, calculate the adjustment coefficient, and then use the adjustment coefficient to perform weighted fusion of the two decoding states to obtain the final decoding state.
[0079] Specifically, in order to dynamically balance the weight of semantics and sentiment when generating each word, an adaptive gating mechanism is introduced:
[0080] in, This is the adjustment coefficient.
[0081] S54 maps the final decoded state to the vocabulary space, obtains the word probability distribution through the Softmax function, and generates an empathetic response that conforms to multimodal emotional cues based on this distribution.
[0082] Specifically, the final decoded state obtained by weighted fusion is mapped to the vocabulary space to obtain the final word probability distribution:
[0083] in, These represent the learnable weight matrix and bias vector, respectively, that map the output layer to the vocabulary space. This refers to the size of the vocabulary.
[0084] In this embodiment of the invention, the generation process of empathic responses employs a multi-task joint optimization mechanism, including: For the given word probability distribution, the negative log-likelihood loss function is used to calculate the response generation loss.
[0085] The multimodal emotion classification loss and response generation loss are weighted and summed to obtain a joint loss function, and the parameters are optimized based on the joint loss function.
[0086] Specifically, to ensure accurate understanding of fine-grained emotions in multimodal contexts while generating fluent responses, this embodiment constructs a joint objective function that includes response generation loss and multimodal sentiment classification loss. The response generation loss uses negative log-likelihood loss, calculated as follows:
[0087] in, The length of the target response; This refers to the final word probability distribution calculated in step 3 above. For the context of the dialogue text, For image, It is for emotional memory.
[0088] The final joint loss function is obtained by weighted summation of response generation loss and sentiment classification loss:
[0089] in, For sentiment classification loss; This is a hyperparameter used to balance the weights of the generation and classification tasks.
[0090] Example 2 This invention also provides a dialogue-oriented graph-text multimodal emotion understanding and empathic response generation device, such as... Figure 5 As shown, the device 10 includes: The multimodal feature encoding module 100 is used to process dialogue text and contextual information through a text encoder, extracting sentence semantic features, contextual semantic features and global contextual semantic features; and to extract deep visual embeddings of facial expression images through a pre-trained visual encoder, which are then mapped to the text semantic space through a learnable linear projection layer to obtain aligned facial expression image features.
[0091] The cross-modal emotion association representation generation module 200 is used to combine the semantic features of the statement with the facial expression image features, the contextual semantic features, the global contextual semantic features, and the previous round of global emotion memory sequence. Through the cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation.
[0092] The global emotion memory iterative update module 300 is used to concatenate the four types of association representations generated in S2 to form a multimodal emotion association set, and add it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues.
[0093] The key multimodal association representation screening module 400 is used to calculate the importance score based on the memory units in the updated global emotional memory, and to obtain the key multimodal association representation by using Top-k screening and weighted aggregation.
[0094] The dual-path empathic response generation module 500 is used to generate semantic decoding state and relational decoding state in parallel through dual-path decoders based on global context semantic features and key multimodal association representations. The two decoding states are dynamically fused through an adaptive gating mechanism, and an empathic response that conforms to multimodal emotional cues is generated based on the fused features.
[0095] Example 3 To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the methods described above.
[0096] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.
[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0099] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A dialogue-oriented, multimodal image-text sentiment understanding and empathic response generation method, characterized in that, Includes the following steps: S1 processes the dialogue text and contextual information through a text encoder to extract sentence semantic features, contextual semantic features, and global contextual semantic features; it extracts deep visual embeddings of facial expression images through a pre-trained visual encoder, maps them to the text semantic space through a learnable linear projection layer, and obtains aligned facial expression image features. S2 combines the semantic features of the statement with the facial image features, contextual semantic features, global contextual semantic features, and the previous round of global emotional memory sequence. Through a cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation. S3 involves sequentially splicing the four types of association representations generated in S2 to form a multimodal emotion association set, which is then added to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues. S4. Based on the memory units in the updated global emotional memory, the importance score is calculated, and Top-k filtering and weighted aggregation are used to obtain the key multimodal association representation. S5, based on global contextual semantic features and key multimodal association representations, generates semantic decoding state and association decoding state in parallel through dual-path decoders, dynamically fuses the two decoding states through an adaptive gating mechanism, and generates empathic responses that conform to multimodal emotional cues based on the fused features.
2. The method according to claim 1, characterized in that, The process involves combining sentence semantic features with facial expression image features, contextual semantic features, global contextual semantic features, and the previous round of global emotional memory sequence. Through a cross-modal second-order interactive attention mechanism, combined with attention calculation and a Top-k selection strategy, it generates image-text association representations, text-context association representations, text-context association representations, and text-historical memory association representations, including: Based on the semantic features of the sentences, the interaction space is mapped through a learnable projection matrix, and the attention score matrix is calculated with the corresponding combined modal features. The Top-k filtering strategy is used to select high response indices and extract key features under that modality. Using the obtained key features as anchors, attention scores are calculated in reverse with the semantic features of the sentences. A Top-k filtering strategy is used to lock in strongly related information in the text to obtain related text features. The key features and associated text features are concatenated along the feature dimension and then linearly mapped to generate the associated representation of the corresponding category.
3. The method according to claim 1, characterized in that, The process involves concatenating the four types of association representations generated in S2 to form a multimodal emotion association set, and then appending it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues, including: Text-context association, text-context association, text-historical memory association, and image-text association are concatenated along the sequence length dimension to form a multimodal emotion association set, preserving the fine-grained features of each association representation; The global emotion memory sequence is iteratively updated by splicing operations. The multimodal emotion association set of the current round is directly appended to the end of the global emotion memory sequence of the previous round, forming an emotion memory that grows dynamically with each round of dialogue.
4. The method according to claim 3, characterized in that, Also includes: Based on the updated global emotion memory sequence, four independent emotion prediction branches are constructed by combining global contextual semantic features, situational semantic features, and facial expression image features. The emotion probabilities output by each branch are multiplied together to obtain the final emotion probability. Based on the final emotion probability and the true label, the multimodal emotion classification loss is calculated using the log-likelihood loss function to assist in parameter optimization.
5. The method according to claim 1, characterized in that, The method, based on global contextual semantic features and key multimodal association representations, generates semantic decoding states and association relationship decoding states in parallel through a dual-path decoder. It then dynamically fuses the two decoding states using an adaptive gating mechanism and generates empathetic responses that conform to multimodal emotional cues based on the fused features. This includes: Based on the global context semantic features, a semantic decoding state is generated through the Transformer decoder to construct the semantic skeleton of the response; Based on the key multimodal association representation, it is mapped to semantic latent state through Transformer decoder to generate association relationship decoding state and supplement emotional expression information; An adaptive gating unit with a Sigmoid activation function is introduced to concatenate features of the semantic decoding state and the relational decoding state, calculate the adjustment coefficient, and then use the adjustment coefficient to perform weighted fusion of the two decoding states to obtain the final decoding state. The final decoded state is mapped to the vocabulary space, and the word probability distribution is obtained through the Softmax function. Based on this distribution, an empathetic response that conforms to multimodal emotional cues is generated.
6. The method according to claim 5, characterized in that, The generation process of the empathic response adopts a multi-task joint optimization mechanism, including: The response generation loss is calculated using the negative log-likelihood loss function for the given word probability distribution. The multimodal emotion classification loss and response generation loss are weighted and summed to obtain a joint loss function, and the parameters are optimized based on the joint loss function.
7. A dialogue-oriented, multimodal image-text emotion understanding and empathic response generation device, characterized in that, include: The multimodal feature encoding module is used to process dialogue text and contextual information through a text encoder to extract sentence semantic features, contextual semantic features, and global contextual semantic features; Deep visual embeddings of facial expression images are extracted by a pre-trained visual encoder and mapped to the text semantic space through a learnable linear projection layer to obtain aligned facial expression image features. The cross-modal emotion association representation generation module is used to combine the semantic features of sentences with the facial expression image features, contextual semantic features, global contextual semantic features, and the previous round of global emotion memory sequence. Through the cross-modal second-order interactive attention mechanism, combined with attention calculation and Top-k selection strategy, it generates image-text association representation, text-context association representation, text-context association representation, and text-historical memory association representation. The global emotion memory iterative update module is used to concatenate the four types of association representations generated in S2 to form a multimodal emotion association set, and add it to the global emotion memory sequence through an iterative update mechanism to dynamically accumulate multiple rounds of emotion association cues. The key multimodal association representation filtering module is used to calculate the importance score based on the memory units in the updated global emotion memory, and uses Top-k filtering and weighted aggregation to obtain key multimodal association representations; The dual-path empathic response generation module is used to generate semantic decoding state and relational decoding state in parallel through dual-path decoders based on global context semantic features and key multimodal association representations. The two decoding states are dynamically fused through an adaptive gating mechanism, and an empathic response that conforms to multimodal emotional cues is generated based on the fused features.
8. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the dialogue-oriented image and text multimodal emotion understanding and empathic response generation method as described in any one of claims 1-6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the dialogue-oriented text-based multimodal emotion understanding and empathic response generation method as described in any one of claims 1-6.