Multi-modal common-situation reply generation method based on emotion evolution sequence backtracking learning

Abstract

The invention relates to a multi-mode common-situation reply generation method based on emotion evolution sequence backtracking learning, and aims to help a dialogue model to better understand user emotion and carry out common-situation communication with a user by utilizing an association relationship of a user emotion feature sequence. The method mainly comprises the following steps: (1) providing a brand-new multi-modal common-situation reply generation framework based on user emotion evolution sequence backtracking learning; and (2) providing a data enhancement method for a multi-modal common-situation data set. And (3) providing a multi-modal user emotion detector based on dot product attention. And (4) providing a common-condition reply generator based on a common-condition strategy backtracking learning mechanism.

Description

Technical Field

[0001] The present invention relates to the fields of dialogue systems and intelligent robots, and in particular to a multimodal empathy response generation method based on emotion evolution sequence backtracking learning. Background Art

[0002] Empathetic response generation (ERG) is one of the basic research tasks in the field of emotional intelligence research and an important way to achieve emotional intelligence human-computer interaction. The empathetic dialogue system takes user speech as input, identifies and detects user emotions, and generates responses containing empathetic emotions. The main challenges of the empathetic response generation task are how to accurately identify and detect user emotions and how to incorporate appropriate corresponding emotions and user empathy into the response. MoEL designs an independent decoder for each individual emotion, and softly combines the output of the decoder to generate empathetic responses, which initially achieves simple empathy with the user. EmpDG further explores the interaction and combination of emotions from a multi-scale perspective, and generates empathetic responses by combining emotions of different granularities. Although early research works achieved simple empathy with users, they did not deeply explore and understand user emotions, and thus could not achieve advanced empathy. Therefore, MkERG introduces the common sense knowledge graph ConceptNet into the empathetic response generation task, and deeply understands the user emotions implied in the context through the common sense knowledge in ConceptNet. ESC introduces Hill's helping skills theory into the empathy response generation task, and proposes a three-strategy empathy response model based on the three-stage goals of exploration, comprehension, and action in the theory, which explores the essence of the problem, soothes the user's emotions, and takes final action. MISC proposes to mix multiple strategies on the basis of a single strategy, and additionally introduces the common sense knowledge reasoning model COMET, which assists in mining and understanding user emotions by building an additional common sense knowledge base from the language model. CoMAE explores the empathy expression paradigm and constructs an empathy expression model based on communication mechanism, dialogue behavior, and emotion. It realizes deep empathy with users by constructing the three elements of empathy expression through hierarchical reasoning. Although in-depth analysis of user emotions can help improve the empathy effect and the quality of empathy responses, it ignores the dynamic evolution trend of user emotions in the time dimension, resulting in the inability to comprehensively and effectively model user emotions. The existing technology has the following three main problems:

[0003] 1. Lack of comprehensive modeling of users’ dynamic emotions: Most existing methods focus on analyzing and understanding users’ static emotions at a certain moment, but do not pay attention to the dynamic changing trends of users’ emotions during the communication process, resulting in the inability to comprehensively and effectively model users’ emotions.

[0004] 2. Lack of mining and utilization of the correlation between emotional feature sequences: The changing trend of user emotions over time during the communication process contains the sequential pattern information of user emotion evolution. Existing methods lack the mining and utilization of the sequential pattern information of emotion evolution.

[0005] 3. Unable to fully utilize multimodal sentiment features to generate empathetic responses: Most existing methods only extract sentiment features of the text modality and integrate them into the generation process to obtain responses. They do not fully extract multimodal information such as audio and further analyze the user’s multimodal emotions to generate empathetic responses. Summary of the invention

[0006] The present invention proposes a multimodal empathy response generation method based on emotion evolution sequence backtracking learning, which aims to utilize the correlation of emotion feature sequences of empathy communication in the dialogue history, explore the correlation relationship of user emotion dynamic change trends in the time dimension, help the dialogue model to better communicate empathetically with users, and generate higher quality empathy responses. The present invention mainly includes the following contents:

[0007] (1) A novel multimodal empathetic response generation framework based on sequential backtracking learning of user emotion evolution is proposed.

[0008] (2) A data enhancement method for a multimodal empathy dataset is proposed.

[0009] (3) A multimodal user emotion detector based on dot product attention is proposed.

[0010] (4) An empathetic response generator based on empathy strategy backtracking learning mechanism is proposed.

[0011] The specific contents are as follows:

[0012] (1) A new multimodal empathetic response generation framework based on user emotion evolution sequence backtracking learning is proposed (e.g. Figure 1 ): The framework mainly consists of three components: a multimodal user emotion detector, an empathy strategy backtracking learning module, and an empathy reply generator. The empathy reply generator is built based on the large language model ChatGLM3. The framework generates empathy replies through the following steps: First, the user's multimodal input and conversation context are input into the user emotion detector to detect the user's multi-emotion sequence, and then backtracking learning is carried out based on the user's multi-emotion sequence to generate three different styles of empathy strategy prompts. The user input text and the three different styles of empathy strategy prompts are concatenated and sent to the empathy reply generator to generate three candidate replies. Finally, the candidate reply with the lowest perplexity PPL (Perplexity) evaluation score and the highest RL (ROUGE-L) evaluation score is selected as the final reply. Among them, for sentence S, N is its sentence length, p(w i) is the probability of generating the i-th word, then the evaluation index PPL(S) is defined as follows:

[0013]

[0014] ROUGE stands for Recall-Oriented Understudy For Gisting Evaluation, which is one of the current mainstream evaluation methods for automatic text summarization. The L in ROUGE-L refers to the longest common subsequence LCS (Longest Common Subsequence). Assuming X is the reference summary sentence and Y is the candidate summary sentence, Rouge-L uses the F-measure to estimate the similarity between two summaries X and Y of length m and n, where R lcs Recall rate, P lcs It represents the accuracy, and the specific calculation method is as follows:

[0015]

[0016] β can be regarded as a measure between recall and precision, usually in F lcs The value of is set relatively large in the calculation of , which is in line with the original design intention of ROUGE that favors recall rate.

[0017] (2) A data enhancement method for a multimodal empathy dataset is proposed: Since there is currently a lack of publicly available datasets for the multimodal empathy response generation task, this paper performs data segmentation, data calibration, collation and merging, and data annotation on the original dialogue segments and corresponding features based on the multimodal empathy dataset MEDIC. The present invention performs data segmentation on the original dialogue segments and corresponding features with reference to the start and end timestamps of each dialogue sentence in the actual dialogue video, and divides the dialogue parties into user input text and psychologist reply text; performs data calibration work such as typo correction and speaker correction; finally, uses three types of communication mechanisms, namely, soothing and guiding, expressing understanding, and exploring, eight types of dialogue behaviors, namely, admitting facts, recognizing the other party, comforting the other party, encouraging the other party, asking questions, giving suggestions, expressing sympathy, and expressing expectations, and ten types of emotions, namely, anger, approval, worry, fear, gratitude, happiness, sadness, surprise, confusion, and neutrality, to perform clause-level multivariate emotion annotation on the dialogue text of the two parties, and finally constructs a multimodal empathy dialogue dataset containing 2,831 dialogue data.

[0018] (3) A multimodal user emotion detector based on dot product attention is proposed (e.g. Figure 2): The present invention feeds the conversation history c and the user's multimodal input x into a dynamically constructed fully connected neural network to detect the user's multi-emotion sequence. Due to the variable length of the user's speech, the coarse-grained sentence-level emotion label sometimes cannot well describe the multi-emotion contained in the user input. The present invention controls the granularity of user emotion detection and calibration at the clause level, thereby more carefully analyzing and extracting the multi-emotion contained in the user input. If the long-term conversation history at time i contains n conversation sentences and the user input contains k clauses, then the conversation history c can be expressed as The user's current multimodal input x can be expressed as

[0019] When detecting the sentiment of the kth clause in the user input at time i, different from the conventional processing methods in previous studies, the present invention further subdivides the conversation history into long-term conversation history and short conversation history Where i, n, and k represent that the conversation history at time i contains n sentences and the user input contains k clauses. The present invention divides the complete sentence input by the user into multiple clauses according to punctuation marks, wherein the conversation content between the two parties before the current input of the user is the long-term conversation history, and all other clauses in the current input of the user before the current clause to be detected are the short-term conversation history.

[0020] First, BERT is used to encode all sentences in the long-term conversation history and all clauses in the short-term conversation history:

[0021]

[0022]

[0023] The text features are transformed through the nonlinear transformation layer With MFCC features Projection transformation to the same vector space (where W T , b T is the text modality transformation weight and bias parameter, W A , b A is the audio modality transformation weight and bias parameter), and the text modality representation p in the same feature space is obtained. i and the audio modality representation q j :

[0024]

[0025] The association weights of the k-th clause with the n sentences in the long-term dialogue history and the k-1 clauses in the short-term dialogue history are calculated by vector dot product, and the calculation results of each association weight are used as components to construct an association weight vector α (where is an n-dimensional vector, is a k-1 dimensional vector. The superscripts TT, TA, AT, and AA represent text and text, text and audio, audio and text, and audio and audio, respectively. The subscripts L and S represent long-term conversation history and short-term conversation history, respectively. For example, formula (9) represents is the text modality feature of the kth clause in the user's current input and the text modality features p1,...,p of n sentences in the long conversation history n The n associated weight calculation scores are vectors composed of components, and formulas (10) to (16) are deduced by analogy):

[0026]

[0027] Based on the association weight vector α, after performing element-wise addition operations, the normalized association weights of all sentences in the long-term dialogue history and all clauses in the short-term dialogue history are calculated. The normalized association weights are used for guidance to weightedly calculate the encodings of all sentences in the long-term dialogue history and all clauses in the short-term dialogue history to obtain the long-term dialogue history encoding h L and the short-term conversation history encoding h S :

[0028]

[0029] A set of weight hyperparameters are used to weight the long-term conversation history, short-term conversation history, user text modal input, and user audio modal input. The calculation results are concatenated and sent to the multi-layer perceptron. The sentiment of the k-th clause is obtained through the softmax function:

[0030]

[0031] After detecting the emotions of all k clauses in the user input at time i according to the above steps, the multivariate emotion sequence EM corresponding to the user input at time i can be obtained. i :

[0032] EM i =(em1,...,em k )#(twenty two)

[0033] (4) An empathy response generator based on empathy strategy backtracking learning mechanism is proposed: This invention uses a fully connected neural network to build a psychologist empathy strategy classifier (such as Figure 3), the psychologist's reply text is encoded using BERT and sent to the fully connected layer, and the 3 possible communication mechanisms and 8 possible types of dialogue behaviors contained in the psychologist's reply are classified by the softmax function. After each round of interaction between the two parties, the multimodal user emotion detector and the empathy strategy classifier are used to detect the user input x and the psychologist's reply r respectively, and the multi-emotion sequence contained in the user's speech and the empathy strategy sequence contained in the psychologist's reply are obtained. Then the multi-emotion sequence and the empathy strategy sequence are constructed into a binary and stored in the empathy strategy backtracking module SBM (Strategy-Backtrack-Module) (such as Figure 4 ). Subsequently, the similarity between the multi-emotion sequence corresponding to the user input and the multi-emotion sequence in all past dialogue scenarios in the empathy strategy backtracking module is calculated by Jaccard similarity. After the similarity calculation results are normalized, the types and proportions of all referenced empathy strategies are counted. Through three different types of backtracking learning strategies, namely dominant backtracking, comprehensive backtracking, and random backtracking, the types and proportions of the learned empathy strategies are converted into three different styles of empathy communication prompts (such as Figure 5 ). The three empathy communication prompts are concatenated at the end of the user input text, and the text is input as a whole into the empathy response generator built based on the fine-tuned ChatGLM3 (e.g. Figure 6 ), three different styles of candidate replies are generated based on three different backtracking learning strategies, and finally the present invention uses PPL and ROUGE-L evaluation indicators to evaluate the three styles of candidate replies, and selects the candidate reply with the lowest PPL score and the highest ROUGE-L score as the final reply.

[0034] Compared with other methods, the present invention has the following significant advantages:

[0035] 1. The present invention innovatively proposes a multimodal empathy response generation framework based on user emotion evolution sequence retrospective learning. This framework integrates three key components: a multimodal user emotion detector, an empathy strategy retrospective learning module, and an empathy response generator, and constructs a new set of empathy response generation processes. The present invention uses user multimodal input and conversation context as input to accurately detect user multi-emotion sequences. At the same time, it generates various types of empathy strategy prompts based on the retrospective learning mechanism, and combines the large language model ChatGLM3 to generate and filter out the optimal response, which can generate semantically clear and high-quality empathy responses. Compared with previous research methods that only focus on analyzing the static emotions of users at a certain moment, the framework proposed by the present invention can better dynamically model user emotions and generate empathy responses with stronger empathy and clearer semantics.

[0036] 2. The present invention adopts a multimodal user emotion detector based on dot product attention. By further dividing the conversation history into long-term conversation history and short-term conversation history, dot product attention is used to calculate the association weight score between the user input and each sentence in the long-term and short-term conversation history, and weighted summation is performed based on the weight calculation results to obtain the long-term and short-term conversation history encoding. By guiding the long-term and short-term conversation history encoding through user input, the key information in the conversation history can be extracted more efficiently, and the multiple emotions contained in the user input can be accurately analyzed.

[0037] 3. The present invention adopts an empathy response generator based on an empathy strategy retrospective learning mechanism to detect the multi-emotion sequence contained in the user input and the empathy strategy sequence contained in the psychologist's response in real time during the interaction process. Past dialogue scenes are retrospected through Jaccard similarity, three types of empathy communication prompts are generated based on three types of retrospective learning strategies, and a large language model ChatGLM3 is used to generate a variety of candidate responses. Finally, the PPL and ROUGE-L evaluation indicators are used to screen out the final response that best suits the current dialogue context. This mechanism of retrospective learning based on the emotional feature sequence in past dialogue scenes and generating adapted responses in the current dialogue scene makes the generated responses more scene-adaptive and can empathize with users more effectively, overcoming the drawbacks of traditional methods that lack mining of correlation relationships between emotional feature sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall framework of the present invention "proposing a multimodal empathic response generation framework based on retrospective learning of emotion evolution sequences".

[0039] Figure 2 Schematic diagram of the detector structure of the present invention "proposing a multimodal user emotion detector based on dot product attention".

[0040] Figure 3 This is a schematic diagram of the classifier structure of the present invention “proposing an empathy strategy classifier”.

[0041] Figure 4 This is a schematic diagram of the module structure of the present invention "proposing an empathy strategy backtracking module".

[0042] Figure 5 This is a schematic diagram of the construction method of the present invention "proposing a construction method for empathy communication prompts".

[0043] Figure 6 A schematic diagram of a generator for the present invention "proposing an empathy response generator based on an empathy strategy backtracking learning mechanism". DETAILED DESCRIPTION

[0044] In order to more clearly show the features and advantages of this patent, a detailed description of an embodiment is provided below. It should be clear that the following detailed description is only used as an example and is intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as those commonly understood by ordinary technicians in the technical field to which this application belongs.

[0045] This embodiment can be implemented according to the following steps, not limited to any programming language. In this example, Python programming language is used as an example to build a model on the Transformers deep learning framework. The specific steps are as follows:

[0046] Step 1: Processing the data set. Processing the data set is divided into four processes: data segmentation, data calibration, collation and merging, and data annotation. First, based on the MEDIC data set, according to the actual start and end timestamps of the dialogue sentences in the original dialogue video, the dialogue segment text and MFCC (Mel-frequency cepstral coefficient) features contained in the original data set are segmented (by time dimension) to obtain sentence-level dialogue text and MFCC features. The data calibration process is mainly to carefully check the dialogue text based on the original dialogue video, and correct typos and speaker errors in the dialogue text. After the subsequent segmentation of the dialogue segment, the dialogue text and MFCC features contained therein are merged into a complete dialogue for collation and merging. Finally, 3 types of communication mechanisms, 8 types of dialogue behaviors and 10 types of emotions are used for manual annotation at the clause level. After the data set is produced, the present invention divides it into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0047] Step 2: Construct the overall model framework. The framework consists of three core components: user emotion detector, empathy strategy backtracking learning module, and empathy strategy generator. The user emotion detector receives user multimodal input and conversation context, finely divides the conversation history into long-term conversation history and short-term conversation history, uses BERT to encode text, uses dot product attention to calculate the associated weight score and weightedly calculate the encoding of each sentence in the conversation history to obtain conversation history features, concatenates the conversation history features and multimodal features, and then sends them to the fully connected layer for classification to accurately detect the multivariate emotion sequence at the clause level. After the interaction round ends, the empathy strategy backtracking learning module combines the user emotion detector and the empathy strategy classifier to convert the types and proportions of the learned empathy strategies into empathy communication prompts of different styles. The empathy reply generator is built based on ChatGLM3. The user input text and three empathy communication prompts are concatenated and sent to the empathy reply generator to generate three candidate replies. Finally, the PPL and ROUGE-L evaluation indicators are used for screening, and the candidate reply with the lowest PPL score and the highest ROUGE-L score is selected as the final reply.

[0048] Step 3: Model training. The model training is performed on the model framework built in step 2, and the training hyperparameters and the optimal model are selected based on the performance on the validation set. The present invention uses 5 fully connected layers (1024-768-512-512-512) to build a user emotion detector, with a training cycle of 3 and a batch size of 32. The reply generator is then built using the 6 billion parameter large language model ChatGLM3 (28 layers, 4096 hidden layer dimensions). During the training process, the present invention uses the Adam optimizer and sets the learning rate to 10 -4 All model training was performed on NVIDIA RTX 4090, with the training epochs set to 5 and the batch size set to 1.

[0049] Step 4: Model testing. The empathy response generation model is comprehensively tested through a large number of experiments on the test set. This step includes two parts: automatic evaluation and manual evaluation. Automatic evaluation is based on PPL and ROUGE-L scores. The present invention has a lower PPL score and a higher ROUGE-L score, indicating that the responses generated by this model are accurate in terms of verbal expression, with clear semantics and easy to understand. In order to be more in line with the way of expression in real life, the present invention also adopts a manual evaluation method to evaluate the quality of the responses generated by the model through fluency, relevance, and empathy. The final experimental results show that the present invention can generate high-quality empathy responses with clear semantics and strong empathy.

[0050] In summary, for the empathy dialogue research task under the open domain dialogue task, this paper proposes a multimodal empathy reply generation method based on emotion evolution sequence backtracking learning. Through data-enhanced multimodal empathy dataset, user emotion detector based on dot product attention and empathy reply generator based on empathy strategy backtracking learning mechanism, high-quality empathy replies with clearer semantics and stronger empathy are generated, solving the related problems of how to dynamically perceive the dialogue context to generate empathy replies.

[0051] Those skilled in the art will appreciate that the protection scope of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after the changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A multimodal empathy response generation method based on emotion evolution sequence backtracking learning, which uses the correlation relationship of user emotion feature sequences to help the dialogue model understand user emotions and conduct empathy communication, specifically including: A multimodal empathy response generation framework based on user emotion evolution sequence backtracking learning is constructed. The framework includes a multimodal user emotion detector based on dot-product attention, an empathy strategy backtracking learning module, and an empathy response generator based on the large language model ChatGLM3. The specific process is as follows: first, a data enhancement method for a multimodal empathy dataset is proposed to preprocess the multimodal empathy dataset MEDIC; then, the user multimodal input and the conversation context are input into the user emotion detector to detect the user's multi-emotion sequence; Based on the user's multi-emotional sequence, retrospective learning is carried out to generate three different types of empathy strategy prompts. Finally, the user input text is concatenated with the prompt and sent to the empathy reply generator built based on the large language model ChatGLM3 to generate three candidate replies. Among the three candidate replies, the one with the lowest perplexity PPL evaluation score and the highest ROUGE-L evaluation score is selected as the final reply.

2. According to claim 1, the multimodal empathy response generation method based on emotion evolution sequence backtracking learning is characterized in that: The multimodal empathy response generation framework implements dataset preprocessing through the proposed data enhancement method of the multimodal empathy dataset, which specifically includes: Based on the multimodal empathy dataset MEDIC, the original dialogue fragments and corresponding features were segmented, calibrated, collated and merged, and labeled. First, the original dialogue fragments and corresponding features were segmented, then calibrated, and then collated and merged. Finally, three types of communication mechanisms, such as comfort and guidance, expression of understanding, and exploration, as well as eight types of dialogue behaviors, such as acknowledging facts and recognizing the other party, and ten types of emotions, such as anger and approval, were used to perform clause-level multi-emotion annotation on the dialogue materials of both parties. Finally, a multimodal empathy dialogue dataset containing 2831 dialogue data was obtained.

3. The multimodal empathy response generation method based on emotion evolution sequence backtracking learning according to claim 1 is characterized in that: The multimodal user emotion detector based on dot product attention detects the user's multi-emotion sequence by feeding the conversation history and the user's multimodal input into a dynamically constructed fully connected neural network, specifically including: The conversation history is further subdivided into long-term conversation history and short-term conversation history. First, BERT is used to encode all sentences in the long-term conversation history and all clauses in the short-term conversation history. Based on the text modal features and audio modal features of the user's current input clause, the dot product attention is used to calculate the correlation weight scores between the above two features and the text modal features and audio modal features of all sentences in the long-term and short-term conversation history. The sum of the weights is combined and then the text modal features of all sentences in the conversation history are weighted to obtain the long-term and short-term conversation history features. Finally, the user input multimodal features and the long-term and short-term conversation history features are concatenated and sent to the fully connected layer for calculation and classification to obtain the user's multi-emotion sequence.

4. The multimodal empathy response generation method based on emotion evolution sequence backtracking learning according to claim 1 is characterized in that: The empathy response generator based on the empathy strategy retrospective learning mechanism generates empathy strategy prompts through the retrospective learning mechanism, and splices the input text with the prompts and sends them to the empathy response generator to generate empathy responses, which specifically includes: An empathy strategy classifier was built using a fully connected neural network, and three possible communication mechanisms and eight possible types of dialogue behaviors were obtained. Then, the binary group consisting of the multi-emotion sequence and the empathy strategy sequence was stored in the empathy strategy backtracking module. The sequence similarity was calculated by Jaccard similarity and normalized. Then, based on three different types of backtracking learning strategies, namely dominant backtracking, comprehensive backtracking, and random backtracking, all the types and proportions of empathy strategies that can be referenced were converted into three different types of empathy communication prompts. Finally, the three empathy communication prompts were spliced ​​at the end of the user's input text, and the text was input as a whole into the fine-tuned large language model ChatGLM3 to generate three different styles of candidate replies. Finally, the candidate reply with the lowest PPL score and the highest ROUGE-L score was selected as the final reply.

Images