A multi-modal empathetic reply generation method based on tool learning

Abstract

The application provides a kind of multi-modal co-empathic reply generation method based on tool learning, it is related to co-empathic dialogue generation technical field.The method constructs the multi-modal co-empathic reply generation framework TOOL-STCIKERCONV based on tool learning, including data modification module, tool calling module and evaluation training module;The method is modified to the constructed dialogue data set, special token is inserted into the content description field Prompt of expression package, so that the multi-modal base model has the ability to think about expression package generation, and the multi-modal base model is learned to actively call the expression package generation tool, the problem of abnormal expression package sending frequency caused by the lack of independent thinking ability of large model in the field of multi-modal co-empathic dialogue generation is solved;In addition, the method uses the expression package generation tool that can be used immediately, realizes the iterative update of expression package generation model and replaces different style text-to-image model.

Description

Technical Field

[0001] This invention relates to the field of empathic dialogue generation technology, and in particular to a multimodal empathic response generation method based on tool learning. Background Technology

[0002] In today's world, chatting has become a crucial form of communication. During a chat, both parties express their emotions by exchanging text messages. However, besides text, images and emoticons can also accurately convey personal feelings and actively promote emotional exchange between the chatter. With the development of artificial intelligence technology, the two parties in a chat are no longer limited to human-to-human interactions. How to achieve human-computer interaction has become a hot topic in the field of dialogue systems. Current chatbots have shown initial success and are widely used in customer service, entertainment, healthcare, and voice assistants. However, current chatbots primarily rely on text replies and lack the ability to proactively respond with emoticons or images, indicating significant room for improvement in emotional and empathetic expression during conversations. Large language models such as Chatgpt and Qwen-vl represent powerful AI technologies that have demonstrated excellent performance in areas such as academic writing, dialogue answering, and intelligent healthcare, but research in the field of multimodal empathetic dialogue remains to be explored.

[0003] In their paper "Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements," Qian et al. conducted a comprehensive empirical study on the performance of LLMs, represented by ChatGPT, in generating empathetic responses. Based on LLMs, they proposed three targeted improvement methods (semantic similarity context learning, two-stage interactive generation, and integration with a knowledge base). The effectiveness of these three improvement methods was verified through experiments. According to the objective evaluation metrics presented in the paper, the three improvement strategies can significantly enhance the model's empathetic response capability and achieve better response results.

[0004] In their paper "A Neural Network Approach to Context-Sensitive Generation of Conversational Responses," Sordoni et al. proposed replacing the original RNN structure with a multi-layer feedforward neural network in the encoder. This allows contextual information to be introduced into the model, alleviating the problem of the model's inability to capture long-distance dependencies. As a result, the final model's response content reflects more of the chat dialogue history, improving the user's chat experience.

[0005] In their paper "Emotional Dialogue Generation with Emotion Embedding," Miao et al. proposed an empathic dialogue generation model based on seq2seq, embedding emotional information into the model's decoder to improve the ability to express empathy during dialogue. However, because the model used in the seq2seq framework at that time was an RNN, it could not maximize the effectiveness of the generative model.

[0006] In their paper "Empathetic Dialogue Generation with Pre-trained RoBERTa-GPT2 and External Knowledge," Liu et al. presented an attention-based empathic response model. They chose the autoencoder RoBERTa as the encoder, hoping to capture more contextual information through the advantages of autoencoder pre-training. Correspondingly, they selected the autoregressive GPT2 as the decoder, aiming to generate higher-quality content through the advantages of autoregressive pre-training. However, the model itself lacks any extraction and processing of user sentiment information, resulting in the overall model merely responding casually based on context, ignoring the user's emotional information.

[0007] In their paper "STICKERCONV: Generating Multimodal Empathetic Responses from Scratch," Zhang et al. proposed Agent4SC, an agent with multimodal empathetic response capabilities. It consists of six main modules: tool module, role module, planning module, memory module, behavior module, and agent management module. This agent successfully simulates human behavior using emojis in chat. The interaction data generated by this agent, after processing, filtering, and selection, yielded a multimodal empathetic dialogue dataset, STICKERCONV, which includes 12.9K dialogues, 5.8K unique emojis, and 2K different dialogue scenarios. Considering the high deployment cost of Agent4SC, a lightweight multimodal empathetic response generation framework, PEGS, was designed by distilling the agent. PEGS uses the STICKERCONV dataset for fine-tuning training to endow it with multimodal empathetic response capabilities. The base model ensures its normal text response capabilities, and general retrieval, generation, and retrieval-enhanced generation methods are used to obtain image responses. However, in practical applications, PEGS suffers from problems such as abnormal emoji response frequency and the huge cost of retraining with a large model when replacing the emoji generation model. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multimodal empathic response generation method based on tool learning. Through tool learning, a large language model learns how humans use emojis, achieving end-to-end multimodal empathic responses and simulating the frequency of human emoji sending in chat.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0010] This invention provides a multimodal empathic response generation method based on tool learning. It constructs a tool-learning-based multimodal empathic response generation framework, TOOL-STCIKERCONV, which includes a data modification module, a tool invocation module, and an evaluation and training module. The data modification module modifies the constructed dialogue dataset and calls the evaluation and training module to filter and screen the modified dataset, ensuring data quality. The tool invocation module decouples the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on tool invocation. The evaluation and training module trains the multimodal base model using the modified dataset and evaluates the training results, objectively assessing the quality and usability of the response content.

[0011] Includes the following steps:

[0012] Step 1: Build a dialogue dataset and call the data transformation module to transform the constructed dialogue dataset, dividing the dialogue dataset into training set, validation set and test set according to the proportion;

[0013] Step 1.1: Obtain the dialogue content input by the user, the description of the emojis used by the chatbot to reply, the storage path of the emojis, and the text information of the chatbot's reply to construct a dialogue dataset;

[0014] The dialogue dataset includes the following fields: "user", "function_call", "observation", and "assistant". The "user" field corresponds to the dialogue content entered by the user, including text and emoji information. The "function_call" field corresponds to the emoji description used by the chatbot to reply. The "observation" field corresponds to the storage path of the emoji used by the chatbot to reply. The "assistant" field corresponds to the text information replied by the chatbot.

[0015] Step 1.2: Call the data transformation module to transform the constructed dialogue dataset, reorganize the content of different fields in the dialogue dataset and concatenate the instructions to obtain the transformed dialogue dataset;

[0016] The specific method for data transformation of the constructed dialogue dataset is as follows:

[0017] Tokenization is performed on the dialogue data in the constructed dataset, breaking down the natural language text into the smallest lexical units, tokens. Tokens composed of words are defined as text tokens, and tokens are defined using special symbols such as <, ·, and .< / ·> The token composed of text is a special token, which is used to represent a specific meaning; the token is converted into a corresponding index token_id under the action of the word list mapping, which is used to obtain the word vector corresponding to the token in the pre-training stage;

[0018] For the user input content in the "user" field, irrelevant tokens are used to adaptively fill the beginning and end of the user input text, so that the length of the content in the "user" field is consistent after the tokens are filled. The irrelevant tokens themselves have no meaning and are only used to ensure that the dialogue dataset meets the fixed parameter specifications.

[0019] For the "function_call" field, a special token insertion method is used to add the "though" field (a model thinking field) and the "prompt" field (an emoji content description field) to the content corresponding to the "function_call" field. The "though" field includes a fixed starting token. <though>", fixed prefix, corresponding emotion, and fixed termination token"< / though> The “prompt” field contains a detailed description of the emoji, serving as a guide for generating emoji content based on the target emotion. <thought> and< / thought> This prompts you to determine whether to reply with an emoji at this stage, and specifies the content of the emoji to reply with. <thought> and< / thought> The middle part is the specific process of determining whether an emoji needs to be output. For the current dialogue scenario, if the result is that an emoji image expressing the corresponding emotion needs to be replied, and a complete description of the content of the emoji image is given, the complete description of the content of the emoji image is then converted into a token_id, and an irrelevant token is used to adaptively fill the token, so that the content length of the function_call field is consistent.

[0020] For the "observation" emoji path field, after using an irrelevant token for adaptive filling, the length of the emoji path corresponding to the emoji path field remains consistent.

[0021] For the "assistant" plain text reply field, the text reply content is converted into the corresponding token_id, and the input and output formats are properly handled to ensure that the content length corresponding to the "assistant" field is consistent.

[0022] Step 1.3: Call the evaluation training module to filter and select the modified dataset, and divide the modified dialogue dataset into training set, validation set and test set according to the proportion;

[0023] Step 2: The tool calls the module to decouple the empathic reply task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call, generating corresponding text reply content and emoji reply content respectively;

[0024] Step 2.1: The tool calls the module to decouple the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call. The multimodal base model is selected to generate text response content.

[0025] In the multimodal chatter subtask based on the multimodal base model, the multimodal base model generates text response content based on the multimodal dialogue history;

[0026] In the multimodal emoji content description generation subtask based on tool invocation, the multimodal base model determines whether to invoke tool-generated emojis based on the current dialogue scenario and dialogue history, and then invokes the emoji generation tool GenerateSticker to generate emoji reply content corresponding to the text reply content;

[0027] The multimodal base model generates text response content based on the multimodal dialogue history, while simultaneously determining whether to respond with an emoji. If the multimodal base model determines that the current scenario requires an emoji response, it enters the multimodal emoji content description generation subtask based on tool invocation, calling the emoji generation tool to directly generate the emoji response content corresponding to the text response content. If the multimodal base model determines that the current scenario does not require an emoji response, it skips the multimodal emoji content description generation subtask based on tool invocation and only generates a text response.

[0028] Step 2.2 The multimodal base model calls the emoji generation tool GenerateSticker, treats the dialogue context as a parameter input to the emoji generation tool, generates the emoji description content description field prompt, and then generates the emoji reply content corresponding to the text reply content;

[0029] The emoji generator GenerateSticker uses a plug-and-play emoji generation model, allowing users to select different emoji generation models based on their specific emoji style requirements.

[0030] Step 3: Call the evaluation and training module, use the modified dialogue dataset to train and evaluate the multimodal base model, and obtain the trained multimodal base model;

[0031] Step 3.1: Transmit the current user input X, dialogue history H, and emoji description field Q to the multimodal base model LLMs to obtain the text response Y and emoji response Z, as shown in the following formula:

[0032] Y,Z = LLMs(X,Q,H)

[0033] The loss function L for multimodal pedestal models (LLMs) is:

[0034]

[0035]

[0036] L = L y +L z

[0037] Where h represents the generated vector features of the input content after passing through the multimodal base model, θ represents a parameter that is fixed in the large model and does not participate in the update, and E LoRA y is a parameter that can be updated during training. i ,z i The i-th token generated from the Prompt field represents the description of the reply content and the emoji content, respectively, where k is the number of tokens, and L is the number of tokens. y ,L z These are the loss functions for the text response part and the loss function for generating the Prompt content description field during the training process, respectively, and L is the final model loss function;

[0038] Step 3.2: Select plain text evaluation metrics, emoji sending frequency evaluation metrics, and image-text reply content consistency evaluation metrics to evaluate the quality and usability of the reply content of the trained multimodal base model. Select the multimodal base model with the best evaluation metrics as the multimodal base model of the TOOL-STCIKERCONV multimodal empathy reply framework, which is used to generate corresponding text replies and emoji replies.

[0039] Step 4: Based on the user's input, use the trained TOOL-STCIKERCONV multimodal empathy response framework to generate corresponding text and emoji responses.

[0040] The beneficial effects of adopting the above technical solution are as follows: The multimodal empathic response generation method based on tool learning provided by this invention, by modifying the dialogue dataset and using the method of inserting special tokens, adds fragments containing special tokens to the Prompt field of the emoji content description, enabling the multimodal base model to have the ability to think about emoji generation, realizing thinking in the latent vector space, and allowing the multimodal base model to learn to actively call emoji generation tools, thus solving the problem of abnormal emoji sending frequency caused by the lack of autonomous thinking ability of large models in the field of multimodal empathic dialogue generation; In addition, this invention utilizes a plug-and-play emoji generation tool to realize the iterative update of the emoji generation model and the replacement of text-to-image models with different styles. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a multimodal empathetic dialogue including emojis provided in an embodiment of the present invention;

[0042] Figure 2 This is a diagram of the TOOL-STCIKERCONV multimodal empathy response framework provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the tool invocation in the multimodal empathy response generation provided in an embodiment of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0045] Empathic response dialogue was first proposed in Facebook's Empathic Dialogue Benchmark in 2019 and is one of the core tasks in the field of emotion. In the traditional field of empathic response, researchers have mainly focused on the plain text domain. Psychological research has shown that introducing other modalities of content, such as images and emojis, into empathic dialogue can achieve better empathic effects. Therefore, this embodiment simulates normal human chat to realize a multimodal empathic response mode of "text + emojis," enhancing the user's interactive experience. Multimodal empathic dialogue, such as... Figure 1 As shown.

[0046] There are two main challenges in the field of multimodal empathic dialogue generation: First, there is a lack of multimodal empathic dialogue datasets. Most current empathic dialogue tasks are purely text-based and lack large-scale multimodal dialogue datasets. Second, most current dialogue models lack the ability to proactively send emojis, demonstrating a lack of consideration for how to send emojis, resulting in dull dialogue content and abnormal emoji sending frequency.

[0047] To address the above issues, this embodiment provides a multimodal empathic response method based on tool learning. It constructs a tool-learning-based TOOL-STCIKERCONV multimodal empathic response generation framework, including a data manipulation module, an evaluation and training module, and a tool invocation module, such as... Figure 2 As shown, the data transformation module is used to transform the constructed dialogue dataset and calls the evaluation and training module to screen and filter the transformed dataset to ensure data quality; the tool invocation module decouples the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on tool invocation; the evaluation and training module uses the transformed dataset to train the multimodal base model and evaluates the training results to objectively assess the quality and usability of the response content.

[0048] The tool-learning-based multimodal empathic response generation method includes the following steps:

[0049] Step 1: Build a dialogue dataset and call the data transformation module to transform the constructed dialogue dataset, dividing the dialogue dataset into training set, validation set and test set according to the proportion;

[0050] Step 1.1: Obtain the dialogue content input by the user, the description of the emojis used by the chatbot to reply, the storage path of the emojis, and the text information of the chatbot's reply to construct a dialogue dataset;

[0051] The dialogue dataset includes the fields “user”, “function_call”, “observation”, and “assistant”. The “user” field corresponds to the dialogue content entered by the user, including text and emoji information. The “function_call” field corresponds to the emoji description used by the chatbot to reply. The “observation” field corresponds to the storage path of the emoji used by the chatbot to reply. The “assistant” field corresponds to the text information replied by the chatbot.

[0052] Step 1.2: Call the data transformation module to transform the constructed dialogue dataset, reorganize the content of different fields in the dialogue dataset and concatenate the instructions to obtain the transformed dialogue dataset;

[0053] The specific method for data transformation of the constructed dialogue dataset is as follows:

[0054] Tokenization is performed on the dialogue data in the constructed dataset, breaking down the natural language text into the smallest lexical units, tokens. Tokens composed of words are defined as text tokens, and tokens are defined using special symbols such as <, ·, and .< / ·> The token composed of text is a special token, which is used to represent a specific meaning; the token is converted into a corresponding index token_id under the action of the word list mapping, which is used to obtain the word vector corresponding to the token in the pre-training stage;

[0055] For the user input content in the "user" field, irrelevant tokens are used to adaptively fill the beginning and end of the user input text, so that the length of the content in the "user" field is consistent after the tokens are filled. The irrelevant tokens themselves have no meaning and are only used to ensure that the dialogue dataset meets the fixed parameter specifications.

[0056] For the "function_call" field, a special token insertion method is used to add the "though" field (a model thinking field) and the "prompt" field (an emoji content description field) to the content corresponding to the "function_call" field. The "though" field includes a fixed starting token. <though> ", fixed prefix, corresponding emotion, and fixed termination token"< / though> The “prompt” field contains a detailed description of the emoji, serving as a guide for generating emoji content based on the target emotion. <thought> and< / thought>This prompts you to determine whether to reply with an emoji at this stage and what the reply emoji should contain; <thought> and< / thought> The middle part is the specific process of determining whether an emoji needs to be output. For the current dialogue scenario, if the result is that an emoji image expressing the corresponding emotion needs to be replied, and a complete description of the content of the emoji image is given, the complete description of the content of the emoji image is then converted into a token_id, and an irrelevant token is used to adaptively fill the token, so that the content length of the function_call field is consistent.

[0057] For the "observation" emoji path field, a similar method to that used for the "user" field is adopted: after adaptive filling with an irrelevant token, the length of the emoji path corresponding to the emoji path field remains consistent.

[0058] For the "assistant" plain text reply field, a similar processing method as that used for the "function_call" field is adopted to convert the text reply content into the corresponding token_id and to properly handle the input and output formats so that the content length corresponding to the "assistant" field is consistent.

[0059] Step 1.3: Call the evaluation training module to filter and select the modified dataset, and divide the modified dialogue dataset into training set, validation set and test set according to the proportion;

[0060] In this embodiment, the preprocessed dialogue dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0061] Step 2: The tool calls the module to decouple the empathic reply task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call, generating corresponding text reply content and emoji reply content respectively;

[0062] Step 2.1: The tool calls the module to decouple the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call. The multimodal base model is selected to generate text response content.

[0063] In the multimodal chatter subtask based on the multimodal base model, the multimodal base model generates text response content based on the multimodal dialogue history;

[0064] In the multimodal emoji content description generation subtask based on tool invocation, the multimodal base model determines whether to invoke tool-generated emojis based on the current dialogue scenario and dialogue history, and then invokes the emoji generation tool GenerateSticker to generate emoji reply content corresponding to the text reply content;

[0065] The multimodal base model generates text responses based on the multimodal dialogue history while simultaneously determining whether to reply with an emoji. If the multimodal base model determines that an emoji response is required in the current scenario, it proceeds to the tool-based multimodal emoji content description generation subtask, directly generating the corresponding emoji response to the text response. If the multimodal base model determines that an emoji response is not required in the current scenario, it skips the tool-based multimodal emoji content description generation subtask and only generates a text response. A diagram illustrating the tool call is shown below. Figure 3 As shown;

[0066] Step 2.2 The multimodal base model calls the emoji generation tool GenerateSticker, treats the dialogue context as a parameter input to the emoji generation tool, generates the emoji description content description field prompt, and then generates the emoji reply content corresponding to the text reply content;

[0067] The emoji generator GenerateSticker uses a plug-and-play emoji generation model, allowing users to select different emoji generation models based on their specific emoji style requirements.

[0068] In this embodiment, Stable Diffusion version 1.6 is used as the emoji generation tool. It can also be replaced with a higher version of Stable Diffusion, or other text-to-image models such as Midjourney.

[0069] Step 3: Call the evaluation and training module, use the modified dialogue dataset to train and evaluate the multimodal base model, and obtain the trained multimodal base model;

[0070] Step 3.1: Transmit the current user input X, dialogue history H, and emoji description field Q to the multimodal base model LLMs to obtain the text response Y and emoji response Z, as shown in the following formula:

[0071] Y,Z = LLMs(X,Q,H)

[0072] The loss function L for multimodal pedestal models (LLMs) is:

[0073]

[0074]

[0075] L = L y +L z

[0076] Where h represents the generated vector features of the input content after passing through the multimodal base model, θ represents a parameter that is fixed in the large model and does not participate in the update, and E LoRA y is a parameter that can be updated during training. i ,z i The i-th token generated from the Prompt field represents the description of the reply content and the emoji content, respectively, where k is the number of tokens, and L is the number of tokens. y ,L z These are the loss functions for the text response part and the loss function for generating the Prompt content description field during the training process, respectively, and L is the final model loss function;

[0077] In this embodiment, the training strategy uses the LLaMA-factory framework with LoRA parameter fine-tuning for efficient model training, ensuring that the original parameters of the multimodal base model remain fixed and training is only performed on newly added parameters. Qwen-VL is used as the pre-trained model, with a hidden layer dimension of 3584 and input and output vocabularies of 15K each. The initial learning rate is 1e-5, the optimizer is Adam, the batch size is set to 4, and the annealing step count is 5K. All experiments are performed on eight NVIDIA A6000 GPUs, and convergence takes approximately 72 hours.

[0078] Step 3.2: Select plain text evaluation metrics, emoji sending frequency evaluation metrics, and image-text reply content consistency evaluation metrics to evaluate the quality and usability of the reply content of the trained multimodal base model. Select the multimodal base model with the best evaluation metrics as the multimodal base model of the TOOL-STCIKERCONV multimodal empathy reply framework, which is used to generate corresponding text replies and emoji replies.

[0079] In this embodiment, Rouge-L, BERT-score, BLEU-1 / 2 / 3, METEOR, CIDEr, and Dist-1 / 2 / 3 / 4 are selected as evaluation metrics for plain text; Accuracy, F1, Recall, and Precision are selected as evaluation metrics for emoji sending frequency; Dist-1 and Dist-2 are selected as consistency evaluation metrics for the emoji content description field prompt; Precision, recall, F1, and clip score are selected as multimodal consistency evaluation metrics; and empathy-txt, empathy-mm, and consistency are selected to evaluate the quality and usability of the base model's response content.

[0080] Step 4: Based on the user's input, use the trained TOOL-STCIKERCONV multimodal empathy response framework to generate corresponding text and emoji responses.

[0081] This embodiment is based on the original dataset from the open-source STICKERCONV, focusing on the chat content, emoji descriptions, and sentiment expression fields. Based on the constructed dialogue dataset, the "function_call" field is supplemented with the model thinking field "though" and the emoji content description field "prompt". The "though" field includes a fixed starting token. <though> The fixed prefix "I need to generate a sticker to express", the corresponding emotion, and the fixed termination token.< / though> The “prompt” field contains a detailed description of the emoji, serving as a guide for generating emoji content based on the target emotion. A specific example of the dialogue dataset is shown in Table 1.

[0082] Table 1 Example of dialogue data

[0083]

[0084] In this embodiment, three large language models, Vicuna, ChatGLM3, and Qwen-VL, were selected as base models. Rouge-L, BERT-score, BLEU-1 / 2 / 3, METEOR, CIDEr, and Dist-1 / 2 / 3 / 4 were selected as evaluation metrics for plain text; Accuracy, F1, Recall, and Precision were selected as evaluation metrics for emoji sending frequency; dist-1 and dist-2 were selected as evaluation metrics for prompt consistency; precision, recall, F1, and clip score were selected as evaluation metrics for multimodal consistency; and empathy-txt, empathy-mm, and consistency were selected as evaluation metrics for large models. The evaluation results are shown in Table 2-6. Qwen-VL achieved better performance in most experiments.

[0085] Table 2. Experimental Results Regarding Text

[0086]

[0087] Table 2 shows that the TOOL-Qwen-VL model achieved the best results. Regarding specific metrics, the three base models showed little difference in the Dist series evaluation metrics, therefore, the Dist series metrics are not very meaningful. The TOOL-Qwen-VL model showed a significant advantage in the BLEU-1 / 2 / 3 metrics, with the performance improvement compared to the TOOL-Vincuna model increasing from 11.63% to 29.41%, demonstrating the TOOL-Qwen-VL model's significant advantage in the standardization of generated content. The TOOL-Qwen-VL model improved the Rouge-L metric by 14.29% compared to TOOL-Vincuna, and its meaning is similar to the BLEU value, also demonstrating the standardization of the TOOL-Qwen-VL model in terms of generated content. In terms of the METEOR metric, the TOOL-Qwen-VL model improved by 22.22% compared to TOOL-Vincuna, demonstrating the standardization of the TOOL-Qwen-VL model in vocabulary-level content. Finally, in terms of the CIDEr metric, the TOOL-Qwen-VL model improved by 50% compared to the TOOL-Vicuna model, indicating that the TOOL-Qwen-VL model has better intent recognition and contextual understanding capabilities in the multimodal domain, and the generated image descriptions are also closest to the reference content.

[0088] The evaluation of the base model in this embodiment regarding the frequency of emoji sending is shown in Table 3:

[0089] Table 3. Experimental Results of Emoji Sending Frequency

[0090]

[0091] As shown in Table 3, the TOOL-Qwen-VL model achieved the best results. Specifically, the TOOL-Qwen-VL model demonstrated significant advantages in Accuracy, F1, Recall, and Precision, indicating that it best reflects real-world emoji sending patterns. Furthermore, it exhibited the smallest error in the core metric pred_freq, which is close to the standard frequency of 0.42 (a US survey showed that college students respond to emojis at a frequency of 0.42 during chat). This further proves that the TOOL-Qwen-VL model most closely matches the reference content in terms of emoji sending frequency distribution, best simulating the frequency of emoji sending in normal chat.

[0092] This embodiment performs consistency evaluation on the emoji content description field "prompt" and multimodal consistency evaluation on the base model. The experimental results of the consistency evaluation of the emoji content description field "prompt" are shown in Table 4, and the results of the multimodal consistency evaluation are shown in Table 5.

[0093] Table 4. Results of the Prompt Consistency Experiment

[0094]

[0095] As shown in Table 4, the three models achieved relatively similar results in terms of consistency of the Prompt field of emoji content description, but the TOOL-Qwen-VL model has some advantages and is closest to the ideal Dist-1 and Dist-2 values.

[0096] Table 5. Results of Multimodal Consistency Experiment

[0097]

[0098] Table 5 shows that the TOOL-Qwen-VL model improves upon TOOL-ChatGLM3 in Precision, Recall, and F1 by 11.03%, 25.07%, and 8.36%, respectively, demonstrating high text-image relevance in the TOOL-Qwen-VL model's responses. Finally, in the clip-score, the TOOL-Qwen-VL model improves upon TOOL-Vicuna by 2.09%, further proving the high text-image consistency of the emojis in the TOOL-Qwen-VL model's responses, indicating an improved ability to understand user emotions and empathetic responses.

[0099] This embodiment performs a large-scale model scoring evaluation on the base model, and the evaluation results are shown in Table 6.

[0100] Table 6 Scoring of the Major Models

[0101]

[0102] As shown in Table 6, the TOOL-Qwen-VL model improves textual empathy by 2.04% and multimodal empathy by 1.32% compared to TOOL-Vicuna, demonstrating that under the comprehensive judgment of the larger model, the TOOL-Qwen-VL model's empathy ability is improved compared to TOOL-Vicuna. Furthermore, in terms of consistency, the TOOL-Qwen-VL model improves by 1.25% compared to TOOL-Vicuna, indicating that the TOOL-Qwen-VL model's responses maintain better contextual consistency and have a lower probability of jumping to unrelated topics.

[0103] To verify the effectiveness of the large model thinking ability proposed in this paper, an ablation experiment was conducted in this embodiment, and multimodal consistency was selected for effect comparison. The results are shown in Table 7.

[0104] Table 7 Ablation Experiment

[0105]

[0106]

[0107] As shown in Table 7, the TOOL-Qwen-VL model incorporating tool-based thinking capabilities improved precision, recall, F1 score, and clip-score by 0.92%, 25.07%, 8.36%, and 3.69% respectively compared to the control model without tool-based thinking capabilities. The most significant improvement in recall was observed after incorporating tool-based thinking capabilities, demonstrating a substantial enhancement in multimodal generation capabilities and making the generated text and images more closely resemble actual chat scenarios. Furthermore, the clip-score also showed a significant improvement, indicating that introducing tool-based thinking capabilities can improve the consistency of generated text and images, i.e., better understand user emotions and generate corresponding emotional responses, thus enhancing the empathy throughout the chat process.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multimodal empathic response generation method based on tool learning, characterized in that: A tool-based learning-based multimodal empathic response generation framework, TOOL-STCIKERCONV, is constructed, comprising a data transformation module, a tool invocation module, and an evaluation and training module. The data transformation module modifies the constructed dialogue dataset and calls the evaluation and training module to filter and select the modified dataset, ensuring data quality. The tool invocation module decouples the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on tool invocation. The evaluation and training module trains the multimodal base model using the modified dataset and evaluates the training results, objectively assessing the quality and usability of the response content. Includes the following steps: Step 1: Build a dialogue dataset and call the data transformation module to transform the constructed dialogue dataset, dividing the dialogue dataset into training set, validation set and test set according to the proportion; Step 2: The tool calls the module to decouple the empathic reply task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call, generating corresponding text reply content and emoji reply content respectively; Step 3: Call the evaluation and training module, use the modified dialogue dataset to train and evaluate the multimodal base model, and obtain the trained multimodal base model; Step 4: Based on the user input, use the trained TOOL-STCIKERCONV multimodal empathy response framework to generate corresponding text and emoji responses; Step 3 includes: Step 3.1: Transmit the current user input content X, dialogue history H, and emoji content description field Q to the multimodal base model LLMs to obtain the text response content. and emoji replies As shown in the formula below: The loss function L for multimodal pedestal models (LLMs) is: in, The generated vector features are the input content after passing through the multimodal base model. These are parameters that are fixed in the large model and do not participate in the update. These are parameters that can be updated during training. , The i-th token generated from the Prompt field represents the reply content and the emoji content description, respectively, where k is the number of tokens. , The loss functions are the text response portion and the Prompt content description field generation portion during the training process, respectively. This is the loss function for the final model; Step 3.2: Select plain text evaluation metrics, emoji sending frequency evaluation metrics, and image-text reply content consistency evaluation metrics to evaluate the quality and usability of the reply content of the trained multimodal base model. Select the multimodal base model with the best evaluation metrics as the multimodal base model of the TOOL-STCIKERCONV multimodal empathy reply framework, which is used to generate corresponding text replies and emoji replies.

2. The multimodal empathic response generation method based on tool learning according to claim 1, characterized in that: Step 1 includes: Step 1.1: Obtain the dialogue content input by the user, the description of the emojis used by the chatbot to reply, the storage path of the emojis, and the text information of the chatbot's reply to construct a dialogue dataset; Step 1.2: Call the data transformation module to transform the constructed dialogue dataset, reorganize the content of different fields in the dialogue dataset and concatenate the instructions to obtain the transformed dialogue dataset; Step 1.3: Call the evaluation training module to filter and select the modified dataset, and divide the modified dialogue dataset into training set, validation set and test set according to the proportion.

3. The multimodal empathic response generation method based on tool learning according to claim 2, characterized in that: The dialogue dataset mentioned in step 1.1 includes the following fields: "user", "function_call", "observation", and "assistant". The "user" field corresponds to the dialogue content input by the user, including text and emoticon information. The "function_call" field corresponds to the emoticon description used by the chatbot to reply. The "observation" field corresponds to the storage path of the emoticons used by the chatbot to reply. The "assistant" field corresponds to the text information replied by the chatbot.

4. The multimodal empathic response generation method based on tool learning according to claim 3, characterized in that: The specific method for data transformation of the constructed dialogue dataset described in step 1.2 is as follows: Tokenization is performed on the dialogue data in the constructed dataset, breaking down the natural language text into the smallest lexical units, tokens. Tokens composed of words are defined as text tokens, and tokens are defined using special symbols such as <, ·, and .< / ·> The token composed of text is a special token, which is used to represent a specific meaning; the token is converted into a corresponding index token_id under the action of the word list mapping, which is used to obtain the word vector corresponding to the token in the pre-training stage; For the user input content contained in the "user" field, irrelevant tokens are used to adaptively fill the beginning and end of the user input text, so that the length of the content contained in the "user" field is consistent after the tokens are filled. The irrelevant tokens themselves have no meaning and are only used to ensure that the dialogue dataset meets the fixed parameter specifications. For the "function_call" field, a special token insertion method is used to add a "though" field (representing the model's thought process) and a "prompt" field (representing the content of the emoji) to the content corresponding to the "function_call" field. The "though" field includes a fixed starting token. <though> ", fixed prefix, corresponding emotion, and fixed termination token"< / though> The "prompt" field contains a detailed description of the emoji, serving as a guide for generating emoji content based on the target emotion. <thought> and< / thought> This prompts you to determine whether to reply with an emoji at this stage, and specifies the content of the emoji to reply with. <thought> and< / thought> The middle part is the specific process of determining whether an emoji needs to be output. For the current dialogue scenario, if the result is that an emoji image expressing the corresponding emotion needs to be replied, and a complete description of the content of the emoji image is given, the complete description of the content of the emoji image is then converted into a token_id, and an irrelevant token is used to adaptively fill the token, so that the content length of the function_call field is consistent. For the "observation" emoji path field, after using an irrelevant token for adaptive filling, the length of the emoji path corresponding to the emoji path field remains consistent. For the "assistant" plain text reply field, convert the text reply content into the corresponding token_id, and handle the input and output formats to ensure that the content length corresponding to the "assistant" field is consistent.

5. The multimodal empathic response generation method based on tool learning according to claim 4, characterized in that: Step 2 includes: Step 2.1: The tool calls the module to decouple the empathic response task into a multimodal chat subtask based on a multimodal base model and a multimodal emoji content description generation subtask based on the tool call. The multimodal base model is selected to generate text response content. Step 2.2: The multimodal base model calls the emoji generation tool. The dialog context is treated as a parameter input to the emoji generation tool, which generates an emoji description field called "prompt", and then generates emoji reply content corresponding to the text reply content.

6. The multimodal empathic response generation method based on tool learning according to claim 5, characterized in that: The specific method for step 2.1 is as follows: In the multimodal chatter subtask based on the multimodal base model, the multimodal base model generates text response content based on the multimodal dialogue history; In the multimodal emoji content description generation subtask based on tool invocation, the multimodal base model determines whether to invoke tool-generated emojis based on the current dialogue scenario and dialogue history, and then invokes the emoji generation tool. Generate emoji replies corresponding to the text reply content; The multimodal base model generates text response content based on the multimodal dialogue history while simultaneously determining whether to reply with an emoji. If the multimodal base model determines that the current scenario requires an emoji reply, it enters the multimodal emoji content description generation subtask based on tool invocation, calling the emoji generation tool to directly generate the emoji response content corresponding to the text response content. If the multimodal base model determines that the current scenario does not require an emoji reply, it skips the multimodal emoji content description generation subtask based on tool invocation and only generates a text response.

7. The multimodal empathic response generation method based on tool learning according to claim 6, characterized in that: The emoji generation tool described in step 2.2 Use a plug-and-play emoji generation model, and select different styles of emoji generation models according to the user's different emoji style requirements.

Images