Psychological accompanying dialogue method based on multi-agent collaboration and storage medium

Abstract

The invention relates to the field of natural language understanding in an artificial intelligence technology man-machine dialogue system, in particular to a psychological accompanying dialogue method based on multi-agent collaboration and a storage medium. The psychological accompanying dialogue with individuation, high naturalness and accurate emotion recognition is provided, the user experience is effectively improved, and the psychological support requirement of the user is met. According to the main scheme, the method comprises the steps of performing multi-mode emotion evaluation on user input to obtain user emotion state description; retrieving related dialogue states and external knowledge from a shared memory module according to user questions and user emotional state description; according to the retrieved dialogue state and external knowledge, the strategy planning module generates a reply strategy to obtain a strategy group; the dialogue content generation module calls a corresponding vertical domain agent according to the generated reply strategy to generate specific reply content; and storing the user questions and model replies in the current round into the short-term memory of the shared memory module, and updating the long-term memory. The method is used for AI dialogues.

Description

Technical Field

[0001] The present invention belongs to the field of natural language understanding in artificial intelligence technology human-computer dialogue systems, and specifically relates to a psychological companionship dialogue method and storage medium based on multi-agent collaboration. Background Art

[0002] In recent years, the mental health issues of contemporary teenagers have attracted widespread attention. With the increase of social pressure and the rapid development of information technology, the factors affecting the mental health of teenagers have become more complex and diverse, including academic pressure, interpersonal relationships, family environment, etc. Teenagers with psychological problems are often reluctant to take the initiative to confide in psychologists or seek help. Therefore, it has become an urgent need to use advanced artificial intelligence technology to monitor the psychological state on campus and provide intelligent psychological counseling services for risk subjects. With the development of the field of artificial intelligence, there are currently multiple deep learning technologies combined with dialogue applications related to the psychological field, as follows:

[0003] Patent CN 111564202B "Psychological counseling method, psychological counseling terminal and storage medium based on human-computer dialogue" mainly uses a human-computer dialogue system and an emotion recognition model to conduct psychological counseling for users. This method uses the target voice and dialogue information input by the user to identify the dialogue emotions from the two dimensions of emotional valence and emotional arousal, and obtains and executes a response plan that matches the user through a psychological conversation technology model. However, the response plan is obtained by matching the speech template and communication steps of the preset target event in the database, which will result in the response strategy being not flexible and targeted enough.

[0004] Patent CN 117932041 A "Emotional support dialogue generation method, system and device based on thought chain reasoning" mainly uses thought chain reasoning to assist in the generation of emotional support dialogue. This method obtains the user's emotional state information and response strategy by constructing emotional thought chains and strategic thought chains. However, the emotional information provided by the dialogue text is limited, and it is necessary to combine tone, expression, etc. to obtain more accurate analysis results. This method only involves dialogue history information when performing emotional thought chain reasoning, and does not perform real-time analysis of the user's voice intonation and facial expressions. In addition, this method only uses prompt words to drive the large model to generate responses. Although it reduces the training cost, it is difficult to guarantee the execution effect of the strategy.

[0005] Patent CN 114999610 A "Method for constructing a dialogue system with emotion perception and support based on deep learning" mainly uses a deep learning model to construct an emotion perception and support dialogue system. This method generates adaptive dialogue strategies by introducing a large model, and replies to user dialogues or makes recommendations based on the generated strategies. However, this method can only perform a one-time strategy selection, and can only select one strategy in a single round, and does not combine the user's psychological state information to generate dialogues, which is insufficient for emotional support for users. In addition, the large model used by this method for strategy prediction is an offline version obtained through pre-training, and cannot be updated in real time based on dialogue feedback, resulting in it not being able to meet the individual needs of users in a real dialogue environment.

[0006] Patent CN117972075A "Method for generating emotional dialogues in collaboration with mental and language agents" mainly utilizes the collaborative operation of multiple agents such as state analysis, strategy planning, and language generation to improve the system's ability to conduct emotional dialogues with users, and ensures the coherence and personality consistency of responses by adding role-setting prompts. In addition, the method also uses similarity retrieval to match training corpus to generate response strategies. However, since the training corpus is relatively fixed, strategy generation cannot be innovated according to actual conditions, resulting in insufficient diversity and richness of the strategy group. In addition, the method requires a single model to complete the generation task of multiple strategies when generating reply texts, which makes it difficult to achieve the desired effect when executing more complex reply strategies, and it is difficult to perform subsequent model optimization based on feedback.

[0007] Most of the current AI emotional dialogue methods use similarity matching or simple reasoning for strategy planning, and are basically driven by prompt words to generate a large language model for reply text. Although this reduces training and operating costs, it also significantly reduces the diversity and targeted effectiveness of emotional dialogue. Therefore, this patent proposes to build a multi-agent system, which uses memory sharing between agents to assist the strategy planning agent, each dialogue state update agent, and each dialogue generation agent to collaboratively generate replies, conduct high-quality multi-round dialogues with users, and provide them with psychological counseling and emotional support. At the same time, multimodal evaluation is used to conduct user psychological profiling, thereby significantly improving the accuracy of user emotion analysis and improving conversation capabilities in psychological counseling scenarios. In addition, an online reinforcement learning method is introduced to train agents to plan response strategies and improve the flexibility and richness of responses. Summary of the invention

[0008] The purpose of this invention is to solve the problem of how to build a psychological companionship dialogue system that can provide personalization, high naturalness and accurate emotion recognition by integrating multi-agent collaborative work and multimodal analysis technology, so as to effectively improve user experience and meet their psychological support needs.

[0009] In order to achieve the above purpose, the present invention adopts the following technical means:

[0010] A psychological companionship dialogue method based on multi-agent collaboration includes the following steps:

[0011] Step 1: Perform multimodal emotion evaluation on user input to obtain a description of the user's emotional state;

[0012] Step 2: According to the user's question and the user's emotional state description, relevant dialogue states and external knowledge are retrieved from the shared memory module;

[0013] Step 3: Based on the retrieved dialogue state and external knowledge, the strategy planning module generates a response strategy to obtain a strategy group;

[0014] Step 4: The dialogue content generation module calls the corresponding vertical domain agent according to the response strategy generated in step 3 to generate specific response content;

[0015] Step 5: Store the user questions and model responses of this round into the short-term memory of the shared memory module, and update the long-term memory.

[0016] In the above scheme, step 1 includes the following steps:

[0017] Step 1.1: Perform BERT encoding on the text input by the user to obtain a semantic vector;

[0018] Step 1.2, extracting features from the voice and image input by the user to obtain an emotion feature vector;

[0019] Step 1.3, combining the semantic vector obtained in step 1.1 and the emotional feature vector obtained in step 2.2, and obtaining a description of the user's emotional state through a multimodal emotion evaluation model;

[0020] In the above scheme, step 2 includes the following steps:

[0021] Step 2.1: Use the BERT encoder to encode the user’s question and the result of step 1 to obtain a retrieval vector. At this point, the retrieval vector will be used to query the vector database in the shared memory module, which stores the vector representation of the conversation history and external knowledge.

[0022] Step 2.2: Based on the result of step 2.1, semantic search is performed on the vector database to obtain relevant dialogue state information and external knowledge;

[0023] Step 2.3: Based on the user input, perform a literal search on the relational database and the graph database to obtain the dialog state information having the same keywords as the user input;

[0024] Step 2.4: weighted sort the results of step 2.2 and step 2.3, and select the top k most relevant texts;

[0025] In the above solution, the shared memory module implementation includes the following steps:

[0026] Step A1, initializing the shared memory module, including the settings of short-term memory and long-term memory;

[0027] Step A2: Storing context and conversation history in short-term memory;

[0028] Step A3: Storing external knowledge documents and dialog states in long-term memory;

[0029] Step A4: When the conversation is in progress, the short-term memory content is called and spliced ​​with the input;

[0030] Step A5: After the response generation is completed, the user input and system output of the current round are written into the conversation history record;

[0031] Step A6: The dialogue state includes the dialogue history record and the processed multi-level dialogue memory;

[0032] Step A7: Multi-level conversation memory includes the original conversation at the lowest level, paragraph summary at the second level, event summary at the third level, emotion summary at the fourth level, and user portrait at the highest level.

[0033] Step A8: After each round of dialogue is generated, call the relevant agent to update the dialogue state of each layer;

[0034] Step A9: After the update is completed, the data cache and management module is called to process, including the update of the graph database, relational database and vector database.

[0035] In the above scheme, step 3 includes the following steps:

[0036] Step 3.1, the strategy planning module generates multiple possible response strategies based on the input dialogue state;

[0037] Step 3.2: Input the result of step 3.1 into the dialogue content generation module to generate specific reply content. The reply content of the optimal strategy is directly used as the model reply for this round of dialogue;

[0038] Step 3.3: Pass the specific response content generated in step 3.2 back to the strategy planning module, use the preference model to sort the results of step 3.1, determine the optimal strategy based on the sorting, form a strategy group, and the strategy planning module performs strategy iteration and parameter update.

[0039] In the above scheme, step 3.1 specifically includes the following steps:

[0040] Step 3.1.1, Strategy Planning Module π t Generate multiple strategies based on the input dialogue state, among which the strategy judged to be the best is recorded as The complete strategy sequence is recorded as The strategy planning module is referred to as the agent;

[0041] Step 3.1.2: The conversation state is called from the shared memory, including the conversation history, conversation event summary, user psychological state, user emotion, and user portrait;

[0042] Each policy output needs to include the name of the agent being called and the information passed to it. The complete input prompt template is:

[0043] {

[0044] The user question in this round is: [User question], and the user’s current mood is [Observed mood];

[0045] Current conversation situation: [memory summary], relevant knowledge: [Useful knowledge];

[0046] The agents in this system are: [Agent Name: {Usage, Input Parameters}, …]

[0047] You are an intelligent scheduling agent. Based on the above information, you need to plan how to use one or more [AgentNames] to respond to the user's questions in this round, and provide emotional support to relieve their negative emotions. Please output your response plan in the following format in order. The plan should include at least one strategy.

[0048] Used agent:

[0049] Task description:

[0050] }.

[0052] In the above scheme, step 3.3 specifically includes the following steps:

[0053] 3.3.1. Use the preference model to sort the strategy sequence results generated in step 3.1 and determine the optimal strategy. The preference model is based on the Bradley-Terry model, and its preference score calculation formula is:

[0054] P(a 1 >a 2 |x,a 1 , a 2 )=σ(r * (x,a1 )-r * (x,a 2 ))

[0055] where r * (x, a) is its inherent reward function, σ(·) is the Sigmoid function, x represents the input dialogue state, a represents the action, and a 1 , a 2 Represents different actions; a 1 >a 2 Indicates action 1 is considered better than action a 2 superior or preferred;

[0056] Step 3.3.2: Add the preference ranking sequence to the strategy preference dataset , and update the parameters of the strategy planning module through direct preference optimization. The optimized loss function is:

[0057]

[0058] where a w =a win , a l =a lose , a w Indicates an action that results in a win or victory, a l Indicates an action that results in loss or failure;

[0059] θ represents the learned policy parameters, π θ represents a parameterized policy, where θ is a set of parameters;

[0060] π0 represents a reference strategy;

[0061] σ(·) is the sigmoid function, which maps real values ​​to the interval (0, 1)(0, 1);

[0062] η is a hyperparameter;

[0063] π0(a * |x) is to take action a in state x * The probability of

[0064] Step 3.3.3: Select the optimal new strategy π by maximizing expected return and considering the diversity of strategies t+1 , the optimal strategy selection process is:

[0065]

[0066] where argmax πIt means choosing the strategy that can maximize the expected return among all possible strategies π;

[0067] Represents the expectation of all possible dialogue states x;

[0068] It represents the expectation of all possible actions a given the strategy π and the dialogue state x;

[0069] η is a hyperparameter used to balance strategy selection and strategy diversity;

[0070] Represents strategy π t The KL divergence relative to the initial policy π0 is used to measure the policy π t The difference between π0.

[0071] In the above scheme, step 4 includes the following steps:

[0072] Step 4.1, the scheduling agent calls the required vertical field agent according to the response strategy;

[0073] Step 4.2: The vertical domain agent selected in step 4.1 generates a response content of a specific strategy based on the relevant information passed in;

[0074] Step 4.3: The scheduling agent calls the vertical domain agent to continue generating the response content of the next strategy, which is spliced ​​into the result of step 4.2;

[0075] Step 4.4: Repeat steps 4.1 to 4.3 until all strategies in the strategy group are used and a complete model response is obtained for this round.

[0076] In the above scheme, step 5 includes the following steps:

[0077] Step 5.1: Store the user's questions and model responses in this round into the conversation history in short-term memory;

[0078] Step 5.2: Update the conversation state in long-term memory according to the update result of step 5.1, including conversation history summary, event summary, emotion summary and user portrait;

[0079] Step 5.3: Store the updated results of step 5.2 into the vector database, relational database and graph database.

[0080] The present invention also provides a storage medium, and when a processor executes a program in the storage medium, the method of psychological companionship dialogue based on multi-agent collaboration is implemented.

[0081] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0082] The present invention introduces multi-agent collaboration and reinforcement learning methods based on human feedback to solve the problems of unstable quality, difficulty in optimization and lack of "humanity" of responses generated by a single large dialogue model. The responses generated by the psychological companionship system are more relevant to the content of the dialogue, and the response model of a single generation strategy can be optimized in a targeted manner. The system can also adjust to a better dialogue mode based on user feedback during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Block diagram of the psychological companionship dialogue system based on multi-agent collaboration;

[0084] Figure 2 Shared memory module;

[0085] Figure 3 Conversation status;

[0086] Figure 4 Knowledge retrieval module;

[0087] Figure 5 Concept map of the strategic planning module;

[0088] Figure 6 The framework of the strategy planning agent running (black arrow) and training (orange arrow) in round t;

[0089] Figure 7 Dialogue content generation module

[0090] Figure 8 Call vertical domain agents to generate responses. DETAILED DESCRIPTION

[0091] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0092] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0093] The present invention relates to a multi-agent collaborative dialogue system in the field of psychological companionship. Figure 1 As shown in the figure, it includes shared memory module, information retrieval module, strategy planning module, and dialogue content generation module. Next, we will introduce the modules in detail:

[0094] 1. Shared Memory Module

[0095] Technical issues to be considered: Multi-agent collaboration requires the transmission of information; dialogue responses require the use of contextual information; and the system needs to store a large amount of diverse data after it is put into use.

[0096] Solution:

[0097] Different data types are stored in order, divided into long and short-term memory, to improve retrieval speed;

[0098] All memories generated by the operation of the agent are stored in this module. If the agent needs to call it during operation, it can directly access all memories to prevent information imbalance between agents and save the transmission step.

[0099] Introduce multi-layer conversation states to perform hierarchical summarization and information extraction on the original conversation; use three data structures to store multi-level states to improve retrieval efficiency.

[0100] The function of this module is to save the conversation history, external knowledge and useful information generated during the operation of the system for each agent to call. Shared memory is divided into short-term memory and long-term memory. Short-term memory content includes context and conversation history, both of which are stored in text form. The context is usually set by the user when starting the conversation, such as roles, scene presets, question background, etc. The context generally does not change in multiple rounds of conversations, and is used to improve the model's understanding of the conversation scenario, thereby improving the language quality and topic relevance of the system's responses. The conversation history records store the complete original conversation and provide historical information to each agent, so that the model's response can be smoothly connected with the previous text. If the current conversation round is the tth round, the conversation history records are the conversation data of the previous t-1 rounds, where each round includes user questions, system responses, and corresponding strategy groups. During the conversation, the system will call the short-term memory content and splice it with the input; after the response is generated, the user input and system output of this round will be written into the conversation history records.

[0101] Long-term memory mainly involves external knowledge documents and dialogue states. External knowledge includes scientific common sense, psychological knowledge, etc. After the knowledge document data is divided into blocks, the corresponding vector expression is obtained through embedding coding and stored in the vector database. During the dialogue generation process, each agent will call relevant documents from the vector database as needed. The dialogue state includes the dialogue history record and the processed multi-level dialogue memory.

[0102] For multi-level conversation memory:

[0103] The lowest level is the original dialogue;

[0104] The second-level paragraph summary is obtained by the "dialogue summarizing agent" summarizing multiple rounds of dialogue (one user question + one model reply is one round);

[0105] The third layer is event summary, which includes entity extraction, event extraction and time extraction. It is mainly obtained by using "entity extraction agent" and "event analysis agent" to analyze the summary content of the next layer;

[0106] The fourth layer is the emotion summary, which includes emotion analysis and psychological state. It is obtained by analyzing events using the "emotion analysis agent" and "multimodal psychological assessment agent" respectively. The specific content includes: the user's attitude towards a topic or event, current psychological state, etc.

[0107] The highest level of conversation state is the user portrait, which summarizes the user's personality traits, thinking style, topics of interest, opinions, etc. from the lower level memory. It is a comprehensive overview of the user's image. The user portrait includes not only a description of the current conversation user, but also relevant information about the entities that appear in the conversation.

[0108] The dialogue state will be stored in three forms in the vector database, relational database and graph database. When the system performs strategy planning and generates dialogue content, it will retrieve and call the contents of these three databases as needed. After each round of dialogue is generated, the relevant intelligent agents will be called to collaborate and update each layer of dialogue state, in order from bottom to top. After the update is completed, the data cache and management module is called for processing: 1) With the "user portrait" as the center point of the graph, connect related events, emotions and specific memories, and store them in the graph database; 2) Establish a user list in the relational database, and establish multiple data tables for each user to store the dialogue state in layers, and each layer of content is the index of the next layer; 3) BERT encode each layer of content (including the original dialogue) and store it in the vector database.

[0109] Information Retrieval Module

[0110] The large language model that constitutes the intelligent agent uses corpora for fixed scenarios or tasks during pre-training and fine-tuning. Therefore, when encountering topics, scientific questions, or real-time data questions outside the corpus field during chats with users, the model's response performance is poor and it is easy to produce nonsense answers. In order to improve the model's hallucinations and improve its ability to correctly understand different topics and complete tasks in different fields, this system has designed an information retrieval module. This module performs queries based on dialogue input. First, the encoder generates semantic vectors, and then searches in the vectorized information to obtain relevant text information to assist the intelligent agent in responding to user questions.

[0111] This module is divided into two parts: semantic retrieval and literal retrieval. Semantic retrieval is based on the retrieval vector encoded by BERT input by the user, and performs vector retrieval on the vector database to obtain dialogue state information related to the user input content (including original dialogue, event summary, related characters, user emotions, psychological state, portrait) and external knowledge. In order to ensure the effectiveness of search matching, external documents and dialogue states also need to be embedded and encoded using the BERT encoder. Literal retrieval is based on the retrieval vector obtained by BM25 encoding of user input, and performs keyword retrieval on the relational database content and graph database content encoded using the same method to obtain dialogue state information with the same keywords as the user input. Then, according to the weighted sorting of the similarity results of the two searches, the top k texts with the highest similarity are selected. Finally, the selected document vector is converted into natural language text, passed to the intelligent agent, and filled in the corresponding part of the prompt template to assist it in performing the task.

[0112] Strategic Planning Module

[0113] Existing problems:

[0114] 1. The existing strategy generation module generates strategy groups by directly using a large language model to generate content or using a Markov chain to predict. The former has poor interpretability and is difficult to optimize. The latter generates strategy groups that are difficult to target different dialogue scenarios and are easily simplistic.

[0115] 2. The text generated by the existing strategy generation module has very average human-like performance (not very similar to the spoken conversations of humans in daily chats). If this is improved only by adjusting the prompt, a lot of experiments are required, and the effect is very small.

[0116] The strategy planning module is the core module of this system. Its main function is to generate strategies for the system to reply to users and guide the calling and output of each intelligent agent. The initial input of the module is a text, which mainly includes three parts: natural language text composed of the dialogue history and related memories (referring to the long-term memory related to this dialogue in addition to the original dialogue history, such as the events / characters / emotions mentioned in the previous text. "Splicing" refers to splitting the input content and filling it into the prompt), retrieved external knowledge, and the description of the user's emotional state. The strategy planning model needs to disassemble the user's questions in the current round, combine the chat history, user emotions, and related psychological knowledge, and determine the system's reply method (such as empathy, cognitive behavioral therapy, recommendation, etc., and the corresponding execution of the intelligent agent name) and the content that needs to be included (such as replying to specific scientific questions, soothing user emotions, etc.). The "reply strategy" output by this module is a fixed-format text that describes the specific content of each task in the strategy group and the intelligent agent planned to be called, and is listed in the order of execution. The output result will be passed to the dialogue content generation module to generate specific reply content.

[0117] The existing emotional support multi-round dialogue systems mainly use Markov chain processes or multi-classification neural networks for strategy planning. However, the generated strategy is simply represented by keywords, the generation effect of subsequent reply content is not good, and it is difficult to improve the human-likeness of the generated content. In order to improve this situation, this patent uses an online reinforcement learning method to construct a strategy planning agent, namely the online RLHF (Reinforcement Learning from Human Feedback) method. By introducing human preference feedback (implemented using a preference model) during the training process, the agent can continuously learn how to generate responses that are more in line with human preferences, thereby improving the ability to provide users with appropriate emotional support and solutions. The overall framework is as follows Figure 6 As shown, assuming that the current dialogue is the tth round, the specific operation and update steps are as follows:

[0118] 1. First, the strategy planning agent (π t ) generates multiple strategies based on the input dialogue state. Among them, the strategy judged to be the best is recorded as The complete strategy sequence is recorded as The dialogue state is called from the shared memory, including the dialogue history, dialogue event summary, user psychological state, user emotion and user profile. Each policy output needs to include the name of the agent called and the information passed to it. The complete input prompt word template is:

[0119] The user’s question in this round is: [User question], the user’s current mood is [Observed mood]. Current conversation situation: [memory summary], relevant knowledge: [Useful knowledge]. The agents in this system are: [Agent Name: {Usage, Input Parameters}, ...]

[0120] You are an intelligent scheduling agent. Based on the above information, you need to plan how to use one or more [AgentNames] to respond to the user's questions in this round, provide emotional support, and relieve their negative emotions. Please output your response plan in the following format (the plan should include at least one strategy).

[0121] Used agent:

[0122] Task description:

[0123] 2. Next, the strategy sequence Π t Enter the dialogue content generation module to generate specific responses. Each strategy generates an optimal response as a representative (the strategy planning agent (large model) has been trained in advance, and the optimal response is directly given by the model). Optimal strategy The response is directly output as the model response for this round of dialogue. Then, all responses (including The preference model is based on the Bradley-Terry (BT) model and is fine-tuned using the response preference dataset collected in advance. It is not updated in the system. The model generates a preference score for each input response, and then sorts the responses from large to small based on this score to simulate sorting responses based on human preferences. The preference model satisfies P(a 1 >a 2 |x,a 1 , a 2 )=σ(r * (x,a 1 )-r * (x,a 2 )). Where r * (x,a) is the inherent reward function, and σ(·) is the Sigmoid function. In this system, x represents the input dialogue state, and a represents the action (corresponding to the name of the dialogue generating agent). The loss function optimized in the process of training the preference model is: where a w The action selected as the best response, a l Actions that were not selected as the best response, represents the expectation, x represents the input dialogue state, logσ(r θ (x,a w )-r θ (x,a l )) measures the agent's choice of a w As a best reply instead of a l degree of preference, It is a policy preference dataset.

[0124] 3. The preference ranking sequence is added as new data to the policy preference dataset D. The parameters of the policy planning agent are updated through direct preference optimization (DPO). The input π0 of this process is the initial policy agent, which is obtained through supervised fine-tuning. The optimized loss function is:

[0125]

[0126] where a w =a win , a l =a lose . Parameters are updated through MLE. The process of selecting the optimal new strategy is:

[0127]

[0128] where argmmax π It means choosing the strategy that can maximize the expected return among all possible strategies π;

[0129] Represents the expectation of all possible dialogue states x;

[0130] It represents the expectation of all possible actions a given the strategy π and the dialogue state x;

[0131] η is a hyperparameter used to balance strategy selection and strategy diversity;

[0132] Represents strategy π t The KL divergence relative to the initial policy π0 is used to measure the policy π t The difference between π0.

[0133] Get the strategy planning agent π used in round t+1 t+1 . Dialogue content generation module

[0134] The main advantages of this module:

[0135] 1. Use multiple intelligent agents to collaborate to generate natural language text in specific scenarios.

[0136] 2. To improve the generation effect of a single agent, the agent of a single strategy can be optimized based on the test response. If the strategy needs to be increased or adjusted, it can be optimized efficiently to improve the fit between the complete response and the user's question and strategy.

[0137] Currently, using a prompt to drive a large model to generate a complete answer containing multiple response strategies is not very effective, and it can only be optimized by constantly adjusting the prompt and observing the output, which is very inefficient. Therefore, we consider training a specialized large model for each response strategy, generating the response content for each strategy separately, and forming a complete response.

[0138] If we directly concatenate these replies, the two sentences may be incoherent. Therefore, we use a scheduling agent to schedule the specialized agent; and during the generation process, we pass the semi-finished replies to the specialized agent, and use prompt to drive it to generate coherent replies that meet the required strategy based on the semi-finished replies.

[0139] In recent years, the fine-tuning technology of large language models has become increasingly mature, and fine-tuning models based on closed-source large language models such as GPT have been applied in various fields. Among them, guiding the model to complete the task in a specified way through a prompt template is the simplest and most common method. However, when faced with multiple tasks in specific scenarios, the prompt template has very limited generation effects and is difficult to optimize. Therefore, in order to improve the conversation quality of the psychological companionship system, this patent proposes to build a multi-agent collaborative conversation content generation module, in which each agent specializes in a subtask and only generates reply content for one strategy.

[0140] In this module, the scheduling agent will call the required vertical domain agents according to the given strategy, and conduct high-quality dialogues with users to achieve the purpose of psychological counseling and emotional support for users. The vertical domain agents of this system include empathy agents, CBT (Cognitive Behavior Therapy) agents, recommendation agents, psychological counseling agents, FAQ agents, and chat companion agents. During the training process, by collecting corpus for each task and using the QLoRA (Quantized Low-Rank Adapter) method to fine-tune the benchmark large model, agents for each task are obtained. Figure 7 The overall framework of this module is shown, and the LangGraph module is used for its specific implementation.

[0141] At the beginning of each round of dialogue, the strategy planning agent usually first calls the multimodal emotion assessment to obtain real-time user emotions and update the memory related to the user's psychological state. Then the vertical domain agent is directly called according to the reply. Agent A first completes the prompt word template according to the relevant information passed in, and then the corresponding large language model (LLM) uses the thinking chain or calls the external tool method to generate a reply in combination with the results of knowledge retrieval. Since each vertical domain agent only focuses on one strategy, a complete reply usually requires the cooperation of multiple agents. For example, if the reply strategy is "empathy-CBT-recommendation", the scheduling agent will first call the empathy agent to generate empathy reply content; then the original dialogue information and the empathy reply will be passed to the CBT agent to generate the CBT reply, which will be spliced ​​after the empathy reply; then the current reply will be passed as part of the input to the recommendation agent, and a specific prompt will be used to guide it to generate recommended content, and splicing will obtain the complete model reply of this round. The model reply will be stored in the "dialogue history record" of the short-term memory together with the user's question in this round; at the same time, it will be passed to the strategy planning module and the dialogue state update module for strategy planning agent iteration and long-term memory update. In the above, the specific prompt is Figure 8 The Prompt Template in the example. This template is a generalized version for vertical domain agents. In each agent, the prompt will be filled with some content. For example, the prompt for the recommendation agent is to add the name, function, and received parameters of the agent to the generalized version.

[0142] The invention relates to a psychological companionship dialogue method and storage medium based on multi-agent collaboration, which aims to build a psychological companionship dialogue system that can provide personalized, highly natural and accurate emotion recognition by integrating multi-agent collaboration and multimodal analysis technology. The following is an analysis of the main features and advantages of the technical solution:

[0143] Multimodal emotion assessment: By combining multimodal emotion assessment with text, voice, and images, it is possible to more accurately identify the user's emotional state and provide more considerate psychological support for the conversation.

[0144] Shared memory module: Utilizes the distinction between short-term and long-term memory to effectively store and retrieve conversation history, external knowledge, and user portraits, improving the coherence and personalization of conversations.

[0145] Strategy planning module: The online reinforcement learning method is used to train the intelligent agent to plan the response strategy, which enhances the flexibility and diversity of the response and improves the pertinence and humanity of the conversation content.

[0146] Dialogue content generation module: Through the collaborative work of multiple vertical field agents, the generation effect of a single agent is improved, and the agent of a single strategy can be optimized based on the test replies, thereby improving the fit between the complete reply and the user's questions and strategies.

[0147] Construction of user portraits: Through multi-level summaries of conversation history (including original conversations, paragraph summaries, event summaries, sentiment summaries, and user portraits), a comprehensive user portrait is constructed, which helps the system better understand user needs and provide personalized services.

[0148] Information Retrieval Module: Combining semantic retrieval and literal retrieval improves the ability to accurately understand and answer user questions, especially when dealing with topics outside the field or scientific problems.

[0149] Real-time optimization and iteration: The system continuously optimizes the strategy planning agent through online learning, enabling it to adjust the dialogue mode according to user feedback, improving the adaptability and effectiveness of the dialogue system.

[0150] To sum up, this invention effectively solves some problems in existing psychological companionship dialogue systems through multi-agent collaboration and multimodal analysis technology, such as insufficient flexibility and diversity of response strategies, and low fit between dialogue content and user needs, thereby improving user experience and psychological support effects.

Claims

1. A psychological companionship dialogue method based on multi-agent collaboration, characterized in that: The following steps are involved: Step 1: Perform multimodal emotion evaluation on user input to obtain a description of the user's emotional state; Step 2: According to the user's question and the user's emotional state description, relevant dialogue states and external knowledge are retrieved from the shared memory module; Step 3: Based on the retrieved dialogue state and external knowledge, the strategy planning module generates a response strategy to obtain a strategy group; Step 4: The dialogue content generation module calls the corresponding vertical domain agent according to the response strategy generated in step 3 to generate specific response content; Step 5: Store the user questions and model responses of this round into the short-term memory of the shared memory module, and update the long-term memory.

2. According to the method of psychological companionship dialogue based on multi-agent collaboration in claim 1, it is characterized in that: Step 1 includes the following steps: Step 1.1: Perform BERT encoding on the text input by the user to obtain a semantic vector; Step 1.2, extracting features from the voice and image input by the user to obtain an emotion feature vector; Step 1.3: Combine the semantic vector obtained in step 1.1 and the emotional feature vector obtained in step 2.2, and obtain a description of the user's emotional state through a multimodal emotion evaluation model.

3. According to the method of psychological companionship dialogue based on multi-agent collaboration in claim 1, it is characterized in that: Step 2 includes the following steps: Step 2.1: Use the BERT encoder to encode the user’s question and the result of step 1 to obtain a retrieval vector. At this point, the retrieval vector will be used to query the vector database in the shared memory module, which stores the vector representation of the conversation history and external knowledge. Step 2.2: Based on the result of step 2.1, semantic search is performed on the vector database to obtain relevant dialogue state information and external knowledge; Step 2.3: Based on the user input, perform a literal search on the relational database and the graph database to obtain the dialog state information having the same keywords as the user input; Step 2.4: weighted sort the results of step 2.2 and step 2.3, and select the top k most relevant texts.

4. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 3 is characterized in that: The shared memory module implementation includes the following steps: Step A1, initializing the shared memory module, including the settings of short-term memory and long-term memory; Step A2: Storing context and conversation history in short-term memory; Step A3: Storing external knowledge documents and dialog states in long-term memory; Step A4: When the conversation is in progress, the short-term memory content is called and spliced ​​with the input; Step A5: After the response generation is completed, the user input and system output of the current round are written into the conversation history record; Step A6: The conversation state includes the conversation history record and the processed multi-level conversation memory; Step A7: Multi-level conversation memory includes the original conversation at the lowest level, paragraph summary at the second level, event summary at the third level, emotion summary at the fourth level, and user portrait at the highest level. Step A8: After each round of dialogue is generated, call the relevant agent to update the dialogue state of each layer; Step A9: After the update is completed, the data cache and management module is called to process, including the update of the graph database, relational database and vector database.

5. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 1 is characterized in that: Step 3 includes the following steps: Step 3.1, the strategy planning module generates multiple possible response strategies based on the input dialogue state; Step 3.2: Input the result of step 3.1 into the dialogue content generation module to generate specific reply content. The reply content of the optimal strategy is directly used as the model reply for this round of dialogue; Step 3.3: Pass the specific response content generated in step 3.2 back to the strategy planning module, use the preference model to sort the results of step 3.1, determine the optimal strategy based on the sorting, form a strategy group, and the strategy planning module performs strategy iteration and parameter update.

6. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 5, characterized in that: Step 3.1 specifically includes the following steps: Step 3.1.1, Strategy Planning Module π t Generate multiple strategies based on the input dialogue state, among which the strategy judged to be the best is recorded as The complete strategy sequence is recorded as The strategy planning module is referred to as the agent; Step 3.1.2: The conversation state is called from the shared memory, including the conversation history, conversation event summary, user psychological state, user emotion, and user portrait; Each policy output needs to include the name of the agent being called and the information passed to it. The complete input prompt template is: { The user question in this round is: [User question], and the user’s current mood is [Observed mood]; Current conversation situation: [memory summary], relevant knowledge: [Useful knowledge]; The agents in this system are: [Agent Name: {Usage, Input Parameters}, …] You are an intelligent scheduling agent. Based on the above information, you need to plan how to use one or more [AgentNames] to respond to the user's questions in this round, provide emotional support, and relieve their negative emotions. Please output your response plan in the following format in order. The plan should include at least one strategy. Used agent: Task description: }。 7. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 5 is characterized in that: Step 3.3 specifically includes the following steps: 3.3.

1. Use the preference model to sort the strategy sequence results generated in step 3.1 and determine the optimal strategy. The preference model is based on the Bradley-Terry model, and its preference score calculation formula is: P(a 1 >a 2 |x,a 1 a 2 )=σ(r * (x,a 1 )-r * (x,a 2 )) where r * (x, a) is its inherent reward function, σ(·) is the Sigmoid function, x represents the input dialogue state, a represents the action, and a 1 , a 2 Represents different actions; a 1 >a 2 Indicates action 1 is considered better than action a 2 superior or preferred; Step 3.3.2: Add the preference ranking sequence to the strategy preference dataset , and update the parameters of the strategy planning module through direct preference optimization. The optimized loss function is: where a w =a win , a l =a lose , a w Indicates an action that results in a win or victory, a l Indicates an action that results in loss or failure; θ represents the learned policy parameters, π θ represents a parameterized policy, where θ is a set of parameters; π 0) represents a reference strategy; σ(·) is the sigmoid function, which maps real values ​​to the interval (0, 1)(0, 1); η is a hyperparameter; π0(a * |x) is to take action a in state x * probability; Step 3.3.3: Select the optimal new strategy π by maximizing expected return and considering the diversity of strategies t+1 , the optimal strategy selection process is: where argmax π It means choosing the strategy that can maximize the expected return among all possible strategies π; Represents the expectation of all possible dialogue states x; It represents the expectation of all possible actions a given the strategy π and the dialogue state x; η is a hyperparameter used to balance strategy selection and strategy diversity; Represents strategy π t The KL divergence relative to the initial policy π0 is used to measure the policy π t The difference between π0.

8. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 1, characterized in that: Step 4 includes the following steps: Step 4.1, the scheduling agent calls the required vertical field agent according to the response strategy; Step 4.2: The vertical domain agent selected in step 4.1 generates a response content of a specific strategy based on the relevant information passed in; Step 4.3: The scheduling agent calls the vertical domain agent to continue generating the response content of the next strategy, which is spliced ​​into the result of step 4.2; Step 4.4: Repeat steps 4.1 to 4.3 until all strategies in the strategy group are used and a complete model response is obtained for this round.

9. The method for psychological companionship dialogue based on multi-agent collaboration according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1: Store the user's questions and model responses in this round into the conversation history in short-term memory; Step 5.2: Update the conversation state in long-term memory according to the update result of step 5.1, including conversation history summary, event summary, emotion summary and user portrait; Step 5.3: Store the updated results of step 5.2 into the vector database, relational database and graph database.

10. A storage medium, characterized in that: When the processor executes the program in the storage medium, it implements a psychological companionship dialogue method based on multi-agent collaboration as described in any one of claims 1-9.

Images