A method, device, equipment and medium for enhancing LLM multi-round dialogue effect by using RAG

Abstract

The present invention discloses a method, device, equipment and medium for enhancing the effect of LLM multi-round dialogue by using RAG, wherein the method comprises: using RAG to retrieve the dialogue history of user dialogue, selecting the most relevant dialogue history segment as the user dialogue context; using RAG to perform Agent retrieval on the user dialogue, selecting the most relevant Agent arrangement knowledge of the user dialogue; finely generating prompts; obtaining the dialogue result output of the LLM model dialogue system. The present invention uses RAG technology to exclude the influence of irrelevant context, introduces Agent arrangement knowledge with relevant guiding significance, improves the execution efficiency and accuracy of LLMAgent, ensures the completion accuracy of complex tasks, and can accurately guide LLMAgent to complete the understanding and execution of complex tasks.

Description

Technical Field

[0001] The present invention relates to the field of application technology of artificial intelligence-large language model system LLM, and in particular to a method, device, equipment and medium for enhancing LLM multi-round dialogue effect by using RAG. Background Art

[0002] With the widespread application of LLM in the field of natural language processing, its customization needs in specific industry information fields have become increasingly prominent. In particular, the ability to complete specific field tasks in combination with domain agent tools has greatly expanded the application scenarios of LLM. However, when dealing with multi-round conversations in complex geographical scenarios, existing LLMs often face problems such as insufficient knowledge depth, limited context memory, and inaccurate understanding of domain-specific terminology, resulting in unsatisfactory conversation results and inaccurate completion of complex tasks.

[0003] RAG (Retrieval-Augmented Generation) technology can provide effective information supplement for LLM and has begun to be used in various application scenarios of LLM, especially in the field of knowledge question and answer. However, in the field of LLM Agent, the value of RAG technology has not been fully utilized.

[0004] The LLMAgent mechanism is an important technical framework for connecting the large model system with the execution interface of the geographic industry. However, the execution of some complex tasks often requires the arrangement of multiple different agent calls in succession. It is difficult to extract some professional agent parameters. Especially when users customize agents based on the geographic industry to conduct multiple rounds of continuous dialogues, the accuracy of task completion and the accuracy of multiple rounds of continuous dialogues are not ideal.

[0005] The main reasons why existing LLMAgents cannot accurately complete complex domain tasks are: (1) the historical records are too long, the dialogue intent span is large, and there are too many records; (2) it is difficult to orchestrate complex domain tasks, which places high demands on the planning ability of LLMAgent; (3) the effect is highly dependent on prompts and requires fine control.

[0006] In order to improve the accuracy of LLMAgent in executing complex multi-agent serial tasks, the present invention proposes a new technical architecture that deeply combines RAG technology with LLMAgent to effectively improve the effect of completing complex tasks of custom Agent. Summary of the invention

[0007] The purpose of the present invention is to provide a method, device, equipment and medium for enhancing the effect of LLM multi-round dialogue by using RAG, using RAG technology in the process of multi-round dialogue of users, eliminating irrelevant influences from the above context, introducing Agent arrangement knowledge, and accurately guiding LLMAgent to complete the understanding and execution of complex tasks.

[0008] The embodiment of the present invention provides a method, apparatus, device and medium for enhancing LLM multi-round dialogue effect by using RAG.

[0009] A first aspect: A method for enhancing LLM multi-round dialogue effect by using RAG, comprising:

[0010] S1, user dialogue input LLM model dialogue system;

[0011] S2, using RAG to retrieve the conversation history of the user's conversation, and selecting the most relevant conversation history segment as the user conversation context;

[0012] S3, use RAG to perform agent retrieval on user dialogues and select the most relevant agent arrangement knowledge for user dialogues;

[0013] S4, based on the most relevant context of the user dialogue and the Agent’s orchestration knowledge, finely generate prompts and input them to the LLMAgent;

[0014] S5. Obtain the dialogue result output of the LLM model dialogue system.

[0015] Optionally, S2 comprises the steps of:

[0016] S21, analyzing the user's conversation history, extracting key information to form conversation history metadata;

[0017] S22. Designing a conversation history index structure suitable for RAG retrieval based on the metadata of conversation history records;

[0018] S23, based on user dialogue, using the LLM model to preliminarily understand the user's dialogue intention and topic, and generate preliminary query words or phrases to provide to RAG;

[0019] S24, RAG retrieves relevant conversation history segments from the conversation history index structure based on the query word or phrase, and sorts them according to relevance and timeliness;

[0020] S25. Sort by relevance and timeliness, and select the most relevant conversation history segment as the context of the current conversation.

[0021] Optionally, S3 comprises the steps of:

[0022] S31, customize the Agent toolset, build the Agent orchestration knowledge base, and transform the acquired Agent orchestration knowledge into a structured form;

[0023] S32, establishing a dedicated retrieval database based on the keywords, descriptions and applicable scenario information of a single piece of Agent arrangement knowledge;

[0024] S33,RAG finds the most relevant Agent-compiled knowledge index from the search database based on the query word or phrase;

[0025] S34. Calling the Agent orchestration knowledge base according to the most relevant Agent orchestration knowledge index.

[0026] Optionally, S4 comprises the steps of:

[0027] S41. Incorporate the most relevant context retrieved by RAG into the prompt design of LLMAgent, making it clear that the LLM model focuses on historical information related to user conversations.

[0028] S42, then integrate the most relevant Agent arrangement knowledge retrieved by RAG into the prompt design of LLMAgent to form a RAG enhanced prompt with clear Agent arrangement knowledge guidance;

[0029] S43. According to the complexity of the multi-round dialogue task of the LLM model user and the basic information of the task, select an appropriate prompt template.

[0030] S44. Use RAG to enhance prompt to dynamically update prompt template, generate prompt requests for user multi-round dialogue tasks in detail, and input them into the LLM model.

[0031] Optionally, the dialogue result output in S5 is returned to the dialogue history.

[0032] Optionally, each piece of Agent orchestration knowledge includes five parts: request user request instance, intent support intention, Agent orchestration instance, tool execution / observation and response reply result.

[0033] Optionally, the Agent toolset custom information includes: tool name, tool description, input parameters and description, output parameters and description, required parameters and remarks information.

[0034] A second aspect: A system for enhancing LLM multi-round dialogue effects by using RAG, comprising:

[0035] RAG conversation history retrieval module selects the most relevant conversation history segments as user conversation context based on user conversations;

[0036] RAGAgent retrieval module, based on user dialogue, selects the most relevant Agent orchestration knowledge of the user dialogue;

[0037] The refined prompt generation module generates prompts according to the most relevant context of the user dialogue and the Agent orchestration knowledge, and inputs them to the LLMAgent;

[0038] The LLM module, based on the LLMAgent input, calls the domain knowledge and obtains the user dialogue results;

[0039] The domain knowledge toolset module provides a collection of retrieval and call databases for the RAG conversation history retrieval module, RAGAgent retrieval module, and LLM module.

[0040] A third aspect: An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the program.

[0041] A fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention proposes a new RAG and LLMAgent combination method and system, and provides a technical solution to improve the effect of complex multi-round Agent calls. The solution of the present invention uses RAG technology to filter relevant conversation contexts from historical conversation records during multi-round user conversations, eliminates the influence of irrelevant contexts, and improves the execution efficiency and accuracy of LLMAgent. In Agentprompt, RAG retrieval enhancement technology is introduced, and Agent arrangement knowledge with relevant guiding significance is introduced and embedded in Agentprompt, so as to ensure the completion accuracy of complex tasks; the effective knowledge extracted by RAG in the above two links can accurately guide LLMAgent to understand and execute complex tasks through fine prompt design.

[0044] 2. The present invention introduces the RAG mechanism in the implementation process of LLMAgent multi-round dialogue, effectively selects historical records when matching the previous records of multi-round dialogue, and can retrieve the Agent arrangement knowledge base in real time when generating dialogue responses, assisting the model system to instantly obtain and integrate the required professional knowledge during the dialogue process, effectively making up for the knowledge blind spots of the simple model system knowledge when completing complex tasks in the field, and enhancing the model system's ability to provide accurate answers when facing complex and detailed geographic problems and tasks. It can effectively improve the LLMAgent mechanism's multi-round dialogue capabilities in complex tasks in the field, improve the geographic information interaction experience, and provide users with more professional, accurate and smooth conversational geographic information services. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a method for enhancing LLM multi-round dialogue effect by using RAG according to the present invention;

[0046] Figure 2 A schematic diagram of the structure of a system for enhancing LLM multi-round dialogue effect by using RAG according to the present invention;

[0047] Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0049] With the widespread application of LLM in the field of natural language processing, its customization needs in specific industries such as geographic information have become increasingly prominent. In particular, the ability to complete specific field tasks in combination with domain agent tools has greatly expanded the application scenarios of LLM. However, when dealing with multi-round conversations in complex geographic scenarios, existing LLMs often face problems such as insufficient knowledge depth, limited contextual memory, and inaccurate understanding of domain-specific terms, resulting in unsatisfactory conversation results and inaccurate completion of complex tasks.

[0050] In view of the above problems, the present invention provides a method for enhancing the LLM multi-round dialogue effect by using RAG. Figure 1 A flow chart of a method for enhancing LLM multi-round dialogue effect by using RAG provided in an embodiment of the present invention, the method comprising:

[0051] S1, user dialogue input LLM model dialogue system; Figure 2As shown in the figure, LLM is a large language model system. Geo LLM is a large language model system applied in the Geo field of the geographic industry. LLM obtains the input based on multiple rounds of user dialogues by calling the domain knowledge toolset, and outputs the results of multiple rounds of user dialogues.

[0052] The domain knowledge toolset is a pre-built domain-related knowledge base and database to support the implementation of the LLM model dialogue system. It mainly includes search engines, agent orchestration knowledge bases, domain databases, agent tools toolsets, and dialogue history records.

[0053] S2, using RAG to retrieve the conversation history of the user's conversation, and selecting the most relevant conversation history segment as the user conversation context; Figure 2 As shown in the figure, when the user dialogue is input into the LLM model system, the RAG retrieval technology is used to interact with the dialogue history of the domain knowledge toolset. In the multi-round dialogue process, the RAG technology is used to filter out the most relevant context of the user dialogue from the historical dialogue records, which can eliminate the interference of irrelevant information and improve the execution efficiency and accuracy of multi-round LLMAgent.

[0054] The specific steps include:

[0055] S21. Analyze the user's conversation history and extract key information, such as topics, question types, entity mentions, etc., to form metadata of the conversation history records.

[0056] S22. Design a conversation history index structure suitable for RAG retrieval based on the metadata of conversation history records, such as inverted index, keyword-document matrix, etc., to ensure efficient retrieval of relevant conversation fragments.

[0057] S23. At the beginning of a multi-round conversation between users, based on the user conversation, LLM is used to preliminarily understand the user's conversation intention and topic, and generate preliminary query words or phrases for RAG retrieval.

[0058] S24, RAG retrieves relevant conversation history segments from the conversation history index structure based on the query word or phrase, and sorts them according to relevance and timeliness;

[0059] S25. Sort by relevance and timeliness, and select the most relevant conversation history segment as the context of the current conversation and input it into the prompt of LLMAgent.

[0060] S3, use RAG to perform agent retrieval on user dialogues and select the most relevant agent arrangement knowledge for user dialogues; Figure 2As shown in the figure, in order to improve the accuracy of LLMAgent in the execution of complex Agent multi-round tasks, RAG retrieval enhancement technology can be introduced, combined with instructive Agent orchestration knowledge, to optimize Agentprompt design.

[0061] The specific steps include:

[0062] S31, customize the Agent toolset, build the Agent orchestration knowledge base, and transform the acquired Agent orchestration knowledge into a structured form;

[0063] The custom information of the Agent toolset includes: tool name, tool description, input parameters and description, output parameters and description, required parameters and remarks, etc.

[0064] The sources of Agent orchestration knowledge for building the Agent orchestration knowledge base mainly include collecting and arranging scheduling instructions, parameter standards, user request implementation cases, etc. related to multi-agent task scheduling, collaboration mode, input and output formats, etc., to form a rich Agent orchestration knowledge base.

[0065] The Agent Orchestration Knowledge Base converts the acquired Agent Orchestration knowledge into structured forms, such as rules, flow charts, case descriptions, etc., to facilitate subsequent retrieval and integration into prompts.

[0066] S32, establishing a dedicated retrieval database based on the keywords, descriptions and applicable scenario information of a single piece of Agent arrangement knowledge;

[0067] As shown in Table 1, each piece of Agent orchestration knowledge mainly includes five parts: request user request instance, intent support intention, Agent orchestration instance, tool execution / observation and response reply result.

[0068]

[0069]

[0070] For the Agent orchestration knowledge base, a dedicated retrieval database is established, which contains the keywords, descriptions, applicable scenarios and other information of a single Agent orchestration knowledge, so that RAG can quickly locate the required key knowledge index information.

[0071] S33, RAG searches the most relevant Agent arrangement knowledge index from the retrieval database based on the query word or phrase; when in use, the RAG retrieval module is integrated with the LLMAgent platform, and the retrieval parameters and recall strategies are set to ensure that the arrangement knowledge related to the current task can be accurately retrieved when the prompt is generated.

[0072] S34. Call the most relevant Agent orchestration knowledge in the Agent orchestration knowledge base according to the most relevant Agent orchestration knowledge index.

[0073] S4, based on the most relevant context of the user dialogue and the Agent's orchestration knowledge, generate prompts in detail and input them to the LLMAgent; Figure 2 shown.

[0074] The specific steps include:

[0075] S41. Incorporate the most relevant context retrieved by RAG into the prompt design of LLMAgent, making it clear that the model focuses on historical information related to the user conversation.

[0076] S42, then integrate the most relevant Agent orchestration knowledge retrieved by RAG into the prompt design of LLMAgent, extract key processes, interaction rules and other information, and form a RAG enhanced prompt with clear Agent orchestration knowledge guidance;

[0077] S43. According to the complexity of the multi-round dialogue task of the LLM model user, as well as the basic task information such as task description, goal, and constraints, an appropriate prompt template is selected to dynamically generate the entire complete request.

[0078] S44. Use RAG to enhance prompt to dynamically update prompt template, generate prompt requests for each round of user dialogue tasks in detail, and input into the LLM model.

[0079] At the same time, according to the LLM model output and user feedback, the RAG retrieval parameters and dialogue management strategies are dynamically adjusted to continuously optimize the dialogue process. The generated dynamic prompt templates are regularly checked for logic and semantic coherence, and manually reviewed and corrected when necessary to ensure that they accurately convey the execution logic and collaboration requirements of multi-agent tasks.

[0080] S5. Obtain the dialogue result output of the LLM model dialogue system.

[0081] The present invention also discloses a system for enhancing the LLM multi-round dialogue effect by using RAG, such as Figure 2 As shown, including:

[0082] RAG conversation history retrieval module selects the most relevant conversation history segments as user conversation context based on user conversations;

[0083] RAGAgent retrieval module, based on user dialogue, selects the most relevant Agent orchestration knowledge of the user dialogue;

[0084] The refined prompt generation module generates prompts according to the most relevant context of the user dialogue and the Agent orchestration knowledge, and inputs them to the LLMAgent;

[0085] The LLM module, based on the LLMAgent input, calls the domain knowledge and obtains the user dialogue results;

[0086] The domain knowledge toolset module provides retrieval and call data sets for the RAG conversation history retrieval module, RAGAgent retrieval module and LLM module.

[0087] The present invention adds an Agent arrangement knowledge base to the domain knowledge tool set. When the user dialogue is input into the LLM model system, the RAG dialogue history retrieval module and the RAGAgent retrieval module interact with the domain knowledge tool set module, perform relevance retrieval according to the user dialogue content, accurately select the most relevant historical dialogue and Agent arrangement knowledge from the user dialogue content, and then generate prompts in detail through the detailed prompt generation module. After that, the domain knowledge tool set performs conventional large model generation replies and Agent execution and other subsequent processes, so that the user dialogue is effectively expanded in the RAG dialogue history retrieval module, the RAGAgent retrieval module and the detailed prompt generation module, so that the Agent execution accuracy of the LLM model architecture can be effectively guaranteed.

[0088] The present invention supports the management of various domain knowledge tools that assist LLMAgent, including user conversation history management, domain tool library management, agent orchestration knowledge management, etc. It supports the integration of RAG technology in multi-round conversations and agent execution scenarios, improves task completion and conversation experience, and has good scalability.

[0089] The present invention also provides an electronic device, Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory to execute the following method:

[0090] S1, user dialogue input LLM model dialogue system;

[0091] S2, using RAG to retrieve the conversation history of the user's conversation, and selecting the most relevant conversation history segment as the user conversation context;

[0092] S3, use RAG to perform agent retrieval on user dialogues and select the most relevant agent arrangement knowledge for user dialogues;

[0093] S4, based on the most relevant context of the user dialogue and the Agent’s orchestration knowledge, finely generate prompts and input them to the LLMAgent;

[0094] S5. Obtain the dialogue result output of the LLM model dialogue system.

[0095] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0096] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0097] S1, user dialogue input LLM model dialogue system;

[0098] S2, using RAG to retrieve the conversation history of the user's conversation, and selecting the most relevant conversation history segment as the user conversation context;

[0099] S3, use RAG to perform agent retrieval on user dialogues and select the most relevant agent arrangement knowledge for user dialogues;

[0100] S4, based on the most relevant context of the user dialogue and the Agent’s orchestration knowledge, finely generate prompts and input them to the LLMAgent;

[0101] S5. Obtain the dialogue result output of the LLM model dialogue system.

[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing the LLM multi-round dialogue effect by using RAG, characterized in that: include: S1, user dialogue input LLM model dialogue system; S2, using RAG to retrieve the conversation history of the user's conversation, and selecting the most relevant conversation history segment as the user conversation context; S3, use RAG to perform agent retrieval on user dialogues and select the most relevant agent arrangement knowledge for user dialogues; S4, based on the most relevant context of the user dialogue and the Agent’s orchestration knowledge, finely generate prompts and input them to the LLMAgent; S5, obtaining the dialogue result output of the LLM model dialogue system; The S2 comprises the steps of: S21, analyzing the user's conversation history, extracting key information to form conversation history metadata; S22. Designing a conversation history index structure suitable for RAG retrieval based on the metadata of conversation history records; S23, based on user dialogue, using the LLM model to preliminarily understand the user's dialogue intention and topic, and generate preliminary query words or phrases to provide to RAG; S24, RAG retrieves relevant conversation history segments from the conversation history index structure based on the query word or phrase, and sorts them according to relevance and timeliness; S25, sorting by relevance and timeliness, selecting the most relevant conversation history segment as the context of the current conversation; The S3 comprises the steps of: S31, customize the Agent toolset, build the Agent orchestration knowledge base, and transform the acquired Agent orchestration knowledge into a structured form; S32, establishing a dedicated retrieval database based on the keywords, descriptions and applicable scenario information of a single piece of Agent arrangement knowledge; S33,RAG finds the most relevant Agent-compiled knowledge index from the search database based on the query word or phrase; S34, calling the Agent arrangement knowledge base according to the most relevant Agent arrangement knowledge index; Each piece of Agent orchestration knowledge includes five parts: request user request instance, intent support intent, Agent orchestration instance, tool execution / observation and response reply result.

2. The method for enhancing the LLM multi-round dialogue effect by using RAG according to claim 1, characterized in that: The S4 comprises the steps of: S41. Incorporate the most relevant context retrieved by RAG into the prompt design of LLMAgent, making it clear that the LLM model focuses on historical information related to user conversations. S42, then integrate the most relevant Agent arrangement knowledge retrieved by RAG into the prompt design of LLMAgent to form a RAG enhanced prompt with clear Agent arrangement knowledge guidance; S43. According to the complexity of the multi-round dialogue task of the LLM model user and the basic information of the task, select an appropriate prompt template. S44. Use RAG to enhance prompt to dynamically update prompt template, generate prompt requests for user multi-round dialogue tasks in detail, and input them into the LLM model.

3. The method for enhancing the LLM multi-round dialogue effect by using RAG according to claim 1, characterized in that: The dialogue result output in S5 is returned to the dialogue history.

4. The method for enhancing the LLM multi-round dialogue effect by using RAG according to claim 3, characterized in that: The Agent toolset custom information includes: tool name, tool description, input parameters and description, output parameters and description, required parameters and remarks.

5. A system for enhancing LLM multi-round dialogue effect by using RAG applied to the method described in any one of claims 1 to 4, characterized in that: include: RAG conversation history retrieval module selects the most relevant conversation history segments as user conversation context based on user conversations; RAGAgent retrieval module, based on user dialogue, selects the most relevant Agent orchestration knowledge of the user dialogue; The refined prompt generation module generates prompts according to the most relevant context of the user dialogue and the Agent orchestration knowledge, and inputs them to the LLMAgent; The LLM module, based on the LLMAgent input, calls the domain knowledge and obtains the user dialogue results; The domain knowledge toolset module provides a collection of retrieval and call databases for the RAG conversation history retrieval module, RAGAgent retrieval module, and LLM module.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for enhancing LLM multi-round dialogue effect by using RAG as claimed in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for enhancing LLM multi-round dialogue effect by using RAG as claimed in any one of claims 1 to 4 are implemented.

Images