Answer generation method and system based on generation-discrimination mechanism and chain thinking

Abstract

The invention relates to an answer generation method and system based on a generation-discrimination mechanism and chain thinking, and the method comprises the steps: obtaining a question proposed by a user, extracting a scene where the question is located and keywords involved in the question, and generating an intermediate representation containing key information; finding out two question and answer pairs most similar to the user question from the knowledge base as sample answers; selecting a corresponding knowledge base according to the extracted scene, finding a document fragment most related to the user question, recalling the found document fragment and reordering the found document fragment; using the first K reordered document segments as reference contexts by using a COT thinking chain method, summarizing replies for answering the user question, and generating a preliminary answer by summarizing all the replies: returning the corresponding answer to the user when the generated preliminary answer or sample answer can be used as an effective reply of the current user question, otherwise, regenerating the answer.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and specifically to an answer generation method and system based on a generation-discrimination mechanism and chain thinking. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] RAG (Retrieval-Augmented Generation) is a technology in the field of natural language processing (NLP). It combines retrieval models and generation models to improve the quality of text generation. It is often used in programs that generate corresponding answers based on user questions.

[0004] When a large-scale language model (LLM) is applied, the performance of the model will improve accordingly as the model scale expands, the amount of data increases, and the computing power improves, which is the so-called Scaling Law. However, although LLM has performed well in many fields, it still has some problems, such as hallucinations, catastrophic forgetting, high training costs, outdated knowledge, and lack of professional problem-solving capabilities. To solve these problems, application methods such as RAG (retrieval-augmented generation) and AGENT have emerged. Among them, RAG is regarded as an effective method to solve the hallucination problem and the unexplainability problem. However, RAG currently faces two major challenges:

[0005] 1. Traditional RAG uses embedding vectors to recall text in the database, but queries often fail to clearly express user intent and lack semantic information, resulting in insufficient and noisy recalled text. This not only wastes token resources, but also slows down the inference speed of large-scale language models.

[0006] 2. In the reply stage, the recall context and user questions are concatenated and summarized by the large language model for reply. However, due to the inaccurate recall problem that occurred in the early stage, and in the absence of a reference, the LLM will combine its own pre-training results to give the basic characteristics of the reply, resulting in inaccurate replies and misleading user results. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an answer generation method and system based on generation-discrimination mechanism and chain thinking, forming a heuristic answer generation and discrimination mechanism, in which one large model + custom COT strategy of step-by-step processing is used as a generator, and another large model + custom COT strategy of step-by-step processing is used as a discriminator. The output of the generator is evaluated and screened, and the answer is returned to the user after confirming that there is no inconsistency between the answer output by the generator and the reference material.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides an answer generation method based on a generation-discrimination mechanism and chain thinking, comprising the following steps:

[0010] Get the question raised by the user, use the front generator model to extract the scenario where the question is located and the keywords involved in the question, and generate an intermediate representation containing key information:

[0011] Based on the intermediate representation, the pre-generator model is used to find the two question-answer pairs most similar to the user's question from the knowledge base as sample answers; the recall model is used to select the corresponding knowledge base based on the extracted scenarios, find the document fragments most relevant to the user's question, and recall and re-rank the found document fragments;

[0012] The post-generator model uses the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies;

[0013] The discriminator model is used to evaluate the answer. When the generated preliminary answer or sample answer can serve as a valid response to the current user's question, the corresponding answer is returned to the user. If not, the answer is regenerated.

[0014] Furthermore, the front-end generator model is used to receive user questions, and the scenario where the question is located and the keywords involved in the question are extracted based on the COT thinking chain method.

[0015] Furthermore, the recall model is used to select the corresponding knowledge base through the extracted scenarios, the keywords extracted by the predecessor generator are vectorized, and the document fragments most relevant to the user's question are found from the knowledge base through the search and matching algorithm, and the recalled document fragments are reordered.

[0016] Furthermore, the post-generator model receives two inputs, namely the recalled document fragment and the user question, and obtains the top K re-ranked results as reference context. The generator summarizes the responses to the user question from each result step by step, and generates a preliminary answer by summarizing all the responses.

[0017] Furthermore, the discriminator model is combined with the COT thinking chain method to evaluate whether the generated preliminary answers or sample answers can serve as effective responses to the current user's questions.

[0018] Furthermore, when the generated preliminary answer or sample answer can serve as a valid response to the current user's question, the answer is qualified, and the qualified answer is returned to the user through the user interface.

[0019] Furthermore, when the generated preliminary answer or sample answer is unqualified, the post-generator model is used to regenerate the answer in a parallel manner.

[0020] The second aspect of the present invention provides an answer generation system based on generation-discrimination mechanism and chain thinking, including.

[0021] The question acquisition module is configured to: acquire questions raised by users;

[0022] The problem analysis module is configured to extract the problem scenario and keywords involved in the problem, and generate an intermediate representation containing key information:

[0023] The sample answer module is configured to: find two question-answer pairs that are most similar to the user's question from the knowledge base according to the intermediate representation as sample answers; select the corresponding knowledge base through the extracted scenario, find the document fragments that are most relevant to the user's question, and recall and re-rank the found document fragments;

[0024] The answer generation module is configured to: use the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies;

[0025] The answer evaluation and output module is configured to: when the generated preliminary answer or sample answer can be used as a valid response to the current user's question, return the corresponding answer to the user; if not, regenerate the answer.

[0026] The third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned answer generation method based on the generation-discrimination mechanism and chain thinking.

[0027] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the above-mentioned answer generation method based on the generation-discrimination mechanism and chain thinking are implemented.

[0028] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0029] 1. Form a heuristic answer generation and discrimination mechanism, in which one large model + custom COT strategy with step-by-step processing is used as the generator, and the other large model + custom COT strategy with step-by-step processing is used as the discriminator. The output of the generator is evaluated and screened, and the answer is returned to the user after confirming that there is no inconsistency between the answer output by the generator and the reference material. Compared with the multi-agent strategy, the workflow is simpler, which can better stimulate the reasoning potential of the large model itself, and can achieve the corresponding effect with a smaller model.

[0030] 2. The pre-generator model combines the COT thinking chain to analyze the problem, extract key information and keywords, and generate a more accurate intermediate representation. It provides high-quality input for the post-generator, further improving the quality of the final answer.

[0031] 3. Using the discriminator model to evaluate the answer quality avoids the resource consumption and management complexity problems in the multi-agent architecture. At the same time, this centralized evaluation mechanism is easier to manage and control, improving the overall robustness and reliability of the system. Compared with the traditional uncertainty scoring method, the discriminator model can be used to comprehensively evaluate multiple factors such as grammatical correctness, contextual relevance, and logical coherence. This comprehensive evaluation method not only reduces hallucinations, but also ensures that the generated answers have high quality in multiple dimensions, which can reduce misjudgments and hallucinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] Figure 1 is a schematic diagram of a search enhancement generation process provided by one or more embodiments of the present invention;

[0034] Figure 2 is a schematic diagram of an algorithm architecture of a search enhancement generation process provided by one or more embodiments of the present invention;

[0035] Figure 3 Schematic diagram of data flow during retrieval enhancement generation provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0038] As introduced in the background technology, RAG is regarded as an effective method to solve the problems of hallucination and unexplainability. The multi-language translation process of the existing technology may introduce errors, resulting in the original answer being misjudged as a hallucination even if it is correct; in addition, the uncertainty score mainly depends on the entropy value of a single token, which may not fully reflect the logic and coherence of the entire answer, thus leading to misjudgment.

[0039] Embodiment 1:

[0040] The answer generation method based on the generation-discrimination mechanism and chain thinking includes the following steps:

[0041] Get the question raised by the user, use the front generator model to extract the scenario where the question is located and the keywords involved in the question, and generate an intermediate representation containing key information:

[0042] Based on the intermediate representation, the pre-generator model is used to find the two question-answer pairs most similar to the user's question from the knowledge base as sample answers; the recall model is used to select the corresponding knowledge base based on the extracted scenarios, find the document fragments most relevant to the user's question, and recall and re-rank the found document fragments;

[0043] The post-generator model uses the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies;

[0044] The discriminator model is used to evaluate the answer. When the generated preliminary answer or sample answer can serve as a valid response to the current user's question, the corresponding answer is returned to the user. If not, the answer is regenerated.

[0045] like Figure 2 As shown, the architecture of this embodiment mainly includes the following core components:

[0046] 1. Knowledge base: A database that stores documents and related information, covering documents and data from multiple business areas.

[0047] 2. Generator Agent:

[0048] 2.1. Question clarification module: Before searching and recalling, the prompt words are combined with the COT (Chain of Thought) thinking chain to analyze the user's question and extract the scenario where the question is located and the keywords involved in the question. The generator deeply analyzes the user's question and generates an intermediate representation containing key information to provide accurate input for the subsequent recall process.

[0049] 2.2. Generator: After the search recall behavior, combined with the COT thinking chain technology, the generator is required to generate responses to user questions step by step based on the contextual information of the recall, and finally summarize all the answers to avoid subjective answers as much as possible. This model extracts key information from relevant document fragments retrieved from the knowledge base and generates the final answer.

[0050] 3. Recall model: It is used to vectorize user questions and document fragments in the knowledge base, and find the document fragments most relevant to the user question through a search and matching algorithm. This model improves the accuracy and efficiency of recall by calculating the similarity between vectors. The recall model performs precise recall based on the keywords extracted by the pre-generator.

[0051] 4. Discriminator model: Combining the idea of ​​the GAN (Generative Adversarial Network) model, the discriminator role is set in the RAG workflow, mainly used to evaluate whether the generated answer can be used as an effective response to the current user's question. The discriminator will comprehensively consider the grammatical correctness, contextual relevance and logical coherence of the answer to determine whether the generated answer can accurately answer the user's question.

[0052] 5. Workflow orchestration: coordinate the workflow of the pre-generator, recall model, post-generator and discriminator, and the logic of retry and error handling to ensure the efficiency and accuracy of the entire question-answering process. Specifically, it includes question analysis, keyword extraction, information recall, answer generation, quality assessment and answer feedback.

[0053] 6. User Interface: Provides an interface for users to interact with the system, through which users can submit questions and receive answers generated by the system.

[0054] This embodiment takes the program of receiving user questions and generating answers as an example. Its working principle involves the collaborative work of multiple modules, such as Figure 1 and Figure 3 As shown, the specific steps are as follows:

[0055] 1. User Question: The user submits a question through the user interface, such as: "How do I apply for sick leave?"

[0056] 2. Problem clarification module analyzes the problem:

[0057] After receiving the user's question, the front-end generator model analyzes the problem based on the COT (Chain of Thought) thinking chain.

[0058] During the analysis process, the pre-generator extracts the context of the question (such as “human resources”) and the keywords involved in the question (such as “application”, “sick leave”).

[0059] The pre-generator generates an intermediate representation containing key information, such as: "The user wants to know how to apply for sick leave in the human resources system."

[0060] 3. Sample search and recall:

[0061] Example search: The example search module finds the two question-answer pairs (examples) most similar to the user's question from the knowledge base based on the generated intermediate representation.

[0062] The recall model performs search and matching: The recall model selects the corresponding knowledge base through the extracted scenario, vectorizes the keywords extracted by the predecessor generator, and finds the document fragment most relevant to the user's question from the knowledge base through the search and matching algorithm.

[0063] The re-ranking model performs recall segment re-ranking: re-rank the recalled document segments to ensure that the most relevant and appropriate segments are ranked first. At the same time, the samples are also ranked according to their similarity to the user's questions to provide reference for the subsequent generator and discriminator.

[0064] 4. The post-generator generates the answer:

[0065] The post-generator model receives two inputs: the recalled document snippet and the user question.

[0066] Combined with COT thinking chain technology, the first K re-ranked results of multi-source recall are used as reference contexts, requiring the generator to summarize the responses to the user's questions from each result step by step, and generate preliminary answers by summarizing all the responses, trying to avoid subjective answers. For example, the preliminary answer generated may be: "To apply for sick leave, please log in to the collaborative office system, fill out the application form and submit it for approval."

[0067] 5. The judge evaluates the answer:

[0068] The discriminator model combines the COT thinking chain to evaluate whether the generated answer or the retrieved sample answer can be used as an effective response to the current user's question, taking into account the grammatical correctness, contextual relevance and logical coherence of the answer.

[0069] If the judge considers the answer to be qualified, the answer will be returned to the user; if the judge considers the answer to be unqualified, the post-generator will regenerate the answer, with two retries, and in order to achieve a higher response speed, the two retries will be performed in parallel.

[0070] 6.Answer feedback:

[0071] The final qualified answer is returned to the user through the user interface.

[0072] Users can see answers generated by the system, such as: "To apply for sick leave, please log in to the collaborative office system, fill out the application form and submit it for approval."

[0073] In this embodiment, Chain of Thought (COT) is a technology that enhances the reasoning ability of the model, and improves the logic and coherence of the generated answers by gradually thinking and explaining the decision-making process of the model. The pre-generator model in the present invention combines the COT thinking chain to analyze the problem and extract key information and keywords.

[0074] In this embodiment, multi-agent: multi-agent debate is an effective way to encourage divergent thinking of LLMs (Liangetal., 2023) and improve the factuality and reasoning ability of LLMs (Duetal., 2023). In both works, multiple LLM reasoning instances are constructed as multiple agents to solve problems through agent debate. Each agent is only an LLM reasoning instance, no tools or humans are involved, and the dialogue between agents needs to follow a predefined order. These works attempt to build LLM applications through multi-agent dialogue

[0075] This embodiment has a GAN-inspired answer generation and discrimination mechanism: Although prompt predefines the step-by-step processing strategy of the large model, this COT processing method simply relies on the logical reasoning ability of a large model and often cannot achieve the desired effect. Inspired by the GAN idea, this embodiment adopts the idea of ​​processing two large models together. That is, one large model + custom COT strategy for step-by-step processing is used as the generator, and another large model + custom COT strategy for step-by-step processing is used as the discriminator to evaluate and screen the output of the generator.

[0076] In the case of recalling 2 reference materials, in order to ensure that the output content of the large model strictly follows the requirements of the reference materials, this embodiment adopts a multi-step cot strategy in the generator. That is:

[0077] 1) Find the answer related to the question in the first reference.

[0078] 2) Find the answer related to the question from the second document

[0079] 3) Summarize the previous two answers, and try not to mix in subjective answers.

[0080] The discriminator also adopts the COT step-by-step strategy, namely:

[0081] 1) Confirm the correlation between the answers output by the generator and the reference materials, and whether there are any inconsistencies.

[0082] 2) If after analysis, you believe that the answer output by the generator is completely derived from the reference material, then reply "completely consistent"; otherwise, reply "not completely consistent" without mixing in any analytical content.

[0083] Compared with the multi-agent strategy, this method has a simpler workflow because only prompt defines a strict paradigm for large models. It can also better stimulate the reasoning potential of large models and achieve good results with models of less than 10B.

[0084] This embodiment can enhance the logic and coherence of the answer. The pre-generator model combines the COT thinking chain to analyze the problem, extract key information and keywords, and generate a more accurate intermediate representation. This provides high-quality input for the post-generator, further improving the quality of the final answer. Although COT can improve the logic and coherence of the generated answers, it relies on the step-by-step reasoning ability of the model. If the model makes an error in a certain step, it may affect the entire reasoning process. In addition, COT may increase the computational complexity, resulting in longer processing time.

[0085] Compared with traditional uncertainty scoring methods, the GAN-inspired discriminator model can comprehensively evaluate multiple factors such as grammatical correctness, contextual relevance, and logical coherence. This comprehensive evaluation method not only reduces hallucinations, but also ensures that the generated answers have high quality in multiple dimensions.

[0086] This embodiment can improve the robustness and reliability of the system. Although the multi-agent architecture can improve the accuracy and robustness of intent recognition, its disadvantages are that it takes up more resources, takes a long time, has strong autonomy, and is difficult to control. In contrast, the GAN-inspired discriminant mechanism uses a centralized method to evaluate the quality of the answer using a powerful discriminator model, avoiding the resource consumption and management complexity problems in the multi-agent architecture. At the same time, this centralized evaluation mechanism is easier to manage and control, improving the overall robustness and reliability of the system.

[0087] This embodiment can reduce misjudgments and hallucinations. Traditional uncertainty scoring mainly relies on the entropy value of a single token, which can easily lead to misjudgments. The GAN-inspired discriminator model can more accurately identify answers with hallucinations by comprehensively evaluating grammatical correctness, contextual relevance, and logical coherence. This comprehensive evaluation method significantly reduces the possibility of misjudgment and improves the reliability of the final answer and user satisfaction.

[0088] Embodiment 2:

[0089] The answer generation system based on the generation-discrimination mechanism and chain thinking includes:

[0090] The question acquisition module is configured to: acquire questions raised by users;

[0091] The problem analysis module is configured to extract the problem scenario and keywords involved in the problem, and generate an intermediate representation containing key information:

[0092] The sample answer module is configured to: find two question-answer pairs that are most similar to the user's question from the knowledge base according to the intermediate representation as sample answers; select the corresponding knowledge base through the extracted scenario, find the document fragments that are most relevant to the user's question, and recall and re-rank the found document fragments;

[0093] The answer generation module is configured to: use the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies;

[0094] The answer evaluation and output module is configured to: when the generated preliminary answer or sample answer can be used as a valid response to the current user's question, return the corresponding answer to the user; if not, regenerate the answer.

[0095] Embodiment three:

[0096] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the answer generation method based on the generation-discrimination mechanism and chain thinking as described in the above-mentioned embodiment 2 are implemented.

[0097] Embodiment 4:

[0098] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the answer generation method based on the generation-discrimination mechanism and chain thinking as described in the above-mentioned embodiment 2 are implemented.

[0099] The steps involved in the above embodiments 2 to 4 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The answer generation method based on the generation-discrimination mechanism and chain thinking is characterized by: The following steps are involved: Get the question raised by the user, use the front generator model to extract the scenario where the question is located and the keywords involved in the question, and generate an intermediate representation containing key information: Based on the intermediate representation, the pre-generator model is used to find the two question-answer pairs most similar to the user's question from the knowledge base as sample answers; the recall model is used to select the corresponding knowledge base based on the extracted scenarios, find the document fragments most relevant to the user's question, and recall and re-rank the found document fragments; The post-generator model uses the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies; The discriminator model is used to evaluate the answer. When the generated preliminary answer or sample answer can serve as a valid response to the current user's question, the corresponding answer is returned to the user. If not, the answer is regenerated.

2. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 1, characterized in that: The pre-generator model is used to receive user questions, and the scenario where the question is located and the keywords involved in the question are extracted based on the COT thinking chain method.

3. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 1, characterized in that: The recall model is used to select the corresponding knowledge base through the extracted scenarios, the keywords extracted by the predecessor generator are vectorized, and the document fragments most relevant to the user's question are found from the knowledge base through the search and matching algorithm, and the recalled document fragments are reordered.

4. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 1, characterized in that: The post-generator model receives two inputs, the recalled document fragment and the user question, and obtains the top K re-ranked results as reference context. The generator summarizes the response to the user question from each result step by step, and generates a preliminary answer by summarizing all the responses.

5. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 1, characterized in that: The discriminator model is combined with the COT thinking chain method to evaluate whether the generated preliminary answers or sample answers can serve as effective responses to the current user's questions.

6. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 1, characterized in that: When the generated preliminary answer or sample answer can serve as a valid response to the current user's question, the answer is qualified and the qualified answer is returned to the user through the user interface.

7. The answer generation method based on generation-discrimination mechanism and chain thinking as claimed in claim 6, characterized in that: When the generated preliminary answer or sample answer is unsatisfactory, the post-generator model is used to regenerate the answer in a parallel manner.

8. The answer generation system based on the generation-discrimination mechanism and chain thinking is characterized by: include: The question acquisition module is configured to: acquire questions raised by users; The problem analysis module is configured to extract the problem scenario and keywords involved in the problem, and generate an intermediate representation containing key information: The sample answer module is configured to: find two question-answer pairs that are most similar to the user's question from the knowledge base according to the intermediate representation as sample answers; select the corresponding knowledge base through the extracted scenario, find the document fragments that are most relevant to the user's question, and recall and re-rank the found document fragments; The answer generation module is configured to: use the COT thinking chain method to take the first K re-ranked document fragments as reference contexts, summarize the replies to answer the user's questions, and generate a preliminary answer by summarizing all the replies; The answer evaluation and output module is configured to: when the generated preliminary answer or sample answer can be used as a valid response to the current user's question, return the corresponding answer to the user; if not, regenerate the answer.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the steps in the answer generation method based on the generation-discrimination mechanism and chain thinking as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the answer generation method based on the generation-discrimination mechanism and chain thinking as described in any one of claims 1 to 7 are implemented.

Images