A method and system for answer reasoning based on a large language model that integrates memory and iterative optimization.

Abstract

This invention discloses a method and system for answer reasoning using a large language model that integrates memory and iterative optimization. First, the invention uses an answer verification module to generate natural language feedback information for the answers to sub-questions, preventing subsequent reasoning steps from deviating from the correct reasoning path after an error occurs in one step. Second, the invention uses a sub-question verification module to verify and prune sub-questions generated during the thought chain reasoning process, preventing irrelevant sub-question information from interfering with the model. Finally, by utilizing the feedback information, the invention iteratively optimizes the model's output, simulating the human process of self-reflection and correction, preventing the large language model from making the same error in the same reasoning step. This invention does not fine-tune the large language model itself, saving the resources needed for fine-tuning the large language model and improving its reasoning performance.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and large language models. Specifically, it relates to a method for answer reasoning of large language models that integrates memory and iterative optimization. The method aims to obtain feedback on the natural language form of the model output based on the checking module, and improve the answering performance of the large language model by prompting the feedback. The method also uses the memory module to optimize the thought chain reasoning process of the large language model. Background Technology

[0002] Large language models are an important technology in the field of artificial intelligence. In natural language processing tasks, large language models can learn more language rules and patterns, thus understanding and generating text more accurately. Furthermore, large language models can improve the fluency and naturalness of dialogue through multi-turn conversations and contextual understanding. In terms of applications, large language models are widely used in various fields, such as natural language translation, intelligent customer service, sentiment analysis, and text generation.

[0003] In existing technologies, large language models such as GPT-3 can generate seemingly reasonable text and perform some simple reasoning tasks. However, these models still have many shortcomings in their reasoning abilities. For example, when faced with relatively complex input questions, large language models often output incorrect answers. First, large language models perform poorly on logical reasoning tasks. Most language models are trained on large-scale corpora, and while they can generate seemingly reasonable text, their performance may be less than ideal for logical reasoning tasks, especially those requiring rigorous logic and proof. For tasks that require understanding complex mathematical structures or concepts, the model needs to be able to understand and manipulate abstract concepts and structures. Second, large language models lack self-reflection capabilities; they cannot self-reflect and correct themselves like humans. Even if the model makes mistakes during reasoning, it cannot consciously recognize and correct these errors, and in the same reasoning step, large language models will repeatedly make the same mistakes.

[0004] Currently, optimization techniques for the reasoning ability of large language models mainly include external knowledge bases, thought chains, and optimization through feedback information. External knowledge base technology supports the model's reasoning process by providing additional information. Thought chain technology guides the model to generate a series of intermediate reasoning steps, simulating the human problem-solving process, thus enabling the model to make more accurate predictions on problems requiring multi-step reasoning. Feedback technology uses multiple feedback generation modules to generate natural language feedback on the model's output, improving the model's accuracy and reliability. However, traditional methods of building external knowledge bases using tools like Faiss struggle to accurately retrieve relevant information; simple thought chain technology still cannot guide the model to solve most complex problems, and errors in one step of the thought chain generation process often prevent the model from outputting the correct answer.

[0005] Therefore, based on the research content, this invention designs a large language model answer reasoning method that integrates memory and iterative optimization, providing technical support for improving the reasoning performance of large language models. Summary of the Invention

[0006] This invention addresses the reasoning process of large language models by utilizing verification modules, memory modules, and feedback information to improve the performance of large models in answer reasoning. This paper presents a method for answer reasoning in large language models that integrates memory and iterative optimization, providing technical support for improving the reasoning performance of large language models.

[0007] This invention provides a method for answer reasoning using a large language model that integrates memory and iterative optimization, comprising the following steps:

[0008] S1, Input question Q, initialize known information K to be empty.

[0009] S2 uses a large language model to generate a set of M sub-questions related to question Q based on known information K and question Q.

[0010] S3. For each subproblem, use the subproblem verification module to verify the relevance of the generated subproblem to problem Q.

[0011] S4. If the subproblem is related to question Q, use the answer verification module to generate the answer to the subproblem.

[0012] In step S5, add the sub-problem and its corresponding answer to the known information to form new known information K. Determine whether the answer to question Q can be obtained based on K. If the answer can be obtained, output the answer; otherwise, repeat steps S2, S3, S4, and S5 until the model outputs the answer to question Q.

[0013] Furthermore, step S2 specifically includes the following sub-steps:

[0014] S21, providing hints for designing large language models. Q This is used to prompt the large language model to generate a sub-problem set S = {q1, q2, q3...q}. l}

[0015] S22, when generating the subproblem set S, prompt P Q It includes an example, known information K, and a question Q. The example is used to prompt the model to generate a set of sub-questions S according to a certain format, and the known information K is initially empty.

[0016] Furthermore, step S3 specifically includes the following sub-steps:

[0017] S31, Using the subproblem verification module V Q The sub-problem set S is examined, and feedback information f is generated, which includes the relevance of the sub-problem to problem Q.

[0018] S32. Add the sub-problems with high relevance to the set of relevant sub-problems S. A middle.

[0019] S33, if S A The number of neutron problems is less than L S The feedback information f, known information K, and problem Q are input into the model to generate new sub-problems, and the sub-problem verification module V is used. Q Check until S A The number of neutron problems is at least L S indivual.

[0020] Furthermore, step S4 specifically includes the following sub-steps:

[0021] S41, Design an external component A to assist the large language model in solving the problem.

[0022] S42, Design Hint P A This is used to prompt the output o of the large language model's generation of sub-problems, and to prompt P. A It includes information I provided by external components, feedback information f, and sub-questions q.

[0023] S43, using the answer verification module V A Check o and generate feedback information f.

[0024] S44, if f represents that o passes the check, then o is treated as a subproblem q. i The answer.

[0025] S45, if f indicates that o failed the check, repeat steps S42, S43, S44 and S45 to regenerate o.

[0026] Furthermore, step S5 specifically includes the following sub-steps:

[0027] S51, add the sub-problem set and the corresponding answer to the known information K.

[0028] S52: Use the model to determine if the answer to question Q can be obtained based on K. If the answer to question Q can be obtained, output the answer. If not, repeat S2, S3, S4, and S5 until the answer to question Q can be obtained based on K.

[0029] Furthermore, in step S2, the generation of the sub-question set can be accomplished using a large language model that provides sub-question decomposition example hints for similar questions. A sub-question decomposition example vector database can be constructed, and related questions can be retrieved as hints based on similarity.

[0030] Furthermore, in step S3, the output f of the subproblem verification module represents the subproblem q. i The relevance of the sub-questions to question Q is determined. The scoring range and relevance threshold for the model are set, and sub-questions with relevance exceeding the threshold are added to a new sub-question set. If the number of sub-questions in the set is insufficient, feedback information f is used to prompt the model to generate new questions. f includes previously generated sub-questions to prevent the model from continuously outputting the same questions.

[0031] Furthermore, in step S4, external component A may include vector databases, knowledge graphs, and search APIs, etc., and different external components can be selected for different types of questions. Based on the input question, each component is used to obtain question-related information. The information provided by each component is integrated, and a large language model is used to extract question-related information from the integrated text.

[0032] Furthermore, in step S4, the output f of the answer verification module represents o. i Is it a subproblem q? i The answer. If o i If an error occurs, the validation fails. Then, based on the prompt f, the model generates a new output, and the validation process is repeated until validation passes or the maximum number of iterations is reached. If validation passes, q is... i and o i The feedback information f is added to the known information K. The purpose of the feedback information f is to prompt the model to generate the correct output based on the existing information. By utilizing the feedback information, the accuracy of the model in solving sub-problems can be improved.

[0033] Furthermore, in step S3, an upper limit T is set for the number of iterations generated for the subproblem. Q The number of times it is repeatedly generated reaches T. Q Then, if the subproblem set S A If the condition is empty, then sort all subproblems in descending order of relevance and take L. s The question was added to S AIn the middle, proceed to step S4; if S A If it is not empty, proceed directly to step S4. In S4, set o. i Upper limit of iteration generation L A In answering subproblem q i At that time, if o i The number of generation times reached L A Then o i As question q i If the answer is correct, proceed to step S5.

[0034] This invention also provides a large language model answer reasoning system that integrates memory and iterative optimization, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the large language model answer reasoning method that integrates memory and iterative optimization as described in the above technical solution.

[0035] Compared with existing research and technology, the present invention has the following advantages:

[0036] This invention generates natural language feedback information for the answers to sub-questions through an answer verification module, thus preventing the large language model from deviating from the correct reasoning path in subsequent steps after an error occurs in a certain step.

[0037] This invention uses a sub-problem verification module to verify and prune sub-problems generated in the thought chain reasoning, thereby preventing irrelevant sub-problem information from interfering with the large language model.

[0038] This invention utilizes feedback information to iteratively optimize the model's output, simulating the human process of self-reflection and correction, thus avoiding large language models from producing the same errors in the same reasoning step. Attached Figure Description

[0039] Figure 1 A schematic diagram of the overall process of the large language model answer reasoning method that integrates memory and iterative optimization provided by the present invention;

[0040] Figure 2 The flowchart for the sub-problem generation process of the prompt model in the large language model answer reasoning method that integrates memory and iterative optimization provided by this invention;

[0041] Figure 3 The flowchart for checking the relevance of sub-questions in the large language model answer reasoning method that integrates memory and iterative optimization provided by the present invention;

[0042] Figure 4 The flowchart for solving sub-problems is shown in the large language model answer reasoning method that integrates memory and iterative optimization provided by this invention. Detailed Implementation

[0043] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, this invention provides a method for answer reasoning using a large language model that integrates memory and iterative optimization, comprising the following steps:

[0045] S1, Input question Q, initialize known information K to be empty.

[0046] S2 uses a large language model to generate M sub-problems related to question Q based on known information K and question Q.

[0047] S2 specifically includes the following steps:

[0048] S21, as Figure 2 As shown, this is a model design hint P. Q This is used to prompt the large language model to generate a sub-problem set S = {q1, q2, q3...q}. l}=M(P Q (Q,K), where l is a hyperparameter representing the upper limit of the size of the subproblem set, and M is the large language model;

[0049] S22, when generating the subproblem set S, prompt P Q It contains an example, known information K, and a question Q, where the example is used to prompt the model to generate a set of sub-questions S according to a format, and the known information K is initially empty;

[0050] Furthermore, in S2, the generation of sub-problem sets can be accomplished using a large language model that provides hints through sub-problem decomposition examples of similar problems. A sub-problem decomposition example vector database can be constructed, and decomposition examples can be retrieved as hints based on similarity.

[0051] S3, for each subproblem, use the subproblem verification module to verify the relevance of the generated intermediate problems to problem Q.

[0052] S3 specifically includes the following steps:

[0053] S31, such as Figure 3 As shown, the sub-problem verification module V is used. Q Each subproblem in S is examined, and feedback information f = M(P) is generated. VQ ,Q,S), where P VQ f is used to score the relevance between the sub-problem and question Q, where f includes the relevance between the sub-problem and question Q;

[0054] S32. Add highly relevant subproblems to the subproblem set S. A middle;

[0055] S33, if S A The number of neutron problems is less than L S The feedback information f, known information K, and problem Q are input into the model to generate new sub-problems, and the sub-problem verification module V is used. Q Check until S A The number of neutron problems is at least L S indivual.

[0056] Furthermore, in S3, the output f of the subproblem verification module represents the subproblem q. i The relevance to question Q. Set the scoring interval [Nmin, Nmax] and relevance threshold FL for the hyperparameter model. If f represents the relevance of q... i If the correlation between Q and FL is greater than that between q and FL, then q will be... i Add to set S A middle.

[0057] Furthermore, in S3, if the number of sub-problems does not meet the requirements, feedback information f is used to prompt the model to generate new sub-problems. f includes sub-problems that have already been generated, preventing the large language model from continuously outputting the same question.

[0058] Furthermore, in step S3, an upper limit T is set for the number of iterations generated for the subproblem. Q The number of times it is repeatedly generated reaches T. Q Then, if the subproblem set S A If the condition is empty, then sort all subproblems in descending order of relevance and take L. S The question was added to S A In the middle, proceed to step S4; if S A If it is not empty, proceed directly to step S4. In S4, set o. i Upper limit of iteration generation L A In answering subproblem q i At that time, if o i The number of generation times reached L A Then o i As question q i If the answer is correct, proceed to step S5.

[0059] S4. If the intermediate question is related to question Q, use the answer verification module to generate the answer to the verified sub-question.

[0060] S4 specifically includes the following steps:

[0061] S41, as Figure 4 The design shown uses external component A to assist in solving the problem.

[0062] S42, Design Hint P A Used to prompt the model to generate pairs of S AAll subproblems {q1,q2,q3...q} l The output of} is {o1,o2,o3...o} l}=M(P A ,I,F,q i ), prompt P A Includes information I provided by external components, feedback information F, and sub-questions q. i .

[0063] S43, using the answer verification module V A For {o1,o2,o3...o l Perform the check and generate feedback information f a ={f1,f2,f3...f l}=M(P VA ,S A (o1, o2, o3... o1), where P VA It is a prompt used to improve the model for inspection;

[0064] S44, if f i This indicates that the check has been passed and o will be... i As a subproblem q i The answer;

[0065] S45, if f i o i If the check fails, f i Join P A Regenerate the output, regenerate o i .

[0066] Furthermore, in S4, external component A may include vector databases, knowledge graphs, and search APIs, etc., and different external components can be selected for different types of questions. Based on the input question, each component is used to obtain information related to the question. The information provided by each component is integrated, and the integrated information format is [[doc1,source...]. i ],[doc2,source2]..], where doc i For the retrieved question, source i For doc i The source of the information is determined using a large language model to extract question-related information from the integrated text. In this invention, a vector database and a search API are used to assist the model in solving the problem.

[0067] Furthermore, in S4, the process of constructing the vector database is as follows:

[0068] 1. Prepare training data. Construct a query dataset and, for each query in the dataset, prepare corresponding positive and negative sample documents;

[0069] 2. Model Training. Two independent BERT models were used as the query encoder (BQ) and document encoder (BA), respectively. The training objective was to minimize the distance between relevant pairs of queries and documents.

[0070] 3. Use the document encoder BA to encode each document in the document collection, converting it into a fixed-length vector p;

[0071] 4. Use query encoder BQ to encode each query in the query set, converting it into a fixed-length vector q;

[0072] 5. Let D be the training data containing m instances, each instance containing a query q. i and a related document And n irrelevant documents The optimized loss function is the negative log-likelihood function, as follows:

[0073]

[0074] Furthermore, in S4, the usage process of the vector database is as follows:

[0075] 1. Use the query encoder BQ to encode the user's question, converting it into a fixed-length vector q;

[0076] 2. Calculate the cosine similarity between q and the encoding vectors of all documents, and take the document with the highest similarity as the result.

[0077] Furthermore, in S4, the workflow for searching the API is as follows:

[0078] 1. Call the API interface to obtain the set of web pages related to question Q, W = {w1, w2, ... w...} n};

[0079] 2. Use the query encoder BQ to encode the question as q, and use BA to encode the text content of the webpage as p, then calculate the similarity e. sim(q,p) Remove web pages with low similarity from W to obtain W. f ;

[0080] 3. Extract W f The text information in each webpage serves as a reference for solving the problem.

[0081] Furthermore, in step S4, the output f of the answer verification module a o iIs it a subproblem q? i The answer, if o i If an error occurs, the validation fails. Then, based on the prompt f, the model generates a new output, and the validation process is repeated until validation passes or the maximum number of iterations is reached. If validation passes, q is... i and o i The feedback information f is added to the known information K. In S4, the role of the feedback information f is to prompt the model to generate the correct output based on the existing information. By utilizing the feedback information, the accuracy of the model in solving sub-problems is improved.

[0082] In step S5, add the sub-problem and its corresponding answer to the known information to form new known information K. Determine whether the answer to question Q can be obtained based on K. If the answer can be obtained, output the answer; otherwise, repeat steps S2, S3, S4, and S5 until the model outputs the answer to question Q.

[0083] S5 specifically includes the following steps:

[0084] S51, add the sub-problem set and the corresponding answers to the known information K;

[0085] S52: Use the model to determine whether the answer to question Q can be obtained based on K. If it can, generate the answer to question Q as the final output. If not, repeat S2, S3, S4 and S5 until the answer to question Q can be obtained based on K.

[0086] This invention also provides a large language model answer reasoning system that integrates memory and iterative optimization, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the large language model answer reasoning method that integrates memory and iterative optimization as described in the above technical solution.

[0087] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for answer reasoning using a large language model that integrates memory and iterative optimization, characterized in that, Includes the following steps: Step S1: Input question Q, initialize known information K to be empty; Step S2: Using a large language model, generate M sub-question sets related to question Q based on known information K and question Q; Step S3: For each sub-problem, use the sub-problem verification module to verify the relevance of the generated intermediate problem to problem Q; Step S3 specifically includes the following sub-steps: S31, Using the subproblem verification module V Q Examine the sub-problem set S and generate feedback information f, where f contains the relevance between the sub-problem and problem Q; S32. Add the sub-problems with high relevance to the set of relevant sub-problems S. A middle; S33, if S A The number of neutron problems is less than L S The feedback information f, known information K, and problem Q are input into the large model to generate new sub-problems, and the sub-problem verification module V is used. Q Check until S A The number of neutron problems is at least L S indivual; Step S4: If the sub-problem is related to question Q, use the answer verification module to generate the answer to the verified sub-problem; Step S4 specifically includes the following sub-steps: S41, Design an external component A to assist the large language model in solving the problem; S42, Design Hint P A This is used to prompt the large language model to generate a set of relevant sub-problems S. A All subproblems {q1,q2,q3...q} l The output of} is {o1,o2,o3...o} l }=M(P A ,I,F,q i ), where l is a hyperparameter representing the upper limit of the size of the subproblem set; Hint P A Includes information I provided by external components, feedback information F, and sub-questions q. i ; S43, using the answer verification module V A For {o1,o2,o3...o l Perform the check and generate feedback information f a ={f1,f2,f3...f l }=M(P VA ,S A ,o1,o2,o3...o l ), where P VA It is used to improve inspection speed; S44, if f i This indicates that the check will pass and o will be... i As a subproblem q i The answer; S45, if f i o i If the check fails, f i Join P A Regenerate the output, regenerate o i ; In step S5, the sub-problem and its corresponding answer are added to the known information to form new known information K. It is then determined whether the answer to question Q can be obtained based on K. If the answer can be obtained, the answer is output. Otherwise, steps S2, S3, S4 and S5 are repeated until the large language model outputs the answer to question Q.

2. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 1, characterized in that: Step S2 specifically includes the following sub-steps: S21, providing hints for designing large language models. Q This is used to prompt the large language model to generate a sub-problem set S = {q1, q2, q3...q}. l }, where l is a hyperparameter representing the upper limit of the size of the subproblem set; S22, when generating the subproblem set S, prompt P Q It contains an example, known information K, and a question Q, where the example is used to prompt the large language model to generate a set of subquestions S according to a format, and the known information K is initially empty.

3. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 1, characterized in that: Sub-problems are generated by using sub-problem decomposition examples of similar problems as hints in a large language model. A sub-problem decomposition example vector database is constructed, and decomposition examples are retrieved based on similarity as hints.

4. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 1, characterized in that: In step S3, the output f of the subproblem verification module represents the subproblem q. i The model is set to score the relevant questions Q and set a relevant threshold. Sub-questions with a relevant score exceeding the threshold are added to a new set of sub-questions. If the number of sub-question sets does not meet the requirements, feedback information f is used to prompt the model to generate new questions. In step S3, an upper limit T is set for the number of iterations generated for the subproblem. Q The number of times it is repeatedly generated reaches T. Q Then, if the subproblem set S A If the condition is empty, then sort all subproblems in descending order of relevance and take L. S The question was added to S A In the middle, proceed to step S4; if S A If it is not empty, proceed directly to step S4.

5. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 1, characterized in that: In step S4, external component A includes a vector database, a knowledge graph, and a search API; different external components are selected for different types of questions. The construction process of the vector database is as follows: (1) Construct a query dataset and prepare corresponding positive and negative sample documents for each query in the query dataset; (2) Two independent BERT models were used as query encoder BQ and document encoder BA, respectively, with the training objective being to make the relevance pairs between queries and documents have a smaller distance; (3) Use document encoder BA to encode each document in the document collection and convert it into a fixed-length vector p; (4) Use query encoder BQ to encode each query in the query set and convert it into a fixed-length vector q; (5) Let D be the training data containing m instances, each instance containing a query q. i and a related document And n irrelevant documents The optimized loss function is the negative log-likelihood function, as follows: sim is the similarity function; The usage process of a vector database is as follows: (a) Use query encoder BQ to encode the user's question and convert it into a fixed-length vector q; (b) Calculate the cosine similarity between q and the encoding vectors of all documents, and take the document with the highest similarity as the result.

6. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 5, characterized in that: The workflow for the search API is as follows: A1, Call the API interface to obtain the set of web pages related to question Q, W = {w1, w2, ... w...} n }; A2, use query encoder BQ to encode the question as q, use BA to encode the text content of the webpage as p, and calculate the similarity e. sim(q,p) Remove web pages with low similarity from W to obtain W. f ; A3, extract W f The text information in each webpage serves as a reference for solving the problem.

7. The large language model answer reasoning method integrating memory and iterative optimization as described in claim 1, characterized in that: In step S4, the output f of the feedback module represents o. i Is it a subproblem q? i The answer, if o i If an error occurs, the verification fails. Then, based on the f prompt, the model generates a new output and repeats the verification steps until the verification passes or the maximum number of iterations is reached. If the verification passes, then q will be... i and o i Add it to the known information K; In step S4, set o i Upper limit of iteration generation L A In answering subproblem q i At that time, if o i The number of generation times reached L A Then o i As question q i If the answer is correct, proceed to step S5.

8. A large language model answer reasoning system that integrates memory and iterative optimization, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the large language model answer reasoning method that integrates memory and iterative optimization as described in any one of claims 1-7.

Images