Long context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompt

Abstract

The invention relates to a long context knowledge reasoning method based on big language model knowledge extraction and self-consistent thinking chain prompt, which extracts high-quality abstract knowledge from a context in a segmentation pre-filling process, enhances the understanding ability of a big language model for a complex context, and improves the understanding ability of the big language model for the complex context in a reasoning stage. Through constructing a thinking chain reasoning prompt, a logically strict transparent reasoning process is gradually generated by using the extracted high-quality abstract knowledge, and the reasoning ability and interpretability of a large language model in the aspect of complex reasoning problems are enhanced. In order to comprehensively cover various aspects of the question, the method encourages to generate a plurality of thinking chains, different thinking chains can perform reasoning through different paths so as to provide diversified visual angles and answers, and then all generated reasoning processes are reflected and verified through self-consistent prompt, so that the model identifies and corrects wrong reasoning paths, and the reasoning efficiency is improved. Therefore, the accuracy of final output is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a knowledge reasoning method, and more specifically to a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts. Background Art

[0002] With the widespread use of computers and the rapid development of the Internet, data and information are growing explosively. Faced with the massive amount of data on the Internet, how to automatically infer useful new knowledge from the massive amount of unprocessed information through machines has gradually become a hot research topic. Information usually refers to unprocessed data or facts, which can be raw, disorganized, or have been preliminarily processed. Knowledge is the product of processing, analyzing and synthesizing information. It is the understanding and insight gained through learning, practice and thinking, and includes the understanding of the meaning, value, relevance and application methods of information. Long-context knowledge reasoning is a task of deriving new knowledge based on large-scale unprocessed information. It aims to extract and integrate information from texts containing a large amount of information and complex relationships, so as to derive new knowledge and get answers to solve problems. Compared with traditional knowledge reasoning, long-context knowledge reasoning includes but is not limited to the following differences: First, in terms of knowledge scope, ordinary knowledge reasoning usually deals with shorter or limited knowledge fragments, focusing on the derivation of single facts or simple relationships; while long-context knowledge reasoning needs to process large and complex text data, covering a wide range of topics and multi-level relationships, and can reason in a larger knowledge context. Secondly, in terms of contextual dependency, general knowledge reasoning has a low reliance on context during the reasoning process, and mainly relies on clear knowledge bases or predefined rules; while long-context knowledge reasoning is highly dependent on contextual information, and needs to understand and integrate associations across paragraphs, chapters, and even documents to ensure the accuracy and consistency of the reasoning results. In addition, in terms of computational complexity, general knowledge reasoning has relatively low computational complexity due to the small amount of information processed, and is suitable for application scenarios with high real-time requirements; while long-context knowledge reasoning needs to process a large amount of data and has high computing resource requirements, and usually requires the use of advanced model architectures and optimization techniques to improve efficiency. As a key component for achieving strong artificial intelligence, long-context knowledge reasoning plays an important role in many fields. In complex question-answering systems, long-context knowledge reasoning can integrate relevant information from different documents to provide accurate and comprehensive answers; in summary generation systems, for long documents such as legal documents, technical reports or academic papers, long-context knowledge reasoning can automatically extract key knowledge, generate summaries or extract core conclusions to help users quickly understand the document content; in information integration systems, for scenarios such as news aggregation and intelligence analysis, long-context knowledge reasoning can extract relevant knowledge from reports from multiple sources and integrate them into a unified view to help users fully understand all aspects of the event; in recommendation systems, by understanding the deep meaning and intention of user queries, long-context knowledge reasoning can find the most relevant content in massive documents, provide more accurate search results, and improve user experience.

[0003] With the continuous advancement of deep learning technology, large language models have learned rich semantic representations and contextual dependencies from massive text data with the help of complex model parameters, showing powerful reasoning capabilities. However, knowledge reasoning methods based on large language models still face many challenges when processing long context inputs, including but not limited to the following: First, most existing large language models use the Transformer architecture, which can capture the global dependencies of each pair of tokens in the input, so that the model has the ability to handle complex relationship sequences. However, as the length of the input sequence increases, the time and space complexity of the Transformer architecture grows quadratically, which limits the context input length of the large language model. Secondly, complex reasoning tasks usually require aggregating multiple document information for multi-stage reasoning, which often reduces the performance of large language models when facing complex reasoning problems. In addition, due to the black box characteristics of large language models, the interpretability of their output is low, and it is difficult to trace the specific decision basis in the reasoning process, which not only affects the trust of the model, but also limits decision support in practical applications. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts.

[0005] The technical solution of the present invention is: a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts, comprising the following steps:

[0006] S1), splitting long contextual cues into several relatively short text paragraphs;

[0007] S2), for the several text paragraphs obtained in step S1), select a text paragraph c according to the order, input c into the large language model, and obtain a pre-filled key value cache;

[0008] S3), for the single text paragraph c obtained in step S2), construct an extraction prompt instruction to extract candidate knowledge from c; then construct a classification prompt instruction to screen the candidate knowledge to obtain summary knowledge related to q;

[0009] S4), continuously repeating the process of steps S2) and S3) until all the contexts are pre-filled, and obtaining a large language model with a complete context key value cache and a set of summary knowledge related to q;

[0010] S5) For the knowledge set obtained in step S4), construct multiple thought chain reasoning prompt instructions, generate multiple reasoning chains through the large language model, and then construct self-consistency prompt instructions to generate the final result.

[0011] In the above method, in step S1), the long context prompt is split into several relatively short text paragraphs, as follows:

[0012] Token is used as the unit of measurement for text length, and the tiktoken tokenizer of GPT-4 is used as a tool to measure the number of text tokens;

[0013] Set L tokens as the maximum length of each text paragraph, and split the long context into several short sequence text paragraphs with a length not exceeding L in sequence.

[0014] In the above method, in step S2), for the multiple text paragraphs obtained in step S1), a text paragraph c is selected according to the order, and c is input into the large language model to obtain a pre-filled key value cache, which is specifically as follows:

[0015] The key-value cache uses the key-value pair cache mechanism to store and reuse previously calculated attention information, avoiding repeated calculation of attention information. It effectively addresses challenges such as high computing resource consumption and slow reasoning speed in the inference phase. The process is as follows:

[0016] The text form of c is first segmented by a tokenizer and converted into a numerical representation. Then, each word is converted into a vector through a word embedding method to obtain the vector representation of c.

[0017] The vector representation c is forward propagated in the model. In the Transformer structure of the model, the input is processed through multiple layers, each of which contains a self-attention mechanism and a feedforward neural network.

[0018] In the self-attention mechanism, the model generates three sets of vectors, "query", "key" and "value", to calculate the degree of "attention" of each word in c to other words, and saves the two sets of vectors, key and value, as c to pre-fill the key-value cache.

[0019] In the above method, in step S3), for the single text paragraph c obtained in step S2), an extraction prompt instruction is constructed to extract candidate knowledge from c, and then a classification prompt instruction is constructed to screen the summary knowledge related to q from the candidate knowledge, as follows:

[0020] The extraction prompt instruction consists of a text paragraph c, a question q, and other prompt texts. The knowledge related to q in c is extracted through a large language model. If the knowledge exists, it is output. If not, a negative answer is generated. Therefore, the candidate knowledge is not completely valid knowledge.

[0021] The classification prompt instruction consists of candidate knowledge, question q and other prompt texts. The large language model is used to verify whether the candidate knowledge is valid knowledge. If the answer is positive, it is retained as summary knowledge for subsequent reasoning. If the answer is negative, the candidate knowledge is discarded.

[0022] In the above method, in step S5), for the summary knowledge set obtained in step S4), multiple thought chain reasoning prompt instructions are constructed, multiple reasoning chains are generated through the large language model, and then self-consistency prompt instructions are constructed to generate the final result, which is as follows:

[0023] Self-consistent chain of thought prompts is an improved chain of thought reasoning method. Its core goal is to select the most consistent reasoning result after generating multiple reasoning chains, thereby improving the accuracy of language models in complex reasoning tasks. In traditional chain of thought prompts, the model often draws conclusions through a single chain of reasoning, while self-consistent chain of thought prompts generates multiple independent reasoning chains to provide multiple solution paths for each problem, then reflects on all generated reasoning chains, identifies and corrects the wrong reasoning steps, and finally returns a verified, self-consistent and consistent reasoning path, thereby effectively reducing the errors caused by randomness and inaccuracy in single reasoning, and improving the robustness and accuracy of the model in complex reasoning tasks. Self-consistent chain of thought improvement is usually divided into two stages: chain of thought prompts and self-consistent prompts. Chain of thought prompts aim to decompose complex problems into multiple solvable sub-problems and generate different reasoning paths for each problem. Each generated reasoning chain may provide different solutions. Self-consistent prompts aim to identify inconsistent conclusions and troubleshoot possible problems in reasoning, ensuring that the final reasoning path is not only logical but also self-consistent.

[0024] The thinking chain reasoning prompt consists of a summary knowledge set, sample questions and answers, real questions q, and a prompt text of "Let's think step by step". According to the different sample questions and answers, multiple thinking chain reasoning prompt instructions are constructed;

[0025] For the large language model with complete context key-value cache obtained in step S4), the first thought chain reasoning prompt instruction is pre-filled into the large model. At this time, the key-value cache includes [complete context, extracted summary knowledge set, example question and answer, real question and "Let's think step by step" prompt text], and the large language model will generate the instruction corresponding reasoning chain;

[0026] Remove the key-value cache portion corresponding to the first thinking chain reasoning prompt instruction from the large language model, that is, restore the key-value cache of the large language model to the state before the cached thinking chain prompt instruction, and then pre-fill the next thinking chain reasoning prompt instruction, and repeat this process until the reasoning chain corresponding to each thinking chain reasoning prompt instruction is obtained;

[0027] The self-consistency prompt consists of all reasoning chains and other prompt texts. By comparing the reasoning process and conclusions of each reasoning chain, inconsistent or erroneous reasoning paths are identified, and the erroneous reasoning steps in these paths are corrected, and finally a verified, self-consistent and consistent reasoning path is returned. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 The present invention is a flowchart of a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts.

[0030] Figure 2 The figure is a flow chart of segment pre-filling and summary knowledge extraction in an embodiment of the present invention.

[0031] Figure 3 It is a flowchart of the self-consistent thinking chain reasoning in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation of the present invention will be further described with reference to the accompanying drawings:

[0033] In view of the deficiencies of the prior art, the present invention reduces the length of the text input into the model each time by pre-filling the long text segment by segment, thereby alleviating the problem of excessive computing resource requirements of the Transformer architecture in the case of long text, freeing the limitation of the input window of the large language model, and extracting high-quality summary knowledge from the context in the process of pre-filling the segments, thereby enhancing the large language model's ability to understand complex contexts. In the reasoning stage, by constructing reasoning prompts of thought chains, the extracted high-quality summary knowledge is used to gradually generate a logically rigorous and transparent reasoning process, thereby enhancing the reasoning ability and interpretability of the large language model on complex reasoning problems. In order to comprehensively cover all aspects of the problem, the present invention encourages the generation of multiple thought chains, and different thought chains can be reasoned through different paths, thereby providing diverse perspectives and answers. Subsequently, all generated reasoning processes are reflected and verified through self-consistency prompts, so that the model can identify and correct incorrect reasoning paths, thereby improving the accuracy of the final output.

[0034] like Figure 1As shown in FIG. 1 , a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts includes the following specific steps:

[0035] S1), splitting long contextual cues into several relatively short text paragraphs;

[0036] S2), for the several text paragraphs obtained in step S1), select a text paragraph c according to the order, input c into the large language model, and obtain a pre-filled key value cache;

[0037] S3), for the single text paragraph c obtained in step S2), construct an extraction prompt instruction to extract candidate knowledge from c; then construct a classification prompt instruction to screen the candidate knowledge to obtain summary knowledge related to q;

[0038] S4), continuously repeating the process of steps S2) and S3) until all the contexts are pre-filled, and obtaining a large language model with a complete context key value cache and a set of summary knowledge related to q;

[0039] S5) For the knowledge set obtained in step S4), construct multiple thought chain reasoning prompt instructions, generate multiple reasoning chains through the large language model, and then construct self-consistency prompt instructions to generate the final result.

[0040] like Figure 2 As shown, the text paragraph c selected by S2) is taken as an example, "Regie Yates, a versatile British actor and TV host, is widely loved by the audience for his role in Doctor Who. This role not only demonstrates his excellent acting skills, but also makes him gain a large number of loyal fans around the world (......)". The text is input into the large language model to obtain the pre-filled key-value cache;

[0041] At the same time, construct the extraction prompt instruction, the example is as follows:

[0042] "(...) Reggie Yates is a versatile British actor and television presenter, who is widely loved by audiences for his outstanding acting skills in "5 pounds", "Doctor Who" and "Wizards vs. Aliens", which has also earned him a large number of loyal fans around the world (...) \n Given the above context, extract all knowledge related to the question: "Which host of "The Voice UK" starred in "Doctor Who"? "\n If the context is not relevant to the question, answer 'I don't know'".

[0043] Through the above extraction prompt instructions, candidate knowledge is extracted from the text paragraph, such as: "Reggie Yates is a versatile British actor and TV host, who is widely loved by the audience for his outstanding acting skills in "5 Pounds", "Doctor Who" and "Wizards vs. Aliens". Then, a classification prompt instruction is constructed. The example is as follows:

[0044] "I asked the Big Language Model if a passage of text was relevant to the question: "Which presenter from The Voice UK starred in Doctor Who?" This is its answer: "Reggie Yates is a versatile British actor and television presenter, known for his roles in £5, Doctor Who, and Wizards vs. Aliens." Should I save it for later use? The rules are as follows: - If the answer contains knowledge about the query, answer "yes" and output the original text. - If the answer says that the section is irrelevant to the query, answer "no"."

[0045] The summary knowledge related to the question is screened from the candidate knowledge through the above classification prompt instructions, such as "Reggie Yates is a versatile British actor and TV host, who is widely loved by the audience for his outstanding acting skills in "5 Pounds", "Doctor Who" and "Wizards vs. Aliens"."

[0046] like Figure 3 As shown, through step S4), a large language model with a complete context key-value cache and all summary knowledge related to the question "Which host of The Voice of Britain starred in Doctor Who?" is obtained, such as: "Summary Knowledge 1: Reggie Yates is a versatile British actor and TV host, who is widely loved by the audience for his outstanding acting skills in "5 Pounds", "Doctor Who" and "Wizards vs. Aliens". Summary Knowledge 2: Past hosts of The Voice of Britain include Emma Willis, Reggie Yates, Tom Jones, Olly Murs, Jennifer Hudson, Danny O'Donnell, etc.";

[0047] Construct multiple different thought chain reasoning prompt instructions, as shown below:

[0048] "Summary Knowledge 1: "Doomsday" is an American fantasy action horror thriller film directed by Peter Hyams and starring Arnold Schwarzenegger, Gabriel Byrne, Robin Tunney, Kevin Pollak, and Rod Steiger.\nSummary Knowledge 2: "Oh My God" is a song released by the Rose band in 1999. It was used as a promotional track for "Doomsday" and was sent to major radio stations.\nQuestion: In which year did the Rose band promote the movie starring Arnold Schwarzenegger? Let's think about it step by step.\nAnswer: According to Summary Knowledge 1, we know that Arnold Schwarzenegger starred in "Doomsday". According to Summary Knowledge 2, we know that the Rose band released a song "Oh My God" in 1999. God", this song was used as the promotional song for the movie "Doomsday" starring Arnold Schwarzenegger. So the answer is 1999.\n\nSummary knowledge 1: Reggie Yates is a versatile British actor and TV host, who is widely loved by the audience for his outstanding acting skills in "5 pounds", "Doctor Who" and "Wizards vs. Aliens".\nSummary knowledge 2: Past hosts of "The Voice of Britain" include Emma Willis, Reggie Yates, Tom Jones, Olly Murs, Jennifer Hudson, Danny O'Donnell, etc.\nQuestion: Which host of "The Voice of Britain" starred in "Doctor Who"? Let's think about it step by step.\nAnswer: "

[0049] Based on the above thought chain prompt instructions, multiple reasoning chains are generated through the large language model, one of which is as follows: "According to summary knowledge 1, we know that Reggie Yates showed excellent acting skills in "Doctor Who". According to summary knowledge 2, the previous hosts of "The Voice of Britain" include Reggie Yates. Therefore, the answer is: Reggie Yates." Then, by constructing a self-consistent prompt instruction, an example is as follows:

[0050] "Inference chain 1: According to summary knowledge 1, we know that Reggie Yates showed excellent acting skills in "Doctor Who". According to summary knowledge 2, the previous hosts of "The Voice of Britain" include Reggie Yates. Therefore, the answer is: Reggie Yates.\nInference chain 2: According to summary knowledge 2, we know that the previous hosts of "The Voice of Britain" include Emma Willis. According to summary knowledge 1, we know that she showed excellent acting skills in "Doctor Who". Therefore, the answer is: Emma Willis.\nInference chain 3: According to summary knowledge 2, we know that Emma Willis, Reggie Yates ... Reggie Yates was the host of The Voice of Britain. According to summary knowledge 1, we know that Reggie Yates starred in Doctor Who. So the answer is: Reggie Yates. \n\nPlease carefully reflect on all the reasoning chains you have generated before, verify whether each reasoning step is reasonable step by step, and check whether the conclusions of these reasoning chains are consistent. By comparing the conclusions of each reasoning path, identify those inconsistent or erroneous paths, and correct the erroneous reasoning steps in these paths. Don't output other content, just return a verified, self-consistent and consistent high-quality reasoning path. "

[0051] Based on the above self-consistency prompt instructions, the model is required to reflect on all reasoning chains, gradually verify whether each reasoning step is reasonable, and check whether the conclusions of these reasoning chains are consistent. By comparing the conclusions of each reasoning path, identify those inconsistent or erroneous paths, and correct the erroneous reasoning steps in these paths. Don't output other content, just return a verified, self-consistent and consistent high-quality reasoning path, such as "According to summary knowledge 2, we know that Reggie Yates was the host of "The Voice of Britain". According to summary knowledge 1, we know that Reggie Yates, the host of "The Voice of Britain", appeared in "Doctor Who". Therefore, the answer is: Reggie Yates."

[0052] The embodiments of the present invention disclosed above are only used to help illustrate the present invention. The embodiments do not describe all the details in detail, nor do they limit the invention to specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompts, characterized in that: The following steps are involved: S1), splitting long contextual cues into several relatively short text paragraphs; S2), for the several text paragraphs obtained in step S1), select a text paragraph c according to the order, input c into the large language model, and obtain a pre-filled key value cache; S3), for the single text paragraph c obtained in step S2), construct an extraction prompt instruction to extract candidate knowledge from c; then construct a classification prompt instruction to screen the candidate knowledge to obtain summary knowledge related to q; S4), continuously repeating the process of steps S2) and S3) until all the contexts are pre-filled, and obtaining a large language model with a complete context key value cache and a set of summary knowledge related to q; S5) For the knowledge set obtained in step S4), construct multiple thought chain reasoning prompt instructions, generate multiple reasoning chains through the large language model, and then construct self-consistency prompt instructions and generate the final result.

2. According to claim 1, a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompting is characterized by: In step S3), for the single text paragraph c obtained in step S2), an extraction prompt instruction is constructed to extract candidate knowledge from c; then a classification prompt instruction is constructed to screen the summary knowledge related to q from the candidate knowledge, as follows: The extraction prompt instruction consists of a text paragraph c, a question q, and other prompt texts. The knowledge related to q in c is extracted through a large language model. If the knowledge exists, it is output. If not, a negative answer is generated. Therefore, the candidate knowledge is not necessarily valid knowledge. The classification prompt instruction consists of candidate knowledge, question q and other prompt texts. The large language model is used to verify whether the candidate knowledge is valid knowledge. If the answer is positive, it is retained as summary knowledge for subsequent reasoning. If the answer is negative, the candidate knowledge is discarded.

3. According to claim 1, a long-context knowledge reasoning method based on large language model knowledge extraction and self-consistent thinking chain prompting is characterized by: In step S5), for the knowledge set obtained in step S4), multiple thought chain reasoning prompt instructions are constructed, multiple reasoning chains are generated through the large language model, and then self-consistency prompt instructions are constructed to generate the final result, which is as follows: The thinking chain reasoning prompt consists of a summary knowledge set, sample questions and answers, real questions q, and the prompt text "Let's think step by step". According to the different sample questions and answers, multiple thinking chain reasoning prompt instructions are constructed; For the large language model with complete context key-value cache obtained in step S4), the first thought chain reasoning prompt instruction is pre-filled into the large model. At this time, the key-value cache includes [complete context, extracted summary knowledge set, example question and answer, real question and "Let's think step by step" prompt text], and the large language model will generate the reasoning chain corresponding to the instruction; Remove the key-value cache portion corresponding to the first thinking chain reasoning prompt instruction from the large language model, that is, restore the key-value cache of the large language model to the state before the cached thinking chain prompt instruction, and then pre-fill the next thinking chain reasoning prompt instruction, and repeat this process until the reasoning chain corresponding to each thinking chain reasoning prompt instruction is obtained; The self-consistency prompt instruction consists of all reasoning chains and other prompt texts. By comparing the reasoning process and conclusions of each reasoning chain, it identifies inconsistent or erroneous paths, corrects the erroneous reasoning steps in these paths, and finally returns a verified, self-consistent and consistent reasoning path.

Images