Large language model-oriented database segmentation and subproblem-based enhanced retrieval method

Abstract

The invention provides a big language model-oriented retrieval method based on database segmentation and subproblem enhancement, which comprises the following steps of: 1, decomposing an input problem into a plurality of related subproblems, and determining a solving sequence of the subproblems by constructing an entity tree; step 2, randomly adding sub-questions in each retrieval step for retrieval; 3, associating the paragraph blocks with the corresponding title blocks to form a hierarchical external knowledge base; 4, in combination with keyword retrieval and vector retrieval, a block most relevant to the input problem is found from an external knowledge base to serve as a cue word; and 5, processing the question based on the cue word by using a large language model, and generating a final answer to the original question. According to the big language model-oriented retrieval method based on database segmentation and sub-problem enhancement, the overall information acquisition efficiency of the user is improved, the performance is excellent in the aspect of improving the retrieval efficiency, and a good effect can be achieved in different scenes needing fine analysis and rapid overview.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a retrieval method based on database segmentation and sub-question enhancement for large language models. Background Art

[0002] With the rise of Large Language Models (LLM), it has become common to use their generative and emergent capabilities to solve professional problems. However, due to the requirement for data freshness, LLM may give outdated answers to some time-sensitive questions. Therefore, enhancing the generative results of LLM by retrieving external relevant information has become a popular solution, which is called retrieval-augmented LLM or retrieval-augmented generation RAG (Retrieval Augmented Generation).

[0003] The retrieval-enhanced LLM method proposed by researchers in the field of information retrieval, Cheng Xiang Zhai and John Lafferty, is to provide LLM with an external database. For the user's question (Query), the information related to the user's question is first retrieved from the external database through information retrieval (IR) technology, and then the LLM is allowed to generate results based on these relevant information. Using retrieval technology to find the most relevant information fragments from a large amount of external data to the input question not only provides a reference for LLM to generate replies, but also filters out some irrelevant information to a certain extent. Compared with directly generating answers by LLM, this method improves the accuracy of generating replies. In addition, the larger the context window, the higher the reasoning cost, so introducing relevant information retrieval steps can also reduce unnecessary reasoning costs. However, for some complex questions, this method of getting answers in one step may lead to low generation accuracy.

[0004] For some complex questions, multi-step question answering is proposed. Multi-step question answering is an important and challenging task in Natural Language Processing (NLP). It obtains and integrates information from multiple sources through multi-step retrieval and information reasoning when answering questions. In addition, in the process of dividing the external database into blocks, the traditional approach is to set a maximum length and then divide the external database into multiple blocks according to the maximum length.

[0005] For multi-step retrieval question answering, despite the improvement of RAG, there are still obvious limitations in the process of generating answers. First, if the multi-hop QA system makes wrong assumptions or understandings during the reasoning or information integration process, this incorrect iteration may lead to inaccurate or biased answers. Second, for some semantically complex or ambiguous questions, due to the system's insufficient understanding of the context, it may be difficult to correctly obtain and integrate information, resulting in reduced answer quality.

[0006] In the process of transferring the external database into the database by setting the size and dividing it into blocks, two chapters with fewer characters will be placed in one block. In the subsequent retrieval process, such blocks will be retrieved and provided to the LLM as context to guide its output. This will undoubtedly introduce too much interference information, resulting in a decrease in the quality of the generated answers.

[0007] In summary, compared with direct use of LLM generation, multi-step question answering has achieved iterative innovation in two aspects: 1) retrieval-guided generation; 2) adopting a multi-step strategy. Even so, for some semantically complex and ambiguous questions, this method may still produce an erroneous process due to insufficient understanding, and the iteration of this erroneous process will lead to obvious quality problems in the final answer. In addition, the problem of external knowledge base division will also introduce a lot of interference information. For this reason, we invented a new retrieval method based on database segmentation and sub-question enhancement for large language models. Summary of the invention

[0008] The purpose of the present invention is to provide a retrieval method based on database segmentation and sub-question enhancement for a large language model, which is applied to the retrieval process in enhancing the generation capability of a large language model through retrieval.

[0009] The object of the present invention can be achieved by the following technical measures: a database segmentation and sub-question enhancement retrieval method for a large language model, the database segmentation and sub-question enhancement retrieval method for a large language model comprising:

[0010] Step 1: Decompose the input problem into multiple related sub-problems and determine the order of solving the sub-problems by building an entity tree;

[0011] Step 2: randomly add sub-questions to search in each search step;

[0012] Step 3, forming a hierarchical external knowledge base by associating the paragraph blocks with the corresponding title blocks;

[0013] Step 4: Combine keyword search and vector search to find the most relevant blocks to the input question from the external knowledge base as prompt words;

[0014] Step 5: Use the large language model to process the question based on the prompt words and generate the final answer to the original question.

[0015] The purpose of the present invention can also be achieved by the following technical measures:

[0016] Step 1 includes:

[0017] Step 11, locate all entities in the question and use these entities as root nodes of the entity tree;

[0018] Step 12, find the paragraphs containing these entity tree root nodes, and associate these entities appearing in the same sentence with the corresponding entity tree root nodes as their child nodes;

[0019] Step 13, starting from these child nodes, repeat the above process until no new child nodes can be added to the tree, forming an entity tree;

[0020] Step 14, add the paragraphs corresponding to the nodes in sequence according to the distance in the tree, so as to obtain the filtered paragraph representation C_QETPS;

[0021] In step 15, use the advanced question-answering task solution AlbertForQuestionAnswer ing to answer each sub-question and get the answer set.

[0022] In step 13, in order to prevent the influence of interfering paragraphs, a query-aware adjustment mechanism is added, and only sub-nodes in the sentence corresponding to the query can be added; at the same time, the F1 value calculated by the longest common subsequence length is used as the similarity, and the occurrence is determined by setting a threshold.

[0023] In step 14, after constructing the entity tree, it is considered that the answer to the i-th jump sub-question is most likely to exist in the paragraph associated with the i-th level node of the entity tree; therefore, the paragraphs corresponding to the nodes are added in sequence according to the distance in the tree, thereby obtaining the filtered paragraph representation C_QETPS.

[0024] In step 15, the answer set obtained is: A = {a_i},

[0025] a_i=Reader(sq_i|C_QETPS^i),i=1,2,3,...

[0026] Here i represents the number of sub-questions, a_i is the i-th sub-answer, sq_i is the i-th sub-question, C_QETPS^i is the i-th filtered paragraph representation, and Reader is the AlbertForQuestionAnswering model.

[0027] In step 2, in each retrieval step of multiple hops, based on the consideration of preventing reasoning hallucinations, a sub-question is randomly added to each retrieval step for retrieval.

[0028] In step 3, by first taking the title as the initial segmentation part and then performing paragraph-level segmentation within each title block, both the subject structure of the title and the detailed content of the paragraph can be retained; the segmented paragraph blocks are associated with other relevant information.

[0029] In step 3, when reading the external document and dividing it into blocks, whenever a line that meets the characteristics of a title is read, it is marked as a title; otherwise it is marked as text; when the next title is read, the previous title and the corresponding text are written into the block; for blocks with text length greater than the specified length, they are split into multiple fixed-size blocks, and the title is added to each block to generate multiple blocks that meet the title-text paradigm.

[0030] In step 4, first use keywords to search the title to get the best block, and the title is the first line of each block; because keyword search can accurately match a small number of characters; then, use vector search to perform a second search on the text in the block; finally, the intersection of the two is considered to be the best relevant knowledge that has both matching keywords and semantic matching; then return to the sub-problem and start again, and stop searching when there is a yes answer in the generated answer.

[0031] In step 5, the retrieved knowledge is used as a prompt generated by the large model and input into the large language model to guide the generation of an answer to the original question.

[0032] The database segmentation and sub-question enhanced retrieval method based on the large language model in the present invention aims at the interference information problem generated in the process of transferring the external knowledge base into the database and dividing it into blocks, and proposes a method of title segmentation and combined retrieval. The method uses the title as the segmentation standard, puts the paragraphs under the same title in one segment, and then uses the combination of keyword retrieval and vector retrieval to retrieve the relevant content. Through this method, not only can the retrieval efficiency be significantly improved, but also the ideal effect can be achieved in fine-grained and coarse-grained retrieval at different levels. In terms of fine-grained retrieval, the paragraphs under the same title are classified into one segment, and combined with keyword and vector retrieval, the information in the database can be refined and analyzed, and the system can identify and extract very specific detail information to meet the user's needs for high precision and detailed information. For example, when a user queries a specific question, the system can accurately locate the relevant paragraphs and sentences and provide detailed and accurate answers. In terms of coarse-grained retrieval, the title segmentation method effectively reduces the retrieval scope and avoids irrelevant information from reducing the retrieval efficiency. At the same time, combined with vector retrieval, the system can quickly extract a large amount of information related to the user's query and provide overall trends and summary results. This is especially important for users who need to quickly understand the general situation or overall trend. For example, when browsing the overall information of a certain topic, the system can quickly provide relevant large blocks of content to improve the user's overall information acquisition efficiency. Therefore, this method not only performs well in improving retrieval efficiency, but also achieves good results in different scenarios that require fine analysis and quick overview.

[0033] On the other hand, in order to address the problem that some semantically complex and ambiguous questions result in low-quality final answers, the present invention proposes a sub-question enhanced retrieval method, which decomposes the input question into multiple sub-questions, and then adds sub-questions in each step to guide the retrieval, replacing the single retrieval guided by the last generation in each step. The purpose of this method is mainly to retrieve from multiple aspects, increase diversity, and prevent stereotyped thinking. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a specific embodiment of the database segmentation and sub-question enhanced retrieval method for a large language model of the present invention;

[0035] Figure 2 It is a schematic diagram of the block division result of the simulation experiment of external knowledge base block division in a specific embodiment of the present invention;

[0036] Figure 3 A comparison chart of the recall rate index between the original method and the method of the present invention in a specific embodiment of the present invention;

[0037] Figure 4This is a comparison chart of the accuracy of a question-answering effect experiment of sub-question enhanced retrieval in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations and / or combinations thereof.

[0040] like Figure 1 As shown, Figure 1 The flowchart of the database segmentation and sub-question enhanced retrieval method for a large language model of the present invention. The method mainly includes five steps: problem decomposition, multi-hop retrieval, establishment of an external knowledge base, retrieval of relevant information and generation of a final answer. The specific steps are as follows:

[0041] Step 1: Decompose the input problem into multiple related sub-problems. The specific steps are as follows:

[0042] 1) Locate all entities in the question and use them as the root nodes of the entity tree.

[0043] 2) Then find the paragraphs containing these entity tree root nodes, and associate these entities appearing in the same sentence with the corresponding entity tree root nodes as their child nodes.

[0044] 3) Starting from these child nodes, the above process is repeated until no new child nodes can be added to the tree, and then the entity tree is formed. In order to prevent the influence of interfering paragraphs, a query-aware adjustment mechanism is added, and only child nodes in the sentence corresponding to the query can be added. At the same time, the F1 value calculated using the longest common subsequence length is used as the similarity, and the threshold is set to determine whether it occurs.

[0045] 4) After constructing the entity tree, it is considered that the answer to the i-th hop sub-question is most likely to exist in the paragraph associated with the i-th level node of the entity tree (the root node is the 0th level). Therefore, the paragraphs corresponding to the nodes are added in sequence according to the distance in the tree, thereby obtaining the filtered paragraph representation C_QETPS.

[0046] 5) Use the advanced question-answering task solution (AlbertForQuestionAnswering) to answer each sub-question and get the answer set: A = {a_i},

[0047] a_i=Reader(sq_i|C_QETPS^i),i=1,2,3,...

[0048] Where i represents the number of sub-questions, a_i is the ith sub-answer, sq_i is the ith sub-question, C_QETPS^i is the ith filtered paragraph representation, and Reader is the AlbertForQuestionAnswering model.

[0049] Step 2: In each retrieval step of multiple hops, the traditional approach is to use the result of the previous hop retrieval-inference to guide the retrieval of the next hop. The present invention is based on the consideration of preventing reasoning hallucinations, and randomly adds a sub-question to perform retrieval in each retrieval step.

[0050] Step 3, build an external knowledge base. Associate each paragraph block with the corresponding title block to form a hierarchical structure. First, by using the title as the initial segmentation part, and then performing paragraph-level segmentation within each title block, the subject structure of the title and the detailed content of the paragraph can be retained at the same time. This combination provides the ability of overall navigation and allows users to read and find paragraph-level information in each title block. Associate the segmented paragraph blocks with other related information.

[0051] In the document, each title is on a separate line, with no spaces at the beginning of the line and no end mark, while the corresponding paragraph is on a separate line or has spaces at the beginning of the line, or has an end mark (such as.,!,?, etc.). Using this feature, when reading external documents and dividing them into blocks, whenever a line that meets the characteristics of a title is read, it is marked as a title; otherwise it is marked as text. When the next title is read, the previous title and the corresponding text are written into the block. For blocks with text length greater than the specified length, it is split into multiple fixed-size blocks, and the title is added to each block to generate multiple blocks that meet the title-text paradigm.

[0052] Step 4: In the specific search process, combine the advantages of keyword search and vector search. First, use keywords to search the title to get the best block, and the title is the first line of each block. Because keyword search can accurately match a small number of characters. Then, use vector search to perform a secondary search on the text in the block. Finally, the intersection of the two is considered to be the best relevant knowledge that has both matching keywords and semantic matching. Then return to the sub-question and start again until the "Answer is" mark is generated and the search stops.

[0053] Step 5: Use the knowledge retrieved above as a prompt for the large model to generate a large language model to guide the generation of an answer to the original question.

[0054] In summary, the present invention proposes three key improvements to solve the problems in the retrieval-generation process:

[0055] 1. New method for dividing external knowledge base:

[0056] Aiming at the problem of interference information generated in the process of transferring external knowledge base into database and dividing it into blocks, the present invention proposes a method of title block division and combined retrieval. The method uses the title as the block division standard and puts the paragraphs under the same title into one block, thereby reducing the introduction of interference information.

[0057] 2. Combination of keyword search and vector search:

[0058] A method is proposed to first use keywords to retrieve the title, and then use vectors to retrieve the relevant paragraph content. This combination utilizes the precise matching of keyword retrieval and the semantic understanding ability of vector retrieval to improve the retrieval effect.

[0059] 3. Sub-question guided search:

[0060] In the multi-step retrieval process, a sub-question guided retrieval method is introduced. Compared with the traditional multi-hop method of generating relevant knowledge, the present invention adds an extracted sub-question as a retrieval question on the basis of each multi-hop knowledge generation, thereby improving the diversity and accuracy of the retrieval.

[0061] The following are several specific embodiments of the present invention.

[0062] Example 1

[0063] In a specific embodiment 1 of the present invention, a simulation test is conducted on the sub-question extraction effect.

[0064] Refer to Table 1 for a simulation experiment of extracting sub-problems from a complex problem. This table shows some examples of sub-problem sets extracted from the original problem.

[0065] Table 1 Example table of sub-questions extracted from the original question

[0066]

[0067]

[0068] Example 2

[0069] In a specific embodiment 2 of the present invention, a simulation experiment is conducted on the external knowledge base blocks in the present invention.

[0070] Reference Figure 2 ,Simulation experiment of external knowledge base partitioning. Figure 2 (a) is the original block result, and 2(b) is the block result of the present invention. By comparing 2(a) and 2(b), it can be seen that the block method of the present invention divides the title and content into the same block, reducing the interference of other irrelevant information.

[0071] Example 3

[0072] In a specific embodiment 3 of the present invention, a simulation experiment is conducted to enhance the retrieval effect of the present invention.

[0073] Reference Figure 3 ,Sub-question enhanced retrieval experiment. Figure 3 It is a comparison chart of the original method and the method of the present invention in terms of recall rate index; Figure 3 (b) is a comparison chart of the accuracy index between the original method and the method of the present invention.

[0074] As can be seen from the comparison chart in 3(a), the retrieval method of the present invention needs to retrieve fewer paragraphs and titles to obtain the final answer, and the recall rate of the retrieval is higher than that of the original method. As can be seen from the comparison chart in 3(b), the accuracy of the retrieval method of the present invention is 7% higher than that of the original method.

[0075] Example 4

[0076] In a specific embodiment 4 of the present invention, a simulation is performed on the question-answering effect of the sub-question enhanced retrieval of the present invention.

[0077] Reference Figure 4 ,Experiments on question answering effect of sub-question enhanced retrieval. Figure 4 is the accuracy comparison between the original method and the enhanced method. Figure 4 It can be seen that the method of the present invention is superior to the previous method, and the accuracy is improved by 6%.

[0078] In summary, it can be seen that the method of the present invention is better than the original method in retrieval effect, and both the recall rate and the accuracy rate are improved.

[0079] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0080] Except for the technical features described in the specification, all other technical features are known technologies to those skilled in the art.

Claims

1. A database segmentation and sub-question enhanced retrieval method for large language models, characterized by: The database segmentation and sub-question enhanced retrieval method for large language models includes: Step 1: Decompose the input problem into multiple related sub-problems and determine the order of solving the sub-problems by building an entity tree; Step 2: randomly add sub-questions to search in each search step; Step 3, forming a hierarchical external knowledge base by associating the paragraph blocks with the corresponding title blocks; Step 4: Combine keyword search and vector search to find the most relevant blocks to the input question from the external knowledge base as prompt words; Step 5: Use the large language model to process the question based on the prompt words and generate the final answer to the original question.

2. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 1, characterized in that: Step 1 includes: Step 11, locate all entities in the question and use these entities as root nodes of the entity tree; Step 12, find the paragraphs containing these entity tree root nodes, and associate these entities appearing in the same sentence with the corresponding entity tree root nodes as their child nodes; Step 13, starting from these child nodes, repeat the above process until no new child nodes can be added to the tree, forming an entity tree; Step 14, add the paragraphs corresponding to the nodes in sequence according to the distance in the tree, so as to obtain the filtered paragraph representation C_QETPS; In step 15, use the advanced question-answering task solution AlbertForQuestionAnswering to answer each sub-question and obtain the answer set.

3. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 2, characterized in that: In step 13, in order to prevent the influence of interfering paragraphs, a query-aware adjustment mechanism is added, and only sub-nodes in the sentence corresponding to the query can be added; at the same time, the F1 value calculated by the longest common subsequence length is used as the similarity, and the occurrence is determined by setting a threshold.

4. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 2, characterized in that: In step 14, after constructing the entity tree, it is considered that the answer to the i-th jump sub-question is most likely to exist in the paragraph associated with the i-th level node of the entity tree; therefore, the paragraphs corresponding to the nodes are added in sequence according to the distance in the tree, thereby obtaining the filtered paragraph representation C_QETPS.

5. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 2, characterized in that: In step 15, the answer set obtained is: A = {a_i}, a_i=Reader(sq_i|C_QETPS^i),i=1,2,3,... Here i represents the number of sub-questions, a_i is the i-th sub-answer, sq_i is the i-th sub-question, C_QETPS^i is the i-th filtered paragraph representation, and Reader is the AlbertForQuestionAnswering model.

6. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 1, characterized in that: In step 2, when performing each retrieval step of multiple hops, based on the consideration of preventing reasoning hallucinations, a sub-question is randomly added to each retrieval step for retrieval.

7. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 1, characterized in that: In step 3, the title is first used as the initial segmentation part, and then paragraph-level segmentation is performed within each title block, while retaining the subject structure of the title and the detailed content of the paragraph; the segmented paragraph blocks are associated with other related information.

8. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 7, characterized in that: In step 3, when reading the external document and dividing it into blocks, whenever a line that meets the characteristics of a title is read, it is marked as a title; Otherwise, it is marked as text. When the next title is read, the previous title and the corresponding text are written into the block. For blocks whose text length is greater than the specified length, it is split into multiple blocks of fixed size, and the title is added to each block to generate multiple blocks that conform to the title-text paradigm.

9. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 1, characterized in that: In step 4, first use keywords to search the title to get the best block. The title is the first line of each block, because keyword search can accurately match a small number of characters. Then, use vector search to perform a second search on the text in the block. Finally, the intersection of the two is considered to be the best relevant knowledge that has both matching keywords and semantic matching. Then return to the sub-problem and start again until the answer is marked as yes in the generation and the search is stopped.

10. The method for retrieval based on database segmentation and sub-question enhancement for a large language model according to claim 1, characterized in that: In step 5, the retrieved knowledge is used as a prompt generated by the large model and input into the large language model to guide the generation of an answer to the original question.

Images