Knowledge retrieval enhancement method and system based on multi-granularity information and hierarchy

Abstract

The invention belongs to the technical field of information, and relates to a knowledge retrieval enhancement method and system based on multi-granularity information and hierarchy. For the problem of single recall granularity in the prior art, a hierarchical storage and recall strategy of long and short context nodes is adopted; for the problem that problems in the prior art are not deeply understood, automatic problem classification based on a large model is introduced, and the number of recall nodes and a threshold value can be dynamically adjusted according to the type and complexity of questions; for the problem of limited recall accuracy in the prior art, a two-stage retrieval strategy is adopted, and high accuracy and practicability of a final result are guaranteed through multi-round retrieval and optimization strategies. According to the method, the defects in recall precision, problem classification, dialogue context association and the like in the prior art are overcome, and a more efficient and intelligent retrieval enhancement solution is provided.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a method and system for enhancing knowledge retrieval technology based on multi-granularity information and hierarchy. Background Art

[0002] Document retrieval and question-answering systems in the prior art mostly rely on search engines combined with natural language processing technology. Common document retrieval frameworks such as Elasticsearch (ES) can store structured or unstructured documents in an index library and retrieve information through keywords and other methods.

[0003] The prior art implementation scheme closest to the present invention is as follows:

[0004] 1. Elasticsearch search engine: ES is one of the mainstream frameworks for document retrieval. It supports full-text search and multiple query strategies through inverted indexing, efficient distributed storage and retrieval methods. However, ES itself mainly relies on keyword matching and lacks in-depth understanding of natural language questions, making it difficult to automatically adjust retrieval strategies to meet complex question-and-answer requirements.

[0005] 2. OpenAI ChatGPT text generation model: ChatGPT and its subsequent models provide powerful capabilities for natural language understanding and generation, and can generate relevant answers based on input questions. However, they are not real-time updated models. This means that these models cannot provide instant information on the latest events, nor can they access the internal data of specific organizations or individuals. Therefore, they have limitations in dealing with problems that require the latest data or knowledge in a specific field, and cannot achieve enhanced effects in knowledge retrieval.

[0006] At present, no retrieval-enhanced question-answering system that combines context node division, question classification, and multi-round recall strategy has been found in the prior art public literature. The prior art has the following deficiencies in complex question-answering scenarios, especially in the processing of large-scale document libraries, unstructured data, and table-based documents:

[0007] 1. Single recall granularity: Most existing systems rely on simple keyword matching and lack in-depth understanding of the context. As a result, when processing long documents or complex questions, relevant information cannot be accurately located, and the recall results deviate from user needs.

[0008] 2. Insufficient understanding of questions: The retrieval system fails to deeply understand the intention of the question, does not adjust the recall strategy according to the question type or complexity, and the answers are not personalized and accurate enough. For example, when dealing with questions about numbers, dates, or tables, it is unable to automatically adjust the search scope and granularity.

[0009] 3. Limited recall accuracy: Traditional retrieval systems usually only use a single-step recall strategy. Rough recall often leads to a large amount of irrelevant information, and lacks subsequent fine-tuning methods, resulting in low accuracy of the final result. Summary of the invention

[0010] The present invention aims to solve the above problems and provides a knowledge retrieval enhancement technology method and system based on multi-granularity information and hierarchicalization. The present invention solves the deficiencies of the prior art in terms of recall accuracy, question classification, and dialogue context association, and provides a more efficient and intelligent retrieval enhancement solution.

[0011] The technical solution adopted by the present invention is as follows:

[0012] A knowledge retrieval enhancement method based on multi-granularity information and hierarchy includes the following steps:

[0013] Split the document into two types of context nodes: long nodes and short nodes, and store them in a vector database or Elasticsearch;

[0014] According to the retrieval scenario, short nodes or long nodes are dynamically selected for recall to obtain retrieval results.

[0015] Furthermore, the dynamically selecting to use short nodes or long nodes for recall according to the retrieval scenario includes:

[0016] In the scenario of full database retrieval of multiple documents, short nodes are used for recall first to improve retrieval efficiency and recall accuracy;

[0017] In a single document conversation scenario, long nodes are used to ensure the completeness and coherence of the context and avoid information loss.

[0018] Furthermore, the method further comprises the following steps:

[0019] Use the big model to classify the questions input by users and determine their question types;

[0020] The number and threshold of recall nodes are dynamically adjusted according to the type of question.

[0021] Furthermore, the dynamic adjustment of the number and threshold of recalled nodes includes: for complex questions, expanding the recall scope and recalling more context nodes; for simple questions, optimizing the retrieval threshold and reducing the interference of irrelevant nodes.

[0022] Furthermore, during the conversation, if it is detected that the next round of questions is ambiguous or unclear, the questions will be automatically rewritten based on the content of the previous round of questions and answers. The rewriting process uses large model generation technology to reorganize the questions to make their semantics clearer, and the rewritten questions will be reused for retrieval to obtain more accurate answers and ensure that in multi-round conversation scenarios, user questions can always receive relevant and coherent answers.

[0023] Furthermore, a two-stage retrieval strategy is adopted in the retrieval process, where the first stage is coarse recall and the second stage is fine re-ranking;

[0024] The first phase includes: using Elasticsearch to perform extensive coarse recall to quickly locate potentially relevant nodes;

[0025] The second stage includes: using a deep learning-based Rerank model to re-rank the coarsely recalled nodes by relevance.

[0026] Furthermore, after retrieving the relevant context nodes, the recalled information is combined with the user question, and reasonable prompt words are designed through Prompt engineering technology, which are input into the large language model and generate answers.

[0027] A knowledge retrieval enhancement system based on multi-granularity information and hierarchy, comprising:

[0028] Document preprocessing and storage module, which is used to split documents into two types of context nodes: long nodes and short nodes, and store them in a vector database or Elasticsearch;

[0029] The multi-granularity selection recall module is used to dynamically select short nodes or long nodes for recall according to the retrieval scenario;

[0030] The question classification and recall optimization module is used to classify the questions input by users using the big model, determine the question type, and dynamically adjust the number and threshold of recall nodes according to the question type;

[0031] The question rewriting and multi-round dialogue module is used to automatically rewrite the question based on the previous round of questions and answers if it is detected that the next round of questions is ambiguous or unclear during the dialogue process, and reuse the rewritten questions for retrieval to obtain more accurate answers, ensuring that user questions can always receive relevant and coherent answers in multi-round dialogue scenarios;

[0032] Two-stage retrieval strategy module, used to perform a two-stage retrieval strategy, in which the first stage uses Elasticsearch to perform extensive rough recall, and the second stage uses the deep learning-based Rerank model to re-rank the nodes in the rough recall;

[0033] The generation enhancement module based on the big model and Prompt engineering is used to combine the recalled information with the user's question after retrieving the relevant context nodes, design reasonable prompt words through Prompt engineering technology, input them into the big language model and generate answers.

[0034] The beneficial effects of the present invention are as follows:

[0035] 1. Regarding the problem of single recall granularity in the prior art: The solution of the present invention adopts a hierarchical storage and recall strategy of long and short context nodes. Short nodes are suitable for full-library retrieval of multiple documents, improving the accuracy and speed of recall, while long nodes ensure that more comprehensive context is provided in a single document dialogue to avoid missing key information related to the sequence. This dynamic adjustment based on the context structure enables the system to flexibly match information according to the question scenario, greatly improving the recall accuracy.

[0036] 2. Regarding the problem of insufficient understanding of questions in the prior art: The solution of the present invention introduces automatic question classification based on a large model. The system can dynamically adjust the number of recall nodes and thresholds according to the type and complexity of the question. For example, for questions such as "the average temperature of each month in 2024", the system can intelligently determine the need for a wider range of node retrieval to ensure that the returned answers are accurate. In this way, the depth of question understanding and the scope of retrieval are flexibly optimized according to the specific problem, achieving a more accurate response.

[0037] 3. Regarding the limited recall accuracy of the prior art: The solution of the present invention designs a two-stage retrieval strategy: ElasticSearch rough recall + Rerank model relevance reordering. In the first stage, Elasticsearch is used to perform extensive rough recall to quickly locate potential related nodes. In the second stage, the relevance of the rough recalled nodes is reordered through the Rerank model to further optimize the recall results and ensure that the most relevant answers are ranked first. This multi-round retrieval and optimization strategy ensures the high accuracy and practicality of the final result, and effectively solves the problem of a single recall strategy in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 1 is a flow chart of the steps of the method of the present invention in an embodiment.

[0039] Figure 2 It is a schematic diagram of document preprocessing and storage.

[0040] Figure 3 It is a schematic diagram of the multi-granularity selection recall strategy.

[0041] Figure 4 It is a schematic diagram of question classification and recall optimization.

[0042] Figure 5 This is a schematic diagram of question rewriting and multi-round dialogue.

[0043] Figure 6 It is a schematic diagram of the two-stage retrieval strategy.

[0044] Figure 7 It is a schematic diagram based on the generation of large models and prompt projects. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0046] The present invention relates to a knowledge retrieval enhancement technology (RAG) based on the combination of retrieval and generation. The technology obtains relevant knowledge from the document library through an effective retrieval strategy, and combines the generation capability of a large language model (LLM) to improve the accuracy of the system's answers to complex questions and its ability to understand context. Specifically, the present invention uses the following technical means to solve the problems in the prior art:

[0047] 1. Hierarchical storage and dynamic recall strategy of document context nodes

[0048] Technical features:

[0049] After the document is read, the document is split into two types of context nodes, long and short, and stored in a vector database or Elasticsearch. The lengths of short nodes and long nodes can be adjusted according to actual needs. For example, the length of a short node can be the length of 1 to 2 paragraphs in an article (i.e., the number of characters or words in 1 to 2 paragraphs), and the length of a long node can be the length of an article (i.e., the number of characters or words in an article). More specifically, in one embodiment, after the text is split, short nodes with a length of less than 2048 Chinese characters or English words represent smaller semantic units, which are suitable for efficient retrieval. Long nodes with a length greater than 2048 and less than 32768 Chinese characters or English words contain more extensive context information, which is suitable for contextual references of a single document.

[0050] Dynamic selection of retrieval strategy: In the scenario of full-database retrieval of multiple documents, the system gives priority to short node recall to improve retrieval efficiency and recall accuracy; while in the case of single document dialogue, long nodes ensure the integrity and coherence of the context to avoid information loss. Through this dynamic recall strategy, the system can flexibly switch the recall method according to different scenarios.

[0051] Innovation:

[0052] Traditional technologies cannot effectively distinguish the granularity of context nodes when processing long documents and multi-document retrieval. The present invention achieves refined management and efficient recall of context through hierarchical storage and dynamic switching of recall strategies.

[0053] The technical advantage of this strategy is that it improves the recall accuracy and response speed of complex questions in large-scale document repositories.

[0054] 2. Automatic question classification and recall optimization based on large models

[0055] Technical features:

[0056] The system has a built-in question classification module based on the Large Language Model (LLM). When a user enters a question, the system first uses the large model to classify the question and determine which type of question it belongs to, such as factual, analytical, tabular, etc. Among them, factual refers to the type of question that can be answered after obtaining relevant information, analytical refers to the type of question that can only be answered after analyzing data from multiple sources and multiple steps, and tabular refers to the type of question that can only be answered after combining tabular data for calculation.

[0057] Depending on the type of question, the system can dynamically adjust the number and threshold of recalled nodes. That is, for more complex questions (such as data extraction questions), the system will expand the recall range and recall more context nodes; for simple questions, the system will optimize the retrieval threshold to reduce the interference of irrelevant nodes.

[0058] Innovation:

[0059] Through question classification and retrieval strategy adjustment, the system can achieve in-depth understanding of questions and flexible response. Especially for data-intensive and multi-information questions, the system can adjust the precision and coverage of recall, significantly improving the accuracy of answers.

[0060] This technology breaks the limitations of existing systems at the question understanding and retrieval levels, and realizes personalized question processing and accurate answer generation.

[0061] 3. Automatic rephrasing of unclear questions and multi-round dialogue support

[0062] Technical features:

[0063] During the conversation, if the system detects that the next round of questions is ambiguous or unclear (for example, lack of context), it will automatically rewrite the questions based on the previous round of questions and answers. The rewriting process uses large model generation technology to reorganize the questions to make their semantics clearer, and then re-use the rewritten questions for retrieval to obtain more accurate answers.

[0064] The rewritten questions are re-retrieved through the retrieval module in the system to ensure that user questions can always receive relevant and coherent answers in multi-round dialogue scenarios.

[0065] Innovation:

[0066] Traditional retrieval systems are usually unable to process contextual information in multi-round dialogues, resulting in incoherent answers or missing important information. This invention introduces a question rewriting mechanism to effectively make up for the comprehension barriers caused by ambiguous questions, enabling the system to demonstrate higher intelligence and coherence in complex dialogue scenarios.

[0067] 4. Two-stage retrieval strategy: Elasticsearch coarse recall + Rerank fine re-ranking

[0068] Technical features:

[0069] Phase 1: Coarse recall. Using vector retrieval technologies such as Elasticsearch, we conduct extensive coarse-grained recall on large-scale document libraries. This phase can quickly find potentially relevant context nodes from massive data within milliseconds to ensure the coverage of the recall.

[0070] Phase 2: Rerank. Based on the recall in the first phase, the recall results are reranked using a deep learning-based Rerank model to filter out nodes with low relevance and ensure that the context nodes most relevant to the user's question are ranked at the top. The reranking model in this phase scores based on context similarity and question understanding to optimize the final retrieval results.

[0071] Innovation:

[0072] Traditional retrieval systems usually rely only on a single round of recall and cannot maintain a balance between efficiency and accuracy in large-scale data. This invention uses a two-stage retrieval strategy to ensure the extensiveness of the recall and achieve accurate sorting of the results through the Rerank technology, thus improving the quality of answers in complex problem scenarios.

[0073] 5. Enhanced generation based on large models and prompt projects

[0074] Technical features:

[0075] After retrieving the relevant context nodes, the system combines the recalled information with the user's question, designs reasonable prompt words through Prompt engineering technology, and inputs them into the large language model to generate answers.

[0076] The large model generation process at this stage can refer to the contextual information of the recall and combine it with the question to generate accurate and contextual answers. Prompt engineering ensures flexibility and controllability in the generation process and enhances the output quality of the system.

[0077] Innovation:

[0078] This invention combines retrieval and generation, fully embodies the core concept of RAG "retrieval + generation" in specific scenarios, and solves the limitation of traditional systems that rely on a single retrieval result. The generation capability of the large model makes the answers more reasonable and natural, and can cover a wider range of application scenarios.

[0079] The overall process of the technical solution of the present invention is as follows Figure 1 As shown, it can be summarized into the following key steps:

[0080] 1. Document preprocessing and storage:

[0081] Split the document into two types of context nodes: long and short. The short nodes and long nodes are stored in a vector database or Elasticsearch, respectively. Figure 2 shown.

[0082] 2. Multi-granularity selection recall strategy:

[0083] According to the search scenario (full database search or single document dialogue), dynamically choose to use short nodes or long nodes for recall, such as Figure 3 As shown in Figure 2, short nodes are used to improve retrieval efficiency and recall accuracy, while long nodes are used to ensure the completeness and coherence of the context.

[0084] 3. Problem classification and recall optimization:

[0085] The large language model (LLM) is used to classify user questions and the number and threshold of recall nodes are dynamically adjusted according to the question type, such as Figure 4 shown.

[0086] 4. Question rephrasing and multi-round dialogue:

[0087] For ambiguous or unclear questions, we automatically rewrite them based on the previous round of questions and answers, use big model generation technology to reorganize the questions to make their semantics clearer, and then re-search them, such as Figure 5 shown.

[0088] 5. Two-stage retrieval strategy:

[0089] The two-stage search strategy adopted in the present invention is as follows Figure 6 As shown, including:

[0090] Phase 1: Use Elasticsearch for coarse recall to quickly find potentially relevant context nodes.

[0091] The second stage: Use the Rerank model to re-rank the recall results and optimize the relevance and ranking of the retrieval results.

[0092] 6. Enhanced large language model generation:

[0093] Combining the recalled context nodes and user questions, we design prompt words through Prompt Engineering technology, input the prompt words into the large language model, and generate accurate and contextual answers, such as Figure 7 shown.

[0094] 7. Output and feedback:

[0095] The generated answers are presented to the user. Iterative optimization is performed based on user feedback to improve system performance.

[0096] In terms of recall accuracy, compared with the recall rate of 66.7% of single-step recall, the multi-step recall strategy of the present invention increases the recall rate to 98.2%, and the accuracy rate increases from 83.1% of single-step recall to 94.6%. In terms of retrieval integrity, the present invention adopts a multi-granularity recall strategy of long and short nodes, using short node information under full-library retrieval, effectively improving retrieval efficiency and recall accuracy, and using long node information under single-article retrieval to ensure the integrity and coherence of the context of the recalled node.

[0097] Another embodiment of the present invention is a knowledge retrieval enhancement system based on multi-granularity information and hierarchy, which includes:

[0098] Document preprocessing and storage module, which is used to split documents into two types of context nodes: long nodes and short nodes, and store them in a vector database or Elasticsearch;

[0099] The multi-granularity selection recall module is used to dynamically select short nodes or long nodes for recall according to the retrieval scenario;

[0100] The question classification and recall optimization module is used to classify the questions input by users using the big model, determine the question type, and dynamically adjust the number and threshold of recall nodes according to the question type;

[0101] The question rewriting and multi-round dialogue module is used to automatically rewrite the question based on the previous round of questions and answers if it is detected that the next round of questions is ambiguous or unclear during the dialogue process, and reuse the rewritten questions for retrieval to obtain more accurate answers, ensuring that user questions can always receive relevant and coherent answers in multi-round dialogue scenarios;

[0102] Two-stage retrieval strategy module, used to perform a two-stage retrieval strategy, in which the first stage uses Elasticsearch to perform extensive rough recall, and the second stage uses the deep learning-based Rerank model to re-rank the nodes in the rough recall;

[0103] The generation enhancement module based on the big model and Prompt engineering is used to combine the recalled information with the user's question after retrieving the relevant context nodes, design reasonable prompt words through Prompt engineering technology, input them into the big language model and generate answers.

[0104] The division of the above modules is only an example. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described in the above method. The specific working process of each module can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0105] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0106] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0107] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A knowledge retrieval enhancement method based on multi-granularity information and hierarchy, characterized in that: The following steps are involved: Split the document into two types of context nodes: long nodes and short nodes, and store them in a vector database or Elasticsearch; Dynamically select short nodes or long nodes for recall according to the retrieval scenario to obtain retrieval results.

2. The method according to claim 1, characterized in that The method of dynamically selecting a short node or a long node for recall according to the retrieval scenario includes: In the scenario of full-database retrieval of multiple documents, short nodes are used for recall first to improve retrieval efficiency and recall accuracy; in the scenario of single-document dialogue, long nodes are used to ensure the integrity and coherence of the context and avoid information loss.

3. The method according to claim 1, characterized in that The following steps are also included: Use the big model to classify the questions input by users and determine their question types; The number and threshold of recall nodes are dynamically adjusted according to the type of question.

4. The method according to claim 3, characterized in that The dynamically adjusting the number and threshold of the recalled nodes includes: For complex questions, the recall range is expanded to recall more context nodes; for simple questions, the retrieval threshold is optimized to reduce the interference of irrelevant nodes.

5. The method according to claim 3, characterized in that: During the conversation, if it is detected that the next round of questions is ambiguous or unclear, the questions will be automatically rewritten based on the content of the previous round of questions and answers. The rewriting process uses large model generation technology to reorganize the questions to make their semantics clearer, and the rewritten questions will be reused for retrieval to obtain more accurate answers and ensure that in multi-round conversation scenarios, user questions can always receive relevant and coherent answers.

6. The method according to claim 1, characterized in that A two-stage retrieval strategy is adopted in the retrieval process, where the first stage is coarse recall and the second stage is fine re-ranking; The first phase includes: using Elasticsearch to perform extensive coarse recall to quickly locate potentially relevant nodes; The second stage includes: using a deep learning-based Rerank model to re-rank the coarsely recalled nodes by relevance.

7. The method according to claim 1, characterized in that After retrieving the relevant context nodes, the recalled information is combined with the user's question, and reasonable prompt words are designed through Prompt engineering technology, which are input into the large language model and generate answers.

8. A knowledge retrieval enhancement system based on multi-granularity information and hierarchy, characterized in that: include: Document preprocessing and storage module, which is used to split documents into two types of context nodes: long nodes and short nodes, and store them in a vector database or Elasticsearch; The multi-granularity selection recall module is used to dynamically select short nodes or long nodes for recall according to the retrieval scenario; The question classification and recall optimization module is used to classify the questions input by users using the big model, determine the question type, and dynamically adjust the number and threshold of recall nodes according to the question type; The question rewriting and multi-round dialogue module is used to automatically rewrite the question based on the previous round of questions and answers if it is detected that the next round of questions is ambiguous or unclear during the dialogue process, and reuse the rewritten questions for retrieval to obtain more accurate answers, ensuring that user questions can always receive relevant and coherent answers in multi-round dialogue scenarios; Two-stage retrieval strategy module, used to perform a two-stage retrieval strategy, in which the first stage uses Elasticsearch to perform extensive rough recall, and the second stage uses the deep learning-based Rerank model to re-rank the nodes in the rough recall; The generation enhancement module based on the big model and Prompt engineering is used to combine the recalled information with the user's question after retrieving the relevant context nodes, design reasonable prompt words through Prompt engineering technology, input them into the big language model and generate answers.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images