A hybrid search enhancement method that combines unstructured text with structured graphs

Abstract

The application discloses a hybrid retrieval enhancement method combining unstructured text and structured graph, and belongs to the technical field of computer data processing and artificial intelligence interaction. The method comprises the following steps: constructing a heterogeneous index of original unstructured documents, generating a text vector index and a knowledge graph index, and establishing a mapping relationship between the two; receiving a user query request, and performing parallel semantic retrieval based on the vector and structure retrieval based on the knowledge graph; obtaining a text block set and a topological subgraph respectively; based on the mapping relationship, dynamically adjusting the text block set and the topological subgraph; performing heterogeneous data fusion and reordering on the adjusted text retrieval result and the graph retrieval result; generating a unified enhanced context information package; inputting the enhanced context information package into a large language model to generate a final answer. The application deeply fuses text and graph retrieval, significantly improves the accuracy, logic and authority of the answer, effectively solves complex problem reasoning, and greatly reduces the large model illusion rate.

Description

Technical Field

[0001] This invention relates to the field of computer data processing and artificial intelligence interaction technology, specifically to a hybrid retrieval enhancement method that combines unstructured text and structured graphs. Background Technology

[0002] In the fields of information retrieval and intelligent question answering, Retrieval-Augmented Generation (RAG) technology has become a key paradigm for connecting large language models with external knowledge bases, aiming to improve the accuracy and timeliness of generated content and reduce illusions. Existing RAG technologies mainly develop along two relatively independent technical paths, each with significant limitations: 1. Semantic fragmentation and lack of context (Text RAG defect): Traditional text chunking often breaks down complete logical relationships. When queries involve complex relationships across paragraphs or documents, vector retrieval can only find literal similar fragments, making it difficult to capture the deep logical connections between entities, resulting in "point-like distribution" of search results and a lack of systematicity.

[0003] 2. Narrow knowledge coverage and high construction cost (Graph RAG drawbacks): Knowledge graphs rely on detailed ontology definitions and entity extraction. For highly unstructured and frequently updated descriptive text, the graph struggles to enumerate all implicit relationships. Once a graph node is missing, the retrieval path is interrupted, leading to insufficient recall.

[0004] 3. Simple fusion strategies lack complementarity: Most existing fusion attempts only involve simple merging of result sets (such as RRF reordering), without achieving deep synergy between "semantics" and "structure" at the underlying retrieval stage. This results in the model generating answers that are often insufficient in depth or have broken logical chains when faced with questions that require both accurate factual derivation and broad contextual awareness.

[0005] To address the aforementioned issues, there is an urgent need for a hybrid retrieval enhancement method that combines unstructured text and structured graphs to resolve the problems associated with traditional methods. Summary of the Invention

[0006] The purpose of this invention is to provide a hybrid retrieval enhancement method that combines unstructured text and structured graphs, deeply integrating text and graph retrieval to significantly improve the accuracy, logic, and authority of answers, effectively solve complex problem reasoning, and greatly reduce the illusion rate of large models.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A hybrid retrieval enhancement method combining unstructured text and structured graphs includes: Step 1: Construct a heterogeneous index for the original unstructured document, simultaneously generate a text vector index and a knowledge graph index, and establish a mapping relationship between the two; Step 2: Receive user query requests and perform vector-based semantic retrieval and knowledge graph-based structural retrieval in parallel to obtain text block sets and topological subgraphs, respectively. Step 3: Based on the mapping relationship, dynamically adjust the text block set and topological subgraph; Step 4: Perform heterogeneous data fusion and reordering on the adjusted text search results and graph search results to generate a unified enhanced context information package; Step 5: Input the enhanced contextual information package into the large language model to generate the final answer.

[0008] Furthermore, in step 1, a heterogeneous index is constructed on the original unstructured document, simultaneously generating a text vector index and a knowledge graph index, and establishing a mapping relationship between the two, specifically as follows: The original unstructured document is segmented into fixed-length or semantically complete text blocks based on the sliding window algorithm. The text blocks are then converted into high-dimensional vectors, stored in a vector database, and each text block is assigned a unique text block ID. Extract entities and relationships from raw unstructured documents and store them in a graph database; Create a mapping table to record the text block ID of each entity.

[0009] Further, in step 2, the user query request is received, and vector-based semantic retrieval and knowledge graph-based structural retrieval are performed in parallel to obtain the text block set and the topological subgraph, respectively. Specifically: Receive user query requests and extract multi-dimensional features from the user query requests; Based on the extracted features, vector-based semantic retrieval and knowledge graph-based structural retrieval are performed to obtain a set of text blocks and a topological subgraph.

[0010] Furthermore, the features obtained from multi-dimensional feature extraction include keyword vectors and core entities.

[0011] Furthermore, based on the extracted features, vector-based semantic retrieval and knowledge graph-based structural retrieval are performed to obtain a set of text blocks and a topological subgraph, specifically: Based on keyword vectors, a Top-K search is performed in the vector database to obtain a set of semantically similar text blocks, where each text block has a semantic similarity score; Using the core entity as the seed node, and starting from the seed node, perform an N-hop query in the graph database to extract the topological subgraph related to the core entity.

[0012] Furthermore, in step 3, the text block set and topological subgraph are dynamically adjusted based on the mapping relationship, specifically as follows: Examine the key nodes in the topological subgraph, and use the mapping table to find text blocks that are not present in the text block set but have high relevance, and add them to the candidate set; Check whether the text content in the text block set supports the relationship in the topological subgraph. If a contradiction exists, reduce the graph path weight of this topological subgraph.

[0013] Furthermore, in step 4, heterogeneous data fusion and re-ranking are performed on the adjusted text retrieval results and graph retrieval results to generate a unified enhanced context information package, specifically: The adjusted semantic similarity scores and graph path weights are normalized. The final score is calculated based on the processed semantic similarity score and graph path weights: ; In the formula, α This is a dynamic adjustment coefficient. Rank For the ranking of each path, k It is a smoothing constant; The top N results are selected from the final scores, from highest to lowest, and encapsulated into entity relationship declarations and in-depth background description text to obtain a unified enhanced context information package.

[0014] Furthermore, in step 5, the enhanced contextual information package is input into the large language model to generate the final answer, specifically as follows: The enhanced contextual information is injected into the prompt word template of the large language model to generate the final answer.

[0015] In summary, the present invention has at least one of the following beneficial technical effects: 1. Achieving deep coupling between "global structure" and "local semantics": By establishing a mapping matrix between vector index and graph index, it can not only retrieve semantically similar text blocks, but also simultaneously activate the associated paths in the graph, ensuring that the retrieval results have both fine-grained descriptions of text and macroscopic logical structures of the graph, significantly improving the completeness of the answer.

[0016] 2. Enhance the multi-hop reasoning ability for complex cross-domain problems. For problems that require "turning corners" of thinking, this invention utilizes the path guidance capability of graph RAG to assist text RAG in breaking through the limitations of single paragraph retrieval. Even if key information is distributed in different documents, it can use graph nodes as bridges to connect scattered text blocks into a complete chain of evidence, effectively solving the problem of "multi-hop question answering".

[0017] 3. To improve the robustness and recall of the retrieval system, a complementary feedback mechanism is introduced. When the matching degree of text retrieval is low due to differences in expression, the graph relationship can provide related entities for topological retrieval compensation. When the graph relationship is sparse, the text semantic vector can fill in the background knowledge outside the structure. This dual-engine complementary mode greatly reduces the probability of information missed detection.

[0018] 4. Significantly reduces the illusion rate generated by large models. By providing the generative model with dual constraints of "fact triples (from graphs) + detailed context (from text)," the large language model is constrained by more rigorous knowledge boundaries when generating answers, thus outputting more authoritative and logically rigorous responses. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a diagram illustrating the text recall results; Figure 3 A schematic diagram of the RAG storage structure. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0021] like Figure 1 As shown, this invention provides a hybrid retrieval enhancement method combining unstructured text and structured graphs, including: Step 1: Construct a heterogeneous index for the original unstructured document, simultaneously generate a text vector index and a knowledge graph index, and establish a mapping relationship between the two; Step 2: Receive user query requests and perform vector-based semantic retrieval and knowledge graph-based structural retrieval in parallel to obtain text block sets and topological subgraphs, respectively. Step 3: Based on the mapping relationship, dynamically adjust the text block set and topological subgraph; Step 4: Perform heterogeneous data fusion and reordering on the adjusted text search results and graph search results to generate a unified enhanced context information package; Step 5: Input the enhanced contextual information package into the large language model to generate the final answer.

[0022] The above methods will be described in detail with reference to specific embodiments: In step 1, a heterogeneous index is constructed on the original unstructured document, simultaneously generating a text vector index and a knowledge graph index, and establishing a mapping relationship between the two, specifically as follows: The original unstructured document is divided into fixed-length or semantically complete text chunks based on the sliding window algorithm. The text chunks are converted into high-dimensional vectors using a pre-trained Embedding model (such as BGE-M3), stored in a vector database, and each text chunk is assigned a unique text chunk ID. Entities and relations are extracted from documents using Named Entity Recognition (NER) and Relation Extraction (RE) models, and the extracted knowledge is stored in a graph database. Establish a mapping table M to record the text block ID of each entity, expressed by the formula: ; This step ensures that graph nodes can directly activate the corresponding text context during subsequent dynamic retrieval.

[0023] In step 2, the user query request is received, and vector-based semantic retrieval and knowledge graph-based structural retrieval are performed in parallel to obtain the text block set and the topological subgraph, respectively. Specifically: Receive user query requests Q (e.g., "What are the financial risks of Company XX and what are its affiliated subsidiaries?"), and extract multi-dimensional features from the user query requests, including: Extract keyword vectors for use in vector path retrieval; Identify the core entities (such as "XX Company") in the user's query request as seed nodes for the graph path; Based on the extracted features, vector-based semantic retrieval and knowledge graph-based structural retrieval are performed to obtain a set of text blocks and a topological subgraph, specifically: 1. Vector Path: Based on keyword vectors, perform Top-K search in the vector database to obtain a set S of semantically similar text blocks. vec Each text block is accompanied by a semantic similarity score. vec ; Using the core entity as the seed node, and starting from the seed node, perform an N-hop query in the graph database to extract the topological subgraph G related to the core entity (e.g., "financial risk" and "subsidiary"). sub .

[0024] In step 3, based on the mapping relationship, the text block set and the topological subgraph are dynamically adjusted, specifically as follows: 1. From graph to text: Examining the topological subgraph G sub The key nodes in the text are found through the mapping table M to be absent from the text block set S.vec Text blocks that are highly relevant but not explicitly related are added to the candidate set; 2. Text-to-Graph Analysis: Verify whether the text content in the text block set supports the topological subgraph G. sub If there is a contradiction in the relationship between the topological subgraphs, then the graph path weight of this subgraph is reduced.

[0025] In step 4, heterogeneous data fusion and re-ranking are performed on the adjusted text search results and graph search results to generate a unified enhanced context information package, specifically: The adjusted semantic similarity scores and graph path weights are normalized. The final score is calculated based on the processed semantic similarity score and graph path weights: ; In the formula, α This is a dynamic adjustment coefficient used to automatically adjust based on whether the query intent leans towards "fact" or "description". Rank For the ranking of each path, k This is a smoothing constant used to prevent the denominator from being zero; The top N results are selected from the final scores, from highest to lowest, and encapsulated as: [Entity Relationship Declaration] + [Deep Background Description Text], resulting in a unified enhanced context information package (Context Bundle).

[0026] In step 5, the enhanced contextual information package is input into the large language model to generate the final answer, specifically: This invention injects enhanced contextual information into the prompt word template of a large language model to generate the final answer. An example is provided: Prompt Template: > "Please answer the question: {User_Query} accurately based on the following structured facts: {Entity_Relationships}, and related detailed background: {Text_Chunks}. If the background information is insufficient, please explain why."

[0027] Furthermore, this invention provides a specific embodiment, whose text recall results are as follows: Figure 2 As shown in the figure, the RAG storage structure is as follows: Figure 3 As shown.

[0028] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0029] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0030] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0031] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0032] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A hybrid retrieval enhancement method combining unstructured text and structured graphs, characterized in that, include: Step 1: Construct a heterogeneous index for the original unstructured document, simultaneously generate a text vector index and a knowledge graph index, and establish a mapping relationship between the two; Step 2: Receive user query requests and perform vector-based semantic retrieval and knowledge graph-based structural retrieval in parallel to obtain text block sets and topological subgraphs, respectively. Step 3: Based on the mapping relationship, dynamically adjust the text block set and topological subgraph; Step 4: Perform heterogeneous data fusion and reordering on the adjusted text search results and graph search results to generate a unified enhanced context information package; Step 5: Input the enhanced contextual information package into the large language model to generate the final answer.

2. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 1, characterized in that, In step 1, a heterogeneous index is constructed on the original unstructured document, simultaneously generating a text vector index and a knowledge graph index, and establishing a mapping relationship between the two, specifically as follows: The original unstructured document is segmented into fixed-length or semantically complete text blocks based on the sliding window algorithm. The text blocks are then converted into high-dimensional vectors, stored in a vector database, and each text block is assigned a unique text block ID. Extract entities and relationships from raw unstructured documents and store them in a graph database; Create a mapping table to record the text block ID of each entity that appears.

3. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 2, characterized in that, In step 2, the user query request is received, and vector-based semantic retrieval and knowledge graph-based structural retrieval are performed in parallel to obtain the text block set and the topological subgraph, respectively. Specifically: Receive user query requests and extract multi-dimensional features from the user query requests; Based on the extracted features, vector-based semantic retrieval and knowledge graph-based structural retrieval are performed to obtain a set of text blocks and a topological subgraph.

4. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 3, characterized in that, The features obtained from multi-dimensional feature extraction include keyword vectors and core entities.

5. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 4, characterized in that, Based on the extracted features, vector-based semantic retrieval and knowledge graph-based structural retrieval are performed to obtain a set of text blocks and a topological subgraph, specifically: Based on keyword vectors, a Top-K search is performed in the vector database to obtain a set of semantically similar text blocks, where each text block has a semantic similarity score; Using the core entity as the seed node, and starting from the seed node, perform an N-hop query in the graph database to extract the topological subgraph related to the core entity.

6. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 5, characterized in that, In step 3, based on the mapping relationship, the text block set and the topological subgraph are dynamically adjusted, specifically as follows: Examine the key nodes in the topological subgraph, and use the mapping table to find text blocks that are not present in the text block set but have high relevance, and add them to the candidate set; Check whether the text content in the text block set supports the relationship in the topological subgraph. If a contradiction exists, reduce the graph path weight of this topological subgraph.

7. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 6, characterized in that, In step 4, heterogeneous data fusion and re-ranking are performed on the adjusted text search results and graph search results to generate a unified enhanced context information package, specifically: The adjusted semantic similarity scores and graph path weights are normalized. The final score is calculated based on the processed semantic similarity score and graph path weights: ； In the formula, α This is a dynamic adjustment coefficient. Rank For the ranking of each path, k It is a smoothing constant; The top N results are selected from the final scores, from highest to lowest, and encapsulated into entity relationship declarations and in-depth background description text to obtain a unified enhanced context information package.

8. The hybrid retrieval enhancement method combining unstructured text and structured graphs according to claim 7, characterized in that, In step 5, the enhanced contextual information package is input into the large language model to generate the final answer, specifically: The enhanced contextual information is injected into the prompt word template of the large language model to generate the final answer.

Images