A method for generating search enhancements based on structured content protection and hybrid search

Abstract

The application discloses a retrieval enhancement generation method based on structured content protection and mixed retrieval, and steps are as follows: original documents in a knowledge base are preprocessed by structured content protection to obtain protected texts; the protected texts are segmented and restored to obtain a plurality of text blocks with complete semantics, and all the text blocks are converted into multidimensional vectors; after a target index type is selected from candidate index types according to the number of the text blocks, a vector index is constructed; the multidimensional vectors and the corresponding protected texts are stored into the vector index to construct an RAG knowledge base; a user question and a keyword in the user question are acquired, a mixed retrieval strategy is adopted to recall contexts from the RAG knowledge base, and an answer is generated based on the contexts. The method avoids missing recall of semantic drift, significantly improves the stability and relevance of recall, effectively reduces the probability of fact-based errors generated by a large language model, and improves the accuracy and robustness of a question and answer system.

Description

Technical Field

[0001] This invention relates to a retrieval enhancement generation method, specifically a retrieval enhancement generation method based on structured content protection and hybrid retrieval. Background Technology

[0002] Traditional large language models (LLMs) such as Qwen and DeepSeek, while possessing powerful language understanding and generation capabilities, are limited by their training data and struggle to handle real-time updates or domain-specific knowledge, leading to potentially outdated or factually incorrect generated content. Retrieval-enhanced generation (RAG) technology is an effective solution to this problem. Its core process involves: segmenting documents from an external knowledge base into text blocks and converting these text blocks into vectors using an embedding model; then, converting the user's question into a vector as well, and retrieving the text blocks most semantically relevant to the question from the knowledge base by calculating the cosine similarity of the vectors; finally, inputting these text blocks along with the question as context into the large language model to constrain the model to generate fact-based answers.

[0003] However, Retrieval Enhanced Generation (RAG) technology has significant drawbacks when processing technical documents containing structured content such as tables and formulas: Document preprocessing schemes typically employ fixed-length or recursive segmentation methods based on simple delimiters, directly dividing the original text. This segmentation method easily disrupts structurally complete and semantically independent tables or LaTeX formulas, causing key information to be fragmented, compromising semantic integrity, and affecting the quality of subsequent retrieval and generation. Furthermore, the retrieval mechanism heavily relies on vector similarity and lacks explicit constraints on key entities in the user's question. This leads to the possibility of missing documents containing key entities in the high-dimensional semantic space due to semantic drift, resulting in unstable recall and directly impacting the accuracy of the final answer. Simultaneously, some vector databases (such as FAISS) use static parameters when building their indexes, making it impossible to adaptively adjust to dynamically changing knowledge bases, thus failing to balance retrieval efficiency and recall. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problems of existing technologies, such as difficulty in preserving the semantic integrity of documents, unstable recall rate, and poor retrieval efficiency, and to provide a retrieval enhancement generation method based on structured content protection and hybrid retrieval.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0006] A retrieval enhancement generation method based on structured content protection and hybrid retrieval, characterized by the following steps:

[0007] Step S1: Perform structured content protection preprocessing on all original documents in the knowledge base to obtain the corresponding protected text;

[0008] The structured protection preprocessing refers to processing the original documents in the knowledge base, identifying the HTML tables and LaTeX mathematical formulas, and then replacing the HTML tables and LaTeX mathematical formulas with unique placeholders.

[0009] Step S2: The protected text is segmented and restored to obtain multiple semantically complete text blocks, and then all text blocks are converted into multidimensional vectors.

[0010] Step S3: After selecting the target index type from the candidate index types based on the number of text blocks, construct the vector index;

[0011] Step S4: Store the multidimensional vector and its corresponding protected text into the vector index to construct the RAG knowledge base;

[0012] Step S5: Obtain the user question and the set of keywords in the user question; use a hybrid retrieval strategy based on cosine similarity, semantic relevance and keywords to retrieve the context from the RAG knowledge base; and generate an answer based on the context to complete the retrieval enhancement generation based on structured content protection and hybrid retrieval.

[0013] Furthermore, step S1 specifically includes:

[0014] Step S1.1: Use MinerU to parse and refine the original documents in the knowledge base to obtain plain text content;

[0015] Step S1.2: Identify HTML tables and LaTeX mathematical formulas in plain text content using regular expressions, and replace the HTML tables and LaTeX mathematical formulas with unique placeholders to obtain the protected text.

[0016] Furthermore, step S2 specifically includes:

[0017] Step S2.1: Set the maximum block length of the recursive character text splitter to 700 characters and the overlap length of adjacent blocks to 120 characters.

[0018] Step S2.2: Using the pre-defined recursive character text splitter, select valid delimiters for each protected text according to the priority order of the preset delimiter list, and split each protected text.

[0019] Step S2.3: After all the protected text has been segmented, the placeholders in all the protected text are restored to HTML tables and LaTeX mathematical formulas to obtain multiple semantically complete text blocks;

[0020] Step S2.4: Use a pre-trained embedding model to convert all text blocks into multi-dimensional vectors.

[0021] Furthermore, the priority order of the preset delimiter list in step S2.2 is as follows:

[0022] First priority: three consecutive newlines, indicating a strong paragraph separator: "\n\n\n";

[0023] Second priority: Two consecutive newlines, representing the standard paragraph separator: "\n\n";

[0024] Third priority: Markdown heading separators arranged in descending order of priority: "\n#", "\n##", "\n###", "\n###", "\n####", "\n####", "\n#####".

[0025] Fourth priority: Common separators arranged in descending order of priority: "\n", ".", ",", " ", "".

[0026] Furthermore, step S3 specifically includes:

[0027] Step S3.1: Define the number of text blocks as N. When N ≥ a preset threshold, select the IVF index of the FAISS vector database as the target index type; when N < the preset threshold, select the planar index of the FAISS vector database as the target index type.

[0028] Step S3.2: When IVF index is selected as the target index type, calculate the number of cluster centers of IVF index, set the number of clusters to be scanned during retrieval, and complete the construction of IVF index;

[0029] When a planar index is selected as the target index type, the vector dimension and distance metric are specified to complete the construction of the planar index.

[0030] Furthermore, the preset threshold in step S3.1 is 10000.

[0031] Furthermore, the specific construction process of the IVF index described in step S3.2 is as follows:

[0032] Calculate the number of basic cluster centers in the IVF index And set the maximum allowed number of clusters. Ensure that each basic cluster center contains at least 30 vectors;

[0033] The final number of cluster centers is calculated based on the basic number of cluster centers and the maximum allowed number of clusters.

[0034] Parameters for setting the number of clusters scanned during retrieval The IVF index was trained and built using the IndexIVFFlat function in the FAISS vector database. The function parameter quantizer in the IVF index was set to faiss.IndexFlatIP, dimension was the input vector dimension, nlist was the final number of cluster centers, and the distance metric was faiss.METRIC_INNER_PRODUCT.

[0035] Furthermore, step S4 specifically includes:

[0036] Step S4.1: Store the multidimensional vector and its corresponding protected text into the vector index;

[0037] Step S4.2: Save the vector index to the local disk in a continuous storage manner to form an RAG knowledge base that can be loaded offline.

[0038] Furthermore, step S5 specifically includes:

[0039] Step S5.1: Obtain the user's question and obtain the set of keywords in the user's question through a named entity recognition algorithm or a keyword extraction algorithm;

[0040] Step S5.2: Use the embedding model to convert the user question into a query vector, and perform cosine similarity retrieval in the RAG knowledge base to recall multiple text blocks with the highest similarity to the query vector;

[0041] Step S5.3: Use a pre-trained reordering model to score the relevance of the recalled text blocks to obtain candidate text blocks;

[0042] Step S5.4: Select text blocks from the candidate text blocks that contain at least one keyword from the keyword set as the final context;

[0043] Step S5.5: Concatenate the final context with the user question, input it into the pre-trained large language model, generate an answer, and complete the retrieval enhancement generation based on structured content protection and hybrid retrieval.

[0044] Furthermore, the rearrangement model in step S5.3 is the bge-reranker-v2-m3 model.

[0045] Compared with the prior art, the present invention has the following beneficial technical effects:

[0046] (1) This invention avoids the problem of structured content breakage caused by traditional segmentation methods through structured content protection preprocessing, providing a complete and effective contextual information foundation for subsequent retrieval and generation. It also dynamically constructs vector indexes based on the number of text blocks, taking into account both retrieval efficiency and recall rate. Combined with knowledge base storage and hybrid retrieval collaboration, it ensures semantic relevance. Furthermore, it avoids missing recall due to semantic drift through explicit keyword constraints, significantly improving the stability and relevance of recall. Finally, it generates answers based on high-quality context of recall, effectively reducing the probability of factual errors generated by large language models and improving the accuracy and robustness of the question answering system.

[0047] (2) This invention replaces structured content with placeholders before segmenting, and restores the placeholders after segmentation. This allows structured elements such as HTML tables and LaTeX mathematical formulas to be completely preserved during text segmentation, without breaking or losing key information due to segmentation operations. This ensures the integrity and coherence of the semantics within the text block, and fundamentally solves the technical problem that traditional RAG systems are prone to damage within structured content when processing technical documents.

[0048] (3) This invention achieves broad semantic recall through a three-level retrieval mechanism of vector similarity retrieval, reordering model scoring and keyword screening, ensuring that no semantically related candidate text blocks are missed, while improving the alignment between the recall results and the user's true intent, and significantly improving the recall rate. Attached Figure Description

[0049] Figure 1 This is a flowchart of a retrieval enhancement generation method based on structured content protection and hybrid retrieval according to the present invention.

[0050] Figure 2 This is a comparison chart of the recall rates of a retrieval enhancement generation method based on structured content protection and hybrid retrieval in step S5 of the present invention, which uses three-level hybrid retrieval, compared with retrieval generation methods using two-level hybrid retrieval and those without hybrid retrieval.

[0051] Figure 3 This is a comparison chart showing the recall rate of a retrieval generation method based on structured content protection and hybrid retrieval, in step S1 of an embodiment of the present invention, which uses MinerU for structured content protection, and a retrieval generation method that does not use MinerU for structured content protection. Detailed Implementation

[0052] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0053] like Figure 1 As shown, this embodiment provides a retrieval enhancement generation method based on structured content protection and hybrid retrieval, including the following steps:

[0054] Step S1: Perform structured content protection preprocessing on the original documents in the knowledge base to obtain protected text;

[0055] Step S1.1: Use MinerU to parse and refine all the original documents in the knowledge base to obtain plain text content;

[0056] Step S1.2: Identify HTML tables in plain text content using regular expressions. .*? The HTML table and LaTeX mathematical formulas (in the form of $$, $$...$$) are replaced with unique placeholders (such as __PROTECTED_BLOCK_0__) to obtain the protected text.

[0057] Step S2: The protected text is segmented and restored to obtain multiple semantically complete text blocks, and then all text blocks are converted into multidimensional vectors.

[0058] Step S2.1: Set the maximum block length of the recursive character text splitter to 700 characters and the overlap length of adjacent blocks to 120 characters.

[0059] Step S2.2: Using the pre-defined recursive character text splitter, select valid delimiters for each protected text according to the priority order of the preset delimiter list, and split each protected text; wherein the priority order of the preset delimiter list is:

[0060] First priority: three consecutive newlines, indicating a strong paragraph separator: "\n\n\n";

[0061] Second priority: Two consecutive newlines, representing the standard paragraph separator: "\n\n";

[0062] Third priority: Markdown heading separators arranged in descending order of priority: "\n#", "\n##", "\n###", "\n###", "\n####", "\n####", "\n#####".

[0063] Fourth priority: Common separators arranged in descending order of priority: "\n", ".", ",", " ", "";

[0064] Step S2.3: After all the protected text has been segmented, the placeholders in all the protected text are restored to the original HTML tables and LaTeX mathematical formulas to obtain multiple semantically complete text blocks.

[0065] Step S2.4: Use the pre-trained embedding model bge-small-zh-v1.5 to convert all text blocks into multi-dimensional vectors.

[0066] Step S3: After selecting the target index type from the candidate index types based on the number of text blocks, construct the vector index;

[0067] Step S3.1: Define the number of text blocks as N. When N≥10000, select the IVF index of the FAISS vector database as the target index type; when N<10000, select the planar index of the FAISS vector database as the target index type.

[0068] Step S3.2: When IVF index is selected as the target index type, calculate the number of cluster centers of IVF index, set the number of clusters to be scanned during retrieval, and complete the construction of IVF index;

[0069] When a planar index is selected as the target index type, the vector dimension and distance metric are specified to complete the construction of the planar index.

[0070] In some specific embodiments, the specific construction process of the IVF index is as follows:

[0071] Calculate the number of basic cluster centers in the IVF index And set the maximum allowed number of clusters. Ensure that each basic cluster center contains at least 30 vectors;

[0072] The final number of cluster centers is calculated based on the basic number of cluster centers and the maximum allowed number of clusters. ;

[0073] Parameters for setting the number of clusters scanned during retrieval The IVF index was trained and built using the IndexIVFFlat function in the FAISS vector database. The function parameter quantizer in the IVF index was set to faiss.IndexFlatIP, dimension was the input vector dimension, nlist was the final number of cluster centers, and the distance metric was faiss.METRIC_INNER_PRODUCT.

[0074] Step S4: Store the multidimensional vector and its corresponding protected text into the vector index to construct the RAG knowledge base;

[0075] Step S4.1: Store the multidimensional vector and its corresponding protected text into the vector index;

[0076] Step S4.2: Save the vector index to the local disk in a continuous storage manner to form an RAG knowledge base that can be loaded offline.

[0077] Step S5: Obtain the user question and the keywords in the user question, retrieve the context from the RAG knowledge base using a hybrid retrieval strategy based on cosine similarity, semantic relevance and keywords, and generate an answer based on the context.

[0078] Step S5.1: Obtain the user's question and obtain the set of keywords in the user's question through a named entity recognition algorithm or a keyword extraction algorithm;

[0079] Step S5.2: Use the embedding model to convert the user question into a query vector, and perform cosine similarity retrieval in the RAG knowledge base to recall multiple text blocks with the highest similarity to the query vector;

[0080] Step S5.3: Use a pre-trained reranking model to score the relevance of the recalled text blocks to obtain candidate text blocks; the reranking model adopts the bge-reranker-v2-m3 model.

[0081] Step S5.4: Select text blocks from the candidate text blocks that contain at least one keyword from the keyword set as the final context;

[0082] Step S5.5: Concatenate the final context with the user question, input it into the pre-trained large language model, generate an answer, and complete the retrieval enhancement generation based on structured content protection and hybrid retrieval.

[0083] To demonstrate the effectiveness of this question-and-answer method, technicians conducted a study based on 50 identical user questions (such as "What are the causes of inlet pressure failure?", "How should I resolve a manual judgment anomaly during inertial navigation system self-test?", "What is the SWA status word related to?", and "How can I determine if the accelerometer is functioning normally?"). After preprocessing the original documents in the fault knowledge base using structured content protection, three-level hybrid retrieval, two-level hybrid retrieval, and no hybrid retrieval were used to retrieve context from the fault knowledge base. Answers were generated based on this contextual information (the local fault knowledge base includes 300 documents covering inertial navigation system and accelerometer fault reports, general knowledge, industry standards, etc., in formats such as PDF and Word; contextual information includes "Inlet pressure failure is due to a problem with the pressure supply equipment..."). The recall rates for the three-level hybrid retrieval, two-level hybrid retrieval, and no hybrid retrieval methods were compared. Figure 2As shown, with 50 identical user questions, the three-level hybrid retrieval method significantly improved the recall rate of user questions in the knowledge base compared to two-level retrieval and no hybrid retrieval method, and reduced the illusion of large model output by a greater probability. The recall rates of using MinerU for structured content protection versus not using MinerU are compared as follows: Figure 3 As shown, it can be seen that the use of MinerU and structured text protection can solve the problems of "semantic fragmentation" and "information loss" when traditional parsing tools process complex documents, thus greatly improving their recall rate.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A retrieval enhancement generation method based on structured content protection and hybrid retrieval, characterized in that, Includes the following steps: Step S1: Perform structured content protection preprocessing on all original documents in the knowledge base to obtain the corresponding protected text; The structured protection preprocessing refers to processing the original documents in the knowledge base, identifying the HTML tables and LaTeX mathematical formulas, and then replacing the HTML tables and LaTeX mathematical formulas with unique placeholders. Step S2: The protected text is segmented and restored to obtain multiple semantically complete text blocks, and then all text blocks are converted into multidimensional vectors. Step S3: After selecting the target index type from the candidate index types based on the number of text blocks, construct the vector index; Step S4: Store the multidimensional vector and its corresponding protected text into the vector index to construct the RAG knowledge base; Step S5: Obtain the user question and the set of keywords in the user question; use a hybrid retrieval strategy based on cosine similarity, semantic relevance and keywords to retrieve the context from the RAG knowledge base; and generate an answer based on the context to complete the retrieval enhancement generation based on structured content protection and hybrid retrieval.

2. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 1, characterized in that, Step S1 is as follows: Step S1.1: Use MinerU to parse and refine all the original documents in the knowledge base to obtain plain text content; Step S1.2: Identify HTML tables and LaTeX mathematical formulas in plain text content using regular expressions, and replace the HTML tables and LaTeX mathematical formulas with unique placeholders to obtain the protected text.

3. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 1, characterized in that, Step S2 is as follows: Step S2.1: Set the maximum block length of the recursive character text splitter to 700 characters and the overlap length of adjacent blocks to 120 characters. Step S2.2: Using the pre-defined recursive character text splitter, select valid delimiters for each protected text according to the priority order of the preset delimiter list, and split each protected text. Step S2.3: After all the protected text has been segmented, the placeholders in all the protected text are restored to the original HTML tables and LaTeX mathematical formulas to obtain multiple semantically complete text blocks. Step S2.4: Use a pre-trained embedding model to convert all text blocks into multi-dimensional vectors.

4. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 3, characterized in that, The priority order of the preset delimiter list in step S2.2 is as follows: First priority: three consecutive newlines, indicating a strong paragraph separator: "\n\n\n"; Second priority: Two consecutive newlines, representing the standard paragraph separator: "\n\n"; Third priority: Markdown heading separators arranged in descending order of priority: "\n#", "\n##", "\n###", "\n####", "\n####", "\n#####", "\n######"; Fourth priority: Common separators arranged in descending order of priority: "\n", ".", ",", " ", "".

5. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 3, characterized in that, Step S3 is as follows: Step S3.1: Define the number of text blocks as N. When N ≥ a preset threshold, select the IVF index of the FAISS vector database as the target index type. When N < preset threshold, the planar index of the FAISS vector database is selected as the target index type; Step S3.2: When IVF index is selected as the target index type, calculate the number of cluster centers of IVF index, set the number of clusters to be scanned during retrieval, and complete the construction of IVF index; When a planar index is selected as the target index type, the vector dimension and distance metric are specified to complete the construction of the planar index.

6. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 5, characterized in that, The preset threshold in step S3.1 is 10000.

7. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 5, characterized in that, The specific construction process of the IVF index in step S3.2 is as follows: a. Calculate the number of basic cluster centers in the IVF index. And set the maximum allowed number of clusters. Ensure that each basic cluster center contains at least 30 vectors; b. Calculate the final number of cluster centers based on the basic number of cluster centers and the maximum allowed number of clusters; c. Set the parameter for scanning the number of clusters during retrieval. The IVF index was trained and built using the IndexIVFFlat function in the FAISS vector database. The function parameter quantizer in the IVF index was set to faiss.IndexFlatIP, dimension was the input vector dimension, nlist was the final number of cluster centers, and the distance metric was faiss.METRIC_INNER_PRODUCT.

8. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 5, characterized in that, Step S4 is as follows: Step S4.1: Store the multidimensional vector and its corresponding protected text into the vector index; Step S4.2: Save the vector index to the local disk in a continuous storage manner to form an RAG knowledge base that can be loaded offline.

9. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 8, characterized in that, Step S5 is as follows: Step S5.1: Obtain the user's question and obtain the set of keywords in the user's question through a named entity recognition algorithm or a keyword extraction algorithm; Step S5.2: Use the embedding model to convert the user question into a query vector, and perform cosine similarity retrieval in the RAG knowledge base to recall multiple text blocks with the highest similarity to the query vector; Step S5.3: Use a pre-trained reordering model to score the relevance of the recalled text blocks to obtain candidate text blocks; Step S5.4: Select text blocks from the candidate text blocks that contain at least one keyword from the keyword set as the final context; Step S5.5: Concatenate the final context with the user question, input it into the pre-trained large language model, generate an answer, and complete the retrieval enhancement generation based on structured content protection and hybrid retrieval.

10. The retrieval enhancement generation method based on structured content protection and hybrid retrieval according to claim 9, characterized in that, The rearrangement model in step S5.3 is the bge-reranker-v2-m3 model.

Images