A context-aware based retrieval augmentation generation method and system
By employing a two-stage retrieval mechanism and document structure indexing, the semantic fragmentation and information gaps of traditional RAG systems are resolved, generating coherent and complete retrieval results and improving retrieval accuracy and user experience.
Patent Information
- Application Number
- CN202511524780.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-03-03
- Estimated Expiration
- 2045-10-24
AI Technical Summary
Traditional RAG systems suffer from semantic fragmentation, information gaps, semantic boundary disruption, and biased relevance assessments, resulting in a lack of coherence and accuracy in search results.
A two-stage retrieval mechanism is adopted. First, the most relevant text blocks are found through vector similarity. Then, the text blocks before and after the document are retrieved based on the document structure and context expansion, restoring the document integrity. Natural text is generated through smooth transition technology.
It significantly improved retrieval accuracy and coherence, enhanced the accuracy, answer completeness, and user experience of the standard question-and-answer test set, reduced query complexity, and improved the system's ability to handle complex problems.
Smart Images
Figure CN121009996B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval technology, and in particular to a context-aware retrieval enhancement generation method and a context-aware retrieval enhancement generation system. Background Technology
[0002] Traditional RAG (Retrieval Enhancement Generation) systems primarily employ vector retrieval methods to find text relevant to user queries. The working principle involves converting both the user query and knowledge base documents into vector representations, and then calculating vector similarity (such as cosine similarity) to find the most similar document fragments.
[0003] These systems typically divide long documents into independent chunks, each usually containing about 500-1000 tokens. During retrieval, the system calculates the similarity between the query and each text chunk and returns the Top K most similar text chunks.
[0004] Traditional RAG systems have the following key drawbacks:
[0005] (1) Semantic fragmentation: Because the document is divided into independent small pieces, the search results are often a group of text fragments that lack coherence and cannot maintain the contextual coherence of the original document.
[0006] (2) Information gaps: Key information may span multiple text blocks. When only some blocks are retrieved, the information will be incomplete, especially in cases involving complex concept explanations, multi-step processes, or conditions that require prerequisites.
[0007] (3) Semantic boundary destruction: The text segmentation process is usually mechanical (such as segmenting by the number of characters or sentences), and may be segmented in inappropriate positions (such as in the middle of a paragraph or inside a sentence), which destroys the semantic integrity.
[0008] (4) One-sided relevance judgment: Relevance is judged only based on the similarity between a single text block and the query, while ignoring the context and information flow within the document, which reduces the accuracy of retrieval. Summary of the Invention
[0009] To address the aforementioned issues, this invention provides a context-aware retrieval enhancement generation method and system. Through a two-stage retrieval mechanism, the initial retrieval ensures the discovery of the most relevant text block to the query, while the context-expanded retrieval restores the complete context of the document by obtaining the preceding and succeeding text blocks. This approach guarantees both relevance and contextual integrity, effectively solving the semantic fragmentation problem of traditional RAG systems and significantly improving retrieval quality.
[0010] To achieve the above objectives, the present invention provides a context-aware retrieval enhancement generation method, comprising:
[0011] The knowledge base documents are divided into blocks, and each text block is assigned a unique identifier and sorting number. The relationship between text blocks is established to save the document structure.
[0012] A structural index of the document is constructed based on the text blocks and their relationships.
[0013] Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance.
[0014] Based on the document structure and the structure index, a context-extended retrieval is performed on the initial set of related text blocks to obtain an extended set of preceding and following text blocks;
[0015] The set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the retrieval result.
[0016] In the above technical solution, preferably, the specific process of segmenting the knowledge base documents includes:
[0017] Natural language processing techniques are used to identify semantic boundaries in the document, including paragraph boundaries, topic boundaries, and concept boundaries.
[0018] Based on the principle of semantic integrity, strongly related content is preserved as a whole as a text block, thus achieving text segmentation.
[0019] In the above technical solution, preferably, a unique identifier and sorting number are assigned to each text block to establish the association between text blocks in order to save the document structure. The specific process includes:
[0020] Each text block is assigned a globally unique identifier and sorting number according to its positional relationship and logical order in the original document.
[0021] The text blocks are associated with and stored together with their corresponding globally unique identifiers and sorting numbers to establish the association between the text blocks.
[0022] In the above technical solution, preferably, the structural index of the document is constructed based on the text blocks and their relationships, and the specific process includes:
[0023] The text block is vectorized and a vector index is constructed.
[0024] Based on the relationships between the text blocks, a structural index is constructed between the corresponding vector indices;
[0025] A fast query index is constructed based on the sorting sequence number of the text block, which supports querying adjacent blocks of the preceding and succeeding text blocks of the text block.
[0026] In the above technical solution, preferably, based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. The specific process includes:
[0027] The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm.
[0028] Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results.
[0029] In the above technical solution, preferably, a context-extended retrieval is performed on the initial set of related text blocks based on the document structure and the structure index to obtain an extended set of contextual text blocks. The specific process includes:
[0030] Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them.
[0031] The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block.
[0032] Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks;
[0033] The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.
[0034] In the above technical solution, preferably, the set of preceding and following text blocks is subjected to deduplication, merging, and reordering processes, and a smooth transition technique is used to obtain natural text as the retrieval result. The specific process includes:
[0035] Identify overlapping text blocks in the set of preceding and following text blocks, and merge the overlapping text blocks.
[0036] The search results are reordered based on the sorting number of the text blocks in the preceding and following text block set to restore the original logical order of the documents;
[0037] The reordered text blocks are smoothed using a smooth transition technique, and the processed natural text is used as the search result.
[0038] In the above technical solution, preferably, natural language processing technology is used to identify the user's query intent and key semantic entities in response to the user's query request;
[0039] Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request;
[0040] The expanded query request is converted into a vector for initial retrieval.
[0041] This invention also proposes a context-aware retrieval enhancement generation system, which applies the context-aware retrieval enhancement generation method disclosed in any of the above technical solutions, including:
[0042] The text block preprocessing module is used to divide the knowledge base document into blocks, assign a unique identifier and sorting number to each text block, and establish the relationship between text blocks to save the document structure.
[0043] A text structure indexing module is used to construct a structural index of the document based on the text blocks and their relationships.
[0044] The initial retrieval module is used to perform an initial retrieval of the query content based on vector similarity according to the user's query request, and obtain the initial set of relevant text blocks with the highest relevance.
[0045] The context-extended retrieval module is used to perform context-extended retrieval on the initial set of related text blocks based on the document structure and the structure index, so as to obtain an extended set of context-extended text blocks;
[0046] The search result optimization module is used to perform deduplication, merging and reordering of the set of preceding and following text blocks, and to obtain natural text as the search result using a smooth transition technique.
[0047] In the above technical solution, preferably, the initial retrieval module is specifically used for:
[0048] The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm.
[0049] Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results;
[0050] The aforementioned front-to-back extended retrieval module is specifically used for:
[0051] Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them.
[0052] The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block.
[0053] Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks;
[0054] The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.
[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0056] (1) The semantic fragmentation problem of traditional RAG systems is effectively solved by a two-stage retrieval mechanism. The first stage, based on vector similarity, finds the most relevant text blocks to the query, while the second stage, based on document structure and context expansion, recovers the complete context of the document by obtaining the preceding and succeeding text blocks. This design improves the accuracy on the standard question-answering test set from 65% to 87% of the traditional system, an increase of 34%, significantly improving retrieval quality.
[0057] (2) By using document structure preservation and semantic boundary recognition technology, the completeness of the answer is greatly improved. Based on assigning a unique identifier and sorting number to each text block, the natural boundaries of paragraphs, topics, and concepts can be accurately identified, avoiding truncation in the middle of important concepts. When dealing with complex questions that require a comprehensive understanding of the document content, this innovation increases the completeness of the answer from 54% to 92%, an improvement of 70%, ensuring the comprehensiveness and accuracy of the generated content.
[0058] (3) A mechanism for dynamically adjusting the size of the context window is adopted, which adaptively determines the expansion range based on the complexity of the content and the density of information, ensuring the integrity of the context while avoiding information overload. Combined with result merging and reordering technology, the coherence of the generated content is significantly improved. The user's rating of the coherence of the generated answer increased from 3.2 / 5 to 4.7 / 5, an increase of 47%, which greatly improves the user experience.
[0059] (4) By using a high-efficiency indexing system based on sort_order, the complexity of context queries is reduced from O(n) to O(1), which greatly improves retrieval efficiency. This optimization can quickly handle complex queries that require understanding the relationships between multiple paragraphs, and the correct answer rate is increased from 42% to 78%, an improvement of 86%, which significantly enhances the system's ability to handle complex problems. Attached Figure Description
[0060] Figure 1 This is a flowchart illustrating a context-aware retrieval enhancement generation method disclosed in one embodiment of the present invention;
[0061] Figure 2 This is a schematic diagram of the architecture of a retrieval enhancement generation method disclosed in one embodiment of the present invention;
[0062] Figure 3 This is a logical schematic diagram of a retrieval enhancement generation method disclosed in one embodiment of the present invention. Detailed Implementation
[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0064] The present invention will now be described in further detail with reference to the accompanying drawings:
[0065] like Figures 1 to 3 As shown, a context-aware retrieval enhancement generation method according to the present invention includes:
[0066] The knowledge base documents are divided into blocks, and each text block is assigned a unique identifier and sorting number. The relationship between text blocks is established to save the document structure.
[0067] A structural index of a document is constructed based on text blocks and their relationships.
[0068] Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance.
[0069] Based on the document structure and structure index, the initial set of related text blocks is expanded by contextual retrieval to obtain an expanded set of preceding and following text blocks;
[0070] The set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the search result.
[0071] In this implementation, a two-stage retrieval mechanism is used. The initial retrieval ensures that the most relevant text block to the query is found, while the context expansion retrieval restores the complete context of the document by obtaining the preceding and succeeding text blocks. This ensures both relevance and contextual integrity, effectively solving the semantic fragmentation problem of traditional RAG systems and significantly improving retrieval quality.
[0072] In the implementation process, firstly, knowledge base documents (such as technical manuals or legal texts) are segmented. A natural language processing library is used to identify the document structure, dividing the document into text blocks of 512 tokens each. Each text block is assigned a globally unique identifier (generated using UUID4) and a sorting number (an integer sequence starting from 1), and relationships between text blocks are established (e.g., recording the IDs of predecessor and successor blocks). Based on this information, a structural index of the document is constructed, storing the text block content and metadata. Simultaneously, a vector index is built using the FAISS library. When a user submits a query request, the Sentence-BERT model is used to convert the query into a 768-dimensional vector. Cosine similarity is used to calculate the similarity with the text block vector, retrieving the top 5 most relevant text blocks as the initial set of relevant text blocks. Subsequently, based on the structural index and sorting number, a contextual expansion retrieval is performed on the initial set, obtaining the two preceding and following text blocks of each text block, forming an expanded set of contextual text blocks. The dataset is deduplicated (e.g., by detecting overlapping regions based on Jaccard similarity), merged (by connecting text content), and reordered (by sorting in ascending order of sorting number). Then, smooth transition techniques (e.g., adding transition sentences or adjusting sentence structure) are applied to generate natural and fluent text as the final search result.
[0073] In the above embodiments, preferably, the specific process of segmenting the knowledge base documents includes:
[0074] Natural language processing techniques are used to identify semantic boundaries in documents, including paragraph boundaries, topic boundaries, and concept boundaries.
[0075] Based on the principle of semantic integrity, strongly related content is preserved as a whole as a text block, thus achieving text segmentation.
[0076] During implementation, the spaCy library can be used to parse documents, identifying paragraph boundaries based on line breaks and indentation. LDA topic modeling can be used to calculate topic change points to identify topic boundaries, and entity recognition and dependency resolution can be used to identify concept boundaries. Based on the principle of semantic integrity, strongly related content (such as a complete paragraph or a concept description) is preserved as a single text block, avoiding truncation in the middle of sentences or concepts. The size of the text blocks is dynamically adjusted, but is typically kept between 500-1000 tokens to ensure semantic integrity. After segmentation, each text block can be stored as a JSON object, containing text content, location information, and boundary type.
[0077] In the above embodiments, preferably, a unique identifier and sorting number are assigned to each text block to establish the association between text blocks in order to save the document structure. The specific process includes:
[0078] Each text block is assigned a globally unique identifier and sorting number according to its positional relationship and logical order in the original document.
[0079] The text blocks are associated with their corresponding globally unique identifiers and sorting numbers for storage, thus establishing the association between the text blocks.
[0080] During implementation, each text block is assigned a globally unique identifier and an integer sorting number according to its logical order in the original document (e.g., from left to right, from top to bottom). The identifier can be generated using the UUID4 algorithm, and the sorting number can increment from 1. A SQL database is used to store text block metadata, including the text block ID, sorting number, predecessor text block ID, successor text block ID, and original document ID. Foreign key constraints are used to establish relationships between text blocks; for example, the predecessor text block ID points to the previous text block, and the successor text block ID points to the next text block, thus preserving the document structure.
[0081] In the above embodiments, preferably, a structural index of the document is constructed based on text blocks and their relationships. The specific process includes:
[0082] Vectorize the text blocks and construct vector indices;
[0083] Based on the relationships between text blocks, construct a structural index between corresponding vector indices;
[0084] A fast query index for text blocks is built based on sorting sequence numbers, supporting queries for adjacent blocks of text blocks that are predecessors and successors of a text block.
[0085] During implementation, text blocks are vectorized using the Sentence-BERT model to generate 768-dimensional vectors and construct vector indexes, such as the IndexFlatIP index type from the FAISS library. Secondly, based on the relationships between text blocks, a graph-structured index can be constructed using the Neo4j graph database, where nodes represent text blocks (attributes include ID, sort number, and content), and edges represent relationships (e.g., NEXT relationships represent successor blocks, and PREVIOUS relationships represent predecessor blocks). Finally, a B-tree index is constructed based on the sort number, supporting fast lookups of adjacent text blocks by sort number, with a query time complexity of O(1).
[0086] In the above implementation, preferably, based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. The specific process includes:
[0087] The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm.
[0088] Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results.
[0089] During implementation, user queries using natural language questions are converted into 768-dimensional vectors using the Sentence-BERT model. Based on the cosine similarity matching algorithm, the similarity score between the query vector and all text block vectors is calculated. This calculation can be performed in batches using the FAISS index to improve efficiency. The top K text blocks with the highest similarity (e.g., K=5, adjustable according to configuration) are identified and grouped into an initial set of relevant text blocks. The initial search results can be returned in JSON format, including the text block ID, similarity score, and content.
[0090] In the above implementation, preferably, a context-expanded retrieval is performed on the initial set of related text blocks based on the document structure and structure index to obtain an expanded set of contextual text blocks. The specific process includes:
[0091] Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and structure index, the sort_order-based indexing system is used to locate the text blocks before and after them.
[0092] The information value of a text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the evaluation results of the information value of the text block.
[0093] Based on the extended scope of the context, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks;
[0094] The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form a set of contextual text blocks that enhance retrieval.
[0095] In the implementation process, firstly, based on the sorting number of each text block in the initial set of related text blocks, a B-tree index is used to quickly locate its predecessor and successor text blocks. Secondly, information entropy is used to calculate the entropy value of the word distribution in the text block, and the keyword density is determined by calculating the occurrence frequency of predefined keywords. The information value of each text block is evaluated based on the entropy value and keyword density; the higher the entropy value or the greater the keyword density, the higher the information value.
[0096] During implementation, the scope of context expansion can be dynamically determined based on the information value assessment results: for text blocks with high information value (e.g., entropy > 5.0 or keyword density > 0.1), expand by 3 text blocks before and after expansion; for text blocks with low information value, expand by 1 text block before and after expansion. Finally, based on the expansion scope, retrieve the preceding text blocks from the initial set of relevant text blocks and the following text blocks from the future, and merge all text blocks into a list to form a set of contextual text blocks for enhanced retrieval.
[0097] In the above embodiments, preferably, the set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the retrieval result. The specific process includes:
[0098] Identify overlapping text blocks in the preceding and following text blocks, and merge the overlapping text blocks.
[0099] The search results are reordered based on the sorting number of the text blocks in the preceding and following text block set to restore the original logical order of the documents;
[0100] The reordered text blocks are smoothed using a smooth transition technique, and the processed natural text is used as the search result.
[0101] During implementation, overlapping text blocks are identified, and the Jaccard similarity of the text block content is calculated based on the word set. If the similarity exceeds a preset threshold (e.g., 30%), the overlapping parts are merged, retaining only the unique content. Secondly, the search results are reordered based on the sorting number of the text blocks, arranged in ascending order to restore the original document's logical order. Finally, smooth transition techniques are applied, such as adding conjunctions (e.g., "in addition," "however") between text blocks or adjusting sentence structure (using a text generation model like GPT-2 for fine-tuning), to ensure the merged text flows smoothly and naturally.
[0102] In the above embodiments, preferably, natural language processing technology is used to identify the user's query intent and key semantic entities in response to the user's query request;
[0103] Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request;
[0104] The expanded query request is converted into a vector for initial retrieval.
[0105] During implementation, a BERT-based classifier can be used to identify the user's query intent, and a NER model can be used to identify key entities, such as names, place names, and technical terms. Based on the identified query intent and key entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request. For example, synonyms can be added, such as expanding "car" to "vehicle," or relevant terms or extended descriptions can be added. The expanded query request is then converted into a vector for initial retrieval, improving retrieval relevance.
[0106] This invention also proposes a context-aware retrieval enhancement generation system, which applies the context-aware retrieval enhancement generation method disclosed in any of the above embodiments, including:
[0107] The text block preprocessing module is used to divide the knowledge base document into blocks, assign a unique identifier and sorting number to each text block, and establish the relationship between text blocks to save the document structure.
[0108] The text structure indexing module is used to build a structural index of a document based on text blocks and their relationships.
[0109] The initial retrieval module is used to perform an initial retrieval of the query content based on vector similarity according to the user's query request, and obtain the initial set of relevant text blocks with the highest relevance.
[0110] The contextual expansion retrieval module is used to perform contextual expansion retrieval on the initial set of related text blocks based on the document structure and structure index, so as to obtain an expanded set of contextual text blocks.
[0111] The search results optimization module is used to deduplicate, merge, and reorder the set of preceding and following text blocks, and uses a smooth transition technique to obtain natural text as the search results.
[0112] In the above embodiments, preferably, the initial retrieval module is specifically used for:
[0113] The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm.
[0114] Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results;
[0115] The front and rear extended search modules are specifically used for:
[0116] Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and structure index, the sort_order-based indexing system is used to locate the text blocks before and after them.
[0117] The information value of a text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the evaluation results of the information value of the text block.
[0118] Based on the extended scope of the context, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks;
[0119] The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form a set of contextual text blocks that enhance retrieval.
[0120] The functions to be implemented by each module of the context-aware retrieval enhancement generation system disclosed in the above embodiments correspond to the steps of the context-aware retrieval enhancement generation method disclosed in the above embodiments. In the implementation process, the above embodiments are referred to for operation, and will not be repeated here.
[0121] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for context-aware retrieval augmentation generation, the method comprising: The application comprises the following steps: The knowledge base document is divided into blocks, and each text block is assigned a unique identifier and a sequence number, and the association between the text blocks is established to save the document structure; The structure index of the document is constructed based on the text blocks and their association; According to the user's query request, the initial retrieval of the query content is carried out based on the vector similarity, and the initial relevant text block set with the highest relevance is obtained; Based on the document structure and the structure index, the initial relevant text block set is subjected to context expansion retrieval, and the expanded context text block set is obtained; The context text block set is subjected to de-duplication, merging and reordering processing, and natural text is obtained as the retrieval result by using a smooth transition technique; The specific process of context expansion retrieval based on the document structure and the structure index to obtain the expanded context text block set comprises the following steps: According to the sequence number of the text block in the initial relevant text block set, the document structure and the structure index are used to locate the adjacent text blocks before and after the text block based on the sort_order index system; The information value of the text block is evaluated based on information entropy and keyword density, and the context expansion range is dynamically determined based on the evaluation result of the information value of the text block; Based on the context expansion range, the predecessor text block is obtained forwardly and the successor text block is obtained backwardly based on the initial relevant text block set; The initial relevant text block set, the predecessor text block and the successor text block are merged to form the context text block set for enhanced retrieval.
2. The context-aware based retrieval augmentation generation method of claim 1, wherein, The specific process of dividing the knowledge base document into blocks comprises the following steps: The semantic boundaries in the document are identified using natural language processing technology, and the semantic boundaries include paragraph boundaries, topic boundaries and concept boundaries; Based on the principle of semantic integrity, the content with strong association is kept as a whole text block to realize text blocking.
3. The context-aware based retrieval augmentation generation method of claim 1, wherein, The specific process of assigning a unique identifier and a sequence number to each text block to establish the association between the text blocks to save the document structure comprises the following steps: According to the position relationship and logical order of each text block in the original document, a globally unique identifier and a sequence number are assigned to each text block; The text blocks and the corresponding globally unique identifiers and sequence numbers are associated and stored to establish the association between the text blocks.
4. The method of claim 3, wherein, The specific process of constructing the structure index of the document based on the text blocks and their association comprises the following steps: The text blocks are represented by vectors, and a vector index is constructed; Based on the association between the text blocks, the structure index between the corresponding vector indexes is constructed; Based on the sequence number, a fast query index of the text blocks is constructed to support the adjacent block query of the predecessor text block and the successor text block of the text block.
5. The context-aware based retrieval augmentation generation method of claim 4, wherein, The specific process of carrying out initial retrieval of the query content based on vector similarity according to the user's query request comprises the following steps: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm; Determine the top pre-set number of text blocks with the highest similarity, and collect the initial relevant text block set as the initial retrieval result.
6. The context-aware based retrieval augmentation generation method of claim 1, wherein, The pre-context and post-context text block set is de-duplicated, merged, and re-ordered, and natural text is obtained as the retrieval result using a smooth transition technique. The specific process includes: Identify overlapping text blocks in the pre-context and post-context text block set, and merge the overlapping text blocks; Reorder the retrieval result based on the sorting order of the text blocks in the pre-context and post-context text block set to restore the original document logical order; Apply a smooth transition technique to the reordered text blocks for smoothing processing, and the processed natural text is taken as the retrieval result.
7. The context-aware based retrieval augmentation generation method of claim 5, wherein, For a user's query request, a natural language processing technique is used to identify the user's query intent and key semantic entities; Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request; The expanded query request is converted into a vector for initial retrieval.
8. A context-aware based retrieval augmentation generation system, comprising: The context-aware retrieval enhancement generation method according to any one of claims 1-7, comprising: A text block preprocessing module for block processing of knowledge base documents, assigning each text block a unique identifier and a sorting order, and establishing an association between text blocks to save the document structure; A text structure index module for constructing a structure index of the document based on the text blocks and their association; An initial query retrieval module for initial retrieval of the query content based on vector similarity according to a user's query request, obtaining an initial relevant text block set with the highest relevance; A pre-context and post-context expansion retrieval module for context expansion retrieval of the initial relevant text block set based on the document structure and the structure index, obtaining an expanded pre-context and post-context text block set; A retrieval result optimization module for de-duplicating, merging, and reordering the pre-context and post-context text block set, and obtaining natural text as the retrieval result using a smooth transition technique.
9. The contextually aware search query enhancement generation system of claim 8, wherein, The initial query retrieval module is specifically configured to: Convert the query request into a vector and calculate the similarity between the query request and the text blocks based on a vector similarity matching algorithm; Determine the top pre-set number of text blocks with the highest similarity, and collect the initial relevant text block set as the initial retrieval result. The pre-context and post-context expansion retrieval module is specifically configured to: Based on the sorting order of the text blocks in the initial relevant text block set, locate the adjacent text blocks before and after it based on the document structure and the structure index using a sort_order-based index system; Evaluate the information value of the text blocks using information entropy and keyword density, and dynamically determine the context expansion range based on the information value evaluation results of the text blocks; Based on the context expansion range, obtain the predecessor text block before and the successor text block after based on the initial relevant text block set; Merge the initial relevant text block set with the predecessor text block and the successor text block to form the pre-context and post-context text block set for enhanced retrieval.
Citation Information
Patent Citations
Knowledge data providing method and device, electronic equipment and storage medium
CN109739964A