Retrieval enhancement generation method and system based on context awareness

Abstract

The invention discloses a retrieval enhancement generation method and system based on context awareness, and relates to the technical field of information retrieval, and the method comprises the following steps: carrying out block processing on a knowledge base document, distributing a unique identifier and a sorting sequence number, and establishing an association relationship between text blocks to save a document structure; constructing a structure index of the document; according to a query request of a user, performing initial retrieval on query content based on the vector similarity to obtain an initial related text block set with the highest relevancy; performing context extension retrieval on the initial related text block set to obtain an extended preamble text block set; and carrying out duplicate removal, merging and reordering processing on the preamble text block set, and adopting a smooth transition technology to obtain a natural text as a retrieval result. Through the technical scheme of the invention, not only is the correlation ensured, but also the context integrity is ensured, the semantic fragmentation problem of a traditional RAG system can be effectively solved, and the retrieval quality is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to a context-aware retrieval enhancement generation method and a context-aware retrieval enhancement generation system. Background Technology

[0002] Traditional RAG (Retrieval Enhancement Generation) systems primarily employ vector retrieval methods to find text relevant to user queries. The working principle involves converting both the user query and knowledge base documents into vector representations, and then calculating vector similarity (such as cosine similarity) to find the most similar document fragments.

[0003] These systems typically divide long documents into independent chunks, each usually containing about 500-1000 tokens. During retrieval, the system calculates the similarity between the query and each text chunk and returns the Top K most similar text chunks.

[0004] Traditional RAG systems have the following key drawbacks: (1) Semantic fragmentation: Because the document is divided into independent small pieces, the search results are often a group of text fragments that lack coherence and cannot maintain the contextual coherence of the original document.

[0005] (2) Information gaps: Key information may span multiple text blocks. When only some blocks are retrieved, the information will be incomplete, especially in cases involving complex concept explanations, multi-step processes, or conditions that require prerequisites.

[0006] (3) Semantic boundary destruction: The text segmentation process is usually mechanical (such as segmenting by the number of characters or sentences), and may be segmented in inappropriate positions (such as in the middle of a paragraph or inside a sentence), which destroys the semantic integrity.

[0007] (4) One-sided relevance judgment: Relevance is judged only based on the similarity between a single text block and the query, while ignoring the context and information flow within the document, which reduces the accuracy of retrieval. Summary of the Invention

[0008] To address the aforementioned issues, this invention provides a context-aware retrieval enhancement generation method and system. Through a two-stage retrieval mechanism, the initial retrieval ensures the discovery of the most relevant text block to the query, while the context-expanded retrieval restores the complete context of the document by obtaining the preceding and succeeding text blocks. This approach guarantees both relevance and contextual integrity, effectively solving the semantic fragmentation problem of traditional RAG systems and significantly improving retrieval quality.

[0009] To achieve the above objectives, the present invention provides a context-aware retrieval enhancement generation method, comprising: The knowledge base documents are divided into blocks, and each text block is assigned a unique identifier and sorting number. The relationship between text blocks is established to save the document structure. A structural index of the document is constructed based on the text blocks and their relationships. Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. Based on the document structure and the structure index, a context-extended retrieval is performed on the initial set of related text blocks to obtain an extended set of preceding and following text blocks; The set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the retrieval result.

[0010] In the above technical solution, preferably, the specific process of segmenting the knowledge base documents includes: Natural language processing techniques are used to identify semantic boundaries in the document, including paragraph boundaries, topic boundaries, and concept boundaries. Based on the principle of semantic integrity, strongly related content is preserved as a whole as a text block, thus achieving text segmentation.

[0011] In the above technical solution, preferably, a unique identifier and sorting number are assigned to each text block to establish the association between text blocks in order to save the document structure. The specific process includes: Each text block is assigned a globally unique identifier and sorting number according to its positional relationship and logical order in the original document. The text blocks are associated with and stored together with their corresponding globally unique identifiers and sorting numbers to establish the association between the text blocks.

[0012] In the above technical solution, preferably, the structural index of the document is constructed based on the text blocks and their relationships, and the specific process includes: The text block is vectorized and a vector index is constructed. Based on the relationships between the text blocks, a structural index is constructed between the corresponding vector indices; A fast query index is constructed based on the sorting sequence number of the text block, which supports querying adjacent blocks of the preceding and succeeding text blocks of the text block.

[0013] In the above technical solution, preferably, based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. The specific process includes: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results.

[0014] In the above technical solution, preferably, a context-extended retrieval is performed on the initial set of related text blocks based on the document structure and the structure index to obtain an extended set of contextual text blocks. The specific process includes: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block. Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.

[0015] In the above technical solution, preferably, the set of preceding and following text blocks is subjected to deduplication, merging, and reordering processes, and a smooth transition technique is used to obtain natural text as the retrieval result. The specific process includes: Identify overlapping text blocks in the set of preceding and following text blocks, and merge the overlapping text blocks. The search results are reordered based on the sorting number of the text blocks in the preceding and following text block set to restore the original logical order of the documents; The reordered text blocks are smoothed using a smooth transition technique, and the processed natural text is used as the search result.

[0016] In the above technical solution, preferably, natural language processing technology is used to identify the user's query intent and key semantic entities in response to the user's query request; Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request; The expanded query request is converted into a vector for initial retrieval.

[0017] This invention also proposes a context-aware retrieval enhancement generation system, which applies the context-aware retrieval enhancement generation method disclosed in any of the above technical solutions, including: The text block preprocessing module is used to divide the knowledge base document into blocks, assign a unique identifier and sorting number to each text block, and establish the relationship between text blocks to save the document structure. A text structure indexing module is used to construct a structural index of the document based on the text blocks and their relationships. The initial retrieval module is used to perform an initial retrieval of the query content based on vector similarity according to the user's query request, and obtain the initial set of relevant text blocks with the highest relevance. The context-extended retrieval module is used to perform context-extended retrieval on the initial set of related text blocks based on the document structure and the structure index, so as to obtain an extended set of context-extended text blocks; The search result optimization module is used to perform deduplication, merging and reordering of the set of preceding and following text blocks, and to obtain natural text as the search result using a smooth transition technique.

[0018] In the above technical solution, preferably, the initial retrieval module is specifically used for: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results; The aforementioned front-to-back extended retrieval module is specifically used for: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block. Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The semantic fragmentation problem of traditional RAG systems is effectively solved by a two-stage retrieval mechanism. The first stage, based on vector similarity, finds the most relevant text blocks to the query, while the second stage, based on document structure and context expansion, recovers the complete context of the document by obtaining the preceding and succeeding text blocks. This design improves the accuracy on the standard question-answering test set from 65% to 87% of the traditional system, an increase of 34%, significantly improving retrieval quality.

[0020] (2) By using document structure preservation and semantic boundary recognition technology, the completeness of the answer is greatly improved. Based on assigning a unique identifier and sorting number to each text block, the natural boundaries of paragraphs, topics, and concepts can be accurately identified, avoiding truncation in the middle of important concepts. When dealing with complex questions that require a comprehensive understanding of the document content, this innovation increases the completeness of the answer from 54% to 92%, an improvement of 70%, ensuring the comprehensiveness and accuracy of the generated content.

[0021] (3) A mechanism for dynamically adjusting the size of the context window is adopted, which adaptively determines the expansion range based on the complexity of the content and the density of information, ensuring the integrity of the context while avoiding information overload. Combined with result merging and reordering technology, the coherence of the generated content is significantly improved. The user's rating of the coherence of the generated answer increased from 3.2 / 5 to 4.7 / 5, an increase of 47%, which greatly improves the user experience.

[0022] (4) By using a high-efficiency indexing system based on sort_order, the complexity of context queries is reduced from O(n) to O(1), which greatly improves retrieval efficiency. This optimization can quickly handle complex queries that require understanding the relationships between multiple paragraphs, and the correct answer rate is increased from 42% to 78%, an improvement of 86%, which significantly enhances the system's ability to handle complex problems. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a context-aware retrieval enhancement generation method disclosed in one embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of a retrieval enhancement generation method disclosed in one embodiment of the present invention; Figure 3 This is a logical schematic diagram of a retrieval enhancement generation method disclosed in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings: like Figures 1 to 3 As shown, a context-aware retrieval enhancement generation method according to the present invention includes: The knowledge base documents are divided into blocks, and each text block is assigned a unique identifier and sorting number. The relationship between text blocks is established to save the document structure. A structural index of a document is constructed based on text blocks and their relationships. Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. Based on the document structure and structure index, the initial set of related text blocks is expanded by contextual retrieval to obtain an expanded set of preceding and following text blocks; The set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the search result.

[0026] In this implementation, a two-stage retrieval mechanism is used. The initial retrieval ensures that the most relevant text block to the query is found, while the context expansion retrieval restores the complete context of the document by obtaining the preceding and succeeding text blocks. This ensures both relevance and contextual integrity, effectively solving the semantic fragmentation problem of traditional RAG systems and significantly improving retrieval quality.

[0027] In the implementation process, firstly, knowledge base documents (such as technical manuals or legal texts) are segmented. A natural language processing library is used to identify the document structure, dividing the document into text blocks of 512 tokens each. Each text block is assigned a globally unique identifier (generated using UUID4) and a sorting number (an integer sequence starting from 1), and relationships between text blocks are established (e.g., recording the IDs of predecessor and successor blocks). Based on this information, a structural index of the document is constructed, storing the text block content and metadata. Simultaneously, a vector index is built using the FAISS library. When a user submits a query request, the Sentence-BERT model is used to convert the query into a 768-dimensional vector. Cosine similarity is used to calculate the similarity with the text block vector, retrieving the top 5 most relevant text blocks as the initial set of relevant text blocks. Subsequently, based on the structural index and sorting number, a contextual expansion retrieval is performed on the initial set, obtaining the two preceding and following text blocks of each text block, forming an expanded set of contextual text blocks. The set is deduplicated (e.g., by detecting overlapping regions based on Jaccard similarity), merged (by connecting text content), and reordered (by sorting in ascending order of sorting number). Then, smooth transition techniques (such as adding transition sentences or adjusting sentence structure) are applied to generate natural and fluent text as the final search result.

[0028] In the above embodiments, preferably, the specific process of segmenting the knowledge base documents includes: Natural language processing techniques are used to identify semantic boundaries in documents, including paragraph boundaries, topic boundaries, and concept boundaries. Based on the principle of semantic integrity, strongly related content is preserved as a whole as a text block, thus achieving text segmentation.

[0029] During implementation, the spaCy library can be used to parse documents, identifying paragraph boundaries based on line breaks and indentation. LDA topic modeling can be used to calculate topic change points to identify topic boundaries, and entity recognition and dependency resolution can be used to identify concept boundaries. Based on the principle of semantic integrity, strongly related content (such as a complete paragraph or a concept description) is preserved as a single text block, avoiding truncation in the middle of sentences or concepts. The size of the text blocks is dynamically adjusted, but is typically kept between 500-1000 tokens to ensure semantic integrity. After segmentation, each text block can be stored as a JSON object, containing text content, location information, and boundary type.

[0030] In the above embodiments, preferably, a unique identifier and sorting number are assigned to each text block to establish the association between text blocks in order to save the document structure. The specific process includes: Each text block is assigned a globally unique identifier and sorting number according to its positional relationship and logical order in the original document. The text blocks are associated with their corresponding globally unique identifiers and sorting numbers for storage, thus establishing the association between the text blocks.

[0031] During implementation, each text block is assigned a globally unique identifier and an integer sorting number according to its logical order in the original document (e.g., from left to right, from top to bottom). The identifier can be generated using the UUID4 algorithm, and the sorting number can increment from 1. A SQL database is used to store text block metadata, including the text block ID, sorting number, predecessor text block ID, successor text block ID, and original document ID. Foreign key constraints are used to establish relationships between text blocks; for example, the predecessor text block ID points to the previous text block, and the successor text block ID points to the next text block, thus preserving the document structure.

[0032] In the above embodiments, preferably, a structural index of the document is constructed based on text blocks and their relationships. The specific process includes: Vectorize the text blocks and construct vector indices; Based on the relationships between text blocks, construct a structural index between corresponding vector indices; A fast query index for text blocks is built based on sorting sequence numbers, supporting queries for adjacent blocks of text blocks that are predecessors and successors of a text block.

[0033] During implementation, text blocks are vectorized using the Sentence-BERT model to generate 768-dimensional vectors and construct vector indexes, such as the IndexFlatIP index type from the FAISS library. Secondly, based on the relationships between text blocks, a graph-structured index can be constructed using the Neo4j graph database, where nodes represent text blocks (attributes include ID, sort number, and content), and edges represent relationships (e.g., NEXT relationships represent successor blocks, and PREVIOUS relationships represent predecessor blocks). Finally, a B-tree index is constructed based on the sort number, supporting fast lookups of adjacent text blocks by sort number, with a query time complexity of O(1).

[0034] In the above implementation, preferably, based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. The specific process includes: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results.

[0035] During implementation, user queries using natural language questions are converted into 768-dimensional vectors using the Sentence-BERT model. Based on the cosine similarity matching algorithm, the similarity score between the query vector and all text block vectors is calculated. This calculation can be performed in batches using the FAISS index to improve efficiency. The top K text blocks with the highest similarity (e.g., K=5, adjustable according to configuration) are identified and grouped into an initial set of relevant text blocks. The initial search results can be returned in JSON format, including the text block ID, similarity score, and content.

[0036] In the above implementation, preferably, a context-expanded retrieval is performed on the initial set of related text blocks based on the document structure and structure index to obtain an expanded set of contextual text blocks. The specific process includes: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of a text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the evaluation results of the information value of the text block. Based on the extended scope of the context, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form a set of contextual text blocks that enhance retrieval.

[0037] In the implementation process, firstly, based on the sorting number of each text block in the initial set of related text blocks, a B-tree index is used to quickly locate its predecessor and successor text blocks. Secondly, information entropy is used to calculate the entropy value of the word distribution in the text block, and the keyword density is determined by calculating the occurrence frequency of predefined keywords. The information value of each text block is evaluated based on the entropy value and keyword density; the higher the entropy value or the greater the keyword density, the higher the information value.

[0038] During implementation, the scope of context expansion can be dynamically determined based on the information value assessment results: for text blocks with high information value (e.g., entropy > 5.0 or keyword density > 0.1), expand by 3 text blocks before and after expansion; for text blocks with low information value, expand by 1 text block before and after expansion. Finally, based on the expansion scope, retrieve the preceding text blocks from the initial set of relevant text blocks and the following text blocks from the future, and merge all text blocks into a list to form a set of contextual text blocks for enhanced retrieval.

[0039] In the above embodiments, preferably, the set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the retrieval result. The specific process includes: Identify overlapping text blocks in the preceding and following text blocks, and merge the overlapping text blocks. The search results are reordered based on the sorting number of the text blocks in the preceding and following text block set to restore the original logical order of the documents; The reordered text blocks are smoothed using a smooth transition technique, and the processed natural text is used as the search result.

[0040] During implementation, overlapping text blocks are identified, and the Jaccard similarity of the text block content is calculated based on the word set. If the similarity exceeds a preset threshold (e.g., 30%), the overlapping parts are merged, retaining only the unique content. Secondly, the search results are reordered based on the sorting number of the text blocks, arranged in ascending order to restore the original document's logical order. Finally, smooth transition techniques are applied, such as adding conjunctions (e.g., "in addition," "however") between text blocks or adjusting sentence structure (using a text generation model like GPT-2 for fine-tuning), to ensure the merged text flows smoothly and naturally.

[0041] In the above embodiments, preferably, natural language processing technology is used to identify the user's query intent and key semantic entities in response to the user's query request; Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request; The expanded query request is converted into a vector for initial retrieval.

[0042] During implementation, a BERT-based classifier can be used to identify the user's query intent, and a NER model can be used to identify key entities, such as names, place names, and technical terms. Based on the identified query intent and key entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request. For example, synonyms can be added, such as expanding "car" to "vehicle," or relevant terms or extended descriptions can be added. The expanded query request is then converted into a vector for initial retrieval, improving retrieval relevance.

[0043] This invention also proposes a context-aware retrieval enhancement generation system, which applies the context-aware retrieval enhancement generation method disclosed in any of the above embodiments, including: The text block preprocessing module is used to divide the knowledge base document into blocks, assign a unique identifier and sorting number to each text block, and establish the relationship between text blocks to save the document structure. The text structure indexing module is used to build a structural index of a document based on text blocks and their relationships. The initial retrieval module is used to perform an initial retrieval of the query content based on vector similarity according to the user's query request, and obtain the initial set of relevant text blocks with the highest relevance. The contextual expansion retrieval module is used to perform contextual expansion retrieval on the initial set of related text blocks based on the document structure and structure index, so as to obtain an expanded set of contextual text blocks. The search results optimization module is used to deduplicate, merge, and reorder the set of preceding and following text blocks, and uses a smooth transition technique to obtain natural text as the search results.

[0044] In the above embodiments, preferably, the initial retrieval module is specifically used for: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results; The front and rear extended search modules are specifically used for: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of a text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the evaluation results of the information value of the text block. Based on the extended scope of the context, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form a set of contextual text blocks that enhance retrieval.

[0045] The functions to be implemented by each module of the context-aware retrieval enhancement generation system disclosed in the above embodiments correspond to the steps of the context-aware retrieval enhancement generation method disclosed in the above embodiments. In the implementation process, the above embodiments are referred to for operation, and will not be repeated here.

[0046] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A context-aware retrieval enhancement generation method, characterized in that, include: The knowledge base documents are divided into blocks, and each text block is assigned a unique identifier and sorting number. The relationship between text blocks is established to save the document structure. A structural index of the document is constructed based on the text blocks and their relationships. Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. Based on the document structure and the structure index, a context-extended retrieval is performed on the initial set of related text blocks to obtain an extended set of preceding and following text blocks; The set of preceding and following text blocks is deduplicated, merged, and reordered, and a smooth transition technique is used to obtain natural text as the retrieval result.

2. The context-aware retrieval enhancement generation method according to claim 1, characterized in that, The specific process of segmenting knowledge base documents includes: Natural language processing techniques are used to identify semantic boundaries in the document, including paragraph boundaries, topic boundaries, and concept boundaries. Based on the principle of semantic integrity, strongly related content is preserved as a whole as a text block, thus achieving text segmentation.

3. The context-aware retrieval enhancement generation method according to claim 1, characterized in that, Assign a unique identifier and sort number to each text block, and establish the association between text blocks to save the document structure. The specific process includes: Each text block is assigned a globally unique identifier and sorting number according to its positional relationship and logical order in the original document. The text blocks are associated with and stored together with their corresponding globally unique identifiers and sorting numbers to establish the association between the text blocks.

4. The context-aware retrieval enhancement generation method according to claim 3, characterized in that, The document's structural index is constructed based on the text blocks and their relationships. The specific process includes: The text block is vectorized and a vector index is constructed. Based on the relationships between the text blocks, a structural index is constructed between the corresponding vector indices; A fast query index is constructed based on the sorting sequence number of the text block, which supports querying adjacent blocks of the preceding and succeeding text blocks of the text block.

5. The context-aware retrieval enhancement generation method according to claim 4, characterized in that, Based on the user's query request, an initial retrieval of the query content is performed based on vector similarity to obtain an initial set of relevant text blocks with the highest relevance. The specific process includes: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results.

6. The context-aware retrieval enhancement generation method according to claim 5, characterized in that, Based on the document structure and the structure index, a context-extended retrieval is performed on the initial set of related text blocks to obtain an expanded set of contextual text blocks. The specific process includes: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block. Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.

7. The context-aware retrieval enhancement generation method according to claim 6, characterized in that, The aforementioned set of preceding and following text blocks undergoes deduplication, merging, and reordering. A smooth transition technique is then used to obtain natural text as the retrieval result. The specific process includes: Identify overlapping text blocks in the set of preceding and following text blocks, and merge the overlapping text blocks. The search results are reordered based on the sorting number of the text blocks in the preceding and following text block set to restore the original logical order of the documents; The reordered text blocks are smoothed using a smooth transition technique, and the processed natural text is used as the search result.

8. The context-aware retrieval enhancement generation method according to claim 5, characterized in that, To address user queries, natural language processing techniques are used to identify the user's query intent and key semantic entities. Based on the query intent and key semantic entities, relevant concepts or synonyms are retrieved from the knowledge base to expand the query request; The expanded query request is converted into a vector for initial retrieval.

9. A context-aware retrieval enhancement generation system, characterized in that, The context-aware retrieval enhancement generation method as described in any one of claims 1 to 8 includes: The text block preprocessing module is used to divide the knowledge base document into blocks, assign a unique identifier and sorting number to each text block, and establish the relationship between text blocks to save the document structure. A text structure indexing module is used to construct a structural index of the document based on the text blocks and their relationships. The initial retrieval module is used to perform an initial retrieval of the query content based on vector similarity according to the user's query request, and obtain the initial set of relevant text blocks with the highest relevance. The context-extended retrieval module is used to perform context-extended retrieval on the initial set of related text blocks based on the document structure and the structure index, so as to obtain an extended set of context-extended text blocks; The search result optimization module is used to perform deduplication, merging and reordering of the set of preceding and following text blocks, and to obtain natural text as the search result using a smooth transition technique.

10. The context-aware retrieval enhancement generation system according to claim 9, characterized in that, The initial retrieval module is specifically used for: The query request is converted into a vector, and the similarity between the query request and the text block is calculated based on the vector similarity matching algorithm. Identify the top preset number of text blocks with the highest similarity and combine them to obtain an initial set of related text blocks as the initial search results; The aforementioned front-to-back extended retrieval module is specifically used for: Based on the sorting number of the text blocks in the initial set of related text blocks, and based on the document structure and the structure index, the sort_order-based indexing system is used to locate the text blocks before and after them. The information value of the text block is evaluated using information entropy and keyword density, and the scope of context extension is dynamically determined based on the information value evaluation results of the text block. Based on the aforementioned extended context range, the preceding text block is obtained forward and the succeeding text block is obtained backward based on the initial set of related text blocks; The initial set of relevant text blocks is merged with the preceding and succeeding text blocks to form the set of preceding and succeeding text blocks for enhanced retrieval.

Images