Hierarchical catalog-based progressive document exploration retrieval enhancement generation system and method

Abstract

The application provides a hierarchical directory-based progressive document exploration retrieval enhancement generation system and method, and relates to the technical field of artificial intelligence and natural language processing. With a directory tree as a navigation framework, an exploration-type Agent as a decision core, and L1 directory level, L2 document level and L3 segment level three-level indexes as an execution basis, through directory summary generation, document global semantic fingerprint construction, segment-level hybrid retrieval, branch pruning and drilling threshold control, range locking and abnormal backtracking mechanism, the Root-L1-L2-L3 layer-by-layer search space is reduced and the interpretable path output is realized. Compared with the existing plane RAG and general two-level retrieval scheme, the application greatly reduces the retrieval time delay in the knowledge base test, improves the accuracy, significantly reduces the local update cost, and is suitable for enterprise knowledge base, regulation document library, product manual library and other scenarios requiring structured exploration retrieval.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular, to the field of large language model retrieval enhancement generation (RAG) technology. Specifically, it relates to a progressive document exploration retrieval enhancement generation system and method based on hierarchical directories. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is one of the main technical approaches to solving the illusion problem of large language models, introducing external timely professional knowledge, and improving the accuracy of question answering. It has become the basic architecture for the application of artificial intelligence in knowledge base question answering, intelligent customer service, and document retrieval scenarios.

[0003] The general technical process of existing conventional RAG systems is as follows: First, massive documents are uniformly sliced ​​according to a fixed character length to obtain standardized text blocks; then, all text blocks are vectorized and encoded using an embedding model, and stored uniformly in a vector database; when a user query is received, the query statement is encoded into a vector, and a full similarity search is performed in the globally unified text block vector space to recall the top K candidate text segments with the highest similarity; finally, the candidate segments are concatenated with the user query and input into a large language model to generate the final question-and-answer answer. This solution is simple to implement, has a low deployment threshold, and has basic usability in small-scale, non-hierarchical fragmented document scenarios.

[0004] As enterprises advance their digital transformation, knowledge bases exhibit four major characteristics: massive scale, cross-domain nature, rigid structural hierarchy, and frequent version iterations. Traditional flat RAGs and their derivative improvement solutions are gradually revealing serious technical shortcomings. Currently, there are no unified national or industry standards that define the retrieval process, hierarchical index construction, and path exploration mechanism in the RAG field. Industry practices typically regard RAGs only as a general framework of "retrieval + generation." Relevant standard documents such as ISO / IEC 22989:2022 and NIST AI RMF 1.0 (2023-01) mainly provide artificial intelligence terminology, risk governance, and system management principles, but do not disclose a directory tree-driven progressive retrieval path control scheme.

[0005] Existing academic research mainly focuses on three major areas: retrieval model optimization, multi-hop link design, and graph structure retrieval, as shown in Table 1:

[0006] Table 1 International Papers and Public Research

[0007]

[0008] In Table 1, the Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks published at NeurIPS 2020 proposes a classic planar RAG framework, establishing a basic paradigm for retrieval and generation, and using retrieval results as an external memory fusion generation model. However, the retrieval object is a set of undifferentiated and uniform text blocks, completely ignoring the inherent directory hierarchy structure of the document.

[0009] The Dense Passage Retrieval for Open-Domain Question Answering published at EMNLP2020 constructs a dual-tower dense retrieval underlying architecture, which has become the basis for vector recall in most RAG systems. However, it is only a single-stage global vector retrieval and fails to solve the problems of document structure preservation and hierarchical range locking.

[0010] The ICLR2021 paper, "Answering Complex Open-Domain Questions with Multi-HopDense Retrieval," improves complex question-answering capabilities through query chain rewriting and multi-step dense retrieval, and optimizes the design of retrieval path length. However, it only focuses on query chain iteration and does not rely on the document directory tree to build a retrieval skeleton with layer-by-layer pruning.

[0011] ICLR2020's Learning to Retrieve Reasoning Pathsover Wikipedia Graph for Question Answering relies on the hyperlink graph structure of Wikipedia to achieve reasoning path retrieval and is adapted to entity-related question answering. However, it depends on the relationship between web page hyperlinks and entity edges and cannot be adapted to the inherent directory hierarchy management system of enterprise documents.

[0012] The Dense Hierarchical Retrieval for Open-Domain Question Answering proposed by Findings of EMNLP 2021 is a two-level dense retrieval approach at the document and paragraph levels. It has basic hierarchical sorting capabilities, but the hierarchy is only used as an auxiliary means of retrieval and sorting. It does not treat directory nodes as independent navigation units and lacks directory exploration agents, abnormal rollback and incremental mounting mechanisms.

[0013] A search of the patent publication website of the China National Intellectual Property Administration (search date: May 8, 2026) revealed that domestic RAG-related patents are concentrated in iterative retrieval, two-level indexing, and multi-source knowledge fusion, and none have broken through the incremental retrieval architecture driven by the directory tree, as shown in Table 2.

[0014] Table 2

[0015]

[0016] In Table 2, the patent for the retrieval enhancement generation method with application number 202511815269X and publication number CN121979968A uses attention feedback to update the query vector and achieve iterative optimization of retrieval, generation, and re-retrieval. However, it does not construct a directory tree hierarchical structure and cannot achieve layer-by-layer retrieval range locking.

[0017] The patent for the retrieval enhancement generation device with application number 2025118759768 and publication number CN121980003A sets up a two-level retrieval system with chapter vector library and sub-block vector library, which has a preliminary hierarchical idea, but it is a fixed two-level index calling mode and lacks the ability to intelligently select directory nodes, explore branches in parallel, fall back in case of anomalies and output a complete tracing path.

[0018] The multi-source knowledge fusion RAG patent with application number 2025119112134 and publication number CN121996750A introduces a long-term memory mechanism and dynamic Top-K adaptive retrieval to enhance the multi-source knowledge fusion capability, but does not use the directory hierarchy as the core skeleton of retrieval navigation and does not have a local incremental update rule for directory branches.

[0019] Existing industry standards and normative documents only define basic terms and provide basic specifications, without addressing engineering design implementation mechanisms, as shown in Table 3:

[0020] Table 3

[0021]

[0022] Table 3 shows that the ISO / IEC 22989:2022 standard on concepts and terminology for artificial intelligence only defines basic terms for artificial intelligence and does not cover the construction of RAG hierarchical indexes or the design of directory-driven retrieval processes.

[0023] The NISTAIRMF1.0 AI Risk Management Framework (released in January 2023) only stipulates the principles for AI system risk governance and management, without disclosing the implementation mechanisms of directory tree retrieval, three-level vector indexing, and exploratory agent decision-making.

[0024] While existing RAG technology has evolved from traditional planar Top-K retrieval to multi-hop dense retrieval, graph path retrieval, and document-paragraph two-level retrieval, the improvement ideas are still limited to the following directions:

[0025] 1. Only optimize the recall model: The solution represented by DPR mainly improves the encoder and vector recall effect, but still searches within a unified set of text blocks and does not utilize the document's table of contents and chapter hierarchy structure;

[0026] 2. Only optimize the query link: For example, multi-hop retrieval makes up for the shortcomings of a single retrieval by rewriting the query and recalling it in multiple rounds, but it relies on manually designing the query link and lacks a stable structured retrieval framework. Complex queries are prone to link breakage.

[0027] 3. Only performing sorting-level optimization: such as a two-level retrieval scheme from document level to paragraph level, which is essentially still a coarse sort followed by a fine sort, without treating directory nodes as exploration units that can be explicitly output, rolled back, and pruned.

[0028] 4. Only enhance knowledge source fusion: Long-term memory and multi-source fusion solutions focus on expanding the number of knowledge sources and adaptive recall strategies, but neglect directory tree path tracing, local update boundaries and structural interpretability.

[0029] In summary, existing technologies all treat structural information (document directory structure) as metadata filtering conditions, rather than as an active navigation object in the agent decision-making process. Therefore, it is difficult to simultaneously meet the requirements of large-scale retrieval efficiency, complex exploratory question answering, path interpretability and traceability, and low-cost incremental updates.

[0030] Especially when dealing with massive amounts of data and complex queries, existing RAG technology has the following obvious shortcomings:

[0031] 1. Lack of structural cognition and loss of contextual information: Many solutions directly insert documents into the database after uniformly slicing them, which destroys the original table of contents, chapters, versions and attribution relationships of the documents. In cross-project and cross-version knowledge base searches, it is easy to cause problems such as similar titles and misattribution, and the question-and-answer context logic is broken.

[0032] 2. It is difficult to balance retrieval accuracy and operational efficiency: As the document size increases, the candidate set in the unified vector space grows nearly linearly, and the number of noisy segments increases. The system has to passively compromise between recall and latency, making it difficult to achieve the dual goals of high accuracy and low latency.

[0033] 3. Insufficient support for complex exploratory queries: Although the existing solution can perform multi-hop or multi-round recall, it lacks an explicit exploratory mechanism that first looks at the major category, then selects the sub-category, and then locates the document. It is not capable of supporting complex queries with vague user intent or unclear expression.

[0034] 4. Unclear scope of incremental updates: After adding or modifying documents or adding a new directory, many existing solutions still need to rebuild most of the index. There is a lack of clear boundaries and triggering rules for updating only the relevant branches. Updates can take up to several hours, which cannot meet the business needs of high-frequency iteration and real-time updates of enterprise knowledge bases.

[0035] 5. Insufficient interpretability of results: Most traditional RAGs only output the final Q&A answers and reference fragments, making it difficult to output the step-by-step decision-making path from the table of contents and documents to fragments. They cannot show the decision-making process and screening path of knowledge base retrieval, and it is difficult to trace the source of the answers, which is not conducive to enterprise auditing and compliance traceability. Summary of the Invention

[0036] In view of this, the purpose of this invention is to propose a progressive document exploration retrieval enhancement generation system and method based on a hierarchical directory. By constructing a three-level hierarchical index, introducing exploratory agent intelligent decision-making, designing a progressive drill-down retrieval process and a local incremental update mechanism, it achieves layer-by-layer shrinking of the retrieval space, accurate and efficient recall, traceable and auditable path, and low-cost local updates. This solves the problems of insufficient utilization of structural information, low retrieval efficiency, poor adaptability to complex queries, high incremental update costs, and uninterpretable results in existing technologies; and is adapted to the retrieval enhancement generation needs of large-scale enterprise-level structured knowledge bases.

[0037] This invention provides a progressive document exploration and retrieval enhancement generation system based on a hierarchical directory, including a user query analysis layer, a knowledge hybrid index layer, a directory exploration engine Agent layer, and a document in-depth reading and answer generation layer;

[0038] The user query analysis layer is used to receive user query statements, complete query intent recognition, complexity assessment, keyword extraction, directory constraint recognition and exploration strategy generation, and transmit the processed query data to the directory exploration engine Agent layer.

[0039] The knowledge hybrid index layer is used to construct and maintain a three-level hierarchical vector index based on the original knowledge base directory tree, consisting of L1 directory-level index, L2 document-level index, and L3 fragment-level index. At the same time, it stores and maintains the directory tree structure, parent-child relationships, and version information through a graph database or relational hierarchical table to provide structured index support for retrieval.

[0040] The directory exploration engine Agent layer has a built-in preset quantitative threshold system and relevance scoring algorithm. It is used to receive the processed query data, evaluate each candidate branch from top to bottom, and intelligently execute branch pruning, parallel exploration, hierarchical drill-down, search range locking and abnormal rollback decisions to gradually shrink the search space.

[0041] The document reading and answer generation layer is used to perform fragment-level hybrid retrieval within the limited retrieval range locked by the Agent layer, recall highly relevant text fragments, generate accurate answers by combining large language models, and simultaneously output the complete exploration path and evidence tracing chain from the root directory to the target fragment.

[0042] Furthermore, each directory node in the L1 directory-level index of the knowledge hybrid index layer is configured with at least the following: a unique directory identifier, a parent directory identifier, a directory name, a complete path from the root to the current directory, a list of direct child documents, a list of subdirectories, a directory summary text, a set of directory keywords, a directory vector identifier, and a version stamp field.

[0043] The L1 directory summary generation steps are as follows: collect the titles of direct documents, first-level headings, document summaries, tags, and high-frequency words from historical queries under the directory. After deduplication, stop word filtering, and keyword extraction, a 200-400 character directory summary is generated through a large language model. 8-15 directory keywords are extracted. The directory name, summary, and keywords are then concatenated and an embedding model is used to generate a 1024-dimensional L1 directory vector.

[0044] Furthermore, each document node in the L2 document-level index of the knowledge hybrid index layer is configured with at least the following: unique document identifier, L1 directory identifier, document title, document storage path, document type, document version, document timestamp, global document summary, document keyword set, number of subordinate L3 segments, subordinate segment identifier set, document vector identifier, and content hash field;

[0045] The L2 document vector is generated using a weighted average normalization method, with the formula V_doc = Norm( Σ w_j× e_j / Σ w_j ): All L3 segment vectors in the document are assigned aggregate weights and summed, then normalized after dividing by the total weights. Here, w_j represents the aggregate weight of the j-th segment, e_j represents the j-th L3 segment vector, and Norm represents the normalization operation. The aggregate weight consists of a base weight of 1 plus a title hit component of 0.4, a keyword coverage component of 0.3, and a position weight component of 0.3, i.e., w_j = 1 + 0.4 × h_j + 0.3 × k_j + 0.3 × p_j, where h_j indicates whether the segment hits a title or subheading, k_j represents the segment's keyword coverage, and p_j` represents the segment's position weight.

[0046] Furthermore, each fragment node in the L3 fragment-level index of the knowledge hybrid index layer is configured with at least the following: fragment unique identifier, document identifier, directory identifier, chapter path, fragment original text, fragment keywords, fragment character length, fragment vector identifier, and text hash field;

[0047] The L3 text segmentation rules are as follows: priority is given to segmenting according to the original title, chapter, and section boundaries; extremely long chapters are segmented a second time by 200-500 Chinese characters; adjacent segments are set to overlap by 50-100 characters; tables, code blocks, and bulleted lists are kept intact and not split.

[0048] Furthermore, the directory exploration engine Agent layer has a built-in preset quantitative threshold system, including: pruning threshold 0.6, drill-down threshold 0.8, document locking threshold 50, fragment locking threshold 200, root node preferred branch number 10, parallel exploration branch limit 3, fragment final recall number 5-20, and maximum exploration depth 3.

[0049] The maximum exploration depth is limited to a three-level path: Root → L1 → L2 → L3. Physical directories with more than three levels are folded and mapped to L1 or L2 nodes to maintain the decision depth.

[0050] Furthermore, the relevance scoring algorithm of the directory exploration engine Agent layer includes semantic similarity calculation, metadata matching degree calculation, and comprehensive score fusion;

[0051] Semantic similarity is calculated using cosine similarity to determine the matching degree between the query vector and the candidate node vector. Metadata matching score M_i is linearly weighted and fused with the following weights: title hit rate t_i (0.4 weight), keyword hit rate k_i (0.3 weight), path hit rate p_i (0.2 weight), and time / version relevance r_i (0.1 weight). The formula is: M_i = 0.4 × t_i + 0.3 × k_i + 0.2 × p_i + 0.1 × r_i, where t_i, k_i, p_i, and r_i all range from [0,1]. The combined score for directory-level and document-level scores is linearly weighted with semantic similarity (0.7) and metadata matching score (0.3). The score for fragment-level scores is linearly weighted with chunk_score_j = 0.8 × semantic_sim_j + 0.2 × bm25_j, where semantic_sim_j is the cosine similarity between the query and the j-th fragment, and bm25_j is the cosine similarity between the query and the j-th fragment. The keyword matching score for each segment, where chunk_score_j is the segment-level score.

[0052] This invention also provides a progressive document exploration and retrieval enhancement generation method based on hierarchical directories, applied to the progressive document exploration and retrieval enhancement generation system based on hierarchical directories as described above, comprising the following steps:

[0053] Step 1: Query parsing and initial positioning. Receive user queries and encode them into query vectors. Extract query keywords, entity words, version time words, and directory constraint words. Search L1 directory child nodes under the root node and return the Top 10 candidate directory branches.

[0054] Step 2: Relevance assessment and branch pruning. Calculate the comprehensive score of each candidate directory. Branches below the pruning threshold of 0.6 are stored in the waiting set and pruned. Branches above 0.6 are included in the drill-down queue. A maximum of 3 high-scoring branches with a score difference ≤ 0.03 are selected for parallel exploration.

[0055] Step 3: Depth control and hierarchical drill-down. Before reaching the maximum exploration depth of 3, traverse the L2 document-level index associated with the drill-down queue branches, calculate the comprehensive document score, and if the high-score branch meets the drill-down threshold of 0.8, continue to explore the lower level. If the threshold is not reached, stop drilling and start range locking judgment.

[0056] Step 4: Scope locking and fragment retrieval. When the number of candidate documents is ≤50 or the number of candidate fragments is ≤200, the retrieval scope is locked. The L3 fragment-level index is only called within the restricted directory / document branch. The top 5-20 highly relevant fragments are recalled by sorting the fragments by their mixed scores. If the number of fragments is insufficient, the high-scoring branches are restored from the waiting set to expand the search scope.

[0057] Step 5: Answer generation and path output. The recall fragment and the original query input are used to generate the module. At the same time, the module outputs the final answer and generates the complete tracing path from Root to L1 directory to L2 document to L3 fragment. The scores of each level of candidates, pruning and drill-down decision basis are recorded.

[0058] Step 6: Anomaly handling and rollback adaptation. For four types of anomaly scenarios, namely no candidate hit, high-scoring branch conflict, drill-down without effective segments, and directory constraints and semantic conflicts, respectively, the following adaptation strategies are implemented: threshold reduction, parallel branch synchronous exploration, upper-level branch rollback, and dual-path result output.

[0059] Specifically, the abnormal adaptation strategy for no candidate hits is as follows: when the score of no branch under the root node reaches 0.6, the number of preferred branches of the root node is expanded from 10 to 15, and the pruning threshold is lowered by 0.05 at one time before re-searching.

[0060] The adaptation strategy for high-scoring branch conflicts is as follows: when multiple branches have similar scores and semantic conflicts, multiple branches are explored in parallel, and the final decision is made by waiting for the L2 or L3 retrieval results. No single selection is forced at the L1 level.

[0061] Furthermore, the progressive document exploration and retrieval enhancement generation method based on hierarchical directory also includes a local incremental update mechanism, specifically including four types of update rules: adding documents, modifying documents, adding directories, and deleting documents.

[0062] When a new document is added, only the corresponding L2 document index and its subordinate L3 fragment index are generated, the L1 directory to which it belongs is bound, and the summary and vector of that directory are updated incrementally, without modifying other branch indexes;

[0063] When the document content modification rate is ≥20%, rebuild the document's L2 vector and all its subordinate L3 segment vectors, and synchronously and incrementally update the L1 directory index.

[0064] When adding a new directory, only the L1 directory node is created and a summary and vector are generated. If the directory has no subordinate documents, the L2 and L3 index updates are not triggered.

[0065] When a document is deleted, the corresponding L2 and L3 nodes are removed, and the summary and vector of the L1 directory to which it belongs are updated incrementally. When only the metadata changes and the content modification ratio is less than 20%, only the metadata fields are updated and the fragment vector is not rebuilt.

[0066] This invention retains the original knowledge base's directory structure, uses directory nodes as retrieval and navigation units, and combines the decision-making capabilities of an exploratory agent to progressively narrow the retrieval scope from top to bottom. It performs fragment-level precise retrieval and answer generation within a limited context, overcoming the shortcomings of existing planar RAGs and existing multi-hop, two-level retrieval technologies.

[0067] This invention does not merely improve the recall algorithm or query rewriting level, but rather designs a unified scheme that integrates structure and reasoning. It uses an explicit directory tree as the retrieval navigation skeleton, an exploratory agent as the focus of path decision-making, and a three-level index as the execution foundation. During the retrieval process, it performs layer-by-layer pruning, drill-down, range locking, fragment refinement, and path output. Compared to existing two-level or multi-hop retrieval methods, this invention differs in that:

[0068] 1. Elevate the directory hierarchy from "passive metadata" to "actively explored objects";

[0069] 2. Incorporate directory-level, document-level, and fragment-level indexes into a unified decision chain;

[0070] 3. Provide explicit quantization thresholds, parallel exploration strategies, anomaly rollback rules, and local update rules;

[0071] 4. Output the interpretable tracing path from Root to L1 to L2 to L3 along with the answer.

[0072] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the progressive document exploration and retrieval enhancement generation method based on hierarchical directories as described above.

[0073] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the progressive document exploration retrieval enhancement generation method based on hierarchical directory as described above.

[0074] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0075] The progressive document exploration and retrieval enhancement generation system and method based on hierarchical directory provided by this invention relies on a hierarchical directory tree to construct a three-level index. Through top-down pruning by an agent, the retrieval space is shrunk layer by layer. The L1 level filters 90% of irrelevant directory branches, the L2 level further filters 80% of irrelevant documents, and finally the L3 fine-check only needs to traverse 2%-3% of the fragment data in the entire database. The average retrieval latency for 100,000 documents is as low as 98ms, and the extrapolation latency for millions of documents is only 230ms, which is 13 times lower than the latency of traditional planar RAG, solving the problems of high latency and slow response in large-scale knowledge base retrieval. This invention integrates semantic similarity, metadata matching, and BM25 keyword multi-scoring, relying on directory hierarchy constraints to avoid cross-domain false recalls; it improves accuracy, increases the hit rate of complex exploratory question answering, and reduces cross-domain document interference rate; it effectively avoids the problems of similar titles but incorrect attribution, contextual fragmentation, and generation illusions in traditional RAG, making it suitable for high-precision question answering needs of professional knowledge bases; this invention can completely output the hierarchical tracing path from Root to L1 directory to L2 document to L3 fragment, recording the score of each candidate branch, pruning basis, and drill-down decision reasons, forming a complete evidence tracing chain; it breaks through the limitation of traditional RAG only outputting answers without source evidence, and can meet the hard requirements of compliance auditing, responsibility tracing, and fault tracing in industries such as finance, government, and state-owned enterprises. This invention establishes standardized local incremental update rules. When adding, modifying, or deleting documents or directories, only the affected branch index is updated, without global reconstruction, reducing the update cost by hundreds to thousands of times compared to traditional RAG, and significantly reducing the difficulty of long-term operation and maintenance and computing power consumption of large-scale knowledge bases. This invention supports progressive exploratory retrieval of users' ambiguous intent and complex queries through exploratory agent parallel exploration, anomaly rollback, and threshold adaptive mechanisms. It breaks through the limitation of existing RAGs that only support precise question answering, simulates the thinking logic of manually browsing catalogs, documents, and fragments, and conforms to the real-world usage scenarios of enterprise employees browsing system regulations, project documents, and product manuals. It is suitable for application scenarios such as large-scale enterprise-level knowledge bases, intelligent question answering systems, complex structured document exploration and retrieval, compliance audit tracing, and knowledge incremental update services. Attached Figure Description

[0076] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0077] In the attached diagram:

[0078] Figure 1This is an example of the overall framework and hierarchical processing flowchart of the progressive document exploration and retrieval enhancement generation system based on hierarchical directory according to an embodiment of the present invention.

[0079] Figure 2 This is a schematic diagram of the three-level hierarchical index tree and the directory-document-fragment mapping relationship structure according to an embodiment of the present invention;

[0080] Figure 3 This is a flowchart illustrating the decision-making process and drill-down strategy of the directory exploration engine agent in this embodiment of the invention.

[0081] Figure 4 This is a schematic diagram comparing the traditional planar RAG full-database single search and the HDR-RAG progressive catalog search of this invention.

[0082] Figure 5 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation

[0083] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and products consistent with some aspects of this disclosure as detailed in the appended claims.

[0084] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0085] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0086] The embodiments of the present invention will be described in further detail below.

[0087] Example

[0088] This invention provides a progressive document exploration and retrieval enhancement generation system based on a hierarchical directory, which mainly includes the following four hierarchical modules:

[0089] User query analysis layer: Receives user query statements and performs intent recognition, complexity assessment, keyword extraction, directory constraint recognition, and exploration strategy generation.

[0090] Knowledge Hybrid Index Layer: Constructs and maintains three types of vector indexes: L1, L2, and L3, while using graph databases or relational hierarchical tables to maintain the directory tree structure, parent-child relationships, and version information.

[0091] Directory exploration engine Agent layer: Based on threshold rules, evaluate each candidate branch layer by layer from top to bottom, and decide on pruning, parallel exploration, drill-down, locking the range or rolling back.

[0092] Document reading and answer generation layer: Performs fragment-level hybrid retrieval within the locked scope, outputting answers, referenced fragments, and complete exploration paths.

[0093] The following key threshold parameters for the Agent exploration engine are preferably set in the embodiments of the present invention:

[0094] `pruning_threshold = 0.6`: If the overall relevance score of a candidate branch is lower than 0.6, then the branch will be pruned.

[0095] `drill_threshold = 0.8`: If the score of the current best candidate branch is not lower than 0.8 and the deepest exploration level has not been reached, then drill down.

[0096] `lock_doc_threshold = 50`: If the number of candidate documents is narrowed down to no more than 50, the directory range is locked and fragment retrieval is initiated;

[0097] `lock_chunk_threshold = 200`: If the number of candidate segments is reduced to no more than 200, then proceed directly to L3 fine-tuning.

[0098] `top_branch_k = 10`: Select the top 10 candidate branches under the root node;

[0099] `parallel_branch_limit = 3`: When multiple high-scoring branches are close, a maximum of 3 branches will be explored in parallel;

[0100] `top_chunk_k = 5~20`: The optimal number of fragments recalled is 5 to 20, with a preferred value of 8;

[0101] `max_depth = 3`: The exploration depth is capped at 3 levels of the three-level index, avoiding infinite drilling that could increase system latency.

[0102] Here, `max_depth = 3` corresponds to the complete path Root→L1→L2→L3. For a very small number of implementations with more than 3 directory levels, deep physical directories can be folded and mapped to the same L1 or L2 node to keep the upper limit of decision depth unchanged.

[0103] The correlation score is calculated as follows:

[0104] (1) Calculate semantic similarity:

[0105] For the query vector `q` and any candidate node vector `v_i`, the semantic similarity is preferably calculated using cosine similarity, with the formula cos_sim(q, v_i) = (q · v_i) / (‖q‖ × ‖v_i‖), which is the vector dot product divided by the product of their magnitudes.

[0106] Where `q` is the vector representation of the user query;

[0107] `v_i` is a vector representation of the candidate directory, document, or fragment node;

[0108] The dot product term is used to measure directional consistency;

[0109] The modulus term is used to eliminate the influence of vector length.

[0110] (2) Calculate metadata matching degree:

[0111] To avoid false cross-domain recalls due to relying solely on semantic similarity, it is preferable to introduce metadata matching degree and fuse the following four items with fixed weights:

[0112] Title hit rate `t_i`, weight 0.4;

[0113] Keyword hit rate `k_i`, weight 0.3;

[0114] Path hit rate `p_i`, weight 0.2;

[0115] Time or version relevance `r_i`, weight 0.1.

[0116] The formula for calculating the metadata matching degree M_i is: M_i = 0.4 × t_i + 0.3 × k_i + 0.2 × p_i + 0.1 × r_i.

[0117] Where `t_i` ranges from [0,1];

[0118] `k_i` has a range of [0,1];

[0119] `p_i` is used to query the degree of consistency between the query and the directory path and the chapter path, with a range of [0,1].

[0120] `r_i` represents the time or version relevance, ranging from [0,1], used to prioritize recent or specified version documents.

[0121] (3) Calculate the comprehensive correlation score:

[0122] Preferably, the comprehensive score of directory-level or document-level candidate nodes is calculated using a linear weighting method of "semantic similarity 0.7 + metadata matching degree 0.3";

[0123] Here, 0.7 and 0.3 represent the weights of semantic similarity and metadata matching, respectively. This weight setting reflects the principle of "semantic dominance and metadata priority"; in other implementations, it can also be adaptively adjusted according to industry data characteristics, but it is preferable to keep the semantic similarity weight greater than the metadata weight.

[0124] (4) Calculate the fragment-level mixed retrieval score:

[0125] Once the directory or document range has been locked, it is preferable to perform mixed scoring on fragment-level candidates, using a linear weighting method of chunk_score_j = 0.8 × semantic_sim_j + 0.2 × bm25_j.

[0126] Where `semantic_sim_j` is the cosine similarity to the j-th segment;

[0127] `bm25_j` is the keyword matching score of the j-th segment, which takes a value in [0,1] after normalization.

[0128] This formula balances semantic recall with precise matching of key terms, making it suitable for text scenarios with a large number of terms and chapter headings, such as specifications, policy documents, and product manuals.

[0129] The main flow of the exploration and decision-making algorithm of this invention is as follows:

[0130] Algorithm input: User query `Q`, hierarchical index tree `T`, threshold parameter set `set(pruning_threshold=0.6, drill_threshold=0.8, lock_doc_threshold=50, lock_chunk_threshold=200, max_depth=3)`.

[0131] Algorithm output: a set of relevant text fragments `S`, a set of exploration paths `P`, and a set of path scores `R`.

[0132] Specifically, the following steps are included:

[0133] Step 1: Query parsing and initial location, including:

[0134] Encode the user query `Q` into the query vector `q`;

[0135] Extract query keywords, entity words, version words, time words, and directory constraint words;

[0136] The Agent performs a vector search on all L1 directory child nodes under the root node, preferably with a search range of no more than 50 root-level directories;

[0137] Return the Top-K candidate branches based on cosine similarity, and preferably return the top 10 candidate directories to form `candidates_L1`;

[0138] If the query explicitly contains directory qualifiers, the corresponding directory will be treated as a strong constraint, and the path hit rate `p_i` will be prioritized.

[0139] Step 2: Relevance assessment and pruning, including:

[0140] Calculate `score_i` for each candidate directory in `candidates_L1`;

[0141] When `score_i < 0.6`, the candidate is written to the temporary standby set `standby_branches` and pruning is performed;

[0142] When `score_i >= 0.6`, the candidate is retained in the drill-down queue `active_branches`;

[0143] If there are multiple candidate branches whose scores differ from the highest score by no more than 0.03, then select no more than 3 branches to explore in parallel to reduce the risk of early misjudgment.

[0144] `standby_branches` are retained until the end of this round of retrieval, in preparation for subsequent secondary exploration or rollback due to abnormalities.

[0145] Step 3: Depth control and drilling decisions, including:

[0146] If the current depth `depth < max_depth`, then call the direct L2 document-level index for each branch in `active_branches`;

[0147] Calculate the similarity between the query vector and each L2 document vector, and sort them according to the overall relevance score;

[0148] If the highest document score in a branch is >= 0.8, then continue drilling down along that branch;

[0149] If all L2 candidate scores are below 0.8 but still above 0.6, then stop drilling down to the current layer and trigger a range locking decision.

[0150] When `depth = max_depth`, further drilling is prohibited, and the process directly enters the range locking and retrieval stage.

[0151] Step 4: Range locking and fragment retrieval, including:

[0152] If the number of candidate documents currently retained does not exceed 50, or the number of candidate fragments currently does not exceed 200, then the retrieval space is considered to have been sufficiently shrunk.

[0153] Use the current directory path, document identifier, or a combination of both as filtering conditions to call the L3 fragment-level index only within a limited scope;

[0154] Perform a hybrid search on the filtered L3 fragment set, sorted by `chunk_score_j`;

[0155] Returning to the Top-K segment, the value of K is preferably between 5 and 20, and preferably 8;

[0156] If there are fewer than 3 fragments, the search scope is expanded again by recovering one or more of the highest-scoring branches from `standby_branches`.

[0157] Step 5: Answer generation and path output, including:

[0158] Input the Top-K fragments and the original query into the answer generation module;

[0159] While generating the final answer, output the complete path: `Root → L1 directory → L2 document → L3 fragment`;

[0160] The path records the candidate score, pruning reason, drill-down reason, and final adoption reason for each level;

[0161] If multiple parallel branches exist, the scores of the main path and alternative paths are output synchronously for manual review or system log recording.

[0162] Step 6: Exception handling and rollback mechanism, including:

[0163] No candidate hit: If the score of no candidate hit under the root node reaches 0.6, then expand `top_branch_k` from 10 to 15, and reduce `pruning_threshold` by 0.05 at once before retrying;

[0164] High-scoring branch conflict: If multiple branches have similar scores but conflict with each other semantically, they should be explored in parallel first and wait for the L2 or L3 layer results to make a decision, instead of forcing a single selection at the L1 layer.

[0165] No valid fragments after drilling down: If enough valid fragments are not obtained after drilling down to L2 or L3, then backtrack to the previous level and resume the exploration on the highest-scoring branch in `standby_branches`;

[0166] Directory constraints and semantic conflicts: If the directory constraints and semantic similarity in the query text are obviously contradictory, then two sets of candidates, "results by directory constraints" and "results by semantic optimality", will be returned simultaneously for further confirmation by the user or the upper-level system.

[0167] Local incremental update mechanisms include:

[0168] 1. Update scope and rules for newly added documents: When a new document is detected, only the following partial updates will be performed:

[0169] (1) Generate L2 document-level summaries, keywords, and document vectors for new documents;

[0170] (2) Bind the document to its corresponding `directory_id` according to the directory mounting rules;

[0171] (3) Segment the newly added document and generate the corresponding L3 fragment and fragment vector;

[0172] (4) Only update the summary and directory vector of the L1 directory node;

[0173] (5) It does not affect the L2 or L3 indexes of other directory branches or other documents.

[0174] Document update scope and rules: When document content modification is detected, a partial update is preferably triggered based on the modification percentage, specifically including:

[0175] (1) Calculate the difference in hash values ​​of the document content before and after modification;

[0176] (2) If the content modification rate is not less than 20%, then regenerate the L2 vector of the document and the L3 segment vectors of the document.

[0177] (3) Simultaneously and incrementally update the L1 directory summary and directory vector of its parent directory;

[0178] (4) Do not rebuild other directories or other documents.

[0179] Update scope and rules for new directories: When a new directory is detected, only the corresponding L1 directory node is added; an L1 summary, keywords and directory vector are generated for the first time for the directory; if there are no documents in the directory, L2 and L3 updates are not triggered; the corresponding L2 and L3 indexes are generated when documents are mounted later.

[0180] Preferably, the system triggers a partial update under the following conditions, specifically as follows:

[0181] Adding a new document triggers a partial update of the new document;

[0182] If the document content modification rate is ≥20%, it will trigger the L2 / L3 reconstruction of the corresponding document and the incremental update of its corresponding L1 level.

[0183] Adding a new directory triggers incremental mounting of the L1 directory-level index;

[0184] Delete a document, remove the corresponding node from its L2 / L3 set, and incrementally update the L1 directory summary.

[0185] Metadata changes only and content change rate < 20%: Only update metadata fields and necessary matching indexes, without rebuilding the entire L3 fragment vector.

[0186] Specifically, in this embodiment of the invention, a three-level hierarchical index structure is constructed: based on the original directory tree, a three-level index system is constructed, which includes L1 directory-level index, L2 document-level index and L3 fragment-level index. The three are physically separated, logically related, and linked by path.

[0187] The three-level vector construction rules have been refined: different text summarization, keyword extraction, and vectorization rules are used for directories, documents, and fragments respectively, in order to adapt to the needs of coarse-grained semantic expression and fine-grained matching at different levels.

[0188] An exploratory agent decision engine was introduced: at each layer, the agent decides whether to continue drilling, explore in parallel, lock the range, or backtrack and expand the range based on relevance scores, pruning thresholds, drill-down thresholds, and maximum depth limits.

[0189] Implement scope limitation and hybrid search: After the directory scope is locked, only documents and fragments under that branch are searched. Fragment-level search adopts the "vector similarity + keyword matching" fusion rule.

[0190] Output complete exploration path and evidence chain: When generating the answer, output the path information from the root directory to the target segment, the hierarchical score and the decision basis simultaneously to improve interpretability and auditability.

[0191] A local incremental update mechanism was established: trigger conditions and local update boundaries were defined for new documents, modified documents, and new directories, and only the affected L1 / L2 / L3 nodes were updated without global reconstruction.

[0192] The specific three-level hierarchical index structure in this embodiment is as follows:

[0193] L1 directory-level index: The L1 index corresponds to a directory node or topic node in the directory tree and is used to perform coarse-grained range filtering. Each L1 node contains at least the following fields:

[0194] `directory_id`: A unique identifier for the directory;

[0195] `parent_directory_id`: Parent directory identifier;

[0196] `directory_name`: Directory name;

[0197] `directory_path`: The complete path from the root node to the current directory;

[0198] `child_document_ids`: List of directly related L2 documents;

[0199] `child_directory_ids`: List of subdirectories;

[0200] `summary_text`: Summary text of the table of contents;

[0201] `keywords`: A collection of keywords for the directory;

[0202] `vector_id`: Directory vector identifier;

[0203] `version_stamp`: Directory version stamp.

[0204] Generate an L1 directory summary using the following steps:

[0205] (1) Collect the titles, first-level headings, document summaries, tags, and high-frequency words of historical queries of the documents directly under this directory;

[0206] (2) Perform deduplication, stop word filtering, and keyword extraction on the collected text;

[0207] (3) Use a large language model to generate a directory summary of 200 to 400 Chinese characters and extract 8 to 15 directory keywords;

[0208] (4) The directory name, abstract text and keywords are concatenated and sent into the embedding model to generate the L1 directory vector.

[0209] Preferably, the L1 vector is generated as a 1024-dimensional vector using a unified embedding model, such as BGE-M3; in other implementations, a 768-dimensional BERT class vector or a 1536-dimensional `text-embedding-ada-002` style vector can also be used. The L1 vector is mainly used to express the topic scope and directory affiliation, and does not directly undertake fragment-level precise recall.

[0210] L2 document-level index: The L2 index corresponds to the source document entity and is used to complete document range positioning and medium-granularity filtering. Each L2 node contains at least the following fields:

[0211] `document_id`: A unique identifier for a document;

[0212] `directory_id`: Identifier of the L1 directory to which it belongs;

[0213] `document_title`: Document title;

[0214] `document_path`: Document path;

[0215] `doc_type`: Document type;

[0216] `doc_version`: Document version;

[0217] `doc_time`: Document timestamp;

[0218] `global_summary`: Document summary;

[0219] `keywords`: A collection of keywords for the document;

[0220] `chunk_count`: Number of L3 fragments;

[0221] `chunk_ids`: The set of chunk identifiers;

[0222] `vector_id`: Document vector identifier;

[0223] `content_hash`: Document content hash.

[0224] Generate an L2 document summary using the following steps:

[0225] (1) Extract the document skeleton text by title, abstract, chapter title, figure and table description, and first and last paragraph;

[0226] (2) Use a large language model to generate a document summary of 300 to 600 Chinese characters and extract 10 to 20 keywords;

[0227] (3) Divide the document into multiple candidate segments and generate segment vectors;

[0228] (4) Based on the fragment position, title coverage and keyword coverage, the fragment vectors are weighted and summarized to generate a global semantic fingerprint of the document.

[0229] L2 vectors are generated using a "weighted average followed by normalization" method: first, all L3 segment vectors in the document are summed according to their weights, then divided by the sum of the weights, and finally vector normalization is performed.

[0230] The formula for calculating the L2 document vector V_doc is: V_doc = Norm( Σ_{j=1}^{n} w_j × e_j / Σ_{j=1}^{n} w_j ), where V_doc represents the document vector, Σ represents the summation operation, and j is the segment index.

[0231] Where `e_j` represents the vector of the j-th L3 segment;

[0232] `w_j` represents the aggregate weight of the j-th segment;

[0233] `n` represents the total number of segments in the document;

[0234] `Norm` represents the normalization operation.

[0235] Preferably, the aggregate weight `w_j` is calculated according to the following rules: the base weight is 1, and three weighted components are superimposed:

[0236] Title hit component: 0.4 multiplied by the title hit indicator value `h_j`;

[0237] Keyword coverage component: 0.3 multiplied by keyword coverage `k_j`;

[0238] Position weight component: 0.3 multiplied by position weight `p_j`.

[0239] The formula for calculating the aggregate weight w_j is: w_j = 1 + 0.4 × h_j + 0.3 × k_j + 0.3 × p_j.

[0240] Where `h_j` indicates whether the segment matches the title or section title, with a value of 1 if it matches and 0 otherwise;

[0241] `k_j` represents the keyword coverage of the fragment, ranging from [0,1].

[0242] `p_j` represents the segment position weight. Key paragraphs at the beginning and end of the document can be assigned higher values, ranging from [0,1].

[0243] L2 vector optimization uses the same embedding model and 1024-dimensional vector space as L1 to ensure that cross-layer similarity is comparable.

[0244] L3 Fragment-Level Index: The L3 index corresponds to fine-grained text fragments that can directly proceed to the answer generation stage. Each L3 node contains at least the following fields:

[0245] `chunk_id`: Unique identifier for the chunk;

[0246] `document_id`: The identifier of the document to which it belongs;

[0247] `directory_id`: Identifier of the directory to which the directory belongs;

[0248] `section_path`: Chapter path;

[0249] `chunk_text`: The original text of the chunk;

[0250] `chunk_keywords`: Chunk keywords;

[0251] `token_count`: Fragment length;

[0252] `vector_id`: The identifier for the fragment vector;

[0253] `text_hash`: Fragment text hash.

[0254] L3 text segmentation rules include:

[0255] (1) Divide the text according to the original title, chapter, and section boundaries;

[0256] (2) If a single chapter is too long, it shall be further divided into 200 to 500 Chinese characters;

[0257] (3) Set 50 to 100 Chinese characters to overlap between adjacent segments to reduce the loss of boundary information;

[0258] (4) Tables, code blocks, and bulleted lists should be kept intact to avoid semantic breaks.

[0259] The L3 vector is preferably generated using the same embedding model as L1 / L2 to generate a 1024-dimensional vector so as to uniformly calculate the cosine similarity; in other implementations, a 768-dimensional or 1536-dimensional embedding model can also be used.

[0260] The logical association rules, storage format, and retrieval rules of a third-level index include:

[0261] Directory-level to document-level mapping: Each L2 document is bound to a unique parent L1 directory via `directory_id`; an L1 directory can be associated with multiple L2 documents.

[0262] Document-level and fragment-level association identifiers: Each L3 fragment is bound to a unique L2 document via `document_id`, and inherits `directory_id` to support cross-level filtering.

[0263] The storage format is as follows: L1, L2, and L3 vectors are stored in independent vector collections, such as `l1_directory_collection`, `l2_document_collection`, and `l3_chunk_collection`.

[0264] Hierarchical relationships are stored in a graph database or relational table, and at least the parent-child edges, sibling edges, path strings, and version information are saved.

[0265] Metadata can be stored in JSON or structured table fields and used to apply filtering conditions during retrieval.

[0266] The calling rules include:

[0267] Root node queries first invoke the L1 directory-level index;

[0268] After determining the directory range, the L2 document-level index under the corresponding directory is invoked;

[0269] After the scope is locked, the L3 fragment-level index under the restricted directory or restricted document is then invoked;

[0270] Filtering can be performed at any level based on `directory_id`, `document_id`, `section_path`, and `version_stamp`.

[0271] Figure 1 This invention illustrates the overall framework and hierarchical processing flow of a progressive document exploration and retrieval enhancement generation system based on a hierarchical directory, according to an embodiment of the present invention. Figure 2This illustrates a three-level hierarchical index tree and its directory-document-fragment mapping relationship according to an embodiment of the present invention. Figure 3 This invention illustrates the directory exploration engine agent decision-making process and the step-by-step drill-down strategy process according to an embodiment of the present invention; Figure 4 The process of traditional planar RAG full-database single search and HDR-RAG progressive catalog search of the present invention is shown in the comparison.

[0272] Compared with existing Flat RAG, query-rewritten multi-hop retrieval, and general two-level retrieval schemes, the embodiments of this invention have achieved significant technical advancements and beneficial effects. The following experimental data comes from an enterprise knowledge base test set containing 120,000 documents, 690,000 fragments, and 5 major business domains, with a question-answering test set of 10,000 question-answer pairs.

[0273] The technical features and their effects are compared in Table 4:

[0274] Table 4

[0275]

[0276] The comparison of retrieval efficiency is shown in Table 5:

[0277] Table 5

[0278]

[0279] As can be seen from the results in Table 5, the embodiments of the present invention achieve the following through hierarchical directory filtering: the L1 stage filters approximately 90% of irrelevant directories on average, the L2 stage further filters approximately 80% of irrelevant documents, and the L3 stage involves only 2% to 3% of the entire library's fragment space for fine-tuning. Compared with planar RAG, this reduces the average latency by approximately 13 times under the same library size.

[0280] The comparison of recall accuracy and anti-interference capability is shown in Table 6:

[0281] Table 6

[0282]

[0283] The results in Table 6 show that the embodiments of the present invention not only improve the accuracy in single-round retrieval and question answering, but also have a lower interference rate for false recall of cross-project, cross-version, and cross-domain documents, and can effectively reduce the problem of "similar titles but incorrect attribution".

[0284] The embodiments of this invention can output the following complete exploration path example:

[0285] Root └─ [L1 score: 0.91] Project Directory: Monetary Authority of Macao - Text Bot └─ [L2 score: 0.87] Document: WX_System_Architecture_v2.pdf └─ [L3 score: 0.94] Excerpt: The core module includes speech recognition, intent understanding, and knowledge retrieval...

[0286] Existing planar RAGs typically only provide the final fragment, making it difficult to explain why the fragment was selected from the entire knowledge base. However, the embodiments of this invention record the candidate scores, pruning reasons, and drill-down reasons at each level, enabling the source of the answer to have a complete chain of evidence, enhancing interpretability and auditability, and making it more suitable for enterprise auditing, regulatory compliance, and fault tracing.

[0287] The cost comparison for dynamic updates is shown in Table 7:

[0288] Table 7

[0289]

[0290] The key innovation of this invention lies not in simply using hierarchical indexing, nor in simply using agent-based reasoning, but in unifying and integrating hierarchical indexing and agent-based reasoning, as detailed below:

[0291] 1. The directory structure provides the Agent with a stable and auditable search navigation framework;

[0292] 2. Agent reasoning transforms the directory structure from a static storage relationship into a dynamic exploration path;

[0293] 3. Quantitative thresholds and rollback mechanisms make the exploration process repeatable, implementable, and adjustable;

[0294] 4. The local incremental update mechanism enables hierarchical indexes to be maintained in the long term.

[0295] Therefore, the embodiments of the present invention not only solve the problem of optimizing existing retrieval algorithms, but also solve the problems of structure-driven retrieval decisions and maintainable knowledge organization.

[0296] This invention also provides a computer device. Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 5As shown, the computer device includes: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the hierarchical directory-based progressive document exploration retrieval enhancement generation method provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0297] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the hierarchical directory-based progressive document exploration and retrieval enhancement generation method described in this embodiment of the invention. The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0298] Input device 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; output device 24 may include display devices such as a display screen.

[0299] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned progressive document exploration and retrieval enhancement generation method based on hierarchical directory.

[0300] The computer equipment provided above can be used to execute the progressive document exploration and retrieval enhancement generation method based on hierarchical directory provided in the above embodiments, and has corresponding functions and beneficial effects.

[0301] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the hierarchical directory-based progressive document exploration retrieval enhancement generation method provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0302] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the progressive document exploration and retrieval enhancement generation method based on hierarchical directory as described in the above embodiments, but can also execute related operations in the progressive document exploration and retrieval enhancement generation method based on hierarchical directory provided in any embodiment of the present invention.

[0303] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0304] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A progressive document exploration and retrieval enhancement generation system based on hierarchical directories, characterized in that, It includes a user query analysis layer, a knowledge hybrid index layer, a directory exploration engine Agent layer, and a document in-depth reading and answer generation layer; The user query analysis layer is used to receive user query statements, complete query intent recognition, complexity assessment, keyword extraction, directory constraint recognition and exploration strategy generation, and transmit the processed query data to the directory exploration engine Agent layer. The knowledge hybrid index layer is used to construct and maintain a three-level hierarchical vector index based on the original knowledge base directory tree, consisting of L1 directory-level index, L2 document-level index, and L3 fragment-level index. At the same time, it stores and maintains the directory tree structure, parent-child relationships, and version information through a graph database or relational hierarchical table to provide structured index support for retrieval. The directory exploration engine Agent layer has a built-in preset quantitative threshold system and relevance scoring algorithm. It is used to receive the processed query data, evaluate each candidate branch from top to bottom, and intelligently execute branch pruning, parallel exploration, hierarchical drill-down, search range locking and abnormal rollback decisions to gradually shrink the search space. The document reading and answer generation layer is used to perform fragment-level hybrid retrieval within the limited retrieval range locked by the Agent layer, recall highly relevant text fragments, generate accurate answers by combining large language models, and simultaneously output the complete exploration path and evidence tracing chain from the root directory to the target fragment.

2. The progressive document exploration and retrieval enhancement generation system based on hierarchical directory as described in claim 1, characterized in that, Each directory node in the L1 directory-level index of the knowledge hybrid index layer shall be configured with at least the following: unique directory identifier, parent directory identifier, directory name, full path from root to current directory, list of direct child documents, list of subdirectories, directory summary text, set of directory keywords, directory vector identifier, and version stamp field; The L1 directory summary generation steps are as follows: collect the titles of direct documents, first-level headings, document summaries, tags, and high-frequency words from historical queries under the directory. After deduplication, stop word filtering, and keyword extraction, a 200-400 character directory summary is generated through a large language model. 8-15 directory keywords are extracted. The directory name, summary, and keywords are then concatenated and an embedding model is used to generate a 1024-dimensional L1 directory vector.

3. The progressive document exploration and retrieval enhancement generation system based on hierarchical directory as described in claim 1, characterized in that, Each document node in the L2 document-level index of the knowledge hybrid index layer is configured with at least the following: unique document identifier, L1 directory identifier, document title, document storage path, document type, document version, document timestamp, global document summary, document keyword set, number of subordinate L3 segments, subordinate segment identifier set, document vector identifier, and content hash field; The L2 document vectors are generated using a weighted average normalization method, with the formula V_doc = Norm( Σ w_j × e_j / Σ w_j ): All L3 segment vectors in the document are assigned aggregate weights and summed, then normalized after dividing by the sum of weights. Here, w_j represents the aggregate weight of the j-th segment, e_j represents the j-th L3 segment vector, and Norm represents the normalization operation. The aggregate weight consists of a base weight of 1 plus a title hit component of 0.4, a keyword coverage component of 0.3, and a position weight component of 0.3, i.e., w_j = 1 + 0.4 × h_j + 0.3 × k_j + 0.3 × p_j, where h_j indicates whether the segment hits a title or subheading, k_j represents the segment's keyword coverage, and p_j` represents the segment's position weight.

4. The progressive document exploration and retrieval enhancement generation system based on hierarchical directory as described in claim 1, characterized in that, Each fragment node in the L3 fragment-level index of the knowledge hybrid index layer shall be configured with at least the following: fragment unique identifier, document identifier, directory identifier, chapter path, fragment original text, fragment keywords, fragment character length, fragment vector identifier, and text hash field; The L3 text segmentation rules are as follows: priority is given to segmenting according to the original title, chapter, and section boundaries; extremely long chapters are segmented a second time by 200-500 Chinese characters; adjacent segments are set to overlap by 50-100 characters; tables, code blocks, and bulleted lists are kept intact and not split.

5. The progressive document exploration and retrieval enhancement generation system based on hierarchical directory as described in claim 1, characterized in that, The directory exploration engine Agent layer has a built-in preset quantitative threshold system, including: pruning threshold 0.6, drill-down threshold 0.8, document locking threshold 50, fragment locking threshold 200, root node preferred branch number 10, parallel exploration branch limit 3, fragment final recall number 5-20, and maximum exploration depth 3. The maximum exploration depth is limited to a three-level path: Root → L1 → L2 → L3. Physical directories with more than three levels are folded and mapped to L1 or L2 nodes to maintain the decision depth.

6. The progressive document exploration and retrieval enhancement generation system based on hierarchical directory as described in claim 1, characterized in that, The relevance scoring algorithm of the directory exploration engine Agent layer includes semantic similarity calculation, metadata matching degree calculation, and comprehensive score fusion; Semantic similarity is calculated using cosine similarity to determine the degree of matching between the query vector and the candidate node vector; Metadata matching score M_i is linearly weighted and fused with title hit score t_i (weight 0.4), keyword hit score k_i (weight 0.3), path hit score p_i (weight 0.2), and time / version relevance score r_i (weight 0.1), i.e., M_i = 0.4 × t_i + 0.3 × k_i + 0.2 × p_i + 0.1 × r_i, where t_i, k_i, p_i, and r_i all range from [0,1]. The comprehensive score of directory-level and document-level scores is linearly weighted with semantic similarity (0.7) and metadata matching score (0.3), and the fragment-level score is linearly weighted with chunk_score_j = 0.8 × semantic_sim_j + 0.2 × bm25_j, where semantic_sim_j is the cosine similarity between the query and the j-th fragment, bm25_j is the keyword matching score of the j-th fragment, and chunk_score_j is the fragment-level score.

7. A progressive document exploration and retrieval enhancement generation method based on hierarchical directories, applied to the progressive document exploration and retrieval enhancement generation system based on hierarchical directories as described in claims 1-6, characterized in that, Includes the following steps: S1, Query parsing and initial positioning: Receive user queries and encode them into query vectors, extract query keywords, entity words, version time words and directory constraint words, search L1 directory child nodes under the root node, and return the Top 10 candidate directory branches; S2. Relevance assessment and branch pruning: Calculate the comprehensive score of each candidate directory. Branches below the pruning threshold of 0.6 are stored in the waiting set and pruned. Branches above 0.6 are included in the drill-down queue. A maximum of 3 high-scoring branches with a score difference ≤ 0.03 are selected for parallel exploration. S3. Depth control and hierarchical drill-down: Before reaching the maximum exploration depth of 3, traverse the L2 document-level index associated with the drill-down queue branch, calculate the comprehensive document score, and if the high-score branch meets the drill-down threshold of 0.8, continue to explore the lower level; if the threshold is not reached, stop drilling and start range locking judgment. S4. Range locking and fragment retrieval: When the number of candidate documents is ≤50 or the number of candidate fragments is ≤200, the retrieval range is locked. The L3 fragment-level index is only called within the restricted directory / document branch. The top 5-20 highly relevant fragments are recalled by sorting the fragments by mixed scores. When the number of fragments is insufficient, the high-scoring branches are restored from the waiting set to expand the search range. S5. Answer generation and path output: The module generates the recall fragment and the original query input, outputs the final answer, and generates the complete tracing path from Root to L1 directory to L2 document to L3 fragment, recording the candidate scores at each level, pruning and drill-down decision basis. S6. Anomaly Handling and Rollback Adaptation: For four types of anomaly scenarios, namely no candidate hit, high-scoring branch conflict, drill-down without effective segments, and directory constraints and semantic conflicts, respectively, the following adaptation strategies are implemented: threshold reduction, parallel branch synchronous exploration, upper-level branch rollback, and dual-path result output.

8. The progressive document exploration and retrieval enhancement generation method based on hierarchical directory as described in claim 7, characterized in that, It also includes a local incremental update mechanism, specifically including four types of update rules: adding documents, modifying documents, adding directories, and deleting documents; When a new document is added, only the corresponding L2 document index and its subordinate L3 fragment index are generated, the L1 directory to which it belongs is bound, and the summary and vector of that directory are updated incrementally, without modifying other branch indexes; When the document content modification rate is ≥20%, rebuild the document's L2 vector and all its subordinate L3 segment vectors, and synchronously and incrementally update the L1 directory index. When adding a new directory, only the L1 directory node is created and a summary and vector are generated. If the directory has no subordinate documents, the L2 and L3 index updates are not triggered. When a document is deleted, the corresponding L2 and L3 nodes are removed, and the summary and vector of the L1 directory to which it belongs are updated incrementally. When only the metadata changes and the content modification ratio is less than 20%, only the metadata fields are updated and the fragment vector is not rebuilt.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the progressive document exploration and retrieval enhancement generation method based on hierarchical directories as described in any one of claims 7-8.

10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the progressive document exploration and retrieval enhancement generation method based on hierarchical directories as described in any one of claims 7-8.

Images