Automatic construction method and system of multi-granularity hierarchical index for unstructured documents

Abstract

The present application relates to the technical field of RAG retrieval enhancement generation, and particularly relates to a multi-granularity hierarchical index automatic construction method and system for unstructured documents, wherein the method first performs standardization preprocessing on the unstructured documents; multi-level titles are recognized, paragraph boundaries are detected, tables and processes are independently parsed, and a complete document tree containing a document, a chapter, a paragraph and special nodes is assembled and generated through multi-feature weighted scoring; three-layer indexes of the document, the chapter and the paragraph are constructed in layers, a three-layer index cross-layer graph map containing multiple association relationships is built, and document, chapter and paragraph drilling retrieval is realized layer by layer; and a chapter-level incremental update mechanism is matched. The present application overcomes the defects of the traditional RAG fixed block that destroys the original level of the document, lacks independent chapter index, loses special structure logic, has poor semantic matching and cannot comply with the source, significantly improves the recall and precision of complex enterprise document question and answer, and is suitable for long-term operation and maintenance of enterprise compliance knowledge base.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, knowledge graph and RAG retrieval enhancement generation technology, and in particular to a method and system for automatically constructing multi-granularity hierarchical indexes for unstructured documents. Background Technology

[0002] Search Enhancement Generation (RAG) is a mainstream technology for building enterprise knowledge bases. The current common process for processing unstructured documents such as procedures and work instructions is: text extraction, fixed-length token mechanical block division, and construction of a single-layer text and vector index. At the same time, the industry has developed layered optimization schemes such as BookRAG, parent-child block RAG, and RAPTOR to improve the retrieval effect of long documents.

[0003] Existing RAG technology suffers from the following common defects: First, mechanical segmentation or virtual semantic clustering destroys the original document heading hierarchy, resulting in the loss of high-value structural logic such as tables and flowcharts, and making it impossible to restore the original chapter numbering system; Second, existing layered solutions only have two virtual layers, lack an independent chapter intermediate index layer, lack a complete document-chapter-paragraph drill-down retrieval link, and rely solely on vector matching, lacking high-level semantic assistance; Third, the hierarchy relies solely on unidirectional ID association, which cannot output a complete traceability path, and document modifications require a full index rebuild, highlighting compliance traceability and maintenance costs.

[0004] In summary, existing solutions are insufficient to meet the actual business needs of enterprises for high-precision knowledge base retrieval, compliance auditing, and low-cost iteration. Therefore, a new technical solution is urgently needed to address these issues. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic construction method and system for multi-granularity hierarchical indexes for unstructured documents, which solves the problems of document structure loss and semantic fragmentation caused by fixed-length segmentation in the prior art, and provides a technical solution that can automatically parse the document tree, construct a three-level index of "document-chapter-paragraph" and establish cross-level associations.

[0006] To achieve the above objectives, the present invention provides the following technical solution: According to one aspect of the present invention, an automatic construction method for multi-granularity hierarchical indexes for unstructured documents is provided, comprising the following steps: S11 preprocesses unstructured documents and outputs standardized text; S12, based on standardized text, uses multi-feature weighted scoring to identify multi-level headings, detects paragraph boundaries and independently parses special structures such as tables and process types, and generates a complete document tree including documents, chapters, paragraphs and special nodes; S13, based on the complete document tree, constructs a three-level index of document, chapter and paragraph. The document index generates semantic tags for answerable question types through reverse reasoning of a large model. Paragraph blocks do not cross chapters and special structures are formed into independent blocks. S14, construct the cross-layer graph association mapping of the three-layer index to realize the step-by-step drill-down retrieval from document to chapter to paragraph.

[0007] According to an embodiment of the present invention, the method further includes step S15 to implement chapter-level incremental updates: when a document is partially revised, only the chapter index and paragraph index of the corresponding chapter are rebuilt and the cross-layer graph association is updated synchronously, and the original data is reused if the index is not changed.

[0008] According to an embodiment of the present invention, the preprocessing in step S11 includes text extraction, OCR image recognition, page noise reduction, and metadata extraction; the page noise reduction specifically includes removing headers, footers, watermarks, page numbers, signatures and stamps, and duplicate directory noise.

[0009] According to an embodiment of the present invention, step S12 specifically includes the following sub-steps: S121, Title level recognition: Employs a weighted fusion scoring system based on multi-dimensional features such as numbering, layout, independent lines, and semantic prompts, and distinguishes titles at each level and generates title nodes based on the score threshold; S122, Paragraph Boundary Detection: Within a single chapter, the boundary score is calculated by combining line breaks, punctuation marks, and logical transition words to automatically divide independent paragraphs and generate paragraph nodes; S123, Special Structure Parsing: Separately identify tables, multi-level lists, and business processes within a document, extract table row and column data, list level numbers, process steps and constraints, and generate independent special structure nodes; S124, Document Tree Assembly: Using the document as the root node, attach chapter nodes corresponding to each level of headings, and associate corresponding paragraph nodes and special structure nodes under each chapter to generate a complete document tree including the document, chapters, paragraphs, and special nodes.

[0010] According to an embodiment of the present invention, step S13, which involves constructing a three-level index, specifically includes the following sub-steps: S131. Construct a document-granular index: Generate a document profile from the root node of the document tree, extract document identifiers, metadata, key topics, and applicable scenarios; use segmented summaries to summarize and merge to generate a global document summary; call the large model for reverse reasoning to generate semantic tags for answerable question types; S132. Construct a chapter-level index: Generate chapter navigation records based on each chapter node, and store the chapter ID, parent-child hierarchy identifier, complete title path, chapter summary, list of key points, and table / process descriptions within the chapter. S133. Construct paragraph-level indexes: Perform semantic segmentation in accordance with the constraint of not crossing chapter boundaries, retain complete natural paragraphs, and treat tables, lists, and processes as independent semantic blocks; generate paragraph indexes containing block identifiers, chapters, title paths, block summaries, original text, and paragraph semantic vectors.

[0011] According to one embodiment of the present invention, the cross-layer graph association mapping in step S14 includes the inclusion relationship between documents and chapters, the inclusion relationship between chapters and paragraphs / special structures, the order relationship between chapters at the same level, and the affiliation relationship between special structures and their respective chapters.

[0012] According to an embodiment of the present invention, the retrieval process in step S14 includes: matching answerable question type tags in the document index to filter candidate documents; matching the chapter index to lock the target chapter; locating paragraph semantic blocks based on cross-layer graphs, and outputting a complete document-chapter-paragraph tracing path.

[0013] On the other hand, the present invention also provides an automatic construction system for multi-granularity hierarchical indexes for unstructured documents, the system comprising: a preprocessing module, a document tree construction module, a three-level index generation module, and a cross-level association storage module; The preprocessing module is used to perform unstructured document preprocessing and output standardized text; The document tree construction module has a built-in title recognition unit, paragraph boundary detection unit, special structure parsing unit, and document tree generation unit. It is used to identify multi-level titles based on standardized text, use multi-feature weighted scoring to detect paragraph boundaries and independently parse special structures such as tables and flowcharts, and generate a complete document tree including documents, chapters, paragraphs, and special nodes. The three-layer index generation module includes a document-level index submodule, a chapter-level index submodule, and a paragraph-level index submodule, which construct a three-layer index of document, chapter, and paragraph based on the complete document tree. The cross-layer associated storage module is used to construct the cross-layer graph association mapping of the three-layer index, so as to realize the drill-down retrieval from document to chapter to paragraph.

[0014] According to one embodiment of the present invention, the system further includes an incremental update module for performing chapter-level incremental update operations.

[0015] According to one embodiment of the present invention, the system further includes a retrieval interaction module for receiving user queries, performing hierarchical drill-down retrieval based on a three-layer index and cross-layer graph, and outputting a complete tracing path.

[0016] The present invention provides an automatic construction method and system for multi-granularity hierarchical indexes of unstructured documents. Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention fully preserves the original document hierarchy and special structural logic. It adopts a multi-feature weighted quantization mechanism to automatically identify multi-level headings, and relies on the original boundaries of the document tree to divide into blocks. Tables and processes are stored as independent nodes, which completely solves the defects of traditional fixed-length segmentation that splits chapters and destroys table and process logic.

[0017] (2) Progressive and accurate retrieval is achieved through a three-layer independent granularity index. This invention adds an independent chapter navigation index layer to form a three-layer retrieval link of "document coarse screening - chapter positioning - paragraph precise hit", which is different from the existing two-layer / virtual level scheme and greatly improves the retrieval accuracy of long procedures and regulations documents.

[0018] (3) Use large models to generate question and answer tags in reverse, pre-label the types of questions that can be answered at the document level, complete the coarse matching of intent in the early stage of retrieval, and solve the problem of insufficient recall caused by the inconsistency between users’ spoken questions and written institutional texts.

[0019] (4) The full-link graph association meets the requirements of compliance traceability. It fully records the four types of hierarchical relationships: inclusion, sequence, and attribution. The search results automatically output a complete tree-shaped traceability path, which is suitable for the audit evidence requirements of highly compliant industries such as power grid, legal affairs, and finance.

[0020] (5) Chapter-level incremental updates significantly reduce operation and maintenance costs. Local modifications to a document only require recalculating the corresponding chapter index and synchronizing the knowledge graph, without the need for full-text parsing and vectorization, which significantly reduces the computational and time overhead of large enterprise knowledge base iterations. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of an automatic construction method for multi-granularity hierarchical indexes of unstructured documents according to an embodiment of the present invention; Figure 2 This is a schematic diagram of an automatic multi-granularity hierarchical index building system for unstructured documents according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the construction process of a multi-granularity hierarchical index for unstructured documents according to an embodiment of the present invention. Detailed Implementation

[0022] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0023] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0024] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0025] like Figure 1 The diagram shows a flowchart of an automatic multi-granularity hierarchical index construction method for unstructured documents. This method includes the following steps: S11 preprocesses unstructured documents and outputs standardized text; Preprocessing includes text extraction, OCR image recognition, page noise reduction, and metadata extraction; the page noise reduction specifically includes removing headers, footers, watermarks, page numbers, signatures and stamps, and duplicate directory noise.

[0026] S12, based on standardized text, uses multi-feature weighted scoring to identify multi-level headings, detects paragraph boundaries and independently parses special structures such as tables and process types, and generates a complete document tree including documents, chapters, paragraphs and special nodes; Specifically, it includes the following sub-steps: S121, Title level recognition: Employs a weighted fusion scoring system based on multi-dimensional features such as numbering, layout, independent lines, and semantic prompts, and distinguishes titles at each level and generates title nodes based on the score threshold; S122, Paragraph Boundary Detection: Within a single chapter, the boundary score is calculated by combining line breaks, punctuation marks, and logical transition words to automatically divide independent paragraphs and generate paragraph nodes; S123, Special Structure Parsing: Separately identify tables, multi-level lists, and business processes within a document, extract table row and column data, list level numbers, process steps and constraints, and generate independent special structure nodes; S124, Document Tree Assembly: Using the document as the root node, attach chapter nodes corresponding to each level of headings, and associate corresponding paragraph nodes and special structure nodes under each chapter to generate a complete document tree including the document, chapters, paragraphs, and special nodes.

[0027] S13, based on the complete document tree, constructs a three-level index of document, chapter and paragraph. The document index generates semantic tags for answerable question types through reverse reasoning of a large model. Paragraph blocks do not cross chapters and special structures are formed into independent blocks. Specifically, it includes the following sub-steps: S131. Construct a document-granular index: Generate a document profile from the root node of the document tree, extract document identifiers, metadata, key topics, and applicable scenarios; use segmented summaries to summarize and merge to generate a global document summary; call the large model for reverse reasoning to generate semantic tags for answerable question types; S132. Construct a chapter-level index: Generate chapter navigation records based on each chapter node, and store the chapter ID, parent-child hierarchy identifier, complete title path, chapter summary, list of key points, and table / process descriptions within the chapter. S133. Construct paragraph-level indexes: Perform semantic segmentation in accordance with the constraint of not crossing chapter boundaries, retain complete natural paragraphs, and treat tables, lists, and processes as independent semantic blocks; generate paragraph indexes containing block identifiers, chapters, title paths, block summaries, original text, and paragraph semantic vectors.

[0028] S14, construct the cross-layer graph association mapping of the three-layer index to achieve layer-by-layer drill-down retrieval from document to chapter to paragraph; the cross-layer graph association mapping includes the inclusion relationship between document and chapter, the inclusion relationship between chapter and paragraph / special structure, the order relationship between sibling chapters, and the affiliation relationship between special structure and its belonging chapter. The retrieval process includes: matching answerable question type tags in the document index to filter candidate documents; matching the chapter index to lock the target chapter; locating paragraph semantic blocks based on the cross-layer graph, and outputting the complete document-chapter-paragraph tracing path.

[0029] S15 enables chapter-level incremental updates: when a document is partially revised, only the chapter index and paragraph index of the corresponding chapter are rebuilt, and the cross-level graph associations are updated synchronously. The original data is reused if the index is not changed.

[0030] like Figure 2 The diagram shows an automatic multi-granularity hierarchical index building system for unstructured documents. This system is used to implement the above method and includes: a preprocessing module, a document tree building module, a three-level index generation module, and a cross-level association storage module. The preprocessing module is used to perform unstructured document preprocessing and output standardized text; The document tree construction module has a built-in title recognition unit, paragraph boundary detection unit, special structure parsing unit, and document tree generation unit. It is used to identify multi-level titles based on standardized text, use multi-feature weighted scoring to detect paragraph boundaries and independently parse special structures such as tables and flowcharts, and generate a complete document tree including documents, chapters, paragraphs, and special nodes. The three-layer index generation module includes a document-level index submodule, a chapter-level index submodule, and a paragraph-level index submodule, which construct a three-layer index of document, chapter, and paragraph based on the complete document tree. The cross-layer associated storage module is used to construct the cross-layer graph association mapping of the three-layer index, so as to realize the drill-down retrieval from document to chapter to paragraph.

[0031] External input includes various unstructured documents such as PDFs, Word documents, and scanned documents, which are input unidirectionally to the preprocessing module.

[0032] The system also includes an incremental update module for performing chapter-level incremental update operations.

[0033] The system also includes a retrieval and interaction module, which receives user queries, performs hierarchical drill-down retrieval based on the three-layer index and cross-layer graph, and outputs a complete tracing path.

[0034] External input includes various unstructured documents such as PDFs, Word documents, and scanned copies, which are fed into the preprocessing module. The preprocessing module receives the raw unstructured documents, performs text extraction, OCR image recognition, page noise reduction, and metadata extraction, and outputs standardized clean text, which is then transmitted down to the document tree construction module. At the same time, the preprocessing module receives revised chapter text data returned by the incremental update module for local reprocessing.

[0035] The document tree construction module receives standardized text output from the preprocessing module and contains three parallel processing units and one aggregation and assembly unit, as follows: The title recognition unit uses a multi-feature weighted fusion scoring mechanism to identify titles at all levels within the document and outputs title node information; the paragraph boundary detection unit calculates paragraph boundary scores within the chapter range, divides independent paragraphs, and outputs paragraph node information; the special structure parsing unit independently identifies and parses tables, multi-level lists, and business processes, extracts row and column constraints, and generates independent special structure nodes; the document tree generation unit receives node data output from the above three units, completes hierarchical assembly with the document as the root node, generates a complete structured document tree containing documents, chapters, paragraphs, and special structure nodes, and sends the document tree data down to the three-level index generation module.

[0036] The three-level index generation module internally runs three independent index sub-modules synchronously, generating three levels of granular index data in parallel: The document-level indexing submodule constructs a macro-level document profile, generating a global summary, key topics, and semantic tags for answerable question types; the chapter-level indexing submodule generates chapter navigation records, storing title paths, chapter summaries, core points, and descriptions of internal special structures; the paragraph-level indexing submodule follows the rule of semantic block division without crossing chapters, treating tables and processes as separate semantic blocks, and generating paragraph indexes containing semantic vectors and block summaries; after all three levels of indexes are generated, they are uniformly output downwards to the cross-level associated storage module.

[0037] The cross-level association storage module receives three types of index data—documents, chapters, and paragraphs—output by the three-level index generation module. It establishes and persistently stores multi-level graph association mapping relationships (including document-chapter inclusion relationships, chapter-paragraph / special structure inclusion relationships, sibling chapter order relationships, and special structure affiliation relationships). It provides a hierarchical association query interface to the retrieval interaction module below and synchronizes index association data to the incremental update module above.

[0038] The retrieval interaction module serves as the system's external interaction point. It receives user retrieval query requests, calls the three-layer index and hierarchical mapping relationship within the cross-layer associated storage module to perform a layer-by-layer drill-down retrieval, and outputs the complete source path of "document-chapter-paragraph" and the target text block after the retrieval is completed, thus completing the question-and-answer interaction.

[0039] When partial content of a document is revised, the incremental update module locates the corresponding chapter boundary and sends the chapter text to be reprocessed to the preprocessing module. After the document tree and three-level index are partially rebuilt, the incremental update module synchronously updates the graph mapping in the cross-level associated storage module. Unchanged index data is directly reused, realizing chapter-level partial incremental updates without the need to reprocess the entire document.

[0040] like Figure 3As shown, a flowchart illustrating the multi-granularity hierarchical index construction process for unstructured documents is presented. The overall process executes sequentially from top to bottom, with the complete chain consisting of original document input, standardized preprocessing, document tree structured parsing, parallel generation of three-layer multi-granularity indexes, cross-layer association graph storage, and hierarchical retrieval application. The starting point of the process is various types of original unstructured documents without fixed hierarchical structures, such as PDFs, Word documents, and scanned documents, which serve as the input data source for the entire index construction system. The specific processing flow includes the following steps: S1, Standardized Preprocessing.

[0041] The original documents are processed uniformly, including text extraction, page noise reduction, OCR image and text recognition, and document metadata extraction. Interference noise from headers, watermarks, page numbers, and stamps is filtered out, and clean and uniform standardized text is output to prepare data for subsequent structure analysis.

[0042] S2, Structure-Aware Document Parsing and Document Tree Construction.

[0043] S21, Heading Level Recognition: A comprehensive weighted scoring mechanism is used to determine the heading level of each line of text, distinguishing between first-level, second-level, and third-level headings and generating heading nodes; the specific determination formula is as follows: Title determination score = α × numbering feature + β × layout feature + γ × independent line feature + δ × semantic prompt feature; For example, α=0.4, β=0.3, γ=0.2, δ=0.1, and the total score threshold is 0.6; if the title judgment score exceeds the preset threshold, it is judged as a title node.

[0044] S22, Paragraph Boundary Detection: Automatically segments independent paragraphs within the same chapter and generates paragraph nodes based on boundary scores. Within title boundaries, it calculates paragraph boundary scores by integrating features such as natural line breaks, end markers, and structural transition words, and then segments the paragraphs accordingly. Similar to title determination scoring, paragraph boundary scores can be calculated, with a paragraph boundary threshold of 0.4.

[0045] S23, Special Structure Extraction: Separately identify and parse tables, multi-level lists, and business processes within the document to generate independent special structure nodes; the special structure specifically includes table headers / rows / columns / data items, list hierarchy / order, and process steps / constraints.

[0046] S24, Constructing the Document Tree: Integrate the document root node, multi-level chapters, paragraphs, and special structure nodes to assemble and generate a tree-structured document tree, and output a standardized document tree object.

[0047] Next, based on the document tree, three independent indexes are generated simultaneously. Using the complete document tree as the data source, independent index libraries of different granularities are generated synchronously in three separate ways: S3, Document Granularity Index (Macro Layer): Generates document profiles, storing global summaries, key document topics, and semantic tags for answerable question types. It generates document profile records based on the document tree, with fields including: document identifier, summary, key topics, and applicable scenarios. Its core feature is the use of a large language model for reverse reasoning to generate "answerable question type" fields (such as scope of application, division of responsibilities, parameter types, etc.). Long documents employ a "segmented summary - aggregation and merging" strategy to generate a global summary.

[0048] S4, Chapter Granularity Index (Mid-level Navigation Layer): Generates chapter navigation records, storing complete title hierarchy paths, chapter summaries, and a list of key points for each chapter; generates chapter navigation records with chapter nodes as boundaries, including fields such as: chapter identifier, parent-child hierarchy identifier, title path, chapter summary, list of key points, and special structure descriptions.

[0049] S5, Paragraph-level Indexing (Micro-level Retrieval Unit): Semantic block segmentation is performed under chapter boundary constraints. The principles are: no crossing of chapter boundaries, priority given to complete paragraphs, and tables / lists / processes treated as independent blocks. Semantic block segmentation without crossing chapter boundaries is performed, generating index records including text block identifiers, their respective chapters, title paths, block summaries, original text content, and paragraph-level semantic representations. The unique identifier of each text block, paragraph summary, original text content, and semantic vector are stored; tables / processes are treated as separate semantic blocks.

[0050] S6, cross-layer relationship registration and storage.

[0051] A unified graph mapping relationship is established and persistently stored for the three-level indexes S3, S4, and S5. The cross-level graph mapping relationship includes three types of logic: document-to-chapter inclusion relationship, chapter-to-paragraph inclusion relationship, sibling chapter order relationship, and special structure affiliation relationship, thus connecting the hierarchical association links of the three-level indexes. It supports drilling down layer by layer according to "document -> chapter -> text block" during retrieval.

[0052] Hierarchical retrieval is achieved by leveraging the cross-layer association mapping of S6 storage. The retrieval chain drills down layer by layer from top to bottom: first, matching the document granularity index to filter candidate documents → then matching the chapter granularity index to lock the target chapter → finally locating the paragraph granularity semantic block, and outputting retrieval results with complete hierarchical tracing path.

[0053] Furthermore, the cross-layer relationship graph supports fine-grained local incremental updates, significantly reducing the computational power consumption and maintenance costs of the enterprise knowledge base. When a document is revised, only the affected chapter boundaries are located, the paragraphs and chapter indexes within that chapter are rebuilt, and the corresponding cross-layer relationships are updated.

[0054] Figure 3In the illustrated process, the original document undergoes preprocessing and structured parsing to generate a unified document tree. Then, three independent indexes—document, chapter, and paragraph—are constructed in parallel. A unified hierarchical association graph is then registered, ultimately supporting a progressive hierarchical retrieval of "document-chapter-paragraph." This fully automates the processing of unstructured documents, preserving their original hierarchical structure, multi-granularity indexing, and compliant source retrieval. It abandons mindless fixed-length segmentation, relying on title and paragraph boundaries for semantic block division, perfectly preserving the logical integrity of images, tables, and flowcharts. It supports hierarchical positioning of "intent -> document -> chapter -> paragraph," greatly improving the retrieval accuracy of long documents and complex regulations. The unique "answerable question type" reverse generation technology, combined with intelligent summarization at each level, effectively solves the semantic gap between user questions and the original text.

[0055] Example 1: Application of knowledge platform for power grid enterprises.

[0056] This embodiment uses a power grid enterprise knowledge platform as the application scenario. The accessed unstructured original documents include "Dispatch Operation Procedures," "Distribution Network Fault Repair Operation Guidelines," and "Procurement Management Measures," etc. These unstructured documents feature multi-level numbering of chapters, sections, and clauses, operation appendices, and fault handling procedures. The specific implementation steps are as follows: S301, Document Standardization Preprocessing: After obtaining documents such as the "Distribution Network Fault Emergency Repair Operation Instruction Manual", the system performs text extraction and OCR recognition. Based on the characteristics of power grid documents, it filters out table of contents, page numbers, stamps and signatures, and duplicate header noise, extracts document metadata, and outputs clean and standardized text.

[0057] S302, Title hierarchy recognition and structured tree construction, generating document tree: The document tree is constructed by identifying first-, second-, and third-level headings through multi-feature weighted scoring, extracting the safety measures checklist and the anomaly reporting process as independent special nodes, and assembling them into a complete document tree. Specifically, the heading level is automatically identified based on the feature scoring formula. For example, "Chapter 3 Emergency Repair Implementation Requirements" is identified as a first-level (parent) node, "Section 2 On-site Handling Requirements" is identified as a second-level (child) node, and "I. Power Outage Testing and Grounding" is identified as a third-level node. At the same time, "Safety Measures Checklist" and "Anomaly Reporting Process" are accurately extracted as special table nodes and process nodes within the text boundaries to construct a complete document tree.

[0058] S303, Generate the first-level (document-level) index: A segmented summary-summary merging strategy is used to generate a document profile record for this "Guideline". Through back-reasoning of a large model, the "answerable question type" feature field of this document is automatically generated, such as: emergency repair prerequisites, power outage permit requirements, on-site safety measures, etc., and written into the document granularity index library.

[0059] S304, Generate the second-level (chapter-level) index: Taking "Section 2 On-site Handling Requirements" as an example, generate chapter navigation records. Extract the "title path (Chapter 3 / Section 2)" and summarize the core points in a structured way (including verifying the permit status, completing the power testing and grounding, clarifying the monitoring responsibilities, etc.), and write it into the chapter granularity index library.

[0060] S305, Generate the third-level (paragraph-level) index: Within the aforementioned chapters, the text is divided into blocks according to the principle of not crossing semantic boundaries. The specific operation of "power outage testing and grounding" is divided into independent text blocks, labeled with the "process step type" structure tag, and the original text location range is recorded.

[0061] S306, Cross-layer retrieval and location linkage: When a business person asks, "What are the requirements for voltage testing and grounding before emergency repair of a distribution network fault?", the system triggers a cross-layer correlation search: Step 1 (Macro Positioning): Target the "On-site Safety Measures" question tag in the "Guideline" document profile; Step 2 (Mesoscopic Positioning): Traverse downwards along the tree structure and hit the navigation record of the "Section 2 On-site Handling Requirements" chapter; Step 3 (Microscopic Drill-down): Finally, accurately locate and return the semantic block of the "Power Outage Testing and Grounding" section and its specific operation list.

[0062] At the same time, the system can also handle comprehensive management issues, such as asking "At what monetary threshold does a procurement project need to be submitted for special review?", and accurately find the corresponding clauses in the "Procurement Management Measures" through the same drill-down approach.

[0063] Example 2: Application of corporate policy database for group companies.

[0064] This embodiment uses a comprehensive system management platform of a large group enterprise as the application scenario. The documents accessed cover unstructured documents with strong regulatory attributes, such as those related to human resources, financial reimbursement, legal review and approval, and project management. These documents are characterized by "numerous conditions and constraints, and detailed hierarchical permissions." Taking the "Contract Review and Approval Management System" as an example, the specific implementation steps are as follows: S401, Analysis of Institutional-Specific Structures: The system performs structure-aware parsing on standardized texts, identifying the structure of standard regulations chapters such as "General Provisions," "Scope of Application," "Division of Responsibilities," "Approval Process," "Exceptions," and "Supplementary Provisions." For common structured expressions in the regulations, the "Countersigning Path List" is extracted as a list node, and the "Approval Responsibility Hierarchy Table" is extracted as a table node.

[0065] S402, Multi-granularity semantic feature extraction and index generation: At the document granularity level, a global document profile is generated, focusing on extracting and solidifying its core entities such as "applicable objects, contract types, and exception approval requirements," and marking its strength in answering "contract review questions."

[0066] At the chapter granularity level, independent chapter navigation records are generated for different management dimensions such as "approval process", "authorization level" and "risk clause review".

[0067] At the paragraph granularity level, high-density paragraph semantic block records are generated based on natural paragraphs and semantic breakpoints for fine-grained clauses such as "specific approval threshold amount", "preconditions for legal intervention", and "order of signing by various departments".

[0068] S403, Related Registration: Establish a strict attribution and inclusion mapping relationship between the approval responsibility hierarchy table (special node) and the "permission hierarchy" chapter.

[0069] S404, Compliance Traceability and Retrieval Application: When legal or business personnel ask, "Under what circumstances must a technical service contract undergo legal review?": The system utilizes a multi-granularity hierarchical index to first pre-match and hit "contract review issues" (corresponding to the "Contract Review and Approval Management System"), then quickly locate the "Risk Clause Review" section, and finally extract the specific clause paragraphs containing "preconditions for legal intervention".

[0070] S405, Application Effect Verification: When returning the final clause content, the front-end interface automatically displays the complete tree path of the clause: "Document Name -> Risk Clause Review Section -> Article X, Section Y". This effectively enhances the traceability of answers and the integrity of the evidence chain in compliance review and legal Q&A scenarios for the group enterprise.

[0071] Compared with existing traditional retrieval technologies based on "fixed-length segmentation + planar vector storage", this invention abandons the mechanical fixed-length segmentation method. Through structure-aware parsing, it fully preserves the logical structure of the document's title hierarchy, chapter boundaries, and special elements such as tables and processes, achieving deep fidelity of document structure, eliminating semantic fragmentation, and fundamentally solving the problem of contextual semantic loss caused by long documents being fragmented.

[0072] This invention relies on a three-layer index—"document granularity—chapter granularity—paragraph granularity"—and its cross-layer association mapping, giving the retrieval system a global perspective. It achieves a hierarchical, progressive retrieval process: first matching the target document, then locating relevant chapters, and finally precisely hitting paragraph blocks. This significantly improves the semantic expression and pre-matching capabilities of the index, greatly enhancing the retrieval accuracy of complex corporate documents. Traditional indexes only contain a single "original text + vector" pattern. This invention introduces higher-level semantics such as summaries, topics, and core points generated based on a large model at each level. In particular, by utilizing reverse generation technology based on "answerable question types," it greatly bridges the semantic gap between user questions and the original wording of regulations, improving retrieval recall.

[0073] This invention addresses the unique structures prevalent in enterprise documents, such as tables, lists, and workflows. It enhances the parsing and utilization of high-density, specialized structural information by treating these structures as independent nodes for feature extraction and attribution. This preserves the internal field relationships and sequential logic, effectively improving the accuracy of question-and-answering for complex logical questions such as those involving parameter thresholds and operational steps. In summary, thanks to the complete tree structure and cross-layer associations, the search results not only return specific text paragraphs but also trace back to and display the complete "document-chapter" path and contextual attribution. This meets the stringent requirements of enterprises (such as power grids, legal departments, and finance departments) for 100% traceability and verifiability of knowledge-based question-and-answer results.

[0074] Furthermore, this invention addresses the scenario of continuous iteration of large enterprise knowledge bases by supporting local incremental updates based on a structured document tree. When the original document is revised, it supports fine-grained and efficient incremental updates, requiring only the reconstruction of the index and relationships of the affected chapters, without the need to recalculate the entire text, thus greatly saving system computing power and index maintenance time costs.

[0075] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and other materials. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0076] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for automatically constructing multi-granularity hierarchical indexes for unstructured documents, characterized in that, Includes the following steps: S11 preprocesses unstructured documents and outputs standardized text; S12, based on standardized text, uses multi-feature weighted scoring to identify multi-level headings, detects paragraph boundaries and independently parses special structures such as tables and process types, and generates a complete document tree including documents, chapters, paragraphs and special nodes; S13, based on the complete document tree, constructs a three-level index of document, chapter and paragraph. The document index generates semantic tags for answerable question types through reverse reasoning of a large model. Paragraph blocks do not cross chapters and special structures are formed into independent blocks. S14, construct the cross-layer graph association mapping of the three-layer index to realize the step-by-step drill-down retrieval from document to chapter to paragraph.

2. The method according to claim 1, characterized in that, The method further includes step S15 to implement chapter-level incremental updates: when a document is partially revised, only the chapter index and paragraph index of the corresponding chapter are rebuilt and the cross-layer graph association is updated synchronously, and the original data is reused if the index is not changed.

3. The method according to claim 1, characterized in that, The preprocessing in step S11 includes text extraction, OCR image recognition, page noise reduction, and metadata extraction; the page noise reduction specifically includes removing headers, footers, watermarks, page numbers, signatures and stamps, and duplicate directory noise.

4. The method according to claim 1, characterized in that, Step S12 specifically includes the following sub-steps: S121, Title level recognition: Employs a weighted fusion scoring system based on multi-dimensional features such as numbering, layout, independent lines, and semantic prompts, and distinguishes titles at each level and generates title nodes based on the score threshold; S122, Paragraph Boundary Detection: Within a single chapter, the boundary score is calculated by combining line breaks, punctuation marks, and logical transition words to automatically divide independent paragraphs and generate paragraph nodes; S123, Special Structure Parsing: Separately identify tables, multi-level lists, and business processes within a document, extract table row and column data, list level numbers, process steps and constraints, and generate independent special structure nodes; S124, Document Tree Assembly: Using the document as the root node, attach chapter nodes corresponding to each level of headings, and associate corresponding paragraph nodes and special structure nodes under each chapter to generate a complete document tree including the document, chapters, paragraphs, and special nodes.

5. The method according to claim 1, characterized in that, Step S13, constructing a three-tiered index, specifically includes the following sub-steps: S131. Construct a document-granular index: Generate a document profile from the root node of the document tree, extract document identifiers, metadata, key topics, and applicable scenarios; use segmented summaries to summarize and merge to generate a global document summary; call the large model for reverse reasoning to generate semantic tags for answerable question types; S132. Construct a chapter-level index: Generate chapter navigation records based on each chapter node, and store the chapter ID, parent-child hierarchy identifier, complete title path, chapter summary, list of key points, and table / process descriptions within the chapter. S133. Construct paragraph-level indexes: Perform semantic segmentation in accordance with the constraint of not crossing chapter boundaries, retain complete natural paragraphs, and treat tables, lists, and processes as independent semantic blocks; generate paragraph indexes containing block identifiers, chapters, title paths, block summaries, original text, and paragraph semantic vectors.

6. The method according to claim 1, characterized in that, In step S14, the cross-level graph association mapping includes the inclusion relationship between documents and chapters, the inclusion relationship between chapters and paragraphs / special structures, the order relationship between sibling chapters, and the affiliation relationship between special structures and their respective chapters.

7. The method according to claim 1, characterized in that, The retrieval process in step S14 includes: matching answerable question type tags in the document index to filter candidate documents; matching the chapter index to lock the target chapter; locating paragraph semantic blocks based on cross-layer graphs and outputting the complete document-chapter-paragraph tracing path.

8. An automatic construction system for multi-granularity hierarchical indexes of unstructured documents, used to implement the method described in any one of claims 1 to 7, characterized in that, The system includes: a preprocessing module, a document tree construction module, a three-level index generation module, and a cross-level association storage module; The preprocessing module is used to perform unstructured document preprocessing and output standardized text; The document tree construction module has a built-in title recognition unit, paragraph boundary detection unit, special structure parsing unit, and document tree generation unit. It is used to identify multi-level titles based on standardized text, use multi-feature weighted scoring to detect paragraph boundaries and independently parse special structures such as tables and flowcharts, and generate a complete document tree including documents, chapters, paragraphs, and special nodes. The three-layer index generation module includes a document-level index submodule, a chapter-level index submodule, and a paragraph-level index submodule, which construct a three-layer index of document, chapter, and paragraph based on the complete document tree. The cross-layer association storage module is used to construct the cross-layer graph association mapping of the three-layer index, so as to realize the drill-down retrieval from document to chapter to paragraph.

9. The system according to claim 8, characterized in that, The system also includes an incremental update module for performing chapter-level incremental update operations.

10. The system according to claim 8, characterized in that, The system also includes a retrieval and interaction module, which receives user queries, performs hierarchical drill-down retrieval based on the three-layer index and cross-layer graph, and outputs a complete tracing path.

Images