A directory-aware long document knowledge base construction method and program product

Abstract

The application discloses a long document knowledge base construction method and program product based on directory perception, and belongs to the field of artificial intelligence and natural language processing. The scheme realizes the automatic construction of the knowledge base and the generation of high-quality search enhancement by sequentially preprocessing the original long document, analyzing the directory structure, hybrid blocking, double-label generation, intelligent optimization of labels, vectorization and meta-information mounting. The application fully utilizes the directory structure perception ability to ensure semantic integrity, combines the double-label system and intelligent optimization cycle to improve the label quality and search efficiency, significantly improves the construction efficiency, search accuracy and result traceability of the knowledge base, and is suitable for intelligent processing and application of long and complex structure documents such as academic monographs, technical documents and policies and regulations.

Description

Technical Field

[0001] This invention relates to knowledge base construction technology, and more particularly to a method and program product for constructing a long document knowledge base based on directory awareness. Background Technology

[0002] Against the backdrop of the rapid development of artificial intelligence, especially large language models, the digitization, structured storage, and efficient retrieval of knowledge have become key infrastructures driving intelligent question answering, automated reasoning, and knowledge-assisted decision-making. Long, unstructured documents (such as academic monographs, standards and specifications, policies and regulations, and industry technical manuals) typically contain important professional knowledge in a specific field. These documents are characterized by their length (hundreds of thousands to millions of words), complex structure (multi-level directories and chapters), large semantic span, and dense use of technical terminology. These documents often constitute important sources of knowledge in vertical domains and serve as the foundational training material and reasoning basis for building specialized large-scale models in those domains.

[0003] Digital processing refers to converting information contained in physical (paper) or existing documents into computer-readable and processable digital text data, followed by cleaning, standardization, and structuring for subsequent computation and processing. Its objectives include knowledge preservation and sharing, searchability and computability, and preparation for automated processing. In large-scale model applications within vertical domains, digital processing is not only about generating machine-readable text, but also about ensuring the semantic integrity and structural fidelity of the data. It ensures that the large model can accurately capture the context of the original knowledge, thereby avoiding misjudgments and information gaps during reasoning and generation.

[0004] Knowledge base construction is the process of organizing information into a set of searchable, reasonable, and maintainable knowledge units after digital processing. The core objective of this process is to enable large models and other intelligent systems to access accurate, authentic, and traceable knowledge through efficient retrieval and reasoning, thereby improving the accuracy and reliability of specialized tasks. In the construction of specialized large models in vertical domains, knowledge bases play the following key roles: accurate retrieval, reasoning support, prevention of illusions, and dynamic updates.

[0005] While general-purpose large models demonstrate strong knowledge coverage and language capabilities in open domains, they still exhibit significant shortcomings in highly specialized vertical domains such as humanities, ethics, medicine, and engineering regulations. This is mainly due to: the need for refined domain semantics and diverse terminology; the low proportion of domain-specific data in large-scale general training data, leading to insufficient domain-specific knowledge identification rate and accuracy; and the lack of standardized and traceable contextual knowledge, resulting in erroneous or unreliable generated results.

[0006] Therefore, before building a high-performance, vertically specialized large-scale model, it is essential to first establish a structured, semantic, and optimizable knowledge base to provide high-quality retrieval support and training materials that incorporate domain context. This knowledge base should possess the following capabilities: retaining the structural hierarchy information of the original document; generating structural and semantic tags; automatically performing tag quality assessment and iterative optimization; providing source tracing information for each knowledge unit; and automating the construction from the original text to the knowledge base via an automated pipeline.

[0007] Although digital processing and knowledge base construction technologies have been widely used in the engineering field, general solutions still suffer from defects such as semantic fragmentation, semantic repetition in tag systems, lack of multidimensional meta-information, and poor interpretability of retrieval and reasoning when supporting long structured documents and large models in vertical domains. As a result, the efficiency of knowledge base construction, retrieval accuracy, and traceability of results need to be further improved. Summary of the Invention

[0008] In view of the problems existing in the prior art, the purpose of this invention is to provide a directory-aware long document knowledge base construction method and program product with higher construction efficiency, retrieval accuracy and result traceability.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0010] A method for constructing a long document knowledge base based on directory awareness includes the following steps:

[0011] (1) Obtain several original long documents and convert all original long documents into standardized text with unified encoding processing and removal of worthless information to form a standardized text set;

[0012] (2) For each standardized text, extract the chapter titles in the text, and take each chapter title, its position and its level as a chapter title node, and generate a chapter tree according to the level of the chapter title;

[0013] (3) For each standardized text, the chapter title node position of the chapter tree is used as the content boundary to divide it into several content blocks. Each content block includes the main text content corresponding to the chapter title, the starting position and the level it belongs to;

[0014] (4) Based on the main text of each content block, extract topic tags and usage type tags. The topic tags are used to represent the topic type of the corresponding content block, and the usage type tags are used to represent the usage attribute of the corresponding content block.

[0015] (5) Quantify the scores of all tags and optimize the scores of tags below the threshold by regularization until the scores of all tags are above the threshold;

[0016] (6) Combine the main text of each content block with the optimized topic tags and usage type tags into a semantic unit, and extract a high-dimensional semantic vector from the semantic unit. Combine the high-dimensional semantic vector with the corresponding meta-information to form a knowledge unit and store it in the knowledge base. The meta-information includes the chapter title, level, starting position, topic tags, usage type tags, standardized text number, standardized text path and timestamp of the corresponding content block.

[0017] Furthermore, step (1) specifically includes:

[0018] (1.1) Obtain several original long documents, parse each original long document and convert its text encoding format to a uniform text encoding format for plain text;

[0019] (1.2) For encoded plain text, the corresponding noise pattern is identified by each noise pattern in the preset noise pattern set, and the identified noise is removed to obtain pure text. The noise pattern set includes several noise patterns, and each noise pattern corresponds to a regular expression rule, which is used to match and identify the corresponding type of worthless information noise.

[0020] (1.3) Use a preset symbol mapping table to replace each special symbol in the clean text with a unified standard encoding symbol to obtain normalized text, wherein the symbol mapping table stores each special symbol and its corresponding unified standard encoding symbol;

[0021] (1.4) Compare the normalized text length with the original long document length and calculate the relative error. If the relative error exceeds the preset error threshold, it is determined that there is an abnormality in the conversion process. Record the abnormality and return to step (1.1) to re-execute. If the relative error does not exceed the preset error threshold, it is determined that the conversion is normal and the normalized text is added to the standardized text set.

[0022] Furthermore, step (2) specifically includes:

[0023] (2.1) For each standardized text in the standardized text set, scan it line by line. During the scan, use the chapter title matching pattern set to determine whether it is a chapter title. Generate a chapter title node by taking the text, position and level of each identified chapter title and store it in the chapter title node list. The chapter title matching pattern set includes several chapter title matching patterns. Each chapter title matching pattern is a combination of chapter title recognition regular expression and finite state machine, which is used to parse the text and identify the corresponding type of chapter title and infer the level of the chapter title based on the chapter title number and indentation.

[0024] (2.2) For each standardized text, generate a chapter tree by taking all chapter title nodes in the chapter title node list according to the level and position of each chapter title node in the chapter title node list; if a discontinuous chapter title sequence number or a missing chapter title is detected, the missing node is automatically filled in using the hierarchical pattern rules, and position information is added to distinguish when chapter titles at the same level are repeated.

[0025] Furthermore, step (3) specifically includes:

[0026] (3.1) For each chapter title node in each chapter tree, extract the content between the current chapter title node and the next chapter title node from the corresponding standardized text as the body text of the current chapter title, and encapsulate it together with the chapter title text and the level into a structure block;

[0027] (3.2) Perform length detection on the main text content of each structural block. When the length is greater than the preset length threshold, the main text content is split into several sub-contents using the preset semantic pause point, and each sub-content, the chapter title text of the structural block and the level of the structural block are encapsulated into a sub-structural block.

[0028] (3.3) Add the start and end positions of the main text content in the standardized text for each structure block or substructure block, and store them in the content block set in the order of their positions in the standardized text.

[0029] Furthermore, step (4) specifically includes:

[0030] (4.1) Input the main text of each content block into the pre-trained first embedding model to extract the topic semantic vector, wherein the first semantic embedding model is used to extract the semantic vector for topic recognition based on the input text;

[0031] (4.2) Input the topic semantic vector into the pre-trained keyword extraction model, thereby extracting several candidate topic words related to the topic semantic vector from the preset topic word set, wherein the keyword extraction model is used to extract several candidate topic words related to the topic semantic vector from the preset topic word set;

[0032] (4.3) Perform topic clustering on all candidate keywords and select the top few candidate keywords as the topic tags for the current content block;

[0033] (4.4) Input the main text of each content block into the pre-trained second embedding model to extract the semantic vector of the purpose, wherein the second embedding model is used to extract the semantic vector of the purpose attribute based on the input text;

[0034] (4.5) Input the use semantic vector into a pre-trained type classifier to extract the use type label, wherein the type classifier is used to match one or more use types and confidence levels from the use semantic vector;

[0035] (4.6) If the number of usage type tags is greater than the threshold, the top few usage type tags with the highest confidence are extracted as the usage type tags of the current content block;

[0036] (4.7) Add topic tags and usage type tags to each content block.

[0037] Furthermore, step (5) specifically includes:

[0038] (5.1) Calculate the quality score for each tag of each content block in the content block set:

[0039] ,

[0040] in, Indicates label Quality rating , Representing tags respectively The semantic accuracy and structural consistency are both within the range of , , The weights are represented by semantic accuracy, which is the semantic similarity between the tag and the standardized text context, and structural consistency, which is the degree of consistency between the tag and the chapter title, level, and global tag system.

[0041] (5.2) Determine if there are any labels whose quality scores are lower than the preset threshold. If so, execute (5.3); otherwise, execute (5.4).

[0042] (5.3) First, standardize the terminology in different types of tags to a standard format; second, add qualifiers or contextual constraints to tags with cross-domain ambiguity to eliminate ambiguity; third, adjust the hierarchical position of tags according to the knowledge base structure to ensure the regularity of the organizational structure; remove redundant or non-keyword parts of tags so that tags can be used directly as index items and retain their contextual relevance; during the optimization strategy execution, the total number of tags remains unchanged; return to step (5.2).

[0043] (5.4) Output the optimized labels.

[0044] Furthermore, step (6) specifically includes:

[0045] (6.1) Merge the main text of each content block with its corresponding optimized topic tags and usage type tags to form a semantic unit;

[0046] (6.2) Input each semantic unit into the embedding generation model that has been pre-trained and optimized for multi-domain semantics to obtain a high-dimensional semantic vector, wherein the embedding generation model is used to extract the high-dimensional semantic vector based on the input;

[0047] (6.3) For each high-dimensional semantic vector, attach the corresponding meta-information, which includes the chapter title, level, starting position, topic tag, purpose type tag, standardized text number, standardized text path and timestamp of the corresponding content block;

[0048] (6.4) Perform a consistency check on each high-dimensional semantic vector and its corresponding meta-information. Once the check passes, store it as a knowledge unit in the knowledge base.

[0049] Furthermore, step (6) is followed by the following step:

[0050] (7) Monitor changes in the original long document in real time, trigger the automated execution of steps (1) to (6), and use a multi-dimensional matching mechanism of structure and semantics for the received user query instructions.

[0051] Furthermore, step (7) specifically includes:

[0052] (7.1) Monitor file change events in real time to trigger the automated execution of steps (1) to (6);

[0053] (7.2) When a user query instruction is received, the user query instruction is converted into a query keyword vector;

[0054] (7.3) Match the query keyword vector with topic tags, usage type tags, and chapter titles, and store the matched content in the structure channel candidate set;

[0055] (7.4) Match the query keyword vector with the high-dimensional semantic vector, and store the high-dimensional semantic vectors with the highest similarity into the semantic channel candidate set;

[0056] (7.5) Merge the candidate sets of structural channels and semantic channels to obtain the final candidate set;

[0057] (7.6) Based on the final candidate set, find the corresponding metadata and generate the query output results based on the metadata;

[0058] (7.7) Return the query output results to the user and write them to the pipeline log at the same time.

[0059] A computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0060] Compared with existing technologies, the beneficial effects of this invention are as follows: The directory-aware long document knowledge base construction method and program product proposed in this invention fully utilize directory structure awareness to ensure semantic integrity, and combine a dual-tag system with intelligent optimization loops to improve tag quality and retrieval efficiency. This significantly improves the knowledge base construction efficiency, retrieval accuracy, and result traceability, making it suitable for the intelligent processing and application of long and complex documents such as academic monographs, technical documents, and policy regulations. It fills the technological gaps in existing technologies regarding structure fidelity, tag optimization, and automated integration, providing a solid knowledge foundation for industry applications. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the method for constructing a long document knowledge base based on directory awareness proposed in an embodiment of the present invention.

[0062] Figure 2 This is a flowchart of step S1 proposed in an embodiment of the present invention;

[0063] Figure 3 This is a flowchart of step S2 proposed in an embodiment of the present invention;

[0064] Figure 4 This is a flowchart of step S3 proposed in an embodiment of the present invention;

[0065] Figure 5 This is a flowchart of step S proposed in an embodiment of the present invention;

[0066] Figure 6 This is a flowchart of step S5 proposed in an embodiment of the present invention;

[0067] Figure 7 This is a flowchart of step S6 proposed in an embodiment of the present invention;

[0068] Figure 8 This is a flowchart of step S7 proposed in an embodiment of the present invention;

[0069] Figure 9 A comparison chart of ablation experimental data is provided for embodiments of the present invention. Detailed Implementation

[0070] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0071] Example 1

[0072] This invention provides a method for constructing a directory-aware long document knowledge base, such as... Figure 1 As shown, it includes the following steps:

[0073] S1. Obtain several original long documents and convert all original long documents into standardized text with unified encoding processing and removal of worthless information to form a standardized text set.

[0074] Among them, such as Figure 2 As shown, this step specifically includes:

[0075] S1.1 Obtain several original long documents Each document can contain multiple chapters, different encoding methods, and potential unstructured information such as page numbers, watermarks, headers and footers, page breaks, and special characters. Each original long document is parsed and its text encoding format is converted into encoded plain text using a unified text encoding format; this can be achieved by calling a pre-set text conversion module. Long documents in formats including but not limited to PDF, DOCX, TXT, and HTML will be uniformly converted into UTF-8 encoded plain text.

[0076] S1.2 For encoded plain text, the corresponding noise patterns are identified using each noise pattern in a preset noise pattern set, and the identified noise is removed to obtain clean text. The noise pattern set includes several noise patterns, specifically... Each noise pattern corresponds to a regular expression rule, used to match and identify the corresponding type of worthless noise. These represent noise patterns for page number markers, headers, footers, and advertising copy, respectively. Specifically, they are used to identify redundant characters in page number markers, headers, footers, and advertising copy that are meaningless to the actual content. Then, preset filtering functions can be called. Obtain pure text In extreme cases, if the proportion of noisy content in a document is too high, the document's anomaly tags will be recorded for secondary processing.

[0077] S1.3. A preset symbol mapping table is used to replace all special symbols in the clean text with unified standard encoded symbols, resulting in normalized text. The symbol mapping table stores each special symbol and its corresponding unified standard encoded symbol; the specific symbol mapping table can be configured. ,in Special symbols representing the original (such as various forms of quotation marks, dashes of different widths, full-width / half-width symbols, special spaces, etc.). This is a unified standard. Encode the symbols. Then call the preset mapping function: Replace these special symbols one by one to obtain the normalized text. This eliminates differences in symbol encoding and ensures consistency in symbol representation across the entire dataset.

[0078] S1.4. Compare the normalized text length with the original long document length and calculate the relative error. If the relative error exceeds the preset error threshold... If the error is not found, the conversion process is deemed abnormal, the abnormality is recorded, and the process returns to step (1.1) for re-execution; if the relative error does not exceed the preset error threshold, the process is considered abnormal. If the conversion is successful, the normalized text will be added to the standardized text set. Medium. Relative error , These represent the normalized text length and the original long document length, respectively.

[0079] This step marks the beginning of the fully automated knowledge base construction process. Its technical objective is to transform complex, long documents from diverse sources and with varying formats into standardized, machine-parsable text representations. Simultaneously, it removes worthless information and unifies symbol encoding, thus providing consistent and clean data input for subsequent S2 and S3 processes. This step significantly reduces parsing error rates through format standardization and noise filtering, minimizes abnormal branches in subsequent modules, and improves the overall pipeline's execution efficiency and stability.

[0080] S2. For each standardized text, extract the chapter titles from the text, and treat each chapter title, its location, and its level as a chapter title node. Generate a chapter tree from all chapter title nodes according to the level of the chapter title.

[0081] like Figure 3 As shown, this step specifically includes:

[0082] S2.1 For each standardized text in the standardized text set Scan line by line, using a set of chapter title matching patterns during the scan. Determine if it is a chapter title, and generate a chapter title node for each identified chapter title, including its text, location, and level, and store it in the chapter title node list.

[0083] The set of chapter title matching patterns includes several chapter title matching patterns, specifically: Matching pattern for each chapter title This invention combines regular expressions and a finite state machine (FSM) to parse text and identify corresponding chapter titles, as well as inferring the chapter title's level based on its number and indentation. For example, it matches "Chapter X", "Section XX", "I," etc., and supports Arabic numerals, Roman numerals (uppercase / lowercase), Chinese numerals, and their mixed forms. This invention combines regular expression matching with a FSM, enabling it to parse both standard titles and titles with additional descriptions (such as "Chapter 3 Technical Specifications (Revised Edition)"). The matching results generate a list of chapter title nodes. , These represent the chapter title text, the level, and the location, respectively.

[0084] S2.2 For each standardized text, generate a chapter tree by combining all chapter title nodes in the chapter title node list according to the level and position of each chapter title node in the chapter title node list.

[0085] For example =1 indicates that the level is a chapter. =2 indicates that the level is a section. =3 indicates the level is a section. Then the preset tree construction function is called. This function inserts nodes into the chapter tree structure sequentially according to their hierarchical relationship and order within the document. Chapter Tree Defined as: , This is a position index, using a character offset to indicate the starting position of the heading in the original text. The position index plays a crucial role in precise boundary positioning during subsequent chunking. During the construction process, if discontinuous numbering or missing headings are detected, the system will automatically complete the intermediate nodes using hierarchical pattern rules and add position information to prevent ambiguity when headings at the same level are repeated.

[0086] The technical goal of this step is to extract the document chapter hierarchy from a preprocessed collection of high-quality plain text and organize it into an ordered chapter tree structure, providing a structural boundary foundation for subsequent S3 processing. Through complete title detection, hierarchy determination, structured storage, and node summary generation, the system can effectively avoid semantic breaks caused by pure length segmentation. Simultaneously, this structural information can serve as a path index during the retrieval phase, enabling original text tracing.

[0087] S3. For each standardized text, the chapter title node position of the chapter tree is used as the content boundary to divide it into several content blocks. Each content block includes the main text content corresponding to the chapter title, the starting position, and the level.

[0088] like Figure 4 As shown, step S3 specifically includes:

[0089] S3.1 For each chapter title node in each chapter tree, extract the content between the current chapter title node and the next chapter title node from the corresponding standardized text as the body text of the current chapter title. If the node is the last chapter, the interval ends at the end of the document, and is encapsulated together with the chapter title text and the level it belongs to into a structure block. These structural blocks are formed based on the natural boundaries of the original directory structure, ensuring that the parent-child node relationship is not broken.

[0090] S3.2 Perform length checks on the text content of each structure block. When the length... Greater than the preset length threshold When this happens, the main text is divided into several sub-contents using preset semantic pause points, and each sub-content, along with its chapter title text and hierarchical level, is encapsulated into a sub-structure block. Specifically, the set of semantic pause points that will be enabled can be set: This set corresponds to natural semantic breakpoint rules for different languages, used to split long blocks of content without compromising semantic integrity. The resulting sub-blocks inherit the title and hierarchy information of the parent chapter and are numbered sequentially. Where j represents the sequence number of the sub-block in the original structure block.

[0091] S3.3 Add the start and end positions of the main text content in the standardized text to each structure block or substructure block, and store them as a content block in the content block set according to their order in the standardized text.

[0092] Specifically, , Content block collection Each entry in the collection retains its respective chapter tag (title, level), content text, and its starting and ending positions in the original text. The chunked collection is globally sorted according to the original text order, ensuring that downstream processes can generate and store tags in the same order as the original text.

[0093] This step utilizes the already parsed chapter tree. The provided precise structural boundaries segment long documents into content blocks that are granularly balanced and semantically complete, providing stable and independently semantically annotated text units for subsequent S4 processing. This method combines two strategies: "structure-driven segmentation" and "semantic pause point segmentation," ensuring that the segmentation boundaries are strictly consistent with the original directory structure while avoiding semantic fragmentation caused by pure length segmentation, thus exhibiting high robustness in multi-format document scenarios.

[0094] S4. Based on the main text content of each content block, extract topic tags and usage type tags. The topic tags are used to represent the topic type of the corresponding content block, and the usage type tags are used to represent the usage attribute of the corresponding content block.

[0095] like Figure 5 As shown, S4 specifically includes:

[0096] S4.1, Set the main text content of each content block Input the first pre-trained embedding model Extract the topic semantic vector The first semantic embedding model is used to extract semantic vectors for topic recognition based on the input text, which can be specifically represented as follows: = ;

[0097] S4.2, Convert the topic semantic vector Input pre-trained keyword extraction model In this way, from the preset keyword set The keyword extraction model extracts several candidate keyword terms related to the topic semantic vector from a preset set of keyword terms. Specifically, it can be expressed as ;

[0098] S4.3, For all candidate keywords Perform topic clustering and select the top The candidate keywords will be used as the topic tags for the current content block. Specifically, it can be expressed as ;

[0099] S4.4, extract the main text content of each content block. Input the pre-trained second embedding model Extract the semantic vector of usage The second embedding model is used to extract a semantic vector of the purpose attribute from the input text, which can specifically represent... ;

[0100] S4.5, Use semantic vector Input pre-trained type classifier Extract usage type tags from the middle. The type classifier is used to match one or more usage types and their confidence levels from the usage semantic vector, which can be represented as follows: ;

[0101] S4.6, If the usage type label Quantity greater than threshold Then extract before The usage type tag with the highest confidence level is used as the usage type tag for the current content block, specifically represented as the usage type tag. This process ensures that the tags reflect the categorization attributes of the content in terms of document structure and information purpose.

[0102] S4.7 Add topic tags and usage type tags to each content block to obtain the content blocks. .

[0103] The goal of this step is to generate two types of tags for each content block from the S3 output—a Topic Tag and a Type Tag—to comprehensively characterize the content's two-dimensional attributes. This dual-tag generation not only improves the accuracy of retrieval and matching but also provides an initial annotation foundation for subsequent S5 iterations and optimizations.

[0104] S5. Quantify the scores of all tags and optimize the scores of tags below the threshold according to rules until the scores of all tags are above the threshold.

[0105] like Figure 6 As shown, this step specifically includes:

[0106] S5.1 Calculate the quality score for each tag of each content block in the content block set:

[0107] ,

[0108] in, Indicates label Quality rating , Representing tags respectively The semantic accuracy and structural consistency are both within the range of , , The weight represents the semantic similarity between the tag and the standardized text context. It measures the degree of matching between the tag and the corresponding chunk content topic, and is calculated based on the BERT Tagging Evaluation algorithm. For high accuracy, when For medium accuracy, when Low accuracy requires verification; structural consistency refers to the degree of consistency between the tag and its corresponding chapter title, hierarchy, and global tag system; weight parameters are set to... and This system prioritizes semantic accuracy in tag quality assessment, combining semantic and structural dimensions to create an objectively quantifiable method for evaluating tag quality.

[0109] S5.2 Determine if there are any labels whose quality scores are lower than the preset threshold of 0.6. If yes, proceed to S5.3; otherwise, proceed to S5.4.

[0110] S5.3. First, standardize the terminology in different types of tags to a standard format, specifically unifying colloquialisms, abbreviations, and spelling variations. Second, add qualifiers or contextual constraints to tags with cross-domain ambiguity to eliminate ambiguity. Third, adjust the hierarchical position of tags according to the knowledge base structure to ensure the regularity of the organizational structure. Remove redundant or non-keyword parts from tags so that tags can be used directly as index items while retaining their contextual relevance. During the optimization strategy execution, the system will also adhere to quantity consistency constraints: regardless of integration, retention, or replacement, the final total number of tags must be consistent with the initial number. That is, deleting a tag must generate an equal number of replacement tags, retaining tags does not change the number, and replacement operations must be completed one-to-one to ensure that the knowledge base retrieval coverage does not decrease and that each knowledge unit always has a corresponding index path. Return to step S5.2 and repeat the closed loop of "scoring - optimization - scoring" until the quality score of all tags is higher than the threshold of 0.6.

[0111] S5.4 Output optimized tags The optimized topic and type tags are denoted as follows: and And sort them globally according to the original text order to form a collection of content blocks. .

[0112] This step involves calculating the dual-label scores, performing rule-based optimization, and iterative loops on the S4 output to ensure the label set simultaneously meets the standards for semantic accuracy, structural consistency, coverage, and quantity control. The process uses a quantifiable evaluation formula to define label quality and initiates a set of optimization actions for labels below a threshold, ultimately outputting a globally optimized label set. This optimization loop not only corrects semantic drift and redundancy that may occur in the initial tag generation stage, but also unifies the tag set by combining the global context, thereby significantly improving the matching degree of subsequent knowledge base storage and retrieval.

[0113] S6. Combine the main text of each content block with the optimized topic tags and usage type tags into a semantic unit, extract a high-dimensional semantic vector from the semantic unit, and combine the high-dimensional semantic vector with the corresponding meta-information to form a knowledge unit, which is then stored in the knowledge base.

[0114] like Figure 7 As shown, this step specifically includes:

[0115] S6.1 Merge the main text of each content block with its corresponding optimized topic tags and usage type tags to form a semantic unit. ;

[0116] S6.2 Input each semantic unit into the embedding generative model that has been pre-trained and optimized for multi-domain semantics. This yields a high-dimensional semantic vector. The embedding generation model is used to extract high-dimensional semantic vectors based on the input.

[0117] S6.3, Each high-dimensional semantic vector Mount the corresponding metadata : Each element represents the chapter title, level, starting position, topic tag, usage type tag, standardized text number, standardized text path, and timestamp of the corresponding content block. This serves as a unique identifier for the original document. This is the path to the original file in the file system. The timestamp generated or updated for this entry serves to strictly bind the vector representation with the original text structure and tag system, achieving three-dimensional traceability of "semantic vector - tag - original text position". This allows the source text and context to be traced while the search results are returned, improving interpretability and reusability.

[0118] S6.4. Combine each high-dimensional semantic vector with its corresponding meta-information. Perform a consistency check; if the check passes, store it as a knowledge unit in the knowledge base. Define the knowledge base storage structure: The knowledge base employs a support vector search storage engine and establishes three types of index tables: tag index, chapter index, and full-text index, to support vector similarity search, tag search, and structured search. Simultaneously, the knowledge base performs consistency checks on all newly added entries (ensuring...). Unique and the set of tags and (Document consistency) ensures data integrity and high availability of retrieval. Ultimately, under any retrieval request, the knowledge base can simultaneously return results using both vector semantic matching and tag structure matching, achieving a high-precision, high-recall response.

[0119] This step receives the final high-quality tag set file output by S5, vectorizes it together with the original text's block content and structural information, attaches metadata, and finally stores it in the knowledge base according to a unified data structure. This process achieves precise matching through high-dimensional semantic vector representation and preserves multi-dimensional associations between tags and the original text, chapters, structure, etc., through metadata attachment, thereby achieving high recall and high precision in retrieval and generation scenarios.

[0120] S7. Monitor changes to the original long document in real time, trigger the automated execution of steps S1 to S6, and use a multi-dimensional matching mechanism of structure and semantics for the received user query commands.

[0121] like Figure 8 As shown, this step specifically includes:

[0122] S7.1 Real-time monitoring of file change events triggers the automated execution of steps S1 to S6;

[0123] S7.2 When a user query command is received At that time, the user's query command will be executed. Convert to query keyword vector , An embedding model used to convert text into vectors;

[0124] S7.3 Match the query keyword vector with topic tags, usage type tags, and chapter titles, and store the matched content in the structural channel candidate set. ;

[0125] S7.4 Match the query keyword vector with the high-dimensional semantic vector, and store the high-dimensional semantic vectors with the highest similarity into the semantic channel candidate set. ;

[0126] S7.5. Merge the structural channel candidate set and the semantic channel candidate set to obtain the final candidate set. , The fusion function performs the following calculations:

[0127] Calculate the tag hit rate of the structural channel candidate set and the semantic channel candidate set. And the vector similarity between the structural channel candidate set and the semantic channel candidate set. Weighted average:

[0128]

[0129] After sorting, the top few are selected as the final candidate set;

[0130] S7.6. Based on the final candidate set, retrieve the corresponding metadata and generate the query output results based on the metadata. The query output result generation model performs enhanced generation based on the provided highly relevant context, including tasks such as direct answering, summarizing, reporting, and tag recommendation. The final output results are encapsulated as follows: in To generate text, For related tags and document information, For integrated scoring, This is the original source path;

[0131] S7.7 Return the query output results to the user and simultaneously write them to the pipeline log.

[0132] This step integrates end-to-end automated task scheduling and Retrieval-Augmented Generation (RAG) to automate the execution of modules S1 through S6 without human intervention. In the retrieval scenario, it employs a multi-dimensional matching mechanism combining structure and semantics, integrating highly relevant content from the retrieval end with the generation model to significantly improve the reliability and accuracy of the final generated results. This module addresses the pain points of traditional processes, such as excessive human intervention, long construction cycles, and a lack of in-depth verification of retrieval results. Through automated pipeline monitoring, file change triggering, vector similarity and tag matching fusion, and traceable output mechanisms, it achieves efficient end-to-end scheduling and high-quality content generation.

[0133] Example 2

[0134] This invention also provides a computer program product, such as an app on a mobile phone or tablet, or an installer on a computer. This product includes a computer program / instructions that, when executed by a processor, implement the method described in Embodiment 1. The code for the computer-executable program used to perform the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] To verify the independent technical effects and necessity of each key step S1 to S7 in the directory-aware long document knowledge base construction and tag intelligent optimization method proposed in this invention, an ablation study was used to conduct a performance comparison test on each step. The aim was to quantitatively demonstrate that the absence of any processing step would cause a significant decrease in knowledge base retrieval performance, thereby proving that the entire process structure of this invention has a synergistic gain effect.

[0136] Dataset: A collection of long and complex documents from the humanities and ethics field, totaling approximately 400,000 characters.

[0137] Experimental systems: Baseline system: Complete execution of the entire S1~S7 process; Ablation variants: Each variant removes a specific step from the process while keeping the remaining steps consistent, such as:

[0138] S1: Remove document preprocessing and directly input raw data.

[0139] S2: Skip directory structure parsing and use fixed-length segmentation instead.

[0140] S3: Remove mixed segmentation and replace it with fixed-length segmentation.

[0141] S4: Remove double tags and use only single tags.

[0142] S5: Remove tag optimization and directly use the initial tags.

[0143] S6: Remove vectorization and metadata loading, leaving only plain text.

[0144] Evaluation Metric: Recall: Measured by the proportion of correctly matched knowledge units in the search results out of all correctly matched knowledge units. The calculation formula is as follows: ,in, (True Positive) represents the number of knowledge units retrieved that match the standard answer. (False Negative) represents the number of standard answer knowledge units that were not retrieved.

[0145] Testing Methodology: For each system version, a retrieval task was performed on the same test query set, and the recall rate was recorded. To avoid the influence of randomness, all results were the average of three measurements.

[0146] Table 1. Test results of the baseline system and various ablation versions

[0147] Experimental version Process steps change Recall rate (%) Relative baseline difference (%) Baseline S1+S2+S3+S4+S5+S6+S7 73.23 0 Remove step S1 Remove document preprocessing module 60.21 -13.02 Remove step S2 Remove directory structure parsing module 58.46 -14.77 Remove S3 step Remove the hybrid block module 64.26 -8.97 Remove step S4 Remove double-label module 65.14 -8.09 Remove S5 steps Remove tag intelligent optimization module 69.32 -3.91 Remove step S6 Remove the vectorization and metadata mounting module 61.31 -11.92

[0148] Figure 9 This invention provides a comparison chart of ablation experimental data.

[0149] Overall trend: The complete process (Baseline) has the highest recall rate (73.23%), and removing each key step will cause varying degrees of decline.

[0150] The most significantly affected components: Removing S1 (-13.02%), S2 (-14.77%), and S6 (-11.92%) had a significant impact on performance, indicating that preprocessing, structure parsing, vectorization, and metadata loading play a crucial role in retrieval performance.

[0151] Moderate impact: Removing S3 (-9.00%) and S4 (-8.09%) also significantly reduces performance, indicating the importance of hybrid segmentation and dual-label system for structural integrity and retrieval accuracy.

[0152] Minor impact: Removing S5 (-3.91%) has the least impact, but tag optimization can maintain system performance stability in the long term.

[0153] Synergistic effect: The cooperation of all modules makes the overall system significantly more effective than any missing solution, proving that the whole process design of this invention is necessary and has synergistic benefits.

[0154] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A method for constructing a long document knowledge base based on directory awareness, characterized in that, Includes the following steps: (1) Obtain several original long documents and convert all original long documents into standardized text with unified encoding processing and removal of worthless information to form a standardized text set; (2) For each standardized text, extract the chapter titles in the text, and take each chapter title, its position and its level as a chapter title node, and generate a chapter tree according to the level of the chapter title; (3) For each standardized text, the chapter title node position of the chapter tree is used as the content boundary to divide it into several content blocks. Each content block includes the main text content corresponding to the chapter title, the starting position and the level it belongs to; (4) Based on the main text of each content block, extract topic tags and usage type tags. The topic tags are used to represent the topic type of the corresponding content block, and the usage type tags are used to represent the usage attribute of the corresponding content block. (5) Quantify the scores of all tags and optimize the scores of tags below the threshold by regularization until the scores of all tags are above the threshold; (6) Combine the main text of each content block with the optimized topic tags and usage type tags into a semantic unit, and extract a high-dimensional semantic vector from the semantic unit. Combine the high-dimensional semantic vector with the corresponding meta-information to form a knowledge unit and store it in the knowledge base. The meta-information includes the chapter title, level, starting position, topic tags, usage type tags, standardized text number, standardized text path and timestamp of the corresponding content block. Step (4) specifically includes: (4.1) Input the main text of each content block into the pre-trained first embedding model to extract the topic semantic vector, wherein the first embedding model is used to extract the semantic vector for topic recognition based on the input text; (4.2) Input the topic semantic vector into the pre-trained keyword extraction model, thereby extracting several candidate topic words related to the topic semantic vector from the preset topic word set, wherein the keyword extraction model is used to extract several candidate topic words related to the topic semantic vector from the preset topic word set; (4.3) Perform topic clustering on all candidate keywords and select the top few candidate keywords as the topic tags for the current content block; (4.4) Input the main text of each content block into the pre-trained second embedding model to extract the semantic vector of the purpose, wherein the second embedding model is used to extract the semantic vector of the purpose attribute based on the input text; (4.5) Input the use semantic vector into a pre-trained type classifier to extract the use type label, wherein the type classifier is used to match one or more use types and confidence levels from the use semantic vector; (4.6) If the number of usage type tags is greater than the threshold, the top few usage type tags with the highest confidence are extracted as the usage type tags of the current content block; (4.7) Add topic tags and usage type tags to each content block.

2. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (1) specifically includes: (1.1) Obtain several original long documents, parse each original long document and convert its text encoding format to a uniform text encoding format for plain text; (1.2) For encoded plain text, the corresponding noise pattern is identified by each noise pattern in the preset noise pattern set, and the identified noise is removed to obtain pure text. The noise pattern set includes several noise patterns, and each noise pattern corresponds to a regular expression rule, which is used to match and identify the corresponding type of worthless information noise. (1.3) Use a preset symbol mapping table to replace each special symbol in the clean text with a unified standard encoding symbol to obtain normalized text, wherein the symbol mapping table stores each special symbol and its corresponding unified standard encoding symbol; (1.4) Compare the normalized text length with the original long document length and calculate the relative error. If the relative error exceeds the preset error threshold, it is determined that there is an abnormality in the conversion process. Record the abnormality and return to step (1.1) to re-execute. If the relative error does not exceed the preset error threshold, it is determined that the conversion is normal and the normalized text is added to the standardized text set.

3. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (2) specifically includes: (2.1) For each standardized text in the standardized text set, scan it line by line. During the scan, use the chapter title matching pattern set to determine whether it is a chapter title. Generate a chapter title node by taking the text, position and level of each identified chapter title and store it in the chapter title node list. The chapter title matching pattern set includes several chapter title matching patterns. Each chapter title matching pattern is a combination of chapter title recognition regular expression and finite state machine, which is used to parse the text and identify the corresponding type of chapter title and infer the level of the chapter title based on the chapter title number and indentation. (2.2) For each standardized text, generate a chapter tree by taking all chapter title nodes in the chapter title node list according to the level and position of each chapter title node in the chapter title node list; if a discontinuous chapter title sequence number or a missing chapter title is detected, the missing node is automatically filled in using the hierarchical pattern rules, and position information is added to distinguish when chapter titles at the same level are repeated.

4. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (3) specifically includes: (3.1) For each chapter title node in each chapter tree, extract the content between the current chapter title node and the next chapter title node from the corresponding standardized text as the body text of the current chapter title, and encapsulate it together with the chapter title text and the level into a structure block; (3.2) Perform length detection on the main text content of each structural block. When the length is greater than the preset length threshold, the main text content is split into several sub-contents using the preset semantic pause point, and each sub-content, the chapter title text of the structural block and the level of the structural block are encapsulated into a sub-structural block. (3.3) Add the start and end positions of the main text content in the standardized text for each structure block or substructure block, and store them in the content block set in the order of their positions in the standardized text.

5. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (5) specifically includes: (5.1) Calculate the quality score for each tag of each content block in the content block set: ， in, Indicates label Quality rating , Representing tags respectively The semantic accuracy and structural consistency are both within the range of , , The weights are represented by semantic accuracy, which is the semantic similarity between the tag and the standardized text context, and structural consistency, which is the degree of consistency between the tag and the chapter title, level, and global tag system. (5.2) Determine if there are any labels whose quality scores are lower than the preset threshold. If so, execute (5.3); otherwise, execute (5.4). (5.3) First, standardize the terminology in different types of tags to a standard format; second, add qualifiers or contextual constraints to tags with cross-domain ambiguity to eliminate ambiguity; third, adjust the hierarchical position of tags according to the knowledge base structure to ensure the regularity of the organizational structure; remove redundant or non-keyword parts of tags so that tags can be used directly as index items and retain their contextual relevance; during the optimization strategy execution, the total number of tags remains unchanged; return to step (5.2). (5.4) Output the optimized labels.

6. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (6) specifically includes: (6.1) Merge the main text of each content block with its corresponding optimized topic tags and usage type tags to form a semantic unit; (6.2) Input each semantic unit into the embedding generation model that has been pre-trained and optimized for multi-domain semantics to obtain a high-dimensional semantic vector, wherein the embedding generation model is used to extract the high-dimensional semantic vector based on the input; (6.3) For each high-dimensional semantic vector, attach the corresponding meta-information, which includes the chapter title, level, starting position, topic tag, purpose type tag, standardized text number, standardized text path and timestamp of the corresponding content block; (6.4) Perform a consistency check on each high-dimensional semantic vector and its corresponding meta-information. Once the check passes, store it as a knowledge unit in the knowledge base.

7. The method for constructing a long document knowledge base based on directory awareness according to claim 1, characterized in that, Step (6) is followed by the following steps: (7) Monitor changes in the original long document in real time, trigger the automated execution of steps (1) to (6), and use a multi-dimensional matching mechanism of structure and semantics for the received user query instructions.

8. The method for constructing a long document knowledge base based on directory awareness according to claim 7, characterized in that, Step (7) specifically includes: (7.1) Monitor file change events in real time to trigger the automated execution of steps (1) to (6); (7.2) When a user query instruction is received, the user query instruction is converted into a query keyword vector; (7.3) Match the query keyword vector with topic tags, usage type tags, and chapter titles, and store the matched content in the structure channel candidate set; (7.4) Match the query keyword vector with the high-dimensional semantic vector, and store the high-dimensional semantic vectors with the highest similarity into the semantic channel candidate set; (7.5) Merge the candidate sets of structural channels and semantic channels to obtain the final candidate set; (7.6) Based on the final candidate set, find the corresponding metadata and generate the query output results based on the metadata; (7.7) Return the query output results to the user and write them to the pipeline log at the same time.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images