Orthogonal structured knowledge extraction method and system based on configurable semantic slots

Abstract

This invention discloses a method and system for extracting orthogonal structured knowledge based on configurable semantic slots. The method includes: recursively parsing a target technical document to generate a set of chapters and removing noise and duplicates; determining the primary topic of each chapter based on a multi-path recall mechanism; calculating the matching score of each chapter on each candidate semantic slot according to classification rules, and routing it to the orthogonal knowledge type semantic slot with the highest matching score; determining local structural clues based on entity recognition rules and mapping them to entity types to construct an initial knowledge topology; employing a multi-stage extraction strategy to perform feature transformation on the chapter text, losslessly converting it into standardized atomic fact records and using them as terminal leaf nodes to be directionally filled under the corresponding entity type nodes; and fusing and outputting multi-level progressive structured knowledge topology data. This invention achieves structured and orthogonal knowledge organization, improving the readability, maintainability, and retrieval of knowledge.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, specifically to a method and system for extracting orthogonal structured knowledge based on configurable semantic slots. Background Technology

[0002] With the ever-growing volume of professional technical documents such as software development documents, chip design documents, industrial manufacturing documents, financial documents, and legal documents, a large amount of knowledge is scattered across documents from different sources in text, table, and semi-structured formats. How to automatically convert the knowledge in massive amounts of unstructured technical documents into searchable, reusable, and reasonable structured knowledge has become an important research direction in the fields of knowledge engineering, intelligent question answering, and knowledge base construction.

[0003] Existing technologies mainly include rule-based knowledge extraction methods, direct summarization methods based on large language models, and methods based on knowledge graph embedding. Rule-based methods suffer from poor generalization ability, high maintenance costs, and difficulty in expressing complex knowledge relationships; large language model-based methods suffer from uncontrollable output structure, inconsistent fact granularity, and difficulty in tracing sources; and knowledge graph embedding methods suffer from high training costs, poor interpretability, and difficulty in expressing complex conditional knowledge. Furthermore, existing solutions generally lack automatic detection and explanation mechanisms for factual conflicts from different sources, making it difficult to achieve highly reliable knowledge organization.

[0004] Therefore, it is necessary to propose an orthogonal structured knowledge extraction method and system based on configurable semantic slots to solve the technical problems existing in the prior art, such as chaotic knowledge organization structure, coupling of knowledge types, untraceable fact sources, inability to automatically handle conflicting facts, and poor cross-domain adaptability, so as to achieve high-quality structured knowledge construction for multi-source technical documents. Summary of the Invention

[0005] This invention provides an orthogonal structured knowledge extraction method and system based on configurable semantic slots, which solves the problems of low extraction accuracy, fragmented contextual semantics, excessive computational overhead, and inability to self-heal conflicts in existing technologies when processing massive, multi-source, and heterogeneous technical documents.

[0006] To address the aforementioned problems, this invention provides an orthogonal structured knowledge extraction method based on configurable semantic slots, comprising:

[0007] Obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and body content, and generate a chapter set containing multiple chapters;

[0008] The chapter set is then subjected to noise reduction and deduplication to obtain a filtered chapter set.

[0009] The primary subject to which each chapter belongs in the filtered chapter set is determined based on a multi-path recall mechanism.

[0010] For each chapter corresponding to the main subject, calculate its matching score on each candidate semantic slot according to the pre-configured classification rules, and route the chapter to the orthogonal knowledge type semantic slot with the highest matching score;

[0011] Based on preset entity recognition rules, the local structural clues of the chapters within the semantic slots are determined, and the local structural clues are mapped to the corresponding entity types to construct an initial knowledge topology structure containing the main subject, semantic slots, chapters, and entity types.

[0012] A multi-stage extraction strategy is adopted to extract atomic facts. Feature transformation is performed on the text content corresponding to each node in the initial knowledge topology, and the text content is losslessly converted into standardized atomic fact records. The standardized atomic fact records are used as terminal leaf nodes and are directionally filled into the corresponding entity type nodes in the initial knowledge topology.

[0013] The atomic fact records are fused with their corresponding chapters, semantic slots, and main themes to output multi-level, progressively structured knowledge topology data.

[0014] This invention also provides an orthogonal structured knowledge extraction system based on configurable semantic slots, and a method for implementing the above technical solutions, comprising:

[0015] The document parsing module is used to obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and the main text content, and generate a chapter set containing multiple chapters;

[0016] The preprocessing module is used to perform noise reduction and deduplication on the chapter set to obtain a filtered chapter set;

[0017] The topic determination module is used to determine the primary topic to which each chapter in the filtered chapter set belongs, based on a multi-path recall mechanism.

[0018] The routing module is used to calculate the matching score of each chapter corresponding to each main subject on each candidate semantic slot according to the pre-configured classification rules, and to route the chapter to the orthogonal knowledge type semantic slot with the highest matching score.

[0019] The topology building module is used to determine the local structural clues of the chapters within the semantic slots based on preset entity recognition rules, and to map the local structural clues to the corresponding entity types, thereby constructing an initial knowledge topology structure that includes the main subject, semantic slots, chapters, and entity types.

[0020] The fact extraction module is used to perform atomic fact extraction using a multi-stage extraction strategy, perform feature transformation on the text content corresponding to each node in the initial knowledge topology, convert the text content into standardized atomic fact records without loss, and use the standardized atomic fact records as terminal leaf nodes to fill the corresponding entity type nodes in the initial knowledge topology.

[0021] The output module is used to merge the atomic fact records with their corresponding chapters, semantic slots and main themes to output multi-level progressive structured knowledge topology data.

[0022] Compared with the prior art, the beneficial effects of the present invention include:

[0023] (1) This scheme achieves orthogonal physical isolation of multi-source heterogeneous technical documents, blocking semantic drift. By constructing an initial topology structure consisting of "main subject - semantic slot - chapter - entity type", the extracted atomic facts are subject to mandatory constraints and are used as the leaf nodes at the very end, which are then oriented, filled, and attached to the specified entity type nodes. The preceding topology applies strong context boundary constraints to the subsequent extraction actions, and different types of technical knowledge are strictly orthogonally organized, eliminating knowledge confusion, semantic drift, and data pollution from the underlying architecture, and ensuring the purity of the multi-level progressive network structure.

[0024] (2) This solution addresses the coexistence of highly structured text (such as register bit field tables and interface parameter matrices) and unstructured free narrative text in technical documents by adopting a multi-stage diversion extraction mechanism. This avoids the global computational loss of large models and effectively controls the system's processing power and hardware storage costs without sacrificing the hard premise of recall and accuracy of deep extraction of unstructured text.

[0025] (3) The structured knowledge topology data finally output by this scheme will rigidly serialize and cascade the six layers of heterogeneous nodes of "topic-semantic slot-chapter-entity-fact-evidence" according to a strict topological order, so that the overall knowledge base naturally has a pointer network for reverse tracing. It automatically extracts highly structured technical knowledge from technical documents of multiple sources and types, and realizes high engineering credibility of the extracted results as they are verifiable and traceable.

[0026] (4) This scheme introduces an adaptive arbitration mechanism based on the technical boundary attribute, provides interpretable multi-source knowledge conflict disambiguation and multi-party evidence chain fusion capabilities, integrates an expert knowledge conflict disambiguation base in multi-level progressive verification, and can adaptively capture conflict feature keys and match the confidence of multi-party factual statements.

[0027] (5) When facing different technical fields or diverse heterogeneous document arrangement types, users do not need to make any intrusive code modifications to the underlying core algorithm. They only need to change the pre-configured semantic slot rules and classification feature words to easily and seamlessly adapt to the new document structure system and knowledge classification paradigm, which greatly improves the reusability and life cycle of the industrial-grade large-scale graph maintenance system. Attached Figure Description

[0028] Figure 1 A flowchart illustrating an orthogonal structured knowledge extraction method based on configurable semantic slots provided by this invention;

[0029] Figure 2 This is a schematic diagram of Topic routing multi-path recall and RRF fusion provided by the present invention;

[0030] Figure 3 A structured knowledge topology diagram illustrating the complete output of this invention;

[0031] Figure 4 This is a schematic diagram of an orthogonal structured knowledge extraction system based on configurable semantic slots. Detailed Implementation

[0032] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0033] Before introducing the embodiments of the present invention, the inventive concept will be explained first.

[0034] This invention provides a method for extracting orthogonal structured knowledge based on configurable semantic slots, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating the orthogonal structured knowledge extraction method based on configurable semantic slots, including:

[0035] Step S101: Obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and body content, and generate a chapter set containing multiple chapters;

[0036] Step S102: Perform noise reduction and deduplication on the chapter set to obtain a filtered chapter set;

[0037] Step S103: Determine the primary subject of each chapter in the filtered chapter set based on the multi-path recall mechanism;

[0038] Step S104: For each chapter corresponding to the main subject, calculate its matching score on each candidate semantic slot according to the pre-configured classification rules, and route the chapter to the orthogonal knowledge type semantic slot with the highest matching score;

[0039] Step S105: Based on the preset entity recognition rules, determine the local structural clues of the chapters within the semantic slots, and map the local structural clues to the corresponding entity types to construct an initial knowledge topology structure containing the main subject, semantic slots, chapters, and entity types;

[0040] Step S106: A multi-stage extraction strategy is used to extract atomic facts. Feature transformation is performed on the text content corresponding to each node in the initial knowledge topology structure. The text content is losslessly converted into standardized atomic fact records. The standardized atomic fact records are used as terminal leaf nodes and are directionally filled into the corresponding entity type nodes in the initial knowledge topology structure.

[0041] Step S107: Merge the atomic fact records with their corresponding chapters, semantic slots and main themes to output multi-level progressive structured knowledge topology data.

[0042] The method in this embodiment achieves structured and orthogonal knowledge organization through a configurable semantic slot system, improving the readability, maintainability, and retrieval of knowledge; through a multi-stage quality increment process, it effectively controls the cost of using LLM while ensuring quality; and through an automatic conflict detection and handling mechanism, it improves the credibility and verifiability of knowledge.

[0043] The above process will be described in detail below with a specific embodiment.

[0044] Step 1: Recursive hierarchical document parsing

[0045] The purpose of recursive hierarchical document parsing of target technical documents is to reconstruct the document's title tree structure by performing AST (Abstract Syntax Tree) preprocessing on unstructured or semi-structured documents. Specifically, this includes the following steps:

[0046] Step S11: Identify the heading lines in the document based on the inherent hierarchical markers (such as heading tags, font size, indentation, etc.), and determine the heading level based on the differences in the markers;

[0047] Step S12: Construct the parent-child relationship tree of the title node: When encountering a title of a higher level, add it as a child node to the current node; when encountering a title of a lower level or the same level, backtrack to the appropriate parent node.

[0048] Step S13: Associate each heading node with its body content and record the starting line number (line_start) and ending line number (line_end) in the document.

[0049] Meanwhile, for untitled or flat documents, virtual chapter nodes are created by paragraph, with each virtual node containing one or more paragraphs;

[0050] Step S14: Calculate the content hash value (content_hash). Use the SHA1 algorithm to perform a hash operation on the concatenated chapter path text and the main text content to generate the content hash value of the corresponding chapter, which will be used for subsequent deduplication.

[0051] It should be noted that, in addition to using the hash value mentioned above for deduplication, the chapter path text and the main text content can also be vectorized using the Embedding model. The similarity between the vectors can be used for deduplication. When the cosine similarity between the corresponding vectors of two chapter path texts and the main text content is greater than a preset threshold, the chapters are determined to be duplicated.

[0052] In the output data structure, each section contains fields as shown in Table 1:

[0053] Table 1. Fields, corresponding types, and descriptions for each chapter.

[0054]

[0055] The recursive hierarchical parsing process described above preserves the original structural information of the document, providing a structural foundation for subsequent knowledge organization.

[0056] Step 2: Chapter Denoising and Deduplication

[0057] The parsed sections are then denoised and deduplicated. The denoising method is as follows:

[0058] BM25 Score Calculation: Preset noise keywords and calculate BM25 scores. Use configurable filter items (such as predefined noise titles and keywords like "Table of Contents," "Legal Notice," and "Disclaimer") as query items (q), and calculate the BM25 relevance score between the main text content (d) of each section and the noise query item;

[0059] Filtering judgment: Pre-set a noise reduction threshold The BM25 relevance score for each section is compared with... A comparison is made. If the BM25 relevance score corresponding to a certain section is greater than or equal to (or exceeds) the denoising threshold... If the section is deemed to be a noisy section unrelated to core technical knowledge, then physical filtering or removal operations will be performed.

[0060] The deduplication method is as follows:

[0061] Duplicate content detection criteria: The unique content hash value obtained in the recursive hierarchical document parsing step 1 (i.e., the content_hash field in Table 1) is used as the unique identifier of the chapter content to identify duplicate content.

[0062] Duplicate content identification: If two or more sections have the same content_hash, they are determined to be sections with completely duplicate content.

[0063] Version selection and deduplication: For sections with completely duplicate content, based on pre-configured source priority rules, the version with the highest source reputation is retained, and other duplicates are filtered out.

[0064] Through the aforementioned noise reduction, deduplication, and version management mechanisms, irrelevant and duplicate content is removed, achieving efficient filtering of duplicate content and optimal version selection, thereby improving the quality of the knowledge base and providing a technical foundation for maintaining the quality of the knowledge base.

[0065] Step 3: Single-homed routing based on multi-path recall strategy

[0066] Each section is assigned a technical topic (hereinafter referred to as Topic, i.e., a predefined topic or category, such as product features, legal terms, financial data, user feedback, etc.). This embodiment uses a multi-path recall strategy to determine the technical topic corresponding to each section. The multi-path recall strategy includes structural prior recall, BM25 lexical recall, and Embedding semantic recall. Finally, ranking fusion is used to improve the accuracy of Topic routing.

[0067] The following section provides a detailed introduction to multi-path recall strategies and ranking integration.

[0068] (1) Structural prior recall

[0069] Since paths, filenames, and chapter titles are generally more reliable than keywords in the text, this method utilizes structured information such as document paths, filenames, and chapter paths to extract key technical terms. For example, the path "docs / api_manuals / xxx_API_Reference_Manual / ... / GPIO.md" is more indicative of GPIO than UART or PWM appearing occasionally in the text. This structured information is a direct output of step 1 (recursive hierarchical document parsing), providing the most reliable structured prior information for Topic routing.

[0070] In practice, this can be achieved by concatenating the first 1600 characters of source_path, doc_slug, heading_title, section_path_text, and body as the route text.

[0071] (2) BM25 lexical recall

[0072] The BM25 algorithm was used to calculate the relevance score between the section text and the keyword set of each topic.

[0073] The BM25 algorithm formula is expressed as follows:

[0074]

[0075] in, The section to be routed Feature keyword set of the target technology topic The overall score of lexical relevance between them; Represents the set of feature keywords Single feature keyword in; The term frequency represents the characteristic keywords. In the chapter The frequency of occurrence in the main text; Indicates the chapter The total text length of the main content; This represents the average document length. =1.5 is the word frequency saturation parameter; b=0.75 is the length normalization parameter; The characteristic keywords represent The inverse document frequency score is calculated using the following formula: , Total number of documents This indicates that the feature keywords are included. The number of chapters.

[0076] BM25 is suitable for handling rare technical terms (such as GPIO_DIR_INPUT, gpio_set_dir, AFIO, DMA2D, SDHOST, PDM, etc.), which are infrequent but highly distinguishable.

[0077] (3) Embedding semantic recall (optional enhancement)

[0078] In other embodiments, locally deployed embedding models (such as BGE-M3) are also used to convert section text and topic descriptions into vector representations, calculate cosine similarity, and perform semantic understanding of the documents.

[0079] Embedding recall can supplement synonyms and match mixed Chinese and English expressions (such as "pin direction configuration" and "GPIO direction", "power domain" and "power domain", "audio input" and "audio capture").

[0080] (4) Ranking Integration

[0081] The Reciprocal Rank Fusion (RRF) algorithm is used to fuse the multi-path recall results, as expressed by the formula:

[0082]

[0083] in, It is a constant (usually taken as 60). For document In the The ranking in the recall results.

[0084] Since RRF only uses ranking information and does not assume that the scores of different recallers are comparable, it can effectively integrate ranking results of different scales.

[0085] Finally, for each section, output the Top-K Topic candidates (usually K=3-5), with the Top 1 being the primary subject (single-subject principle) to avoid knowledge redundancy caused by copying the same content to multiple peripherals. Candidate scores are retained for subsequent auditing and manual review.

[0086] It's important to note that within the same Topic, different knowledge types must be stored in separate slots. Otherwise, the final knowledge structure will suffer from confusion between different categories of knowledge (such as entity attributes, operational processes, constraints, and definition descriptions), thus affecting knowledge orthogonality and retrieval efficiency. Figure 2 As shown, Figure 2 The diagram illustrates the process of integrating Topic routing multi-path recall with RRF.

[0087] Step 4: Orthogonal routing within a Topic's Slots

[0088] This solution routes the chapters to the most matching orthogonal knowledge type semantic slot. Specifically: First, the semantic slot (hereinafter referred to as slot) is configured.

[0089] Slots are defined by a configuration file (taxonomy), and the properties of the slot configuration file are shown in Table 2:

[0090] Table 2 Slot Configuration File

[0091]

[0092] The definition of each slot in the slot definition list is shown in Table 3:

[0093] Table 3 Definition of each slot

[0094]

[0095] Secondly, Slot routing determines the final slot for a section by comprehensively analyzing multiple signals, including chapter path, title, slot alias, matching degree of focused terms, and local keywords in the text content.

[0096] 1. Chapter Path Matching: By extracting the complete title chain from the root node to the current node, the hierarchical path structure of the document (such as paths containing specific structural types of words) can bind the chapter to the context of a specific technical component, providing a globally unique physical topology location for the originally isolated chapter and preventing cross-module confusion of subsequently extracted facts.

[0097] 2. Chapter Title Matching: This involves identifying the content characteristics of chapter titles (such as keywords related to specific knowledge categories or operations). Since technical document authors inherently include strong knowledge classification characteristics in their chapter titles, title matching can identify the technical attribute bias of a chapter with high confidence, even before the main text is parsed (e.g., directly determining whether it belongs to a structure definition or an API function), thus providing a decisive basis for classification.

[0098] 3. Slot Alias ​​and Focused Terminology Matching: This aims to resolve the issue of synonyms with different names. Because different vendors and technical backgrounds use different names for the same type of engineering knowledge in their documents (for example, for the orthogonal semantic slot "interface parameters," it might be called "Arguments" in document A, "Parameters" in document B, and "input-type bit field" in document C), this solution first configures an alias extension set for each preset orthogonal semantic slot. During routing decisions, based on the characteristics of the chapter title text, a string similarity metric (such as edit distance or word vector cosine similarity) is calculated between the title text and the current semantic slot's alias extension set. If the current chapter title matches any alias in the alias extension set, an alias matching signal is triggered, assigning a score to the corresponding alias matching dimension for that semantic slot.

[0099] 4. Local Keyword Matching in the Main Text: Local keywords refer to specific terms with high information density only in the local context of the current chapter, a specific table field, or a specific code enumeration block. In implementation, the main text features are first segmented locally. An improved TF-IDF algorithm or TextRank algorithm can be used to extract words that appear with significantly abnormally high frequency within the current chapter or are located in specific grammatical positions (such as the key column of a table, bold text, or text within parentheses) after excluding globally common interference words. These words are then used as local keywords. By comparing whether the dense local keywords within the main text match specific patterns, it helps determine whether the current chapter belongs to a "definition-type entity" or a "relational entity."

[0100] Based on the extracted features of each dimension, the signal scores of each slot in terms of chapter path matching, title matching, slot alias matching, and focused term matching are calculated respectively.

[0101] Finally, a weighted summation algorithm is used to calculate the total score, i.e.:

[0102]

[0103] in, This represents the overall matching score of the current chapter on a specific candidate semantic slot; This represents the total number of signal dimensions involved in the evaluation, and It is an integer greater than 1. This indicates that the current chapter is in the [number]th [section]. Dimensional matching score on each signal dimension; Indicates the corresponding number Preset feature weights for each signal dimension.

[0104] The routing decision method is as follows: compare the total scores of all Slot candidates, and select the Slot with the highest score that exceeds the preset threshold as the final routing target for that section.

[0105] For example, when the input chapter level path feature is an enumeration type and its title text feature is "gpio pindirection", the chapter is routed to the "data structure semantic slot data_structures" because it has the highest score in the title matching and path matching dimension of the data structure semantic slot.

[0106] When the input chapter level path is of enumeration type and the title text feature is "gpio_irq_event" (interrupt event), its strong hardware interrupt attribute is identified by matching the focus terms such as "irq" and "event" in the title, and it is directed to the "interrupt semantic slot interrupts".

[0107] By storing different knowledge types in slots, orthogonal organization of knowledge is achieved, avoiding knowledge confusion.

[0108] Step 5: Section / Entity Hierarchy Recognition

[0109] To express the complete knowledge hierarchy, this application adds Section and Entity levels to the "Topic -> Slot -> Fact" hierarchy, thus forming a complete topology of "Topic -> Slot -> Section -> Entity -> Fact -> Evidence".

[0110] Entity type identification rules are configurable, allowing customization for users in different domains. These rules are typically defined in system configuration files (such as Taxonomy) to map local structural clues in documents to predefined entity types. Specifically, the mapping relationship between local structural clues and predefined entity types is shown in Table 4:

[0111] Table 4 Mapping Relationship between Local Structural Clues and Predefined Entity Types

[0112]

[0113] Table 4 lists the rules as a default example. In actual use, users can add or modify the mapping relationship between local structural clues and Entity types (e.g., "Contains "Requirement"" -> "Constrained Entity") in the configuration file according to domain requirements, achieving flexible domain adaptation. Taking GPIO enumeration as an example, its default expression is:

[0114] Topic: GPIO

[0115] Slot: data_structures

[0116] Section: gpio pin direction

[0117] Entity (enum): gpio pin direction

[0118] Fact: GPIO_DIR_INPUT sets_direction_to normal input mode

[0119] Fact: GPIO_DIR_OUTPUT sets_direction_to normal output mode

[0120] The above expression clearly shows that GPIO_DIR_INPUT is a member of some enum, rather than an isolated fact.

[0121] Step 6: Atomic Fact Extraction

[0122] The natural language content, tables, and lists in the section are converted into comparable factual records. This step employs a two-stage extraction strategy to balance quality and cost. The two-stage extraction strategy includes rule-based baseline extraction and LLM extraction. Specifically:

[0123] (1) Rule baseline extraction

[0124] The rule baseline is fast, suitable for rapid building and debugging. The principle is to extract the main text content of relevant sections based on the slot's focus_terms, generating preliminary facts from short sentences, list items, and table rows. The actual implementation method is as follows:

[0125] Segment the body of the section into sentences;

[0126] Filter relevant sentences based on focus_terms;

[0127] Extract key-value pairs from the table;

[0128] Generate preliminary fact records.

[0129] (2) LLM extraction

[0130] The following information is constructed as prompts and sent to the large language model, which is then required to output structured facts in JSON format according to the augmented fact schema. Specifically:

[0131] Input fields include:

[0132] topic_id: To which topic belongs

[0133] topic_title: Topic title

[0134] slot_id: Slot to which it belongs

[0135] slot_title: Slot title

[0136] source_path: Path to the source document

[0137] section_path: Chapter path

[0138] body: Main content

[0139] The output fields, i.e., the structure of each standardized atomic fact record, are shown in Table 5:

[0140] Table 5 Field Structure of Standardized Atomic Fact Records

[0141]

[0142] Evidence chain tracing: The `source_path`, `section_path`, `line_start`, and `line_end` information in the `evidence` field comes directly from the AST preprocessing results in step 1, ensuring that each atomic fact has complete source traceability.

[0143] Furthermore, the extraction rules for large models are as follows:

[0144] 1. Only extract facts explicitly supported by the source text;

[0145] 2. Do not complete information that is not in the document;

[0146] 3. A single fact expresses only one assertion;

[0147] 4. The value should be a specific object, numerical value, enumeration value, pattern, or constraint, and should not be an entire explanation;

[0148] 5. Retain topic / slot unless there is an obvious routing error.

[0149] LLM's semantic understanding capabilities can significantly improve the accuracy and completeness of fact extraction.

[0150] Because LLM extraction is prone to missing table rows, enumeration items, and condition information during the initial extraction, this application uses Gleaning to self-check and complete the extraction results.

[0151] Step 7: Gıeaning (multi-round iterative) quality enhancement

[0152] Processing flow: Initial extraction → Validation prompts → Gleaning round 1 → Validation prompts → Gleaning round 2 → Corrected JSON Validation Highlights:

[0153] 1. Are any enumerations, macros, parameters, or return values ​​missing?

[0154] 2. Should multiple assertions be combined into a single fact?

[0155] 3. Is the condition field missing?

[0156] 4. Are the subject / predicate / value fields missing?

[0157] 5. Should an entire sentence of natural language be inserted into the value field?

[0158] 6. Does it violate the known topic / slot affiliation?

[0159] Multiple rounds of iteration significantly improve extraction quality and reduce omissions and errors.

[0160] Step 8: Fact Standardization and Conflict_Key Generation

[0161] The extracted facts are standardized to prepare for subsequent deduplication and conflict detection.

[0162] Standardized processing procedures:

[0163] 1. Remove meaningless whitespace characters;

[0164] 2. Standardize the format of the subject, predicate, and condition fields (e.g., ensure consistent capitalization, remove extra spaces, etc.);

[0165] 3. Generate a conflict_key for subsequent deduplication and conflict detection;

[0166] 4. Merge completely duplicate facts (all fields are the same).

[0167] 5. Conflict_key generation rules:

[0168] conflict_key = normalize (subject) + "::" + normalize (predicate) +"::" + normalize (condition)

[0169] It is important to note that you cannot simply use subject + predicate, because different conditions may represent different facts (such as parameter values ​​under different working modes).

[0170] Standardization and conflict_key generation provide a foundation for subsequent fact alignment and conflict detection.

[0171] Step 9: Blocking, Deduplication and Conflict Detection

[0172] A blocking strategy is used to avoid full pairwise comparisons, thereby improving processing efficiency.

[0173] Blocking strategy:

[0174] Facts are grouped using topic_id + slot_id + conflict_key as the blocking key. That is, only facts with the same blocking key can be duplicated or conflicted, so pairwise comparisons can be performed only within the same blocking group, avoiding the total comparison complexity of O(n^2).

[0175] Intra-group comparison: Comparing the similarity of facts within the same blocking group. Strategies include:

[0176] 1. If they are completely identical, they are considered duplicates, and the version with the highest confidence level is retained;

[0177] 2. If the same key has different values, it is considered a conflict, and the conflict group is recorded;

[0178] 3. If the descriptions are similar, it is determined to be a semantic cluster.

[0179] Furthermore, token similarity or embedding similarity can be used to cluster facts with similar expressions.

[0180] Token similarity formula:

[0181] similarity (A, B) = |tokens (A) intersect tokens (B) | / |tokens (A)union

[0182] tokens (B) | Technical effect:

[0183] By employing a blocking strategy, the efficiency of conflict detection is significantly improved, enabling the system to handle large-scale fact sets.

[0184] Step 10: Conflict Resolution

[0185] In this scheme, when different data sources output conflicting multi-party fact claims for the same atomic fact using the same feature key, the system captures the feature key and extracts the confidence scores and data source family attributes corresponding to each party's fact claim value.

[0186] Based on the physical attribute type indicated by the feature key, the corresponding arbitration strategy is dynamically matched in the preset disambiguation rule matrix. If the feature key indicates a hardware limit parameter, the hard arbitration strategy of matching the data manual takes precedence over the user guide.

[0187] According to the arbitration strategy, the statement values ​​belonging to the data manual family and meeting the confidence score requirements are selected from the multi-party fact statement values ​​and retained as the final standardized atomic fact records. At the same time, the conflict disambiguation process is solidified as historical evidence into the status field of the atomic fact record, and the disambiguation status of the current atomic fact record is marked as resolved.

[0188] It should be noted that these rules are configurable, allowing users to dynamically customize the priority order of rulings based on the type of fact and the credibility of the document source, to ensure domain adaptability and accuracy. The conflict resolution rules are shown in Table 6:

[0189] Table 6 Examples of Conflict Resolution Rules

[0190]

[0191] Adjudication Rules:

[0192] 1. Prioritize sources based on fact type;

[0193] 2. Select the value from the highest priority source as the ruling result;

[0194] 3. If conflicts also exist within high-priority sources, the conflict status should be preserved, automatic adjudication should not be performed, and this should be explained in the report.

[0195] By dynamically adjudicating source priorities, an interpretable conflict resolution mechanism is provided, avoiding silent overwrite errors.

[0196] Step 11: Structured Topology Synthesis

[0197] The processed facts are synthesized into a complete knowledge topology:

[0198] Topic->Slot->Section->Entity->Fact->Evidence

[0199] This completes the synthesis of a comprehensive knowledge topology structure, consisting of a top-down, nested structure of technical topic layer, semantic slot layer, chapter layer, entity layer, atomic fact layer, and evidence chain layer, with complete vertical cohesion. For example... Figure 3 As shown, Figure 3 This diagram illustrates the structured knowledge topology of the final complete output.

[0200] According to the preset output protocol, the complete knowledge topology can be losslessly serialized into a JSON-formatted structured dataset containing the aforementioned six-level nested relationships. By utilizing the structured dataset to construct a technical entity index in memory space, cross-document knowledge retrieval, differential comparison, and graph quantitative analysis support can be provided.

[0201] Through structured topological synthesis, a complete and traceable knowledge base was generated.

[0202] Step 12: Coverage Assessment and Search

[0203] Step S121: Coverage Assessment

[0204] The purpose of this step is to generate a Topic / Slot level coverage report, checking whether the requiredSlots for each Topic have been covered, including:

[0205] Extract each primary subject from the multi-level progressive structured knowledge topology data, and retrieve the preset knowledge pattern metadata to obtain the set of mandatory semantic slots corresponding to each primary subject.

[0206] The semantic slot hierarchy under the main subject is traversed vertically, the instance filling status of each mandatory semantic slot is compared, and the slot coverage metric of the main subject is calculated and output.

[0207] Capture the required semantic slots that have not been filled by any chapter data, generate a list of missing required slots, and based on the semantic slot attributes associated with the list of missing required slots, reverse deduce and output suggestions for knowledge defect type completion.

[0208] The final report contains the following:

[0209] 1. Topic list and slot coverage for each topic;

[0210] 2. Missing list of required slots;

[0211] 3. Types of knowledge to be supplemented.

[0212] Step S122: Fact Graph Construction

[0213] Construct a lightweight fact graph with nodes including Topic, Slot, Section, Entity, Fact, and Evidence, and edges representing the relationships between them.

[0214] The diagram structure is as follows:

[0215] Topic --has_slot--> Slot

[0216] Slot --has_section--> Section

[0217] Section --defines_entity--> Entity

[0218] Entity --has_fact--> Fact

[0219] Fact --supported_by--> Evidence

[0220] Step S123: Multidimensional and Multimodal Knowledge Retrieval Application

[0221] 1. Precise retrieval: Filter by fields such as topic_id, slot_id, subject, and predicate;

[0222] This mode provides deterministic field filtering channels to perform precise retrieval, namely:

[0223] In response to an input exact retrieval request, at least one hard-filtered field is extracted from the exact retrieval request, including topic ID (topic_id), slot ID (slot_id), assertion subject (subject), or relational predicate (predicate), and prefix key-value matching or graph path filtering is performed in the hierarchical structured knowledge topology data to return the corresponding leaf node data;

[0224] 2. Semantic retrieval: Similarity retrieval is performed using embedding vectors.

[0225] This mode provides a dense vector space mapping channel to perform semantic retrieval: in response to an input free natural language retrieval request, the free natural language retrieval request is converted into a low-dimensional dense semantic vector using a preset text embedding model, and a topological distance metric is performed in the pre-established vector space using an approximate nearest neighbor search algorithm to recall the atomic fact records with the closest semantics.

[0226] Through the above processing methods, this solution can perform coverage assessment and fact graph construction, thereby providing comprehensive knowledge management and retrieval capabilities.

[0227] Example 2

[0228] like Figure 4 As shown, this embodiment of the invention also provides an orthogonal structured knowledge extraction system 400 based on configurable semantic slots, used to implement the method described in the above technical solution, including:

[0229] The document parsing module 401 is used to obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and the main text content, and generate a chapter set containing multiple chapters;

[0230] Preprocessing module 402 is used to perform noise reduction and deduplication on the chapter set to obtain a filtered chapter set;

[0231] The topic determination module 403 is used to determine the primary topic to which each chapter in the filtered chapter set belongs based on a multi-path recall mechanism.

[0232] The routing module 404 is used to calculate the matching score of each chapter corresponding to each main subject on each candidate semantic slot according to the pre-configured classification rules, and to route the chapter to the orthogonal knowledge type semantic slot with the highest matching score.

[0233] The topology building module 405 is used to determine the local structural clues of the chapters within the semantic slots based on preset entity recognition rules, and to map the local structural clues to the corresponding entity types, thereby constructing an initial knowledge topology structure that includes the main subject, semantic slots, chapters, and entity types.

[0234] The fact extraction module 406 is used to perform atomic fact extraction using a multi-stage extraction strategy, perform feature transformation on the text content corresponding to each node in the initial knowledge topology, convert the text content into standardized atomic fact records without loss, and use the standardized atomic fact records as terminal leaf nodes to be directionally filled under the corresponding entity type nodes in the initial knowledge topology.

[0235] The output module 407 is used to fuse the atomic fact record with its corresponding chapter, semantic slot and main subject to output multi-level progressive structured knowledge topology data.

[0236] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting structured knowledge based on a configurable semantic slot orthogonal structure, characterized in that, include: Obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and body content, and generate a chapter set containing multiple chapters; The chapter set is then subjected to noise reduction and deduplication to obtain a filtered chapter set. The primary subject to which each chapter belongs in the filtered chapter set is determined based on a multi-path recall mechanism. For each chapter corresponding to the main subject, calculate its matching score on each candidate semantic slot according to the pre-configured classification rules, and route the chapter to the orthogonal knowledge type semantic slot with the highest matching score; Based on preset entity recognition rules, the local structural clues of the chapters within the semantic slots are determined, and the local structural clues are mapped to the corresponding entity types to construct an initial knowledge topology structure containing the main subject, semantic slots, chapters, and entity types. A multi-stage extraction strategy is adopted to extract atomic facts. Feature transformation is performed on the text content corresponding to each node in the initial knowledge topology, and the text content is losslessly converted into standardized atomic fact records. The standardized atomic fact records are used as terminal leaf nodes and are directionally filled into the corresponding entity type nodes in the initial knowledge topology. The atomic fact records are fused with their corresponding chapters, semantic slots, and main themes to output multi-level, progressively structured knowledge topology data. 2.The configurable semantic slot based orthogonal structured knowledge extraction method according to claim 1, wherein, The target technical document is recursively parsed hierarchically to extract the title level and body content, generating a chapter set containing multiple chapters, including: Identify heading nodes and heading levels based on the hierarchical markers in the target technical document; Construct a parent-child relationship tree for chapter nodes based on the title nodes and title levels; Associate the title and text content corresponding to each chapter node in the relation tree, and generate the path text for each chapter; The path text of the chapter is concatenated with the text content corresponding to the chapter node, and the SHA1 algorithm is used to perform a hash operation to generate the content hash value of the corresponding chapter. 3.The configurable semantic slot based orthogonal structured knowledge extraction method of claim 2, wherein, The aforementioned collection of chapters undergoes noise reduction processing, including: Calculate the BM25 relevance score between the main text content of each chapter and the preset noise keywords, and filter out chapters with BM25 relevance scores greater than or equal to the preset noise reduction threshold. The collection of chapters is deduplicated, including: Compare the content hash values ​​corresponding to each chapter; If two or more chapters have the same content hash value, they are considered duplicate chapters, and the chapter version with the highest source reputation is retained according to the pre-configured priority rules.

4. The method of claim 1, wherein the method further comprises: The process of determining the primary subject of each chapter in the filtered chapter set based on a multi-path recall mechanism includes: For each filtered chapter, structural prior recall, lexical recall, and semantic vector recall are performed in parallel to obtain the corresponding first topic recall results, second topic recall results, and third topic recall results, respectively. The inverse ranking fusion algorithm is used to calculate the comprehensive score of the recall results of the first, second and third topics, and the topic with the highest comprehensive score is taken as the main topic of the chapter.

5. The orthogonal structured knowledge extraction method based on configurable semantic slot according to claim 2, characterized in that, For each chapter corresponding to the primary topic, its matching score on each candidate semantic slot is calculated according to pre-configured classification rules. The chapter is then routed to the orthogonal knowledge type semantic slot with the highest matching score, including: Extract the path features, title features, and text content features of each chapter whose main topic has been determined; Calculate the path matching dimension score, title matching dimension score, alias matching dimension score, and focus term matching dimension score of the chapter on each candidate semantic slot; The overall matching score for each candidate semantic slot is obtained by weighted summation of the scores on each dimension. The candidate semantic slot with the highest overall matching score that exceeds a preset threshold is used as the belonging semantic slot, and the chapter is directed to the belonging semantic slot.

6. The orthogonal structured knowledge extraction method based on configurable semantic slots according to claim 1, characterized in that, A multi-stage extraction strategy is employed to extract atomic facts. Feature transformation is performed on the text content corresponding to each node in the initial knowledge topology, losslessly converting the text content into standardized atomic fact records, including: If the main text contains tables or key-value pairs that conform to a preset structure, the scheduling rule parsing engine performs hard feature stripping and directly outputs the corresponding standardized atomic fact records. If the main text is a natural language narrative paragraph, then entity type and local structural clues are used as input constraint variables, and a large language model is called to perform single assertion semantic transformation, outputting a standardized atomic fact record containing fact subject, relational predicate, specific value and fact establishment conditions; The standardized atomic fact records output by the rule parsing engine and the large language model are combined and used as leaf nodes to fill the corresponding entity nodes.

7. The orthogonal structured knowledge extraction method based on configurable semantic slots according to claim 6, characterized in that, After the standardized atomic fact records are used as terminal leaf nodes and directionally populated under the corresponding entity type nodes in the initial knowledge topology structure, the method further includes: The atomic fact records initially extracted and the corresponding chapter text content are re-input into the large language model. Based on the preset anti-omission self-check prompts, multiple rounds of iterative reconciliation and completion checks are performed on elements with missing facts, multiple assertion merging, and incomplete fields, and the corrected and enhanced atomic fact records are output.

8. The orthogonal structured knowledge extraction method based on configurable semantic slots according to claim 7, characterized in that, Following the output of the corrected and enhanced atomic fact record, it also includes: Each atomic fact record is standardized by field and then concatenated to generate a composite conflict detection key; Facts are grouped using the identifier of the main subject, the identifier of the semantic slot, and the composite conflict detection key as the block key; Within each group, if the composite conflict detection keys are the same but the specific values ​​are different, then a conflict group is constructed. For the conflict group, the priority of the adjudication is dynamically determined by matching the preset mapping relationship between fact type and document source credibility, and the value corresponding to the source with the highest priority is retained.

9. The orthogonal structured knowledge extraction method based on configurable semantic slots according to claim 1, characterized in that, Also includes: Extract each primary subject from the multi-level progressive structured knowledge topology data, retrieve preset knowledge pattern metadata, and obtain a preset set of semantic slots corresponding to each primary subject. The set of semantic slots under the main topic is traversed vertically, the filling status of each preset semantic slot is compared, and the slot coverage metric of the main topic is calculated and output. Identify the mandatory semantic slots that have not been filled by any chapter data, generate a list of missing mandatory slots, and based on the semantic slot attributes associated with the list of missing mandatory slots, deduce and output suggestions for knowledge defect type completion in reverse.

10. A system for extracting orthogonal structured knowledge based on configurable semantic slots, characterized in that, For implementing the method as described in any one of claims 1-9, comprising: The document parsing module is used to obtain the target technical document, perform recursive hierarchical document parsing on the target technical document, extract the title level and the main text content, and generate a chapter set containing multiple chapters; The preprocessing module is used to perform noise reduction and deduplication on the chapter set to obtain a filtered chapter set; The topic determination module is used to determine the primary topic to which each chapter in the filtered chapter set belongs, based on a multi-path recall mechanism. The routing module is used to calculate the matching score of each chapter corresponding to each main subject on each candidate semantic slot according to the pre-configured classification rules, and to route the chapter to the orthogonal knowledge type semantic slot with the highest matching score. The topology building module is used to determine the local structural clues of the chapters within the semantic slots based on preset entity recognition rules, and to map the local structural clues to the corresponding entity types, thereby constructing an initial knowledge topology structure that includes the main subject, semantic slots, chapters, and entity types. The fact extraction module is used to perform atomic fact extraction using a multi-stage extraction strategy, perform feature transformation on the text content corresponding to each node in the initial knowledge topology, convert the text content into standardized atomic fact records without loss, and use the standardized atomic fact records as terminal leaf nodes to fill the corresponding entity type nodes in the initial knowledge topology. The output module is used to merge the atomic fact records with their corresponding chapters, semantic slots and main themes to output multi-level progressive structured knowledge topology data.

Images