Knowledge base construction method and system based on ai text analysis and hybrid retrieval

Abstract

This invention relates to the field of database construction technology, and discloses a knowledge base construction method and system based on AI text parsing and hybrid retrieval. By integrating semantic relevance and document layout features through adaptive block segmentation, it achieves accurate identification of core semantic units in the text and maintains logical integrity in the division, fundamentally avoiding information fragmentation and laying the foundation for high-quality knowledge organization. Secondly, through a hybrid retrieval mechanism combining keyword, semantic vector retrieval, and multi-dimensional re-ranking, it effectively balances retrieval response speed with the depth and accuracy of results, meeting users' multi-level query needs from rapid location to in-depth correlation mining. Finally, through dynamic updates and automatic association mapping functions, it achieves real-time synchronization and intelligent association of newly added knowledge, not only ensuring the timeliness of the knowledge system but also proactively building a cross-document knowledge network, thereby breaking down information silos and enhancing the overall utilization value and discovery capability of knowledge.

Description

Technical Field

[0001] This invention relates to the field of database construction technology, and more specifically to a knowledge base construction method and system based on AI text parsing and hybrid retrieval. Background Technology

[0002] In traditional information-based knowledge management practices, the processing of unstructured text typically relies on document segmentation methods based on fixed lengths or simple paragraph rules. While this coarse-grained approach achieves basic data entry, it often crudely severs the inherent semantic coherence and logical structure of the text. This results in knowledge fragments that are inherently lacking in information and context, creating inherent deficiencies in subsequent retrieval processes. For example, keyword-based literal matching retrieval techniques struggle to handle complex linguistic phenomena such as polysemy and synonym substitution of technical terms, often returning results accompanied by a large amount of irrelevant information or missing key content, making it difficult to guarantee retrieval accuracy. Furthermore, the static knowledge base construction model makes the integration of new or revised text content cumbersome and delayed, hindering the dynamic evolution of the knowledge system and its ability to respond promptly to business needs. Ultimately, this leads to a predicament of numerous information silos and low efficiency in knowledge reuse. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a knowledge base construction method and system based on AI text parsing and hybrid retrieval, so as to solve the technical problems of poor text segmentation logic and overly simple semantic elements in the construction of current knowledge bases for retrieval, especially normative domain knowledge bases, which leads to low subsequent retrieval efficiency and low accuracy of retrieval results.

[0004] The first aspect of this invention discloses a method for constructing a knowledge base based on AI text parsing and hybrid retrieval, the method comprising:

[0005] Acquire text data that has not yet been entered into the database, and perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not yet been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors;

[0006] Based on semantic association information and document layout feature information, the text data not yet entered into the database is divided into text blocks, and the keyword index and semantic vector of each text block are determined.

[0007] The knowledge base is constructed by dividing the text blocks into blocks, using the keyword index of each block, and semantic vectors.

[0008] Hybrid retrieval is performed based on user query information and a constructed knowledge base.

[0009] Furthermore, the hybrid retrieval process specifically includes:

[0010] Extract keywords from the user's input query, perform keyword matching based on the keyword index, and obtain a first result set;

[0011] The query statement is converted into a query vector, and vector similarity retrieval is performed based on the semantic vector to obtain a second result set.

[0012] The first result set and the second result set are merged, and the merged result set is sorted based on a preset reordering optimization mechanism to output the final search results.

[0013] Furthermore, the document layout feature extraction operation includes:

[0014] The document layout of each page is identified, and visual layout element information and logical structure indication information are extracted from the document; wherein, the visual layout element information includes at least one of font format, paragraph format, bullet points, table borders, and chart positions; the logical structure indication information includes at least one of title numbering, header, footer, column break, and section break.

[0015] Furthermore, before performing the document layout feature extraction operation, the method further includes:

[0016] Determine the text type of the texts not yet entered into the database;

[0017] When the text type is a policy text, after extracting the document layout features of the policy text, the format marks representing the hierarchical structure in the policy text are identified based on the document layout feature information;

[0018] Based on the format tags, text segments that conform to the clause format specifications are identified as the smallest text block units, and a tree-like hierarchical block relationship including chapters, sections, and clauses is established based on text semantic association information.

[0019] Furthermore, after establishing a tree-like hierarchical block relationship including chapters, sections, and articles, the method further includes:

[0020] For each text block of the policy document, explicit citation statements that cite other clauses within the policy document are extracted using preset citation pattern recognition rules.

[0021] Based on the explicit citation statement, determine the hierarchical positioning identifier of the cited clause, and search for the corresponding target text block in the tree-like hierarchical block relationship based on the hierarchical positioning identifier;

[0022] The correspondence between the hierarchical positioning identifier of the current text block and the hierarchical positioning identifier of the target text block is used as reference-related metadata and stored in association with the content, keyword index, and semantic vector of the text block.

[0023] Furthermore, the semantic parsing operation is performed through a pre-trained semantic parsing model;

[0024] When the text type is institutional text, the semantic parsing model is also used to parse the text content of the smallest text block unit into element information, and associate and store the element information with the corresponding text block; wherein, the type of element information includes applicable conditions, acting subject, normative behavior, behavioral consequences, and exceptions.

[0025] Furthermore, the hybrid retrieval process also includes:

[0026] The query statement type of the user is determined. When it is determined to be a contextual description statement, the contextual description statement is parsed, and the contextual entities and relationships corresponding to the type of element information are extracted to generate a contextual query element set.

[0027] The matching degree between the contextualized query element set and the element information is calculated, and the matching degree calculation result is used as the ranking factor of the re-ranking optimization mechanism.

[0028] Furthermore, when the text type is an institutional text, the sorting factors of the reordering optimization mechanism also include institutional effectiveness level factors, job role suitability factors, clause status and timeliness combination factors, and citation logic correlation strength factors.

[0029] Furthermore, before performing semantic parsing, keyword extraction, and document layout feature extraction on the institutional texts that have not been entered into the database, the method also includes performing a conflict retrieval operation on the institutional texts that have not been entered into the database.

[0030] The conflict retrieval operation includes:

[0031] Each clause in the policy text that is not yet in the database is used as a query statement for a similarity-mixed search. Target clauses in the knowledge base with a semantic similarity to the query statement that are higher than a preset threshold are identified and highlighted to prompt users to review whether there are any potential conflicts.

[0032] The second aspect of this invention discloses a knowledge base construction system based on AI text parsing and hybrid retrieval, the system comprising:

[0033] The text processing module is used to acquire text data that has not been entered into the database, and to perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors;

[0034] The text segmentation module is used to divide the unstored text data into text blocks based on semantic association information and document layout feature information, and to determine the keyword index and semantic vector of each text block.

[0035] The building module is used to construct a knowledge base from the content of the divided text blocks, the keyword index of each text block, and the semantic vector;

[0036] The retrieval module is used for hybrid retrieval based on the user's query information and the constructed knowledge base.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention achieves accurate identification of core semantic units in text and maintains logical integrity through adaptive segmentation that integrates semantic relevance and document layout features, fundamentally avoiding information fragmentation and laying the foundation for high-quality knowledge organization. Secondly, the constructed hybrid retrieval mechanism, combining precise keyword matching, semantic vector retrieval, and multi-dimensional re-ranking, effectively balances retrieval response speed with the depth and accuracy of results, meeting users' multi-level query needs from rapid location to in-depth relational mining. Finally, through dynamic updates and automatic association mapping, real-time synchronization and intelligent association of newly added knowledge are achieved, ensuring not only the timeliness of the knowledge system but also proactively building a cross-document knowledge network, thereby breaking down information silos and enhancing the overall utilization value and discovery capability of knowledge. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a knowledge base construction method based on AI text parsing and hybrid retrieval, as disclosed in an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0042] Example 1

[0043] The first aspect of this invention discloses a method for constructing a knowledge base based on AI text parsing and hybrid retrieval, which belongs to the field of database construction technology. Please refer to... Figure 1 , Figure 1This is a flowchart illustrating a knowledge base construction method based on AI text parsing and hybrid retrieval disclosed in an embodiment of the present invention. The method includes:

[0044] Acquire text data that has not yet been entered into the database, and perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not yet been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors;

[0045] Based on semantic association information and document layout feature information, the text data not yet entered into the database is divided into text blocks, and the keyword index and semantic vector of each text block are determined.

[0046] The knowledge base is constructed by dividing the text blocks into blocks, using the keyword index of each block, and semantic vectors.

[0047] Hybrid retrieval is performed based on user query information and a constructed knowledge base.

[0048] Preferably, the semantic parsing operation described above is performed using a pre-trained semantic parsing model based on the Transformer architecture. When building the model, a model pre-trained on a large-scale general corpus is chosen as the foundation, whose multi-layer self-attention mechanism can effectively capture the contextual dependencies and deep semantics of the text. To adapt it to the dual task of simultaneously outputting semantic association information and semantic vectors, this invention fine-tunes the model within a multi-task learning framework. The model structure adds two parallel output layers on top of the pre-trained model. One output layer generates fixed-dimensional semantic vectors for text fragments, obtained by converging the hidden states of the last layer of the model; the other output layer is used for semantic association prediction, taking a pair of text fragments as input and outputting a score representing the semantic closeness between them after model encoding. The training data contains a large number of manually or semi-automatically labeled text pairs. Each data point includes text fragment A, text fragment B, and their semantic association labels, such as relevance, irrelevance, or continuity scores.

[0049] During training, the following loss functions are jointly optimized: first, a vector representation learning loss based on contrastive learning or cosine similarity, aiming to make semantically similar texts also similar in vector space; second, a mean squared error loss for relevance prediction. Through this training, the model not only learns to generate high-quality text vectors rich in semantics, but also internalizes the ability to evaluate the strength of semantic logical connections between any two text segments, thus providing crucial semantic connection information for subsequent adaptive segmentation.

[0050] Furthermore, regarding the keyword extraction process described above, a hybrid keyword extraction strategy is preferably adopted in this embodiment of the invention. Specifically, for all text, general keywords representing the topic are extracted using statistical methods (improved TF-IDF or TextRank algorithms). Based on named entity recognition rules, keywords with specific types and importance are extracted; these keywords include, but are not limited to, dates and times, amounts and numbers, codes, proper nouns, and core terms. In addition, for texts in specific domains, such as policy documents, a domain dictionary is loaded to improve the accuracy and weight of core terms in areas such as "legal responsibility," "effective date," and "right of interpretation." Finally, all extracted keywords, their locations, and type tags are organized and indexed to form a keyword index library that supports efficient and accurate matching.

[0051] Furthermore, the hybrid retrieval process specifically includes:

[0052] Extract keywords from the user's input query, perform keyword matching based on the keyword index, and obtain a first result set;

[0053] The query statement is converted into a query vector, and vector similarity retrieval is performed based on the semantic vector to obtain a second result set.

[0054] The first result set and the second result set are merged, and the merged result set is sorted based on a preset reordering optimization mechanism to output the final search results.

[0055] It is understandable that a single search mode has inherent limitations and cannot simultaneously meet users' dual requirements for efficiency and accuracy. In this embodiment of the invention, a hybrid search mechanism combining keyword matching and semantic vector retrieval is employed to address these shortcomings. Specifically, keyword matching, based on literal comparison, while fast and effective at hitting text containing explicit terms, dates, numbers, and other rigid information, cannot handle synonyms, near-synonyms, or semantically similar but differently expressed words, risking insufficient recall. Semantic vector retrieval, on the other hand, matches queries with the deeper meaning of the text through in-depth understanding, significantly improving recall and semantic relevance. However, its computational complexity is relatively high, and it may not be as direct as keyword matching when dealing with precise matching of proper nouns. Therefore, this invention executes both in parallel to construct a complementary search pipeline: keyword matching performs rapid filtering, locking in explicit relevant results (the first result set) immediately, ensuring the immediacy of the search; semantic vector retrieval performs in-depth mining, simultaneously capturing potential semantically related results (the second result set), ensuring the comprehensiveness and intelligence of the search.

[0056] Furthermore, after merging the two result sets, this invention introduces a re-ranking optimization mechanism. It is understandable that simple merging may lead to redundant results or unreasonable sorting. The re-ranking mechanism uses a pre-defined optimization algorithm, considering multiple dimensions such as the original relevance score of each result based on semantic vector similarity, timeliness based on text publication time, authority based on data source validity level, and even user behavior feedback, to globally and meticulously re-evaluate and re-rank the merged list. This process not only integrates the advantages of both retrieval paths but also injects business logic and user needs into the ranking strategy, resulting in a final list of results that combines high relevance, high timeliness, high authority, and a good user experience.

[0057] Through the above operations, this invention achieves a significant improvement in retrieval accuracy, namely precision and recall, while ensuring response efficiency. It can also output dynamically optimized intelligent ranking results that conform to business logic based on actual application scenarios.

[0058] Furthermore, the document layout feature extraction operation includes:

[0059] The document layout of each page is identified, and visual layout elements and logical structure indicators are extracted. The visual layout elements include, but are not limited to, font formatting, paragraph formatting, bullet points, table borders, and chart positions. The logical structure indicators include, but are not limited to, title numbering, headers, footers, column breaks, and section breaks.

[0060] Furthermore, before performing the document layout feature extraction operation, the method further includes:

[0061] Determine the text type of the texts not yet entered into the database;

[0062] When the text type is a policy text, after extracting the document layout features of the policy text, the format marks representing the hierarchical structure in the policy text are identified based on the document layout feature information;

[0063] Based on the format tags, text segments that conform to the clause format specifications are identified as the smallest text block units, and a tree-like hierarchical block relationship including chapters, sections, and clauses is established based on text semantic association information.

[0064] In this embodiment of the invention, visual typography element information specifically refers to the formatting features directly extracted from the visual presentation of a document. These features are crucial for the human eye to recognize the document structure. Specifically, font formatting, such as bold, font size, and font type, can distinguish between headings and body text; paragraph formatting, such as first-line indentation, line spacing, and alignment, can indicate paragraph beginnings and logical groupings; bullet points, table borders, and chart positions clearly define the physical boundaries of specific content blocks such as lists, data areas, and illustrations. Logical structure indication information goes further, referring to explicit markers that clearly encode the document's logical organizational structure. For example, heading numbers indicate the hierarchy and order of chapters; headers and footers often contain document titles or chapter names; column breaks and section breaks are explicit instructions used in typesetting software to divide pages or chapters. All of this information collectively constitutes a page layout semantics that differs from a pure text sequence and reflects the document's design intent.

[0065] In practical implementation, for texts identified as institutional documents, this invention utilizes the extracted format markers, such as matching specific numbering patterns like "Article X" or "Section X" using regular expressions, combined with their accompanying specific font, bolding, and centering attributes, to accurately locate the start and end positions of each clause, thereby dividing each independent and complete clause into the smallest indivisible text block unit. Subsequently, the nesting relationships of these format markers (e.g., "Chapter 1" contains "Article 1" to "Article N") and the semantic relationships between clauses are analyzed. Then, the semantic relationship information is used to analyze the coherence of the clause content, citation relationships, and consistency of the core theme to assist in verifying and correcting misjudgments that might arise from relying solely on formatting. Based on the above operations, a clear tree-like hierarchical block relationship data structure is constructed, with chapters as direct child nodes of the root node, sections as child nodes of chapters, and clauses as direct child nodes of sections or chapters. This data structure not only records the text content of the nodes but also their hierarchical paths and the hierarchical relationships between nodes.

[0066] In this invention, by performing the aforementioned operations on regulatory texts, the integrity of each regulation clause as an independent semantic unit is ensured, laying a solid foundation for subsequent high-precision retrieval and citation. Secondly, the established tree-like hierarchical block relationship enables browsing based on chapter navigation, hierarchical range retrieval, and precise cross-clause citation jumps, greatly improving the logical organization of knowledge. Locating the corresponding level based on the ambiguity of the user's query statement improves the efficiency of outputting retrieval results. This parsing method, combining visual formatting and deep semantics, significantly enhances the ability to understand and process complex and standardized documents.

[0067] Furthermore, after establishing a tree-like hierarchical block relationship including chapters, sections, and articles, the method further includes:

[0068] For each text block of the policy document, explicit citation statements that cite other clauses within the policy document are extracted using preset citation pattern recognition rules.

[0069] Based on the explicit citation statement, determine the hierarchical positioning identifier of the cited clause, and search for the corresponding target text block in the tree-like hierarchical block relationship based on the hierarchical positioning identifier;

[0070] The correspondence between the hierarchical positioning identifier of the current text block and the hierarchical positioning identifier of the target text block is used as reference-related metadata and stored in association with the content, keyword index, and semantic vector of the text block.

[0071] Specifically, the aforementioned pre-defined citation pattern recognition rules are preferably a composite rule set integrating regular expression matching, syntactic pattern analysis, and context inference. Specifically, based on statistical analysis of a large number of institutional texts (such as laws and regulations, company articles of association, and contract templates), common explicit citation expression paradigms are summarized. The core of these rules consists of a series of regular expression templates used to accurately match fixed phrases such as "in accordance with Article XX of these Measures" or "...the provisions of the preceding paragraph shall apply." Furthermore, the rule set also includes the identification of common citation indicators (such as "refer to," "see," "the following," and "the above") in Chinese legal texts, and determines whether they constitute valid citation statements based on their syntactic structure.

[0072] In the specific extraction process, each segmented clause text is traversed. First, the aforementioned regular expression template is used for scanning to initially capture all possible citation fragments. Next, context binding and entity parsing are performed. For matched patterns, the citation target description is extracted, such as Clause XX. If the description is absolutely positioned, it is converted into a standard hierarchical positioning identifier. If the description is relatively positioned or vaguely referenced, such as the preceding paragraph or the aforementioned provisions, the context inference engine is activated. Based on the current clause's position in the chapter-section-clause tree structure, it traverses upwards or to its sibling level, combining the semantic relevance of the clause title to dynamically parse out the most likely target clause. Finally, using the established hierarchical segmentation relationship, the parsed hierarchical positioning identifier is quickly mapped to a unique target text block entity, thereby establishing a precise, machine-readable citation link.

[0073] In this invention, the above operations enable the automated construction of the internal logical network of institutional texts, transforming the originally isolated and static knowledge of clauses into an interconnected dynamic knowledge system. This not only allows users to immediately obtain the superior basis cited and the detailed rules cited when querying a certain clause, achieving three-dimensional navigation and in-depth understanding of knowledge, but also provides a solid data foundation for subsequent advanced applications such as automatic compliance checks, impact scope analysis, and intelligent question answering, greatly enhancing the intelligence level and practical value of knowledge management.

[0074] Furthermore, the semantic parsing operation is performed through a pre-trained semantic parsing model. In the case of institutional texts, the semantic parsing model is also used to parse the text content of the smallest text block unit into element information, and associate and store the element information with the corresponding text block; wherein, the type of element information includes, but is not limited to, applicable conditions, acting subjects, normative behaviors, behavioral consequences, and exceptional circumstances.

[0075] To achieve deep, structured analysis of institutional texts, this invention further enhances the semantic parsing model with domain adaptability and functional improvements. Specifically, at the top level of the model, a parallel sequence labeling output layer is added, specifically for parsing institutional clause elements. This layer employs a labeling system such as BIO to classify each token in the input text sequence, identifying and labeling the text span belonging to predetermined element types such as "applicable conditions," "acting subjects," "normative behavior," "behavior consequences," and "exceptional circumstances."

[0076] During model training, a specialized, high-quality dataset of labeled institutional text elements was used. Each sample in this dataset is a complete institutional clause text, with precise boundary annotations and type assignments for its elements by domain experts. During training, a third loss function was introduced into the existing multi-task learning framework: the sequence labeling cross-entropy loss for the element parsing task. Model parameters were updated by jointly optimizing the following objectives: contrastive learning loss to ensure the semantic vector accurately represents the overall meaning of the text; loss to enable the model to judge the correlation between text pairs; and sequence labeling loss to enable the model to accurately identify and classify elements within the clause. Through this joint training, the model not only maintains strong general semantic understanding and association analysis capabilities but also deeply integrates the ability to recognize the unique logical structure of institutional texts. In application, the smallest text block unit identified as institutional text is input into this enhanced model. While generating its semantic vector and association information, the model automatically analyzes the various elements and their specific textual content contained within the clause through the sequence labeling output layer. Subsequently, this structured element information is associated and stored with all other data of the text block, forming a knowledge representation rich in logical semantics that can be directly reasoned by the machine.

[0077] Furthermore, the hybrid retrieval process also includes:

[0078] The query statement type of the user is determined. When it is determined to be a contextual description statement, the contextual description statement is parsed, and the contextual entities and relationships corresponding to the type of element information are extracted to generate a contextual query element set.

[0079] The matching degree between the contextualized query element set and the element information is calculated, and the matching degree calculation result is used as the ranking factor of the re-ranking optimization mechanism.

[0080] Furthermore, when the text type is an institutional text, the sorting factors of the reordering optimization mechanism include, but are not limited to, institutional effectiveness level factors, job role suitability factors, clause status and timeliness combination factors, and citation logic correlation strength factors.

[0081] Specifically, among the ranking factors in the aforementioned re-ranking optimization mechanism, the institutional validity level factor refers to the ranking weight assigned based on the legal or administrative validity of the institutional text. For example, constitutional laws take precedence over administrative regulations, and company bylaws take precedence over departmental rules. Clauses with higher validity levels are prioritized during ranking to ensure the authority and fundamental nature of the search results. The job role suitability factor refers to the matching calculation based on the query user's job identity (e.g., legal, human resources, finance) and the explicit or implicit applicable objects of the clauses in the knowledge base (e.g., all employees, financial personnel, department managers). Clauses most relevant to the current user's role are prioritized, achieving personalized search results. The clause status and timeliness combination factor is a composite indicator that comprehensively considers the specific status of the clause (e.g., currently valid, repealed, revised) and time attributes (e.g., publication date, effective date, revision date). This ensures that currently valid clauses receive the highest priority during ranking, and clauses in the same status are re-ranked based on timeliness to guarantee the current validity of the results. The logical association strength factor quantifies the closeness of the citation relationship between the search result clauses and the query context or other result clauses. For example, if result clause A is directly cited by result clause B, or both cite the same underlying clause, then the logical association between them is strong, and they will be promoted or displayed adjacently in the sorting to maintain logical coherence.

[0082] In this embodiment of the invention, by incorporating the matching degree calculation results of contextualized query elements into the aforementioned multi-dimensional domain factors into the re-ranking mechanism, the final search result ranking becomes an intelligent decision-making process that deeply integrates contextual accuracy, legal authority, personal relevance, current validity, and logical completeness. Users not only obtain the clauses that best match the described business scenario in terms of semantics and logical structure, but the presentation order of these clauses also ensures that they have the highest legal effect, best align with the user's own responsibilities, are the most recently effective, and have the most complete logical context. This greatly enhances the decision support value and practical reliability of the knowledge retrieval system in professional fields.

[0083] Furthermore, before performing semantic parsing, keyword extraction, and document layout feature extraction on the institutional texts that have not been entered into the database, the method also includes performing a conflict retrieval operation on the institutional texts that have not been entered into the database.

[0084] The conflict retrieval operation includes:

[0085] Each clause in the policy text that is not yet in the database is used as a query statement for a similarity-mixed search. Target clauses in the knowledge base with a semantic similarity to the query statement that are higher than a preset threshold are identified and highlighted to prompt users to review whether there are any potential conflicts.

[0086] Specifically, when an old clause that is semantically highly similar to the new clause or may be replaced or repealed by it is located through hybrid retrieval, an impact scope analysis is automatically triggered. At this time, the reference association metadata established in the knowledge association mapping step is invoked to quickly query all other old clauses in the knowledge base that point to the target clause in the reference relationship, that is, those text blocks that explicitly reference the target clause in their content.

[0087] Based on the above operations, this invention, through automated and precise impact scope positioning, can instantly generate a list of affected related clauses and visually display the citation chain. This not only greatly improves the efficiency and completeness of institutional revision work, but also fundamentally ensures the dynamic consistency and logical rigor within the institutional system, preventing implicit systemic conflicts caused by partial modifications, and achieving systematic and intelligent maintenance of the institutional knowledge ecosystem.

[0088] As a further preferred embodiment, to present the knowledge association network to users in an intuitive and structured way, this invention sets up a visual knowledge graph to support the differentiated display of multiple types of associations. Specifically, the graph data includes nodes (text blocks) and edges (associations), and each edge has a pre-defined association type attribute. During visualization rendering, different association types are significantly distinguished according to predefined visual encoding rules. For example, solid blue arrows are used to represent explicit reference associations to simulate the explicit guidance relationships in institutional clauses; dashed green lines are used to represent the evolutionary associations between old and new text versions, indicating the historical link of the same clause between different historical versions; and wavy orange lines are used to represent semantically similar associations, implying their potential connections calculated based on content similarity. Meanwhile, nodes themselves can be rendered with different colors or shapes according to their metadata (such as level of effectiveness, affiliated system). The graph interface provides interactive exploration functions; users can click on any node to highlight all its associated edges, or hover to view association details, or filter the display by association type. The above settings transform complex, multi-dimensional, interconnected data into a clear visual semantic network, greatly assisting users, especially legal and compliance personnel, in intuitively grasping the internal logical structure, historical evolution, and cross-domain semantic connections of the system, thereby supporting in-depth knowledge discovery, conflict tracing, and impact analysis.

[0089] As another preferred implementation, to ensure the timeliness of the knowledge base content, this invention also supports the function of importing new text from multiple channels and automatically triggering processing. Through various standard interfaces, such as deploying a background daemon process for local files to monitor specified folder directories, any added or modified file events (such as dragging or saving files) will be captured and a pending task will be generated; for online documents, a secure API endpoint is exposed to receive file push notifications from authorized sources or directly uploaded file streams. All un-included text arriving through the above channels is assigned a globally unique transaction ID and enters an asynchronous task queue to await processing.

[0090] Subsequently, file format parsing, text block segmentation, and knowledge base update operations are performed. The status and result of each processing step are bound to a transaction ID and recorded. Once a text block completes all processing, its generated keyword index, semantic vector, structured elements, and reference-related metadata are written to the corresponding database as atomic operations and marked as the latest version. This ensures that newly added knowledge can be retrieved by subsequent search requests the instant processing is completed, achieving rapid synchronization.

[0091] Furthermore, a structured update log is generated after each successful data import transaction. This log records not only metadata such as transaction ID, file source, import time, and operator, but also, with fine granularity, the unique identifier and version number of each text block added, modified, or abolished during the operation. An incremental version management strategy is used to assign a version sequence to each text block. When the same policy is revised and re-imported, the hierarchical positioning identifier of the clauses is compared to create new versions for clauses whose content has changed, and a substitution association is established between the old and new versions. All historical versions are retained, and users can clearly trace the evolution history of any clause through a version comparison view, achieving precise version control and audit trail from the document level to the clause level.

[0092] Example 2

[0093] The second aspect of this invention discloses a knowledge base construction system based on AI text parsing and hybrid retrieval, the system comprising:

[0094] The text processing module is used to acquire text data that has not been entered into the database, and to perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors;

[0095] The text segmentation module is used to divide the unstored text data into text blocks based on semantic association information and document layout feature information, and to determine the keyword index and semantic vector of each text block.

[0096] The building module is used to construct a knowledge base from the content of the divided text blocks, the keyword index of each text block, and the semantic vector;

[0097] The retrieval module is used for hybrid retrieval based on the user's query information and the constructed knowledge base.

[0098] It should be noted that the specific implementation process of Example 2 is similar to that of Example 1, and will not be repeated in Example 2.

[0099] Finally, it should be noted that the above-described embodiments include multiple parallel implementations of the present invention. Deleting or otherwise adjusting one or more implementations will not affect the implementation of the solution. Furthermore, the knowledge base construction method and system based on AI text parsing and hybrid retrieval disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge base construction method based on AI text analysis and hybrid retrieval, characterized by, The method includes: Acquire text data that has not yet been entered into the database, and perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not yet been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors; Based on semantic association information and document layout feature information, the text data not yet entered into the database is divided into text blocks, and the keyword index and semantic vector of each text block are determined. The knowledge base is constructed by dividing the text blocks into blocks, using the keyword index of each block, and semantic vectors. Hybrid retrieval is performed based on user query information and a constructed knowledge base. 2.The AI text parsing and hybrid search-based knowledge base construction method of claim 1, wherein, The hybrid retrieval process specifically includes: Extract keywords from the user's input query, perform keyword matching based on the keyword index, and obtain a first result set; The query statement is converted into a query vector, and vector similarity retrieval is performed based on the semantic vector to obtain a second result set. The first result set and the second result set are merged, and the merged result set is sorted based on a preset reordering optimization mechanism to output the final search results. 3.The AI text parsing and hybrid search-based knowledge base construction method of claim 1, wherein, The document layout feature extraction operation includes: The document layout of each page is identified, and visual layout element information and logical structure indication information are extracted from the document; wherein, the visual layout element information includes at least one of font format, paragraph format, bullet points, table borders, and chart positions; the logical structure indication information includes at least one of title numbering, header, footer, column break, and section break. 4.The AI text parsing and hybrid search-based knowledge base construction method of claim 1, wherein, Before performing the document layout feature extraction operation, the method further includes: Determine the text type of the texts not yet entered into the database; When the text type is a policy text, after extracting the document layout features of the policy text, the format marks representing the hierarchical structure in the policy text are identified based on the document layout feature information; Based on the format tags, text segments that conform to the clause format specifications are identified as the smallest text block units, and a tree-like hierarchical block relationship including chapters, sections, and clauses is established based on text semantic association information. 5.The AI text parsing and hybrid search-based knowledge base construction method of claim 4, wherein, After establishing a tree-like hierarchical block relationship including chapters, sections, and articles, the method further includes: For each text block of the policy document, explicit citation statements that cite other clauses within the policy document are extracted using preset citation pattern recognition rules. Based on the explicit citation statement, determine the hierarchical positioning identifier of the cited clause, and search for the corresponding target text block in the tree-like hierarchical block relationship based on the hierarchical positioning identifier; The correspondence between the hierarchical positioning identifier of the current text block and the hierarchical positioning identifier of the target text block is used as reference-related metadata and stored in association with the content, keyword index, and semantic vector of the text block. 6.The AI text parsing and hybrid search-based knowledge base construction method of claim 2, wherein, The semantic parsing operation is performed through a pre-trained semantic parsing model; When the text type is institutional text, the semantic parsing model is also used to parse the text content of the smallest text block unit into element information, and associate and store the element information with the corresponding text block; wherein, the type of element information includes applicable conditions, acting subject, normative behavior, behavioral consequences, and exceptions. 7.The AI text parsing and hybrid search-based knowledge base construction method of claim 6, wherein, The hybrid retrieval process also includes: The query statement type of the user is determined. When it is determined to be a contextual description statement, the contextual description statement is parsed, and the contextual entities and relationships corresponding to the type of element information are extracted to generate a contextual query element set. The matching degree between the contextualized query element set and the element information is calculated, and the matching degree calculation result is used as the ranking factor of the re-ranking optimization mechanism. 8.The AI text parsing and hybrid search-based knowledge base construction method of claim 7, wherein, When the text type is an institutional text, the sorting factors of the reordering optimization mechanism also include institutional effectiveness level factor, job role suitability factor, clause status and timeliness combination factor, and citation logic correlation strength factor. 9.The AI text parsing and hybrid search based knowledge base construction method of claim 8, wherein, Before performing semantic parsing, keyword extraction, and document layout feature extraction on the institutional texts that have not been entered into the database, the method further includes performing a conflict retrieval operation on the institutional texts that have not been entered into the database. The conflict retrieval operation includes: Each clause in the policy text that is not yet in the database is used as a query statement for a similarity-mixed search. Target clauses in the knowledge base with a semantic similarity to the query statement that are higher than a preset threshold are identified and highlighted to prompt users to review whether there are any potential conflicts.

10. An AI text analysis and hybrid retrieval-based knowledge base construction system, the system being implemented based on the method of any one of claims 1-9, characterized in that, The system includes: The text processing module is used to acquire text data that has not been entered into the database, and to perform semantic parsing, keyword extraction, and document layout feature extraction on the text data that has not been entered into the database; among them, the results of semantic parsing include semantic association information and semantic vectors; The text segmentation module is used to divide the unstored text data into text blocks based on semantic association information and document layout feature information, and to determine the keyword index and semantic vector of each text block. The building module is used to construct a knowledge base from the content of the divided text blocks, the keyword index of each text block, and the semantic vector; The retrieval module is used for hybrid retrieval based on the user's query information and the constructed knowledge base.

Images